Some of you may have read by previous post on Mirroring with TFS at http://blogs.msdn.com/sudhir/archive/2006/10/30/using-sql-mirroring-with-tfs.aspx. This post is for Orcas or VS2008 release.
TFS leverages SQL Server for storing data. This allows our customer to leverage various failover options that come with SQL Server. SQL Mirroring is a great option. You can use clustering if you need automatic failover but if a small downtime is acceptable Mirroring is a good solution. TFS does not support automatic failover with mirroring. Few manual steps need to be executed when the primary server fails with mirroring in TFS. Following are the high level steps required to use mirroring.
The process of configuring a TFS deployment for mirroring can be defined in following steps
- Backup databases from the main Data Tier Server(Principal server)
- Restore backed up databases on mirror server.
- Setup Mirroring for each database between Principal and Mirrored servers
Perform manual Failover to enable Mirrored Server as principal server.
- Manual failover when primary server is available can be performed using Management studio
Manual failover when primary server is unavailable(Forced service Failover) can be performed using ALTER DATABASE command in TSQL Statements.
ALTER DATABASE database_name SET PARTNER FAILOVER, where database_name is the mirrored database.
- Map Application Tier to talk to new Data Tier(Mirror Server) using TFSAdminUtil RenameDT
Now you have the databases active but we store some messages in sysmessages table in Master DB and we also have few SQL jobs which will not get created by just restoring the databases. To restore these you have couple of options
- You can run repair on Application Tier machine which will fix the Data tier
You can run TFSDB utility which is in Tools folder for TFS. This is quicker and much faster.
TfsDb.exe repair /server:<servername> /property: “TFS_SERVICE_ACCOUNT=<TFSServiceAC>; TFS_REPORTING_ACCOUNT=<TFSReportingAC>; LCID=1033; VSTF_AS_INSTANCE=<ASInstance>; VSTF_AS_DATABASE=TfsWarehouse;VSTF_AS_ACCOUNT=”
TFS Databases: TfsBuild, TfsIntegration, TfsVersionControl, TfsWarehouse, TfsWorkItemTracking, TfsWorkItemTrackingAttachments, TfsActivityLogging
Team Foundation Server 2008 can be deployed in different configurations. In 2008 release we have made the product very flexible from deployment perspective. That means now you will have many components to think about in fail over or redundancy. In this post I will focus on TFS only components for SQL Reporting services and WSS you can follow on line MSDN docs for TFS or SQL RS/WSS.
TFS can be deployed in Single server configuration or dual server configuration. We do not provide any fail over options in single server configuration (MSDN Doc: http://msdn2.microsoft.com/en-us/library/ms316473(vs.90).aspx)
In dual server scenario you have 2 machines Application Tier and Data Tier:
Data Tier: All TFS data is stored in SQL Server. Therefore you can use any of the standard SQL fail over or high availability options. You can use Clustering, SQL Mirroring or Log shipping. The failover in case of Mirroring and Log Shipping cannot be automatic. We need to perform couple of steps before mirrored databases are activated. I will have posts on Mirroring and Log shipping coming soon.
Application Tier: TFS provides warm standby Application Tier support. If you run a TFS Orcas install and if databases are already present and mapped to different application tier we automatically setup a Warm Standby server for you. Warm standby will be non functional server deployment of TFS. If your primary application tier server fails you can use TFSAdminUtil ActivateAt to activate your Standby Server.
Hope this information was helpful.
TFS can be deployed in single server or dual server deployment. Dual server is when your application tier and data tier are on 2 different physical machines. Single server deployment is when everything is on same machine.
My last 3 posts covered recovery in various scenarios of dual server deployment. Let’s take a look at single server deployment now. What happens if the machine hosting TFS in single server configuration dies? Following are steps you can follow to recover in that situation. All this information is only round TFS and does not include SQL Reporting services and SharePoint, if you need details on everything you should access online MSDN documents.
- Reinstall TFS on single server machine. You can use the install guide to do this. At end of this step you should have a running TFS single server configuration.
- Install any service packs or patches you had installed on the crashed machine
- Now recover the databases from your backup
- If your machine name remained same you should be ready to go.
- If you changed your machine name you need to rum TFSAdminUtil with RenameDT and ActivateAT options
In my last post I blogged about recovering from a crashed AT. TFS also supports a warm standby of AT. This helps you fail over to the warm standby AT when the primary AT fails. This is really great as it reduces the recovery time. You can setup a warm standby by running TFS 2008 setup and provide existing DT machine as input. TFS setup will detect the primary AT is different machine and automatically install a warm standby for you.
If you have warm standby and your primary AT machine crashes you can activate the standby AT and make it the primary AT by running TFSAdminUtil command line tool with ActivateAt option.
- TfsAdminUtil ActivateAt <standby server>
Team Foundation Server can be deployed in dual server environment with Application Tier and Data Tier on 2 different servers. In my last post I wrote about recovering from a DT crash. Now let’s look at how we can recover from crash of AT machine.
In case of AT crash, the easiest way to recover is to reinstall TFS 2008. The best part about our TFS 2008 Setup is you can reinstall TFS with existing databases without any problems.
Steps to perform:
- Install Pre-requisites as defined in Install Guide
- Run TFS Install and provide existing database server as input.
Team Foundation server can be setup in dual server environment with application tier running on one machine and the data tier running on different machine. We use SQL Server as our backend and we support named instances for SQL Server in VS2008. We also recommend taking regular backups of your TFS databases. In VS2008 you can use existing WSS and SQL Reporting Services. This post is going to summarize the steps you can take to recover just the TFS part of data.
In VS 2008 unlike 2005 we just have 1 SKU for all types of install. You can install TFS on single server or dual server from 1 setup that runs on the application tier machine. In case your Data Tier machine crashes you can follow following steps to recover your environment.
Now your system should be ready for use. If you changed the name of the DT server while recovering the new system you will have to run TFSAdminUtil renameDT command.
Let’s start this post with good old Murphy’s Law “Anything that can go wrong will go wrong”. Therefore we need to figure out how we can fix things once they have gone wrong. In TFS there are few scenarios that come to mind when you think about disaster recovery
- Recover from Data Tier crash
- Recover from Application Tier Crash
- Recover from Application Tier Crash when you have Warm Standby
- Recover from whole system crash(AT & DT on same machine)
I feel there are few areas of TFS deployment which don’t have enough guidance or documentation. We are always working on improving the documents but I want to take a stab at these areas and provide some guidance on these. Please give me feedback on other areas you would like to know more about. The areas are
- Disaster recovery
- Redundancy or Fail Over
- Move: There are MSDN documents available on this topic. I will cover the high level options in this blog but for more details you should follow MSDN docs.
We have put in lot of efforts in Orcas. There were many features we thought we would not be able to achieve but we finally squeezed them in. One such feature which customers have been asking for is flexibility with how we use SQL Reporting Services. Many customers want to move SQL Reporting services to other machines that TFS AT or use existing Reporting infrastructure. We have added lot of flexibility in how you use Reporting Services. First of all we were not able to get all the changes required into the UI so you will have to fall back to the MSIProperty.ini file. Sorry about that.
- Open MSIProperty.ini from InstallMedia\AT
- Change following properties to set Reporting information
- VSTF_RS_REPORTS_URI=http://<reportserver>/<somepath or reports>
- VSTF_RS_REPORTSERVER_URI= http://<reportserver>/<somepath or reportserver>
- Run setup and install TFS
Scenario we have tested:
- Running Reporting Services on DT machine. This was not supported in Whidbey and now this is something we have made sure it works.
- Running Reporting services on machine anywhere on the network.
- Running Reporting services and WSS on remote machine
To configure reporting on remote box we make WMI calls that require us to have remote admin rights on the machine. I hope this is not a big problem to our customers. I know in large enterprises all ports and access rights are secured and this maybe an issue. I hope customers can work around this problem.