Backing Up the Directory
In the good old days of classic NT, you could capture the Registry hives on a single Emergency Repair Disk in a smallish domain and on the hard drive using RDISK -S if the Security Account Manager (SAM) was too big to fit on a floppy. Those days are long gone. In fact, RDISK is no longer even available.
Active Directory operations rely heavily on the Registry and many different system binaries, so you must back up and restore whole swatches of the system directory to retain integrity. Microsoft calls these files the System State. Figure 10.4 shows the list of System State files as they are displayed in NT backup. They include the following:
All files protected by the Windows File Protection system (virtually everything in C:\Windows)
The system files Ntdetect.com, Ntldr, and Bootsect.dat
The Active Directory database, Ntds.dit, and its associated log and checkpoint files
Selected Internet Explorer files and other shared files in the Program File directory
All Registry hives from the \Windows\System32\Config directory (updated copies are placed in \Windows\Repair)
The COM+ class registration database
The contents of the Sysvol directory
The Certificate service files (if installed)
The Cluster service files (if installed)
Figure 10.4. NTbackup management window showing the list of files backed up as part of System State.
Forcing you to back up all these files in one scoop does maintain consistency between the various data storehouses, but it means that if you make a minor mistake that affects a portion of the Registry out of reach of the Last Known Good Configuration rollback, you might be forced to restore all the System State files, including Active Directory.
This topic covers backing up and restoring the System State files using Ntbackup. This is a stripped-down version of Veritas Backup Exec. Third-party backup utilities are available that perform the same functions. This includes the full version of Backup Exec, Computer Associates ArcServe, Legato Networker, Tivoli Storage Manager, BEI Corporation UltraBac, CommVault Galaxy, and others. All of these vendors make backup utilities that are Windows Server 2003 compliant. Make sure you upgrade to the most current version, because a Windows 2000 backup utility may not be fully compatible with Windows Server 2003.
Why Active Directory Backups Are Necessary
You may be wondering why AD backups are required at all. After all, if you design your system correctly, you should always have at least three domain controllers in a domain. Each of these domain controllers has a full copy of Active Directory. If any of those servers crash, or even if two of them go down, you still have a third to fall back on. You can quickly promote another Windows member server to a domain controller and regain fault tolerance.
The most obvious answer to "why backup a domain controller" is that Mr. Murphy knows if you don't do backups and may arrange the universe so that all three of your domain controllers crash at the same time. Or, you may get a fire that takes your server room and you need a way to recover Active Directory at a remote site. You never want to be without a full backup in case the stars ever do align in a manner that is not in your best interests.
The more likely occurrence, though, is a mistake on the part of an administrator. I've done it. You'll do it. You get tired or distracted and you delete or modify the wrong object and well, needless to say, things can get a little ugly. If you accidentally delete an entire OU with thousands of user and computer and group objects in it, your mistake will merrily replicate to all the other domain controllers. The only way you'll get that information back is from tape.
You do not need to back up all your domain controllers. Select two or three from your domain and backup System State on those machines. Don't run applications on the domain controllers that would require them to be backed up by local administrators.
Performing a System State Backup
A System State backup is always performed with the machine online and operating normally. The Plug-and-Play Manager should find your tape drive and install the proper drivers for it. You may need to use the Removable Storage Manager to juggle your media pools. See Chapter 21, "Recovering from System Failures," for details.
To perform a System State backup using Ntbackup, follow Procedure 10.6.
Procedure 10.6 Performing a System State Backup
Mount a fresh tape in the tape drive.
Start Windows 2000 Backup via START | PROGRAMS | ACCESSORIES | SYSTEM TOOLS | BACKUP. The Backup window opens.
Click the Advanced button to avoid the backup wizard.
Select the Backup tab.
Expand the tree to reveal the System State option, which is the last option under the drive letters.
Place a check next to the System State option.
Under Backup Destination, select the tape unit you are using for the backup.
If you are backing up to a file, select File and then enter the path and filename under Backup Media or File Name. Be sure to save the files on reliable removable media or on a network drive that is sure to be available if you need to recover. Don't back up to the local hard drive.
From the menu, select TOOLS | OPTIONS. The Options window opens.
Select the Backup Type tab.
Under Default Backup Type, select the Copy option. This keeps the system from resetting the archive bit so that you'll be sure to capture the System State files on your normal nightly backup.
Select the Backup Log tab.
Select the Detailed radio button. This gives lots of information, including the name of every file included in the backup. Normally this gives too much information to be useful, but it is worth doing at least once to see exactly what is included in the System State backup.
Click OK to save the changes and return to the main Backup window.
We want to run this backup job interactively rather than scheduling it, so click Start Backup.
If this is the first time you've used this tape, you may get a message stating that There is no unused media with the selected type, but unrecognized media is available. You might also be prompted to confirm the overwrite of existing data if the tape is recognized. If you're sure you haven't inserted a tape with valuable data on it, click OK. Different tape systems have different formatting and retrieval schemes, so the messages may vary. Refer to your system documentation for specific instructions. Ntbackup uses the Microsoft Tape Format (MTF).
When the system can see the tape or the file location if you are using removable media, the backup begins. The Backup Progress window opens to show you what's happening.
When the backup is complete, you get a final window showing the statistics. The Status field will probably state that files were skipped. Click Report to view the backup log. The most likely culprit is the DO_NOT_REMOVE_NtFrs_Preinstall_Directory under Sysvol. This folder holds files that were copied as part of the initial migration of files and folders into the File Replication System database. It can be skipped.
Review the list of files in the log to see what is included. Note that the Active Directory files do not appear individually on the backup log. There is just a single line entry indicating that Active Directory was included in the backup.
The Registry files are first copied to a special folder called \Registry from their normal location in \WindowsWindows\System32\Config. The \Registry folder is a temporary, volatile folder used to capture a snapshot of the Registry for backup. This prevents a file lock during backup from blocking access to the Registry.
Restoring Active Directory
There are two basic scenarios requiring a recovery of the Active Directory database:
A complete loss of a domain controller necessitating a full recovery from tape, including System State.
A mistake on the part of an administrator that results in loss of or damage to a portion of Active Directory.
Although any Windows Server 2003 compliant backup utility can do a satisfactory backup and restore of System State, it is often impractical to do a quick restore. For example, you might have a big tape library with dozens or hundreds of servers in the backup catalog. Doing an across-the-wire restore of a large Ntds.dit file along with the other System State files can take time, and often requires repermissioning the agent on the domain controller because Active Directory is not available to authenticate the service.
To speed up the recovery, many organizations schedule a file-based backup of the System State files on their domain controller prior to doing the nightly tape backup. This puts a set of *.bkf files on a remote server where they can be captured to tape. Recovering a single file to a landing pad where it can be transferred to the domain controller for restoration is often simpler than doing the restore directly from tape.
In the first scenario, the tape restore of the System State files (which includes Active Directory) is done in conjunction with the restore of the operating system. When the newly restored domain controller comes back online, its replication partners update it with the changes that have occurred since the tape backup was run. For this reason, you should be able to restore from a relatively old tape (three or four days, even a week old) just in case you have problems with your tape library that prevent you from using last night's backup.
Directory Services Restore Mode Password
When restoring Active Directory at a domain controller, you must be able to log on to the server using the Administrator account in the local SAM. You may not know the password for this account.
To remedy this problem, Microsoft included a new feature in the Ntdsutil utility for resetting the local Administrator password. This feature is called Reset DSRM Administrator Password. If you have administrator privileges on a domain controller, you can use this feature to change the Administrator account in the SAM of the domain controller.
Ordinarily, non-administrators are not permitted to log on locally at a domain controller. If a non-administrator does succeed in getting access to a domain controller console and attempts to reset the DSRM password, this is the result:
Reset DSRM Administrator Password: reset pas on ser dotnet-rc-es
Please type password for DS Restore Mode Administrator Account:
Setting password failed.
WIN32 Error Code: 0x6ba
Error Message: The RPC server is unavailable.
A standard System State restoration does "recover" objects you have deleted from Active Directory. When the newly restored domain controller replicates from its partners, any undeleted objects are promptly moved back to the Deleted Objects container and all your hard work goes for naught.
If you need to restore a portion of Active Directory (a single object, for instance, or an accidentally deleted OU), or the entire database in the event that it has become irresolutely corrupted, you must ensure that the objects and their properties on the tape replicate outward to the other domain controllers. This is accomplished with an authoritative restore.
An authoritative restore is not a restoration as such. It is performed after the actual tape restoration to mark the objects that will overwrite the replicas on other domain controllers. It does this by adding 100000 to the Property Version Number (PVN) of each property on the authoritatively restored objects.
The following is a REPADMIN listing showing how a sample object looks before and after an authoritative restore. The ObjectClass attribute is not touched by the authoritative restore because it is a special attribute used for indexing:
Loc.USN Originating DSA Org.USN Org.Time/Date Ver Attribute
======= =============== ======= ============= ====== =========
2562 Houston\HOU-DC-01 2562 2001-10-23 21:21.29 1 objectClass
3112 Houston\HOU-DC-01 3112 2001-10-24 09:29.12 2 cn
3112 Houston\HOU-DC-01 3112 2001-10-24 09:29.12 2 sn
2562 Houston\HOU-DC-01 2562 2001-10-23 21:21.29 1 description
2562 Houston\HOU-DC-01 2562 2001-10-23 21:21.29 1 givenName
2562 Houston\HOU-DC-01 2562 2001-10-23 21:21.29 1 instanceType
Loc.USN Originating DSA Org.USN Org.Time/Date Ver Attribute
======= =============== ======= ============= ====== =========
2562 Houston\HOU-DC-01 2562 2001-10-23 21:21.29 1 objectClass
28905 Houston\HOU-DC-01 28905 2001-10-29 19:59.32 100002 cn
28905 Houston\HOU-DC-01 28905 2001-10-29 19:59.32 100002 sn
28905 Houston\HOU-DC-01 28905 2001-10-29 19:59.32 100001 description
28905 Houston\HOU-DC-01 28905 2001-10-29 19:59.32 100001 givenName
28905 Houston\HOU-DC-01 28905 2001-10-29 19:59.32 100001 instanceType
Use great caution when doing an authoritative restore of the Configuration naming context or of the entire Active Directory database. Make sure you understand the consequences of overwriting any system configuration parameters. This is especially true if you are running Exchange 2000 or other Active Directory-dependent applications. It is not necessary to restore the entire database to get back a single OU. Be selective.
Who Can Restore Active Directory
If you have a distributed IT organization with administrators in many regions, you may wonder who can perform a Directory restore. You may want to make sure that only qualified individuals do a tape restore on a domain controller. You may also want to find out if you can recover from a mistake without calling up the central IT staffers at headquarters to pull you out of a jam.
Performing a System State restore (which includes the Active Directory database) requires having the local Administrator password in the SAM on the domain controller. This is called the Directory Service Recovery password. If you know this password, you can also perform an authoritative restore of any or all of the Active Directory database, regardless of whether you have rights to a particular portion of the Directory tree. This is because the Active Directory service is not running when you do the authoritative restore, so the system has no way of collecting or validating your credentials.
The Schema naming context cannot be authoritatively restored. If you want to recover a previous copy of the schema, you can do a standard System State restore on the Schema Master. Any changes made to the schema after the tape backup are lost.
For instance, let's say you install a new network monitoring application that makes changes to the Active Directory schema. You find a bug in the program that didn't show up in lab testing and you want to remove it from the schema. You should delete the new classes and attributes individually using the Schema Management console. You could also do a System State restore at the Schema Master using a tape taken before you installed the application. A System State restore done at the Schema Master will overwrite the schema for the entire forest.
Keep this operation of the Schema Master in mind when you do tape restores. If you do not want to restore the schema, do your System State restore on another domain controller.
You should guard the local Administrator password on domain controllers closely and change it frequently, but be aware that utilities such as ERD Commander permit anyone with physical access to a domain controller to set the local Administrator password to get full access. You should take the following actions to guard your Active Directory database:
Never give anyone but highly trusted administrators the right to back up or restore System State on domain controllers.
Only back up the System State on a few selected domain controllers. The administrators performing this backup should be among the most trusted administrators in the organization.
Do not permit physical access to domain controllers in remote locations. You should make these headless servers without floppy or CD drives.
Explain to other administrators that domain controllers are like routers. They are appliances controlled by a certain cadre of administrators.
Time Limit for Using Backup Tapes
It happens in every IT organization at one time or another. You need to do a tape restore and last night's tapes are either unreadable or the backup didn't run. You go back into the library and find that some of the tapes are so old you can see daylight though the oxide layer. And some of the good tapes got jammed up in the library robot assembly and didn't actually get anything written to them. You will eventually find a mountable tape, but it might be a week or two old. You then wonder if you can safely restore Active Directory from that tape.
The good news is that you can restore System State from a somewhat older tape without compromising Active Directory. As soon as you finish the System State restoration and restart the domain controller, the restored copy of Active Directory will obtain updates from its replication partners and get caught up to date. If you perform an authoritative restore, on the other hand, you will lose any changes such as password resets, group membership changes, new domain trusts, and so forth that happened in the meantime. Be very careful when using an older tape for an authoritative restore.
This ability to use an older copy of Active Directory has two limits. The first limit is tied to the tombstone lifetime for deleted objects in Active Directory.
When an AD object is deleted, the ESE engine does a little Dante number on it. The object is stripped of all but a couple of attributes and hurled into the depths of the Deleted Objects container where it sits and rots for 60 days. At the end of 60 daysЧthe standard tombstone lifetimeЧthe object is removed completely from the database by the garbage collection process.
This means that you must not restore System State (and by inference, Active Directory) from a tape that is more than 60 days old. Let me give an example to show the risk.
Let's say you delete an object from AD and then perform a System State restore from last night's backup. As long as you do not do an authoritative restore of that object, it will be redeleted as soon as the domain controller pulls updates from its replication partners. The updates say, in effect, "This object is in my Deleted Objects container. Move your copy of the object to the Deleted Objects container, as well."
But, if you restore from a tape that is more than 60 days old, the replication partners will have removed some of the tombstoned objects from the Deleted Objects container. They then have no way to tell the newly restored domain controller to delete the old objects. The newly restored go into limbo. They are deleted on all replicas of the naming context except for the restored machine. This corrupts Active Directory.
Keep this 60-day window in mind if you have disaster recovery procedures that could potentially introduce an older copy of Active Directory into your system. For instance, do not make images of domain controllers for use in disaster recovery. The CD that holds the image stops being your best friend and starts being your worst nightmare on day 61 after it was burned. If you want to use imaging as a recovery strategy, take an image before you promote a server to a domain controller. Restore the image then promote the machine to a domain controller. This replicates a fresh copy of the Directory from another domain controller.
If you lose all copies of Active Directory during a catastrophe of some sort, your only alternative is to restore from the most recent tape you can find and deal with any changes that have been made in the interim, such as password changes or group membership changes.
The 60-day interval for tombstone lifetime and the 12-hour garbage collection interval are default settings in Active Directory. You can change these default values by adding attributes to the cn=Directory Service, cn=Windows NT,cn=Services,cn=Configuration,dc=<domain_name>,dc=<root> object. The two attributes are as follows:
Use great caution when changing the tombstone lifetime. The value must be long enough to accommodate using an older System State tape backup. It is very important that you never restore Active Directory from a tape that is older than the tombstone lifetime.
Active Directory Restores and Trust Relationships
A second time limit also affects your ability to do System State restores from older tapes. If you have external trust relationships to other forests or NT domains, these trust relationships have passwords that change every 7 days. When a new password is negotiated, the old password is retained. For this reason, if you restore from a tape that is older than 14 days, you will need to verify the external trust relationships and force a renegotiation of the password. In the worst case, you must remove and re-install the trust.
Authoritative Restoration and Sysvol
The File Replication Service, or FRS, replicates the contents of Sysvol among all domain controllers in a domain. If you authoritatively restore the Group Policy Container (GPC) objects in Active Directory without returning Sysvol to the condition that existed when the System State backup was obtained, you risk a mismatch between the GPC objects in AD and the GPT files in Sysvol.
There is no such thing as an authoritative restore of Sysvol. Instead, you must restore a copy of the System State files, including the Sysvol files, to an alternative location and then copy them over the Sysvol files after you restart following an authoritative restore of the AD. To see how it works, take a look at Procedure 10.7.
Procedure 10.7 Performing an Authoritative Restore of Sysvol
Perform a standard System State restore to the original location. Do not restart.
Do a second System State restore to an alternate location. This retains the older Sysvol files so you can use them following restart.
Perform an authoritative restore of the AD objects or containers you want to recover. This list should include group policy related objects, otherwise there is no reason to include Sysvol in your recovery plans.
Restart the domain controller and ensure that the Sysvol contents get updated from the replication partners. This is very important. The contents of Sysvol must fully converge on all domain controllers before proceeding.
At the domain controller where you did the authoritative restore, copy the contents of Sysvol from the alternate location to the standard Sysvol location. This overwrites the existing files. You can also delete the Policies and the Scripts folders and copy them from the alternate location if you want to remove any GPT files.
Wait for the contents of Sysvol to converge on all domain controllers.
Verify that clients receive the correct group policies.
Restoring System State
When you restore the System State files, you overwrite the existing files on the machine. Many of these files are databases that are normally locked when the server is operating. This includes the Active Directory database, the Registry hives, the Certificate database, and others.
To do a System State restoration, then, you must boot the machine into a configuration where these databases are not running. This is done using Directory Service Restore Mode, one of the safe mode options in Windows. Select this option from a menu by pressing F8 at the standard boot menu. If you do not see a boot menu, press F8 as soon as the machine completes POST. This puts the keystroke in the keyboard buffer where it will be seen when NTLDR is activated.
The Directory Service Restore Mode option loads the network drivers so you can do a tape restore across the network but does not start Active Directory and other critical databases. This mode also sets the environment variable SAFEBOOT_OPTION to DSREPAIR. This variable must be set or Ntdsutil will not perform an authoritative restore.
Because Active Directory is not available when booting to Directory Services Restore Mode, you must provide the password for the Administrator account in the local SAM. This is the password you entered when you promoted the server to be a domain controller.
If you forget the Restore Mode password, or if you inherit a server and you do not know the password that was originally entered during DCPROMO, you can change the password using a utility from SysInternals (www.sysinternals.com) called ERD Commander. There are two versions of this utility. The free version does not allow you to change the Administrator password. The for-fee version does.
You might want to do what I do to remember these sorts of passwords. I jot them down on a slip of paper and lock them inside the server case. I also tape up a screen print of the Disk Management console inside the case, as well, if another administrator needs to know the original partition layout if the machine crashes.
When you have logged on to the Safe mode console, perform a System State restoration as shown in Procedure 10.8.
Procedure 10.8 Performing a System State Restore
At the domain controller where you want to restore Active Directory, start the system in Directory Services Repair mode.
Insert the backup tape with the System State files.
Open the Run window and enter Ntbackup.
Select the Restore tab.
Expand the tree to show the drive and list of previously submitted jobs. When you expand the tree at the tape icon, you might need to wait a while for the tape to rewind. If you have a catalog on the disk, the system will compare the signature in the catalog to the signature on the tape. If you do not have a catalog on the disk, you will need to catalog the tape. Right-click the media and select CATALOG.
In the tree displayed by the tape catalog, place a check next to the System State option.
Click Restore Now. You are prompted with a message warning that the System State files will always overwrite the existing files unless restoring to an alternate location. I do not recommend restoring to an alternate location because too many files and too many variables are involved if you try to copy them by hand. Hold your breath and enable the system to overwrite where it wants.
Click Yes to acknowledge the message and proceed. The Restore Progress window opens and the tape drive or removable media begins responding.
When the restore is complete, click Report and review the log to make sure that all files were restored without error.
Close Notepad and return to the Restore Progress window. Click Close to finish the restore. You are prompted to shut down the system. Do not restart. Instead, click No to bypass the restart option and close the Backup program.
At this point, you have completed the System State restoration. If you do not need to restore individual objects or containers, you can restart the domain controller and let it replicate any changes that have occurred since the backup was run. If you want to restore individual components of the Directory, proceed to the next section.
Performing an Authoritative Directory Restoration
Because the authoritatively restored properties have higher PVNs than the replicas on other domain controllers, the properties replicate outward and overwrite other replicas. For this reason, an authoritative restore can result in lots of replication traffic. Schedule it for after hours and pay particular attention to domain controllers at the wrong end of slow WAN links. When you're ready to start, follow Procedure 10.9.
Procedure 10.9 Performing an Authoritative Restore
Open a command session.
From the Ntdsutil: prompt, enter authoritative restore. This opens the authoritative restore: prompt:
authoritative restore: ?
? - Show this help information
Help - Show this help information
Quit - Return to the prior menu
Restore database - Authoritatively restore entire database
Restore database verinc %d - ... and override version increase
Restore object %s - Authoritatively restore an object
Restore object %s verinc %d - ... and override version increase
Restore subtree %s - Authoritatively restore a subtree
Restore subtree %s verinc %d - ... and override version increase
Enter the option corresponding to the item you want to restore. You'll need to know the distinguished name of the item. For instance, if you want to restore a single object, enter restore object cn=BigBoss,ou=Executives,dc=Company,dc=com.
A message window opens prompting you to verify the action. Click Yes to begin the authoritative restore. An example output looks like this:
Opening DIT database...............Done.
The current time is 2001-10-29 19:59.32.
Most recent database update occurred at 2001-10-29 14:27.52.
Increasing version numbers by 100000.
Counting records that need updating...
Records found: 000001
Found 000001 records to update.
Successfully updated 000001 records.
Authoritative Restore completed successfully.
If the restore does not complete successfully, you may have a corrupted database. Try doing a soft restore and then a hard repair (as described later in this chapter). Next, retry the authoritative restore. If it still refuses to work, try restoring from tape one more time. Next, try restoring at a different domain controller. Next, call Microsoft Support Services for advice.
Quit out of Ntdsutil and restart the domain controller. When it comes back online, manually force replication.
When the changes converge at all the domain controllers, you should see the restored objects in their original locations.