Global Catalog Server could not be located: All GC's down

[guest]

when ever I try to add a new user into our active directory i get the following error " windows cannot verify that the user name is unique because the following error occurred while trying to contact the global catalog
server: The server is not operational ..."

When I check in the ntds settings properties there is a tick on the global catalog.

when I do a dcdiag i get the following

Domain Controller Diagnosis

Performing initial setup:
* Verifying that the local machine nwscdc2, is a DC.
* Connecting to directory service on server nwscdc2.
* Collecting site info.
* Identifying all servers.
* Identifying all NC cross-refs.
* Found 2 DC(s). Testing 2 of them.
Done gathering initial info.

Doing initial required tests

Testing server: Default-First-Site-Name\NWSCDC
Starting test: Connectivity
* Active Directory LDAP Services Check
* Active Directory RPC Services Check
The clock difference between the home server NWSCDC2 and target
server

NWSCDC is greater than one minute. This may cause Kerberos

authentication failures. Please check that the time service is
working

properly. You may need to resynchonize the time between these
servers.

......................... NWSCDC passed test Connectivity

Testing server: Default-First-Site-Name\NWSCDC2
Starting test: Connectivity
* Active Directory LDAP Services Check
* Active Directory RPC Services Check
......................... NWSCDC2 passed test Connectivity

Doing primary tests

Testing server: Default-First-Site-Name\NWSCDC
Starting test: Replications
* Replications Check
* Replication Latency Check
DC=ForestDnsZones,DC=nwsc,DC=co,DC=ug
Latency information for 4 entries in the vector were ignored.
4 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
DC=DomainDnsZones,DC=nwsc,DC=co,DC=ug
Latency information for 4 entries in the vector were ignored.
4 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
CN=Schema,CN=Configuration,DC=nwsc,DC=co,DC=ug
Latency information for 9 entries in the vector were ignored.
9 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
CN=Configuration,DC=nwsc,DC=co,DC=ug
Latency information for 9 entries in the vector were ignored.
9 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
DC=nwsc,DC=co,DC=ug
Latency information for 8 entries in the vector were ignored.
8 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
* Replication Site Latency Check
......................... NWSCDC passed test Replications
Starting test: Topology
* Configuration Topology Integrity Check
[Topology Integrity Check,NWSCDC] Intra-site topology generation is
disabled in this site.
* Analyzing the connection topology for
DC=ForestDnsZones,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the connection topology for
DC=DomainDnsZones,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the connection topology for
CN=Schema,CN=Configuration,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the connection topology for
CN=Configuration,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the connection topology for DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
......................... NWSCDC passed test Topology
Starting test: CutoffServers
* Configuration Topology Aliveness Check
* Analyzing the alive system replication topology for
DC=ForestDnsZones,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the alive system replication topology for
DC=DomainDnsZones,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the alive system replication topology for
CN=Schema,CN=Configuration,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the alive system replication topology for
CN=Configuration,DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
* Analyzing the alive system replication topology for
DC=nwsc,DC=co,DC=ug.
* Performing upstream (of target) analysis.
* Performing downstream (of target) analysis.
......................... NWSCDC passed test CutoffServers
Starting test: NCSecDesc
* Security Permissions Check for
DC=ForestDnsZones,DC=nwsc,DC=co,DC=ug
(NDNC,Version 2)
* Security Permissions Check for
DC=DomainDnsZones,DC=nwsc,DC=co,DC=ug
(NDNC,Version 2)
* Security Permissions Check for
CN=Schema,CN=Configuration,DC=nwsc,DC=co,DC=ug
(Schema,Version 2)
* Security Permissions Check for
CN=Configuration,DC=nwsc,DC=co,DC=ug
(Configuration,Version 2)
* Security Permissions Check for
DC=nwsc,DC=co,DC=ug
(Domain,Version 2)
......................... NWSCDC passed test NCSecDesc
Starting test: NetLogons
* Network Logons Privileges Check
......................... NWSCDC passed test NetLogons
Starting test: Advertising
The DC NWSCDC is advertising itself as a DC and having a DS.
The DC NWSCDC is advertising as an LDAP server
The DC NWSCDC is advertising as having a writeable directory
The DC NWSCDC is advertising as a Key Distribution Center
Warning: NWSCDC is not advertising as a time server.
Warning: NWSCDC has not finished promoting to be a GC.
Check the event log for domains that cannot be replicated.
Warning: NWSCDC is not advertising as a global catalog.
Check that server finished GC promotion.
Check the event log on server that enough source replicas for the
GC are available.
......................... NWSCDC failed test Advertising
Starting test: KnowsOfRoleHolders
Role Schema Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role Domain Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role PDC Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role Rid Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role Infrastructure Update Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
......................... NWSCDC passed test KnowsOfRoleHolders
Starting test: RidManager
* Available RID Pool for the Domain is 6606 to 1073741823
* nwscdc2.nwsc.co.ug is the RID Master
* DsBind with RID Master was successful
* rIDAllocationPool is 6106 to 6605
* rIDPreviousAllocationPool is 6106 to 6605
* rIDNextRID: 6158
......................... NWSCDC passed test RidManager
Starting test: MachineAccount
* SPN found :LDAP/nwscdc.nwsc.co.ug/nwsc.co.ug
* SPN found :LDAP/nwscdc.nwsc.co.ug
* SPN found :LDAP/NWSCDC
* SPN found :LDAP/nwscdc.nwsc.co.ug/NWSC
* SPN found
:LDAP/db891d27-9bea-4ae8-a1ef-6819076984e0._msdcs.nwsc.co.ug
* SPN found
:E3514235-4B06-11D1-AB04-00C04FC2DCD2/db891d27-9bea-4ae8-a1ef-6819076984e0/nwsc.co.ug
* SPN found :HOST/nwscdc.nwsc.co.ug/nwsc.co.ug
* SPN found :HOST/nwscdc.nwsc.co.ug
* SPN found :HOST/NWSCDC
* SPN found :HOST/nwscdc.nwsc.co.ug/NWSC
* SPN found :GC/nwscdc.nwsc.co.ug/nwsc.co.ug
......................... NWSCDC passed test MachineAccount
Starting test: Services
* Checking Service: Dnscache
* Checking Service: NtFrs
* Checking Service: IsmServ
* Checking Service: kdc
* Checking Service: SamSs
* Checking Service: LanmanServer
* Checking Service: LanmanWorkstation
* Checking Service: RpcSs
* Checking Service: w32time
* Checking Service: NETLOGON
......................... NWSCDC passed test Services
Starting test: OutboundSecureChannels
* The Outbound Secure Channels test
** Did not run Outbound Secure Channels test
because /testdomain: was not entered
......................... NWSCDC passed test OutboundSecureChannels
Starting test: ObjectsReplicated
NWSCDC is in domain DC=nwsc,DC=co,DC=ug
Checking for CN=NWSCDC,OU=Domain Controllers,DC=nwsc,DC=co,DC=ug in
domain DC=nwsc,DC=co,DC=ug on 2 servers
Object is up-to-date on all servers.
Checking for CN=NTDS
Settings,CN=NWSCDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
in domain CN=Configuration,DC=nwsc,DC=co,DC=ug on 2 servers
Object is up-to-date on all servers.
......................... NWSCDC passed test ObjectsReplicated
Starting test: frssysvol
* The File Replication Service SYSVOL ready test
File Replication Service's SYSVOL is ready
......................... NWSCDC passed test frssysvol
Starting test: frsevent
* The File Replication Service Event log test
......................... NWSCDC passed test frsevent
Starting test: kccevent
* The KCC Event log test
An Information Event occured. EventID: 0x40000617
Time Generated: 04/11/2008 13:58:19
(Event String could not be retrieved)
An Information Event occured. EventID: 0x4000062A
Time Generated: 04/11/2008 13:58:19
(Event String could not be retrieved)
An Information Event occured. EventID: 0x40000456
Time Generated: 04/11/2008 13:58:19
(Event String could not be retrieved)
An Error Event occured. EventID: 0xC0000466
Time Generated: 04/11/2008 13:58:53
(Event String could not be retrieved)
......................... NWSCDC failed test kccevent
Starting test: systemlog
* The System Event log test
An Error Event occured. EventID: 0x00000010
Time Generated: 04/11/2008 13:42:43
(Event String could not be retrieved)
......................... NWSCDC failed test systemlog


Testing server: Default-First-Site-Name\NWSCDC2
Starting test: Replications
* Replications Check
* Replication Latency Check
DC=ForestDnsZones,DC=nwsc,DC=co,DC=ug
Latency information for 4 entries in the vector were ignored.
4 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
DC=DomainDnsZones,DC=nwsc,DC=co,DC=ug
Latency information for 4 entries in the vector were ignored.
4 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
CN=Schema,CN=Configuration,DC=nwsc,DC=co,DC=ug
Latency information for 9 entries in the vector were ignored.
9 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
CN=Configuration,DC=nwsc,DC=co,DC=ug
Latency information for 9 entries in the vector were ignored.
9 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
DC=nwsc,DC=co,DC=ug
Latency information for 8 entries in the vector were ignored.
8 were retired Invocations. 0 were either: read-only
replicas and are not verifiably latent, or dc's no longer replicating this
nc. 0 had no latency information (Win2K DC).
* Replication Site Latency Check
......................... NWSCDC2 passed test Replications

Starting test: NetLogons
* Network Logons Privileges Check
......................... NWSCDC2 passed test NetLogons
Starting test: Advertising
The DC NWSCDC2 is advertising itself as a DC and having a DS.
The DC NWSCDC2 is advertising as an LDAP server
The DC NWSCDC2 is advertising as having a writeable directory
The DC NWSCDC2 is advertising as a Key Distribution Center
The DC NWSCDC2 is advertising as a time server
Warning: NWSCDC2 has not finished promoting to be a GC.
Check the event log for domains that cannot be replicated.
Warning: NWSCDC2 is not advertising as a global catalog.
Check that server finished GC promotion.
Check the event log on server that enough source replicas for the
GC are available.
......................... NWSCDC2 failed test Advertising
Starting test: KnowsOfRoleHolders
Role Schema Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role Domain Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role PDC Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role Rid Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
Role Infrastructure Update Owner = CN=NTDS
Settings,CN=NWSCDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=nwsc,DC=co,DC=ug
......................... NWSCDC2 passed test KnowsOfRoleHolders
Starting test: RidManager
* Available RID Pool for the Domain is 6606 to 1073741823
* nwscdc2.nwsc.co.ug is the RID Master
* DsBind with RID Master was successful
* rIDAllocationPool is 5606 to 6105
* rIDPreviousAllocationPool is 5606 to 6105
* rIDNextRID: 5797
......................... NWSCDC2 passed test RidManager
Starting test: MachineAccount
* SPN found :LDAP/nwscdc2.nwsc.co.ug/nwsc.co.ug
* SPN found :LDAP/nwscdc2.nwsc.co.ug
* SPN found :LDAP/NWSCDC2
* SPN found :LDAP/nwscdc2.nwsc.co.ug/NWSC
* SPN found
:LDAP/7ea18a42-4c90-4ef9-bf5e-cbf19374b3fb._msdcs.nwsc.co.ug
* SPN found
:E3514235-4B06-11D1-AB04-00C04FC2DCD2/7ea18a42-4c90-4ef9-bf5e-cbf19374b3fb/nwsc.co.ug
* SPN found :HOST/nwscdc2.nwsc.co.ug/nwsc.co.ug
* SPN found :HOST/nwscdc2.nwsc.co.ug
* SPN found :HOST/NWSCDC2
* SPN found :HOST/nwscdc2.nwsc.co.ug/NWSC
* SPN found :GC/nwscdc2.nwsc.co.ug/nwsc.co.ug
......................... NWSCDC2 passed test MachineAccount
Starting test: Services
* Checking Service: Dnscache
* Checking Service: NtFrs
* Checking Service: IsmServ
* Checking Service: kdc
* Checking Service: SamSs
* Checking Service: LanmanServer
* Checking Service: LanmanWorkstation
* Checking Service: RpcSs
* Checking Service: w32time
* Checking Service: NETLOGON
......................... NWSCDC2 passed test Services

Starting test: frsevent
* The File Replication Service Event log test
There are warning or error events within the last 24 hours after the

SYSVOL has been shared. Failing SYSVOL replication problems may
cause

Group Policy problems.
An Warning Event occured. EventID: 0x800034C4
Time Generated: 04/11/2008 13:24:11
(Event String could not be retrieved)



Running partition tests on : DomainDnsZones
Starting test: CrossRefValidation
......................... DomainDnsZones passed test
CrossRefValidation
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom

Running partition tests on : Schema
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom

Running partition tests on : Configuration
Starting test: CrossRefValidation
......................... Configuration passed test
CrossRefValidation
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom

Running partition tests on : nwsc
Starting test: CrossRefValidation
......................... nwsc passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... nwsc passed test CheckSDRefDom

Running enterprise tests on : nwsc.co.ug
Starting test: Intersite
Skipping site Default-First-Site-Name, this site is outside the scope

provided by the command line arguments provided.
......................... nwsc.co.ug passed test Intersite
Starting test: FsmoCheck
Warning: DcGetDcName(GC_SERVER_REQUIRED) call failed, error 1355
A Global Catalog Server could not be located - All GC's are down.
PDC Name: \\nwscdc2.nwsc.co.ug
Locator Flags: 0xe00003f9
Time Server Name: \\nwscdc2.nwsc.co.ug
Locator Flags: 0xe00003f9
Preferred Time Server Name: \\nwscdc2.nwsc.co.ug
Locator Flags: 0xe00003f9
KDC Name: \\nwscdc2.nwsc.co.ug
Locator Flags: 0xe00003f9
......................... nwsc.co.ug failed test FsmoCheck

[guest]

1) unedited ipconfig /all from all DCs is below

ip config /all for Domain Controller NWSCDC:

Windows IP Configuration

Host Name . . . . . . . . . . . . : nwscdc
Primary Dns Suffix . . . . . . . : nwsc.co.ug
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : nwsc.co.ug
co.ug

Ethernet adapter Local Area Connection 2:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : BCM5703 Gigabit Ethernet
Physical Address. . . . . . . . . : 00-0B-CD-4E-15-04
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : 10.0.0.11
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . : 10.0.0.6
DNS Servers . . . . . . . . . . . : 10.0.0.12
10.0.0.11

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : 3Com EtherLink XL 10/100 PCI For
Complete
PC Management NIC (3C905C-TX)
Physical Address. . . . . . . . . : 00-04-76-D6-C3-07
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Autoconfiguration IP Address. . . : 169.254.55.17
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . :

ip config /all for Domain Controller NWSCDC2:

Windows IP Configuration

Host Name . . . . . . . . . . . . : nwscdc2
Primary Dns Suffix . . . . . . . : nwsc.co.ug
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : nwsc.co.ug
co.ug

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : HP NC7781 Gigabit Server Adapter
Physical Address. . . . . . . . . : 00-17-08-54-EE-39
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : 10.0.0.12
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . : 10.0.0.6
DNS Servers . . . . . . . . . . . : 10.0.0.12
10.0.0.11

2) The firewall is not enabled on any of the domain controllers

3) Did I change anything before this occurred?
I have just inherited this network but from what i gathered from the
previous admin, the DC with the GC role collapsed and there was no chance to
recover it. Not sure if this is the correct info.

4) How many sites do I have: I have 1 site

Have synchronized the clocks using the net time command, but still get the GC
error, thanks

[guest]

Check that your DC's have in total the 5 FSMO roles. You talk about an old
crashed DC's Are there any entries in AD or DNS from that machine?

[guest]

I think a few other things should also be looked at:
DNS - somebody else had asked about DNS and if the crashed DC is still left
as a remnant. When a DC won't become a GC, the roles and other dc's in the
environment are something you really start to wonder about. I've noticed
that you also have dual-homed dc's - that's a bad thing, but at least at
this point the secondary nic seems to be disconnected. You should check to
see that it wasn't connected in the past and registered an address.

While on the subject of NICs, I notice you have the broadcom nics. Those
have some known issues with older drivers that may need to be updated when
done here.

You may also want to run dcdiag with the /v switch to help you see more
information.

When you get it all sorted out, you may also want to revisit the dns
settings of you DC's - they are a single point of failure at the moment. :)

Roles, resolution, and remnants. Those are three things to focus on for now
though, so that you do not confuse while troubleshooting.

[guest]

I actually used the ntdsutil command / metadata cleanup and identified that
there was a subdomain DC=kw,DC=nwsc,DC=co,DC=ug. The domain controller server
for this subdomain was the one taken offline some time before. Apparently, it
was taken off without due process.

Because of this the global catalog server could not come up, as it was
always waiting for the KCC to up it with a readonly copy of the directory
partition for this subdomain.

I removed this subdomain details using ntdsutil.metadata cleanup, and now
the global catalog is up and working fine.

I would like to thank all of you for the support you have rendered. Am
grateful.

[guest]

Yes, ntdsutil.metadata cleanup fixes my problem too.