@martius - my thoughts

Sometime ago I ran into a WinRM problem when adding two Hyper-V R2 nodes to an existing Hyper-V R2 Failover cluster.

The errors which I got in System Center Virtual Machine Manager 2008 R2 (SCVMM) where: 

 Error (2916)

VMM is unable to complete the request. The connection to the agent hvn-srv001.domain.local was lost.

(Unknown error (0×80338126))

Recommended Action

Ensure that the WS-Management service and the agent are installed and running and that a firewall is not blocking HTTP traffic. If the error persists, reboot hvn-srv001.domain.local and then try the operation again.

Error (2927)
A Hardware Management error has occurred trying to contact server hvn-srv001.domain.local.
(Unknown error (0×80338171))
Recommended Action
Check that WinRM is installed and running on server hvn-srv001.domain.local. For more information use the command “winrm helpmsg hresult”.

This was strange because the failover cluster already consisted out of five nodes, so why these errors all of a sudden!?!

What did I check? Well actually not that much because the messages only refer to the firewall and a hardware management error when contacting the server. A way to see what “0×80338171” means by using the” winrm helpmsg 0×80338171” command. This has as result:

The WinRM client received an HTTP bad request status (400), but the remote service did not include any other information about the cause of the failure.

Okay…something is not going as it should! I tried to disable the firewall completely…with no satisfying results unfortunately! So the firewall is out for being the reason!
When searching the interwebs with the “0×80338171” error code I ended up at this article http://support.microsoft.com/kb/970875. This article is saying that because of a large Kerberos token it is possible that in some domain environments the user may be a member of so many security groups that the security token used to authenticate the user to the server may be > 16k. This is causing either http.sys or WinRM server to reject the request.

D-I-S-C-O!

Why? Due to a weird nested group construction my account had a membership of 400+ Domain Local groups which where nested by serveral Global groups! Don’t ask why! Don’t!

A script to find out the group membership count can be found here http://forums.techarena.in/active-directory/1074988.htm#post4089330

I remembered that because of this insane amount of group memberships a hotfix was applied that fixed an issue with connection via RDC to a server. Or actually, we encountered a problem when connecting to server using RDC from a Windows 7 laptop. The cause of this was a very large number of group memberships. The hotfix can be found here: http://support.microsoft.com/kb/978918. This hotfix only solved the logon issues, not the issue I encountered next when opening dsa.msc e.g. and change domain controller. This was solved my setting the MaxTokenSize to a Microsoft recommended maximum value of 65535 decimal or FFFF hexadecimal.

Name: MaxTokenSize
Type: REG_DWORD
Value: 65535
Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters

When this was added all issues seemed to be gone….except the WinRM issues.

The quick fix is to remove as many groups so that your token size is less than 16KB. Another fix is to change some WinRM related settings of the http.sys. All of this is described in http://support.microsoft.com/kb/971244 but the “Applies to” does not included any flavor of Windows Server 2008 R2. The entries that should be added to resolve the WinRM error are:

Name: MaxFieldLength
TYPE: REG_DWORD
Value: default (16384). Range (64 to 65534)
Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

Name: MaxRequestBytes
Type: REG_DWORD
Value: default (16384). Range (64 to 65534)
Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

For both entries I entered a value of 65534. Reboot the machine to make sure both http.sys and WinRM pick up the changes or restart the http and WinRM services.

After this all was fine!

All done….well no!

Bottom line…it’s all about Token size. Mine was large then the standard 16KB so Kerberos, RDP and WinRM where not functioning as they should. The reason for this was this insane high number of group membership. This was the cause of all evil and should be addressed. I resolved this by applying different folder permissions which made it possible to substitute some 230 Domain Local groups by only 1 without compromising Delegation of Control or any other security principal.

AGDLP is good…but be aware of the token size when applying!