@martius - my thoughts

No doubt you all have experienced a BSOD. Often a dump file is written with information about what, how and why of the BSOD. These files can be found in C: \Windows\Minidump for Windows 7 and Windows Server 2008 R2. If not then you have to search for *. dmp in C: \Windows .

If you want to read these files you the “Debugging Tools for Windows” are required: http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx#b or http://www.microsoft.com/whdc/devtools/debugging/install64bit.mspx.

Besides these “Debugging Tools for Windows” you also need some symbols etc. These can be found here: http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx#d

Install these symbols etc at a location you can easily find, eg C: \Symbols. This path you need later to import the symbols in windbg.

Figured out what you need, I always just install everything:-D. Very easy! After installing the tools you have access to a tool called windbg.exe.

First add the symbols by opening File menu> “Symbol File Path”, specify: SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols

NO SPACES!

When you don’t specify the symbol path, windbg will show you all kinds of error saying symbols cannot be found etc. So, first specify the Symbol path!

Once the symbol path is set a DMP file can be loaded and examination of its contents can start to find out what was the reason for the BSOD.

Below an example

*********************************************************************************************************Loading Dump File [C:\minidumps\051810-13057-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are availableWARNING: Whitespace at end of path element
Symbol search path is: SRV*D:\Symbols*http://msdl.microsoft.com/download/symbolsExecutable search path is:
Windows 7 Kernel Version 7600 MP (16 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 7600.16539.amd64fre.win7_gdr.100226-1909
Machine Name:
Kernel base = 0xfffff800`01c55000 PsLoadedModuleList = 0xfffff800`01e92e50
Debug session time: Tue May 18 16:48:49.648 2010 (UTC + 1:00)
System Uptime: 0 days 0:22:41.011
Loading Kernel Symbols
………………………………………………………
……………………………………………………….
……..
Loading User Symbols
Loading unloaded module list
……….
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************Use !analyze -v to get detailed debugging information.

BugCheck A, {4, 2, 1, fffff80001cd71c1}

Probably caused by : msiscsi.sys ( msiscsi!iSpProcessWMIRequestTimeout+71e )  <<<<<<< D I S C O !!!!!

Followup: MachineOwner
———

8: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000000000000004, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, bitfield :
 bit 0 : value 0 = read operation, 1 = write operation
 bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff80001cd71c1, address which referenced memory

Debugging Details:
——————
WRITE_ADDRESS: GetPointerFromAddress: unable to read from fffff80001efd0e0
 0000000000000004

CURRENT_IRQL:  2

FAULTING_IP:
nt!IoReleaseRemoveLockEx+21
fffff800`01cd71c1 f0834304ff      lock add dword ptr [rbx+4],0FFFFFFFFh

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR:  0xA

PROCESS_NAME:  System

TRAP_FRAME:  fffff8800233d180 — (.trap 0xfffff8800233d180)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
rdx=fffffa80323dc820 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80001cd71c1 rsp=fffff8800233d310 rbp=0000000000000000
 r8=0000000000000020  r9=fffff80001fc2804 r10=fffff80001c55000
r11=fffff8800233d2e0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz ac pe cy
nt!IoReleaseRemoveLockEx+0×21:
fffff800`01cd71c1 f0834304ff      lock add dword ptr [rbx+4],0FFFFFFFFh ds:00000000`00000004=????????
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff80001cc4b69 to fffff80001cc5600

STACK_TEXT: 
fffff880`0233d038 fffff800`01cc4b69 : 00000000`0000000a 00000000`00000004 00000000`00000002 00000000`00000001 : nt!KeBugCheckEx
fffff880`0233d040 fffff800`01cc37e0 : fffffa80`3a8ed270 00000000`00000000 fffffa80`6f952b80 00000000`00000000 : nt!KiBugCheckDispatch+0×69
fffff880`0233d180 fffff800`01cd71c1 : 00000000`00010009 fffff800`01e6a5f8 fffffa80`31c1cb60 fffffa80`31c1cc68 : nt!KiPageFault+0×260
fffff880`0233d310 fffff880`055bf2b6 : fffffa80`703df010 fffffa80`703df010 00000000`00000002 fffffa80`323dc820 : nt!IoReleaseRemoveLockEx+0×21
fffff880`0233d380 fffff880`055be68b : fffffa80`3a400018 fffff880`0233d400 00000000`00000000 00000000`00000000 : msiscsi!iSpProcessWMIRequestTimeout+0x71e
fffff880`0233d420 fffff800`01cb0493 : 00000000`00000000 00000000`00000003 00000000`00000001 00000000`ffffffff : msiscsi!iSpTickHandler+0x11f
fffff880`0233d460 fffff800`01cd16a6 : 00000000`00000002 fffff880`0233d618 00000000`00000000 00000000`00000100 : nt!IopTimerDispatch+0×132
fffff880`0233d570 fffff800`01cd0a26 : fffffa80`31ecc6e8 fffffa80`31ecc6e8 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0×66
fffff880`0233d5e0 fffff800`01cd157e : 00000003`2b39f2be fffff880`0233dc58 00000000`000154cb fffff880`02318ee8 : nt!KiProcessExpiredTimerList+0xc6
fffff880`0233dc30 fffff800`01cd0d97 : 00000003`2dafbfc5 fffff880`000154cb 00000000`00000001 00000000`000000cb : nt!KiTimerExpiration+0x1be
fffff880`0233dcd0 fffff800`01ccddfa : fffff880`02315180 fffff880`023202c0 00000000`00000002 fffff800`00000000 : nt!KiRetireDpcList+0×277
fffff880`0233dd80 00000000`00000000 : fffff880`0233e000 fffff880`02338000 fffff880`0233dd40 00000000`00000000 : nt!KiIdleLoop+0x5a
STACK_COMMAND:  kb

FOLLOWUP_IP:
msiscsi!iSpProcessWMIRequestTimeout+71e
fffff880`055bf2b6 33d2            xor     edx,edx

SYMBOL_STACK_INDEX:  4

SYMBOL_NAME:  msiscsi!iSpProcessWMIRequestTimeout+71e

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: msiscsi

IMAGE_NAME:  msiscsi.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bcad7

FAILURE_BUCKET_ID:  X64_0xA_msiscsi!iSpProcessWMIRequestTimeout+71e

BUCKET_ID:  X64_0xA_msiscsi!iSpProcessWMIRequestTimeout+71e

Followup: MachineOwner
———

 *********************************************************************************************************

A quick search on the Interwebs show this result >>> “Stop error message on a Computer That is Running Windows 7 or Windows Server 2008 R2 and iSCSI storage That HAS: “0x0000000A”” with a hotfix! Jeuh! Ofcourse you have to meet the “Sympthoms” and “Applies to”.

Hopefully with this post you will find it easier to debug these DMP files created with a BSOD.

Sometime ago I ran into a WinRM problem when adding two Hyper-V R2 nodes to an existing Hyper-V R2 Failover cluster.

The errors which I got in System Center Virtual Machine Manager 2008 R2 (SCVMM) where: 

 Error (2916)

VMM is unable to complete the request. The connection to the agent hvn-srv001.domain.local was lost.

(Unknown error (0×80338126))

Recommended Action

Ensure that the WS-Management service and the agent are installed and running and that a firewall is not blocking HTTP traffic. If the error persists, reboot hvn-srv001.domain.local and then try the operation again.

Error (2927)
A Hardware Management error has occurred trying to contact server hvn-srv001.domain.local.
(Unknown error (0×80338171))
Recommended Action
Check that WinRM is installed and running on server hvn-srv001.domain.local. For more information use the command “winrm helpmsg hresult”.

This was strange because the failover cluster already consisted out of five nodes, so why these errors all of a sudden!?!

What did I check? Well actually not that much because the messages only refer to the firewall and a hardware management error when contacting the server. A way to see what “0×80338171” means by using the” winrm helpmsg 0×80338171” command. This has as result:

The WinRM client received an HTTP bad request status (400), but the remote service did not include any other information about the cause of the failure.

Okay…something is not going as it should! I tried to disable the firewall completely…with no satisfying results unfortunately! So the firewall is out for being the reason!
When searching the interwebs with the “0×80338171” error code I ended up at this article http://support.microsoft.com/kb/970875. This article is saying that because of a large Kerberos token it is possible that in some domain environments the user may be a member of so many security groups that the security token used to authenticate the user to the server may be > 16k. This is causing either http.sys or WinRM server to reject the request.

D-I-S-C-O!

Why? Due to a weird nested group construction my account had a membership of 400+ Domain Local groups which where nested by serveral Global groups! Don’t ask why! Don’t!

A script to find out the group membership count can be found here http://forums.techarena.in/active-directory/1074988.htm#post4089330

I remembered that because of this insane amount of group memberships a hotfix was applied that fixed an issue with connection via RDC to a server. Or actually, we encountered a problem when connecting to server using RDC from a Windows 7 laptop. The cause of this was a very large number of group memberships. The hotfix can be found here: http://support.microsoft.com/kb/978918. This hotfix only solved the logon issues, not the issue I encountered next when opening dsa.msc e.g. and change domain controller. This was solved my setting the MaxTokenSize to a Microsoft recommended maximum value of 65535 decimal or FFFF hexadecimal.

Name: MaxTokenSize
Type: REG_DWORD
Value: 65535
Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters

When this was added all issues seemed to be gone….except the WinRM issues.

The quick fix is to remove as many groups so that your token size is less than 16KB. Another fix is to change some WinRM related settings of the http.sys. All of this is described in http://support.microsoft.com/kb/971244 but the “Applies to” does not included any flavor of Windows Server 2008 R2. The entries that should be added to resolve the WinRM error are:

Name: MaxFieldLength
TYPE: REG_DWORD
Value: default (16384). Range (64 to 65534)
Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

Name: MaxRequestBytes
Type: REG_DWORD
Value: default (16384). Range (64 to 65534)
Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

For both entries I entered a value of 65534. Reboot the machine to make sure both http.sys and WinRM pick up the changes or restart the http and WinRM services.

After this all was fine!

All done….well no!

Bottom line…it’s all about Token size. Mine was large then the standard 16KB so Kerberos, RDP and WinRM where not functioning as they should. The reason for this was this insane high number of group membership. This was the cause of all evil and should be addressed. I resolved this by applying different folder permissions which made it possible to substitute some 230 Domain Local groups by only 1 without compromising Delegation of Control or any other security principal.

AGDLP is good…but be aware of the token size when applying!

A very simple way to see what command line options client Windows Update Agent (wuauclt.exe) has is to drop the executable in notepad. In the beginning of  the notepad (Word wrap enabled) you will find them!

In Windows 7 these options are available:

/DetectNow
/ReportNow
/RunHandlerComServer
/RunStoreAsComServer
/ShowSettingsDialog
/ResetAuthorization
/ResetEulas
/ShowWU
/ShowWindowsUpdate
/CloseWindowsUpdate
/SelfUpdateManaged
/SelfUpdateUnmanaged
/UpdateNow
/ShowWUAutoScan
/ShowFeaturedUpdates
/ShowOptions
/ShowFeaturedOptInDialog
/DemoUI 

Many of them are undocumented. What some of them mean can be found on the interwebs, others I have no clue!

So, what does this picture mean?

By default, volumes are spread across three members in one pool. So if you have six arrays in a pool, a different combination of three members will be used to handle different volumes.

There is a maximum of 16 nodes in a Hyper-V cluster which means every node in the cluster will get 6 persistent rervervations on a volume (96/16 = 6).

These 6 persistent reservations per node are then spread over a maximum of 3 members in a pool. We will then get 2 connections per member. If we apply redundancy this will result in 1 connection per NIC per Hyper-V cluster node.

If the above is correct then my previous post (http://www.delltechcenter.com/thread/4007957/Microsoft+Windows+2008+R2+CSV+and+Equallogic+SAN/ or http://www.martius.nl/?p=1896) on how to calculate the persistent reservations in wrong. This is because of the spread over max 3 members which actually means no mather how many Equallogics you are using(max. 16 in a pool) only 3 will be used to locate the volume.

The new calculation would then be:

Number of Equallogics   = 3 (max spread)
Number of NICs    = N
Number of MPIO    = M (default HIT = 2)
Number of Servers   = S

3 * N * M * S = 96

The numbers will then be:

3 * 2 * 1 * 16 = 96

Obvious question then is why should you use the DSM of Equallogic above the use of the standard Microsoft Multipath capabilities? The Equallogic DSM has a default MPIO per NIC of 2 while only one is available.

Another question is if this is the way Equallogic supports a 16 node Hyper-V cluster?

What are your thoughts? Please share them in getting this Equallogic – Persistent Reservation mystery clarified.

Here is a very simple PowerShell script to remotly reset local user passwords:

$data = import-csv C:\Scripts\ResetAdminPassword.csv

foreach ($row in $data) {

$admin=[adsi](“WinNT://” + $row.Server + “/administrator, user”)

$admin.psbase.invoke(“SetPassword”, $row.Password)

$admin.psbase.CommitChanges()

}

The CSV file is build with as top row the values “Server,Password“

Sometime ago I had an issue with a target on one of my customers servers (Windows Server 2008 Core) and an Equallogic PS5000XV. The issues was the error message below.

Event 7.4.3
ERROR
1-7-09 13:57:28
000-SAN002
iSCSI login to target ’192.168.199.13:3260, iqn.2001-05.com.equallogic:0-8a0906-bbb8ad503-b53d4d74e944a34d-000-srv666-hd00′ from initiator ’192.168.199.103:49913, iqn.1995-05.com.broadcom.iscsiboot’ failed for the following reason:
CHAP user ‘mpio-000-SRV123-0-8a0906-bbb8ad503-b53d4d74e944a34d-91d1a2′ authentication failed.

Obviously somewhere a something went wrong or a bit fell over. Anyhow…this error sucks because little info can be found on the Internet regarding this issue.

What to do?

First of all: Before you start make sure no active applications, services or Virtual Machines are using the iSCSI target on the specific server.

Remove the persistent tartget from the server with is giving the error. In my case ’192.168.199.103′.

1. Retrieve the session list with: iSCSIcli sessionlist > C:\Temp\sessionlist.txt. Below my SessionID example:
2. Look for the following info: Initiator Name, TargetName and Target Portal for the target that is reporting the issues;
3. Fill in the found info in the following iSCSICLI command: iSCSIcli RemovePersistentTarget <Initiator Name> <TargetName> * TargetPortal 3260. In my case this would be: iscsicli RemovePersistentTarget Root\ISCSIPRT\0000_0 iqn.2001-05.com.equallogic:0-8a0906-bbb8ad503-b53d4d74e944a34d-000-srv666-hd00 * 192.168.199.240 3260;
4. Log-out all existing sessions that the target is using with: iSCSIcli LogoutTarget <SessionId>. In my case this would be: iSCSIcli LougoutTarget fffffa8029687018-400001370000041d. Logout ALL found SessionID’s for the specific TargetName.
5. Open regedit.exe and make a back-up of HKEY_LOCAL_MACHINE\SOFTWARE\EqualLogic\EHCM\MpioChap
6. Search in regedit.exe on the troubled server for “‘mpio-000-SRV123-0-8a0906-bbb8ad503-b53d4d74e944a34d-91d1a2″ and delete it! Mine looked like this in an export.
7. After delting the registry key rebuild the persistent target on the troubled server to the specific target. Do this with iSCSIcli PersistentLoginTarget TargetName T * * * * * * * * * * * * * * * 0. Mine looked like this iSCSIcli PersistentLoginTarget iqn.2001-05.com.equallogic:0-8a0906-bbb8ad503-b53d4d74e944a34d-000-srv666-hd00 T * * * * * * * * * * * * * * * 0.

As you can check in your Equallogic Group Manager you will find that now for each NIC in your server which is connected to the iSCSI VLAN a Session is made.

The 192.168.199.103 and 192.168.199.104 have been reconnected. Previouslu only the 192.168.199.104 session would show up!

To start off! Out of the box Apple does not support the configuration of a wired-wireless connection to the Apple Express (AE). So this was challenge one!

The second challenge was to create a virtual machine of Windows 7 Build 7201 with the firewall enabled.

The last challenge was to create all this in a stable way so it would work “quick” and should be “easy” to set-up.

My used hardware/software is:

• Normal x64 PC with Microsoft Windows Server 2008 R2 RC
• Microsoft Hyper-V virtual machine with Windows 7 build 7201, fixed IP address, firewall enabled
• Apple Express 802.11n with firmware 7.4.2. The AE is joined to my existing network (from the Davolink) with an WPA-TKIP configuration. No MAC address filtering is used. Fixed IP address.
• Apple iTunes 8.2.0.23
• Apple Airport Utility 5.4.2
• Davolink DV-2020 WiFi AP, firmware unknown

There is a wire from the PC to the Davolink WiFi AP. The link between the Davolink link and the Apple Express is wireless (DUHHH).

After setting up all the bits and hardware I ran into some nasty errors like:

“An unknown error (-15006) occurred while connecting to the remote speaker“, “The remote speaker cannot be found on your network. Please verify your network settings and AirTunes configuration or select another speaker from the menu at the bottom of the iTunes window.” and the most nasty one (because there was no direct real error message) the loosing of sound and connection to the AE.

Than the troubleshooting started! Firewall on, firewall off, different versions of iTunes, upgrade firmware on AE, multiple reboots of all the hardware and software in many orders etc etc.

Then the EUREKA moment arrived eventually after 18 hours! What is it, what is it, what is it!!!

I found the magic sequence in how to get the sound blasting out of my speakers.

1. Setup the AE to join an existing wireless network
2. Boot all the hardware and software in a not-specific order.
3. Fire-up the Apple Airport Utility and manually connect to the AE.
4. Reboot or update the AE without making any changes!
5. Start iTunes and check if you can select the remote speakers at the right bottom
6. Stop the Bonjour service! (The holly grail!!!) but leave the start-up type at Automatic.

So….surprised?

In the end I was because the Bonjour service should keep the remote speakers discovered for eternity because that’s the whole point of the service, not to “disconnectâ€? them after an hourâ€?â€?.

This worked for me…will it work for you? Please let me know!

UPDATE:

After reading many, many, many postings on Apple forums I applied the folowing tip. This seems to make it even more stable.

Open iTunes > Edit > Preferences > Advanced Tab > Streaming Buffer Size > Set this to LARGE!

The larger the “chunks”, the lesser the posibility that set-recieve errors can occure…..sounds fine to me!

When a user is using Office 2007 on a Remote Desktop Server(RDS) (with an GPO implemented that deletes the locally created profile at logoff) and customizes the Quick Access Toolbar a file is created with the extension qat. Different qat files are made for every application in the Office 2007 suite that has a Quick Access Toolbar.

These files are saved at this locations:
XP:
C:\Documents and Settings\<username>\Local Settings\Application Data\Microsoft\Office\*.qat
Vista:
C:\Users\<username>\AppData\Local\Microsoft\Office\*.qat

The problem is that when you use Office 2007 on a RDS and implemented Roaming TS profiles which are deleted locally on the RDS at logoff these files are also deleted. Result is that users must re-customize the Quick Access Toolbar every time the logon.

Microsoft has an hotfix and an KB article for this:
http://support.microsoft.com/kb/958062
http://support.microsoft.com/kb/955142

Apply the hotfix when there are no open Office applications and add an Registry GPO preference that updates the user’s registry at logon.

Problem solved!

Realy old, but realy handy.

How to show hidden device in Windows Device Management

1. Click Start, click Run, type cmd.exe, and then press ENTER. If using Windows Server 2008 and newer, Run As Administrator.
2. Type set devmgr_show_nonpresent_devices=1, and then press ENTER.
3. Type Start DEVMGMT.MSC, and then press ENTER.
4. Click View, and then click Show Hidden Devices.
5. Expand the Network Adapters tree.
6. Right-click the dimmed network adapter, and then click Uninstall.