I ran into an interesting issue this evening where the Spooler service on a customer's print server was crashing constantly immediately after logging an event about unpublishing a specific printer. Since this was a Windows 2000 machine, troubleshooting this was a bit more involved than usual on 2003.
In general, you troubleshoot these issues by collecting a dump at the time of the process crash with adplus. Adplus is a VBScript available in the Debugging Tools for Windows package. To start, you'll need the PID of the Spooler process – you can get this from Task Manager. Look for spoolsv.exe, if the PID is missing, go to View>Select Columns to get it. The syntax to attach adplus to the process and wait for it to crash goes like this: cscript adplus.vbs –quiet –crash –p pid.
If you're running under Windows 2000 (or NT4), there is an extra step if you're also connecting to the problem server via Remote Desktop. The extra step here involves another tool in the Debugging Tools for Windows package called remote.exe. The steps are:
- Create a batch file on the problem server, let's call it remotecmd.cmd, and save it in c:\debuggers. In the batch file, put this line in c:\debuggers\remote.exe /s "cmd.exe" remoteshell.
- Use AT to schedule the batch file to run in one minute. If the current server time is 23:15, run this command AT 23:16 c:\debuggers\remotecmd.cmd.
- Wait for remote.exe to spawn the command prompt (you'll see remote.exe running in task manager).
- Connect from another machine to the remote. Run remote.exe /c ProblemServer remoteshell. Replace ProblemServer with the name of the machine having the issues and running the remote session.
- From the remote shell, launch adplus - cscript adplus.vbs –quiet –crash –p pid.
ADPlus will create a folder inside c:\debuggers (or wherever you launch it from). The folder will be named Crash_Mode__Date…. Inside of here there will be a number of files including a couple of dumps. There will be mini-dumps and full ones, grab the full dump, probably named similar to PID-7996__SPOOLSV.EXE__2nd_chance_AccessViolation__full_1710_2009-01-04_21-30-10-290_1f3c.
Open the dump using WinDbg (Ctrl+D), and if you don't have symbols configured, issue these two commands:
- .symfix c:\symbols
- .reload
Chances are the !analyze –v debugger will be able to pinpoint the correct thread and display the stack where the crash occurred. In my case this was the stack:
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0182f430 01db23e8 000db6d8 000dad90 000d2b1c zsdzplui!BlankSheetProc+0x71e
0182f49c 7780564d 00000000 0182f4d0 00000000 zsdzplui!BlankSheetProc+0x766
0182f4b4 77805797 00000000 0182f4d0 00000000 winspool!DocumentPropertySheets+0x126
0182f4f4 7614fc1c 00000000 000d2b1c 016261b8 winspool!DocumentPropertiesW+0xdc
0182f524 7614cfd2 016261b8 10000000 016260a0 localspl!ThisIsAColorPrinter+0x32
0182ff30 7614ed5d 01648778 016260a0 016260a0 localspl!UpdateDsDriverKey+0x2bb
0182ff58 7614144c 0182ffac 01648778 7c57b400 localspl!DsUpdatePrinter+0x142
0182ff6c 7614edd0 00cd2130 0182ffac 7614ebfd localspl!RunForEachPrinter+0x2c
0182ff7c 761413fa 0182ffac 00cd2130 7c57b400 localspl!DsUpdateSpooler+0x19
0182ff90 7614eec0 0182ffac 7614edb7 00070000 localspl!RunForEachSpooler+0x21
0182ffb4 7c57b3bc 0162b360 00070000 00070748 localspl!DsUpdate+0xec
0182ffec 00000000 7614edd6 0162b360 00000000 KERNEL32!BaseThreadStart+0x52
The important piece of information here is the module at the top of the stack (zsdzplui) which is a third party print driver. You can use the lmvm command to get more information:
0:012> lmvm zsdzplui
start end module name
01db0000 01dbe000 zsdzplui (export symbols) zsdzplui.dll
Loaded symbol image file: zsdzplui.dll
Image path: C:\WINNT\system32\spool\drivers\w32x86\2\zsdzplui.dll
Image name: zsdzplui.dll
Timestamp: Fri Dec 30 05:05:36 2005 (43B51480)
CheckSum: 0001B3AB
ImageSize: 0000E000
File version: 1.0.0.40
Product version: 1.0.0.40
File flags: 2 (Mask 3F) Pre-release
File OS: 10001 DOS Win16
File type: 3.2 Driver
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Number Five Software
ProductName: ZUD Printer Driver
InternalName: ZUDUI
OriginalFilename: zsdui.dll
ProductVersion: 1, 0, 0, 40
FileVersion: 1, 0, 0, 40
PrivateBuild: 1, 0, 0, 40
SpecialBuild: 1, 0, 0, 40
FileDescription: ZUD Printer Driver UI
LegalCopyright: Copyright © Number Five Software 2001-2005
LegalTrademarks: Copyright © Number Five Software 2001-2005
Comments: Copyright © Number Five Software 2001-2005
The first thing to notice here is that this driver is three years old – that's really old. Now unfortunately there's no real easy way for most folks to figure out which printer driver this file corresponds to, especially on a busy print server. It's at least somewhat simpler if you know what make of printer to start looking under – in this case that's not obvious either. The good news is that I happen to know that with this particular driver you should be looking under Zebra, so if this is your exact problem, part of the work is done. If it's a different vendor, a couple tricks for doing this at home:
- Go in the Printers Control Panel, right click and select Server Properties. Go to the Drivers tab and start pulling the Properties of the installed drivers – the files the driver uses will be listed.
- Open up Regedit and browse to HKLM\System\CurrentControlSet\Control\Print\Environments. Search on the DLL in question – it will be listed under here somewhere and you can get the name of the driver from the parent key.
Once you've identified the driver in question (in our case it happens to be a Zebra 170XiIII), obtain an update from the manufacturer and see if that helps.
In the case of the problem I had, the spooler service was crashing constantly and thus nobody would be able to print. Based on the stack it was obvious that something was going wrong when the Spooler was trying to publish this printer in Active Directory. What's happening is that quite a bit of printer information is published in AD and in order to collect some of that data, Windows has to ask the print driver to provide it.
The obvious solution was of course to go in the Printers Control Panel, and access the properties of each printer using the problem driver, and uncheck List in the Directory. The problem is that Explorer came back with an error and refused t load the printer properties.
So what to do in this situation? There's a tool called setprinter.exe in the Windows 2003 Resource Kit Tools download. This particular tool lets you mess with the properties of printers in an unintuitive but direct way. I ran the command below on each printer using the problem driver to just disable the publishing all together for those printers. This band-aid'ed the problem such that the spooler could continue the publishing sequence for each printer on the server and stay up.
C:\Program Files\Windows Resource Kits\Tools>setprinter \\ProblemServer\BustedPrinter 7 "dwAction=unpublish"
Set printer on '\\ProblemServer\BustedPrinter' succeeded (DS operation pending).