HP WSD printer port type screws up Windows Server 2012 domain controllers!

No response from serverI don’t think this can be that uncommon a scenario: a Windows Server 2008 R2 domain, with mainly HP printers. New domain controller added (at new site), this time running Windows Server 2012 R2; HP printers there too.

This was the position I found myself in earlier this year. On paper, there’s nothing unusual about this set-up. Adding new 2012 DCs and standard HP workgroup printers shouldn’t be a problem. That’s what we all thought.

Until the domain controller started becoming non-responsive.

Cue many, many hours on TechNet and various other similar sites, chasing down what I became increasingly sure must be some latent fundamental corruption in Active Directory (horrors!), revealed only by the introduction of the newer o/s. There were many intermediate hypotheses. At one point, we thought maybe it was because we were running a single DC (and it was lonely). Or that the DC was not powerful enough for its file serving and DFS replication duties. So I provisioned a second DC. Ultimately I failed all services over to that because the first DC was needing increasingly frequent reboots.

And then the second domain controller developed the same symptom.

Apart from the intermittent loss of replication and certain other domain duties, the most obvious symptom was that the domain controller could no longer initiate DNS queries from a command prompt. Regardless of which DNS server you queried. Observe:

C:\Users\rob>nslookup bbc.com
Server: UnKnown
Address: 192.168.1.1

*** UnKnown can’t find bbc.com: No response from server

C:\Users\rob>nslookup bbc.com 192.168.1.2
Server: UnKnown
Address: 192.168.1.2

*** UnKnown can’t find bbc.com: No response from server

C:\Users\rob>nslookup bbc.com 8.8.8.8
Server: UnKnown
Address: 8.8.8.8


*** UnKnown can't find bbc.com: No response from server

Bonkers, right? Half the time, restarting AD services (which in turn restarts file replication, Kerberos KDC, intersite messaging and DNS) brought things back to life. Half the time it didn’t, and a reboot was needed. Even more bonkers, querying the DNS server on the failing domain controller worked, from any other machine. DNS server was working, but the resolver wasn’t (so it seemed).

I couldn’t figure it out. Fed up, I turned to a different gremlin – something I’d coincidentally noticed in the System event log a couple of weeks back.

Ephemeral port exhaustion

Event ID 4266, with the ominous message “A request to allocate an ephemeral port number from the global UDP port space has failed due to all such ports being in use.”

What the blazes is an ephemeral port? I’m just a lowly Enterprise Architect. Don’t come at me with your networking mumbo jumbo.

Oh wait, hang on a minute. Out of UDP ports? DNS, that’s UDP, right?

With the penny slowly dropping, I turned back to the command line. netstat -anob lists all current TCP/IP connections, including the name of the executable (if any) associated to the connection. When I dumped this to a file I quickly noticed literally hundreds of lines like this:

TCP 0.0.0.0:64838 0.0.0.0:0 LISTENING 4244
[spoolsv.exe]

As it happened, this bit of research coincided with the domain controller being in its crippled state. So I restarted the Print Spooler service, experimentally. Lo and behold, the problem goes away. Now we’re getting somewhere.

Clearly something in the printer subsystem is grabbing lots of ports. Another bell rang – I recalled when installing printers on these new domain controllers that instead of TCP/IP ports, I ended up with WSD ports.

WSD ports

What on earth is a WSD port?! (Etc.)

So these WSD ports are a bit like the Bonjour service, enabling computers to discover services advertised on the network. Not at all relevant to a typical Active Directory managed workspace, where printers are deployed through Group Policy. WSD ports (technically monitors, not ports) are however the default for many new printer installations, in Windows 8 and Server 2012. And as far as I can tell, they have no place in an enterprise environment.

Anyway, to cut a long story short (no, I didn’t did I, this is still a long story, sorry!), I changed all the WSD ports to TCP/IP ports. The problem has gone away. Just like that.

I spent countless hours trying to fix these domain controllers. I’m now off to brick the windows* at Microsoft and HP corporate headquarters.

Hope this saves someone somewhere the same pain I experienced.

Peace out.

*Joke

Easily host your own web sites | part 3: networking

Contents


network cablesSo far in this series, we’ve set up our server hardware and installed and configured the operating system and web hosting application. In today’s article, we’ll look at how you enable people to browse content on your server (wherever they and it may be).

Port forwarding

Most SOHO routers are configured to prevent unsolicited inbound traffic. If you want people to be able to view web sites on your server therefore, you need to create a conduit through which web traffic can pass. This is usually called “port forwarding”. Broadly speaking, ports segregate different types of network traffic. Web traffic comes in on port 80 (TCP), SSH traffic on port 22 (TCP), encrypted web traffic (HTTPS) on port 443 (TCP) and so on. So you need to configure your router to forward port 80 traffic to the static IP address you configured for your server (see part 2). You may also wish to forward port 10000 (Webmin) and port 22 (SSH) so you can administer your server remotely, but be warned that as soon as you do, so-called “script kiddies” will start trying to break into your server. So proceed with caution, and read up on the risks. You’re pretty safe forwarding port 80, provided your web sites are well designed.

To find out how to set up port forwarding on your router, read the manual, or check out the references at portforward.com. Once you’ve done that, check that the port is forwarded by browsing to your network’s external IP address. How do you find that out? From within the network, browse to icanhazip.com.

If you’ve forwarded the port correctly, and if you’re not caught by the hairpin NAT gotcha (see below), you should see the default web page from your new web server. On my new server, the page looks like this:

It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

Dynamic vs. Static IP addresses

If you are going to be hosting any kind of internet service (like a web server), it is easiest to have a static IP address. DNS (the Domain Name System) converts “human-friendly” web addresses such as “pomeroy.me” into “computer-friendly” IP addresses such as “178.248.39.218”. This is great if your network’s external IP address never changes, but for many customers on cable, broadband or dial-up connections, their IP address is “dynamic”, meaning that it might be different each time the modem or router reboots. There is a workaround for dynamic IP addresses (see the section on Dynamic DNS, below), but in the long run, it might be easiest simply to ask your ISP for a static IP address. In many cases that costs little or no more than a dynamic address.

Dynamic DNS

With Dynamic DNS (DDNS), your router or server contacts a DDNS service provider on the internet and reports its current external IP address. The DDNS server then transmits this information through the internet via the DNS system. Changes can take a while to take effect, but it’s better than nothing. My current DDNS provider of choice is changeip.com. ChangeIP.com has a free service, where you use a sub-domain from one of several domains they have available (e.g. fredbloggs.changeip.org).

Many routers can speak DDNS, meaning that as soon as the router reboots, it can check in with the DDNS provider and let it know if the IP address has changed. See your router’s manual for more information. If your router doesn’t directly support this, you can install software on your server that will periodically update the DDNS service. Read ChangeIP.com’s page on DDNS clients. You’ll need to install one of the scripts on your server, and set up a cron job to run the script as often as you like (providing you don’t breach the DDNS provider’s terms of service). You can of course use Webmin to set up your cron job.

Regular DNS

If you have your own domain registered, like me, and you’re using a static IP address, all you need to do is point your domain to your external IP address. Log into your domain host and add “A records” as needed.

Note: Did you know that the customary “www.” prefix on many websites is an unnecessary hangover from earlier in the internet’s history? You can use www.mydomain.com if you like, or simply mydomain.com. Note however that when you set up a “mydomain.com” web site using Virtualmin, Virtualmin automatically makes a “www.mydomain.com” alias, so that either will resolve to the same web site code. If you want DNS to work with both, you’ll need to add two A records: one for “@” and one for “www”.

Hairpin NAT

For most people, that’s it. You should now be able to create web sites on your server, use DNS or DDNS to broadcast those web sites to the outside world, and receive visitors from all around the world. But there’s a very significant “gotcha” with some routers, when you’re attempting to browse your web site from your LAN. If you’re on the same physical network as your web server, when you enter the web address in your browser, it should attempt to browse to your network’s external IP address. In other words, your web query goes out and attempts to come back in again. This out-and-back-again behaviour is known as “hairpin NAT” and not all routers can handle it.

If you’re one of the unlucky ones and you don’t want to swap your router for a better one, you have two main options. Technically, the best option is to run your own private DNS server (which you can do on the same server), but that is complicated and a bit of a hassle. If you’re only browsing from a single PC, the quick-and-dirty solution is to edit your hosts file. This file is located somewhere like “C:\Windows\System32\drivers\etc\hosts”. There are a lot of resources on the internet that will explain to you more about this file, but let me help you over one little hurdle: if you’re using Windows Vista or later, you need to edit this file as an Administrator. To do that, right-click Notepad and choose “Run as Administrator”. Then open the hosts file from within Notepad. You’ll need to change the files filter to “*.* (All files)”.

The default hosts file contains details about the format of entries. Suffice it to say that you’ll put here the DNS name of your web site and the internal LAN IP address of your web server. Your web browser, when looking up an IP address, should check the hosts file first, so in this scenario, you won’t be going out and back again. If you then take that computer somewhere else (e.g. it’s a laptop and you’ve taken it to work), you’ll need to remove the relevant hosts file entries, in order to browse to the external IP address of your network.


Networking is a complicated area and I have only scratched the surface here, in an effort to get most people up and running quickly. By all means if you’re stuck at a particular point, post a comment and, time permitting, I’ll try to point you in the right direction. In the meantime, stay tuned for part 4 of this series in which I will provide some initial pointers on building your own web site.

Cables image copyright © Pascal Charest, licensed under Creative Commons. Used with permission.