Your basic ITPro blog... What's going on at work, what I'm interested in.

Thursday, June 18, 2009

BAD_ADDRESS = bad!

I was working to deploy some new IP phones on our Gilbert campus, and kept getting DHCP address assignment errors. The phones would sit there ‘configuring IP’… Just sitting there. Meanwhile, my DHCP scopes was filling up with leases to “BAD_ADDRESS”. Do a web search for “DHCP BAD_ADDRESS” and you will get a good idea of the problem.

While some reported this problem being associated with Mac clients or other IPv6 clients on the network, this was not my problem at all. My problem was simple duplicate IP addresses on the network. The tough part of this was that there were no DNS entries for the offending IP address and no valid DHCP leases for these IPs. Yet, I was able to ping the addresses, so something out there was using these addresses.

I tried using ping/arp to find the devices on the network, but did not have any success until a network engineer I was talking to suggested that go to my core router/switch on the network and do my ARP lookups on that device. I had been doing them from my workstation and a couple of edge switches. This was the key and I had struck gold. My core switch (managing all of my VLANs) had all of these IP/MAC entries in its ARP table.

From there, I was able to find the actual devices that has these BAD_ADDRESSes. This exposed the root problem that turned out to be an interesting residual from a previous issue I had worked on. It turns out that there were a number of phones on my network that were still configured to use the now-defunct IP address from our old multi-homed configuration. So, essentially, their DHCP server no longer existed. Thus, they had little choice but to hold on to their assigned IP address for dear life, hoping and praying that, someday, their long-lost DHCP server would return. Little did they know that the server was sitting right next to them, just with a new IP address. I quickly generated a list of these devices and rebooted them. They immediately found the DHCP server and got an IP address.

But, back to the BAD_ADDRESS issue… My DHCP scope had no record (no active leases) for these residual IP addresses being held by these orphaned devices. So, when I plugged a new phone in, my DHCP server was more than happy to attempt to hand those IP address out. From what I have gathered, the basic steps in DHCP go something like this (super-simplified and possibly not even right):

  • Client makes request
  • Server pulls an unused address from the appropriate scope
  • Server responds to client with this IP address and associated network configuration
  • Client verifies that IP address is actually available (not currently on the network)
    • SUCCESS! Client keeps the network configuration and is happily on the network
    • FAILURE! Client reports back to DHCP server that IP is already in use
      • DHCP adds entry in its DHCP lease DB for this IP address, assigning it to ‘BAD_ADDRESS’
      • Start process over with next available IP address

Once all devices were talking to the correct DHCP server, this problem simply went away. My new phones were immediately configured and working.

No comments:

Additional Info

My photo
email: support (AT) mangrumtech (DOT) com
mobile: 480-270-4332