Ever since I first installed my Dell PowerEdge 2900 with Virtual Server I have been experiencing issues with the Broadcom Nextreme II Gigabit adapter also known as the BCM5708C. After my latest disaster at a client with Hyper-V RTM I think it’s time to send out an alert.
In my case I the two hosts I was installing with Hyper-V ended with corrupted network stacks on Virtual Machines and hosts that initially worked flawlessly and all of a sudden caused strange error messages and in the end forced me to completely reinstall from scratch.
Here is my story:
I installed two clean HP ML370G5 Servers with Windows Server 2008 and Hyper-V. At first all installations went smooth. Installed Windows Server 2008 on both hosts. Installed Hyper-V (first RC1 later with RTM). Installed the first VM with Windows Server 2008 and IIS on one host and another VM on the other host.
Both VMs initially work fine. Then I start setting up a new VM on the first host. All of a sudden I notice that the first VM on the same host is no longer responding to the network. I have a look at the eventlog and notice that ASP.Net crashed with an unhandled exception (event ID 1309). I reboot the VM. After the reboot, the system tray shows a disconnected network icon. In the Eventlog I see that the DHCP Client service could not start due to an Access Denied event (ID 1004). Two services fail to start on the VMs:
- Base Filtering Engine
- DHCP Client
Now comes the real surprise:
I can no longer start any other VM on the host!
When I start a VM that contains a NIC (virtual or synthetic) I get the following message: "No available MAC address for ‘Network Adapter’". The message occurs before the BIOS startup of the VM.
Now I am stuck. I cannot start any VM with a Network Adapter on this host and my first VM with IIS is broken. I tried to take the VHD to another Hyper-V server, but I was never able to recover network connectivity.
When I started creating VMs on the other host I got the same errors there.
Both hosts are HP Proliant ML370G5 with dual Broadcom 5708C Nextreme II GigE network adapters connected to a 100 MB half duplex switch without VLANs. I did clean installs of Windows Server 2008 with no extra drivers or software from HP installed. After the first panic, I gave the situation some thought. That’s when I started to seriously suspect the Broadcom adapters of screwing up my game. Especially because they messed up for me before on my Dell server with the same NICs in Hyper-V Beta and RC0.
So I did another test:
- Clean install of Windows Server 2008 on one of the hosts
- Install Hyper-V RTM
- Create one virtual network connected to a Broadcom NIC
- Create the first VM with a synthetic NIC and boot from the Windows Server 2008 ISO
- Create another VM with a synthetic NIC
When I start the second VM I get the message "No available MAC address…". The message is logged in the Hyper-V-Worker Admin log with Event ID 12564, followed by event Ids 12010 and 12030.
In order to proof my theory that my issues were caused by the Broadcom Nextreme II adapters I did the following:
- I obtained an extra network adapter with a different chipset and added it to both servers
- Created a Virtual Network connected to the new NIC
- Created and started multiple VMs
This went smooth without issues. I can now run Hyper-V and I’m a happy camper!
The Broadcom Nextreme II is built-in in servers from both Dell and HP. I don’t know if other vendors are using the chipset in their systems and what other systems from Dell and HP contain them besides the Dell PowerEdge 2900/2950 and HP Proliant ML370 G5.