Zhao's Space

ESX 3.5 Update 4

zhaospace — Tue, 31 Mar 2009 18:04:00 +0000

Just surprisingly found out VMware just released ESX 3.5 Update 4 yesterday on 30 Mar 2009. A couple months back, a consultant told me that VMware ESX 3.5 Update 3 was the last VI3 updates, and the next release is going to be ESX 4, or vSphere. Well, we will have to wait for a while before ESX 4 come out. It is not like I can’t wait for ESX 4 coming out (well, maybe I do), I just wish VMware does a better job in QA. There is no need to have a buggy major release. I think I can wait.

After glancing at the release note, I found a couple things that I don’t want to use ESX 3.5U4 in my production environment. Using these will japordize my currently stable environment. Normally when VMware has new update release for ESX, the first thing I would check out is What’s New, which will contain the major change. The second thing is Known Issues, which will determine how soon I will be putting it into our production environment.

In this realse ESX 3.5U4, What’s New mainly brings out more hardware support, and Network drivers, HBA drivers updates, along with a couple newer version of OS support. What really stop me from putting ESX 3.5U4 into production is the Known Issues.

And here are the major Known Issues that I don’t like.

x64-based versions of Windows Vista and Windows Server 2008 guest operating systems require Microsoft hotfix x64-based versions of Windows Vista and Windows Server 2008 guest operating systems without Microsoft hotfix http://support.microsoft.com/kb/950772 might encounter a situation where the guest operating system stops responding and returns the following error:

MONITOR PANIC: vcpu-3:ASSERT vmcore/vmm/cpu/segment.c:430

Linux guest operating systems lose network connectivity after automatic tools upgrade

If the version of VMware tools in a Linux guest operating system becomes out of date and an automatic tools upgrade is performed, the guest operating system loses network connectivity. After an automatic tools upgrade, the guest operating system stops the network service and does not restart the service automatically after tools upgrade.

Workaround: Restart the network service in the guest operating system manually or reboot the guest operating system after an automatic tools upgrade.

The cursor disappears after VMware Tools has been configured

This issue has been observed on ESX Server 3.5 Update 4 with SUSE Linux Enterprise Server 8 (with or without SP4) guests. Immediately after you configure VMware Tools, the mouse cursor is not visible.

Workaround: Reboot the virtual machine.

Virtual Machines Might Not Power on After a Failover When the Host Is Isolated

Virtual machines might not start after a failover when a host is isolated and the isolation response is set to Guest Shutdown, which is the default configuration for a cluster. This might occur on clusters with fewer than five nodes and can happen on virtual machines
that take more time to complete the guest shutdown.

Workaround: Set the Isolation Response to either Leave powered on or Power off for clusters which have fewer than five nodes. To set the Isolation Response for a virtual machine, select the cluster, click the Edit Settings link, and select Virtual Machine Options under VMware HA. From the Isolation Response pop-up menu, select either Leave powered on or Power off options for the specific virtual machine.

There might be more dislikes in the Known Issues sections, since here is just what I have at the first glance. Another thing that make me think twice before an upgrade is the VMware tools upgrade. It is painful for Windows and Linux. Well, back to reality. All of these known issues wont’ prevent me from installing it in my lab.

Could not power on VM II

zhaospace — Mon, 30 Mar 2009 18:22:00 +0000

Here is a revisit the problem with not being able to power on a VM. In my previous post, Could not power on VM, it use a Linux command to kill the process so that the locked swap file can be release. Today I found an even easier way to fix the problem.

In my situation, I have an ESX server has some physical problem, which I haven’t figured out what caused it yet. Basically two hostd and one vpxa processes are taking up 100% usage of CPU and 95% of 800MB memory and 50% of 1.6GB swap in service console. Contacted VMware technical support, and after he tried a couple things that I did to kill the hostd processes and vpxa without success, he gave me a solution, which is to reboot the ESX server. That is right, the solution is to reboot the ESX. In VC, the ESX has a “not responding” status. I have to export a list of the VM running in this server, record what datastore these VMs reside in. After that, I disconnected the ESX, and removed it from VC. Now I need to use RDP or SSH to connect to VM, and to shutdown the VM so that I can register the VMs in another ESX and power it on. Most of the VMs are working fine. But there always one or two VMs doesn’t behave the way they are supposed to. One VM gave me an error of “Could not power on VM: No swap file. Failed to power on VM.”. When I tried to remove the file from service console, it gave me an error of “Device or resource busy”. The service console in this ESX isn’t 100% fully functioning. “ps -ef | grep VM_name” doesn’t returned anything. It means, killing the process won’t work in this case. The solution I found is to rename the VM’s vswp file.

Error: Could not power on VM: No swap file. Failed to power on VM.
Solution: Rename to vm_name-xxxxxx.vswp file name to something else, and VM can be powered on.

Here are two articles about how to kill a VM:
Gabe’s virtual world and Technoblog.

NeApp ESX Host Utilities Kit 6.0

zhaospace — Fri, 13 Mar 2009 17:45:00 +0000

NetApp is going to have an ESX Host Utilities (6.0 replease) vCenter Plugin, with release date dependent upon VMware official release of vSphere. It was NetApp’s showcase in VMworld Europe 2009.

This Host Utilities needs only a single installation to vCenter as a plugin instance, eliminates the hassle of the needs to install it into every ESX host.

Check out the video here

Improving ESX performance by unloading USB

zhaospace — Wed, 11 Mar 2009 19:13:00 +0000

Many systems have shared interrupts between the USB controller and other components. In order to reduce any performance impact due to the shared interrupts, we can configure the ESX to avoid loading the USB drivers.

Configure the USB controller setting in BIOS to enable USB, so that the system supports USB devices both during and after ESX boot process even if USB drivers are not loaded by the ESX software.

Remove the following USB modules aliases from /etc/modules.conf

alias usb-controller usb-uhci

alias usb-controller1 ehci-hcd

Save /etc/modules.conf and restart the system

This VMware KB 1290 has more detail information
on how you can tell if your system has shared interrupts.

Disabling ESX WebAccess

zhaospace — Tue, 27 Jan 2009 17:51:00 +0000

Haven’t thought about it, but it is interesting to know how to do that. Here is a discussion about it in VMware communities. Since vmware-hostd is using port 80 and 443, blocking firewall ports don’t seem to be a right way to do it. Replacing the index page looks like a right way to go.

Another ESX bug

zhaospace — Wed, 21 Jan 2009 18:14:00 +0000

I was planning to upgrade my VirtualCenter from version 2.5 Update 1 to version 2.5 Update 3, and then I received an email from VMware. This email states there is an issue with ESX 3.5 U3 could adversely effects my environment with I/O failure on SAN LUNs. I guess ESX 3.5 U3 can’t go in my environment, and I won’t be getting any benefits from VC 2.5 U3. I told my boss to hold off on this upgrade to VC 2.5 U3.

Here is the detail of this email:

ISSUE DETAILS:

VMware ESX and ESXi 3.5 U3 I/O failure on SAN LUN(s) and LUN queue is blocked indefinitely. This occurs when VMFS3 metadata updates are being done at the same time failover to an alternate path occurs for the LUN on which the VMFS3 volume resides. The effected releases are ESX 3.5 Update 3 and ESXi 3.5 U3 Embedded and Installable with both Active/Active or Active/Passive SAN arrays (Fibre Channel and iSCSI).

PROBLEM STATEMENT AND SYMPTONS:

ESX or ESXi Host may get disconnected from Virtual Center

All paths to the LUNs are in standby state

Esxcfg-rescan might take a long tome to complete or never complete (hung)

VMKernel logs show entries similar to the following:

Queue for device vml.02001600006006016086741d00c6a0bc934902dd115241 49442035 has been blocked for 6399 seconds.

Please refer to KB 1008130.

SOLUTION:

A reboot is required to clear this condition.

VMware is working on a patch to address this issue. The knowledge base article for this issue will be updated after the patch is available.

NEXT STEPS:

If you encounter this condition, please collect the following information and open an SR with VMware Support:

1. Collect a vsi dump before reboot using /usr/lib/vmware/bin/vsi_traverse.

2. Reboot the server and collect the vm-support dump.

3. Note the activities around the time where a first “blocked for xxxx seconds” message is shown in
the VMkernel.

How to check VMware tools version in a Linux VM not running X win

zhaospace — Thu, 11 Dec 2008 18:11:00 +0000

Since in a Linux VM not running X windows, we can’t use vmware-toolbox or .vmx to check the version of the VMware tools. We can check the version of VMware tools listed in vmware-config-tools.pl, although it could be the case that VMware tools got installed, but haven’t been configured yet. Run the following command to grab the version of VMware tools have been installed.

#grep buildNr /usr/bin/vmware-config-tools.pl
my $buildNr;
$buildNr = ’3.5.0 build-82663′
return remove_whitespaces($buildNr);

3.5.0 build-82663 is the VMware tools version number.

ESX intermittently disconnects from VirtualCenter

zhaospace — Tue, 18 Nov 2008 14:09:00 +0000

Error:

Sometimes an ESX host could intermittently disconnects from VirtualCenter. There are many things could cause it, but here are two common, but haven’t paying much attention places that we need to look for.

Solution:

1) Use “df -h” Check to see if / directory in ESX Service Console is filling up. After / directory is clean up and wait for a while before reconnecting ESX to VirtualCenter.

2) When seeing a message like below in /var/log/vmware/hostd.log, it is time to upgrade the memory in service console, and change its default soft and hard limit. If the memory checker reach its hard limit, bad things could happen to ESX.

/var/log/vmware/hostd.log

“[2008-11-14 22:05:13.319 'Memory checker' 20937648 warning] Current value 138020 exceeds soft limit 122880.”

“Unable to fork: Cannot allocate memory”

SC memory: With VirtualCenter 2.5 and above, it is easy to upgrade the ESX service console memory to 800MB, but it requires a reboot. See KB1003313 and KB1003501 in VMware’s website.

Soft/Hard limit: Edit the config.xml under /etc/vmware/hostd and add the following into the section:

200

250

Do a restart on hostd process by:

service mgmt-vmware restart

Creating a floppy image

zhaospace — Wed, 12 Nov 2008 19:52:00 +0000

Occasionally need to create floppy image, and an easy way to do it is to use Linux to create it. And you can mount it right there to add any files to it to.

The following command is to create a blank floppy, filled with zero.

/bin/dd if=/dev/zero of=/root/BlankFloppy.flp bs=512 count=2880

And this blank floppy can be formatted in Linux too.

/sbin/mkfs.vfat -v -c /root/BlankFloppy.flp

Mount the floppy image so that you can put files into it.

mount -o loop /root/BlankFloppy.flp /media/floppy

In Windows, you can use some Winimage, Floppyimage, undisker to create image of floppy.

Replacing O2 sensor in 2005 Honda CR-V LX

zhaospace — Wed, 16 Jul 2008 03:26:00 +0000

After the Just Brakes scam, I decided to DIY on my car if it has problems. It didn’t take long before I had a chance.

My 2005 Honda CR-V LX 2WD had an engine light came on one day. I went over to the dealership close to my house the next morning. I was told that I would be charged $99 for diagnostic to find out why the light came on, and I would have to come back again since that day’s schedule was full. I was glad that he told me to come back; otherwise I would have to come up with my own reason to give him that I need to come back later. I don’t like they charge their customers like this. I remembered my co-worker told me that AutoZone offers free diagnostic on cars. After work I bought my car over to AutoZone, and the clerk gave me a printout of code P0135, which said it was O2 sensor’s error. I came back home and get online for some research. It took me a while. I spent quite a bit to time to see if my car is still under emission warranties. Finally, I downloaded my car’s Emission Warranty from Honda Owner Link website, and it confirmed with the information I got from two dealers’ service departments and America Honda – my car’s O2 sensor (A/F sensor) has 3 year/36K miles warranty only.

In the beginning I ordered the O2 sensor for $100 from The Parts Authority in Amazon. But it just wasted my time since the one they shipped me wasn’t a right one. Calling them back and found out that they don’t know which one is for my car. Finally they told me that they don’t have it. I ended up with paying both ways shipping for nothing. Finally I placed my order with Majestic Honda at www.hondaautomotiveparts.com. They have diagrams in their website for each part they are selling. It took about a week to get to me. I went to AutoZone to rent the O2 sensor socket, anti-seize and prepared to have a hard time to take off the original O2 sensor. I ended up taking less than one minute to take it off. Putting the new one on was very easy too. The engine light kept on in that day, but it went off in the next early morning. I felt pretty good about it. This is my first time doing maintenance on my car, and I am saving about $190 replacing it myself. Honda dealer was going to charge me $400 if I let them do it, plus $99 diagnostic.