Archive for the ‘ESX’ Category

Many people think the key to the virtual data centre is about getting as many virtual machines (VMs) on a physical server as possible. It is still the case, however, that prohibitive memory costs are a roadblock to high consolidation ratios, and virtualisation availability tools still lag behind business need. They have not yet cancelled out the “eggs-in-one-basket” scenario that virtualisation endemically brings with it.

Read more…

In this article, we eschew the normal blue-sky 2010 technology predictions for something a bit more everyday that will affect your daily virtual life.

Scale up, up, up and away
Firstly, it’s no surprise that by mid-year there is likely to be a major rerelease of vSphere4 with a strong emphasis on increased scalability. Building on top of vSphere4’s current scalability I wouldn’t be surprised to see the number of vCPUs a single ESX host can support go beyond the 128 core range. I think it’s likely that by the end of 2010 or the beginning of 2011 we will be looking at more than 8 vCPUs to a VM – with VMware pushing the amount of RAM per-VM into the 512 GB to 1 TB range and the ESX host supporting 1 TB or 2 TB of physical RAM.

Read on…

In case you don’t know Scott is a VMware employee – he focuses on performance and scalability – and taking on Citrix/Microsoft at the how-got-the-best-product game…

Anyway, Scott’s a good resource for information (in fact some of it quite technical, but written in an accessible way) on performance.

Performance of Thin Provisioned Disks

This article debunks the popularly held myth that thin provisioning = poor performance. It doesn’t, so long as the underlying storage and VMFS is configured correctly. Actually, the performance is comparable to the old thick format. Don’t about you – but I’d rather be thin, than thick any day.

Additionally, my pal and fellow blogger – Steve Beaver has some interesting (and some would say more guarded) thoughts on thin-provisioning.

http://www.thevirtualblackhole.com/vwire/some-thoughts-on-thin-provisioning

esxtop Analysis With esxplot

This one allows you to download a graphing tool that take the raw date from an esxtop -b output – and doesn’t suffer from the column and row limitations of conventional systems – such as Microsoft Excel

Following from my popular (according to me!) blog post theme – of “what I learned this week…” a irregular round-up of stuff that seems really obivious – but actually has escaped me.

vCenter DB Retention Policy:

A couple of weeks ago I made a point about how the vCenter DB grows in size day-by-day and week-by-week – and that wasn’t away to control it size or FIFO its data. I turned out I was wrong. If you look in the vCenter Server Settings. In vCenter 4.0 there is actually something called the retention policy.

What I was hoping for was the ability to FIFO the performance data. As far as I know the only way to clear performance data is with SQL Transact commands to delete the redundant data.

VMware Tools won’t upgrade:

I’ve a couple of VMs who’s VMware Tools won’t upgrade. I think it maybe because the uninstall data was deleted – erm, by me. Opps. Anyway, I think I might have found a work around in this KB article.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001354

ESXi lacks RPMs for VMware Tools for Redhat Linux 64-bit:

This is little bit of an obscure one – and it was drawn to my attention by one of my students. If you create a Red Hat Linux 64-bit (RHEL5) on an ESX ‘Classic’ host you will find you have both the ability to install VMware Tools with either an .RPM or extract it from a .TGZ file. NOW, if you try the SAME thing on ESXi host you will find the RPM version isn’t there. Presumably this is done by VMware to reduce the footprint of ESXi.

Find out if your virtual disks are the right type for VMware Fault Tolerance:

This one came thru twitter. Specifically from William Lam. Thanks William this is great find. William has a great resource of scripts on his website:

http://www.engr.ucsb.edu/~duonglt/vmware/

My previous method of finding this out was pretty lame. It involved using vmkfstools and vmkernel log files – and looking at very obscure values therein. This is MUCH easier; MUCH clearer, and MUCH simpler.

Why is this important. Well, let say your VM has the ordinary thick type of disk (technical term is zeroedthick) or you have used the thin type – AND these disks are quite large (let say 2TB). To convert these disks the Fault Tolerance format (eagerzeroedthick format) requires a power off of the VM, and it can take a long time. That’s got to be factored into your maintenance window.

Of course none of this matter if the vSphere Client/vCenter actually showed the correct format. Unfortunately, both the zeroedthick and eagerzeroedthick format are both referred to as “thick” in the GUI.

Here’s how you do it.

Just cat & grep the last vmware.log file of the VM which you can see in the directory where the VM is located. So here’s a VM with three types of disk. Thin (scsi0:0), EagerZeroedthick (scsi0:1) and Thick (scsi0:2). In the screen grab you can see scsi(0:0) is thin

If you then PuTTy to a ESX host, and the type the following command:

[root@esx1 mail03]# cat vmware-1.log  | grep  ‘FT enable’

This will produce the output of:

Dec 02 00:38:08.505: vmx| VM has thin disk scsi0:0; FT enable will be disallowed
Dec 02 00:38:08.507: vmx| VM has zeroedthick disk scsi0:2; FT enable will be disallowed

Alternatively, if you use

[root@esx1 mail03]# cat vmware-1.log  | grep  ’scsi0:0′

[root@esx1 mail03]# cat vmware-1.log  | grep  ’scsi0:1′

[root@esx1 mail03]# cat vmware-1.log  | grep  ’scsi0:2′

This will print out much detailed information. The difference between each output is the “allocation type” field. These are are as follows:

allocation type: 0 – EagerZeroedthick Virtual Disk

allocation type: 1 – Zeroedthick Disk (Default)

allocation type: 2 – Thin Virtual Disk

I was actually in the process of writing an article about my upgrade. Fortunately, I caught wind of this upgrade bug before hand. I’ve decided its too important to wait for the completion of the article. The people in the US are lucky they are on holiday – and so might be with their families rather than in datacenter doing upgrades – that said holiday times are ideal for upgrades…

http://kb.vmware.com/kb/1016070

This week I was asked in an email the following question:

[By the way I hate Wordpress - even when I put paragraph returns in this post it ****ing well removes them!]

Hope you been doing well. I have been busy, busy, busy.

Got a question for you. Preparing to start a new vSphere farm and trying to decide do I use stand switches or start with the DVS. Especially when I see this: http://www.ntpro.nl/blog/archives/1283-vSphere-DvSwitch-caveats-and-best-practices!.html So the main question is do we think this is mature enough to go forward with yet? Can I use Host profiles with DVS safely?

Just appreciate your thoughts.

Apologises for my tardiness in getting back to you. Snowed under. You get the picture. So I’ve written you this lovely long email to make up for that. Gee, this could be blog post! [How ironic - it did become a blogpost!]

A couple of things come to mind. Firstly, Eric’s blog post – I do respect Eric’s blog (and others). Things have gotten so big in VMware – that it would impossible to know of all the known/unknown issues – after all no man is an island, and no-one ever see’s all the possible configurations and screw-ups….

BUT. (you could tell that was coming). Untimely, ripping an ESX host out of vCenter (right-click disconnect/right-click remove). Isn’t the recognised procedure. Yes, I know tardy/slapdash admins might have done this in vCenter1/2 but isn’t something that’s recommended or recognised as the right way. It should always be regarded as the LAST resort.

So if people first removed ESX from the DVS, before removing the ESX host this problem would NOT happen. Basically, what Eric is saying is if you don’t step A followed by Step B followed by Step C then problems can occur. But problems will always occur if you not aware of relationships.

So Eric post would not put me off using DVS for Service Console/Management Network traffic or vmkernel traffic for that matter. I think there’s tendency when people read blog post about new technology that we focus on the bad news, rather than the good. Also people forget (rather conveniently) about what was awful about the previous technology (in this case Standard vSwitches) which they moaned and complained about before the advent of the new. So there’s tendency to re-write the past as some golden period before all this new stuff came about….

I’ve used host profiles with DvS without a problem. For me the bigger question is not whether they can be “depended on” or “safely” used in production- but the reality of what they are like to use on daily basis…. Do they make your admin/configuration any easier. Whilst I love DvSwitches, I’m not at all armoured of Host Profiles. Neither are helped by being shackled to the most expensive SKU of vSphere you can buy!

I like DvS… I especially like them for virtual machine networking. But I see them as being less useful for Management and Vmkernel Ports (not bad, just less useful) because you still have to configure the IP configuration of the ESX host.

Of course, VMware’s answer to that would be to use host profiles. For me – I look at the different ways I might get that IP configuration on to the ESX host. I know the more I manually have to do stuff, the more errors I’m likely to make thru fat-finger-syndrome. Also, If I have to hand the build process to someone less experienced I want to automate as much possible to prevent accidental errors…. This is also tempered by what type of ESX the customer is using. If they using ESXi it rules out kickstart %post scripting and using my UDA appliance…

So lets say I’m doing bulk admin. Building 100’s of ESX hosts, do I really want to sit there apply the host profile to each and every host. As I applied the host profile to N host, it would stall as it asked for my management, VMotion, iSCSI, FT logging, HA Address IP addresses. If you think about the number of IP’s you need for an ESX box (if you follow to the recommendations to letter you end up with quite a lot). Five IP addresses (and their associated subnet masks!)

So when I think of bulk admin & host profiles –  I don’t think they add up. Have I ever created an IP conflict by applying the host profile to many ESX hosts? Yes, an I only have 4 bloody ESX hosts!

I have PowerCLI scripts which automates EVERYTHING I would want to do (and more) than Host Profiles are current capable off. It did take some time to learn PowerCLI and test my scripts. But time is something I have plenty of when I’m not teaching. I wouldn’t say I was the most gifted of scripters – but I don’t care as long as it works, and works reliably.

My PowerCLI script will add a host into vCenter and completely configure it – include things that host profiles cannot do… such as iSCSI and DPM settings. The other thing I don’t like about host profiles – is the fact the host has to be maintenance mode to apply them.   So they can’t really be used as a reconfiguration tool – because who wants to move 20-30 VMs just to add a port group – when something like PowerCLI could do that infinitely quicker with just a couple of lines of script. That more or less relegates the host profiles to being a “consistency” check too – like ConfigControl without the licensing (another product that should be included in the core vSphere4 product!!!).

What’s irritating about host profiles – is that I think if you did have 100’s of ESX hosts you would want something that was more automated than host profiles. The smaller/medium shops won’t buy enterprise+ so they won’t benefit. When I teach VMware, it’s at this moment that I mime to my students – taking a gun from the wall – loading it up – and then firing it at my own foot. It always makes my students laugh.

Can you see what I’m doing here? I’m not arguing against DvS/Host Profiles per se. I’m looking at the technologies and what I might trying to achieve with them – and seeing if they help or hinder. Like anything it depends on the environment and the skills of the person involved. So here’s what I’m recommending in the book:

• New to Product/Admin with no time/aptitude for Scripting
  ESXi+vCenter4+DvS+Host Profiles. Essentially, drag-and-drop management strategy….
• Admin with plenty of time/aptitude for scripting/100’s of Servers
  ESXi or ESX “Classic”+vCenter+DvS+PowerCLI. I know this guy would not be disappointed with PowerCLI. They are adding another 60 cmdlet in time for vSphere4.1….
• ESX 2/3 Admin who has an existing kickstart scripted environment” –
  Don’t re-invent the wheel. Use ESX Classic+KickStart (UDA)- with a view to moving over to PowerCLI, ready for the demise of the COS in vSphere5 (circa 1012?).

  R.I.P. COS

I don’t know what category you fall into. You see host profiles could benefit everyone – the more ESX hosts – the more benefits they are. But then you have to buy this damn + product to get them. It’s just madness.

As for DvS. You could take a hybrid attitude on that. This appears to be the consensus view at the moment. That you use Standard vSwitches for management & vmkernel – and leave DvS just for VMs. This seems to be because Standard vSwitches are more of a “known quantity” than DvSwitches. I had hoped that host profiles would see of end of scripting and scripted installation, but nothing seems offer the same flexibility of things like the EDA/UDA and PowerCLI…. If you really mad about scripting then PowerCLI is the way to go. Hence my recent move away from “Classic” to ESX4i, and using PowerCLI.

And Finally…. I’ve been using DvSwitches for everything since the beta. I’m firm believer in using new technology in lab environments like my own – the more exposure you get to technology the more comfortable/confident you become with it. Plus it gives you the edge on folks who might be more production/conservatively minded. I’ve had no problems until I make an admin error. Like the day I removed vmnic0 from DvSwitch0. But hey, I could have done that with Standard vSwitches as well.

I now know how to fix both it they get broken, and that does impress my students.

Occasionally, (more often than I would really like) I get people emailing asking me to help them on specific problems. Unless these are former customers/students I generally hit delete key!!! Sorry, I’m not being unhelpful but I’m not a one-man help desk guys!

Then occasionally, when I’m bored/intrigued I do respond. It helps if email comes from an interesting domain name. Look there I said it. So if the email comes from my-business-is-to-cheap-to-pay-for-training-or-consultancy.com I’m like to hit the delete key. But if it come from i-work-a-mega-corp.com then I don’t.

That probably makes me sound like a total arse/git. But the reality is there is only so many hours in the day, and I have work that pays to do as well. Still I wouldn’t like to see the end of these emails – because occasionally they spark my interest.

Anyway, what follows is cut and paste from an email I wrote this morning – which I thought some people might benefit from. Here’s the question:

I have picked up a lot of great information from your website and it has helped me over the past couple of years a great deal. I have a quesiton and was wondering if you plan on doing anything with a kickstart installation and vSphere 4 there have been some changes to the way the script is structured and using vm manual is quite daunting at times. My main problem seems to be configuring the hard disks.

Here is another question you might be able to answer for us we have been trying to understand v4. The question is this when I am at the console and I do a alt f1 and log on as root am I on the physical server or am I in the esxconsole and if we are in the console how am I able to see the vmdk file which controls the esxconsole. The follow up to this is if I am not in the esxconsole when I log on then what is the esxconsole and what is it use for..

You do not have to spend a lot of time explaining this you can be brief like hay dummys here is what you need to know oh and we have read the manual.

Thank you sir your time is greatly appreciated.

And this is my response:

You won’t be surprised to hear that during the beta programme information from VMware on scripted installations was decidedly thin on the ground. I worked with very closely with the creators of the EDA and UDA install appliances to get both PXE booting, and scripted installations working. I was lucky to have contacts in VMware Education at the time of the ESX4 beta, who were using kickstart to build their own lab environments to squeeze a little bit of information out them that help. But much was achieved by research – in other words repeatedly retrying scripted installations – until they worked. It’s not the way I like to learn technology myself, but if a vendor is unforthcoming then I won’t let that stop me. You quite right to indicate that whilst the documentation has improved, it still isn’t comprehensive. Quite how VMware imagine folks are going to deploy 100’s of ESX hosts without being more forthcoming is anyone’s guess. It has always amazed me what little effort VMware puts into assisting people in deploying ESX – perhaps they leave that for their PSO work. Nice.

Anyway, just to clarify. The service console now uses a virtual disk. Logically this means that a VMFS volume must be selected or created during the installation, prior to the creation of the partition tables for the ESX host. That VMFS volume must be big enough to hold the esxconsole.vmdk file, and itself must be large enough to hold the partitions within it. Not everything about ESX is held in the .VMDK.

There are some system partitions which can only be custom created by kickstart – in other words they don’t appear in the standard GUI installation. So if a GUI installation is done you just get these standard partitions. These partitions reside OUTSIDE of the virtual disk

/boot
vmkcore
/extended partition

If you do standard installation, the partitions that reside inside the virtual disk are:

swap
/var/log
/

Historically, its this partition table that we have “customized”. I was somewhat unsure whether folks would want to do that – in the end I decided that most would want to create a custom partition scheme in the .vmdk file. So I also invested sometime working out how to do it with kickstart scripts. This new disk structure does introduce some interesting experiences at the Service Console CLI. Say if you run the ‘mount’ command on HP Proliant.

/dev/sdh8 on / type ext3 (rw)

None on /proc type proc (rw)
None on /sys type sysfs (rw)
None on /dev/pts type devpts (rw)
/dev/cciss/c0d0p1 on /boot type ext3 (rw)
/dev/sdh5 on /home type ext3 (rw)
/dev/sdh7 on /opt type ext3 (rw)
/dev/sdh6 on /tmp type ext3 (rw)
/dev/sdh1 on /var/log type ext3 (rw,errors=panic)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

Normally, in ESX3 /dev/sdh5 would have been /dev/cciss/c0d0pN. But as its now a virtual disk you see the more common /dev/sdN syntax… I guess what must be happening is this. That the vmkernel loads, then loads the VMFS Driver – which then allows for the vmkernel to mount the VMDK file, and then access the remainder of the system. The boot loader is grub as you know, and the boot strap environment appears to be the same “busybox” environment used to load ESXi. Again, there is really NO architecture information on how this done. Their might have been a VMWorld session on this – but I wouldn’t know and to be honest I really don’t care too much how it is done so long as it works!

Anyway, I’m hoping that clarifies what’s going on here…. Now to the real meat – scripted installations…

I’ve taken the liberty of attaching my sample KS script to this email. It’s very similar to the one on RTFM associated with the UDA… Here’s the part that does my custom partition scheme. I’ve put comments in to explain what I am doing – hopefully this will clarify things for you. The UDA uses “variables” to hold common parameters like hostname, IP and so on. I’ve removed these [VARIABLES] with the plain text to make it look more like a bog standard kickstart file.

# Clear Partitions
clearpart –drives=/dev/cciss/c0d0 –overwritevmfs

The clearpart as you might know – destroys partitions. Quite a dangerous command. But if I was doing a re-install and want the install to completely wipe my installation – it would fail without the use of new parameter introduced in ESX4.x called –overwritevmfs. This destroys the partition scheme on /dev/cciss/c0d0… including the VMFS volume that holds the esxconsole.vmdk….

# BootLoader ( The user has to use grub by default )

bootloader –location=mbr –driveorder=/dev/cciss/c0d0

Driver order sets where the MBR record goes to – without this script installations can stall.

# Manual Paritioning
part /boot –fstype=ext3 –size=250 –ondisk=/dev/cciss/c0d0
part None –fstype=vmkcore –size=100 –ondisk=/dev/cciss/c0d0
part esx1_local –fstype=vmfs3 –size=20000 –ondisk=/dev/cciss/c0d0 –grow

So here I’m creating the 3 partitions that make up the physical partition scheme. All quite straight foreward in fact my settings here are more less exactly the SAME as what you would get with a manual installation – but I need them in script and is good know they can be change. The odd one is the last one – where I same make the vmfs volume 20000MB, followed by –grow. It appear to be the case that a positive integer must be supplied, and that this number is check against the total partition space defined below – EVEN though the grow means use the remainder of the disk. As word of caution logically only ONE partition can be designated to grow – for obvious reasons!

virtualdisk vd1 –size=15000 –onvmfs=esx1_local
part swap –fstype=swap –size=1600 –onvirtualdisk=vd1
part /opt –fstype=ext3 –size=2048 –onvirtualdisk=vd1
part /tmp –fstype=ext3 –size=2048 –onvirtualdisk=vd1
part /home –fstype=ext3 –size=2048 –onvirtualdisk=vd1
part / –fstype=ext3 –size=5120 –onvirtualdisk=vd1 –grow

VD1 says to create the 1^st virtual disk. It is theoretically possible to give create multiple virtual disks – I just don’t see the point in doing so. I create a virtual disks which is 15GB in size. I then using the part command create the partitions scheme. This does mean there is free space available in the VMFS for local virtual machines if I’m so inclined. The partitions actually add up to about 12.6GB. As for the scheme itself I’m more or less creating the same partition scheme in the virtual disk, as I would have done on the physical disk in ESX 3/2. Although the partition scheme is 12.6GB in totally, I actually let the / volume grow – so once swap, opt, tmp, and home have been created / then uses the remainder of the disk.

Finally, there will be many people who have different ideas about how the ESX console should be partitioned, as there stars in the firmament.  What interests me is the technology that allows that to happens – and how it is done. I will leave it to you to make those decision…

I want to draw ALL my readers attention to Stu Radridge’s blog and in particular to a particular problem in Windows 2008. Although I hate the phrase this is real heads up…

Basically, Stu has discovered that VM created on ESX4 with “Virtual Hardware 7 presents virtual disks as SAN devices – not as local disks. This resulted in the default SAN policy of W2K8 offlining everything bar the boot volume during installation…”

Stu’s article points to a recent KB article that explains this issue, and the MS KB article that explains how to modify the behaviour should you need too

http://vinternals.com/2009/11/virtual-hardware-7-undocumented-feature-revealed/

A couple of days ago I commented on MS recent spate of articles about the surface footprint of the hypervisor.

http://www.rtfm-ed.co.uk/?p=1614

One of the comments on my blog pointed out a salient point. That’s if ESXi is so small – how come it needs a 2GB memory stick, and occupies more space than the alleged 30-60MB claimed by VMware?

Well, I think Eric Gray over at vcritical.com has made the definitive answer…

http://www.vcritical.com/2009/08/if-vmware-esxi-4-is-so-small-why-is-it-so-big/

It appears much of what is resident on the stick, does not include what is resident in memory. For example the ESXi stick contains a copy of the Vi Client and also the .ISO images used by VMware to install VMware Tools the guest operating system. Additionally, as ESXi can be “rolled back” after an upgrade/patch – there has to space on the stick to hold this roll back data.

Of course, that might lead to say – “ah, you see VMware are telling fibs”… But IF you recall the original argument/case for a small hypervisor – is to do with its vulnerability/weakness in terms of performance, patching and security. The small the hypervisor is the less vunerable it is. Unfortunately, this view of the hypervisor is decidedly NOT shared by Microsoft. Microsoft firmly believe the virtualization layer should live inside the operating system. It’s an ideology that many Microsoft bloggers public subscribe to and endorse. I don’t.

I subscribe to the VMware ideology – that the hypervisor should be superslim. With APIs like vStorage/vNetwork that allow third parties like EMC, NetApp and Cisco to “hook” into. Without bloating out the hypervisor with 3rd party unchecked code, that responble for so much instability in the Microsoft platform. If you don’t believe in the VMware ideology, what you believe in the same vulnerabilities and patch management that bedevils a vanilla operating system like Windows, Linux, Solaris and Netware…

Update:

I actually wrote this post early in the morning when I couldn’t sleep (does that show?) and this morning I’ve got something add. And its this. And it’s not going to be pretty. IF Microsoft will persist on patiently avioding the ftechncial acts on their blogs in effort to make their product look as good as/better than VMware. I think we can discount this kind claptrap about MS not being free to present their products at next weeks VMworld, as the unmitigate bullshit that it is… (Are you liking this Stu)

This one has been knocking about in my head from sometime. One of the main manual configurations I have to do with ESXi is settings a number of IP related parameters:

• Hostname
• IP
• Subnet Mask
• Default Gateway
• Primary & Secondary DNS Addresses
• DNS Suffix

Before I can run my PowersHell configuration scripts against the ESX it has to have an IP address and hostname that can be resolved, otherwise my PowersHell scripts go somewhat doolally. So for sometime I have been toying with despensing with a static configuration, and using DHCP to configure the ESXi host.

Horrifying isn’t it? If your from EMEA like me, the idea of giving an physical server a DHCP address is unspoken tabboo. You just DON’T create that kind of dependency between the physical server and the outside world. Interestingly, I stuck rigidly to this view until last year, when I was stopped in my tracks by PowerPoint slide which said “use DHCP to solve this problem”. I was aghast. You’ll never get that past anyone in EMEA, I said – some of them can’t even get approval for DHCP for PXE booting in a server room LAN. And… I guess that for many that will remain the case. It seems this culture is perhaps different across the pond…

In a very informal poll of my US chums – the issue of using DHCP to configure ESXi seemed to garner the response – if it that’s works for you, what’s the problem – I do it here… Perhaps all these years I’ve got stuck in a rut with DHCP?

So today, I thought I would experiment with reconfiguring the Linux DHCP service that runs in my UDA in my lab environment to lease a static “client-reservation” to my ESX host. I did that by adding:

host esx4 {
option host-name “esx4″;
option domain-name “vi4book.com”;
option routers 192.168.3.130, 192.168.3.199;
option domain-name-servers 192.168.3.130, 192.168.3.199;
hardware ethernet 00:15:60:AC:E4:40;
fixed-address 192.168.3.104;
}

I got the MAC address of the ethernet card from the HP ILO. Then I did a factor reset of the ESXi host, which left me with just the root password to setup on boot-up. Everything now is done by combination of DHCP with PowersHell….