I’ve pulled the article until I have time to properly benchmark all three systems on identical hardware, hopefully after the holidays. Although I used different hardware, I firmly believe my numbers provided a reasonable estimate of what performance you would see using identical hardware. I’d had verbal feedback from others that in fact they saw similar results but nothing to substantiate those comments.
The differences between RAID, CPU’s, Drives, etc all tended to balance out in my opinion. I welcome anyone to take on this task as well as I’d love to have something to compare my numbers against and prove my numbers as wrong. So put your money where your mouth is and let’s see what you got.
Are you being serious with these results? Have you ever done benchmarking before? I'm not going to argue which is better but I do want to point out some issues. First, you need to compare apples to apples – same memory, CPU, disks, and number of virtual instances running. You might say this is marketing FUD but it is engineering – simple engineering. The numbers you get don't mean anything unless you do a scientific run on these systems.
Second, you might want to take a look at this article: http://www.virtualization.info/2006/09/licensing-…. You MUST pay for every virtual instance even on Virtuozzo – if you don't then you're pirating software. Don't let the SWSoft FUD tell you otherwise. I've seen this bite other hosters in the butt.
Lastly, you are not only in violation of VMware's EULA if you post benchmark results without their permission but you are also in violation of Microsoft's EULA.
A little research next time and some more scientific results unless you make commission from Virtuozzo and just want to spread their FUD.
Hey Bob,
Thanks for the feedback.
I appreciate you visiting my site and leaving feedback. The reason for this blog is so that I can get feedback and be further educated by others on the Internet that are more knowledgeable than me.
I ask that you please assist in helping me to setup a 'proper' benchmarking environment and performing accurate tests. I have asked SWsoft and I emailed [email protected] as well asking that they too provide me guidance/information on how to best do this. I honestly doubt anyone will give me the time of day.
I also have a brand new DELL 1950 with 6GB of memory, dual xeon processors and SAS drives on order and plan to perform clean installs of the host OS and VM application to perform a more accurate benchmark if nothing else but for my own knowledge and I acknowledged this in my blog entry that these results were only early numbers. Bob, you're probably an engineer so our methods are probably similar. When you develop a new product do you not first create a rude mock up as proof of concept and then present this to your Boss. Once you get the go ahead and funding you then build a prototype system and perform bench or field testing. Once the prototype and field test results prove useful you then present your results to your boss's bosses who decide whether or not to proceed with development, marketing and eventually distribution? Consider these tests as nothing more than my initial mock up.
Please understand I have only the utmost respect for VMware, their products and the trail they blazed in virtualization. Without question VMware has been doing virtualization long before these others and I have happily used it for many years. I can understand their view that they don't want people just throwing benchmarks out there and distributing 'rubbish' but at the same time I think we all want to see how things stack up so someone should step up to the plate and publish some real benchmarks. PC manufacturers do it, do they not?
I did google VMware and VirtualPC to see if anyone is providing any VMware related benchmarks and I found many:
http://www.flexbeta.net/main/articles.php?action=… http://usuarios.lycos.es/hernandp/articles/vpcvs…. http://arstechnica.com/reviews/apps/vm.ars/4 http://usuarios.lycos.es/hernandp/articles/vpcvsI… http://capitalhead.com/articles/benchmarking-vmwa… http://www.windowsitpro.com/Windows/Article/Artic…
I will review all three EULA's this evening and if as you mention I am in violation of Microsoft's EULA I will remove those results from my website as well. In fact, I emailed all three companies asking if I am in error and if I should remove this and I suspect before long I'll get an email from one of them agreeing with your views which in a nutshell are: I suck, this is wrong and I should burn in hell for even posting it. (Okay, maybe I'm exaggerating)
Virtualization and Microsoft Licensing has been a topic of much discussion between me and various individuals at Microsoft. This is part of the reason we have not released VPS hosting yet. At one point we were going to release VPS hosting on Windows 2003 Enterprise using Microsoft Virtual Server. But the lack of customer tools made that impossible. Recently at a hosting conference I asked an individual who says he leads the hosting 'sector' of Microsoft about Virtualization and Microsoft's view on licensing and if the SWsoft comments of "you only have to pay the one license" were correct and the response I got was: "Keep doing what you're doing now and we'll get back to you". It's up in the air!! Quite honestly Bob, I've had this conversation with multiple people at Microsoft over the past 2 years and depending on who you speak with, the time of day and the angle at which the sun reflects from the water, the response differs. Now, I did read this article on virtualization.info and I also read the comments where it appears one person from SWsoft responded with their view based on the PUR. But I will heed your advice and talk to Microsoft once again on the topic. Better safe than sorry.
Jess
Bob,
I just downloaded MSVS R2 and read the EULA and couldn't find mention of not being allowed to benchmark it and post the benchmarks.
Here's the version ID from the one I read. Perhaps you can point it out to me?
EULAID:VS05R2_RM.1_STD-ENT_RTL_EN
I did find that you're not allowed to offer commercial hosting on it, but then I already knew that.
Jess
The benchmark is reasonably true.
I have seen similar result from a consultant working for a bank evaluating Virtuozzo for Linux vs VMware ESX and gained result that Virtuozzo is 4 times faster than ESX.
The test was based on identical hardware configuration except Virtuozzo is using slower SATA disk and ESX using faster SCSI disk, making it even more impressive!
I appreciate the intent of this article, but this is really not a good article for many reasons. My enterprise uses virtualization widely, Virtuozzo, MS & VMware (at least for a little longer) and perf is important but there's a lot you're missing.
Licensing you have all wrong. All OSes need licenses whether they're running Virtuozzo or MS or VMware.
Virtuozzo provides nice isolation, but doesn't even come close to MS/VMware. With Virtuozzo you can only run the same identical OS for each virtual container. With MS/VMware you can run completely different OSes, say Windows 2K3 and RH Linux on the same server.
Finally, you're methodology is all over the place. You're comparing different virtualization technologies running on different hardware and trying to make a conclusion?
Again, I do appreciate what you're trying to say and I will even agree that Virtuozzo performs with lower overhead than MS/VMware, but you're comparing apples and oranges…
Thank you Adam, I don't disagree with you at all. I have not had the opportunity to test all of the platforms on the same hardware, my time just hasn't allowed it. And I tried to make it abundantly clear that this was the case, that the hardware was all different. This was really just a preliminary results if nothing else. I have 2 dozen new dell servers arriving this week so hopefully I can find a few hours to pull one aside and work on this. hopefully…
I also agree that Virtuozzo has shortcomings when compared to VMware and MSVS, the lack of hardware virtualization can cause problems when you need to run other guest operating systems, vpns, etc. In fact I use both Vmware and Microsoft technologies on my desktops for this, not virtuozzo. I've also run into issues running tools like regmon and filemon on virtuozzo, they simply won't run because of the OS virtualization. So it's by no means the perfect solution.
However, for a hosting environment where all of your clients are going to run the same operating system and perform similar tasks it has an advantage and the lower overhead imposed by OS virtualization is a big boost to hosts as it means that the clients get better performance and we'll be able to get a better VE density.
Thank you for the comments.
Because of these dumb benchmarks, the ISV are putting statements in their EULA that benchmarks can not be published. What a nonsense artcile.
If you compare apples with oranges, why not include bananas? If you only care about performance, run native!
I will not even start listing all the strange comparisons in your article.. come on, raid5 against raid1.. duh raid1 wins.
And then just 'downplay' that VPN connections do not work against these 'virtual environments'… but what about patching, os level upgrades, running mix of terminal server and non terminal server scheduling..
Virtuozzo is absolutly not a bad product, buy why do you want to compare it against something different??? Compare it with something simulair, like terminal services, because that is what virtuozzo is.
Thank you for the feedback Anykey. I'm giving in and pulling the article. Anykey is correct in that I am not providing an accurate comparison and that this is in fact the very type of benchmark that VMware is trying to prevent.
When time permits I'll compare all three on identical hardware and not post the article again until then.
It should be noted, there is not going to be a one platform is the best solution. That's just not going to happen, all three are going to have their place and provide a better solution for a particular task. I use all three systems on a regular basis but for the particular task in question Virtuozzo was the more elegant solution.
Actually, some of these points are valid and others are not. The two approaches to virtualization are in fact apples and oranges. The point is, which one is best for your business need? OS virtualization, such as Sun Containers and Virtuozzo, do allow a homogeneous OS environment to leverage a single OS instance across multiple VPSs. Hardware emulators, on the other hand do not. Hardware emulators do give you the benefit of running different OS versions side by side in the VPS. Is this a benefit or or not? It depends on your need. As for licensing, The EULA from Microsoft clearly states that licensing is based on the number of instances running, virtual or physical. MS has in fact verified this contrary to previous posts.
From a performance standpoint, OS virtalization is faster since there is no virtalized hardware and multiple OS instances to deal with. Security has been reviewed for Virtuozzo by Deloitte & Touche as well as Verisign. I have seen Verisign document and the VPSs and host could not be compromised by one another.
VMware as well, has a single point of failure with their hypervisor layer. This is actually a custom Linux kernel.
As Jsss pointed out, why is VMware so anal about benchmarking? If platforms are not the same or something is completely off, the reader should be able to figure it out. The fact that VMWare won't let any results be published by users in production environments, leads me to wonder what they are hiding.
There is a need for both approaches in business. Why pretend there is not?