The Challenge of Fair Platform Comparisons
Following my initial comparison of Windows virtualization platforms—Virtuozzo, VMware, and Microsoft Virtual Server—I received valuable feedback highlighting the limitations of my methodology. The critique was warranted: my comparison wasn't conducted on equivalent hardware platforms, making it an inherently flawed analysis.
This experience taught me two fundamental lessons that remain relevant for any technology professional conducting platform evaluations.
Key Lessons for Technology Evaluations
1. Methodology Transparency is Critical
When publishing performance comparisons, organizations must explicitly acknowledge testing limitations. My original benchmark used different hardware configurations without clearly stating this constraint upfront. For enterprise decision-makers relying on such comparisons, this transparency is essential for proper evaluation.
2. License Agreements Contain Publication Restrictions
More importantly, I discovered that major virtualization vendors include specific clauses governing benchmark publication in their End User License Agreements (EULAs) and Product Use Rights (PURs).
Vendor-Specific Licensing Restrictions
VMware's Benchmark Policy
VMware's EULA specifically states:
"You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware at benchmark@VMware.com to request such review."
This means any public benchmarking requires pre-approval from VMware, including methodology review.
Microsoft's Complex Requirements
Microsoft's Product Use Rights document—a comprehensive 66-page legal document—contains similarly restrictive language regarding benchmark publication and methodology requirements.
Strategic Implications for IT Leaders
These licensing restrictions have significant implications for enterprise technology evaluation:
- Internal Testing: Organizations can conduct internal performance testing for decision-making purposes
- Publication Limitations: Sharing detailed benchmark results publicly may violate licensing terms
- Vendor Relations: Formal benchmark publication requires vendor coordination and approval
Why This Still Matters Today
While virtualization technology has evolved dramatically since 2006, the fundamental principle remains unchanged: major enterprise software vendors continue to include benchmark publication restrictions in their licensing agreements. IT leaders must understand these constraints when planning technology evaluations and considering public disclosure of performance data.
The lesson extends beyond virtualization to any enterprise software evaluation—always review licensing terms before conducting and sharing comparative analysis.