Xen Dual VGA Passthrough

From Ubisoft Quebec City Studio Cloud Computing Team (currently hiring):

This video presents one of our experiments featuring 2 fully 3D hardware accelerated VMs using Xen and GFX passthrough.

The host is a Lenovo ThinkStation S20 with 2 NVIDIA GeForce GTX 460 running a xenified (Dom0) version of Ubuntu 10.10 64-bit.
Each VM is running a vanilla Windows XP 32-bit with 4Go of RAM and 4 VCPUs.
The native baseline results are very similar.

The video shows the Unigine Heaven Benchmark 2.5 using the following settings:
• Renderer: DirectX 9
• Resolution: 1680 x 1050
• Shaders: High
• Textures: High
• Filter: Trilinear
• Anisotropy: 4x
• Occlusion: Enabled
• Refraction: Enabled
• Volumetric: Enabled

If you are interested to join our team, please contact: marie-eve.bolduc at ubisoft dot com

--------------------------------------------------------------------------------­-----------

For those interested in reproducing our experiment, here is how we did it:

1. Build a Dom0 kernel for your favorite distro and activate Xen options

We used:
• Ubuntu 10.10 64-bit (Maverick Meerkat)
• Jeremy Fitzhardinge's stable pv-ops kernel

2. Extract your video card BIOS

We used:
• NVFlash

3. Download, patch, build and install Xen

We used:
• Stable Xen 4.1.0
• Tobias Geiger's patches

4. Create a guest image

We used:
• Windows XP 32-bit

5. Passthrough GFX Adapter using pciback

YMMV

Category:

Gaming

Tags:

• Xen
• GFX
• VGA
• Passthrough
• Virtualization
• Ubuntu
• VM
• Cloud
• GPU
• Unigine
• Heaven
• Benchmark
• Hardware
• Acceleration
• QEMU
• VT-d

[54v]

If you Ubisoft folks don't mind sharing, I'm currently doing this to replace aging desktops in my gaming room: Four Radeon 5850's plugged into an MSI 890FXA-GD70 board. The results are *amazing.*

I'm using VMware for it though, and a major drawback is that I can't use VMware's USB passthrough implementation (which doesn't use IOMMU, it's a USB packet capture/routing thing) refuses to pass HID devices. How do you do it with Xen?

Great video, guys :)

[ubicloud]

@54v We updated the description of the video so that it now contains information on how we actually did it.

[nbhusain]

Could you guys release a patch to get this working, ive been trying to do this with my dual 470gtx without success :(

[ubicloud]

@nbhusain We updated the description of the video so that it now contains information on how we actually did it.

[PvtPapaSmurf]

A guide for making this work would be a much appreciated contribution. Let me know if there is anything I can do to help with such a guide.

[ubicloud]

@PvtPapaSmurf We updated the description of the video so that it now contains information on how we actually did it.

[me1ne]

If I'm getting this right this could mean "bye bye dual boot" for some gaming folks out there.

[crcmachine]

Real GPU performance in a VM. This is really cool stuff.

[Glad1kun]

There should be another GPU for the host Ubuntu...

[archimedesmp]

So, I just upgraded my rig to support vt-d and I really look forward some dual gaming with my gf on one machine.

But: c'mon, Ubisoft pays you guys to make Windows Games run in Windows VMs on Linux hosts? Why not just build games that run on Linux and save the licence fees for the VMs, plus maybe not needing *any* VMs? :P That way I would even buy some Ubisoft games during the next Steam sale again ;)

But still, I like what you made out of Xen there :) Keep up the good work

[54v]

@craftyguy1 I had the exact same problem. There are what appear to be updated patches (just checked, not sure where I found them), but I'm currently trying to work this against 4.2-unstable as the notes show significant progress in this area, and maybe the patches won't be necessary. We'll see, and I'll post back when I know more :P

[craftyguy1]

@ubicloud What modifications did yall make to Tobias Gieger's patches? If applied against stable Xen 4.1.0, the make will fail on pass-through.c

[54v]

@ubicloud Very nice! I looked more into it and saw how Xen has the unique "VGA Passthrough" tech that actually exposes the VGA device, option rom and all into the VM; far more sophisticated than just tossing another PCIe device into the machine through IOMMU.

I was really hoping that I could hack my way into pushing HID devices to the guest machines through ESXi, but VMKernel "won't allow it." Something about the kernel eating the USB HID device before the arbitrator can even see it :(

[ubicloud]

@SmartestButtOfAll We updated the description of the video so that it now contains information on how we actually did it.