Note: This Tech Preview has now been superseded. In XenServer 7.4, AMD’s MxGPU is fully supported for production use. For more information, see the XenServer 7.4 documentation.
XenServer 7.2 contains a technical preview of AMD’s multi-user GPU technology, MxGPU. This technical preview is suitable for early adopters to evaluate the technology with Windows 7 SP1 (64-bit) and Windows 10 (64-bit) guests, but may not be suitable for production environments.
AMDs MxGPU enables multiple Virtual Machines (VM) to have direct access to a portion of a single physical GPU, by making use of Single Root I/O Virtualisation. The same AMD graphics driver deployed on non-virtualized Operating Systems can be used inside the guest.
Guest VMs use MxGPU GPUs in the same manner as a physical GPU that has been passed through by the hypervisor; an AMD graphics driver loaded in the guest VM provides direct access to the GPU for performance critical fast paths.
Note: This MxGPU technical preview does not currently have supported hardware configurations with XenServer 7.2. Details on supported configurations will be available on the XenServer Hardware Compatibility List
1. Licensing Note
MxGPU is available for XenServer Enterprise edition customers, or those who have access to XenServer through their XenApp/XenDesktop entitlement. To learn more about XenServer editions, and to find out how to upgrade, visit the Citrix website here. For detailed information on Licensing, refer to the XenServer Licensing FAQ on the Citrix Product Documentation website.
2. Available AMD MxGPU vGPU Types
AMD MxGPU cards can contain multiple Graphics Processing Units (gpu). For example, S7150 cards contain one physical GPUs, and S7150x2 cards contain two GPUs. Each physical GPU (pgpu) can host several different types of virtual GPU (vgpu). vgpu types split a pgpu into a pre-defined number of vgpus, each with an equal share of the framebuffer and graphics processing capabilities. The different vgpu types are targeted at different classes of workload with the vgpu types that split a pgpu into a fewer number of pieces being more suitable for intensive workloads.
Note: At any given time, vgpus hosted on a physical gpu must all be of the same type. However, there is no corresponding restriction between physical gpus on the same card. This restriction is automatic and may cause unexpected capacity planning issues.
3. AMD MxGPU System Requirements
- AMD FirePro S7100-series GPUs
- XenServer Enterprise edition (or access to XenServer through a XenDesktop/XenApp entitlement).
- A server capable of hosting XenServer and AMD MxGPU cards. The list of servers validated by AMD can be found on AMDs website (http://www.amd.com/citrix).
- AMD MxGPU host driver public key (Attached to this KB article)
- AMD MxGPU host drivers for XenServer
- AMD FirePro in-guest drivers, suitable for MxGPU on XenServer
- To run XenDesktop with VMs running AMD MxGPU, you will also need: XenDesktop 7.13 or later, full installation.
- System BIOS configured to support SR-IOV, with the MxGPU configured as the secondary adapter.
4. Preparation Overview
- Install XenServer
- Install the AMD MxGPU host driver public key
- Install the AMD MxGPU host drivers for XenServer
- Reboot the XenServer host
5. Installation on XenServer 7.2
XenServer 7.2 is available for download from the XenServer Downloads page.
Customers should install:
- XenServer 7.2 Base Installation ISO
- XenCenter 7.2 Windows Management Console
Refer to the XenServer Installation Guide on the Citrix Product Documentation website for comprehensive details on installation. Please see section 1 for details on licensing requirements.
6. Installing the AMD MxGPU host driver for XenServer
Before installing the MxGPU host driver, the GPG key must be installed in dom0. Once this feature is formally supported, and out of tech preview, this step will not be required. In order to run the following, make sure you download the RPM-GPG-KEY-AMD-MXGPU attachment on this article, copy it to the host and reference the correct location in the following commands.
Note: The names and versions below are examples and may be different in your environment.
Before installing the MxGPU host driver, the GPG key must be installed in dom0. Once this feature is formally supported, and out of tech preview, this step will not be required. In order to run the following, make sure you copy RPM-GPG-KEY-AMD-MXGPU to the host and referencing the correct location
[root@xenserver ~]# /opt/xensource/debug/import-update-key RPM-GPG-KEY-AMD-MXGPU
gpg: key 09DD72A4: public key “RPM-GPG-KEY-AMD-MXGPU (GPG key for AMD MxGPU) <MxGPU@amd.com>” imported
pub 2048R/09DD72A4 created: 2017-03-16 expires: never usage: SC
trust: ultimate validity: unknown
[ unknown] (1). RPM-GPG-KEY-AMD-MXGPU (GPG key for AMD MxGPU) MxGPU@amd.com
The Update can then be installed using XenCenter (Tools > Install Update > Select update or supplemental pack for disk) or using the xe CLI command xe-install-supplemental-pack. As with the GPG key, copy the Update to the host and reference the correct location.
[root@xenserver ~]# xe-install-supplemental-pack mxgpu-0.9.amd.iso
[root@xenserver ~]# xe update-list
uuid ( RO) : b9c2969d-9ed3-4432-9b86-8e9399b0e8d7
name-label ( RO): mxgpu
name-description ( RO): MxGPU Host Driver
version ( RO): 0.9
installation-size ( RO): 595329
hosts (SRO): 369a34f4-f273-4c0f-9c77-0c0f59220412
Reboot the XenServer host:
[root@xenserver ~]# shutdown -r now
Broadcast message from root (pts/1) Mon May 09 13:05:31 2016):
After rebooting the XenServer host, verify that the MxGPU package has installed and loaded correctly by checking that the gim kernel driver can be loaded: [root@xenserver ~]# modinfo gim
[root@xenserver ~]# modprobe gim
Verify that the gim kernel driver successfully created MxGPU Virtual Functions, which will then be provided to the guests. The Virtual Functions will have the “S7150V” identifier, as in the following output:
[root@xenserver ~]# lspci | grep “FirePro S7150”
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
05:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
05:02.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
05:02.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
Use the GPU tab in XenCenter to confirm that MxGPU Virtual GPU types are listed as available on the system:
7. Create a MxGPU Enabled VM
Before configuring a VM to use MxGPU, the guest VM must be installed. Note that this technical preview of MxGPU will only work with Windows 7 SP1 (64-bit) and Windows 10 (64-bit).
Once the VM is installed, follow the XenServer “Configuring Graphics” guide for XenServer 7.2 section “5. Create a Virtual GPU or GPU Pass-Through Enabled VM” to complete the configuration.
8. Known Restrictions
- A host must either use MxGPU or Passthrough of a GPU; you cannot dynamically switch between these. The host must be rebooted before switching between these modes.
- Various features, including VM Suspend/Resume, Live Migration and High Availability are not compatible with MxGPU.
- Some systems may require a vendor-specific VBIOS. Ensure that the latest VBIOS available from the system vendor is applied.
9. Providing Feedback and Raising Issues
As this feature is a technical preview, feedback should be provided via https://tracker.citrix.com rather than with Citrix Support.
- Create an account: Visit https://tracker.citrix.com, click the “Sign up” link and follow the wizard for help in signing up
- Create an issue in the XenServer Feedback (XSB) project, selecting the appropriate issue type (Enquiry, Bug or Improvement)
- Select the “AMD MxGPU” component
- Fill in as many details about the environment and problem you are seeing
- Attach a Server Status Report or bugtool to the ticket (see https://support.citrix.com/article/CTX125372)