This article explains changes introduced in XenServer 6.5 to the handling of power settings and describes how users with specific needs can change the configuration.
XenServer’s default power settings are optimal for most users, so do not normally need to be modified.
By default, XenServer 6.5 and later has the following power settings.
For Intel processors:
Use all available C-states regardless of BIOS settings
Use the “performance” CPU frequency governor
Use frequency scaling and Turbo mode as controlled by the BIOS settings
For AMD processors:
Use the BIOS settings to know which C-states are supported
Use the more power-friendly “ondemand” CPU frequency governor
Use frequency scaling and Turbo mode as controlled by the BIOS settings
It is recommended to use XenServer’s default configuration and to enable OS-controlled frequency scaling and Turbo mode in the BIOS.
For Intel processors, XenServer 6.5 and later uses all the available C-states regardless of the BIOS C-state settings.
In previous versions of XenServer, Xen used information from the BIOS to know which C-states the processor supported. In XenServer 6.5 and later, that information from the BIOS is not used for Intel processors by default. This means that enabling or disabling C-states and C1E in the BIOS has no effect for Intel processors, unlike in previous releases.
Note: For AMD processors, the C-state configuration remains the same as for previous versions of XenServer. Changing C-state settings in the BIOS does have an effect for AMD processors.
XenServer 6.5 and later use Xen Project Hypervisor 4.4’s new means of enumerating and using C-states for newer Intel processors. Xen Project Hypervisor 4.4 comes preloaded with the C-states and performance characteristics of each processor family (like Sandy Bridge, Haswell, etc). It chooses the correct information at boot based on the processor used. This improves performance because Xen can make more accurate decisions about which C-state to use. For example, C1 and C1E are treated as two separate states which gives Xen more flexibility in the choice of C-state. Modern CPUs can transition between C-states very quickly so the use of deep C-states has minimal performance impact but provides great potential for power saving.
Falling back to use of BIOS C-state settings
In order to revert to the C-state configuration for Intel processors that was used by previous versions of XenServer, use the following command in dom0:
/opt/xensource/libexec/xen-cmdline –set-xen mwait-idle=false
A reboot is required for this change to take effect. After this, controlling C-states in the BIOS will have an effect on XenServer. This is not a recommended configuration as it is likely to lead to reduced performance.
To go back to the default configuration, use the following and reboot:
/opt/xensource/libexec/xen-cmdline –delete-xen mwait-idle
Exploiting Turbo mode
Using all the available C-states gives the processor most thermal headroom to enter Turbo mode. Hence the default configuration that uses all available C-states should give the best performance for processors supporting a Turbo mode.
For maximum performance, it is recommended that Turbo mode is enabled in the BIOS.
To determine whether Turbo mode is enabled, use the following command in dom0:
# xenpm get-cpufreq-para | fgrep turboturbo mode : enabled
To verify that Turbo mode is being used stress the CPU in a VM and check that the CPU frequency is higher than the rated value:
# grep "model name" /proc/cpuinfomodel name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
With VMs idle:
# xenpm start 1|grep "Avg freq" Avg freq 3503030 KHz Avg freq 2856840 KHz Avg freq 2788820 KHz Avg freq 2312680 KHz Avg freq 2210650 KHz Avg freq 3537040 KHz Avg freq 3469020 KHz Avg freq 2584760 KHz
With VMs busy:
# xenpm start 1|grep "Avg freq" Avg freq 3673080 KHz Avg freq 3571050 KHz Avg freq 3503030 KHz Avg freq 3537040 KHz Avg freq 3469020 KHz Avg freq 3537040 KHz Avg freq 3469020 KHz Avg freq 3571050 KHz
Avoiding Turbo mode
Some users might wish to avoid using Turbo mode, for example when running workloads that require low-latency wakeups, or when desiring consistent rather than maximal performance. In practice, this can normally be achieved by forcing the processor to stay in C0.
The maximum C-state can be set as in previous versions of XenServer. To immediately force the processor to stay in C0, execute the following command in dom0:
xenpm set-max-cstate 0
Or, to make this change persist across reboots, use the following command and reboot:
/opt/xensource/libexec/xen-cmdline –set-xen max_cstate=0
Note that forcing the processor to stay in C0 will significantly increase the power consumption and may reduce the lifespan of the processor (From Section 4.2.2 of the 4th Generation Intel Core Datasheet Volume 1: “Long term reliability cannot be assured unless all the Low-Power Idle States are enabled”).
When setting a maximum C-state, note that C-state numbers do not necessarily correspond directly with ACPI C-state names. The C-states mentioned in the output of “xenpm get-cpuidle-states” are simply a numerical list of all the C-states that Xen knows about, listed as C0, C1, C2, etc. This means that ACPI C0 will appear as C0, ACPI C1 will appear as C1, ACPI C1E will appear as C2, and so on. Furthermore, some states might not be individually addressable with “xenpm set-max-cstate <X>” and setting the maximum C-state to X does not necessarily mean that only C0..CX will be used.
Frequency scaling and Turbo mode are controlled by the BIOS settings, as with previous versions of XenServer. XenServer controls each processor core’s frequency by adjusting the P-state.
Citrix recommends configuring the BIOS frequency scaling settings to use “OS control” mode (or equivalent) and to enable Turbo mode.
In its default configuration, XenServer 6.5 and later use the “performance” governor for Intel processors and the more power-friendly “ondemand” governor for AMD processors. This is the recommended configuration.
The performance governor always sets the processor to its maximum P-state which results in the highest frequency being used.
The ondemand governor dynamically downclocks the cores. This can provide some power savings at the cost of performance.
For AMD processors, the “ondemand” governor is normally preferred as it gives a good compromise between performance and power-saving. By contrast, Intel processors typically save power by entering deep C-states so the “performance” governor is preferred.
Some users may wish to change the CPU frequency governor, e.g. in order to sacrifice performance to make power savings, or vice versa.
In order to use a different governor at runtime, run the following command in dom0:
xenpm set-scaling-governor performance
xenpm set-scaling-governor ondemand
In order to make this change persist across reboots, use:
/opt/xensource/libexec/xen-cmdline –set-xen cpufreq=xen:performance
/opt/xensource/libexec/xen-cmdline –set-xen cpufreq=xen:ondemand
Setting the CPU frequency governor to “performance” should not have a significantly detrimental effect on the ability of the processor to enter Turbo mode for Intel processors. Since entering Turbo mode usually relies on other cores being idle, having the idle cores at a higher frequency does not matter because the default C-state configuration allows the processor to stop its clocks when idle.
In order to maximize performance, CPUs should be allowed to run at maximum frequency. However, many CPUs will decrease their operating frequency if they overheat. Overheating is more likely to happen if cores are forced to run in P0 and C0 or while there is a sustained high CPU load.
In order to avoid overheating, the CPU and chassis fans should be configured to maximum speed. This will ensure CPUs stay cool at the cost of increased power consumption. Note that “adaptive” or “automatic” fan speed settings may not always go to maximum speed so do not necessarily perform maximal cooling.