Tag: X86 architecture

This article explains how to handle your power settings and how you can change the configuration to fit your specific needs.

Citrix Hypervisor (formerly XenServer) has the following power settings:

For Intel processors:

• Use all available C-states regardless of BIOS settings

• Use the “performance” CPU frequency governor

• Use frequency scaling and Turbo mode as controlled by the BIOS settings

For AMD processors:

• Use the BIOS settings to know which C-states are supported

• Use the more power-friendly “ondemand” CPU frequency governor

• Use frequency scaling and Turbo mode as controlled by the BIOS settings

It is recommended to use Citrix Hypervisor’s default configuration and to enable OS-controlled frequency scaling and Turbo mode in the BIOS.

C-states

• For Intel processors, Citrix Hypervisor uses all the available C-states regardless of the BIOS C-state settings.
• For AMD processors, changing C-state settings in the BIOS does have an effect for AMD processors.

Citrix Hypervisor uses Xen Project Hypervisor’s means of enumerating and using C-states for newer Intel processors. Xen Project Hypervisor comes preloaded with the C-states and performance characteristics of each processor family (like Sandy Bridge, Haswell, etc). It chooses the correct information at boot based on the processor used. This improves performance because Xen can make more accurate decisions about which C-state to use. For example, C1 and C1E are treated as two separate states which gives Xen more flexibility in the choice of C-state. Modern CPUs can transition between C-states very quickly so the use of deep C-states has minimal performance impact but provides great potential for power saving.

Falling back to use of BIOS C-state settings

To revert to the C-state configuration for Intel processors that was used by older versions of XenServer (6.2 and earlier), use the following command in dom0:

/opt/xensource/libexec/xen-cmdline --set-xen mwait-idle=false

A reboot is required for this change to take effect. After this, controlling C-states in the BIOS will have an effect on Citrix Hypervisor. This is not a recommended configuration as it is likely to lead to reduced performance.

To go back to the default configuration, use the following and reboot:

/opt/xensource/libexec/xen-cmdline --delete-xen mwait-idle

Exploiting Turbo mode

Using all the available C-states gives the processor most thermal headroom to enter Turbo mode. Hence the default configuration that uses all available C-states should give the best performance for processors supporting a Turbo mode.

For maximum performance, it is recommended that Turbo mode is enabled in the BIOS.

To determine whether Turbo mode is enabled, use the following command in dom0:

# xenpm get-cpufreq-para | fgrep turboturbo mode : enabled

To verify that Turbo mode is being used stress the CPU in a VM and check that the CPU frequency is higher than the rated value:

# grep "model name" /proc/cpuinfomodel name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

With VMs idle:

# xenpm start 1|grep "Avg freq" Avg freq 3503030 KHz Avg freq 2856840 KHz Avg freq 2788820 KHz Avg freq 2312680 KHz Avg freq 2210650 KHz Avg freq 3537040 KHz Avg freq 3469020 KHz Avg freq 2584760 KHz

With VMs busy:

# xenpm start 1|grep "Avg freq" Avg freq 3673080 KHz Avg freq 3571050 KHz Avg freq 3503030 KHz Avg freq 3537040 KHz Avg freq 3469020 KHz Avg freq 3537040 KHz Avg freq 3469020 KHz Avg freq 3571050 KHz

Avoiding Turbo mode

Some users might wish to avoid using Turbo mode, for example when running workloads that require low-latency wakeups, or when desiring consistent rather than maximal performance. In practice, this can normally be achieved by forcing the processor to stay in C0.

The maximum C-state can be set as in previous versions of XenServer (6.2 and earlier). To immediately force the processor to stay in C0, execute the following command in dom0:

xenpm set-max-cstate 0

Or, to make this change persist across reboots, use the following command and reboot:

/opt/xensource/libexec/xen-cmdline --set-xen max_cstate=0

Note that forcing the processor to stay in C0 will significantly increase the power consumption and may reduce the lifespan of the processor (From Section 4.2.2 of the 4th Generation Intel Core Datasheet Volume 1: “Long term reliability cannot be assured unless all the Low-Power Idle States are enabled”).

When setting a maximum C-state, note that C-state numbers do not necessarily correspond directly with ACPI C-state names. The C-states mentioned in the output of “xenpm get-cpuidle-states” are simply a numerical list of all the C-states that Xen knows about, listed as C0, C1, C2, etc. This means that ACPI C0 will appear as C0, ACPI C1 will appear as C1, ACPI C1E will appear as C2, and so on. Furthermore, some states might not be individually addressable with “xenpm set-max-cstate <X>” and setting the maximum C-state to X does not necessarily mean that only C0..CX will be used.

CPU Frequencies

Frequency scaling and Turbo mode are controlled by the BIOS settings. Citrix Hypervisor controls each processor core’s frequency by adjusting the P-state.

Citrix recommends configuring the BIOS frequency scaling settings to use “OS control” mode (or equivalent) and to enable Turbo mode.

In its default configuration,Citrix Hypervisor uses the “performance” governor for Intel processors and the more power-friendly “ondemand” governor for AMD processors. This is the recommended configuration.

• The performance governor always sets the processor to its maximum P-state which results in the highest frequency being used.

• The ondemand governor dynamically downclocks the cores. This can provide some power savings at the cost of performance.

For AMD processors, the “ondemand” governor is normally preferred as it gives a good compromise between performance and power-saving. By contrast, Intel processors typically save power by entering deep C-states so the “performance” governor is preferred.

Changing governor

Some users may wish to change the CPU frequency governor – for example, to sacrifice performance to make power savings, or vice versa.

In order to use a different governor at runtime, run the following command in dom0:

xenpm set-scaling-governor performance

or:

xenpm set-scaling-governor ondemand

In order to make this change persist across reboots, use:

/opt/xensource/libexec/xen-cmdline --set-xen cpufreq=xen:performance

or:

/opt/xensource/libexec/xen-cmdline --set-xen cpufreq=xen:ondemand

Setting the CPU frequency governor to “performance” should not have a significantly detrimental effect on the ability of the processor to enter Turbo mode for Intel processors. Since entering Turbo mode usually relies on other cores being idle, having the idle cores at a higher frequency does not matter because the default C-state configuration allows the processor to stop its clocks when idle.

Thermal effects

In order to maximize performance, CPUs should be allowed to run at maximum frequency. However, many CPUs will decrease their operating frequency if they overheat. Overheating is more likely to happen if cores are forced to run in P0 and C0 or while there is a sustained high CPU load.

In order to avoid overheating, the CPU and chassis fans should be configured to maximum speed. This will ensure CPUs stay cool at the cost of increased power consumption. Note that “adaptive” or “automatic” fan speed settings may not always go to maximum speed so do not necessarily perform maximal cooling.