How To Troubleshoot High Packet or Management CPU Issue on Citrix ADC

CPU is a finite resource. Like many resources, there are limits to a CPU’s capacity. The NetScaler appliance has two kinds of CPUs in general: The Management CPU and Packet CPU.

Wherein, the Management CPU is responsible for processing all the Management traffic on the appliance and the Packet CPU(s) are responsible for handling all the data traffic for eg. TCP , SSL etc.

When diagnosing a complaint involving high CPU, start by gathering the following fundamental facts:

  1. CPUs impacted: nsppe (one or all) & management.
  2. Approximate time stamp/duration.

The following command o/p are quintessential for troubleshooting the high CPU issues:

  • Output of top command: Gives the CPU utilization percentage by the processes running on the NetScaler.
  • Output of stat system memory command: Gives the memory utilization percentage which can also contribute in the CPU utilization.
  • Output of stat system cpu command: This gives the stats about the current CPU utilization in total on the appliance.

Sample o/p of stat cpu command:

> stat cpuCPU statisticsID Usage1 29

The above o/p indicates that there is only 1 CPU (utilized for both Management and Data traffic) and the percentage of utilization is 29%.

The CPU ID is 1.

Now, there are appliances with multiple cores (nCore ) wherein more than single core is allocated to the appliance and then we see multiple CPU IDs on the “stat system cpu ” o/p.

*The high CPU seen when running a “top” command does not impact the performance of the box. It also “does not” mean that the NetScaler is running at high CPU or consuming all of the CPU. The NetScaler Kernel runs on top of BSD and that is what is being seen. Although it appears to be using the full amount of the CPU, it is actually not.

We can further follow the below steps for understanding the CPU usage:

  1. Check the following counters to understand CPU usage.

    CLASSIC:

    master_cpu_use

    cc_appcpu_use filter=cpu(0)

    (If AppFW or CMP is configured, then looking at slave_cpu_use also makes sense for classic)

    nCORE:

    (For an 8 Core system)

    mgmt_cpu_use (CPU0 – nscollect runs here)

    master_cpu_use (average of cpu(1) thru cpu(7))

    cc_cpu_use filter=cpu(1)

    cc_cpu_use filter=cpu(2)

    cc_cpu_use filter=cpu(3)

    cc_cpu_use filter=cpu(4)

    cc_cpu_use filter=cpu(5)

    cc_cpu_use filter=cpu(6)

    cc_cpu_use filter=cpu(7)

  2. How to look for CPU use for a particular CPU?

    Use the nsconmsg command and search for cc_cpu_use and grep for the CPU you are interested in.

    The output will look like the following:

    Index rtime totalcount-val delta rate/sec symbol-name&device-no
    320 0 209 15 2 cc_cpu_use cpu(8)
    364 0 205 -6 0 cc_cpu_use cpu(8)
    375 0 222 17 2 cc_cpu_use cpu(8)
    386 0 212 -10 -1 cc_cpu_use cpu(8)
    430 0 216 6 0 cc_cpu_use cpu(8)
    440 0 201 -15 -2 cc_cpu_use cpu(8)
    450 0 208 7 1 cc_cpu_use cpu(8)
    461 0 202 -6 0 cc_cpu_use cpu(8)
    471 0 209 7 1 cc_cpu_use cpu(8)
    482 0 238 29 4 cc_cpu_use cpu(8)
    492 0 257 19 2 cc_cpu_use cpu(8)
  • Look at the total count (third) column and divide by 10 to get the CPU percentage. For eg. in the last line above, 257 implies that 257/10 = 25.7% CPU is used by CPU(8).

    Run the following command to investigate the nsconsmg counters for CPU issue:

    nsconmsg –K newnslog –g cpu_use –s totalcount=600 –d currentnsconmsg –K newnslog –d current | grep cc_cpu_use
  • Look at the traffic, memory and CPU in conjunction. We may be hitting platform limits if it sustained high CPU usage. Try to understand if the CPU has gone up because of traffic. If so, try to understand if it is genuine traffic or any sort of attack.
  • We can further check for the Profiler o/p to understand who is taking the CPU.

    For details on the profiler o/p , logs , refer to the below article:

    https://support.citrix.com/article/CTX212480

  • We can further use the CPU counters mentioned in the below article for more details:

    https://support.citrix.com/article/CTX133887


Profiling FAQs

1. What is Constant profiling?

This refers to the running of CPU profiler at all times, as soon as the NetScaler device comes up. At the boot time, the profiler is invoked and it keeps running. Any time any of the PE’s associated CPU exceeds 90%, the profiler captures the data into a set of files.

2. Why is this needed?

This was necessitated with the issues seen at some customer sites and in internal tests. With customer issues, it’s hard to go back and request the customer to run the profiler when the issue is seen again. Hence, we have felt the need of a profiler running to be able to see the functions triggering high CPU. With this feature now, the profiler will be running always and the data gets captured when the high CPU usage occurs.

3. Which releases/builds contain this feature?

TOT (Crete) 44.2+

9.3 – all builds

9.2 52.x +

Only nCore builds are affected.

4. How do we know the profiler is already running?

Run the ps command to check if nsproflog and nsprofmon are running. The number of nsprofmon processes should be the same as the number of PEs running.

root@nc1# ps -ax | grep nspro36683 p0 S+ 0:00.00 grep nspro79468 p2- I 0:00.01 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79496 p2- I 0:00.00 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79498 p2- I 0:00.00 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79499 p2- I 0:00.00 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79502 p2- S 33:46.15 /netscaler/nsprofmon -s cpu=3 -ys cpuuse=800 -ys profmode=cpuuse -O -k /v79503 p2- S 33:48.03 /netscaler/nsprofmon -s cpu=2 -ys cpuuse=800 -ys profmode=cpuuse -O -k /v79504 p2- S 32:20.63 /netscaler/nsprofmon -s cpu=1 -ys cpuuse=800 -ys profmode=cpuuse -O -k /v

5. Where is the profiler data?

The profiled data is collected in /var/nsproflog directory. Here is a sample output of the list of files in that folder. At any point of time, the currently running files are newproflog_cpu_<penum>.out. Once the data in these files exceed 10MB in size, they are archived into a tar file and compressed. The roll over mechanism is similar to what we have for newnslog files.

newproflog.0.tar.gz newproflog.5.tar.gz newproflog.old.tar.gznewproflog.1.tar.gz newproflog.6.tar.gz newproflog_cpu_0.outnewproflog.2.tar.gz newproflog.7.tar.gz nsproflog.nextfilenewproflog.3.tar.gz newproflog.8.tar.gz nsproflog_optionsnewproflog.4.tar.gz newproflog.9.tar.gz ppe_cores.txt

The current data is always captured in newproflog_cpu_<ppe number>.out. Once the profiler is stopped, the newproflog_cpu_* files will be archived into newproflog.(value in nsproflog.nextfile-1).tar.gz.

6. What is nsprofmon and what’s nsproflog.sh?

Nsprofmon is the binary that interacts with PE, retrieves the profiler records and writes them into files. There are a myriad of options present which are hard to remember. The wrapper script nsproflog.sh is easier to use and remember. Going forward, it is recommended to use the wrapper script, if it’s limited to collecting CPU usage data.

7. Should I use nsprofmon or nsproflog.sh?

In earlier releases (9.0 and earlier), nsprofmon was heavily used internally and by the support groups. Some internal scripts that devtest use, refer to nsprofmon. It is recommended to use nsproflog.sh, if it’s limited to collecting CPU usage data.

8. Will the existing scripts be affected?

It will affect the existing scripts if they try to invoke the profiler. Please see the next question.

9. What if I want to start the profiler with a different set of parameters?

There can be only one instance of profiler running at any time. If the profiler is already running (invoked at boot time with constant profiling), and if we want to invoke again, it flags an error and exits.

root@nc1# nsproflog.sh cpuuse=900 startnCore ProfilingAnother instance of profiler is already running.If you want to run the profiler at a different CPU threshold, please stop the current profiler using# nsproflog.sh stop... and invoke again with the intended CPU threshold. Please see nsproflog.sh -h for the exact usage.


Similarly, nsprofmon is also modified to check if another instance is running. If it is, it exits flagging an error.

If the profiler needs to be run again with a different CPU usage (i.e. 80%), the running instance needs to be stopped and invoked again:

root@nc1# nsproflog.sh stopnCore ProfilingStopping all profiler processesRemoving buffer for -s cpu=1Removing profile buffer on cpu 1 ... Done.Saved profiler capture data in newproflog.5.tar.gzSetting minimum lost CPU time for NETIO to 0 microsecond ... Done.Stopping mgmt profiler process
root@nc1# nsproflog.sh cpuuse=800 start

10. How do I view the profiler data?

In /var/nsproflog, unzip and untar the desired tar archive. Each file in this archive should correspond to each PE.

Caution: When we unzip and untar the older files, the files from the archive will overwrite the current ones. The names stored inside the tar archive are the same as the ones to which currently running profiler keeps writing into. To avoid this, unzip and untar into a temporary directory.

The simplest way to see the profiled data is

# nsproflog.sh kernel=/netscaler/nsppe display=newproflog_cpu_<ppe number>.out

11. How do we collect this data for analysis?

The showtech script has been modified to collect the profiler data. When customer issues arrive, var/nsproflog can be checked to see if the profiler has captured any data.

12. Anything else that I need to know?

Collecting traces and profiler data are made mutually exclusive. When nstrace.sh is run to collect traces, profiler is automatically stopped and restarted when nstrace.sh exits. We wouldn’t have the profiler data during the time of collecting traces.

13. What commands get executed when profiler is started?

Initialization:

For each CPU, the following commands are executed initially:

nsapimgr -cnsapimgr -ys cpuuse=900 nsprofmon -s cpu=<cpuid> -ys profbuf=128 -ys profmode=cpuuse

Capturing:

For each CPU, the following are executed:

nsapimgr -cnsprofmon -s cpu=<cpuid> -ys cpuuse=900 -ys profmode=cpuuse -O -k /var/nsproflog/newproflog_cpu_<cpuid>.out -s logsize=10485760 -ye capture

After the above, nsprofmon processes will be running till any one of the capture buffers is full.

nsproflog.sh waits for any of the above child processes to exit

Stopping:

Kill all nsprofmon processes (killall -9 nsprofmon)

For each CPU, the following commands are executed:

nsprofmon -s cpu=<cpuid> -yS profbuf

Profiler capture files are archived:

nsapimgr -ys lctnetio=0

Related:

Need a guide for the meaning of each OID in the MIB and its tree/relations

I need a solution

is there a guide for all proxysg or SSLV OIDs and its relations/tree

trap OID, polling OID

For example: how to know which OID is mapped to the sensor name of the CPU temperature and which OIDs is mapped for the CPU temprature units and value

Thanks in advance,

Islam

0

Related:

How To Troubleshoot High Packet or Management CPU Issue on NetScaler Appliance

CPU is a finite resource. Like many resources, there are limits to a CPU’s capacity. The NetScaler appliance has two kinds of CPUs in general: The Management CPU and Packet CPU.

Wherein, the Management CPU is responsible for processing all the Management traffic on the appliance and the Packet CPU(s) are responsible for handling all the data traffic for eg. TCP , SSL etc.

When diagnosing a complaint involving high CPU, start by gathering the following fundamental facts:

  1. CPUs impacted: nsppe (one or all) & management.
  2. Approximate time stamp/duration.

The following command o/p are quintessential for troubleshooting the high CPU issues:

  • Output of top command: Gives the CPU utilization percentage by the processes running on the NetScaler.
  • Output of stat system memory command: Gives the memory utilization percentage which can also contribute in the CPU utilization.
  • Output of stat system cpu command: This gives the stats about the current CPU utilization in total on the appliance.

Sample o/p of stat cpu command:

> stat cpuCPU statisticsID Usage1 29

The above o/p indicates that there is only 1 CPU (utilized for both Management and Data traffic) and the percentage of utilization is 29%.

The CPU ID is 1.

Now, there are appliances with multiple cores (nCore ) wherein more than single core is allocated to the appliance and then we see multiple CPU IDs on the “stat system cpu ” o/p.

*The high CPU seen when running a “top” command does not impact the performance of the box. It also “does not” mean that the NetScaler is running at high CPU or consuming all of the CPU. The NetScaler Kernel runs on top of BSD and that is what is being seen. Although it appears to be using the full amount of the CPU, it is actually not.

We can further follow the below steps for understanding the CPU usage:

  1. Check the following counters to understand CPU usage.

    CLASSIC:

    master_cpu_use

    cc_appcpu_use filter=cpu(0)

    (If AppFW or CMP is configured, then looking at slave_cpu_use also makes sense for classic)

    nCORE:

    (For an 8 Core system)

    mgmt_cpu_use (CPU0 – nscollect runs here)

    master_cpu_use (average of cpu(1) thru cpu(7))

    cc_cpu_use filter=cpu(1)

    cc_cpu_use filter=cpu(2)

    cc_cpu_use filter=cpu(3)

    cc_cpu_use filter=cpu(4)

    cc_cpu_use filter=cpu(5)

    cc_cpu_use filter=cpu(6)

    cc_cpu_use filter=cpu(7)

  2. How to look for CPU use for a particular CPU?

    Use the nsconmsg command and search for cc_cpu_use and grep for the CPU you are interested in.

    The output will look like the following:

    Index rtime totalcount-val delta rate/sec symbol-name&device-no
    320 0 209 15 2 cc_cpu_use cpu(8)
    364 0 205 -6 0 cc_cpu_use cpu(8)
    375 0 222 17 2 cc_cpu_use cpu(8)
    386 0 212 -10 -1 cc_cpu_use cpu(8)
    430 0 216 6 0 cc_cpu_use cpu(8)
    440 0 201 -15 -2 cc_cpu_use cpu(8)
    450 0 208 7 1 cc_cpu_use cpu(8)
    461 0 202 -6 0 cc_cpu_use cpu(8)
    471 0 209 7 1 cc_cpu_use cpu(8)
    482 0 238 29 4 cc_cpu_use cpu(8)
    492 0 257 19 2 cc_cpu_use cpu(8)
  • Look at the total count (third) column and divide by 10 to get the CPU percentage. For eg. in the last line above, 257 implies that 257/10 = 25.7% CPU is used by CPU(8).

    Run the following command to investigate the nsconsmg counters for CPU issue:

    nsconmsg –K newnslog –g cpu_use –s totalcount=600 –d currentnsconmsg –K newnslog –d current | grep cc_cpu_use
  • Look at the traffic, memory and CPU in conjunction. We may be hitting platform limits if it sustained high CPU usage. Try to understand if the CPU has gone up because of traffic. If so, try to understand if it is genuine traffic or any sort of attack.
  • We can further check for the Profiler o/p to understand who is taking the CPU.

    For details on the profiler o/p , logs , refer to the below article:

    https://support.citrix.com/article/CTX212480

  • We can further use the CPU counters mentioned in the below article for more details:

    https://support.citrix.com/article/CTX133887


Profiling FAQs

1. What is Constant profiling?

This refers to the running of CPU profiler at all times, as soon as the NetScaler device comes up. At the boot time, the profiler is invoked and it keeps running. Any time any of the PE’s associated CPU exceeds 90%, the profiler captures the data into a set of files.

2. Why is this needed?

This was necessitated with the issues seen at some customer sites and in internal tests. With customer issues, it’s hard to go back and request the customer to run the profiler when the issue is seen again. Hence, we have felt the need of a profiler running to be able to see the functions triggering high CPU. With this feature now, the profiler will be running always and the data gets captured when the high CPU usage occurs.

3. Which releases/builds contain this feature?

TOT (Crete) 44.2+

9.3 – all builds

9.2 52.x +

Only nCore builds are affected.

4. How do we know the profiler is already running?

Run the ps command to check if nsproflog and nsprofmon are running. The number of nsprofmon processes should be the same as the number of PEs running.

root@nc1# ps -ax | grep nspro36683 p0 S+ 0:00.00 grep nspro79468 p2- I 0:00.01 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79496 p2- I 0:00.00 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79498 p2- I 0:00.00 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79499 p2- I 0:00.00 /bin/sh /netscaler/nsproflog.sh cpuuse=800 start79502 p2- S 33:46.15 /netscaler/nsprofmon -s cpu=3 -ys cpuuse=800 -ys profmode=cpuuse -O -k /v79503 p2- S 33:48.03 /netscaler/nsprofmon -s cpu=2 -ys cpuuse=800 -ys profmode=cpuuse -O -k /v79504 p2- S 32:20.63 /netscaler/nsprofmon -s cpu=1 -ys cpuuse=800 -ys profmode=cpuuse -O -k /v

5. Where is the profiler data?

The profiled data is collected in /var/nsproflog directory. Here is a sample output of the list of files in that folder. At any point of time, the currently running files are newproflog_cpu_<penum>.out. Once the data in these files exceed 10MB in size, they are archived into a tar file and compressed. The roll over mechanism is similar to what we have for newnslog files.

newproflog.0.tar.gz newproflog.5.tar.gz newproflog.old.tar.gznewproflog.1.tar.gz newproflog.6.tar.gz newproflog_cpu_0.outnewproflog.2.tar.gz newproflog.7.tar.gz nsproflog.nextfilenewproflog.3.tar.gz newproflog.8.tar.gz nsproflog_optionsnewproflog.4.tar.gz newproflog.9.tar.gz ppe_cores.txt

The current data is always captured in newproflog_cpu_<ppe number>.out. Once the profiler is stopped, the newproflog_cpu_* files will be archived into newproflog.(value in nsproflog.nextfile-1).tar.gz.

6. What is nsprofmon and what’s nsproflog.sh?

Nsprofmon is the binary that interacts with PE, retrieves the profiler records and writes them into files. There are a myriad of options present which are hard to remember. The wrapper script nsproflog.sh is easier to use and remember. Going forward, it is recommended to use the wrapper script, if it’s limited to collecting CPU usage data.

7. Should I use nsprofmon or nsproflog.sh?

In earlier releases (9.0 and earlier), nsprofmon was heavily used internally and by the support groups. Some internal scripts that devtest use, refer to nsprofmon. It is recommended to use nsproflog.sh, if it’s limited to collecting CPU usage data.

8. Will the existing scripts be affected?

It will affect the existing scripts if they try to invoke the profiler. Please see the next question.

9. What if I want to start the profiler with a different set of parameters?

There can be only one instance of profiler running at any time. If the profiler is already running (invoked at boot time with constant profiling), and if we want to invoke again, it flags an error and exits.

root@nc1# nsproflog.sh cpuuse=900 startnCore ProfilingAnother instance of profiler is already running.If you want to run the profiler at a different CPU threshold, please stop the current profiler using# nsproflog.sh stop... and invoke again with the intended CPU threshold. Please see nsproflog.sh -h for the exact usage.


Similarly, nsprofmon is also modified to check if another instance is running. If it is, it exits flagging an error.

If the profiler needs to be run again with a different CPU usage (i.e. 80%), the running instance needs to be stopped and invoked again:

root@nc1# nsproflog.sh stopnCore ProfilingStopping all profiler processesRemoving buffer for -s cpu=1Removing profile buffer on cpu 1 ... Done.Saved profiler capture data in newproflog.5.tar.gzSetting minimum lost CPU time for NETIO to 0 microsecond ... Done.Stopping mgmt profiler process
root@nc1# nsproflog.sh cpuuse=800 start

10. How do I view the profiler data?

In /var/nsproflog, unzip and untar the desired tar archive. Each file in this archive should correspond to each PE.

Caution: When we unzip and untar the older files, the files from the archive will overwrite the current ones. The names stored inside the tar archive are the same as the ones to which currently running profiler keeps writing into. To avoid this, unzip and untar into a temporary directory.

The simplest way to see the profiled data is

# nsproflog.sh kernel=/netscaler/nsppe display=newproflog_cpu_<ppe number>.out

11. How do we collect this data for analysis?

The showtech script has been modified to collect the profiler data. When customer issues arrive, var/nsproflog can be checked to see if the profiler has captured any data.

12. Anything else that I need to know?

Collecting traces and profiler data are made mutually exclusive. When nstrace.sh is run to collect traces, profiler is automatically stopped and restarted when nstrace.sh exits. We wouldn’t have the profiler data during the time of collecting traces.

13. What commands get executed when profiler is started?

Initialization:

For each CPU, the following commands are executed initially:

nsapimgr -cnsapimgr -ys cpuuse=900 nsprofmon -s cpu=<cpuid> -ys profbuf=128 -ys profmode=cpuuse

Capturing:

For each CPU, the following are executed:

nsapimgr -cnsprofmon -s cpu=<cpuid> -ys cpuuse=900 -ys profmode=cpuuse -O -k /var/nsproflog/newproflog_cpu_<cpuid>.out -s logsize=10485760 -ye capture

After the above, nsprofmon processes will be running till any one of the capture buffers is full.

nsproflog.sh waits for any of the above child processes to exit

Stopping:

Kill all nsprofmon processes (killall -9 nsprofmon)

For each CPU, the following commands are executed:

nsprofmon -s cpu=<cpuid> -yS profbuf

Profiler capture files are archived:

nsapimgr -ys lctnetio=0

Related:

How to pin Citrix Hypervisor Virtual CPUs to specific Physical CPUs

Citrix Hypervisor maps vCPUs to pCPUs by default in a semi-even way to distribute VM load on the host. In some cases it may be needed to have a specific mapping, for example, if some VMs will be CPU intensive while other wont, the intensive VMs can be mapped to exclusive physical CPUs while the others share resources. This can lead to performance improvements in the Hypervisor.

The following example will be used to change hard affinity. This means where the vCPU is allowed to run. In this sense, once a vCPU is pinned to a pCPU with hard affinity, the vCPU won’t be able to run in any other pCPU.

Soft affinity is used to define where does the vCPU prefers to run but it doesn’t restrict where it is allowed to run.

To determine which type of mapping to use, analyze the type of workload that the guest VMs are going to be running. Ultimately the best way to determine the best mapping is through testing different configurations with real workloads and observing the results. There is not a perfect ratio since workloads change depending on many factors, such as OS type, patches, apps run by users, employee work schedules, etc.

Related:

Can i deployment SSL intercept with single CPU

I need a solution

Hi all.

I want to ask, can my Proxy SG with single CPU doing the SSL intercept ?

I try find the document requirement for SSL intercept but i can not find the article.

Our Proxy SG type is SG S200-20. 

Because i have an experience with Proxy SG with single CPU and the CPU is full 100%.

When i check utilization for CPU, there are SSL and Cryptography proccess which consumes more than 45% of CPU resources.

Best Regards

Indra Pramono

0

Related:

  • No Related Posts

Virtualize SMP Server

I need a solution

I’m considering virtualizing my SMP server that has 2 CPUs with 12 cores each to a VMware infrastructure.  VCenter Converter will set the cores to 24 like the physical box but for those who’ve gone through this P2V, did you decrease, say, the cores per processor so as not to utilize too much of the resources on the ESXi host?  I know memory tends to have more of an impact on an ESX host where my physical box has 24 gigs which the host can handle.

Anyhow, I’m more concerned about CPU usage on the host should I leave the cores as-is or the performance impact it’ll have on the VM by reducing the total number of cores.  I don’t believe we have a VM with 24 cores in our VMware environment so not sure if leaving it at “24” is reasonable.  Also, thought the more cores you specify in a VM could have a negative impact as the ESX host needs to wait to grab ALL of the CPU resources from the host when needed.  Just not sure what to set the number of CPUs and cores to in Converter which makes sense.

0

Related:

How to Use host-cpu-tune to Fine tune XenServer 6.2.0 Performance

Pinning Strategies

  • No Pinning (default): When no pinning is in effect, the Xen hypervisor is free to schedule domain’s vCPUs on any pCPUs.
    • Pros: Greater flexibility and better overall utilization of available pCPUs.
    • Cons: Possible longer memory access times, particularly on NUMA-based hosts. Possible lower I/O throughput and control plane operations when pCPUs are overcommitted.
    • Explanation: When vCPUs are free to run on any pCPU, they may allocate memory in various regions of the host’s memory address space. At a later stage, a vCPU may run on a different NUMA node and require access to that previously allocated data. This makes poor utilization of pCPU caches and incur in higher access times to that data. Another aspect is the impact on I/O throughput and control plane operations. When more vCPUs are being executed than pCPUs that are available, the Xen hypervisor might not be able to schedule dom0’s vCPUs when they require execution time. This has a negative effect on all operations that depend on dom0, including I/O throughput and control plane operations.
  • Exclusive Pinning: When exclusive pinning is on effect, the Xen hypervisor pins dom0 vCPUs to pCPUs in a one-to-one mapping. That is, dom0 vCPU 0 runs on pCPU 0, dom0 vCPU 1 runs on pCPU 1 and so on. Any VM running on that host is pinned to the remaining set of pCPUs.
    • Pros: Possible shorter memory access times, particularly on NUMA-based hosts. Possible higher throughput and control plane operations when pCPUs are overcommitted.
    • Cons: Lower flexibility and possible poor utilization of available pCPUs.
    • Explanation: If exclusive pinning is on and VMs are running CPU-intensive applications, they might under-perform by not being able to run on pCPUs allocated to dom0 (even when dom0 is not actively using them).

Note: The exclusive pinning functionality provided by host-cpu-tune will honor specific VM vCPU affinity configured using the VM parameter vCPU-params:mask. For more information, refer to the VM Parameters section in the appendix of the XenServer 6.2.0 Administrator’s Guide.

Using host-cpu-tune

The tool can be found in /usr/lib/xen/bin/host-cpu-tune. When executed with no parameters, it displays help:

[root@host ~]# /usr/lib/xen/bin/host-cpu-tune

Usage: /usr/lib/xen/bin/host-cpu-tune { show | advise | set <dom0_vcpus> <pinning> [–force] }

show Shows current running configuration

advise Advise on a configuration for current host

set Set host’s configuration for next reboot

<dom0_vcpus> specifies how many vCPUs to give dom0

<pinning> specifies the host’s pinning strategy

allowed values are ‘nopin’ or ‘xpin’

[–force] forces xpin even if VMs conflict

Examples: /usr/lib/xen/bin/host-cpu-tune show

/usr/lib/xen/bin/host-cpu-tune advise

/usr/lib/xen/bin/host-cpu-tune set 4 nopin

/usr/lib/xen/bin/host-cpu-tune set 8 xpin

/usr/lib/xen/bin/host-cpu-tune set 8 xpin –force

[root@host ~]#

Recommendations

The total number of pCPUs and advise as follows:

# num of pCPUs < 4 ===> same num of vCPUs for dom0 and no pinning

# < 24 ===> 4 vCPUs for dom0 and no pinning

# < 32 ===> 6 vCPUs for dom0 and no pinning

# < 48 ===> 8 vCPUs for dom0 and no pinning

# >= 48 ===> 8 vCPUs for dom0 and excl pinning

The utility works in three distinct modes:

  1. Show: This mode displays the current dom0 vCPU count and infer the current pinning strategy.

    Note: This functionality will only examine the current state of the host. If configurations are changed (for example, with the set command) and the host has not yet been rebooted, the output may be inaccurate.

  2. Advise: This recommends a dom0 vCPU count and a pinning strategy for this host.

    Note: This functionality takes into account the number of pCPUs available in the host and makes a recommendation based on heuristics determined by Citrix. System administrators are encouraged to experiment with different settings and find the one that best suits their workloads.

  3. Set: This functionality changes the host configuration to the specified number of dom0 vCPUs and pinning strategy.

    Note: This functionality may change parameters in the host boot configuration files. It is highly recommended to reboot the host as soon as possible after using this command.

    Warning: Setting zero vCPUs to dom0 (with set 0 nopin) will cause the host not to boot.

Resetting to Default

The host-cpu-tune tool uses the same heuristics as the XenServer Installer to determine the number of dom0 vCPUs. The installer, however, never activates exclusive pinning because of race conditions with Rolling Pool Upgrades (RPUs). During RPU, VMs with manual pinning settings can fail to start if exclusive pinning is activated on a newly upgraded host.

To reset the dom0 vCPU pinning strategy to default:

  1. Run the following command to find out the number of recommended dom0 vCPUs:

    [root@host ~]# /usr/lib/xen/bin/host-cpu-tune advise

  2. Configure the host accordingly, without any pinning:
    • [root@host ~]# /usr/lib/xen/bin/host-cpu-tune set <count> nopin
    • Where <count> is the recommended number of dom0 vCPUs indicated by the advise command.
  3. Reboot the host. The host will now have the same settings as it did when XenServer 6.2.0 was installed.

Usage in XenServer Pools

Settings configured with this tool only affect a single host. If the intent is to configure an entire pool, this tool must be used on each host separately.

When one or more hosts in the pool are configured with exclusive pinning, migrating VMs between hosts may change the VM's pinning characteristics. For example, if a VM are manually pinned with the vCPU-params:mask parameter, migrating it to a host configured with exclusive pinning may fail. This could happen if one or more of that VM's vCPUs are pinned to a pCPU index exclusively allocated to dom0 on the destination host.

Additional commands to obtain information concerning CPU topology:

xenpm get-cpu-topology

xl vcpu-list

Related:

7022546: Updating microcode in Xen environments.

SLES12SP2 and newer Xen environments:

Beginning withSLES12SP2, Dom0 is now a PVOPS based kernel (kernel-default), whichhas no interface for microcode updates while running as a Dom0.However, if the initrd contains an updated microcode, and Xen is madeaware of its existence, the update will be applied during the Xenearly boot process. Updates using this method required a host rebootafter correctly adding the microcode to the initrd.

Installing a microcode update in SLES12SP2 and newerenvironments:

1. Determine current microcode level:

# grep -m1 microcode/proc/cpuinfo

microcode : 0x2000011

2. Install updated microcode package (ucode-intel, or ucode-amd).

3. Rebuild initrd using `mkinitrd`.

NOTE – The `lsinitrd` command can beused to verify the microcode is correctly inserted into the initrd.

#lsinitrd /boot/initrd-4.12.14-23-default

Image:/boot/initrd-4.12.14-23-default:11M

================================================================

EarlyCPIOimage

================================================================

drwxr-xr-x 1 root root 0 Jul 13 13:05 .

-rw-r–r– 1root root 2 Jul 13 13:05 early_cpio

drwxr-xr-x 1 root root 0 Jul 13 13:05 kernel

drwxr-xr-x 1root root 0 Jul 13 13:05 kernel/x86

drwxr-xr-x 1 root root 0 Jul 13 13:05kernel/x86/microcode

-rw-r–r– 1 root root 31744Jul 13 13:05kernel/x86/microcode/GenuineIntel.bin

================================================================

4. Edit /etc/default/grub, and add “ucode=scan” to Xenhypervisor command line:

GRUB_CMDLINE_XEN_DEFAULT=”vga=gfx-1024x768x16crashkernel=202M<4G ucode=scan”

5. Reboot.

6. Verify microcode is updated:

# grep -m1 microcode/proc/cpuinfo

microcode : 0x200004a

7. Verify new speculative mitigation features are availablethrough `xl dmesg`.

# xl dmesg | grep Speculative-A5

(XEN) Speculative mitigation facilities:

(XEN) Hardware features: IBRS/IBPB STIBP SSBD

(XEN) Compiled-insupport: INDIRECT_THUNK

(XEN) Xen settings: BTI-Thunk JMP,SPEC_CTRL: IBRS+ SSBD-, Other: IBPB

(XEN) Support for VMs: PV:MSR_SPEC_CTRL RSB, HVM: MSR_SPEC_CTRL RSB

(XEN) XPTI (64-bitPV only): Dom0 enabled, DomU enabled

Pre-SLES12SP1 Xen environments:

In SLES12SP1 and older(including SLES11), the Dom0 kernel (kernel-xen) is based onxenlinux. This environment can upgrade microcode from Dom0 atrun-time. However, the CPU is not re-sampled after such an update,and therefore guests cannot use new features exposed with an onlinemicrocode update. To avoid this problem, micocode updates should bedone using the following steps:

Installing a microcode update in SLES12SP1 and olderenvironments:

1. Install updated microcode package (microcode_ctrl).

2. Determine correct microcode file:

# grep -E ‘family|model|stepping’ -m 3/proc/cpuinfo

cpu family : 6

model : 62

model name :Intel(R) Xeon(R) CPU E7-4890 v2 @ 2.80GHz

stepping : 7

Intel microcode is named “[cpufamily]-[model]-[stepping]”, using hexadecimal values. In the aboveoutput, this would be “06-3e-07”.

AMD microcode is named”microcode_amd_fam[NN]h.bin”, where [NN] is the hexadecimalvalue of the CPU family. For example:

# grep -E ‘cpu family|model name’ -m 2/proc/cpuinfo

cpu family : 23

model name :AMD EPYC 7601 32-Core Processor

For the AMD CPU above, the applicablemicrocode would be /lib/firmware/amd-ucode/microcode_amd_fam17h.bin.

3. Copy the microcode file from /lib/firmware/intel-ucode to/boot as GenuineIntel.bin. (For AMD environments, use/lib/firmware/amd-ucode and AuthenticAMD.bin.)

# cp/lib/firmware/intel-ucode/06-3e-07 /boot/GenuineIntel.bin

NOTE – For EFI boot environments,the microcode should be copied to the EFI boot partition anddirectory used in booting. This is typically /boot/efi/efi/SuSE.

4. Edit /etc/default/grub, and make the following 2 changes:

– Add thefollowing module line in the Xen boot section, following the initrdmodule:

module /boot/GenuineIntel.bin

– Add “ucode=2” (where “2” is the “module” line number containing the GenuineIntel.bin string, starting from 0) to Xen hypervisor command line:

“kernel/boot/xen.gz vga=mode-0x317 ucode=2”

NOTEfor EFI boot environments, add the following line to the Xen efi bootconfiguration (/boot/efi/efi/SuSE/xen.cfg)entries.

“ucode=GenuineIntel.bin”

5. Reboot.

6. Verify new speculative mitigation features are availablethrough `xm dmesg`.

# xm dmesg | grep Speculative-A5

(XEN) Speculative mitigation facilities:

(XEN) Hardware features: IBRS/IBPB STIBP SSBD

(XEN) Xen settings:BTI-Thunk N/A, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB

(XEN) Support for VMs: PV: MSR_SPEC_CTRL RSB, HVM: MSR_SPEC_CTRL RSB

(XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled

NOTE: Multiple vendors may provide updated microcode. Ultimately,only the updates which matches the running CPU (using hex cupidcomparison) will be applied during the update process.

Related:

How to manage SMT (hyper-threading) in XenServer?

This article describes how to explicitly manage simultaneous multithreading (SMT) or hyper-threading on supported hardware.

SMT or hyper-threading is a technology that is designed to improve CPU performance by enabling parallelization of computations. When hyper-threading is enabled, each physical processor is split into multiple logical processors.

If you are running a XenServer host with hyper-threading enabled, disabling hyper-threading will reduce the number of CPUs available on the host. Before you disable hyper-threading, you must carefully evaluate the workload and decide whether disabling is the best option for your XenServer deployment.

Citrix recommends that you do not run a VM with more virtual CPUs (vCPUs) than the number physical CPUs available on the XenServer host. For more information, see CTX236977 – Overcommitting pCPUs on individual XenServer VMs.

Related: