7023078: Security Vulnerability: “L1 Terminal Fault” (L1TF) ??? Hypervisor Information (CVE-2018-3620, CVE-2018-3646, XSA-273).

Full mitigation for this issue requires a combination of hardware and software changes. Depending on the guest type, software changes may be required at both the Hypervisor and guest level.

Updated Intel microcode (provided through your hardware / BIOS vendor or by SUSE) introduces a new feature called “flush_l1d”. Hypervisors and bare-metal kernels use this feature to flush the L1 data cache during operations which may be susceptible to data leakage (e.g. when switching between VMs in Hypervisor environments).

Software mitigations exist for the Linux Kernel and for Hypervisors. These mitigations include support for new CPU features, passing these features to guests, and support for enabling/disabling/tuning the mitigations. Recommended mitigations vary depending on the environment.

For the Linux kernel (on both bare metal and virtual machines) L1TF mitigation is controlled through the “l1tf” kernel boot parameter. For complete information on this parameter, see TID 7023077.

KVM

For KVM host environments, mitigation can be achieved through L1D cache flushes, and/or disabling Extended Page Tables (EPT) and Simultaneous MultiThreading (SMT).

The L1D cache flush behavior is controlled through the “kvm-intel.vmentry_l1d_flush” kernel command line option:

kvm-intel.vmentry_l1d_flush=always

The L1D cache is flushed on every VMENTER.

kvm-intel.vmentry_l1d_flush=cond

The L1D cache is flushed on VMENTER only when there can be leak of host memory between VMEXIT and VMENTER. This could still leak some host data, like address space layout.

kvm-intel.vmentry_l1d_flush=never

Disables the L1D cache flush mitigation.

The default setting here is “cond”.

The l1tf “full” setting overrides the settings of this configuration variable.


L1TF can be used to bypass Extended Page Tables (EPT). To mitigate this risk, it is possible to disable EPT and use shadow pages instead. This mitigation is available through the “kvm-intel.enable_ept” option:
kvm-intel.enable_ept=0

The Extended Page tables support is switched off.
As shadow pages are much less performant than EPT, SUSE recommends leaving EPT enabled, and use L1D cache flush and SMT tuning for full mitigation.


To eliminate the risk of untrusted processes or guests exploiting this vulnerability on a sibling hyper-thread, Simultaneous MultiThreading (SMT) can be disabled completely.

SMT can be controlled through kernel boot command line parameters, or on-the-fly through sysfs:

On the kernel boot command line:

nosmt

SMT is disabled, but can be later reenabled in the system.

nosmt=force

SMT is disabled, and can not be reenabled in the system.

If this option is not passed, SMT is enabled. Any SMT options used with the “l1tf” kernel parameter option overrides this “nosmt” option.


SMT can also be controlled through sysfs:

/sys/devices/system/cpu/smt/control

This file allows to read the current control state and allows to disable or (re)enable SMT.

Possible states are:

on

SMT is supported and enabled.

off

SMT is supported, but disabled. Only primary SMT threads can be onlined.

forceoff

SMT is supported, but disabled. Further control is not possible.

notsupported

SMT is not supported.

Potential values that can be written into this file:

on

off

forceoff

/sys/devices/system/cpu/smt/active

This file contains the state of SMT, if it is enabled and active, where active means that multiple threads run on 1 core.

Xen

For Xen hypervisor environments, mitigation is enabled by default and varies based on guest type. Manual adjustment of the “smt=” parameter is recommended, but the remaining parameters are best left at default values.A description of all relevant parameters are provided in the event any changes are necessary.

PV guests achieve mitigation at the Xen Hypervisor level. If a PV guest attempts to write an L1TF-vulnerable PTE, the hypervisor will force shadow mode and prevent the vulnerability. PV guests which fail to switch to shadow mode (e.g. due to a memory shortage at the hypervisor level) are intentionally crashed.

pv-l1tf=[ <bool>, dom0=<bool>, domu=<bool> ]

By default, pv-l1tf is enabled for DomU environments and, for stability and performance reasons, disabled for Dom0.

HVM guests achieve mitigation through a combination of L1D flushes, and disabling SMT.

spec-ctrl=l1d-flush=<bool>

This parameter determines whether or not the Xen hypervisor performs L1D flushes on VMEntry. Regardless of this setting, this feature is virtualized and passed to HVM guests for in-guest mitigation.

smt=<bool>
This parameter can be used to enable/disable SMT from the hypervisor. Xen environments hosting any untrusted HVM guests, or guests not under the full control of the host admin, should either disable SMT (through BIOS or smt=<bool> means), or ensure HVM guests use shadow mode (hap=0) in order to fully mitigate L1TF. It is also possible to reduce the risk of L1TF through the use of CPU pinning, custom CPU pools and/or soft-offlining of some hyper-threads.
These approaches are beyond the scope of this TID, but are documented in the standard Xen documentation.

WARNING – The combination of Meltdown mitigation (KPTI) and shadow mode on hardware which supports PCID can result in a severe performance degradation.

NOTE – Efforts are ongoing to implement scheduling improvements that allow hyper-thread siblings to be restricted to threads from a single guest. This will reduce the exposure of L1TF, and the requirement to disable SMT in many environments.

Related:

  • No Related Posts

AppLayering 4.x – OS layer creation fails with error “An error occurred. The required file ‘OSType.txt’ was not found on your OS disk. Please obtain the application layering OS machine tools, run ‘Setup_x64.exe’ on your OS image”

AppLayering only supports MBR and Hyper-V Generation 1 machines. It does not supports GPT and Generation 2 machines.

Capture ELM logs and in camlogfile we could see, if GPT is being used.

2017-11-17 11:43:46,085 INFO Threadpool worker Shell: EXEC: Device Boot Start End Blocks Id System

2017-11-17 11:43:46,085 INFO Threadpool worker Shell: EXEC: /dev/nbd1279p1 1 4294967295 2147483647+ ee GPT

Note: App Layering will only look at the first partition or two, so if we are running setup64.exe and it’s saving on a 3rd or 4th partition, it’ll never look at that partition by design, so it will never find ostype.txt there.

Related:

  • No Related Posts

How to Perform Reverse Imaging on a Provisioning Services Target Device for Windows and its Applicable Usages

When a Provisioning Services Target Device for Windows is booted from Provisioning Services (across the network), it is not possible to perform any software updates that affect the network stack, since the network stack changes will drop the connection to the vDisk.

The following provides a list of known network affecting software that periodically requires updating, this is not necessarily a complete list:

  • Hypervisor Tools/NIC Drivers (e.g. VMware Tools, XenServer Tools, VirtIO, etc.)
  • Provisioning Services Target Device Software for Windows – If the Provisioning Services Target Device Software for Windows is version 7.6.1 or newer, then reverse imaging is no longer needed to update the Provisioning Services Target Device Software. In this case, create a new maintenance version of your vDisk, boot it, and run the new Provisioning Services target device installer to do an in-place upgrade.
  • Windows 10 SAC releases upgrades
  • Antivirus definition updates
  • Firewall/Network security software

To update network stack-affecting software, you must first convert (clone) the Provisioning Services vDisk to a traditional virtual machine local disk. The process to convert from vDisk to local disk is sometimes called Reverse Imaging. Once booted from local disk (without going through the network), you can do whatever you want with the NIC. In this state, it’s just a regular virtual machine and no longer connected to the Provisioning Services server.

After Provisioning Services target device software is uninstalled and the system is rebooted to local disk, proceed to upgrade hypervisor tools, NIC driver, Provisioning services target device software, Windows 10, or update antivirus definitions.

Related:

  • No Related Posts

7022293: Error ‘CDB: Unmap/Read sub-channel TIMEOUT_ERROR’ on Nutanix Virtual Machines

This document (7022293) is provided subject to the disclaimer at the end of this document.

Environment


SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2)

SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

Situation

This error is observed when performing fstrim of xfs file system LUN presented by Nutanix Acropolis hypervisor (AHV):

CDB: Unmap/Read sub-channel TIMEOUT_ERROR

Resolution

Fixed in:-

SUSE Linux Enterprise Server Service Pack 2: kernel-default-4.4.103-92.53.1

SUSE Linux Enterprise Server Service Pack 3: kernel-default-4.4.92-6.18.1

Cause

Unaligned SCSI unmap requests from fstrim cause disk congestion, I/O aborts and in extreme cases virtual machine hangs.

Disclaimer

This Support Knowledgebase provides a valuable tool for NetIQ/Novell/SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented “AS IS” WITHOUT WARRANTY OF ANY KIND.

Related:

  • No Related Posts

Re: Hyper-V Disaster Recovery with DELLEMC Unity

VMware uses Storage Replication Adapters, or plugins that specifically can handle using storage-based replication to secure the virtual machines, and handle the failover.

When looking at Hyper-V, the only materials that I could find were these:

https://docs.microsoft.com/en-us/azure/site-recovery/hyper-v-vmm-disaster-recovery

Look at page 38 of this guide:

https://www.emc.com/collateral/white-papers/h12557-storage-ms-hyper-v-virtualization-wp.pdf

Which basically makes it sound like Hyper-V Replica handles the replication on a per-guest/VHD basis. Alternatively you might want to look at Veeam or Zerto for VM-level recovery, but still it’s not storage-level replication.

You could also look at EMC Storage Integrator (ESI):

https://www.emc.com/data-center-management/storage-integrator-for-windows-suite.htm#!interoperability

I’m not saying that using storage-level replication won’t work. Maybe it will, but I can’t find anything that specifically says it will, or that seems to make it easy like it is with VMware.

~Chris

Related:

  • No Related Posts

Hotfix XS74E001 – For XenServer 7.4

Who Should Install This Hotfix?

This is a hotfix for customers running XenServer 7.4. All customers who are affected by the issues described in CTX234679 – Citrix XenServer Multiple Security Updates should install this hotfix.

Information About this Hotfix

Component Details
Prerequisite None
Post-update tasks Restart Host
Content live patchable* No
Baselines for Live Patch N/A
Revision History Published on May 08, 2018
* Available to Enterprise Customers.

Issues Resolved In This Hotfix

This security hotfix addresses the vulnerabilities as described in the Security Bulletin above.

Installing the Hotfix

Customers should use either XenCenter or the XenServer Command Line Interface (CLI) to apply this hotfix. When the installation is complete, see the Post-update tasks in the table Information About this Hotfix for information about any post-update tasks you should perform for the update to take effect. As with any software update, back up your data before applying this update. Citrix recommends updating all hosts within a pool sequentially. Upgrading of hosts should be scheduled to minimize the amount of time the pool runs in a “mixed state” where some hosts are upgraded and some are not. Running a mixed pool of updated and non-updated hosts for general operation is not supported.

Note: The attachment to this article is a zip file. It contains the hotfix update package only. Click the following link to download the source code for any modified open source components XS74E001-sources.iso. The source code is not necessary for hotfix installation: it is provided to fulfill licensing obligations.

Installing the Hotfix by using XenCenter

Choose an Installation Mechanism

There are three mechanisms to install a hotfix:

  1. Automated Updates
  2. Download update from Citrix
  3. Select update or Supplemental pack from disk

The Automated Updates feature is available for XenServer Enterprise Edition customers, or to those who have access to XenServer through their XenApp/XenDesktop entitlement. For information about installing a hotfix using the Automated Updates feature, see the section Applying Automated Updates in the XenServer 7.4 Installation Guide.

For information about installing a hotfix using the Download update from Citrix option, see the section Applying an Update to a Pool in the XenServer 7.4 Installation Guide.

The following section contains instructions on option (3) installing a hotfix that you have downloaded to disk:

  1. Download the hotfix to a known location on a computer that has XenCenter installed.
  2. Unzip the hotfix zip file and extract the .iso file
  3. In XenCenter, on the Tools menu, select Install Update. This displays the Install Update wizard.
  4. Read the information displayed on the Before You Start page and click Next to start the wizard.
  5. Click Browse to locate the iso file, select XS74E001.iso and then click Open.
  6. Click Next.
  7. Select the pool or hosts you wish to apply the hotfix to, and then click Next.
  8. The Install Update wizard performs a number of update prechecks, including the space available on the hosts, to ensure that the pool is in a valid configuration state. The wizard also checks whether the hosts need to be rebooted after the update is applied and displays the result.
  9. Follow the on-screen recommendations to resolve any update prechecks that have failed. If you want XenCenter to automatically resolve all failed prechecks, click Resolve All. When the prechecks have been resolved, click Next.

  10. Choose the Update Mode. Review the information displayed on the screen and select an appropriate mode.
  11. Note: If you click Cancel at this stage, the Install Update wizard reverts the changes and removes the update file from the host.

  12. Click Install update to proceed with the installation. The Install Update wizard shows the progress of the update, displaying the major operations that XenCenter performs while updating each host in the pool.
  13. When the update is applied, click Finish to close the wizard.
  14. If you chose to carry out the post-update tasks, do so now.

Installing the Hotfix by using the xe Command Line Interface

  1. Download the hotfix file to a known location.
  2. Extract the .iso file from the zip.
  3. Upload the .iso file to the Pool Master by entering the following commands:

    (Where -s is the Pool Master’s IP address or DNS name.)

    xe -s <server> -u <username> -pw <password> update-upload file-name=<filename>XS74E001.iso

    XenServer assigns the update file a UUID which this command prints. Note the UUID.

    85893d4f-4ab6-460d-9074-ebd4847078c0

  4. Apply the update to all hosts in the pool, specifying the UUID of the update:

    xe update-pool-apply uuid=<UUID_of_file>

    Run the following command if you would like to apply the hotfix for a individual host:

    xe update-apply host-uuid=<UUID_of_host> uuid=<UUID_of_file>

    Alternatively, if you need to update and restart hosts in a rolling manner, you can apply the update file to an individual host by running the following:

    xe upload-apply host-uuid=<UUID_of_host> uuid=<UUID_of_file>

  5. Verify that the update was applied by using the update-list command.

    xe update-list -s <server> -u root -pw <password> name-label=XS74E001

    If the update is successful, the hosts field contains the UUIDs of the hosts to which this patch was successfully applied. This should be a complete list of all hosts in the pool.

  6. If the hotfix is applied successfully, carry out any specified post-update task on each host, starting with the master.

Files

Hotfix File

Component Details
Hotfix Filename XS74E001.iso
Hotfix File sha256 82a41f4610ae03051215df0a22baba5c374eb868ecfe49711737ca93859893dc
Hotfix Source Filename XS74E001-sources.iso
Hotfix Source File sha256 fe721e2fe6b8453bcc707981e3b214f052d8c8abb653964541c6487253477107
Hotfix Zip Filename XS74E001.zip
Hotfix Zip File sha256 3d4f51cbcd76155ae1ec521f4ecee3ffef40c40f1f42d8abc8804915f518fe25
Size of the Zip file 5.59 MB

Files Updated

microcode_ctl-2.1-22.xs1.x86_64.rpm
xen-dom0-libs-4.7.5-4.2.x86_64.rpm
xen-dom0-tools-4.7.5-4.2.x86_64.rpm
xen-hypervisor-4.7.5-4.2.x86_64.rpm
xen-libs-4.7.5-4.2.x86_64.rpm
xen-tools-4.7.5-4.2.x86_64.rpm

More Information

If you experience any difficulties, contact Citrix Technical Support.

Related:

  • No Related Posts

Hotfix XS73E004 – For XenServer 7.3

Who Should Install This Hotfix?

This is a hotfix for customers running XenServer 7.3.

All customers who are affected by the issues described in CTX234679 – Citrix XenServer Multiple Security Updates should install this hotfix.

Information About this Hotfix

Component Details
Prerequisite None
Post-update tasks* Restart Host
Content live patchable** No
Baselines for Live Patch N/A
Revision History Published on May 8, 2018
* Important: If you have previously disabled microcode loading on your XenServer host or pool. You must enable microcode loading again after applying this hotfix. For more information, see How to disable microcode loading on a XenServer pool.
** Available to Enterprise Customers.

Issues Resolved In This Hotfix

This security hotfix addresses the vulnerabilities as described in the Security Bulletin above.

This hotfix also includes the following previously released hotfix:

Installing the Hotfix

Customers should use either XenCenter or the XenServer Command Line Interface (CLI) to apply this hotfix. When the installation is complete, see the Post-update tasks in the table Information About this Hotfix for information about any post-update tasks you should perform for the update to take effect. As with any software update, back up your data before applying this update. Citrix recommends updating all hosts within a pool sequentially. Upgrading of hosts should be scheduled to minimize the amount of time the pool runs in a “mixed state” where some hosts are upgraded and some are not. Running a mixed pool of updated and non-updated hosts for general operation is not supported.

Note: The attachment to this article is a zip file. It contains the hotfix update package only. Click the following link to download the source code for any modified open source components XS73E004-sources.iso. The source code is not necessary for hotfix installation: it is provided to fulfill licensing obligations.

Installing the Hotfix by using XenCenter

Choose an Installation Mechanism

There are three mechanisms to install a hotfix:

  1. Automated Updates
  2. Download update from Citrix
  3. Select update or Supplemental pack from disk

The Automated Updates feature is available for XenServer Enterprise Edition customers, or to those who have access to XenServer through their XenApp/XenDesktop entitlement. For information about installing a hotfix using the Automated Updates feature, see the section Applying Automated Updates in the XenServer 7.3 Installation Guide.

For information about installing a hotfix using the Download update from Citrix option, see the section Applying an Update to a Pool in the XenServer 7.3 Installation Guide.

The following section contains instructions on option (3) installing a hotfix that you have downloaded to disk:

  1. Download the hotfix to a known location on a computer that has XenCenter installed.
  2. Unzip the hotfix zip file and extract the .iso file
  3. In XenCenter, on the Tools menu, select Install Update. This displays the Install Update wizard.
  4. Read the information displayed on the Before You Start page and click Next to start the wizard.
  5. Click Browse to locate the iso file, select XS73E004.iso and then click Open.
  6. Click Next.
  7. Select the pool or hosts you wish to apply the hotfix to, and then click Next.
  8. The Install Update wizard performs a number of update prechecks, including the space available on the hosts, to ensure that the pool is in a valid configuration state. The wizard also checks whether the hosts need to be rebooted after the update is applied and displays the result.
  9. Follow the on-screen recommendations to resolve any update prechecks that have failed. If you want XenCenter to automatically resolve all failed prechecks, click Resolve All. When the prechecks have been resolved, click Next.

  10. Choose the Update Mode. Review the information displayed on the screen and select an appropriate mode.
  11. Note: If you click Cancel at this stage, the Install Update wizard reverts the changes and removes the update file from the host.

  12. Click Install update to proceed with the installation. The Install Update wizard shows the progress of the update, displaying the major operations that XenCenter performs while updating each host in the pool.
  13. When the update is applied, click Finish to close the wizard.
  14. If you chose to carry out the post-update tasks, do so now.

Installing the Hotfix by using the xe Command Line Interface

  1. Download the hotfix file to a known location.
  2. Extract the .iso file from the zip.
  3. Upload the .iso file to the Pool Master by entering the following commands:

    (Where -s is the Pool Master’s IP address or DNS name.)

    xe -s <server> -u <username> -pw <password> update-upload file-name=<filename>XS73E004.iso

    XenServer assigns the update file a UUID which this command prints. Note the UUID.

    ba963670-fd85-4530-82c1-a1f70c401e34

  4. Apply the update to all hosts in the pool, specifying the UUID of the update:

    xe update-pool-apply uuid=<UUID_of_file>

    Run the following command if you would like to apply the hotfix for a individual host:

    xe update-apply host-uuid=<UUID_of_host> uuid=<UUID_of_file>

    Alternatively, if you need to update and restart hosts in a rolling manner, you can apply the update file to an individual host by running the following:

    xe upload-apply host-uuid=<UUID_of_host> uuid=<UUID_of_file>

  5. Verify that the update was applied by using the update-list command.

    xe update-list -s <server> -u root -pw <password> name-label=XS73E004

    If the update is successful, the hosts field contains the UUIDs of the hosts to which this patch was successfully applied. This should be a complete list of all hosts in the pool.

  6. If the hotfix is applied successfully, carry out any specified post-update task on each host, starting with the master.

Files

Hotfix File

Component Details
Hotfix Filename XS73E004.iso
Hotfix File sha256 8888537f03ce37a90fe29ee03fa5e33f19d9bc180abd889b709f8550c6ef6421
Hotfix Source Filename XS73E004-sources.iso
Hotfix Source File sha256 a8f9ed98a1fdf8b8b42f009291503ee91ea622f8ca040cfc82e00fdc8899a0f0
Hotfix Zip Filename XS73E004.zip
Hotfix Zip File sha256 fb3979beeeddaeceb879db9a259a5ff5fa2c2b1ab5922ca924e380cb402c8172
Size of the Zip file 31.65 MB

Files Updated

linux-firmware-20170622-3.noarch.rpm
dracut-033-360.el7.centos.xs13.x86_64.rpm
dracut-network-033-360.el7.centos.xs13.x86_64.rpm
xen-libs-4.7.5-3.4.x86_64.rpm
xen-tools-4.7.5-3.4.x86_64.rpm
xen-hypervisor-4.7.5-3.4.x86_64.rpm
microcode_ctl-2.1-22.xs1.x86_64.rpm
xen-dom0-tools-4.7.5-3.4.x86_64.rpm
xen-dom0-libs-4.7.5-3.4.x86_64.rpm

More Information

If you experience any difficulties, contact Citrix Technical Support.

Related:

  • No Related Posts

Windows Media Redirection Fails When Playing .MTS Videos Through VDA

Launched a Server OS VDA session and opened Windows Media Player to play a .MTS format video.

Windows Media redirection failed and resulted in a “Windows media player cannot play the file.” error.

User-added image

The issue was not observed over RDP or in the hypervisor console for the server.

With the Citrix Windows Media redirection policy disabled, the video played fine on the server.

The issue was observed with XenApp 7.6.0, 7.6.200 & 7.6.300.

Related:

  • No Related Posts

How to Perform Reverse Image in Provisioning Services Target Device for Windows and its Applicable Usages

When a Provisioning Services Target Device for Windows is booted from Provisioning Services (across the network), it is not possible to perform any software updates that affect the network stack, since the network stack changes will drop the connection to the vDisk.

The following provides a list of known network affecting software that periodically requires updating, this is not necessarily a complete list:

  • Hypervisor Tools/NIC Drivers (e.g. VMware Tools, XenServer Tools, VirtIO, etc.)
  • Provisioning Services Target Device Software for Windows – If the Provisioning Services Target Device Software for Windows is version 7.6.1 or newer, then reverse imaging is no longer needed to update the Provisioning Services Target Device Software. In this case, create a new maintenance version of your vDisk, boot it, and run the new Provisioning Services target device installer to do an in-place upgrade.
  • Windows 10 SAC releases upgrades
  • Antivirus definition updates
  • Firewall/Network security software

To update network stack-affecting software, you must first convert (clone) the Provisioning Services vDisk to a traditional virtual machine local disk. The process to convert from vDisk to local disk is sometimes called Reverse Imaging. Once booted from local disk (without going through the network), you can do whatever you want with the NIC. In this state, it’s just a regular virtual machine and no longer connected to the Provisioning Services server.

After Provisioning Services target device software is uninstalled and the system is rebooted to local disk, proceed to upgrade hypervisor tools, NIC driver, Provisioning services target device software, Windows 10, or update antivirus definitions.

Related:

  • No Related Posts