7023078: Security Vulnerability: “L1 Terminal Fault” (L1TF) ??? Hypervisor Information (CVE-2018-3620, CVE-2018-3646, XSA-273).

Full mitigation for this issue requires a combination of hardware and software changes. Depending on the guest type, software changes may be required at both the Hypervisor and guest level.

Updated Intel microcode (provided through your hardware / BIOS vendor or by SUSE) introduces a new feature called “flush_l1d”. Hypervisors and bare-metal kernels use this feature to flush the L1 data cache during operations which may be susceptible to data leakage (e.g. when switching between VMs in Hypervisor environments).

Software mitigations exist for the Linux Kernel and for Hypervisors. These mitigations include support for new CPU features, passing these features to guests, and support for enabling/disabling/tuning the mitigations. Recommended mitigations vary depending on the environment.

For the Linux kernel (on both bare metal and virtual machines) L1TF mitigation is controlled through the “l1tf” kernel boot parameter. For complete information on this parameter, see TID 7023077.

KVM

For KVM host environments, mitigation can be achieved through L1D cache flushes, and/or disabling Extended Page Tables (EPT) and Simultaneous MultiThreading (SMT).

The L1D cache flush behavior is controlled through the “kvm-intel.vmentry_l1d_flush” kernel command line option:

kvm-intel.vmentry_l1d_flush=always

The L1D cache is flushed on every VMENTER.

kvm-intel.vmentry_l1d_flush=cond

The L1D cache is flushed on VMENTER only when there can be leak of host memory between VMEXIT and VMENTER. This could still leak some host data, like address space layout.

kvm-intel.vmentry_l1d_flush=never

Disables the L1D cache flush mitigation.

The default setting here is “cond”.

The l1tf “full” setting overrides the settings of this configuration variable.


L1TF can be used to bypass Extended Page Tables (EPT). To mitigate this risk, it is possible to disable EPT and use shadow pages instead. This mitigation is available through the “kvm-intel.enable_ept” option:
kvm-intel.enable_ept=0

The Extended Page tables support is switched off.
As shadow pages are much less performant than EPT, SUSE recommends leaving EPT enabled, and use L1D cache flush and SMT tuning for full mitigation.


To eliminate the risk of untrusted processes or guests exploiting this vulnerability on a sibling hyper-thread, Simultaneous MultiThreading (SMT) can be disabled completely.

SMT can be controlled through kernel boot command line parameters, or on-the-fly through sysfs:

On the kernel boot command line:

nosmt

SMT is disabled, but can be later reenabled in the system.

nosmt=force

SMT is disabled, and can not be reenabled in the system.

If this option is not passed, SMT is enabled. Any SMT options used with the “l1tf” kernel parameter option overrides this “nosmt” option.


SMT can also be controlled through sysfs:

/sys/devices/system/cpu/smt/control

This file allows to read the current control state and allows to disable or (re)enable SMT.

Possible states are:

on

SMT is supported and enabled.

off

SMT is supported, but disabled. Only primary SMT threads can be onlined.

forceoff

SMT is supported, but disabled. Further control is not possible.

notsupported

SMT is not supported.

Potential values that can be written into this file:

on

off

forceoff

/sys/devices/system/cpu/smt/active

This file contains the state of SMT, if it is enabled and active, where active means that multiple threads run on 1 core.

Xen

For Xen hypervisor environments, mitigation is enabled by default and varies based on guest type. Manual adjustment of the “smt=” parameter is recommended, but the remaining parameters are best left at default values.A description of all relevant parameters are provided in the event any changes are necessary.

PV guests achieve mitigation at the Xen Hypervisor level. If a PV guest attempts to write an L1TF-vulnerable PTE, the hypervisor will force shadow mode and prevent the vulnerability. PV guests which fail to switch to shadow mode (e.g. due to a memory shortage at the hypervisor level) are intentionally crashed.

pv-l1tf=[ <bool>, dom0=<bool>, domu=<bool> ]

By default, pv-l1tf is enabled for DomU environments and, for stability and performance reasons, disabled for Dom0.

HVM guests achieve mitigation through a combination of L1D flushes, and disabling SMT.

spec-ctrl=l1d-flush=<bool>

This parameter determines whether or not the Xen hypervisor performs L1D flushes on VMEntry. Regardless of this setting, this feature is virtualized and passed to HVM guests for in-guest mitigation.

smt=<bool>
This parameter can be used to enable/disable SMT from the hypervisor. Xen environments hosting any untrusted HVM guests, or guests not under the full control of the host admin, should either disable SMT (through BIOS or smt=<bool> means), or ensure HVM guests use shadow mode (hap=0) in order to fully mitigate L1TF. It is also possible to reduce the risk of L1TF through the use of CPU pinning, custom CPU pools and/or soft-offlining of some hyper-threads.
These approaches are beyond the scope of this TID, but are documented in the standard Xen documentation.

WARNING – The combination of Meltdown mitigation (KPTI) and shadow mode on hardware which supports PCID can result in a severe performance degradation.

NOTE – Efforts are ongoing to implement scheduling improvements that allow hyper-thread siblings to be restricted to threads from a single guest. This will reduce the exposure of L1TF, and the requirement to disable SMT in many environments.

Related:

  • No Related Posts

NetWorker 9.x: Unable to select VBA, the VBA list is blank for VMware backups in Workflow>Action

Article Number: 486491 Article Version: 4 Article Type: Break Fix



NetWorker 9.0,NetWorker 9.0.1,NetWorker 9.0.50

After a Policy is created, Policy’s Workflow & Action should be created using the wizard.

But in the second step of Action creation the list of VBAs is blank. Not allowing the policy to be created.

This is seen due to a mismatch of the Hypervisor resource in NetWorker Server and vCenter name in VBA.

For example:

nsradmin

p:type:NSR Hypervisor

name:xyzvcenter

username:abc

Password:*****

Above is the output of printing Hypervisor resource.

But when the VBA is printed:

p type:NSR VBA Server

name:xyzvba;

vcenter:
xyzvcenter.test.domain

As shown above, there is a mismatch between the NetWorker Server’s Hypervisor resource(created with a shortname of the vCenter), and VBA resource(has the FQDN of the vCenter).

There can be various configuration changes done to match the configuration between resources. These could involve:

1. Changing the ‘nsr hypervisor’ resource

OR

2. Re-deploying the VBA

NOTE: Below workarounds involve changing your resource database- it is recommended to make copies of ‘nsrresnsrdb’ directory before attempting changes.

Workaround 1: Change the Hypervisor resource:

  1. Delete the groups created with current vCenter name
  2. Delete the current resource:

    nsradmin

    > delete type: nsr hypervisor; name: vcenter-name


    Delete? Yes
  3. Create new resource that matches your VBA:

    > create type: nsr hypervisor; name: vcenter-name.test.domain

    create> Yes

    > p type:nsr hypervisor; name: vcenter-name.test.domain

    > update username: abc@test.domain

    update> yes

    > update password: test12

    update > yes

    > update proxy: nw-server-name

    update: yes

    > p type:nsr hypervisor; name: vcenter-name.test.domain
  4. Verify your resource has the right user/password
  5. Refresh the NMC > Configuration > VMware View
  6. Create new groups as needed

    You should now be able to assign policies with these new groups to VBAs

Workaround 2: If you have recently registered a new VBA and have other VBAs working fine- you may not want to remove ‘nsr hypervisor’ resource. In this case- you can choose to re-deploy your new VBA:

NOTE: This involves deleting your new VBA and its backup data from networker. If you have any backups taken through this new VBA- you will not able to restore them after below change

  1. Contact EMC Support for assistance with ‘decommission’ your VBA. See https://support.emc.com/kb/335878
  2. Deploy a new VBA and during the initial wizard- select the vCenter name that matches your networker’s ‘nsr hypervisor’ resource

Related:

  • No Related Posts

Netscaler VPX 1000 – Azure – Slowness getting through Netscaler.


With 12.0 builds, we have changed default yield behavior for PE vCPUs. vCPU will not yield to hypervisor, even though if there is less/moderate traffic in 12.0 build, which was not the case for 11.1 builds. That’s the reason, VPX vCPU is always 100% on hypervisor. However, vCPU is allocated to management core might not be 100%.

NetScaler yields PE vCPUs to hypervisor in sparse/moderate traffic cases. Since we have observed Tx overflow/congestion, it’s somewhat related to scheduling, we thought not yielding vCPU helps in improving the situation.

– set ns vpxparam -cpuyield NO

Upgrade to 12.0.53.X+

Related:

  • No Related Posts

New Revised Positioning for Our Comprehensive SDS Portfolio

We’ve listened to our customers and Partners, and we’re making some adjustments to simplify how we deliver and position our outstanding Software Defined Storage (SDS) portfolio.

Did you know that our VMware vSAN-based offerings now have a commanding lead of the SDS market and continues to experience exceptional growth? We’re very proud of this momentum and the opportunity it brings to our channel partners. The time is right to simplify our portfolio positioning, giving you a simpler story to tell your customers.

But does the market success of VMware vSAN mean ScaleIO is going away? Absolutely not. ScaleIO remains an important part of our portfolio. We’re simply making it easier for you to sell our portfolio, and easier for your customers to make the right purchase.



The new Dell EMC and VMware SDS strategy

Our simplified SDS positioning:

  • Lead with VMware vSAN-based solutions, VxRail and VxRack SDDC
  • Position VxRack FLEX for the following use cases: server SAN/2-tier architecture, heterogeneous hypervisor/bare metal, or high-performance database
  • Do not sell ScaleIO Ready Nodes and ScaleIO software to new customers, except in the case of large deals (e.g., >$2M software-only, >$10M software plus ScaleIO Ready Nodes) positioned specifically in use cases of server SAN/2-tier architecture, heterogeneous hypervisor/bare metal, or high-performance database
  • We will continue delivery of the ScaleIO roadmap: ScaleIO is an important component of VxRack FLEX

Additional sales guidance:

  • We will honor ScaleIO software and ScaleIO Ready Node deals in flight for Q1 close
  • We will offer a smaller entry point for VxRack Flex via an appliance in 2HFY19

Reassure your ScaleIO customers that:

  • The ScaleIO roadmap remains unaffected; we will continue to deliver important functionality, such as space efficiency.
  • Current customers can continue to purchase expansions, additions and upgrades, as well as receive support through the full lifecycle of their contracts

Looking ahead

While we’re continuing to support current ScaleIO customers, we also know that turnkey appliances and engineered systems provide the best customer and support experience for software-defined storage consumption. That’s why we encourage existing ScaleIO software and ScaleIO Ready Node customers and prospects to consider pivoting to VxRack FLEX.

VxRack FLEX is a go-forward offer in our portfolio and includes ScaleIO Software as its operating system. A number of large customers who started with ScaleIO software are now embracing the fully engineered experience to achieve better business outcomes.

With further questions, please get reach out to your designated Dell EMC contact.

Related:

  • No Related Posts

Does Netscaler VPX have support for ESXi 6.0 Patch 6 (Build number 6921384) as hypervisor?

Question: Does Netscaler VPX have support for ESXi 6.0 Patch 6 (Build number 6921384) as hypervisor

Answer: Supported Hypervisors, Features, and Limitations

12.0: https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/supported-hypervisors-features-limitations.html

11.1: https://docs.citrix.com/en-us/netscaler/11-1/deploying-vpx/supported-hypervisors-features-limitations.html

Related:

  • No Related Posts

How to fix App Layering tasks stuck at “Stalled” status

In the management console’s tasks menu, open the task by clicking the ‘i’ icon to the left of the task’s name

Click on the “X” to cancel the subtask.

Click “Cancel All” to cancel all subtasks. This may need to be done twice for stubborn tasks.

This should send the task into a “Cancelling” state, and then a “Failed” state.

If “Cancelling” also stalls, please reboot the ELM appliance from the hypervisor.

Follow this article for more details on how to cancel a task – https://support.citrix.com/article/CTX225301

Related:

  • No Related Posts

FAQ: Personal vDisk in XenDesktop

Q: Can multiple PvDs be associated to a device/user?

A: There can only be one PvD per Virtual Machine. The PvD is assigned to a Virtual Machine when building the catalog of desktops. The pool type for a PvD catalog is a pooled static, which the desktop is assigned to the user on first use.

Q: Is the PvD a 1-1 mapping per user?

A: Actually, it is a 1:1 mapping to a Virtual Machine in a catalog, which is then assigned to the user on first use. A PvD is attached to a Virtual Machine assigned to the user. The administrator can move a PvD to a new virtual machine in a recovery situation.

Q: If you create a catalog for pooled with PvD, it does not mean that the user is always required to be assigned to that Virtual Machine defeating one of the benefits of a pooled?

A: The base image is still shared and updated across the pool. However, once the user makes an initial connection to a Virtual Machine, the Virtual Machine is kept assigned to the user.

Note: You must connect early in the starting stage long before you know who the user is in order to maximize the application compatibility for services, devices etc.

Q: How does the pooled with personal vDisk catalog affect idle pool?

A: After the user connects, this user is kept assigned to the Virtual Machine.

You must connect early in the starting stage long before you know who the user is in order to maximize the application compatibility for services, devices etc. So for hypervisor resource management, instead of idle pool management, you would use power management to handle Virtual Machine idle workloads.

Q: What Operating Systems are supported for PvD?

A: Windows 7 x86, Windows 7 x64, and Windows 10 up to v1703.

Q: Is PvD only for Desktop Operating Systems or will it also work with Server Operating Systems?

A: It is only supported on Desktop Operating Systems.

Design and Deploy

Q: What kinds of risks are there for BSODs with PvDs?

A: PvD is architected to be compatible with a wide range of Windows software, including software that loads the drivers. However, drivers that load in phase 0 or software that alters the networking stack of the machine (through the installation of additional miniports or intermediate or protocol drivers) might cause PvD to not operate as expected. You must install these types of software in the base Virtual Machine image.

Related:

  • No Related Posts

App Layering – PVS Blue Screen SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (CVhdMp.sys)

The issue seems to be regarding the PVS template, the OS/Platform layer, or both.

Solution 1

The first scenario that was seen for this issue is outlined in Jira UNI-45882. The issue here is related to PVS requiring the VMXnet3 NIC to be in the correct PCI slot address of 192 on the PVS template. A PVS template is simply a VM in vSphere that is a VMware template. Any VM can be used for this but some of the known requirements are as follows. It must have a VMXnet3 NIC as PVS does not support E1000 and is very sensitive to VM’s that have had an E1000 assigned. The NIC must be using PCI address 192 and the SCSI controller should be on 160. To check this please have the template VM powered down (you may need to convert the template VM back to a normal VM first). Then select Edit Settings > Options > General > Configuration. The ethernet0.pciSlotNumber should be 192 and scsi0.pciSlotNumber should be 160. If you do not see the SCSI line then add a blank disk to the VM and power it on. It will boot to the PXE boot screen, just power it down at that point. You should now have an SCSI controller and see the configuration entries with the correct numbers. Make this VM a template again and the image should boot (there are a few other steps in PVS that I do not have documented but the customer should know how to apply the new template and boot from the Unidesk image).

Solution 2

In this case, the customer had two NICs in their PVS template. This is one of many valid PVS configurations. The first NIC is used for the PXE boot and likely will not have any additional network access beyond that. The second is for general data and should be used the same as your standard NIC when accessing the network. The template may still need to be modified per solution 1 before the image will boot.

The customer reported the following got them up and running:

  1. Created a new OS version. Booted the packaging VM and checked the NIC. Only one present but was connected to data vLAN, not PVS. Edited VM settings so this NIC was connected to PVS streaming LAN.
  2. Shut down VM. Checked config parameters to see ensure the ethernet0 was at pci192. It was.
  3. Booted VM. Clicked finalize.
  4. Finalised OS layer on ELM.
  5. Repeated steps 2-4 on platform layer that contains VDA and PVS drivers.
  6. Created a new template with OS new layer.
  7. Published to PVS with new platform layers.
  8. I didn’t have any PVS targets so I ran XenDesktop Setup Wizard to create a target. After initial automatic boot and shutdown to format the write cache I started VM and it booted.
  9. Logged in successfully and checked build. All components present and both NICs present and connected correctly.


NOTES:

I have a feeling you do not need to create a new OS layer as he did in step 1. Likely you only need to add a version to the platform layer and make the vLAN changes as he outlined. Our code is doing a lot in the platform layer regarding NIC merge and Unidesk 4 OS layers should be platform agnostic. Unidesk as of 4.0.3 and for the foreseeable future, by design, will only provide you with a single NIC when editing a layer. This is fine. The key seems to be that the packaging machine for the platform layer must have the “data” vLAN set and not the PVS PXE boot vLAN.

Solution 3

Make sure the hypervisor tools are installed and working. Check device manager and confirm the active NIC is the correct NIC. If you see a 3rd party NIC like Realtek or Intel instead of Xen PV device or VMXNet3, then the tools are not installed/working. Having the wrong NIC will cause this BSOD.

Solution 4

If the above solutions do not work, another option is to publish the image to your hypervisor and then use the PVS imaging wizard to upload the image to PVS.

Related:

  • No Related Posts

Error: “Power State Unknown” “CDS_EVENT_HOSTING_FAILED_POWER_ACTION” in XenDesktop

To update the correct host machine ID on the DDC, complete one of the following solutions:

Solution 1

Restart the Citrix Site services on all the DDCs.

Note: This may result in momentary disruptions of new connections, however current sessions are not affected.

Open PowerShell as admin and run the following commands:

Get-Service Citrix* | Stop-Service -Force

Get-Service Citrix* | Start-Service

Solution 2

This can be caused by changes made on the hypervisor to VM metadata. If the VM’s unique ID has changed then the XenDesktop database may be unaware of this UID mismatch. This process will verify the UID known to XenDesktop for the VMs and compare against the UID provided by the hypervisor.

Warning! Back up the XenDesktop database before completing these actions.

  1. Open DDC using the PowerShell console and run the following commands to display all machine IDs of the virtual machines from the hypervisor. .

    asnp Citrix*$ErrorActionPreference=ContinueGet-ChildItem -Path XdHyp: -force -recurse | ?{ $_.IsMachine } | Out-File –Filepath c:xdhyp.txt
  2. The xdhyp.txt output file contains the correct machine IDs from the hypervisor. Open that file and press Ctrl+F or Edit > Find. Search for the name of the Virtual Machine, in this case the name of the Virtual Machine is PVS0003.

    Example output

    PSPath : Citrix.Host.Admin.V1Citrix.Hypervisor::XDHyp:ConnectionsXenServerPVS0003.vmPSParentPath : Citrix.Host.Admin.V1Citrix.Hypervisor::XDHyp:ConnectionsXenServerPSChildName : PVS0003.vmPSDrive : XDHypPSProvider : Citrix.Host.Admin.V1Citrix.HypervisorPSIsContainer : TrueName : PVS0003FullName : PVS0003.vmObjectType : VmId : 7d1d6004-5319-7a7e-59cb-2662e212a3e5IsContainer : TrueIsMachine : TrueIsSnapshotable : TrueObjectPath : /PVS0003.vmFullPath : XDHyp:ConnectionsXenServerPVS0003.vmIsSymLink : FalseAdditionalData : {}

    Note: The machine ID is as follows:

    Id : 7d1d6004-5319-7a7e-59cb-2662e212a3e5.

    Your result will vary.

  3. Run the following command:

    Get-BrokerMachine -PowerState Unknown

    This identifies the machines that have the unknown power state.

    + Note the “HostedMachineId “ from the output.

    + Now comparing the “ID” from Step1, and the “HostedMachineId “ from this step, You’ll find that the IDs are different.

    + Correct “Id” is from the Step-1 and incorrect value is present in Database (from step-3)

    + We can also verify the same by Browsing the below tables in SQL site database, and confirming the values.

    Chb_Config.Workers >> HostedMachineId >>

    DesktopUpdateManagerSchema.ProvisionedVirtualMachine >> VMId >>

  4. Run the following command to change the XendDesktop Database’s record for the machine ID to match the Hypersvisor’s Machine ID:

    Set-BrokerMachine -MachineName ‘MyDomainMyMachine’ -HostedMachineId [machine ID from preceding output]

    This corrects the HostedMachineId for the problem machines using the ID that was retrieved from xdhyp.txt.

  5. Check Desktop Studio or Desktop Director and refresh the list of results.

    The power state must now match the state indicated in the hypervisor.

Note: It might be necessary to restart the Citrix Broker Service on all DDCs and/or restart the virtual machine.

Solution 3

Remove the affected virtual machines from the Desktop Group in Desktop Studio and add them again.

NOTE: Removing machines from an MCS catalog cannot be reversed. Once the VM is removed you will only be able to add that machine to a catalog of the “Existing” type.

Solution 4

Ensure the SCVMM console version and hotfix level, installed on the DDCs, is the same version and hotfix level as the SCVMM server.

For example: Install the upgraded version of the SCVMM Console, version 8, KB3097539 on both controllers, which matched the SCVMM server hotfix level.

Solution 5

Run “Get-BrokerHypervisorConnection”, and check the output for Hypervisor “state,

If for any hypervisor connection the state is, anything else then ON or OFF,

Then try to put that connection in maintenance mode for few secs and then turn off the maintenance mode again.

User-added image

Solution 6

Understanding of the Broker > Hypervisor communication:

  • The Broker service runs on every Delivery Controller in the site (DDC). It has many subcomponents, one of which is the Hosting Management sub component.
  • The broker service must communicate to the Hypervisor using the VM/Machine ID
  • The UUID/Machine ID of the VM can be obtained by running “Get-BrokerMachine” cmdlet from any of the DDC’s in the site.
  • It needs to match with the BIOS file of the VM on the hypervisor to be managed properly by DDC’s in the Site.

“Get-BrokerHypervisorConnection”

  • If the Certificate is updated on the VSphere server the same needs to be updated on all the DDC’s in the Site. Certificate mismatch can also cause the Broker to change the power states to “Unknown” and Hypervisor connection state to “Unavailable”.
  • If there is a Host/VSphere Server which is put under maintenance or is down for any reasons, the Broker will change the Power state to “Unknown” and Hypervisor connection state to “Unavailable”.
  • If there is an issue with broker service on one controller, broker service from other controller will serve as the Preferred controller to control the Power and pool for the site.

Steps to remediate the situation if the issue occurs:

  • If there is a network or VMware host issue which has corrected itself, the broker service won’t be able to re-establish the communication on its own if the disruption is for a longer period of time. In that case the broker service needs to be restarted on all the DDC’s in the site.
  • Alternatively you may run the command below cmdlet on any of DDC’s using Power Shell.
  • Update-HypHypervisorconnection – LiteralPath “The Actual Path of hypervisor Connection”
  • -LiteralPath of the Hypervisor connection can be obtained by running the below cmdlet on PowerShell of any DDC.

cd Xdhyp:

cd ./Connections

  • for instance: Update-HypHypervisorConnection -LiteralPath “XDHyp:connectionsConnection”
  • Alternatively you may do the following using Studio to update the connection:

Click on Hosting tab on Studio -> Right click the connection -> Click on Edit Connection -> Without making any changes click on OK.

Related:

  • No Related Posts

A way to use NAT network by using Oracle virtualBox

That is true.

Vmware is easy and confortable tools to make vitrual machines than Oracle virtual box. But Oracle vitrual box is free to use and vmware needs to buy license.

Now, here is a way to use NAT network by using Oracle virtualBox.

face_vbox.JPG.jpg

I am sorry that my interface is chinese words. I can do to make them to be understand easily.

Yes, When you first time finish install Oracle virtual box , you just see one vitrual network card .

host_network.JPG.jpg

See the the rounded rectangle, it is exist after fininshing install on your host machine network setting area.

You also can see just only one host only network setting for each guests machine.

vbox_network.JPG.jpg

The red line is the original one.

If we were let it go, for example , we had created two guest machines “linux3” and “linux4”. The “linux3” guest ip is 10..0.2.15 and the “linux4” ip is 10.0.2.25.

Then we just add extra one more vbox host-only network card on vitrual box console, like yellow line marked.. The we can see the yellow rounded rectangle marked network adapter show on host machine.

After that., we just need to find this two guest machines network adapters setting and set NAT port mapping rule. Just as follows.

guest_linux3_network.JPG.jpg

, guest_linux4_network.JPG.jpg

After all that finished, we are suprised that we can ssh from host machine to Oracle vitrual machine box.guest machine.

so we can see:

from_host1.JPG.jpg

from_host2.JPG.jpg

Then problem solved.

Thanks

Related:

  • No Related Posts