Randomly some of the Power Managed Server VDA's receive a shutdown request approximately 20 mins after the scheduled reboot and remain Turned Off

Randomly some of the Power Managed Server VDA’s receive a shutdown request approximately 20 mins after the scheduled reboot and do not power on automatically and remain Turned Off.

Scout logs detect the following:

Some Power Actions are marked as failures by Broker possibly because VDAs are not registering within the timeout. XD Controller may shutdown these VDAs due to multiple registration failures. MaxRegistrationDelayMin timeout may need to be tweaked for some environments. Refer to http://support.citrix.com/article/CTX138738 for the registry keys that can be tweaked.For other Registration failures refer to https://support.citrix.com/article/CTX126992

As per the event logs on Delivery Controllers, the Scheduled shutdown shows the origin:Schedule. The second shutdown that is triggered due to registration timeout shows origin:Policy.

Below is the sequence of events logged on the Delivery Controllers.

04/12/2017 04:05:31 xxxxxxx 3013 Citrix Broker Service The Citrix Broker Service successfully performed power action “Shutdown” (origin: Schedule) on virtual machine “yyyyyyy”. Information

04/12/2017 04:06:11 xxxxxxx 3013 Citrix Broker Service The Citrix Broker Service successfully performed power action “TurnOn” (origin: Schedule) on virtual machine “yyyyyyy”. Information


2017-04-12 04:30:16 xxxxxxx 3015 Citrix Broker Service The Citrix Broker Service marked a power action as failed because virtual machine “yyyyyyy” failed to register with the site after being started, restarted, or resumed. Information


04/12/2017 04:30:18 xxxxxxx 3012 Citrix Broker Service The Citrix Broker Service detected that power action “Shutdown” (origin: Policy) on virtual machine “yyyyyyy” failed.

This problem is most likely due to a host issue. Check that the configuration of the virtual machine on the host does not prohibit the requested power action. Verify that there are no storage problems on the host. Check that the host connection information is correct.

Error details:

Exception “Operation not available due to missing tools” of type “PluginUtilities.Exceptions.OperationNotAvailableException”. Warning


04/12/2017 04:30:25 xxxxxxx 3013 Citrix Broker Service The Citrix Broker Service successfully performed power action “TurnOff” (origin: Policy) on virtual machine “yyyyyyy”. Information

CDF Logs show below

04:23:26:40117,4972,4460,-1,FailOverdueRegistration(xxxxxxxxxx): Enqueue shutdown action result: Success, 1 failed registrations”,””

Related:

  • No Related Posts

How to Configure LHC to use Custom Port

Follow below steps to configure Citrix Broker and Citrix High Availability services to use custom port.

• Modify the default storefront port of Citrix Broker service via cmd or PowerShell by using below command

User-added image
• Modify the storefront port of High Availability service to be same as that of Broker service. Although SF port for this gets updated automatically but still change it explicitly by using below command.

User-added image
• Launching Citrix studio will prompt you for “Automatic Site Upgrade” and continuing with the upgrade should update the broker service port changes returning with 1 successful task.

User-added image

• Follow the above steps on all the delivery controllers and complete automatic site upgrade via Citrix studio.

• Run netsh http show urlacl on all the DDCs and ensure you see Reserved URLs of all the broker WCF endpoints for CitrixHighAvailabilityService and they are in ‘Listening’ state.

Example:


• Modify the Storefront GUI port via SF console to match with the Storefront port configured for Citrix Broker and Citrix HA services.

User-added image
• Validate and confirm, you are able to enumerate and launch applications successfully.

• Switch to LHC outage mode and monitor the events on DDC to confirm secondary broker service has taken over to serve user requests.

• Validate and confirm, you are able to enumerate and launch applications successfully in LHC outage mode.

LHC Outage Mode:

During the period directly after the database connection is lost, resource (Application/Desktop) enumerations in StoreFront and Session launches may fail. In practice it takes ~2 minutes before the site starts working (Resources enumerating and session launching). The site will not be fully functional until all the machines have re-registered which may take up to 10 minutes (You can decrease this interval via registry) . Recovery from the outage follows a similar flow.

Step Timeline Description
1 0 seconds DDCs lose connectivity to the Site Database.
2 0 – 60 seconds Services are unable to contact site database and report errors in the event log.
3 120 seconds Broker Service hands over the XML communication to the High Availability Service (Secondary Broker).
4 120 seconds StoreFront is no longer able to communicate with the Secondary DDCs. All Enumerations are going to the Primary DDC.
5 120 – 600 seconds VDAs begin to register with the Secondary Broker on the Primary DDC.
6 120 – 600 seconds Site is ready.

The process of exiting outage

Step Timeline Description
1 0 seconds DDCs gain connectivity to the Site Database.
2 0 – 60 seconds Services connect to the Site Database and report connection in the event log.
3 120 seconds High Availability Service (Secondary Broker) hands over the XML communication to the Broker Service.
4 120 seconds StoreFront is able to communicate with all the DDCs
5 120-600 seconds VDAs begin to register with the Broker Service a DDC in their List of DDCs.
6 120-600 seconds Site is ready.

Related:

VDAs will not register due to WMI Repository Corruption

Broker agent Won’t start, crashing after power outtage on network switch.

Exception:

System.Management.ManagementException Invalid class

at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)

at System.Management.ManagementObjectCollection.ManagementObjectEnumerator.MoveNext()

at Citrix.MetaInstaller.CoreUtilities.IsVirtualMachine(VMType type)

at Citrix.MetaInstaller.MetaInstallerApplication.Run(String[] args)

at Citrix.MetaInstaller.MetaInstallerApplication.InstallResultMain(String[] args)


Application: BrokerAgent.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.Reflection.TargetInvocationException

Stack:

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at Citrix.Cds.PluginLoader.PluginProxy.StartPlugin()

at Citrix.Cds.PluginLoader.PluginProxy.StartPlugin()

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

at System.Threading.ThreadHelper.ThreadStart()

Faulting application name: BrokerAgent.exe, version: 7.1.0.4019, time stamp: 0x5255694d

Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb1677

Exception code: 0xe0434352

Fault offset: 0x000000000000940d

Faulting process id: 0x1988

Faulting application start time: 0x01d019c5969618fd

Faulting application path: C:Program FilesCitrixVirtual Desktop AgentBrokerAgent.exe

Faulting module path: C:Windowssystem32KERNELBASE.dll

Report Id: d5d91837-85b8-11e4-86a5-b499babae614

Related:

  • No Related Posts

How to enable debug logging for Citrix WEM Cloud Authentication Service and Citrix WEM Cloud Messaging Service on Cloud Connectors

The Workspace Environment Management (WEM) service is a Citrix Workspace product.

Similar to on-premise WEM, the WEM Service Agent needs to connect to the WEM service Broker.

In order to do so, the WEM Service Agent must first request the WEM service Broker’s URL and a service key from a Citrix Cloud Connector (CCC) before it can connect to the WEM service Broker.

After retrieving the Broker URL and service key from the CCC, the Agent connects to the WEM service Broker using the provided URL. The Agent presents the service key, the WEM service Broker validates the service key, and communications between the Agent and WEM service Broker commence.

The Citrix WEM Cloud Authentication Service on the CCC handles the Broker URL/service key requests.

Note: After Agent to Broker comms has completed, the service key is no longer valid, and so the Agent must request a new WEM service Broker URL and service key, each time it needs to communicate with the WEM service Broker.

After enabling debug logging for the Citrix WEM Cloud Authentication Service on the CCC, the log file only contains entries related to these Agent requests. Refer to the instructions below.

Related:

CitrixBrokerService event 1152 after importing additional license file

Please check if following messages are outputted by Citrix Broker Service after above error/warining:

EventID 1150

Message:The Citrix Broker Service successfully contacted the license server ‘xxxxxxxx’.

​EventID 1156

Message:The Citrix Broker Service is successfully communicating with the license server ‘xxxxxxxx’. This controller is no longer in an emergency licensing grace period.

If yes, that means Citrix Broker Service successfully communicating with license server, and above error/warining can be safely ignored, no action needed.

Related: