How to Ensure Citrix ADC Traces will Decrypt

The intent of this article is to provide instructions to ensure your Citrix ADC traces will decrypt.

Background

There are several configuration settings for SSL Vservers and ServiceGroups that must be made in order to ensure they decrypt, as well as a few other requirements when taking traces. We also provide recommendations on what additional information to provide and how to take the trace so it provides relevant data.

Instructions to ensure NetScaler traces will decrypt

When taking a NetScaler trace, you must start by determining what you will be taking a trace of, and all the components involved. For example, a Content Switch may redirect to a Load Balancing Vserver with a Service Group, or a Gateway Vserver might point it’s Session Profile to the Load Balancing Vserver, which uses a Service Group. We typically recommend removing the Load Balancer to simplify troubleshooting, however, if that is not possible because you suspect the Load Balancer as part of the problem, then you have 3 items to modify to ensure decryption in the second example: The Gateway Vserver, the Load Balance Vserver, and the Load Balance Service Group. We will cover the Gateway Vserver, Load Balancing Vserver, and Service Groups. The steps are the same for Content Switches and other SSL Vservers.

  1. Let us start with the Gateway Vserver. Modify the SSL Parameters section to those depicted below.
    • Disable DH param *
    • Disable DH Key Expire Size *
    • Disable Ephemeral RSA
    • Disable Session Reuse
    • Disable TLS 1.2, but ensure TLS 1.1 and/or 1.0 is enabled. **

      User-added image

  2. Now let’s adjust the Load Balancing SSL Vserver in the same way. The settings are the same as for a Gateway Vserver.
    • Disable DH param *
    • Disable DH Key Expire Size *
    • Disable Ephemeral RSA
    • Disable Session Reuse
    • Disable TLS 1.2, but ensure TLS 1.1 and/or 1.0 is enabled. **

      User-added image

  3. Finally, we adjust the Service Group. Note that if using individual Services, you have to repeat this step for each Service, so it would be easier to move to using a Service Group. Additionally, if using an SSL Profile, you will have to adjust the SSL Profile, either for each Service or for the Service Group
    • Service Group SSL Parameters (no SSL Profile)
      • Disable Session Reuse
      • Disable TLS 1.2, but ensure TLS 1.1 and/or TLS 1.0 is enabled. **

        User-added image

    • Service Group SSL Profile or Individual Services
      • Disable DH param *
      • Disable DH Key Expire Size *
      • Disable Ephemeral RSA
      • Disable Session Reuse
      • Disable TLS 1.2, but ensure TLS 1.1 and/or 1.0 is enabled. **

        User-added image

  4. Now we need to set up the trace. These are the settings we want on the trace modified, leave all other settings as they are:
    • Packet Size: 0
    • File Size: 0
    • Trace Filtered connections peer traffic: Checked
    • Capture SSL Master Keys: Checked ***
    • Click Start. You will receive a warning. Click Ok.
    • You will receive notice the trace started.
  5. Now that the trace has started, open a NEW Incognito browser window. This is so the SSL Client Hello is not missed. If using a Mobile device, FULLY CLOSE all apps before beginning, for the same reason. This is very important. If we don’t capture the beginning of the SSL handshake, we cannot decrypt. So Ensure the trace is started BEFORE accessing a login webpage or opening any mobile app.

  6. Login and duplicate the issue. Once the issue has been duplicated, WAIT 1-2 minutes, especially if the issue is some sort of interruption or loading error or ICA timeout. This is because often this results in a RESET packet being sent and if you don’t wait, we will miss the reset which can contain valuable information.

  7. Once the issue is replicated and you have waited as recommended, then stop the trace. Download both the sslkeys files (there can be more than 1) and the nstrace file.

  8. Gather IPs of all relevant devices: NetScaler, Client, Gateway URL, Backend Server, VDA if ICA traffic, Storefront, etc., and provide these.

  9. Now that you have the Traces and IPs, Generate a new Support file from the NetScaler. You want to do this after taking the traces. This allows us to correlate logs and counters from the support file with events in the traces and can be essential when troubleshooting complex issues.

    NOTES:

    * – DH – Using the “Capture SSL Master Keys” option in the trace often allows us to decrypt, but not always. If unable to disable DH, then you can proceed to take a trace but know there is a chance they will not decrypt.

    ** – TLS 1.2 is not required to disable, it’s just more likely to decrypt without it.

    *** – “Capture SSL Master Keys” option does Not capture private keys, they are session keys only. Capturing these session keys are the best way to ensure we can decrypt.

What is the purpose/impact of these settings

SSL Session Reuse

This is a mechanism that speeds up the SSL Transaction, which speeds up client communications. The impact of disabling is that some clients using SSL Reuse may have to reconnect to the SSL Session. However, we cannot always decrypt when Session Reuse is enabled.

DH parameters

DH refers to DIffie-Hellman, and ciphers which use DH often do not decrypt. To ensure we decrypt it’s best to disable them. Using the “Capture SSL Master Keys” option in the trace often allows us to decrypt, but not always. Users will reconnect if using DH ciphers when they are disabled.

Ephemeral RSA

Disabling this should prevent the NetScaler from trying to use ECC curve ciphers, which we cannot decrypt. Users will reconnect if using ECC Curves when disabled.

TLS1.2

This is not actually necessary to disable, however, we tend to have fewer decryption issues when disabling TLS 1.2. Users will have to reconnect if using TLS 1.2 when disabled.

Related:

Leave a Reply