NTP status displays “No association ID ” error message on Secondary NetScaler

On the Secondary NetScaler, ” No association ID error” gets displayed when “Show NTP Status command ” is executed

Primary NetScaler Appliance:

=======================

> show ntp status

remote refid st t when poll reach delay offset jitter

=======================================================

adljj.john.com .LOCL. 1 u 9 64 7 0.293 -212012 2.175


Secondary NetScaler Appliance:

===========================

> show ntp status

No association ID’s returned

Done

Log Analysis:

==============

1) From the logs, we found that, NTP was configured after upgrade and during that time secondary device interface was down.

2) We can see that interface was down in the time interval of10:01 – 11:18 A.M. In that interval, none of the command gets propagated. Because of that ntp config was missing from secondary.

3) As per current design, even if the Secondary comes UP and the NTP configurations are Synchronized through HA Synchronization, we have to manually restart the NTP Daemon to get the NTP status on Secondary. Which is a current limitation on NetScaler.

4) Hence, Enhancement request was raised to address this limitation. 5) The limitation was fixed in the following versions: 12.1 50.x 12.0 60.x 11.1 60.x

Logs from Primary:

—————————–

var/log/ns.log

ns.log.0:649:Apr 23 10:15:59 <local0.info> X.X.X.X 2018:01:15:59 GMT NetScaler-Internal-TDC-01 0-PPE-1 : default GUI CMD_EXECUTED 136 0 : User nsroot – Remote_ip X.X.X.20 – Command “add ntp server X.X.X.3 -minpoll 6 -maxpoll 10 -devno 32833536” – Status “Success”

ns.log.0:651:Apr 23 10:15:59 <local0.info> X.X.X.X 04/23/2018:01:15:59 GMT NetScaler-Internal-TDC-01 0-PPE-1 : default GUI CMD_EXECUTED 137 0 : User nsroot – Remote_ip X.X.X.20 – Command “unset ntp server X.X.X.3 -autokey” – Status “Success”

Logs from secondary:

——————————–

var/log/ns.log

Apr 23 10:00:34 <local0.info> X.X.X.25 04/23/2018:01:00:34 GMT NetScaler-Internal-TDC-02 0-PPE-1 : default CLI CMD_EXECUTED 131 0 : User nsroot – Remote_ip 127.0.0.1 – Command “logout” – Status “Success”

Apr 23 10:01:13 <local0.notice> X.X.X.25 04/23/2018:01:01:13 GMT NetScaler-Internal-TDC-02 0-PPE-0 : default EVENT DEVICEDOWN 79 0 : Device “interface(0/1)” – State DOWN

Apr 23 10:01:13 <local0.notice> X.X.X.25 04/23/2018:01:01:13 GMT NetScaler-Internal-TDC-02 0-PPE-1 : default EVENT DEVICEDOWN 132 0 : Device “interface(0/1)” – State DOWN

Apr 23 11:18:15 <local0.notice> X.X.X.25 04/23/2018:02:18:15 GMT NetScaler-Internal-TDC-02 0-PPE-1 : default EVENT DEVICEUP 133 0 : Device “interface(0/1)” – State UP

Apr 23 11:18:15 <local0.notice> X.X.X.25 04/23/2018:02:18:15 GMT NetScaler-Internal-TDC-02 0-PPE-0 : default EVENT DEVICEUP 80 0 : Device “interface(0/1)” – State UP

Apr 23 11:18:29 <local0.info> X.X.X.25 04/23/2018:02:18:29 GMT NetScaler-Internal-TDC-02 0-PPE-1 : default AAA Message 134 0 : “rba authentication : user nsroot response_len-0 cmdPolicyLen-0, partitionLen-0 PromptLen-0 timeout 805307268 authPolicyLen-0 authActionLen-0 ssh_pubkey_len

Related:

The following error occurred during an authentication attempt for user:domain.comabc with realm:

At the Storefront server open a command line and run the following command:

>set u

There would be two fields called USERDOMAIN and USERDNSDOMAIN

And these will be like this:

USERDNSDOMAIN=DOMAIN.COM

USERDOMAIN=DOMAIN

Open Netscaler Gateway Virtual server session profile.

Go to Published applications tab and look for SSODomain field

As per the error it would be domain.com

We need to change it to domain, and save the configuration on Netscaler.

Also confirm that Storefront has either “Any” domain selected or has “domain.com” and “domain” added as trusted domain.

Related:

NetScaler GSLB Static Proximity Does Not Work After Upgrading to 11.0/11.1 Firmware

To resolve this issue delete the nslocation.* files from the /var/netscaler/locdb/ directory and then re-run the configuration to add the location file.

root@NS-Cumulus1# cd /var/netscaler/locdb/

root@NS-Cumulus1# ls

GeoIPCountryWhois.csv GeoLite2-City-Locations-en.csv IP2LOCATION-LITE-DB1.CSV nslocation.ck nslocation.db

root@NS-Cumulus1# rm nslocation.*

> add locationfile /var/netscaler/locdb/GeoIPCountryWhois.csv -format geoip-country

Related:

FAQ: NetScaler Surge Queue

Q: What is NetScaler Surge queue?

A: A Surge queue is a path in the NetScaler appliance through which all client connections are sent, irrespective of the condition of the target service, such as service being loaded or service has reached the maximum connections state. When the number of requests to the servers is low, the connections are not observed in the Surge queue because the connections are sent to the servers quickly and the Surge queue build up is not observed.

Q: When connection is in Surge queue, is there a way to change the number of retries before giving up a connection (default is 7)?

A: No, this is as per design and it is not recommended to change the number of retries.

Q: What is the total maximum interval of 7 attempts of retransmit before NetScaler gives up on a connection? How long does the 7 retries take in total?

A: When there is a SYN without a response, the time is doubled for the retransmit and the time keeps doubling for every SYN without a response.

If you were to capture an nstrace for analysis then you can see the following retry pattern interval – 1 second, 2 seconds, 4 seconds, 8 seconds, 16 seconds, 32 seconds, 64 seconds and then a RST is sent. This works as per exponential back off algorithm.

Q: How many connections can NetScaler surge queue handle?

A: Surge queue is essentially a list of memory buffer thus there is no hard limit and it can go on building as far as there is memory in the connection pool (NSB/PCB). Till date there is no failover or crash grade issues observed with Surge queue.

Related:

ShareFile Connector SSO to Network Shares and SharePoint using Kerberos (KCD)

Summary of items

  1. Configure SharePoint for KCD
  2. Create an additional “Internal Content Switch” on the NetScaler
  3. Configure SplitDNS to resolve to the new Internal Content Switch
  4. StorageZone Controller IIS changes
  5. AD Delegation
  6. Web Browsers configs

1. Configure SharePoint for KCD

SharePoint config steps:

  1. On the Central Administration page, on the Quick Launch click Security, and in the General Security section click Specify authentication providers.
  2. On the Authentication Providers page, select the zone for which you want to change authentication settings.
  3. On the Edit Authentication page, and in the Authentication Type section ensure this is set to Windows (selected by default).
  4. In the IIS Authentication Settings section, select Negotiate (Kerberos).

    NOTE: If you selectNegotiate (Kerberos)you must perform additional steps to configure authentication (below).
  5. Click Save.

Set the SPN to the service account for SharePoint config steps:

NOTE:this is a standard SharePoint requirement which references the service account used during the installation of SharePoint itself). The service account used below is usually the one that SharePoint has been initially installed with.

  1. From any server, open CMD (elevate with account with the appropriate SharePoint rights)
  2. Type the following:

    SetSPN -S HTTP/SharePoint domainserviceaccountname

    SetSPN -S HTTP/SharePoint.citrix.lab domainserviceaccountname

2. Create an additional “Internal Content Switch” on the NetScaler

Before creating this, you should have run the wizard to create an External Content Switch as you would need to split the traffic, to split External and Internal traffic. The main reason being is to have AAA configured for Connectors externally, but for Internal use, not to have AAA enabled on the Connectors, especially if you would like to enable Web Access to Connectors and have a seamless SSO in all web browsers.

NOTE: AAA requires a NetScaler Enterprise license to use.

External Content Switch (usually created by the inbuilt ShareFile wizard on the NS).

NOTE: If Web Access to Connectors are required then additional configuration is needed in addition to the wizard. Please see this
article in section “Configure NetScaler for restricted zones or web access to Connectors”.

The External config would typically have:

  • 1 x Content Switch, with Policies, Responders, Callouts.
  • 3 x LBVIP’s
    • ShareFile Data LBVIP.
    • Connectors LBVIP with AAA enabled.
    • OPTIONS LBVIP.

Internal Content Switch (in this scenario, created manually)

The internal config would typically have:

  • 1 x Content Switch, with Policies, Responders, Callouts.
  • 2 x LBVIP’s
    • ShareFile Data LBVIP.
    • Connectors LBVIP (No AAA enabled).
    • No OPTIONS LBVIP required (even if SSO to “Web Access to Connectors” is needed).

Create the Internal Content Switch config steps:

Create the Virtual Servers (one for ShareFile Data and another for Connectors)

  1. Log onto the NetScaler and browse to:

    +Traffic Management

    +Load Balancing

    Virtual Servers
  2. Click Add to create the ShareFile Data LBVIP:

    Name: _SF_SZ_LB_INT

    Protocol: SSL or HTTP

    IP Address Type: Non Addressable
  3. Click OK.
  4. Click on the “No Load Balancing Virtual Server Binding”
  5. On the Select Server option click the arrow next to Click to select field
  6. Select the appropriate StorageZone Controller node(s) and click Bind
  7. Select the Certificate and click Bind, click Continue
  8. Click on the +Method option, change the Load Balancing Method to Token
  9. Add the expression REQ.URL.QUERY.VALUE(“uploadid”), click OK
  10. Click on the +Persistence option, and change the Persistence field to SSLSESSION
  11. Click OK
  12. Click Add to create the ShareFile Connector LBVIP:

    Name: _SF_CIF_SP_LB_INT

    Protocol: SSL or HTTP

    IP Address Type: Non Addressable
  13. Click OK
  14. Click on the “No Load Balancing Virtual Server Binding”
  15. On the Select Server option click the arrow next to Click to select field
  16. Select the appropriate StorageZone Controller node(s) and click Bind
  17. Select the Certificate and click Bind, click Continue
  18. Click on the +Method option, change the Load Balancing Method to LEASTCONNECTION
  19. Click on the +Persistence option, and change the Persistence field to COOKIEINSERT
  20. Click OK

Create the HTTP Callouts

  1. Browse to :

    +AppExpert

    HTTP Callouts
  2. Click Add to create the first callout:

    Name: _SF_CALLOUT_INT

    Server to receive callout request:

    Virtual Server and choose _SF_SZ_LB_INT

    Request to send to the server:

    Request Type:Attribute-Based

    Method: GET

    HostExpression: FQDN of the SSL cert internally Place quotes around ie: “sz.company.com”

    URLStemExpression: “/validate.ashx?RequestURI=” + HTTP.REQ.URL.BEFORE_STR(“&h”).HTTP_URL_SAFE.B64ENCODE + “&h=”+ HTTP.REQ.URL.QUERY.VALUE(“h”)

    Parameter:

    Scheme: HTTP

    ServerResponse

    ReturnType: BOOL

    Expression to extract data from the response: HTTP.RES.STATUS.EQ(200).NOT
  3. Click Create:

    Name: _SF_CALLOUT_INT_Y

    Server to receive callout request:

    Virtual Server and choose _SF_SZ_LB_INT

    Request to send to the server:

    Request Type:Attribute-Based

    Method: GET

    HostExpression: FQDN of the SSL cert internally Place quotes around ie: “sz.company.com”

    URLStemExpression: “/validate.ashx?RequestURI=” + HTTP.REQ.URL.HTTP_URL_SAFE.B64ENCODE + “&h=”

    Parameter:

    Scheme: HTTP

    ServerResponse

    ReturnType: BOOL

    Expression to extract data from the response: HTTP.RES.STATUS.EQ(200).NOT
  4. Click Create.
  5. Click Add to create the second callout (note: this is the same as the other except for the Name and URL Stem Expression)
  6. Click Add to create the first callout:

    Name: _SF_CALLOUT_INT_Y

    Server to receive callout request:

    Virtual Server and choose _SF_SZ_LB_INT

    Request to send to the server:

    Request Type: Attribute-Based

    Method: GET

    Host Expression: FQDN of the SSL cert internally Place quotes around ie: “sz.company.com”

    URL Stem Expression: “/validate.ashx?RequestURI=” + HTTP.REQ.URL.HTTP_URL_SAFE.B64ENCODE + “&h=”

    Parameter:

    Scheme: HTTP

    Server Response


    Return Type: BOOL

    Expression to extract data from the response: HTTP.RES.STATUS.EQ(200).NOT
  7. Click Create.

Create the Responder policy

  1. Browse to :

    +AppExpert

    +Responder

    Policies
  2. Click Add to create the responder:

    Name: _SF_RESPONDERPOL_INT

    Action: DROP

    Expression: HTTP.REQ.URL.CONTAINS(“&h=”) && HTTP.REQ.URL.CONTAINS(“/crossdomain.xml”).NOT&& HTTP.REQ.URL.CONTAINS(“/validate.ashx?requri”).NOT&& SYS.HTTP_CALLOUT(_SF_CALLOUT_INT) || HTTP.REQ.URL.CONTAINS(“&h=”).NOT && HTTP.REQ.URL.CONTAINS(“/crossdomain.xml”).NOT&& HTTP.REQ.URL.CONTAINS(“/validate.ashx?requri”).NOT&& SYS.HTTP_CALLOUT(_SF_CALLOUT_INT_Y)
  3. Click Create:

    Bind the Responder policy


    +Traffic Management

    +Load Balancing

    Virtual Servers
  4. Open _SF_SZ_LB_INT
  5. Click on the +Policies option
  6. Click Add Binding, Select the policy _SF_RESPONDERPOL_INT
  7. Click Bind, then Close.
  8. Click Done to complete.

Create the Content Switch policies

+Traffic Management

+Content Switching

Policies

  1. Click Add.

    Name: _SF_SZ_CSPOL_INT

    Expression: HTTP.REQ.HOSTNAME.CONTAINS(“sz.company.com”) && HTTP.REQ.URL.CONTAINS(“/cifs/”).NOT && HTTP.REQ.URL.CONTAINS(“/sp/”).NOT

    Note: DON’T FORGET TO CHANGE TO THE CORRECT EXTERNAL FQDN
  2. Click Create and then Add.

    Name: _SF_CIF_SP_CSPOL_INT

    Expression: HTTP.REQ.HOSTNAME.CONTAINS(“sz.company.com”) && (HTTP.REQ.URL.CONTAINS(“/cifs/”) || HTTP.REQ.URL.CONTAINS(“/sp/”))

    NOTE: Don’t forget to change to the correct external FQDN.
  3. Click Create.

Create the Content Switch vServer

+Traffic Management

+Content Switching

Virtual Server

  1. Click Add to create the Content Switch vServer:

    Name: _SF_CS_ShareFile_INT

    Protocol: SSL

    IP Address: Internal IP of DNS name

    Port:443
  2. Click OK
  3. Under Content Switching Policy Binding click on the No Content Switching Bound option:

    Select Policy:_SF_SZ_CSPOL_INT

    Target Load Balancing Virtual Server: _SF_SZ_LB_INT

    Click Bind

    Select Policy:_SF_CIF_SP_CSPOL_INT

    Target Load Balancing Virtual Server: _SF_CIF_SP_LB_INT

    Click Bind
  4. Click OK
  5. Click on the +Certificates option, add a certificate by clicking the No Server Certificate option
  6. Select the Certificate and click Bind, click Continue.

3. Configure SplitDNS to resolve to the new Internal Content Switch

This is important as you need to direct traffic internally to the NetScaler for internal clients. Create a Host A entry for the StorageZone FQDN to point to the IP of the Internal Content Switch created in section 2.

  1. Log into the Domain Controller and open dsa.msc.
  2. Browse to Forward Lookup Zones to find the one which correlates to the StorageZone FQDN (sz.company.com)
  3. Add a New Host (A or AAAA)… and enter the FQDN for the StorageZone.
  4. Enter the IP, this should be the one of the Internal Content Switch created in section 2.
  5. To test, open CMD from another desktop/server, run ipconfig/flushdns and ping the StorageZone FQDN. Does it resolve to the correct IP?

4. StorageZone Controller IIS changes

Config steps:

  1. Log onto the StorageZone Controller(s) and open IIS.
  2. Click on the Default web site then to the SP virtual directory.
  3. Click on Authentication, then ensure Anonymous and Windows Authentication are Enabled.
  4. Right-click on the WindowsAuthentication option and select Providers
  5. Highlight Negotiate and Move Up to the top of the list. Click
  6. Ensure Basic Authentication is set to Disabled.
  7. Click on the CIFS virtual directory, then on Authentication.
  8. Ensure Anonymous and Windows Authentication are Enabled.
  9. Right-click on the WindowsAuthentication option and select Providers.
  10. Highlight Negotiate and Move Up to the top of the list. Click
  11. Ensure Basic Authentication is Disabled.

    NOTE: If Using port 80 on your StorageZone Controller for Load Balancing communication, see section 5 of this article.
  12. Then right-click the Default Web Site and select Edit Bindings.
  13. Add a new binding on port 80, assign the IP address and insert a host header (which is the fqdn of storagezone).

    NOTE: Editing the existing binding on port 80 will upset the NTLM Path configured within the NetScaler IdP
    article on page 14 .
  14. On the StorageZone Controller, run CMD, then type:

    setspn –a http/sz.company.com SZCServer1

    setspn –a http/”fqdn of storagezone”hostname of storagezone controller”

    where “fqdn of storagezone” = sz.company.com

    and “hostname of storagezone controller” = SZCServer1)

5. AD DELEGATION

Changes need to be actioned on the SZC AD object(s), and all the servers used for Network Shares and SharePoint need to be added. Config steps shown in this procedure.

NOTE:

  • Ensure that any File servers hosting any Network Shares, are added to the delegation as CIFS.
  • Ensure any SharePoint servers that need to be accessed, are also entered as HTTP.

6. Browsers

Config steps:

Internet Explorer

  1. Open Internet Options, Security, Local Intranet, Sites, Advanced then enter the following:

    ShareFile site – subdomain.sharefile.com

    FQDN StorageZone – sz.company.com

    FQDN of AAAVIP – aaavip.company.com

    Note: If this is locked down, configure via GPO which will be actioned on the User Configuration.
  2. Open GPMC and select the GPO controlling the behavior of IE.
  3. Browse to Computer Configuration/Administrative Templates/System/Group Policy and Enabled the policy Configure user group policy loopback processing mode and select Replace.
  4. Then browse to User Configuration/Administrative Templates/Windows Components/Internet Explorer/Internet Control Panel/Security Page and edit the Site to Zone Assignment List as follows:

    User-added image

    NOTE: The number in the Value field denotes the number of the zone. MS breaks them down as follows:

    1 – Intranet zone – sites on your local network.

    2 – Trusted Sites zone – sites that have been added to your trusted sites.

    3 – Internet zone – sites that are on the Internet.

    4 – Restricted Sites zone – sites that have been specifically added to your restricted sites.

  5. For external IE browsers, extra configuration is required as follows:

    Click on the Internet/Custom Level and ensure that:

    Miscellaneous/Access data sources across domains is Enabled.

    User Authentication/Log on/Prompt for Username and Password is selected.
  6. Click OK twice.

Firefox

  1. Launch Firefox. In the Address Bar, instead of typing a URL, enter:

    about:config
  2. This will open the configuration interface. You may need to agree to a security warning in order to proceed.
  3. Double-click the line labeled automatic-ntlm-auth.trusted-uris and enter the following:

    ShareFile site – subdomain.sharefile.com

    FQDN StorageZone – sz.company.com

    FQDN of AAAVIP – aaavip.company.com

    NOTE: Separate individual URLs with commas, but do not put spaces between them, for example:

    subdomain.sharefile.com, sz.company.com
  4. Click OK when you’re finished.
  5. Double-click the line labeled negotiate-auth.trusted-uris. Enter the same information you entered in the previous step, with the URLs separated by commas and with no spaces. Click OK.

Chrome

This should work. CORS should be enabled by default on Chrome but you can add the plugin into Chrome here.

Opera

This should work.

Related:

Unable to access Storefront through NetScaler Gateway and getting ” Could reach the page ” error.

– After upgrading to 12.0 build 58.15 , unable to access the Storefront server through NetScaler Gateway and getting ” Could reach the page ” error.

NOTE: On NetScaler Gateway Session profile, the Storefront URL is configured with Storefront Load balancing server IP.

– If Storefront Load balancer IP is replaced with Actual Storefront Server IP, then Storefront is accessible through NetScaler gateway.

In the following nstrace screenshot, we could see that the Storefront Load balancer has sent Export cipher in the Server Hello. For which, we could see a FATAL Error message from NetScaler gateway Vserver.

User-added image

Related:

How Do I Configure end-to-end SSL on NetScaler?

NetScaler CLI

Complete the following steps to configure end-to-end SSL on NetScaler using CLI:

  1. Enable SSL Offloading feature:

    enable ns feature ssl

  2. Add SSL based services:

    Note: The service that is configured must use SSL protocol to ensure that the backend connection is secure. If configured as HTTP service, then it will not support NetScaler to backend server security and hence it will not be an end to end SSL configuration.

    > add service servicessl1 10.102 .216.29 SSL 443

    Done

    > add service servicessl2 10.102 .216.30 SSL 443

    Done

  3. Add an SSL virtual server.

    add lb vserver vserverssl SSL 10.102.216.180 443

    Done

  4. Add a certificate-key pair:

    > add SSI certKey sslckey -cert ns -server. cert -key ns-server.key -password ssl -expiryMonitor ENABLED -notificationperiod 30

    Done

  5. Bind the SSL key pair to the SSL vserver.

    bind ssl vs vserverssl -certkeyName sslckey

    Done

  6. Bind the SSL services to the SSL virtual server.

    > bind 1b vserver vserverssl servicessl1

    Done

    > bind 1b vserver vserverssl servicessl2

    Done

NetScaler GUI

Complete the following steps to configure end-to-end SSL on NetScaler using GUI:

  1. Enable SSL Offloading feature.

    Go to System > Settings > Configure Basic Features > check SSL Offloading.

    Note: Ensure that load balancing is checked as well.

    User-added image

  2. Add SSL based services.

    Note: The service that is configured must use SSL protocol to ensure that the backend connection is secure. If configured as HTTP service, then it will not support NetScaler to backend server security and hence it will not be an end to end SSL configuration.

    Go to Traffic Management > Load Balancing > Services > Add.

    User-added image

  3. Add an SSL virtual server.

    Go to Traffic Management > Load Balancing > Virtual Servers > Add.

    User-added image

  4. Add a certificate-key pair.

    On NetScaler GUI: Go to Traffic Management > SSL > Certificates > Install.

    User-added image

  5. Bind the SSL key pair to the SSL vserver.

    Go to Traffic Management > Load Balancing > Virtual Servers > select the virtual server you wish to bind the certificate to > Edit > Certificates > Server Certificates > select the certificate you wish to bind to the virtual server > Bind.

    User-added image

    User-added image

    User-added image

    User-added image

  6. Bind the SSL services to the SSL virtual server.

    Go to Traffic Management > Load Balancing > Virtual Servers > select the virtual server you wish to bind the services to > Edit > Service Binding > select the services to be bound to virtual server > Bind.

    User-added image

For additional configuration details refer to Citrix Documentation – Configuring SSL Offloading.

Additional/Optional Configuration Steps

There are two additional key features on backend SSL which you can configure:

  • Performing server certificate authentication on NetScaler by enabling it on NetScaler.
  • Sending client certificate to the backend sever for authentication.

Server Certificate Authentication on NetScaler

The server certificate authentication can be enabled on a NetScaler SSL service when the NetScaler wants to verify that the certificate sent by the backend server is for the same hostname as requested by the client.

Go to Traffic Management > Load Balancing > Services > select the SSL service on which you wish to enable Server Certificate Authentication > Edit > SSL Parameters > check Enable Server Authentication.

User-added image

Sending Client Certificate to the Backend Server

Usually this option need not be enabled if NetScaler and Server reside in the same secure zone. If not the case, then this option can be enabled for additional security. The bound Client Certificate would be sent to the backend sever when the server demands a certificate from the client (in this case NetScaler) to authenticate its identity.

Go to Traffic Management > Load Balancing > Services > select the SSL service on which you wish to enable Server Certificate Authentication > Edit > Certificates > Client Certificates.

User-added image

User-added image

Related:

How to Configure Double Hop on NetScaler

  • Under Published Applications, you have to add the second hop NetScaler under Next Hop Servers with Port as 443 and Secure checked.

    The moment you configure Next Hop, this NetScaler will understand that it is the first NetScaler in the double hop environment. Also the STA needs to be configured on the first hop NetScaler.

    Note: The Secure Ticketing Authority (STA) will show as down until you configure the second hop NetScaler because until then the connection will not be proxied.

    Note2: After configuring the Second Hop Gateway, if the STAs still show as down, add a route on the Second Hop appliance to the SNIP of the First Hop, to ensure both SNIP (First Hop) and Gateway VIP (Second Hop) can reach one another

    User-added image

  • Related:

    How to Configure NetScaler Gateway Session Policies for StoreFront

    Complete the following procedures to configure NetScaler Gateway with StoreFront:

    User-added image
    I. Policy for Web
    User-added image
    II. Policy for Receiver
    User-added image
    III. Authentication
    User-added image
    IV. Virtual Server
    User-added image
    V. StoreFront

    NetScaler Gateway

    I. The following steps details how to create the Session Policy for Web Browser Based Access.

    1. To create session policy, navigate to NetScaler Gateway > Policies > Session.
    2. In the Session Policies field, click Add.
    3. In the Name field, type the name of the Session Policy. For example, Web_Browser_Policy.
    4. Click the box with the + sign.

      User-added image

    5. Type in the Name of the new Session Profile in the Configure NetScaler Gateway Session Profile window.

      User-added image

    6. In the Client Experience tab, enable the following settings:

      • Enable Clientless Access and set it to Allow
      • Enable Single Sign-on to Web Application
      • Enable Plug-in Type to Windows/MAC OS X

      User-added image

    7. In the Security tab, enable Default Authorization Actions and set it to ALLOW.

      User-added image

    8. In the Published Application tab, enable the following settings:

    • Enable ICA Proxy and set it to ON.
    • Enable and configure Web Interface Address – FQDN of the Storefront server followed by the path to the store for web
    • Enable and configure Single Sign-on Domain – NetBIOS name for the domain
    • Click Create

    User-added image

    1. If you are using Classic Policy expression, In the expression field, add the information listed below and click Create.

      REQ.HTTP.HEADER User-Agent NOTCONTAINS CitrixReceiver

      User-added image
      If using Advanced Policy expression, In the expression field, add the information listed below and click Create.

      HTTP.REQ.HEADER(“User-Agent”).CONTAINS(“CitrixReceiver”).NOT​


      User-added image
      Note: This policy is needed in order for the NetScaler to differentiate between web browser based and Citrix Receiver based connections. This policy will be applied to web browser based connections.

    Top of Page

    II. The following steps details how to createthe Session Policy for Citrix Receiver for Windows or Mac, and Mobile Devices on NetScaler Gateway:

    1. Navigate to NetScaler Gateway > Policies > Session.
    2. In the Session Policies field, click Add.
    3. In the Name field, type the Name of the Session Policy. For example, Receiver_Policy
    4. Click the box with the + sign.

      User-added image

    5. Type in the Name of the new Session Profile in the Configure NetScaler Gateway Session Profile window.

      User-added image

    6. In the Client Experience tab, enable the following settings:

      User-added image

      • Set the Home Page to None
      • Enable Split Tunnel and Set to OFF
      • Enable Clientless Access and set it to Allow
      • Enable Single Sign-on to Web Application
      • Set Plug-inType to Java
      User-added image
      • Uncheck Choices Choices
    7. In the Security tab, enable Default Authorization Actions and set it to ALLOW.

      User-added image

    8. In the Published Application tab, enable the following settings:
      • Enable ICA Proxy and set it to ON
      • Enable and configure Web Interface Address – FQDN of the Storefront server followed by the path to the store for web
      • Enable and configure Single Sign-on Domain – NetBIOS name for the domain
      • Enable and configure Account Services Address. The last back slash is important
      • Click Create.
      User-added image
    9. If using Classic Policy expression, In the expression field, add the information listed below and click Create.

      REQ.HTTP.HEADER User-Agent CONTAINS CitrixReceiver


      User-added image

      If using Advanced Policy expression, In the expression field, add the information listed below and click Create.

      HTTP.REQ.HEADER(“User-Agent”).CONTAINS(“CitrixReceiver”)​

      User-added image


      Note: This policy is needed in order for the NetScaler to differentiate between web browser based and Citrix Receiver based connections. This policy will be applied for Citrix Receiver based connections.

    Top of Page

    III. The following steps details how to configure authentication on the NetScaler appliance.

    Click on the following link for latest information on how to configure
    LDAP authentication on the NetScaler appliance.

    Top of Page

    IV. The following steps details how to createNetScaler Gateway Virtual Server and bind the Session Policies.

    1. Navigate to NetScaler Gateway > Virtual Server and click Add to add a new virtual server.

      User-added image

      User-added image

    2. After the virtual server is created, bind the specific session policy to the virtual server based on your company’s requirements.

    Top of Page

    StoreFront

    V. The following steps details how to configure authentication for StoreFront.

    1. Enable the pass-through authentication from NetScaler Gateway on StoreFront. For more information, refer to Citrix Documentation – Create and configure the authentication service.

      Note: StoreFront must trust the issuer of the NetScaler Gateway virtual server’s bound certificate (Root and/or Intermediate certificates) for the Authentication Callback service.
    2. Add NetScaler Gateway to StoreFront. For more information, refer to Citrix Documentation – Add a NetScaler Gateway connection.

      Note: The Gateway URL must match exactly what the users are typing into the web browser address bar.
    3. Enable remote access on the StoreFront store. For more information, refer to Citrix Documentation – Manage remote access to stores through NetScaler Gateway.

    Top of Page

    User-added image

    Related:

    Troubleshooting NetScaler High Availability (HA) Issues

    When troubleshooting issues related to the high availability feature of a NetScaler appliance, consider the following points:

    Avoid different NetScaler software releases and builds on the NetScaler appliances. Citrix recommends that you do not install different NetScaler software releases and builds on the NetScaler appliances forming the high availability setup. Such inconsistency can lead to undesired performances, such as failovers and missing configurations.

    You can prevent this by using the stay primary and stay secondary commands on the respective appliances. For more information refer to Citrix Documentation – Forcing the Primary Node to Stay Primary and Forcing the Secondary Node to Stay Secondary.

    Back to top

    ​If necessary, you can retrieve the original primary appliance configuration from a backup copy present on the hard disk of the appliance. The appliance saves the last five copies of the ns.conf file in the /nsconfig directory. These ns.conf files are named ns.conf.0 through ns.conf.4. The ns.conf.0 file contains the most recently saved configuration. For more information on NetScaler configuration file refer to Citrix Documentation.

    To retrieve and update the configuration of the appliance from a backup, complete the following procedure:

    -rw------- 1 root wheel 4671 Feb 28 20:54 /nsconfig/ns.conf.0-rw------- 1 root wheel 4671 Feb 28 20:54 /nsconfig/ns.conf.0-rw------- 1 root wheel 4671 Feb 28 20:42 /nsconfig/ns.conf.2-rw------- 1 root wheel 4671 Feb 28 20:41 /nsconfig/ns.conf.1-rw------- 1 root wheel 4671 Feb 28 20:40 /nsconfig/ns.conf.4root@ns#

    In the output of the preceding command, notice the date and time stamp for the ns.conf file.

    1. Log on to NetScaler CLI and issue the following command to switch to the shell prompt of the NetScaler appliance:

      shell

    2. Run the following command to determine the name of the latest backup copy of the ns.conf file:

      root@ns# ls -ltr /nsconfig/ns.conf.?

    3. Run the following command to make a copy of the latest ns.conf backup file:

      root@ns# cp /nsconfig/ns.conf.0 /nsconfig/copyns.conf

    4. Run the following command to switch to the command line interface of the appliance:

      root@ns# exit

    5. Run the following command to read the contents of the copyns.conf file and run each line as an individual command:

      > batch –filename /nsconfig/copyns.conf

    6. Run the following command to save the running configuration to the /nsconfig/ns.conf file:

      save config

    Back to top

    The nsroot password must be the same for the primary and secondary NetScaler appliance in the high availability setup. When the password of the nsroot user account is changed on either of the appliances, the change must also be performed on the peer appliance because the password synchronization is required between the appliances.

    For more information refer to Citrix Documentation – Resetting the Default Administrator (nsroot) Password.

    Back to top

    Synchronization is a process of duplicating the configuration of the primary node on the secondary node. The purpose of synchronization is to ensure that there is no loss of configuration information between the primary and the secondary nodes, regardless of the number of failovers that occur. Synchronization uses port 3010.

    User-added image
    The ha_err_sync_failure counter increments when a NetScaler high availability synchronization failure is detected. The ha_err_sync_failure counter tracks the number of times the primary and secondary appliance failed to synchronize the configuration after the last transition. A synchronization failure results in mismatched configuration. For a complete list of NetScaler high availability counters refer to CTX131802NetScaler High Availability Counters.

    If there are synchronization issues, verify the following information:

    • Run the sync ha files all command twice and examine the results. Occasionally an issue occurs when the file synchronization does not finish in a minute or when you manually run the sync ha files command simultaneously. To confirm this issue, you must stop the periodic synchronization by commenting the appropriate line in the crontab file.
    • Ensure that the primary and secondary appliances can communicate with each other. The management and heartbeat packets are sent on the L2 layer. The L2 layer connectivity between the two appliances in the high availability setup must allow the heartbeat packets to be received within 3 seconds on the Port 3003.
    • Verify port 22 is not blocked between the primary and the secondary appliance by using ACL or firewall policies. Port 22 is used by the rsync process. For detailed information on all ports that should be open refer to CTX101810 – Communication Ports Used by Citrix Technologies.
    • Ensure that any configured ACLs permit communication between the pair.
    • Ensure that nsconf, nsfsyncd and nssync process is running.
    • Ensure that SSL files are not missing on the secondary appliance.
    • Ensure that there is no temporary network connectivity loss between primary and secondary appliance.
    • Run the following command to verify that the nsnetsvc process is running:

      root@GA-NS4# ps auxw | grep -i nsnetsvc | grep -v grep

      root 256 0.0 0.2 18568 5668 ?? Ss Wed05PM 0:14.33 /netscaler/nsnetsvc

    Complete the following procedures to resolve file synchronization problems by analyzing the network trace to verify the communication between the appliances:

    1. Log on to the NetScaler appliances using an SSH utility, such as PuTTY and specifying the NetScaler IP (NSIP). Use the nsroot credentials to log on to the appliance.
    2. Terminate the nsfsyncd process on both the primary and the secondary appliances and restart it. Ensure that the process is running on both appliances:

      /netscaler/nsfsyncd –d
    3. Ensure that Keep Alive is enabled on TCP parameters on both appliances. Occasionally if this variable is disabled, the nsfsyncd process is terminated. Run the following command to verify if the process is enabled:

      >show tcpparam | grep KA
    4. Comment the nsfsyncd –p line in the /etc/crontab file. To comment or hash out nsfsyncd, log on to the NetScaler appliance through any SFTP client like WinSCP and edit the file with a text editor.

      User-added image

    5. Disable high availability synchronization on the secondary appliance.

      Note: The reason for disabling synchronization is to clearly identify file synchronization ioctl and its operations.

      > set ha node -hasync DISABLED
    6. Run a trace from both appliances and then run the sync ha files all command locally from the secondary and the primary appliance.

      Note: You can use this trace to analyze or verify the communication between the appliances.
    7. You can then uncomment the nsfsyncd process and enable high availability synchronization.

    For more information on troubleshooting this issue, refer to CTX138748 – File Synchronization in NetScaler High Availability Setup and Citrix Documentation – Configuring Synchronization.

    Back to top

    The configuration file of the primary and secondary NetScaler appliances is synchronized with the following exceptions:

    • The primary and secondary NetScaler appliances must be configured with unique NetScaler IP (NSIP) addresses. Therefore, this information is not synchronized between the appliances. The information about the interfaces are also omitted.
    • For each NetScaler appliance, configure the other high availability NetScaler appliance node. The node ID and associated IP address must reflect the node ID and IP addresses of the peer node. For example, NetScaler1 configured with a unique node ID and IP address of NetScaler2 and NetScaler2 configured with a unique node ID and IP address of NetScaler1.

    User-added image

    Back to top

    On both NetScaler appliances in the high availability setup, there might be a need to have a set of common configuration or certificate files depending on the deployment needs. If this is a requirement, then specific files present in the same location on both the appliances need to be manually synchronized.

    For example, if SSL offload is enabled, then SSL certificates must be copied to the same location on both the appliances. Similarly, the vsr.html file for Sure Connect, any manually customized files, or any other batch files containing configuration commands should be manually synchronized. You can use secure file transfer utility, such as the WinSCP, to transfer the files. You can run the following commands on the primary appliance to synchronize files with the secondary appliance:

    sync HA files ALL

    For all the operations that can be performed on “HA files” command refer to Citrix Documentation.

    Back to top

    Ensure that the RPC node password is the same on NetScaler appliances.

    If you have configured Global Server Load Balancing (GSLB), then the RPC node passwords should be configured on high availability NetScaler appliances for additional security, else the default password is enforced. Initially, all NetScaler appliances are configured with the same default RPC node password.

    Note: In NetScaler 11.0 hash value or encrypted string for RPC node password will look different even though they are configured to be the same.This is by design.

    For more information refer to Citrix Documentation – Creating or Changing an RPC Node Password.

    Back to top

    On new NetScaler appliances, all interfaces are enabled by default. Ensure that only used interfaces are enabled. Run the disable interface <Interface_Number> command to disable any unused interfaces to ensure that failover can occur when required.

    Disable monitoring for the interfaces whose failure should not cause a failover in the high availability setup by running the following command from the command line interface of the appliance, set interface -hamonitor OFF.

    For more information refer to Citrix Documentation – Configuring Network Interfaces.

    Note: Repeat this step for each appliance interface that is used and whose failure should not cause failover, such as the management only interface.

    Back to top

    Mapped IP addresses (MIP) are used for server-side connections. A MIP can be considered a default Subnet IP (SNIP) address, because MIPs are used when a SNIP is not available or Use SNIP (USNIP) mode is disabled.

    For more information refer to Citrix Documentation – Configuring Mapped IP Addresses (MIPs).

    Back to top

    If the NetScaler appliances are failing over unexpectedly, run the nsconmsg –d event command from the shell prompt to display the current events that might be causing the failover.

    The following are the possible causes:

    • Interface is down.
    • SSL acceleration card is down.
    • System stopped responding.

    Back to top

    If command propagation fails, it can be a result of connectivity issues, duplex mismatches, packet drops, or the /netscaler/nsnetsvc process is not running.

    For more information refer to Citrix Documentation – Configuring Command Propagation and Troubleshooting Command Propagation.

    Back to top

    Do not connect the appliances by a cross-over cable, because it can result in a bridging loop.

    Back to top

    Examine the cable on both the appliances and if required change the cable. Worn-out/damaged cables can cause failovers.

    Back to top

    You can examine the nic_cur_ha_MAC interface on the primary NetScaler in HA to see which interface has the HA MAC address. NetScaler ARPs for peer device interface MAC before sending heartbeat. In general the 0/1 interface MAC address of secondary should be learnt by 0/1 interface of primary and vice versa.

    If we notice that an interface MAC address of one device is learnt on multiple interfaces of the other appliance in HA then we will see this issue.

    User-added image

    For example, 0/1 interface and 10/1 interface are in use on primary (NS1) and secondary (NS2) NetScalers where, 0/1 interface is used for management and 10/1 interface is used for data traffic.

    If the NS1 learns the MAC address of 0/1 interface of NS2 then it would send all the heartbeats to only 0/1 interface of NS2. Due to this there will be no heartbeats seen on the 10/1 interface of NS2.

    You can address such issues by separating the interfaces using VLANs. For example, use VLAN100 for 0/1 interface and use VLAN200 for 10/1 interface. Make sure you create L3 VLANs..

    Back to top

    Ensure that the VLANs are tagged accurately. For more information refer to CTX122921 – Citrix NetScaler Interface Tagging and Flow of High Availability Packets and CTX214033 – NetScaler Networking and VLAN Best Practices.

    Back to top

    Point to Note

    • ​The secondary NetScaler appliance drops all traffic except for the high availability management and heartbeat packets. This is evident from the fact that the secondary appliance displays constantly increasing packet drops on all ports. This is the expected behavior of the secondary appliance.

    Back to top

    Related: