NetScaler MAS Troubleshooting Guide

Table of Contents

FAQs During Diagnosis and Troubleshooting of NetScaler MAS Issues Common NetScaler MAS Issues and Troubleshooting Steps

Frequently Asked Questions During Diagnosis and Troubleshooting of NetScaler MAS Issues

The following section lists some of the frequently asked questions during diagnosis and troubleshooting of NetScaler MAS issues:

Q: How to verify the NetScaler MAS build version using CLI and support file?

A: On a live device check the version.conf file under /mps:

bash-2.05b# cat version.conf

11.1-52.15

In support file you can open /var/nslog/dmesg.boot and grep for flash. You should see the build version.

bash-2.05b# grep flash dmesg.boot

/mas-11.1-52.15 -> /flash/mas-11.1-52.15

Back to Top

Q: How does MAS fetch all the dashboard related data from connected devices?

A: MAS does Nitro calls to instances and gets all the details.

To confirm how many vservers each instance has and communicated by NetScaler can be seen in mps_inventory.log:

Wednesday, 22 Mar 17 19:08:02.287 +0530 [Debug] [Emon[#60]] HTTP Request Protocol: https, ContentType: , Method: GET, URL: https://10.107.100.131/nitro/v1/config/lbvserver?attrs=name,ipv46,port,servicetype,effectivestate,curstate,health,tickssincelaststatechange,comment,lbmethod,persistencetype,totalservices,activeservices&pagesize=10000&pageno=1&format=json

Wednesday, 22 Mar 17 19:08:02.371 +0530 [Debug] [Emon[#60]] EMON_LB_VIP: 10.107.100.131, db_objects: 9, new_objects: 8

In the above output you will see differeance in db_objects and new_objects as a vserver was deleted from NetScaler when this call was last made. For performance related dashboard and graphs output look at /var/mps/mps_perf.log

Back to Top

Q: How to capture NetScaler tcpdumps on MAS?

A: Use the following command to capture tcpdumps on MAS:

Tcpdump –i 1 <options>

The switch -i specifies the interface and 1 represents the first interface on the MAS appliance. For further reference please review https://www.freebsd.org/cgi/man.cgi?tcpdump(1)

For example:

tcpdump -i 1 udp and src 10.107.100.131

tcpdump -i 1 tcp

tcpdump -i 1 src NSIP or dst NSIP

Back to Top

Q: How does NetScaler MAS communicate with different devices after adding them?

A: NetScaler MAS periodically communicates through Nitro calls for each and every instance and resource added to it. You can see these calls in mps_inventory.log and mps_config.log

Back to Top

Q: How to upgrade different instances through NetScaler MAS?

A: Complete the following steps to upgrade different instances through NetScaler MAS:

  1. Go to Infrastructure > Configuration Jobs > Maintenance tasks.
  2. Now select the device that you want to upgrade.

Back to Top

Q: How to run techsupport script from CLI of NetScaler MAS?

A: Use the following command to run the techsupport script:

cd /mps/scripts

./techsupport.pl

Back to Top

Q: How to capture effective debug logs to troubleshoot insight related isues?

A: Complete the following steps to capture debug logs:

  1. Whenever you are capturing a support file to troubleshoot an issue, enable ‘Collect Debug logs’ in the tech-support page:

    User-added image

  2. When you enable this, additional debug level afdecoder logs are captured. This will help troubleshoot the issue further.
  3. The time that you will specify is the amount of time these additional debug logs will be captured after clicking on OK.
  4. So if you are troubleshooting HDX insight issue, give a suitable time and click on OK which will start the debugging:

    User-added image

  5. Now replicate the issue within that time interval so that debug logs pertaining to the issue are captured. Support file will automatically be generated after this timer expires.

Back to Top

Q: What are the new features which assists in troubleshooting NetScaler MAS issues?

This is available only from 12.0 onwards.

  • Go to System > Diagnostics > Troubleshooting.
  • You can then select the instance you want to diagnose and get the Diagnostics chart.
  • Remember, only the vservers and instances where AppFlow is enabled will work with this.
  • It will show you different AppFlow related parameters enabled on the device and different vserver states.

User-added image

Back to Top

Q: What are the different processes on NetScaler MAS and how to troubleshoot different issues using mps_*.log in /var/mps/log?

A: The following are the different processes on NetScaler MAS:

  • Control SubSystem (mps_control.log) : Initialize+monitor+stop other subsystems and the database. It is responsible for restarting any subsystem if it crashes.
  • Service SubSystem (mps_service.log) : It has an inbuilt HTTP(s) Request/Response handler. It listens on port 80 and 443. Any request from UI/API will hit the Service SubSystem. Based on the type of request, it might process the request itself or pass it on to the other appropriate SubSystem. Response always goes back via Service SubSystem.
  • Inventory SubSystem (mps_inventory.log) : It does inventory from NetScaler/SD-WAN instances and updates instances’ information in the database. It retrieves build/system information from NetScaler Instances. It runs complete inventory every 30 minutes by default. This subsystem also retrieves statistics from instances to show CPU/Memory usage etc.
  • Config SubSystem (mps_config.log) : It processes any configuration request that is received from Service Process. Configuration request can be adding instance, or any other operations on NetScaler instane or on MAS itself. It is also related to admin user management, device profiles, external authentication server config etc.
  • Event SubSystem (mps_event.log) : It raises internal events in case of any SubSystem failure or configuration changes. This subsystem also registers itself with NetScaler/SD-WAN instances and SDX appliances to receive syslogs and provide event based reporting. All traps and syslogs events come to this process.
  • Perf SubSystem (mps_perf.log) : This subsystem is responsible for performance reporting of NetScaler/SD-WAN instances. This retrieves instance stats every 5 mins and aggregates them on minutely, hourly, daily and weekly basis. There are pre-defined reports.
  • afdecoder SubSystem (mps_afdecoder.log) : This subsystem is responsible to receive AppFlow traffic from NetScaler/SD-WAN instances and process that data.
  • afanalytics SubSystem (mps_afanalytics.log) : This subsystem is responsible for analytics reporting of NetScaler/SD-WAN instances. This subsystem aggregates the data on minutely, hourly, daily and weekly basis. There are pre-defined reports.

Back to Top

Q: How to verify NetScaler MAS configuration using the DB?

A: All the config is stored in a csv files under /var/mps/mpsdb directory.You can open each file and check the config.

Back to Top

Q: What does various NetScaler MAS process states refer to (Output of ‘PS –ax’)?

A: Here are the different values that the s, stat and state output specifiers (header “STAT” or “S”) will display to describe the state of a process:

  • D uninterruptible sleep (usually IO)
  • R running or runnable (on run queue)
  • S interruptible sleep (waiting for an event to complete)
  • T stopped, either by a job control signal or because it is being traced.
  • W paging (not valid since the 2.6.xx kernel)
  • X dead (should never be seen)
  • Z defunct (“zombie”) process, terminated but not reaped by its parent.

For BSD formats and when the stat keyword is used, additional characters may be displayed:

  • < high-priority (not nice to other users)
  • N low-priority (nice to other users)
  • L has pages locked into memory (for real-time and custom IO)
  • s is a session leader
  • l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
  • + is in the foreground process group.

Back to Top

Common NetScaler MAS Issues and Troubleshooting Steps

The following section lists some of the most common NetScaler MAS issues and the steps to troubleshoot these issue:

Unable to see all the virtual servers that are present on an instance

  1. Check the licenses for MAS and the virtual servers it supports. You can see that under Infrastructure > Licenses > System Licenses.

    User-added image

  2. Remember the license limit that you see in the above snapshot comprises of all the vservers on the instances including Gateway VIP and GSLB VIP.
  3. Click Modify Licensed Virtual Servers and see if you can find the missing Virtual Server by clicking Add Virtual Servers.

    User-added image

  4. See if you find the missing virtual server there.
  5. You can then go to the above snapshot page again and select the virtual servers you do not need. Then click Mark Unlicensed. You can then replace it with the virtual server that you need.
  6. Alternately, you can go to /var/log/ns.log and try to search for these errors:

    Mar 21 18:22:34 <local0.err> NetScaler MAS-1 mas_event: 10.107.143.118 03/21/2017:12:52:34 GMT : EVENT VIPLICENSELIMITWARNING : 127.0.0.1:VIPLicenses – System discovered more Virtual Servers than license limit, dropping 1 vips

Back to Top

Unable to add instance to NetScaler MAS

  1. You can take a snapshot of the error that you see on GUI when adding an instance.
  2. The following is the flow:

    User-added image

  3. In the collector file you can confirm this by looking at /var/ns.log file.
  4. To understand where the addition failed, examine the /var/mps/mps_Inventory.log and /var/mps/mps_config.log files.
  5. You can also take tcpdump on MAS by using the following command:

    tcpdump -i 1 src NSIP or dst NSIP where NSIP = your NetScaler IP

Back to Top

Service/load balancing state is not shown as DOWN even if that resource is DOWN on NetScaler

  • MAS will only know about the vserver being down when it polls that instance.

    Every 30 minutes, entities are polled by NetScaler MAS by using NITRO calls. An entity is either a policy, virtual server, service, or action attached to a NetScaler instance. While this poling interval is configurable, you cannot set it to less than 10 minutes. To configure it, navigate to Networks > Network Functions > Settings > Configure Polling Interval for Entities. You can also poll the entities configuration when required by navigating to Networks > Network Functions > Load Balancing > Entities Configuration > Poll Now.

    User-added image

  • But you will see the status change in the Application dashboard of MAS. Because it monitors the applications in real time.

    User-added image

  • You will also receive traps for this event and you can see that under Infrastructure > Events > Event Messages.
  • You can see the output of the MAS poll in mps_config.log and mps_inventory.log files.

Back to Top

Cannot see any data on HDX, TCP, Gateway, and Web Insight

The troubleshooting procedure is similar to how you troubleshoot Insight.

  1. Verify if AppFlow is configured on NetScaler vservers. Check if MAS IP is in AppFlow collector list.
  2. Make sure that appropriate AppFlow policies and actions are in place.
  3. Examine mps_afdecoder.log and mps_afanalytics.log under /var/mps/log for any errors.
  4. Verify if traffic is reaching MAS from NetScaler on UDP port 4739. Use tcpdump on MAS to determine this.

Back to Top

Related:

Leave a Reply