This article contains information that you must collect for troubleshooting an issue with the NetScaler appliance.
You must collect the following information to troubleshoot any issues with the NetScaler appliance:
-
NetScaler hardware model – (from FreeBSD, runsysctl -a netscaler).
-
NetScaler software version including the build – (from NetScaler CLI, runshow version).
-
Serial number of the appliance – (from FreeBSD, run sysctl -a netscaler).
-
Production setup or new installation.
-
Whether an application/service that was working is now broken or the user wants to configure an application/service.
-
Network topology information.
-
Any change(s) performed on the NetScaler appliance prior to the issue.
-
Any change(s) performed on the connected switches, upstream router, or back end server prior to the issue.
-
Collect the “ns.conf” file after saving the configuration – (from NetScaler CLI, run save configuration).
Notes:
- If the appliance is installed with NetScaler software release 6.x or 7.x, then the configuration and license files are located in the /flash/nsconfig directory.
- If the appliance is installed with NetScaler software release 8.x or 9.x, then the configuration file is located in the /flash/nsconfig directory and the license file(s) are located in the /flash/nsconfig/license directory.
Additional Information Required for Specific Issues
The following is a list of specific categories of issues for which you must collect additional information:
Unable to Access NetScaler for Management Purposes
High Availability (HA) Issues
Load Balancing Issues
Global Server Load Balancing (GSLB) Issues
Content Switching Issues
Caching Issues
SSL Related Issues
SSL VPN Related Issues
Web Logging Related Issues
Hardware Related Issues
Performance Related Issues
Command Line Interface Access
GUI and Dashboard Access
Examples of such issues include:
-
HA synchronization failing.
-
Both appliances appear as primary or both appliances appear as secondary.
-
Continuous change of state between the nodes.
-
You run a command on the primary appliance which fails on the secondary appliance (command propagation failure).
The data required to troubleshoot these issues include:
-
show node – (from NetScaler CLI on both nodes).
-
show interface <interface ID> – (from NetScaler CLI on both nodes).
-
“ns.conf” file from both the NetScaler appliances.
- Latest newnslog files from both the NetScaler appliances.
Examples of such issues include:
-
Services flapping up and down.
-
Virtual IP address flapping up and down.
-
Uneven load balancing.
-
Slow response when accessing applications through a virtual IP address.
The data required to troubleshoot these issues include:
-
newnslog file.
-
show lb vserver <vservername> for issues where virtual IP address is flapping up or down.
-
show service <servicename> for issues where a service is flapping up or down.
-
nstracefor issues related to packet flow and handling. Refer to Additional Resources section of this article to collect an nstrace.
-
If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.
Examples of such issues include:
-
Network traffic does not reach the intended site.
-
Metric Exchange Protocol (MEP) not being formed.
-
Remote site not coming up.
-
Proximity not working.
The data required to troubleshoot these issues include:
-
The “ns.conf” file from all the sites participating in GSLB along with basic information listed in the Overview section for all the sites.
-
show ns license output from the NetScaler CLI to verify if you have purchased proximity based GSLB license.
-
Latest newnslog files from all the NetScaler appliances.
-
show gslb site output from each NetScaler appliance participating in GSLB.
-
show gslb runningconfig output from each NetScaler appliance participating in GSLB.
Examples of such issues include:
The data required to troubleshoot these issues include:
Examples of such issues include:
The data required to troubleshoot these issues include:
-
The newnslog file.
-
If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.
-
stat cache output from the NetScaler CLI.
-
date (from FreeBSD shell) output for the NetScaler appliance and back end server.
Examples of such issues include:
-
Access to SSL virtual server failing.
-
Users receiving certificate related warnings when accessing HTTPS site.
-
Unable to bind certificate to SSL virtual server.
The data required to troubleshoot these issues include:
-
The newnslog file.
-
A copy of the certificate if the issue is related to SSL certificate installation/binding.
-
An nstrace for problems related to packet flow and handling. Refer to Additional Resources section of this article to collect an nstrace.
-
If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.
-
show ssl stats output from the NetScaler CLI.
Examples of such issues include:
-
Unable to access the server through the SSL VPN.
-
FTP or data transfer through the SSL VPN fails.
-
SSL VPN authentication related issues.
The data required to troubleshoot these issues include:
-
The “sslvpn.txt” file (for Windows platform) or “mpSSLVpn” and “Netscaler.log” files (for MAC platform) from the client computer if issues are related to the SSL VPN.
-
An nstrace for problems related to packet flow and handling. Refer to Additional Resources section of this article to collect an nstrace.
-
If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.
Examples of such issues include:
-
Web logging does not occur.
-
Web logging stops intermittently.
-
Required details are not logged.
-
Irrelevant IP addresses or domains are logged.
The data required to troubleshoot these issues include:
-
Details of the server on which the Web Logging service is installed and the version of the Web Logging client installed.
>nswl –version Version: Netscaler Weblogging Application(nswl) NS9.2: Build 47.11, Date: Aug 10 2010, 21:55:57 (release) [Linux] Done !!
-
The “log.conf” file and debug (file name: nswl.log-ddmmyyyyHHMM) file if the issue is related to Web Logging. These files reside on the Web Logging server in the /etc and /bin folders respectively (if Web Logging is installed as a Windows service, debug files are created in the WINDOWSsystem32 folder).
Examples of such issues include:
-
The interface is not detected.
-
The interface is always flapping.
-
Continuous console messages referring to a hardware component.
The data required to troubleshoot these issues include:
-
The newnslog file.
-
dmesg(from the FreeBSD shell) output if it is a hardware related issue. For example, if the interface is not detected or disk related issues.
-
show node output from the NetScaler CLI if the issue is related to SSL card failures.
Examples of such issues include:
To troubleshoot these issues collect the newnslog file.After the preliminary analysis is complete, you might require additional information and logs based on the nature of the problem. For instance:
-
Caching related issue:icstats and nscachemgroutput.
-
High CPU utilization: nsprofmonoutput.
-
NetScaler crashes: vmcore, kernel, nslog.log, ns.reboots files.
-
“ns.conf.x” file if you are not sure of the recent changes done on the NetScaler appliance or if you want to verify the changes yourself.
|
File name
|
Remark
|
Location in NetScaler Software Release 8.x/9.x/10.x
|
|
ns.conf
|
configuration file
|
/flash/nsconfig
|
|
ns.conf.x
|
older configuration file
|
/flash/nsconfig
|
|
newnslog
|
main log file (data format)
|
/var/nslog
|
|
newnslog.xx.gz
|
archived newnslog file
|
/var/nslog
|
|
ns.lic
|
license file
|
/flash/nsconfig/license
|
|
nstrace.sh
|
script to collect nstrace
|
/netscaler
|
|
nstcpdump.sh
|
script to collect tcpdump
|
/netscaler
|
|
nstrace.x
|
packet trace
|
/var/nstrace
|
|
vmcore.x.gz
|
core dump during a crash
|
/var/crash
|
|
kernel.x
|
kernel dump during a crash
|
/var/crash
|
|
process-pid
|
user process core file
|
/var/core
|
|
savecore.log
|
core dump log file
|
/tmp
|
|
pitboss.debug
|
open pipe for debug info
|
/tmp
|
|
aaad.debug
|
open pipe for debug info
|
/tmp
|
|
ns.log
|
system syslog file
|
/var/log
|
|
messages
|
all logged entries
|
/var/log
|
|
auth.log
|
authentication/authorization
|
/var/log
|
|
dmesg.*
|
hardware error/boot sequence
|
/var/nslog
|
Additional Resources
How to Obtain Performance Statistics and Event Logs from NetScaler Appliance
To record a network packet trace on a NetScaler appliance, complete the following procedure:
-
Run the following command from the shell prompt of the appliance:
/netscaler/nstrace.sh -sz 0
Note: The trace file is stored in the /var/nstrace directory with the name nstrace.x, where “x” is a number.
-
Reproduce the problem scenario.
-
Press CTRL+C to end the trace.
You must collect the trace file along with details such as source IP address, destination IP address, and virtual IP address.
Network trace collected can be decrypted using Wireshark tool. Refer to the following links:
How to Decrypt SSL and TLS Traffic Using Wireshark
Filter Expressions for Wireshark When Using NetScaler Appliance
How to Extract an SSL Certificate from a Network Packet Trace File in Wireshark
NetScaler Packet Trace Format in NetScaler 11.0 Release
