NetScaler 12.1 : NSPEE Crashes and Reboots Continuously After Upgrading to 12.1.48.13

After upgrading firmware from 12.0-57.19 to 12.1-48.13, NetScaler NSPEE crashes and goes in to a continuous reboot loop. The crash is happening in nsbasemem_free when trying to check for “Dup free Condition”.

There’s no user impact, but this prevents firmware upgrade. Entering the single user mode, and reverting the kernel to 12.0-57.19 stops the reboot loop.

Back Trace

Sample 1

Program terminated with signal 11, Segmentation fault.

#0 0x000000000197802c in kill ()

(gdb) bt

#0 0x000000000197802c in kill ()

#1 0x000000000174af89 in ns_stub_kill (pid=1866, sig=11) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_libc_stubs.c:703

#2 0x0000000001749691 in nsppe_signal_handler (signo=<optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/nsppe_utils.c:375

#3 <signal handler called>

#4 nsbasemem_free (uaddr=0x1dbaca900, sz=128, ID=<optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_basemem.c:759

#5 0x000000000116df3c in ns_freemem (mp=0x1dbaca900 “”, memsz=0, ID=45) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_mem.c:616

#6 0x0000000000f31751 in ns_mp_run_ioctls_queued () at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_krpc.c:1487

#7 0x00000000015079ac in packet_engine () at /home/build/rs_121_48_11_RTM/usr.src/sys/nic/ns_pe.c:3627

#8 0x000000000174814d in ns_enter_main (argc=<optimized out>, argv=<optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/nsppe.c:890

#9 0x000000000174c329 in main (argc=-609441536, argv=0x0) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_libc_stub_main.c:28

(gdb) f 4

#4 nsbasemem_free (uaddr=0x1dbaca900, sz=128, ID=<optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_basemem.c:759

759 /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_basemem.c: No such file or directory.

(gdb) info args

uaddr = 0x1dbaca900

sz = 128

ID = <optimized out>

(gdb) info local

smdp = <optimized out>

mdp = 0x0

block_id = 1

inx = 0

mask = <optimized out>

addr = 7980493056

descp = 0x20b7e20 <ns_base_mem_descr_arr>

(gdb) set $ADDR=(unsigned long long)uadddr

No symbol “uadddr” in current context.

(gdb) set $ADDR=(unsigned long long)uaddr

(gdb) p $ADDR

$1 = 7980493056

(gdb) p/x $ADDR

$2 = 0x1dbaca900

(gdb) p/x *$ADDR

$3 = 0x0

(gdb) p $ADDR+sz

$4 = 7980493184

(gdb) p mdp

$5 = (ns_memdesc_t *) 0x0

(gdb) f 6

#6 0x0000000000f31751 in ns_mp_run_ioctls_queued () at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_krpc.c:1487

1487 /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_krpc.c: No such file or directory.

(gdb) info args

No arguments.

(gdb) info local

nsb_rec = 0x1dbaca900

(gdb) p ns_mp_ioctl_nsb_q.ns_cur_nsb

$6 = 0

(gdb) f 4

#4 nsbasemem_free (uaddr=0x1dbaca900, sz=128, ID=<optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_basemem.c:759

759 /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_basemem.c: No such file or directory.

(gdb) info args

uaddr = 0x1dbaca900

sz = 128

ID = <optimized out>

(gdb) info local

smdp = <optimized out>

mdp = 0x0

block_id = 1

inx = 0

mask = <optimized out>

addr = 7980493056

descp = 0x20b7e20 <ns_base_mem_descr_arr>

(gdb) p descp->memlist

$7 = (ns_memdesc_t *) 0x20b6a80 <nsmem_memlist>

(gdb) p descp->memlist.nmd_memlist

$8 = {{cle_next = 0x100000000}, {cle_prev = 0x1dbacae00}}

(gdb) p mdp->ns_memdesc_nmdaddr

Cannot access memory at address 0x0

Sample 2 :

(gdb) bt

#0 0x000000000197802c in kill () at /home/build/rs_121_48_11_RTM/usr.src/contrib/ustl/ustl-2.2/sistream.cc:123

#1 0x000000000174af89 in ns_stub_kill (pid=<value optimized out>, sig=<value optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_libc_stubs.c:703

#2 0x0000000001749691 in nsppe_signal_handler (signo=<value optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/nsppe_utils.c:375

#3 <signal handler called>

#4 nsbasemem_free (uaddr=0x11f3b9e80, sz=128, ID=<value optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_basemem.c:759

#5 0x000000000116df3c in ns_freemem (mp=0x11f3b9e80 “”, memsz=48, ID=45) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_mem.c:616

#6 0x0000000000f31751 in ns_mp_run_ioctls_queued () at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_krpc.c:1487

#7 0x00000000015079ac in packet_engine () at /home/build/rs_121_48_11_RTM/usr.src/sys/nic/ns_pe.c:3627

#8 0x000000000174814d in ns_enter_main (argc=<value optimized out>, argv=<value optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/nsppe.c:890

#9 0x000000000174c329 in main (argc=<value optimized out>, argv=<value optimized out>) at /home/build/rs_121_48_11_RTM/usr.src/sys/netscaler/ns_libc_stub_main.c:28

Current language: auto; currently c++

Messages Log :

Messages shows“unable to connect to nsnetsvc: Connection refused”

19:32:44 <local0.err> FH2-NETSCALER1 nsconfigd: _dispatch(): Connection failed

Jun 30 19:32:44 <local0.err> FH2-NETSCALER1 nsconfigd: _dispatch(): Connection failed

Jun 30 19:32:44 <local0.crit> FH2-NETSCALER1 nsconfigd: netsvcfd(): unable to connect to nsnetsvc: Connection refused

Jun 30 19:32:44 <local0.err> FH2-NETSCALER1 nsconfigd: _dispatch(): Connection failed

Jun 30 19:32:44 <local0.crit> FH2-NETSCALER1 nsconfigd: netsvcfd(): unable to connect to nsnetsvc: Connection refused

Jun 30 19:32:44 <local0.err> FH2-NETSCALER1 nsconfigd: _dispatch(): Connection failed

Jun 30 19:32:44 <local0.crit> FH2-NETSCALER1 nsconfigd: netsvcfd(): unable to connect to nsnetsvc: Connection refused

Jun 30 19:32:44 <local0.err> FH2-NETSCALER1 nsconfigd: _dispatch(): Connection failed

Jun 30 19:32:44 <local0.crit> FH2-NETSCALER1 nsconfigd: netsvcfd(): unable to connect to nsnetsvc: Connection refused

Jun 30 19:32:44 <local0.err> FH2-NETSCALER1 nsconfigd: _dispatch(): Connection failed

Jun 30 19:32:44 <local0.crit> FH2-NETSCALER1 nsconfigd: netsvcfd(): unable to connect to nsnetsvc: Connection refused

NS Log :

Ns.log is flooded with the message

cpu” – Status “Success”

Jul 1 10:00:58 <local0.info> FH2-NETSCALER1 httpd: Failed to open file:/flash/nsconfig/.callhome.conf, Permission denied

Jul 1 10:00:58 <local0.info> 10.17.172.207 07/01/2018:10:00:58 GMT FH2-NETSCALER1 0-PPE-0 : default API CMD_EXECUTED 4103 0 : User nsroot – Remote_ip 10.172.2.20 – Command “stat ssl” –

Status “Success”

Jul 1 10:01:59 <local0.info> FH2-NETSCALER1 httpd: Failed to open file:/flash/nsconfig/.callhome.conf, Permission denied

Jul 1 10:01:59 <local0.info> 10.17.172.207 07/01/2018:10:01:59 GMT FH2-NETSCALER1 0-PPE-0 : default API CMD_EXECUTED 4104 0 : User nsroot – Remote_ip 10.172.2.20 – Command “stat ns” –

Status “Success”

Jul 1 10:01:59 <local0.info> FH2-NETSCALER1 httpd: Failed to open file:/flash/nsconfig/.callhome.conf, Permission denied

Jul 1 10:01:59 <local0.info> 10.17.172.207 07/01/2018:10:01:59 GMT FH2-NETSCALER1 0-PPE-0 : default API CMD_EXECUTED 4105 0 : User nsroot – Remote_ip 10.172.2.20 – Command “stat system

” – Status “Success”

Jul 1 10:01:59 <local0.info> FH2-NETSCALER1 httpd: Failed to open file:/flash/nsconfig/.callhome.conf, Permission denied

Jul 1 10:01:59 <local0.info> 10.17.172.207 07/01/2018:10:01:59 GMT FH2-NETSCALER1 0-PPE-0 : default API CMD_EXECUTED 4106 0 : User nsroot – Remote_ip 10.172.2.20 – Command “stat system

memory” – Status “Success”

Jul 1 10:01:59 <local0.info> FH2-NETSCALER1 httpd: Failed to open file:/flash/nsconfig/.callhome.conf, Permission denied

Jul 1 10:01:59 <local0.info> 10.17.172.207 07/01/2018:10:01:59 GMT FH2-NETSCALER1 0-PPE-0 : default API CMD_EXECUTED 4107 0 : User nsroot – Remote_ip 10.172.2.20 – Command “stat system

cpu” – Status “Success

Related:

Leave a Reply