3078409: Handling ndsd (eDirectory) core files on Linux and Solaris

Sometimes the reason ndsd crashes is due to memory corruption. If this is the case, it is necessary to add variables setting to the ndsd environment to put the memory manager into a debug state. This will help to ensure that ndsd generates a core at the time the corruption occurs so the module that caused the corruption can more easily be identified in the core.

If ndsd cores due to stack corruption, Novell Technical Support will request that you add the appropriate memory manager setting and wait for another core to re-submit.

Linux

To set the necessary memory checking variable on Linux:

Systemd – SLES 12 / Redhat 7 or later: Modify the “env” file located in the /etc/opt/novell/eDirectory/conf directory, then restart the eDirectory instance. ( See 2nd bullet under “Please refer to the following notes:” for details. )

MALLOC_CHECK_=3



SysVinit
– SLES 11 / RedHat 6 or earlier: Modify the pre_ndsd_start script and the following at the very top, then restart the eDirectory instance.

MALLOC_CHECK_=3

export MALLOC_CHECK_

Please refer to the following notes:

  • The contents of the pre_ndsd_start script are sourced into ndsd at the time ndsd loads. Be aware that any permanent settings will be overwritten if left in the ndsd script the next time an eDirectory patch is applied while the pre_ndsd_start script will not be modified. For this reason changes to the ‘ndsd’ script itself should not be made. This is the purpose of the pre/post_ndsd_start scripts.

  • eDirectory on SLES 12 or RHEL 7: You must add all environment variables required for the eDirectory service in the env file located in the /etc/opt/novell/eDirectory/conf directory.

  • MALLOC_CHECK_=3 should NOT be left permanently. Once the cores have been gathered, remove this setting from the modified script and restart ndsd. This environment variable can have a performance impact on some systems due to the increased memory checking. In eDirectory 8.8, it will cause ndsd to revert back to using malloc instead of tcmalloc_miminal which was added to enhance performance.

    Another side effect of using MALLOC_CHECK_=3 is the possibility of increased coring. Malloc will cause ndsd to core whenever a memory violation is detected whether or not it would have caused ndsd to crash under normal running conditions.

    To verify this ndsd environment variable is set properly while ndsd is running, do the following as the user running the eDirectory instance (‘root’ most of the time):

    strings /proc/`pgrep ndsd`/environ | grep -i MALLOC_CHECK_

    The command above will not work on a server with multiple eDirectory instances (or ndsd processes). To check a particular instance find that instance’s process’s PID and use that directly. For PID 12345 the command would be the following:

    strings /proc/12345/environ | grep -i MALLOC_CHECK_

    After ndsd has cored, to verify the core file had the ndsd environment variable set, do the following:

    strings core.#### | grep -i MALLOC_CHECK_

    Bundle the core with MALLOC_CHECK_=3 set as in step 2.

    For more information on Malloc check see: TID 3113982 – Diagnosing Memory Heap Corruption in glibc with MALLOC_CHECK_

  • eDirectory 8.8.5 ftf2 (patch2) the location of the pre_ndsd_start has been moved from /etc/init.d to /opt/novell/eDirectory/sbin/.

Solaris

In current code, eDirectory uses libumem as the memory manager.

To configure libumem for debugging add the following to the pre_ndsd_start script at the top and restart ndsd:

UMEM_DEBUG=default

UMEM_LOGGING=transaction

export UMEM_DEBUG UMEM_LOGGING

Submit a new core with these settings in place.

Changing the location where cores files are generated

In certain situations it may be desirable to change the location where core files are generated. By default ndsd core files are placed in the dib directory. If space in this directory is limited or if another location is desired, the following can be done:

mkdir /tmp/cores

chmod 777 /tmp/cores

echo “/tmp/cores/core”> /proc/sys/kernel/core_pattern

This example would now generate the core. <pid> file in /tmp/cores

To revert back to placing cores in default location:

echo core > /proc/sys/kernel/core_pattern

Symbol build of ndsd libriaries



In some cases, a core file generated while running libraries with symbols included may be necessary to analyze the core.

This is particularly true when analyzing cores generated by the 64 bit version of ndsd since the parameters aren’t located at a specific location.

The symbol versions of the libraries can be obtained from Novell eDirectory backline support.

Related:

Leave a Reply