How to Obtain Performance Statistics and Event Logs from NetScaler

NetScaler CLI

You can run the nsconmsg command from NetScaler shell prompt without naming a file, to report events in real time.

  • Use the following syntax to read a historical file:

    /netscaler/nsconmsg -K /var/nslog/newnslog -d event

    Displaying event informationNetScaler V20 Performance DataNetScaler NS10.5: Build 57.7.nc, Date: May 14 2015, 07:35:21 rtime: Relative time between two records in millisecondsseqno rtime event-message event-time11648 16310 PPE-0 MonServiceBinding_10.104.20.110:443_(tcp-default)

    Notes: The preceding command uses -d event to request display of major events. The parameter -K (uppercase) is for reading and -k (lowercase) is for writing. If you accidentally use -k then you might overwrite any information.

  • To view the time span covered by a given newnslog file, use the syntax as in the following example:

    /netscaler/nsconmsg -K /var/nslog/newnslog -d setime

    The current data is appended to the /var/nslog/newnslog file. NetScaler archives the newnslog file automatically every two days by default. To read the archived data, you must extract the archive as shown in the following example:

    cd /var/nslog – This is the command to go to a particular directory from NetScaler Shell Prompt.

    tar xvfz newnslog.100.tar.gz – This is the command to extract the tar file.

    /netscaler/nsconmsg -K newnslog.100 -d setime – This is the command to check time span covered by the particular file, in this example newnslog.100.

    “ls -l”command can be used to check all the logs file and time stamp associated with those files

    root@NETSCALER# cd /var/nslog

    root@NETSCALER# ls -l

     wheel 461544 Aug 7 2014 newnslog.1.tar.gz-rw-r--r-- 1 root wheel 191067 Aug 7 2014 newnslog.10.tar.gz-rw-r--r-- 1 root wheel 11144873 Apr 26 22:04 newnslog.100.tar.gz-rw-r--r-- 1 root wheel 11095053 Apr 28 22:04 newnslog.101.tar.gz-rw-r--r-- 1 root wheel 11114284 Apr 30 22:04 newnslog.102.tar.gz-rw-r--r-- 1 root wheel 11146418 May 2 22:04 newnslog.103.tar.gz-rw-r--r-- 1 root wheel 11104227 May 4 22:04 newnslog.104.tar.gz-rw-r--r-- 1 root wheel 11297419 May 6 22:04 newnslog.105.tar.gz-rw-r--r-- 1 root wheel 11081212 May 8 22:04 newnslog.106.tar.gz-rw-r--r-- 1 root wheel 11048542 May 10 22:04 newnslog.107.tar.gz-rw-r--r-- 1 root wheel 11101869 May 12 22:04 newnslog.108.tar.gz-rw-r--r-- 1 root wheel 11378787 May 14 22:04 newnslog.109.tar.gz-rw-r--r-- 1 root wheel 44989298 Apr 11 2014 newnslog.11.gz

    T

  • Use the nsconmsgcommand to only display a span of time within the given file, as shown in the following example:

    /netscaler/nsconmsg -K /var/nslog/newnslog -s time=22Mar2007:20:00 -T 7 -s ConLb=2 -d oldconmsg

    Where

    • s time=22Mar2007:20:00:00 is start at March 22, 2007 at exactly 8 p.m.
    • T 7 is display seven seconds of data
    • s ConLb=2 is a detail level for load balancing statistics
    • d oldconmsg is display statistical information

    NOTE: From ADC release 12.1 you need add at the “time” seconds as well, i.e.: 22Mar2007:20:00:00

    The statistical information provided by the -d oldconmsg parameter is recorded every seven seconds. The following is a sample output:

    VIP(10.128.58.149:80:UP:WEIGHTEDRR): Hits(38200495, 18/sec) Mbps(1.02) Pers(OFF) Err(0)Pkt(186/sec, 610 bytes) actSvc(4) DefPol(NONE) override(0)Conn: Clt(253, 1/sec, OE[252]) Svr(3)S(10.128.49.40:80:UP) Hits(9443063, 4/sec, P[2602342, 0/sec]) ATr(5) Mbps(0.23) BWlmt(0 kbits) RspTime(112.58 ms)Other: Pkt(36/sec, 712 bytes) Wt(10000) RHits(31555)Conn: CSvr(42, 0/sec) MCSvr(20) OE(16) RP(11) SQ(0)S(10.128.49.39:80:UP) Hits(9731048, 4/sec, P[2929279, 0/sec]) ATr(9) Mbps(0.27) BWlmt(0 kbits) RspTime(161.69 ms)Other: Pkt(41/sec, 756 bytes) Wt(10000) RHits(31555)Conn: CSvr(32, 0/sec) MCSvr(19) OE(13) RP(4) SQ(0)S(10.128.49.38:80:UP) Hits(9341366, 5/sec, P[2700778, 0/sec]) ATr(4) Mbps(0.27) BWlmt(0 kbits) RspTime(120.50 ms)Other: Pkt(42/sec, 720 bytes) Wt(10000) RHits(31556)Conn: CSvr(37, 0/sec) MCSvr(19) OE(13) RP(9) SQ(0)S(10.128.49.37:80:UP) Hits(9685018, 4/sec, P[2844418, 0/sec]) ATr(3) Mbps(0.23) BWlmt(0 kbits) RspTime(125.38 ms)Other: Pkt(38/sec, 670 bytes) Wt(10000) RHits(31556)Conn: CSvr(32, 0/sec) MCSvr(20) OE(10) RP(7) SQ(0)

    Note: The reason the client connection count of the individual services do not add up to the client connection count of the virtual server is because of session reuse between the NetScaler appliance and the back-end service.

Description of the Output

Virtual Server Sample Output

VIP(10.128.58.149:80:UP:WEIGHTEDRR): Hits(38200495, 18/sec) Mbps(1.02) Pers(OFF) Err(0) Pkt(186/sec, 610 bytes) actSvc(4) DefPol(NONE) override(0) Conn: Clt(253, 1/sec, OE[252]) Svr(3)

The following list describes the virtual server statistics:

  • VIP (IP address:port:state:Load balancing method): The IP address and port of the Virtual IP address as it was configured; State of the virtual server or virtual IP address such as UP, DOWN, or OUT OF SERVICE; Load balancing method, configured for the Virtual IP address..

  • Hits (#): Number of requests that reached the virtual server.

  • Mbps (#): Total traffic Volume on the Vserver (Rx + Tx) converted into Mbits/s

  • Pers: Type of persistence configured.

  • Err (#): Number of times error page was generated by the virtual server.

  • Pkt (#/sec, # bytes): Volume of network traffic in terms of packets passing through the virtual server; average size of the packets, flowing through the virtual server.

  • actSvc(#): Number of active services that are bound to the virtual server.

  • DefPol (RR): Indicates whether default load balancing method is active. Default load balancing method is used for some number of initial requests to smooth the behavior of the other methods.

  • Clt (#, #/sec): Number of current client connections to the virtual server rate.

  • OE [#]: Number of server connections from the virtual server in open established state.

  • Svr (#): Number of current server connections from the virtual server.

In the preceding output, Svr(3) indicates that when the command collected the statistical sample, there are three active connections for the virtual server to the back-end server, even though there are four services in total. When a client has an “open established” connection to the virtual server, it is not necessary that the client is sending or receiving any network traffic at that point of time when the command collected the information. Therefore, it is common to see the Svr counter lower than the OE[] number. The connections that are actively making or receiving transactions are represented by Svr counter. The Mapped IP address (MIP) or Subnet IP address (SNIP) makes the connection to the associated back-end server, but the NetScaler tracks which virtual server is connected to the back-end server and calculates the counter.

Service Sample Output

S(10.128.49.40:80:UP) Hits(9443063, 4/sec, P[2602342, 0/sec]) ATr(5) Mbps(0.23) BWlmt(0 kbits) RspTime(112.58 ms)Other: Pkt(36/sec, 712 bytes) Wt(10000) RHits(31555)Conn: CSvr(42, 0/sec) MCSvr(20) OE(16) RP(11) SQ(0)

The following list describes the service statistics:

  • S (IP address:port:state): IP address, port, and state of the service such as, DOWN, UP, or OUT OF SERVICE.

  • Hits (#, P[#]): Number of hits directed to the service, Number of hits directed to the service due to configured server persistence.

  • ATr (#): Number of active connections to the service.

    Note: Active connections are those which have outstanding request to the service or currently have traffic activity.

  • Mbps (#.##): Total traffic Volume on the Service (Rx + Tx) converted into Mbits/s

  • BWlmt ( # kbits): Defined bandwidth limit.

  • RspTime ( # ms): Average response time of the service in milliseconds.

  • Pkt(#/sec, #bytes): Traffic volume in terms of packets per second going to the service; Average size of the packets.

  • Wt (#): Weight index, used in load balancing algorithm.

    Note: If you divide this value by 10,000, then you get the actual configured weight of the service.

  • RHits (#): Running hits counter used in Round Robin load balancing algorithm.

  • CSvr (#, #/sec): Number of connections to the service rate.

  • MCSvr (#): Maximum number of connections to the service.

  • OE (#): Number of connections to the service in established state.

  • RP (#): Number of connections to the service, residing in the reuse pool.

  • SQ (#): Number of connections to the service, waiting in the surge queue.

NetScaler GUI

The log files and various troubleshooting data can be obtained from NetScaler Configuration Utility too, To download specific files using GUI Navigate to System>Diagnostics>Maintenance>Delete/Download log files. You can download the specific files and can share the same with support.

User-added image
Select a file and click on Download to download the file:

User-added image

Related:

How to Free Space From /var Directory of NetScaler Appliance When Unable to Log on to NetScaler GUI

  • Run the following commands to view the contents of the /var directory:

    cd /varls -l


    The directories that are usually of interest are as follows:

    /var/nstrace – This directory contains trace files.This is the most common reason for HDD being filled on the NetScaler. This is due to an nstrace being left running for indefinite amount of time. All traces that are not of interest can and should be deleted. To stop an nstrace, go back to the CLI and issue stop nstrace command.

    /var/log – This directory contains system specific log files.

    /var/nslog – This directory contains NetScaler log files.

    /var/tmp/support – This directory contains technical support files, also known as, support bundles. All files not of interest should be deleted.

    /var/core – Core dumps are stored in this directory. There will be directories within this directory and they will be labeled with numbers starting with 1. These files can be quite large in size. Clear all files unless the core dumps are recent and investigation is required.

    /var/crash – Crash files, such as process crashes are stored in this directory. Clear all files unless the crashes are recent and investigation is required.

    /var/nsinstall – Firmware is placed in this directory when upgrading. Clear all files, except the firmware that is currently being used.

  • Related:

    OneFS SmartDedupe: Deduplication Assessment

    To complement the actual SmartDedupe job, a dry-run Dedupe Assessment job is also provided to help estimate the amount of space savings that will be seen by running deduplication on a particular directory or set of directories. The dedupe assessment job reports a total potential space savings. The dedupe assessment does not differentiate the case of a fresh run from the case where a previous dedupe job has already done some sharing on the files in that directory. The assessment job does not provide the incremental differences between instances of this job. Isilon recommends that the user should run the assessment job once on a specific directory prior to starting an actual dedupe job on that directory.

    The assessment job runs similarly to the actual dedupe job, but uses a separate configuration. It also does not require a product license and can be run prior to purchasing SmartDedupe in order to determine whether deduplication is appropriate for a particular data set or environment. This can be configured from the WebUI by browsing to File System > Deduplication > Settings and adding the desired directories(s) in the ‘Assess Deduplication’ section.

    smartdedupe_assess_1.png



    Alternatively, the following CLI syntax will achieve the same result:

    # isi dedupe settings modify –add-assess-paths /ifs/data

    Once the assessment paths are configured, the job can be run from either the CLI or WebUI. For example:

    smartdedupe_assess_2.png

    Or, from the CLI:

    # isi job types list | grep –I assess

    DedupeAssessment Yes LOW

    # isi job jobs start DedupeAssessment

    Once the job is running, it’s progress and be viewed by first listing the job to determine it’s job ID.

    # isi job jobs list

    ID Type State Impact Pri Phase Running Time

    —————————————————————

    919 DedupeAssessment Running Low 6 1/1 –

    —————————————————————

    Total: 1

    And then viewing the job ID as follows:

    # isi job jobs view 919

    ID: 919

    Type: DedupeAssessment

    State: Running

    Impact: Low

    Policy: LOW

    Pri: 6

    Phase: 1/1

    Start Time: 2019-01-21T21:59:26

    Running Time: 35s

    Participants: 1, 2, 3

    Progress: Iteration 1, scanning files, scanned 61 files, 9 directories, 4343277 blocks, skipped 304 files, sampled 271976 blocks, deduped 0 blocks, with 0 errors and 0 unsuccessful dedupe attempts

    Waiting on job ID: –

    Description: /ifs/data

    The running job can also be controlled and monitored from the WebUI:

    smartdedupe_assess_3.png

    Under the hood, the dedupe assessment job uses a separate index table from the actual dedupe process. Plus, for the sake of efficiency, the assessment job also samples fewer candidate blocks than the main dedupe job, and obviously does not actually perform deduplication. This means that, often, the assessment will provide a slightly conservative estimate of the actually deduplication efficiency that’s likely to be achieved.

    Using the sampling and consolidation statistics, the assessment job provides a report which estimates the total dedupe space savings in bytes. This can be viewed for the CLI using the following syntax:

    # isi dedupe reports view 919

    Time: 2019-01-21T22:02:18

    Job ID: 919

    Job Type: DedupeAssessment

    Reports

    Time: 2019-01-21T22:02:18

    Results:

    Dedupe job report:{

    Start time = 2019-Jan-21:21:59:26

    End time = 2019-Jan-21:22:02:15

    Iteration count = 2

    Scanned blocks = 9567123

    Sampled blocks = 383998

    Deduped blocks = 2662717

    Dedupe percent = 27.832

    Created dedupe requests = 134004

    Successful dedupe requests = 134004

    Unsuccessful dedupe requests = 0

    Skipped files = 328

    Index entries = 249992

    Index lookup attempts = 249993

    Index lookup hits = 1

    }

    Elapsed time: 169 seconds

    Aborts: 0

    Errors: 0

    Scanned files: 69

    Directories: 12

    1 path:

    /ifs/data

    CPU usage: max 81% (dev 1), min 0% (dev 2), avg 17%

    Virtual memory size: max 341652K (dev 1), min 297968K (dev 2), avg 312344K

    Resident memory size: max 45552K (dev 1), min 21932K (dev 3), avg 27519K

    Read: 0 ops, 0 bytes (0.0M)

    Write: 4006510 ops, 32752225280 bytes (31235.0M)

    Other jobs read: 0 ops, 0 bytes (0.0M)

    Other jobs write: 41325 ops, 199626240 bytes (190.4M)

    Non-JE read: 1 ops, 8192 bytes (0.0M)

    Non-JE write: 22175 ops, 174069760 bytes (166.0M)

    Or from the WebUI, by browsing to Cluster Management > Job Operations > Job Types:

    smartdedupe_assess_4.png

    As indicated, the assessment report for job # 919 in this case discovered the potential of 27.8% in data savings from deduplication.

    Note that the SmartDedupe dry-run estimation job can be run without any licensing requirements, allowing an assessment of the potential space savings that a dataset might yield before making the decision to purchase the full product.

    Related:

    Sophos Anti-Virus for Linux /UNIX: Running and Configuring on-demand scans

    Overview

    This article describes the steps to configure on-demand scans on Sophos anti-virus for UNIX. An on-demand scan is a scan that you initiate. You can scan anything from a single file to everything on your computer that you have permission to read. You can either manually run an on-demand scan or schedule it to run unattended.

    The command that you type to run an on-demand scan is savscan.

    The following sections are covered:

    Applies to the following Sophos products and versions

    Sophos Anti-Virus for Unix

    Run an on-demand scan of the computer

    • To run an on-demand scan of the computer, type:

    savscan /

    Scan a particular directory or file

    • To scan a particular directory or file, specify the path of the item. For example, type:

    savscan /usr/mydirectory/myfile

    You can type more than one directory or file in the same command.

    Scan a filesystem

    • To scan a filesystem, specify its name. For example, type:

    savscan /home

    You can type more than one filesystem in the same command.

    In this section, where path appears in a command, it refers to the path to be scanned.

    • To see a full list of the options that you can use with an on-demand scan, type:

    man savscan

    Scan all file types

    By default, Sophos Anti-Virus scans only executables. To see a full list of the file types that Sophos

    Anti-Virus scans by default, type savscan -vv.

    • To scan all file types, not just those that are scanned by default, use the option -all. Type:

    savscan path -all

    Note:This makes scanning take longer, can compromise performance on servers, and can cause

    false virus reports

    Scan a particular directory or file

    • To scan a particular directory or file, specify the path of the item. For example, type:

    savscan /usr/mydirectory/myfile

    You can type more than one directory or file in the same command.

    Scan inside all archive types

    You can configure Sophos Anti-Virus to scan inside all archive types.

    • To see a list of these archive types, type:

    savscan -vv.

    Note: The threat detection engine only scans archived files that are up to 8GB (when decompressed).

    This is because it supports the POSIX ustar archive format, which does not accommodate larger files.

    • To scan inside all archive types, use the option -archive. Type:

    savscan path -archive

    Archives that are “nested” within other archives (for example, a TAR archive within a ZIP archive) are scanned recursively. If you have numerous complex archives, the scan may take longer to run. Bear this in mind when scheduling unattended scans.

    Scan inside a particular archive type

    • To scan inside a particular archive type, use the option that is shown in the list. For example, to scan inside TAR and ZIP archives, type:

    savscan path -tar -zip

    Archives that are “nested” within other archives (for example, a TAR archive within a ZIP archive) are scanned recursively. If you have numerous complex archives, the scan may take longer to run. Bear this in mind when scheduling unattended scans.

    Scan remote computers

    By default, Sophos Anti-Virus does not scan items on remote computers (that is, does not traverse remote mount points).

    • To scan remote computers, use the option –no-stay-on-machine. Type:

    savscan path --no-stay-on-machine

    Turn off scanning of symbolically linked items

    By default, Sophos Anti-Virus scans symbolically linked items.

    • To turn off scanning of symbolically linked items, use the option –no-follow-symlinks. Type:

    savscan path --no-follow-symlinks

    To avoid scanning items more than once, use the option –backtrack-protection.

    Scan the starting filesystem only

    Sophos Anti-Virus can be configured not to scan items that are beyond the starting filesystem (that is, not to traverse mount points).

    • To scan the starting filesystem only, use the option –stay-on-filesystem. Type:

    savscan path --stay-on-filesystem

    Excluding items from scanning

    • You can configure Sophos Anti-Virus to exclude particular items (files, directories, or filesystems) from scanning by using the option -exclude. Sophos Anti-Virus excludes any items that follow the option in the command string. For example, to scan items fred and harry, but not tom or peter, type:

    savscan fred harry -exclude tom peter

    • You can exclude directories or files that are under a particular directory. For example, to scan all of Fred’s home directory, but exclude the directory games (and all directories and files under it), type:

    savscan /home/fred -exclude /home/fred/games

    • You can also configure Sophos Anti-Virus to include particular items that follow the option -include. For example, to scan items fred, harry, and bill, but not tom or peter, type:

    savscan fred harry -exclude tom peter -include bill

    Scan file types that UNIX defines as executables

    By default, Sophos Anti-Virus does not scan file types that UNIX defines as executables.

    • To scan file types that UNIX defines as executables, use the option –examine-x-bit. Type:

    savscan path --examine-x-bit

    Sophos Anti-Virus still scans files that have filename extensions that are in its own list as well. To see a list of these filename extensions, type savscan -vv.

    If you’ve spotted an error or would like to provide feedback on this article, please use the section below to rate and comment on the article.

    This is invaluable to us to ensure that we continually strive to give our customers the best information possible.

    Related:

    • No Related Posts

    Sophos Anti-Virus for UNIX: Running and Configuring on-demand scans

    Overview

    This article describes the steps to configure on-demand scans on Sophos anti-virus for UNIX. An on-demand scan is a scan that you initiate. You can scan anything from a single file to everything on your computer that you have permission to read. You can either manually run an on-demand scan or schedule it to run unattended.

    The command that you type to run an on-demand scan is savscan.

    The following sections are covered:

    Applies to the following Sophos products and versions

    Sophos Anti-Virus for Unix

    Run an on-demand scan of the computer

    • To run an on-demand scan of the computer, type:

    savscan /

    Scan a particular directory or file

    • To scan a particular directory or file, specify the path of the item. For example, type:

    savscan /usr/mydirectory/myfile

    You can type more than one directory or file in the same command.

    Scan a filesystem

    • To scan a filesystem, specify its name. For example, type:

    savscan /home

    You can type more than one filesystem in the same command.

    In this section, where path appears in a command, it refers to the path to be scanned.

    • To see a full list of the options that you can use with an on-demand scan, type:

    man savscan

    Scan all file types

    By default, Sophos Anti-Virus scans only executables. To see a full list of the file types that Sophos

    Anti-Virus scans by default, type savscan -vv.

    • To scan all file types, not just those that are scanned by default, use the option -all. Type:

    savscan path -all

    Note:This makes scanning take longer, can compromise performance on servers, and can cause

    false virus reports

    Scan a particular directory or file

    • To scan a particular directory or file, specify the path of the item. For example, type:

    savscan /usr/mydirectory/myfile

    You can type more than one directory or file in the same command.

    Scan inside all archive types

    You can configure Sophos Anti-Virus to scan inside all archive types.

    • To see a list of these archive types, type:

    savscan -vv.

    Note: The threat detection engine only scans archived files that are up to 8GB (when decompressed).

    This is because it supports the POSIX ustar archive format, which does not accommodate larger files.

    • To scan inside all archive types, use the option -archive. Type:

    savscan path -archive

    Archives that are “nested” within other archives (for example, a TAR archive within a ZIP archive) are scanned recursively. If you have numerous complex archives, the scan may take longer to run. Bear this in mind when scheduling unattended scans.

    Scan inside a particular archive type

    • To scan inside a particular archive type, use the option that is shown in the list. For example, to scan inside TAR and ZIP archives, type:

    savscan path -tar -zip

    Archives that are “nested” within other archives (for example, a TAR archive within a ZIP archive) are scanned recursively. If you have numerous complex archives, the scan may take longer to run. Bear this in mind when scheduling unattended scans.

    Scan remote computers

    By default, Sophos Anti-Virus does not scan items on remote computers (that is, does not traverse remote mount points).

    • To scan remote computers, use the option –no-stay-on-machine. Type:

    savscan path --no-stay-on-machine

    Turn off scanning of symbolically linked items

    By default, Sophos Anti-Virus scans symbolically linked items.

    • To turn off scanning of symbolically linked items, use the option –no-follow-symlinks. Type:

    savscan path --no-follow-symlinks

    To avoid scanning items more than once, use the option –backtrack-protection.

    Scan the starting filesystem only

    Sophos Anti-Virus can be configured not to scan items that are beyond the starting filesystem (that is, not to traverse mount points).

    • To scan the starting filesystem only, use the option –stay-on-filesystem. Type:

    savscan path --stay-on-filesystem

    Excluding items from scanning

    • You can configure Sophos Anti-Virus to exclude particular items (files, directories, or filesystems) from scanning by using the option -exclude. Sophos Anti-Virus excludes any items that follow the option in the command string. For example, to scan items fred and harry, but not tom or peter, type:

    savscan fred harry -exclude tom peter

    • You can exclude directories or files that are under a particular directory. For example, to scan all of Fred’s home directory, but exclude the directory games (and all directories and files under it), type:

    savscan /home/fred -exclude /home/fred/games

    • You can also configure Sophos Anti-Virus to include particular items that follow the option -include. For example, to scan items fred, harry, and bill, but not tom or peter, type:

    savscan fred harry -exclude tom peter -include bill

    Scan file types that UNIX defines as executables

    By default, Sophos Anti-Virus does not scan file types that UNIX defines as executables.

    • To scan file types that UNIX defines as executables, use the option –examine-x-bit. Type:

    savscan path --examine-x-bit

    Sophos Anti-Virus still scans files that have filename extensions that are in its own list as well. To see a list of these filename extensions, type savscan -vv.

    If you’ve spotted an error or would like to provide feedback on this article, please use the section below to rate and comment on the article.

    This is invaluable to us to ensure that we continually strive to give our customers the best information possible.

    Related:

    • No Related Posts

    Why is brptviewer displaying an error message when processing csm files?.

    Why is brptviewer displaying an error message when processing csm files?.

    dataloader1 -> /opt/bmi/brptviewer/brptviewer.sh -f 20160622101500 -t 20160622114000

    Error in reading webproxy brpt from package /dataloader_in/csm/pkg_20160612_11204.tar.gz.

    No webproxy file /dataloader_in/csm/pkg_20160622_11204.tar.gz

    Skipping it

    Error in reading webproxy brpt from package /dataloader_in/csm/pkg_20160622_11203.tar.gz.

    No webproxy file /dataloader_in/csm/pkg_20160622_11203.tar.gz

    Skipping it

    This is an informational message, so it’s not related to any issue.

    Dataloader looks into all packages available in the input directory, so Dataloader doesn’t know the webproxy brpt is not in there until it looks into all files.

    Related:

    OneFS IntegrityScan

    Under normal conditions, OneFS typically relies on checksums, identity field, and magic numbers to verify file system health and correctness. Within OneFS, system and data integrity, can be subdivided into four distinct phases:

    integrityscan_2.png

    Here’s what each of these phases entails:

    Phase

    Description

    Detection

    The act of scanning the file system and detecting data block instances that are not what OneFS expects to see at that logical point. Internally, OneFS stores a checksum or IDI (Isi data integrity) for every allocated block under /ifs.

    Enumeration

    Enumeration involves notifying the cluster administrator of any file system damage uncovered in the detection phase. For example, logging to the /var/log/idi.log file.

    Isolation

    Isolation is the act of cauterizing the file system, ensuring that any damage identified during the detection phase does not spread beyond the file(s) that are already affected. This typically involves removing all references to the file(s) from the file system.

    Repair

    Repairing any damage discovered and removing the damaged file(s) from OneFS. Typically a DSR (Dynamic Sector recovery) is all that is required to rebuild a block that fails IDI.

    Focused on the detection phase, the primary OneFS tool for uncovering system integrity issues is IntegrityScan. This job is run on across the cluster to discover instances of damaged files, and provide an estimate of the spread of the damage.

    Unlike traditional ‘fsck’ style file system integrity checking tools (including OneFS’ isi_cpr utility), IntegrityScan is explicitly designed to run while the cluster is fully operational – thereby removing the need for any downtime. It does this by systematically reading every block and verifying its associated checksum. In the event that IntegrityScan detects a checksum mismatch, it generates an alert, logs the error to the IDI logs (/var/log/idi.log), and provides a full report upon job completion.

    IntegrityScan is typically run manually if the integrity of the file system is ever in doubt. By default, the job runs at am impact level of ‘Medium’ and a priority of ‘1’ and accesses the file system via a LIN scan. Although IntegrityScan itself may take several hours or days to complete, the file system is online and completely available during this time. Additionally, like all phases of the OneFS job engine, IntegrityScan can be re-prioritized, paused or stopped, depending on its impact to cluster operations. Along with Collect and MultiScan, IntegrityScan is part of the job engine’s Marking exclusion set.

    integrityscan_1.png

    OneFS can only accommodate a single marking job at any point in time. However, since the file system is fully journalled, IntegrityScan is only needed in exceptional situations. There are two principle use cases for IntegrityScan:

    • Identifying and repairing corruption on a production cluster. Certain forms of corruption may be suggestive of a bug, in which case IntegrityScan can be used to determine the scope of the corruption and the likelihood of spreading. It can also fix some forms of corruption.

    • Repairing a file system after a lost journal. This use case is much like traditional fsck. This scenario should be treated with care as it is not guaranteed that IntegrityScan fixes everything. This is a use case that will require additional product changes to make feasible.

    IntegrityScan can be initiated manually, on demand. The following CLI syntax will kick off a manual job run:

    # isi job start integrityscan

    Started job [283]

    # isi job list

    ID Type State Impact Pri Phase Running Time

    ————————————————————

    283 IntegrityScan Running Medium 1 1/2 1s

    ————————————————————

    Total: 1

    With LIN scan jobs, even though the metadata is of variable size, the job engine can fairly accurately predict how much effort will be required to scan all LINs. The IntegrityScan job’s progress can be tracked via a CLI command, as follows:

    # isi job jobs view 283

    ID: 283

    Type: IntegrityScan

    State: Running

    Impact: Medium

    Policy: MEDIUM

    Pri: 1

    Phase: 1/2

    Start Time: 2018-09-05T22:20:58

    Running Time: 31s

    Participants: 1, 2, 3

    Progress: Processed 947 LINs and approx. 7464 MB: 867 files, 80 directories; 0 errors

    LIN & SIN Estimate based on LIN & SIN count of 3410 done on Sep 5 22:00:10 2018 (LIN) and Sep 5 22:00:10 2018 (SIN)

    LIN & SIN Based Estimate: 1m 12s Remaining (27% Complete)

    Block Based Estimate: 10m 47s Remaining (4% Complete)

    Waiting on job ID: –

    Description:

    The LIN (logical inode) statistics above include both files and directories.

    Be aware that the estimated LIN percentage can occasionally be misleading/anomalous. If concerned, verify that the stated total LIN count is roughly in line with the file count for the cluster’s dataset. Even if the LIN count is in doubt. The estimated block progress metric should always be accurate and meaningful. If the job is in its early stages and no estimation can be given (yet), isi job will instead report its progress as ‘Started’. Note that all progress is reported per phase.



    A job’s resource usage can be traced from the CLI as such:



    # isi job statistics view

    Job ID: 283

    Phase: 1

    CPU Avg.: 30.27%

    Memory Avg.

    Virtual: 302.27M

    Physical: 24.04M

    I/O

    Ops: 2223069

    Bytes: 16.959G

    Finally, upon completion, the IntegrityScan job report, detailing both job stages, can be viewed by using the following CLI command with the job ID as the argument:

    # isi job reports view 283

    IntegrityScan[283] phase 1 (2018-09-05T22:34:56)

    ————————————————

    Elapsed time 838 seconds (13m58s)

    Working time 838 seconds (13m58s)

    Errors 0

    LINs traversed 3417

    LINs processed 3417

    SINs traversed 0

    SINs processed 0

    Files seen 3000

    Directories seen 415

    Total bytes 178641757184 bytes (166.373G)

    IntegrityScan[283] phase 2 (2018-09-05T22:34:56)

    ————————————————

    Elapsed time 0 seconds

    Working time 0 seconds

    Errors 0

    LINs traversed 0

    LINs processed 0

    SINs traversed 0

    SINs processed 0

    Files seen 0

    Directories seen 0

    Total bytes 0 bytes

    In addition to the IntegrityScan job, OneFS also contains an ‘isi_iscan_report’ utility. This is a tool to collate the errors from IDI log files (/var/log/idi.log) generated on different nodes. It generates a report file which can be used as an input to ‘isi_iscan_query’ tool. Additionally, it reports the number of errors seen for each file containing IDI errors. At the end of the run, a report file can be found at /ifs/.ifsvar/idi/tjob.<pid>/log.repo.

    The associated ‘isi_iscan_query’ utility can then be used to parse the log.repo report file and filter by node, time range, or block address (baddr). The syntax for the isi_iscan_query tool is:

    /usr/sbin/isi_iscan_query filename [FILTER FIELD] [VALUE]

    FILTER FIELD:

    node <logical node number> e.g. 1, 2, 3, …

    timerange <start time> <end time> e.g. 2018-09-05T17:38:02Z 2018-09-06T17:38:56Z

    baddr <block address> e.g. 2,1,185114624:8192

    Related:

    Re: Re: How to add VM client from CLI

    I will start by assuming that you’ll use the same VMware vCenter name and same Domain in your list, and only the VM name will change. For this, I would create a text file called “vm.txt” that contained each VM name on its own line and no other characters, i.e.

    VM1

    VM2

    VM3

    etc.

    If you’re trying to build this file on a Windows computer in a text editor, be 100% sure your file is encoded for ASCII.

    I would next make a file called “rename.sh” on the Utility Node and put in the following:

    #!/bin/bash

    for i in `cat vm.txt`

    do

    java -jar proxycp.jar –addvm –vc my.vcenter.org –vm $i –domain my.domain.org

    done



    What this does is simply read each line in the file and run the command against the VM name. In the above script, you may also need to provide the full path to the java command or to the proxycp.jar file, depending on your shell environment and path settings.



    Let us know if that helps!

    Karl

    Related:

    Searching for Symlinks

    Recently received an inquiry from the field about how to find out if a data on a cluster contains symlinks, and thought it was worth sharing.

    Probably the simplest method to determine whether a set of data contains symbolic links is by using the ‘find’ utility from the OneFS command line interface, or from an NFS client. The following syntax will confirm that symlinks are present under a directory (in this case /ifs/test), and provide a count of the number of links the dataset contains:

    # find /ifs/test -type l | wc -l

    OneFS (and most Linux flavors) has the extended find command, where the –ls flag will return the link, target path and metadata for each symlink:

    # find /ifs/test -type l -ls

    438343 0 lrwxr-xr-x 1 root wheel 26 Jul 18 14:27 /ifs/test/slink1 -> /ifs/home1/file1

    438352 0 lrwxr-xr-x 1 root wheel 26 Jul 18 14:31 /ifs/test/dir1/slink2 -> /ifs/home2/file2

    453127 0 lrwxr-xr-x 1 root wheel 26 Jul 18 14:31 /ifs/test/dir2/slink3 -> /ifs/home3/file3

    364228 0 lrwxr-xr-x 1 root wheel 26 Jul 18 14:27 /ifs/test/slink4 -> /ifs/home4/file4

    For UNIX clients that don’t support the extended find command, the following syntax will provide similar output:

    # find /ifs/test -type l -exec ls -al {} ;

    lrwxr-xr-x 1 root wheel 26 Jul 18 14:27 /ifs/test/slink1 -> /ifs/home1/file1

    lrwxr-xr-x 1 root wheel 26 Jul 18 14:31 /ifs/test/dir1/slink2 -> /ifs/home2/file2

    lrwxr-xr-x 1 root wheel 26 Jul 18 14:31 /ifs/test/dir2/slink3 -> /ifs/home3/file3

    lrwxr-xr-x 1 root wheel 26 Jul 18 14:27 /ifs/test/slink4 -> /ifs/home4/file4

    The find command can easily be filtered to return just the symlink and target file. For example:

    # find /ifs/symmlink -type l -exec ls -al {} ; | awk -F ‘ ‘ ‘{print $9,$10,$11}’

    /ifs/test/slink1 -> /ifs/home1/file1

    /ifs/test/dir1/a.txt -> /ifs/home2/file2

    /ifs/test/dir2/c.txt -> /ifs/home3/file3

    /ifs/test/slink4 -> /ifs/home4/file4

    Symbolic linking (or soft linking) has been a feature in POSIX operating systems for decades, but was only fully added to Windows and the NFTS filesystem much later, in the Vista and SMB2 timeframe.

    From the Windows command shell, the following command will show the symbolic links (or ‘reparse points’ in Windows vernacular) on an SMB share mapped to drive O:

    > dir /AL /S O:

    Or, using Windows PowerShell instead:

    > Dir O: -Force –Recurse –ErrorAction ‘silentlycontinue’ |

    Where { $_.Attributes –match “ReparsePoint”}

    Symlinks are represented as follows in the ‘dir’ output:

    07/14/2018 04:01 PM <SYMLINK> slink1.txt [file1.txt]

    Unlike shortcut files, Windows symbolic links have no size and are not transparent to all applications. While symbolic links do directly connect to the data blocks of their target, their handling is determined by the symbolic link’s file extension. For example, a symbolic link created using:

    >mklink slink1.jpg file1.doc

    It’s worth noting that when accessing ‘slink1.jpg’ above, a Windows host will attempt to open the file with the default viewer for .jpg files rather than .doc files.

    Windows symlinks are created via the SMB2 protocol by writing a regular file in the initial Create request, then converting it into a symlink with a ‘Set Reparse Point’ request.



    Whenever a server completing and SMB2 request encounters a symlink, it returns a ‘Symbolic Link Error Response’, which includes the following:



    • Target path (stored in the inode)
    • The remaining section of the original file path
    • Whether the target path is relative or absolute.



    From here, the client is expected to formulate another request to the server using the details from this response. If there are three symlinks in the path, this process will be repeated three times as each link in the path is reached.

    As with the UNIX ‘ls –l’ command, Windows ‘dir’ command will also display a symlink target. When performing the dir command, a find request of info level SMB2_FIND_ID_BOTH_DIRECTORY_INFO is used to enumerate all the files in a directory. Included with this are file attributes, which can identify a file as a reparse point.

    Windows and SMB make a distinction between links to files and links to directories. Once a link is set to be of a certain type, this flag cannot change. Directory links are fully supported in OneFS 8.0 and beyond.

    Other Windows link constructs include ‘widelinks’ and ‘absolute links’. Absolute links have no parallel in POSIX, and are links to specific drive letters or UNC paths. OneFS 7.2 and later supports absolute links to both UNC paths and drive letters.

    A widelink is a reparse point that is constructed using a lookup table and is designed for interoperability between SMB and POSIX systems.

    For instance, if the translation is:

    /mnt/serverA ===> \serverA

    Then the same link can point to 2 different places:

    link1 -> /mnt/serverA/share1/ (NFS)

    link1 -> \serverAshare1 (SMB)

    Note: OneFS does not yet support widelinks.

    OneFS provides the ability to disable all SMB symlink features via a registry key. This key is called SMB2Symlinks and can be accessed using the OnefsConfigSMB2Symlinks function. To enable or disable symlinks, run the lwregshell utility from the OneFS CLI as follows:

    # cd /usr/likewise/bin

    # ./lwregshell

    > cd HKEY_THIS_MACHINEServiceslwioParametersDriversonefs

    ls /* to check the current value */

    set_value “SMB2Symlinks” 0 /* 0 to disable, 1 to enable */

    Returning to the original question, we demonstrated how to determine if there are symlinks with a OneFS dataset. However, there is no simple way to determine whether Windows clients are actually trying to use and/or create them. That said, there are a couple of options available:



    • Trace the client’s activity via the Windows “Process Monitor” tool.
    • Take packet captures and analyze them with a protocol sniffer like Wireshark to find the appropriate RPCs.



    For example, when selecting a linked file, a windows client creates a request which includes the name of the symbolic link. The cluster then sends a response of an error type 0x8000002D (Note: Wireshark doesn’t appear to know how to classify this error and instead designates it as a malformed packet).



    Length of Response-¬

    Start-¬ Reserved ⌐-Byte Count

    0070 00 00 00 00 00 00 00 00 00 00│09 00│00 00│a8 00 …………….

    SymLink Length¬ SymLink ErrTag¬ Reparse Tag ⌐Reparse Data Length

    0080 00 00│a4 00 00 00│53 59 4d 4c│0c 00 00 a0│98 00│ ……SYML……

    A-¬ B-¬ C-¬ D-¬ E¬ Flags¬ Path Buffer¬ Print Name-¬

    0090 00 00│46 00│46 00│00 00│46 00│01 00 00 00│5c 00 ..F.F…F……

    00a0 74 00 65 00 73 00 74 00 5c 00 64 00 6f 00 63 00 t.e.s.t..d.o.c.

    00b0 73 00 4c 00 69 00 6e 00 6b 00 5c 00 57 00 69 00 s.L.i.n.k..W.i.

    00c0 6e 00 4c 00 69 00 6e 00 6b 00 5c 00 74 00 65 00 n.L.i.n.k..t.e.

    00d0 73 00 74 00 46 00 69 00 6c 00 65 00 2e 00 74 00 s.t.F.i.l.e…t.

    Substitute Name-¬

    00e0 78 00 74 00 5c 00 74 00 65 00 73 00 74 00 5c 00 x.t..t.e.s.t..

    00f0 64 00 6f 00 63 00 73 00 4c 00 69 00 6e 00 6b 00 d.o.c.s.L.i.n.k.

    0100 5c 00 57 00 69 00 6e 00 4c 00 69 00 6e 00 6b 00 .W.i.n.L.i.n.k.

    0110 5c 00 74 00 65 00 73 00 74 00 46 00 69 00 6c 00 .t.e.s.t.F.i.l.

    0120 65 00 2e 00 74 00 78 00 74 00 e…t.x.t.

    SymLink Length: The length of the response excluding itself.

    Symlink ErrTag: Must be set to 0x4C4D5953.

    Reparse Tag: Type of link, must be set to 0xA000000C.

    Reparse Data Length: The size in bytes of PathBuffer+A+B+C+D+Flags.

    A: Unparsed Path Length: The length in bytes of the unparsed portion of the path remaining after the symbolic link. As the link is

    at file1.txt, this is 0.

    B: Substitute Name Offset: The offset in bytes from the beginning of the path buffer field at which the substitute name is located.

    C: Substitute Name Length: The length in bytes of the substitute name.

    D: Print Name Offset: The offset from the path buffer field at which the user friendly name is located.

    E: Print Name Length.

    Flags: Defines if the substitute name is relative or absolute. 0x00000001 = SYMLINK_FLAG_RELATIVE while 0 = absolute.

    Path Buffer: Variable length unicode string holding the substitute and print name. For an absolute name, the format must be

    ??UNCservershare…, not a local evaluation. According to the rules of the error, a relative path should not start with ,

    but apparently that is not true in this case.



    In this example, a Windows client is searching a cluster share for a symbolic link named ‘testfile1.txt’, pointing to ‘testdocsLinkWinLinktestFile.txt’.

    Related:

    DELL EMC Data Domain Cleaning

    Deep dive into Data Domain GC or cleaning



    When your backup application (such as NetBackup or NetWorker) expires data, the data is marked by the Data Domain system for deletion. However, the data is not deleted immediately; it is removed during a cleaning operation.

    • During the cleaning operation, the file system is available for all normal operations including backup (write) and restore (read).
    • Although cleaning uses a significant amount of system resources, cleaning is self throttling and gives up system resources in the presence of user traffic.
    • Data Domain recommends running a cleaning operation after the first full backup to a Data Domain system. The initial local compression on a full backup is generally a factor of 1.5 to 2.5. An immediate cleaning operation gives additional compression by another factor of 1.15 to 1.2 and reclaims a corresponding amount of disk space.
    • When the cleaning operation finishes, a message is sent to the system log giving the percentage of storage space that was reclaimed.

    As files are written to a Data Domain system, they are deduplicated which means



    • Duplicate data in the file is replaced with a pointer to existing data on disk.
    • Only unique data is placed in ~128 Kb compression regions which sit in 4.5 Mb containers on disk.



    This reflects unique data uses space on disk and unique data written by one file maybe referred to by one or more other files (written later, however contains the same data)Cleaning works on data on disk which is completely unreferenced, this data is deemed to be ‘dead’ and therefore superfluous to the system.Garbage Collection/Cleaning is used to find ‘dead’ data on the system (enumeration) and remove this data and make the corresponding space available for re-use (copy forward).



    GC.jpg







    History of Garbage Collection (GC) algorithms:



    The biggest challenge with cleaning we had was its speed, it was slow and there is no other way to free space.



    Currently we have 12 phases of cleaning.



    Cleaning: phase 1 of 12 (pre-merge)

    Cleaning: phase 2 of 12 (pre-analysis)

    Cleaning: phase 3 of 12 (pre-enumeration)

    Cleaning: phase 4 of 12 (pre-filter)

    Cleaning: phase 5 of 12 (pre-select)

    Cleaning: phase 6 of 12 (merge)

    Cleaning: phase 7 of 12 (analysis)

    Cleaning: phase 8 of 12 (candidate)

    Cleaning: phase 9 of 12 (enumeration)

    Cleaning: phase 10 of 12 (filter)

    Cleaning: phase 11 of 12 (copy)

    Cleaning: phase 12 of 12 (summary)



    DD OS 5.4 and earlier



    DD OS 5.4 and previous versions used to have Full cleaning algorithm.

    • 10 phases.
    • File centric (clean enumerates each file on the system to work out which data is live).
    • Did not scale well for systems with a very large data set or a lot of small files.

    DD OS 5.5.- 5.7



    Data Domain starting DD OS 5.5 up to DD OS 5.7 uses Physical cleaning (PGC)

    • 12 phases.
    • Data centric (clean enumerates metadata within the file system to work out which data is live).
    • Scaled better for system with a very large data set or a lot of small files but perhaps not as well as was originally envisaged.



    DD OS 6.0 and later



    Starting DD OS 6.0 engineers came up with yet another way to constantly improve the cleaning process. This is called perfect physical cleaning (PPGC)PPGC is essentially the same as PGC, however only changes are how memory is used. This allows certain phases to be skipped meaning that clean can run more quickly on the majority of the systems.



    The need of sampling



    Clean is allocated a fixed amount of memory to hold its data structures, In most cases this memory is not sufficient to track whether all data on a Data Domain is live or dead, this means that most DD units leveraging FULL or PGC have to perform ‘sampling’ .



    • The file system is split into logical chunks.
    • Each chunk is sampled to work out how dead data it is likely to contain.
    • Clean selects the chunk expected to give best return in free space
    • This chunk is then cleaned (i,e enumerated/ copied forwards)

    Thus sampling means that we need to run a number of phases twice and wastes a lot of time.With PPGC (DD OS 6.0 and later) optimizes the above approach and avoids sampling by tracking a much larger amount of data in the same amount of GC memory (approx 4.4x more segments compared to PGC). This reduces overall clean duration by 20-50 %.



    Some Prerequisites of using PPGC

    • DD OS 6.0 and later
    • Index files must be in the index 2.0 format – this will always be the case as indices must be upgraded to index 2.0 before the upgrade to DD OS 6.x can proceed (https://support.emc.com/kb/495604).
    • Segment information for the entire data set must be able to fit in GC memory – Whilst PPGC can track approximately 4.4x more segments in the same format of RAM as PGC there will still be some very large systems where this is still not the case.
    • On applicable systems PPGC will be automatically enabled – users do not need to ‘turn on’ or ‘switch to’ PPGC, other systems will automatically fall back on PGC (and sampling).

    Once a DDR with index files in the index 1.0 format is upgraded to DDOS 5.5.x/5.6.x/5.7.x index files will automatically be converted to the index 2.0 format. Note, however, that:

    • Index file conversion is performed when clean/garbage collection is performed on the DDR
    • The conversion is performed in a lazy style (i.e. the entire conversion is not performed at once) with only ~1% of index files being converted during each clean

    As a result it can take considerable time for all index files to be fully converted to the index 2.0 format and until this is complete upgrades to DDOS 6.x will fail during pre-check (with the error shown in above). Systems in this state can either:



    • Be left to run as they are (i.e. on DDOS 5.5.x/5.6.x/5.7.x) until conversion of index files naturally completes – at this point an upgrade to DDOS 6.x will be allowed to proceed. Note, however, that with standard user interfaces (i.e. the Data Domain System Manager/Enterprise Manager/Management Center/command line shell) it is not possible to determine the format/state of index files and therefore it is not easy to determine when the system is ready for upgrade.

    • Have their index files forcibly converted to the index 2.0 format – this then allows an upgrade to DDOS 6.x to take place immediately

    Forcibly converting index files requires elevated access to the Data Domain Operating System (DDOS) via the command line shell (and therefore cannot be performed by end users/administrators). There is no way to tell whether PGC or PPGC is being used other than the number of phases run during the operation.



    When clean starts, it will report that it is running all 12 phases, however will skip phases, for example jump straight from

    phase 5 (pre-select) to phase 11 (copy), PGC will continue to run all 12 phases.



    GC1.jpg





    Disabling PPGC

    • To disable PPGC and force a system to use PGC

    # filesys disable# se sysparam set GC_PPGC_IS_ENABLED=FALSE# filesys enable



    • To disable PPGC and PGC and force a system to use traditional/full GC)

    # filesys disable

    # se sysparam set GC_PHYSICAL_ENABLED=FALSE

    # filesys enable

    Note that traditional/full GC should be avoided on long term retention (LTR) enabled systems as this can cause DDFS panics, this is purely based on experience.



    Hope it helps!!

    Related: