Accessing FSA results through Platform API

Many Isilon administrators have been asking about how to utilize data that are already being gathered through the FS Analyze (FSA) job engine job. Many have wondered if it was possible to directly access the on-cluster FSA database to obtain file/directory counts and all the goodness that comes with knowing what’s on your cluster.

For good reasons, Isilon Engineering has not published the details of the database schema for the FSA database, as it was not designed to be user-facing.

With “PAPI-fication” of FSA results since OneFS v8.0, there is no longer a need to publish the FSA database schema. “FSA PAPI-fication” is our fancy way of saying we now have Platform API handlers to access specific functional portions of FSA results. This was done for OneFS v8.0 and beyond as a way for InsightIQ to obtain FSA results through Platform API – thus obsoleting the need to mount an NFS share from the InsightIQ server/appliance to the cluster for obtaining FSA results.

This blog will outline some examples of getting at the FSA results through Platform API handlers.

To invoke the FSA resources in Platform API, the API user needs to have the “read” permissions on ISI_PRIV_STATISTICS. Ensure that the API user has been granted this privilege in Membership & Roles. This example user below has the correct privileges to invoke the FSA resources in the OneFS API:

api-stats-role.png

My role “API_stats_role” has exactly these 2 privileges associated with it. My user, “apiuser”, is then assigned to this role.

The list of FSA Platform API resources are outlined in the OneFS API documentation. For OneFS v8.0.0.x, the online version of the doc is located here — http://doc.isilon.com/onefs/8.0.0/api/en-us/03-ifs-br-system-config-api.htm. From the navigation pane on the right, find the “FSA” section for a list of FSA-related API handlers.

The API handlers are self-documenting. For most of the API handlers listed in the FSA section, you could obtain the schema information by invoking the API resource with the “?describe” query.

As an example, here’s a quick script that will perform a session authentication and then invoke three different API handlers with the “?describe” query.

[17:00:43] rchang@USENCHANGR4D2C:[~/api-work/riptide/tme-sandbox]:cat get-fsa.sh

# my isilon target

my_cluster=”xxx.xxx.xxx.xxx”

# my creds

username=apiuser

password=apipass

# prep auth.json file

echo “{

“username”:”${username}”,

“password”:”${password}”,

“services”:[“platform”,”namespace”]

} ” > auth.json

# get session cookie

curl -k –insecure -H “Content-Type: application/json” -c cookiefile -X POST -d @auth.json https://${my_cluster}:8080/session/1/session

echo ===== calling /platform/3/fsa/results?describe =====

curl -k –insecure -b @cookiefile https://${my_cluster}:8080/platform/3/fsa/results?describe

echo

echo ===== calling /platform/3/fsa/results/0?describe =====

curl -k –insecure -b @cookiefile https://${my_cluster}:8080/platform/3/fsa/results/0?describe

echo

echo ===== calling /platform/3/fsa/results/0/directories?describe =====

curl -k –insecure -b @cookiefile https://${my_cluster}:8080/platform/3/fsa/results/0/directories?describe

echo

The output is extensive, and describes the schema elements.

A special note — here’s a very useful short-cut for referencing the latest, completed FSA result. We can reference the latest, completed FSA result by using result ID of 0. This saves us from having to first obtaining a list of completed FSA results and then selecting one by referencing its result ID.

In the example script above, the three API calls all referenced the result ID of 0, getting at the latest completed FSA result set.

For our practical example, let’s find the file and subdirectory count for a specific path, using the latest FSA result set.

The API handler we’ll use is outlined in red below (from the API documentation). We will use the query path argument to inquire about a specific path.

FSA-PAPI-resource-example.png

Using the script provided in the previous section, the schema elements we’ll need to examine are:

[09:39:33] rchang@USENCHANGR4D2C:[~/api-work/riptide/tme-sandbox]:. get-fsa.sh | grep usage

“total_usage”: {

“description”: “Disk usage from root.”,

“usage_data”: {

“description”: “Disk usage for all of immediate children of the current directory.”

“dir_usage”: {

“description”: “Disk usage for current directory.”,

This is the API call: /platform/3/fsa/results/0/directories

For the directory path of: /ifs/System/VLAB/administrator

===== calling /platform/3/fsa/results/0/directories?path=/ifs/System/VLAB/administrator =====

{

“attribute_count”: 0,

“begin_time”: 1495663284,

“dir_depth”: 5,

“dir_usage”: {

“ads_cnt”: 0,

“dir_cnt”: 3,

“file_cnt”: 6,

“lin”: 4379901960,

“log_size_sum”: 8902,

“log_size_sum_overflow”: 0,

“name”: “administrator”,

“other_cnt”: 0,

“parent”: 4363452425,

“phys_size_sum”: 216576

},

“path_parts”: [

[

“System”,

4363452424

],

[

“VLAB”,

4363452425

],

[

“administrator”,

4379901960

]

],

“total_usage”: {

“ads_cnt”: 0,

“dir_cnt”: 27,

“file_cnt”: 77,

“lin”: 2,

“log_size_sum”: 914039713,

“log_size_sum_overflow”: 0,

“name”: “”,

“other_cnt”: 0,

“parent”: 1,

“phys_size_sum”: 1836782592

},

“usage_data”: [

{

“ads_cnt”: 0,

“dir_cnt”: 2,

“file_cnt”: 6,

“has_subdirs”: true,

“lin”: 4383834137,

“log_size_sum”: 8876,

“log_size_sum_overflow”: 0,

“name”: “myfolder”,

“other_cnt”: 0,

“parent”: 4379901960,

“phys_size_sum”: 214016

}

]

}

See that the dir_usage, total_usage, and usage_data objects are returned with their perspective metrics. In this example, we want to find the file count and subdirectory count for the path “/ifs/System/VLAB/administrator”. We need to examine the usage_data object. The latest FSA result reports that there are 2 subdirectories and 6 files under this path. Let’s verify.

[root@isilon-1 /ifs/System/VLAB/administrator/myfolder/api-work]# ls -lR /ifs/System/VLAB/administrator

total 3

drwx—— 3 VLABadministrator VLABdomain users 54 May 23 20:39 myfolder

/ifs/System/VLAB/administrator/myfolder:

total 37

drwxrwx— 2 root wheel 144 May 23 22:36 api-work

-rwx—— 1 VLABadministrator VLABdomain users 168 May 10 23:05 myfile.txt

/ifs/System/VLAB/administrator/myfolder/api-work:

total 106

-rwxrwx— 1 root wheel 87 May 25 20:25 auth.json

-rwxrwx— 1 root wheel 215 May 25 20:25 cookiefile

-rwxrwx— 1 root wheel 1100 May 23 22:45 get-fsa.sh

-rwxrwx— 1 root wheel 1373 May 23 21:13 get-quotas.sh

-rwxrwx— 1 root wheel 5735 May 23 21:00 myquotas.txt

Indeed there are 2 subdirectories (highlighted in cyan) and 6 regular files (highlighted in green) under the path /ifs/System/VLAB/administrator.

From OneFS 8.0 and beyond, the ability to query FSA results using Platform API has opened a new way of programmatic extraction of FSA results beyond the exclusive consumption by InsightIQ. Other FSA metrics that were only available for reporting through InsightIQ can now be obtained over the API. Savvy administrators can now script and report on specific areas for embedding into custom reports and alerts.

For related information and resources, please see:

Related:

Leave a Reply