Hello everyone in the Isilon-land,
I was helping a customer who experienced time-outs during a lengthy datastore import process. I figure I’d share the problem and the proposed solution here as a blog.
Problem: The problem is that the datastore import/export functionality in InsightIQ v3.2 uses the tar.gz format to compress the exported datastore. The tar.gz compression format cannot be indexed to speed up the importing process. As a result, some large, long-running import process can time out.
Solution: A Dell EMC engineer in the Global Services team (customer support) wrote a tool to tackle this problem. Essentially the tool allows for unpacking of the tar.gz datastore export and repack it into a .zip format, which allows for indexing. Importing a datastore using the .zip format has shown to greatly speed up the import process, and thus reducing the potential of timeouts during import.
If you are currently running an old version of InsightIQ (v3.2), the database structure behind the datastore is based on PostgreSQL, same as the current version of InsightIQ (v4.1.1). There is also no database schema changes from v3.2 to v4.1.1. You should be able to use this utility to import a re-packed datastore export into InsightIQ v3.2. But if you are intending on deploying the current version of InsightIQ (v4.1.1), I would perform this unpack/repack/import process AFTER you’ve raised your InsightIQ v4.1.1 instance, using the repacked export from v3.2.
The utility is located here:
The documentation page is located here:
The utility you care about is this one:
Pasting the “readme” section here:
Starting with InsightIQ 3.2, you could export a cluster’s database from one instance, then import it later or on another InsightIQ instance. Initially, the exported data was in tar file format, but in InsightIQ 4.1 we switched to using a zip file. The switch was to resolve a bug where importing large exports would time out. The data contained within the tar and the zip files is identical; only the compression format has changed. This means that if we convert an old tar export to zip, we can use that archive in newer versions of InsightIQ.
Use cases for this script:
Instead of upgrading an existing deployment, you export the data on your old instance, use this script to convert the format, and then import that data on a new deployment of InsightIQ. This approach is ideal for OVA deployments of InsightIQ because the newer OVAs for InsightIQ have the latest security patches applied, and the root partition is configured with LVM.
Maintain Legacy Exports
With the upgrade to 4.1, any datastore exports created on the older version of InsightIQ are no longer compatible. This script will update the format of those older datastore exports so you can continue to use them in newer versions of InsightIQ.