DFS Replication as Migration Method

I’ve seen several old posts about DFS Replication and the Isilon but I’m not sure if what I want to do is what is being described. It’s been some time since I’ve worked with DFS. The theme I see most is that DFS replication isn’t supported by the Isilon, nor by any non-Windows system, but I’ve seen this work with an NS20 (long time ago) but I wasn’t the main admin.

The idea is to take some current shares on another file server and replicate them to new shares on the Isilon using DFS. We’ve done this in the past with other storage appliances, and were successful. Some of our techs prefer this method as they find it non-disruptive. Adding a folder target is no problem, but when we go set up replication, we have to pick a computer object in AD. We find our object for the Isilon (was created when I set up the AD authentication provider, has all correct SPN records) but selecting it results in an error about not being able to resolve.

Is this strictly a DNS issue where I request some sort of alias or other record from the AD record name to my SmartConnect zone alias, if that is even possible? Or is there something deeper to DFS replication that means it truly won’t work even if it could resolve?

If a DNS issue alone, what do I request? Something pointing to the SIP? Something pointing to my delegated zone for the Isilon?

Related:

App Layering: Supporting DFS File Shares

For load distribution and data availability, many customers are choosing to use DFS file shares instead of single-server shares. And for security purposes, customers are choosing to disable SMB 1.0 on those servers. Although the ELM cannot access a DFS namespace when SMB 1.0 is disabled, App Layering still can leverage DFS. Since App Layering uses native Windows for file share access, as long as Windows can access your DFS shares, and the data is there in the DFS namespace, features like Elastic and User layers will work fine.

Since the ELM itself does not support DFS, your only option is to point it directly to a shared folder on one of the servers in the namespace. The ELM will then upload the layers and configuration files normally to that specific server. On the backend, DFS replication will manage duplicating the data to the rest of the DFS servers. The data is actually available in your DFS namespace even if the ELM could not directly talk to DFS. And if the specific server the ELM is configured to talk to becomes unavailable, the file server information can be changed at will to point to another server in the cluster.

And that means that your published images absolutely can take advantage of DFS namespaces, for load sharing and data redundancy. The published images don’t need to find the files in the exact place the ELM put them, they simply need to find the files where they are told to look for them. So you just need to reconfigure your published images to read data from DFS instead of the direct file server. When publishing an image, the ELM puts in the registry the name of the share it uploaded to, so that the layering software in the image will download its elastic layers from the same location. However, that registry key can be changed after publishing, by a script or a GPO.

See https://support.citrix.com/article/CTX222107 for the registry information as well as a sample GPO template.

Related:

Re: Difference between Avamar/DD replication DDR “stream” parameters?

I’m having some difficulty understanding the difference between a couple of the various vamar/DD replication “stream” parameters.

If I recall correctly, the “max-streams” parameter is the one that determines how many clients are going to be able to run at the same time. That is, if you only wanted to replicate a single client in a replication policy, you could just choose “1”. If you wanted to replicate a number of clients but limit the number of concurrent replications to “4”, you would select “4”. And if you had a lot of clients, and you wanted to replicate as many as possible in that replication policy, you would select “8” (because you can only select values between 1 and 8).

What I’m having trouble with is the difference between the following parameters:

“max-ddr-streams” – which I always thought was how many streams each client “instance” was going to be able to use on the Data Domain(s); the default value for this appears to be “8”

“ddr-repl-max-parallel-streams” – which seems to be the same parameter, in terms of how many DDR streams will be allowed, but seems to relate specifically to replication (hence the naming, I guess); also, the default for this value is listed as “6”

They are definitely two separate parameters, so they must relate to separate circumstances. However, I can’t seem to find any documentation that clarifies the difference between them. What is also confusing me is that the first parameter does actually show up as an advanced option when configuring a replication policy, and the “help” dialog for it mentions AMS replication; however, in several KB articles about AMS replication, they make a point of saying that one has to manually add the second parameter “to allocate additional Data Domain streams” for the replication.

Can someone provide some additional clarification and insight? Thank you.

Related:

Difference between Avamar/DD replication DDR “stream” parameters?

I’m having some difficulty understanding the difference between a couple of the various vamar/DD replication “stream” parameters.

If I recall correctly, the “max-streams” parameter is the one that determines how many clients are going to be able to run at the same time. That is, if you only wanted to replicate a single client in a replication policy, you could just choose “1”. If you wanted to replicate a number of clients but limit the number of concurrent replications to “4”, you would select “4”. And if you had a lot of clients, and you wanted to replicate as many as possible in that replication policy, you would select “8” (because you can only select values between 1 and 8).

What I’m having trouble with is the difference between the following parameters:

“max-ddr-streams” – which I always thought was how many streams each client “instance” was going to be able to use on the Data Domain(s); the default value for this appears to be “8”

“ddr-repl-max-parallel-streams” – which seems to be the same parameter, in terms of how many DDR streams will be allowed, but seems to relate specifically to replication (hence the naming, I guess); also, the default for this value is listed as “6”

They are definitely two separate parameters, so they must relate to separate circumstances. However, I can’t seem to find any documentation that clarifies the difference between them. What is also confusing me is that the first parameter does actually show up as an advanced option when configuring a replication policy, and the “help” dialog for it mentions AMS replication; however, in several KB articles about AMS replication, they make a point of saying that one has to manually add the second parameter “to allocate additional Data Domain streams” for the replication.

Can someone provide some additional clarification and insight? Thank you.

Related:

Share File Encryption and DFS Replication – HELP

I need a solution

Dear Sir/Madam,

We are using Symantec Encryption Desktop, and we need to configure DFS Replication to work with feature Share File Encryption (Symantect Encryption Desktop).

In detail:

  • Step 1: We have 2 servers: Server A in DC and Server B in DR, both servers are joined in the same domain.
  • Step 2: On Server A we created a shared folder names SHARE with the owner right is granted to ENCRYPT (a domain account)
  • Step 3: We configured DFS Replication to replicate folder SHARE from Server A to server B.
    • All existing content in SHARE from Server A is replicated to Server B.
    • Whenever a new content created in SHARE on Server A, it is replicated automatically to Server B. (DFS work perfectly at this moment)
  • Step 4: We added folder SHARE for encryption to SED with user access is ENCRYPT (above domain account).
    • Then add a new content to SHARE on Server A, that new content WON’T be replicated to Server B anymore. (this is the problem)
  • Step 5: Remove folder SHARE from SED
    • Whenever a new content created in SHARE on Server A, it is replicated automatically to Server B. (DFS work perfectly again)

 I search around and notice that people are successful but not us.

Please give us some advices, thank you.

0

Related:

Isilon and Cloudera Backup and Disaster Recovery Integration – Hive Metastore and Data Replication

In Isilon and Cloudera Backup and Disaster Recovery Integration we reviewed Cloudera BDR integration for HDFS replication between a DAS cluster and an Isilon Cluster. In this post we will close the loop on BDR replication and review how to setup and integrate Hive replication

Assumptions:

CDH 5.8 and greater

UID/GID parity – through local accounts or LDAP, parity in uid and gid is important to maintain consistent access across storage

DAS Cloudera cluster setup complete

Isilon Cloudera cluster setup complete

DNS Name resolution fully functional – all host, forward and reverse

Both the source and destination clusters must have a Cloudera Enterprise license

Note the following when using replication jobs for clusters with Isilon:

• hdfs user is mapped to root on Isilon, If you specify alternate users with the Run As option when creating replication schedules, those users must also be superusers.

• Always select the ‘Skip Checksum Checks’ property when creating replication schedules.

• Kerberos authentication is fully supported from CDH 5.8 and higher, the account used to replicate data will need a principal and keytab to enable authentication against the target, see the Cloudera documentation for additional information on configuring this.

• Data replication can fail if the source data is modified during replication, it is therefore recommended to leverage snapshots as the source of data replication. If enabled replication can automatically make use of snapshots to prevent this issue. For more details see the following Cloudera documentation Using Snapshots with Replication

• Source clusters that use Isilon storage do not support HDFS snapshots. Since snapshots are used to ensure data consistency during replications in scenarios where the source files are being modified. Therefore, when replicating from an Isilon cluster source, it is recommended that you do not replicate Hive tables or HDFS files that could be modified before the replication completes without taking additional steps to ensure data replication succeeds effectively. Additional options would be to leverage SyncIQ to replicate data between Isilon clusters or using Isilon native snapshots in conjunction with metastore replication.

In our example we have loaded a sample set of data for use by Impala on our DAS Cloudera cluster, since Impala shares the Hive metastore database we can use BDR Hive replication to replicate this Impala database and the HDFS data to our Isilon Cloudera cluster. This illustrates that both Hive and Impala based databases and the HDFS based tables can be replicated with BDR.

1. In Hue, we see the tpcds_parquet database in the impala/hive metastore

2.png

2. The tpcds__parquet table definition and information can be seen here in Hue

3.png

3. The data for the tables is seen here in the /user/hive/warehouse

4a.png

4. Run a sample Impala query to validate the data on the DAS cluster8.png

5. On the Isilon cluster, the tpcds_parquet database, tables and HDFS data do not exist

11.png

12.png

6. Since we have already created a replication Peer in blog post 1 we can move straight on to setting up Hive/Impala replication using the Cloudera Backup tools

a.png

7. Select the DAS cluster as source; a replication schedule and which databases to replicate can be defined here. Also the Run As Username; any user will need superuser permissions and kerberos enablement if the clusters use kerberos.

b.png

Again, make sure to always check “Skip Checksum Checks” as the target is Isilon.

You also have the option to override the location of the exported metadata and location of the HDFS data is replicated to, for more details see: Hive/Impala Replication

c.png

If the source HDFS data is not enabled for snapshots, you’ll see the following information. It is highly recommended to use snapshots with Hive/Impala replication. To configure this, make the source HDFS data default location – /user/hive/warehouse snapshottable. BDR will now automatically make use of this feature when replicating data.

d.png

We have enabled snapshots on the default location for data: /user/hive/warehouse

8. Having defined the schedule, execute it

e.png

9. The replication then executes copying the metadata & data: we see it copy the database, tables and HDFS data

f.png

10. We can now see the tpcds__parquet database in the metastore, the BDR job take care of location specific URI and paths relating to the metadata and data now being on a different Hadoop cluster, this is the critical piece of Hive/Impala replication and why using BDR is so useful.



g.png

11. Running a simple SQL query against the customer table on both clusters validates the database, table and HDFS data replication was successful.

On DAS cluster

h.png

On Isilon Cluster

i.png

Hive Replication and Incremental Replication

1. Drop a Hive/Impala Table on the Isilon cluster

a1.png

a2.png

2.Execute replication, Incremental Replication only copies the data for the missing table

a3.png

3. Update Hue’s view of the metastore data

a4.png

The table is now present and can be queried

a6.png

Having now replicated the Hive/Impala metastore data and underlying table data on HDFS to the Isilon cluster, we can again leverage exciting native Isilon features to protect this data further; Snapshots, SyncIQ, NDMP backup etc..This short demonstration illustrate how Cloudera BDR can be used to backup and replicate HDFS data between Hadoop DAS clusters and Isilon integrated Hadoop clusters easily.

Using Hadoop with Isilon – Isilon Info Hub

russ_stevenson

Isilon

Related:

Isilon and Cloudera Backup and Disaster Recovery Integration

Backing up Hadoop data can be challenging, the size and scope of the data set can present challenges. This blog will illustrate how the Cloudera Manager Backup and Recovery tools can be used with an Isilon cluster to backup native HDFS data and Hive/Impala data and metadata. Once the data has landed on Isilon, additional features native to OneFS can be used to backup and protect the data further. See the Isilon Hadoop Enterprise Features whitepaper for additional details.

Additional information on Cloudera Backup and Disaster Recovery can be found here on Cloudera’s site: BDR introduction

In our scenario, we have a DAS cluster and an Isilon integrated Hadoop cluster and we will backup from our active DAS cluster to an archive Isilon cluster.

0.jpg

It is possible to backup the data the other way also, DAS < — Isilon. If this methodology is used additional considerations will need to be addressed around active data. Since Cloudera BDR utilizes distcp, active data can present challenges that must be accounted for in the design if the replication jobs.

Assumptions:

CDH 5.8 and greater

UID/GID parity – through local accounts or LDAP, parity in uid and gid is important to maintain consistent access across storage

DAS Cloudera cluster setup complete

Isilon Cloudera cluster setup complete

DNS Name resolution fully functional – all host, forward and reverse

Both the source and destination clusters must have a Cloudera Enterprise license

Note the following when using replication jobs for clusters with Isilon:

• hdfs user is mapped to root on Isilon, If you specify alternate users with the Run As option when creating replication schedules, those users must also be superusers.

• Always Select the ‘Skip Checksum Checks’ property when creating replication schedules.

• Kerberos authentication is fully supported from CDH 5.8 and higher, the account used to replicate data will need a principal and keytab to enable authentication against the target, see the Cloudera documentation for additional information on configuring this.

• Data replication can fail if the source data is modified during replication, it is therefore recommended to leverage snapshots as the source of data replication. If enabled replication can automatically make use of snapshots to prevent this issue. For more details see the following Cloudera documentation Using Snapshots with Replication

• Source clusters that use Isilon storage do not support HDFS snapshots. Since snapshots are used to ensure data consistency during replications in scenarios where the source files are being modified. Therefore, when replicating from an Isilon cluster source, it is recommended that you do not replicate Hive tables or HDFS files that could be modified before the replication completes without taking additional steps to ensure data replication succeeds effectively. Additional options would be to leverage SyncIQ to replicate data between Isilon clusters or using Isilon native snapshots in conjunction with metastore replication.

In our example here /user/test1; the source is native HDFS so we can enable snapshots on the directory to be replicated, Cloudera can then automatically make use of the ‘directory enabled for snapshots feature’ and use a snapshot as the source of replication.

Review the directory with the HDFS file browser in Cloudera Manager

21.png

Enable Snapshots either via the HDFS file browser or via the cli.

22.png

The directory /user/test1 is now snapshot enabled

212.png

In our example, we use a local user to generate some test data, a corresponding user on Isilon exists with the same uid and gid membership.(this could be an LDAP user also)

$ su – test1

$ cd /opt/cloudera/parcels/CDH/jars

$ yarn jar /hadoop-mapreduce-examples-2.6.0-cdh5.11.1.jar teragen 1000000 /user/test1/gen1

$ yarn jar /hadoop-mapreduce-examples-2.6.0-cdh5.11.1.jar terasort /user/test1/gen1 /user/test1/sort1

$ yarn jar /hadoop-mapreduce-examples-2.6.0-cdh5.11.1.jar teravalidate /user/test1/sort1 /user/test1/validate1

Now lets setup replication of this data from the DAS cluster to Isilon:

1. Configure a Replication Peer on the Source (Isilon Cluster), Select Peers from the backup Tab on the Isilon Cloudera Manager

1.png

2. Add a Peer

2.png

3. Name the Peer, in this example we use ‘DAS’ to make it easy, add the peer URL and the credentials to logon to the Target(DAS) Cloudera Manager

3.png

4. The Peer is validated as connected

4.png

5. Now, lets create a HDFS Replication Schedule from the Backup menu

5.png

6. From the drop select the Source; the ‘DAS’ cluster, the source path, destination ‘Isilon’ cluster and the destination path to replicate to:

A schedule can be set as needed; we select daily at 00:00AM PDT

We run this job as hdfs, since we wish to replicate the source Permissions the Run As User must have superuser privilege on the target cluster; if kerberos is in use additional steps need to be completed to enable the run as user to authenticate successfully against the target cluster.

6.png

Select the Advanced Tab

Select ‘Skip Checksum Checks’ — this must be done, otherwise replication will fail

7.png

Additional setting can be used that are specific to your environment and your requirements

7. The replication policy is now available

8.png

8. Before executing a data copy, we can execute a dry run to validate and evaluate the replication policy.

9.png

a.png

9. On execution of a successful dry run, the job can be run manually or wait for the scheduled job to run to copy data

66.png

Review the job on completion, the details of the distcp and options can be seen along with additional other information regarding the job

b.png

10. Compare the Source and Target directories; we see the data has been replicated maintaining permissions.

Source DAS cluster – /user/test1

cc.png

Target Isilon cluster – /DAS/user/test1

dd.png

Using HDFS replication is incremental aware.

1. Add new data to DAS – /user/test1 – gen2, sort2,validate2, tpcds

Reviewing the Source DAS cluster data – /user/test1

e.png

2. execute a replication and review the results, only the new data was copied as expected

hh.png

As can be seen using HDFS replication is pretty straightforward and can be used to maintain a well structured and scheduled backup methodology for large HDFS data sets. Now, since the data is resident on Isilon additional backup methodologies can be leveraged; SyncIQ copies to other Isilon clusters, Isilon Snapshots, NDMP backups and tiering.

In the next post we will look at how Hive replication is enabled for integration between two Cloudera clusters.

Using Hadoop with Isilon – Isilon Info Hub

russ_stevenson

Isilon

Related:

Event ID 5706 — Netlogon Share Creation

Event ID 5706 — Netlogon Share Creation

Updated: November 25, 2009

Applies To: Windows Server 2008 R2

The Netlogon service creates the Netlogon and SYSVOL shares during the domain controller promotion process. The Netlogon share stores the logon script and possibly other files. The SYSVOL share is the shared directory on domain controllers that contains Group Policy and logon script information.

Event Details

Product: Windows Operating System
ID: 5706
Source: NetLogon
Version: 6.0
Symbolic Name: NELOG_NetlogonFailedToCreateShare
Message: The Netlogon service could not create server share %1. The following error occurred: %2

Resolve
Ignore error after upgrade

If Event ID 5076 appears immediately after an upgrade, there may not be a problem. Use the information under “Verify” to ensure that the Netlogon and SYSVOL shares are available.

If this situation is not a result of an upgrade, see the following articles in the Microsoft Knowledge Base for additional troubleshooting information:

Verify

Confirm that the NetLogon service was able to create the Netlogon and SYSVOL shares. To view the shares that available on the domain controller, run the command net share. NETLOGON and SYSVOL should appear in the list of shares.

Related Management Information

Netlogon Share Creation

Active Directory

Related: