RecoverPoint: XtremIO: Unable to created new snapshots, SRM “RE” failover fails due to unavailable bookmark

Article Number: 480610 Article Version: 3 Article Type: Break Fix



RecoverPoint,RecoverPoint SE,RecoverPoint Adaptor for VMware Site Recovery Manager,XtremIO

When XtremIO replica copy configured with ‘Number of Desired Snapshots: 2’, customer is unable to create new snapshots while in Test a Copy mode (or Image Access mode) and SRM fail-over operation also fails.


Symptoms found in the logs:

In replication logs on replica copy site:

2016/03/10 16:45:41.866 – #1 – 8747/8560 – XioSnapConsolidation::findBestSnapToDelete: didn’t find user snap to delete …

2016/03/10 16:45:41.866 – #2 – 8747/8560 – XioDistributorHandler::checkSnapDelete: going to delete the latest snapshot!

Affected versions:
All RecoverPoint versions since XtremIO was introduced (4.1.1)

When maximal number of snapshots is 2 the following will happen:

– After the second snapshot is created there are 2 snapshots in snapshot list.

– When the system try to create the 3rd snapshot – it can’t, so snapshot creation is delayed and internal variable indicating number of reserved snapshots is set to 1.

– The system then delete one snapshot and only then will try to create the new one.

– If the pipe is closed before the system handle the delayed request, number of reserved snapshots will remain 1.

From that moment on the system can only hold 1 snapshot at any given moment. So every time a new snapshot is created the previous is deleted.

Workaround:

Increase ‘Number of Desired Snapshots’ for the relevant Consistency Group using RecoverPoint CLI (via config_copy_policy command)

Permanent Fix:
RecoverPoint version 4.4 SP1

Related:

  • No Related Posts

RecoverPoint: RPA replacement failing with “GetArrayCredentialsOfRepositoryCommand FAILED: Array of repository does not exists in settings”

Article Number: 492948 Article Version: 3 Article Type: Break Fix



RecoverPoint,RecoverPoint CL 4.1 SP3,RecoverPoint EX 4.1 SP3,RecoverPoint SE 4.1 SP3

Deployment Manager fails to replace an RPA on step 2 with following error:

ERROR – GetArrayCredentialsOfRepositoryCommand FAILED: Array of repository does not exists in settings

The issue is that at 4.1.3 any volume starting with “RP_” is filtered from the view.

RPA replacement on 4.1.3 code, etc.

On the cluster that is failing in RecoverPoint GUI, go to VNX storage and find the VNX IP address.

Login to VNX Unisphere with credentials (defaults are sysadmin/syadmin unless changed, if so, then client has to log in).

Once logged in, go to Storage -> Storage Configuration -> Storage Pools.

On bottom under Details -> LUNs, type in RP_ to filter out the LUNs that start with this prefix.

Find the repository volume, click on Properties and rename so that it doesn’t have RP_.

Save your changes, then go to RP CLI as admin user and run rescan_san choosing full/all options.

How to rename VNX repository volume

CE should then retry replacement wizard from DM and it should go through.

Related:

  • No Related Posts

RecoverPoint: Deployment Manager fails with “Unable to distribute ISO file. RPA is already receive a file from another Deployment Manager.”

Article Number: 490925 Article Version: 4 Article Type: Break Fix



RecoverPoint

DM is unable to distribute the ISO file to the RPAs.

RPA thinks it is already receiving a file from another instance of DM even though there is only one instance running.

From DM logs:

2016-10-12 10:34:58,826 [pool-1-thread-7] (BaseInstallationActionWithStatusLogicCommand.java:85) DEBUG – Failing the current status object: StepStatus- _title=Downloading ISO to RPA number 2 at cluster B1, _status=RUNNING, message=null, _progress=0.22514111910289922, DownloadISOStatus [_rpaSettings=RPA [ip=172.23.1.32, isCredentails=Credentials [user=boxmgmt, password=*****], fapiCredentials=null, alternativeISCredentails=null, number=1, siteName=B1, version=null, boxWWNs=null, rpaState=null, generation=UNKNOWN]], failureMessage=Unable to distribute ISO file. RPA is already receiving a file from another Deployment Manager.

2016-10-12 10:34:58,826 [pool-1-thread-7] (BaseDMCommand.java:235) DEBUG – Command DistributeISOToRPACommand FINISHED with returnValue=[ReturnValue [_isSuccess=true, _message=null, _data=null, _throwable=null, _failureReason=UNKNOWN]]

2016-10-12 10:34:58,827 [pool-1-thread-7] (BaseDMCommand.java:205) DEBUG – Command:DistributeISOToRPACommand Finished

Upgrading from one version of RP to another, in this case, it was from 4.0.2.1 to 4.1.3.

Workaround:

Reboot the RPA that DM fails to apply the ISO to.

Login as boxmgmt and reboot the RPA.

Run in a cmd prompt the following without quotes, “ping RPA IP address -t” to see when it comes up and then retry DM.


If a reboot does not work, this could be because of a file transfer issue. The user should change the DM tweak SHOULD_REMOVE_EXISTING_ISO_FILES to true and restart DM.

This is done by going to conf folder and then editing dm.properties and installation.properties and changing SHOULD_REMOVE_EXISTING_ISO_FILES tweak to true and saving the files and restarting DM.


Resolution/Fixed at:

EMC engineering is currently investigating this issue. A permanent fix is still in progress. Contact the EMC Customer Support Center or your service representative for assistance and reference this solution ID.

Related:

  • No Related Posts

RecoverPoint: Error “the requested action cannot be performed on any consistency group”

Article Number: 488249 Article Version: 4 Article Type: Break Fix



RecoverPoint

Unable to run image access.

Error “the requested action cannot be performed on any consistency group”.

RecoverPoint License was expired

N/A



Uninstall the expire license and install the permanent license.

Contact EMC Licensing Service to obtain a permanent license. See EMC Knowledgebase article 325653 RecoverPoint: How to activate, reactivate, or obtain a RecoverPoint license

Related:

  • No Related Posts

RecoverPoint: Can’t use VMAX3 devices as journals.

Article Number: 486577 Article Version: 4 Article Type: Break Fix



RecoverPoint,RecoverPoint CL,RecoverPoint EX

Isuue: Cannot use VMAX3 devices as RecoverPoint journals.

In our code we check for tagging for all Symmetrix (when VMAX2 was implemented, there was no VMAX3…). In case of Symmetrix volume, if the device is not tagged, it is filtered out from the SAN view, so users can’t use it even as journal.

Splitter type(s): All Recoverpoint Splitters (Doesn’t have to be Symmetrix)

Symptoms found in the logs: N/A

Root cause: In RP code, we don’t differentiate between VMAX2 & VMAX3 devices. So we check for the tag that should be enabled by the user for all Symmetrix devices, even though user cannot tag VMAX 3 devices. Since these devices are not tagged, we filter them out from the SAN view so users can’t use them

Affected versions: Day 1 bug.

None

Workaround: The workaround is to change t_SanDiscoveryIgnoreSymmTagging to false. However, be aware that when ignoring tagging, we ignore tagging also for Symmetrix VMAX2 devices, so users must carefully pick their volumes and pay attention not to protect irrelevant devices. All devices will appear in the RP volumes list.

Resolution:

EMC engineering is currently investigating this issue. A permanent fix is still in progress. Contact the EMC Customer Support Center or your service representative for assistance and reference this solution ID.

Related:

  • No Related Posts

Re: EMC NetWorker 9.1: Can’t get exchange 2010 backups to work

Hi guys,

I’ve been spending a few hours reading docs and trying to get Exchange backups going without success.

From the Exchange server, I correctly configured a NMM Exchange Admin, I can validate an existing admin successfully.

Also when I run nsrnmmsv -P, I get successful message (no errors) -> Successfully listed supported save sets.

From the NetWorker console, I configured a client with Microsoft Exchange Server application enabled, I left most settings default.

From the Objects window, I selected all available:

APPLICATIONS:Microsoft Exchange 2010

APPLICATIONS:Microsoft Exchange 2010Database-Name1



Backup Objects: APPLICATIONS:Microsoft Exchange 2010

backup command: nsrnmmsv

application information:

NSR_EXCH_CHECK=yes

NSR_ESE_CC_METHOD=DB_AND_LOGS

NSR_ESE_UTIL_SEQUENTIAL=False

NSR_ESE_UTIL_THROTTLE=False

Name: Exchange-Server.domain.local

Backup Type: Microsoft Exchange Server

Snapshot Management: false

Priority: 100

Parallelism: 6

Block based backup: true

Client direct: true

Parallel save streams: false

Data Domain interface: IP



Actually I can do a Filesystem backup successfully..

But when I try an Application: Exchange backup (to backup the Exchange database), I get the errors you see below.



I believe it has to do with my storage pool, I currently use local, adv_file media type..



After reading some posts on this forum, I can confirm that save -b POOL_NAME -a DIRECT_ACCESS=yes C:” is working well from the client computer.



Is it related to the fact that I didn’t configure a Remote user in the Device properties? If so, I don’t understand how to do it, please help.



Thank you!

Victor

——————————–

107092:nsrnmmsv: Version information for C:Program FilesEMC NetWorkernsrbinnsrnmmsv.exe: Original file name: nsrnmmsv.exe Version: 9.1.0 (ntx64) Comments: Supporting Microsoft Volume Shadow Copy Service

127159:nsrnmmsv: Created temp path C:Program FilesEMC NetWorkernsrtmpnmm2018-08-16_15-00-12_5564-6044.

nsrnmmsv: multithreaded save set parallelism==6

145369:nsrnmmsv: Initialization success — Exchange shell successfully initialized. Required for Exchange.

133296 1534446013 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 26 Starting backup operation. 0

145369 1534446014 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 89 Initialization success — Exchange shell successfully initialized. Required for Exchange. 0

132558 1534446016 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 75 Initialization process successful. Required for getting required log range. 0

115938 1534446016 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 71 The maximum number of Consistency Check process threads is set to ‘%d’. 1 1 1 4

129273 1534446016 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 41 Consistency Check Throttling is disabled. 0

114457 1534446017 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 89 ‘%s’: adding to the file list with the minimum range as = %d and the maximum range as %d. 3 0 52 APPLICATIONS:Microsoft Exchange 2010Database-Name1 1 3 109 1 3 110

98519 1534446018 2 5 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR warning 47 Unable to setup direct save with server %s: %s. 2 12 33 Backup-Server.domain.local 49 294 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 0 10 4269132226 11 13 DIRECT_ACCESS 0 3 Yes

90018 1534446018 5 5 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 55 Cannot open a %s session with NetWorker server ‘%s’: %s 3 20 8 nsrnmmsv 12 33 Backup-Server.domain.local 49 294 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 0 10 4269132226 11 13 DIRECT_ACCESS 0 3 Yes

0 1534446018 1 5 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR notice 50 %s%s%s: %-*s%s%s%s %*s %2.2ld:%2.2ld:%2.2ld %6s %s 15 0 0 0 0 20 8 nsrnmmsv 1 1 0 23 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 0 7 level= 0 4 full 0 1 , 1 1 0 0 4 0 KB 30 1 0 30 1 0 30 1 0 36 1 0 0 5 files

110812 1534446018 5 0 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 44 Backup of Saveset ‘%s’ failed with error %s. 2 51 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 24 240 Exchange-Server.domain.local:APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles: Unable to perform backup for save-set ID ‘4269132226’ with DIRECT_ACCESS=Yes, check device and pool configuration to enable Client direct.

98519 1534446018 2 5 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR warning 47 Unable to setup direct save with server %s: %s. 2 12 33 Backup-Server.domain.local 49 289 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 0 10 4252355010 11 13 DIRECT_ACCESS 0 3 Yes

90018 1534446018 5 5 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 55 Cannot open a %s session with NetWorker server ‘%s’: %s 3 20 8 nsrnmmsv 12 33 Backup-Server.domain.local 49 289 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 0 10 4252355010 11 13 DIRECT_ACCESS 0 3 Yes

0 1534446018 1 5 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR notice 50 %s%s%s: %-*s%s%s%s %*s %2.2ld:%2.2ld:%2.2ld %6s %s 15 0 0 0 0 20 8 nsrnmmsv 1 1 0 23 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 0 7 level= 0 4 full 0 1 , 1 1 0 0 4 0 KB 30 1 0 30 1 0 30 1 0 36 1 0 0 5 files

110812 1534446018 5 0 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 44 Backup of Saveset ‘%s’ failed with error %s. 2 51 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 24 235 Exchange-Server.domain.local:APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles: Unable to perform backup for save-set ID ‘4252355010’ with DIRECT_ACCESS=Yes, check device and pool configuration to enable Client direct.

140124 1534446020 0 0 0 6076 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 89 Performing the cleanup of ‘%s’ to avoid inconsistency in the indexes for child save sets. 1 51 52 APPLICATIONS:Microsoft Exchange 2010Database-Name1

139527 1534446027 3 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR error 51 The backup operation did not complete successfully. 0

102333 1534446027 3 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR error 21 Exiting with failure. 0

2018-08-16 3:00:27 PM Removing files under temp path C:Program FilesEMC NetWorkernsrtmpnmm2018-08-16_15-00-12_5564-6044.

Related:

  • No Related Posts

NVP-vProxy: one or more VMs in a policy fail to back up with the message: no matching devices for save of client `vCenter_Server_Name’; check storage nodes, devices or pool

Article Number: 502312 Article Version: 3 Article Type: Break Fix



NetWorker 9.1,NetWorker 9.2

The NetWorker VMware Protection integration is configured with the vProxy Appliance. One or more virtual machine backups are failing with the error noted below:

no matching devices for save of client `[VCENTER_SERVER]‘; check storage nodes, devices or pool

The vCenter Server has a NetWorker Client resource configured in the NetWorker Management Console (NMC) Protection tab. The vCenter Server NetWorker Client “Storage Nodes” attribute, under the “Globals (2 of 2)” tab, has the default blank settings.

The specific cause is unknown. Some of the virtual machines require the vCenter Server NetWorker Client “Storage Nodes” attribute to be configured with the appropriate Storage nodes.

To work around the backup failures, the vCenter Server NetWorker Client “Storage Nodes” attribute should be updated:

  1. Open the NetWorker Management Console.
  2. Select the Protection Tab.
  3. In the left hand tree view, select “Clients“.
  4. Select the vCenter server, right click, and select “Modify Client Properties“.
  5. Go to the “Globals 2 of 2” tab
  6. Add the individual storage node names with the target backup devices for the vProxy backups. Include nsrserverhost (default for NetWorker Server) as the last entry: (One server name per line)
For Example:
NW_SNode1.yourdomain.com

NW_SNode2.yourdomain.com

nsrserverhost

This modification will explicitly zone all the required devices, on the storage nodes that the vProxy backups use, as available to the vCenter server.

Related:

  • No Related Posts

NetWorker 9.0: When the retention policy set to greater than 21 years (making clretent go past the year 2038), the Policy fails with “mm/dd/yyyy hh:mm:ss Action backup traditional ‘backup’ has initialized as ‘backup action job’ with job id “

Article Number: 499678 Article Version: 3 Article Type: Break Fix



NetWorker 9.0

Retention is set to 50 years for the Action (second page, “Backup options”).

Client properties has a retention policy that is also set to 50 years.

The Policy failure is instantaneous (with in 8 seconds).

There are no other details in the Show messages.

After the upgrade from NetWorker 8.2.3 to NetWorker 9.0.1.8 the policies with these “50 year” settings are failing with

“mm/dd/yyyy hh:mm:ss Action backup traditional ‘backup’ has initialized as ‘backup action job’ with job id <Number>”

The NetWorker nsrjobd was failing to interpret the retention that is greater than the epoch of January 1, 2038

Reconfigured the Policy’s retention period to less than 2038.

Reconfigured any clients that possessed a retention policy defined to be greater than 2038.

Related:

  • No Related Posts

CBT + DDBoost + BBBackups not equal to “almost zero”

Hello,

Just wondering on the space used on a Data Domain when you do several backup of the same server in a short period of time.

Since a while (exactly from the 7 of June) we experiment an increase of the space used on our DD not in relation with data growing.

DD space used.png

I discover that on the date where we had a capacity increased we had several manual launch of some backup.

What is “bizarre” is the fact that the capacity used on the DD seems to increase accordingly to the number of backups… (which is “acceptable” in case of traditional file level backup)

Capacity report V2.png

If you look at the number of backup version we have for the server “Server one” (as an example) you can see that on the 22 of June we had 5 backup of this server (don’t ask me why 5…) and every backup has a Data Domain copy (which is confirmed by the column AE of the Excel report.

Server one.png

I was expecting that with all CBT + Dedup and whatsoever optimization we will have just few blocks added and not the full disk.

(We are using Networker 9.2.1.4)

Questions:

1) Is CBT an option to be setup on every VM (found an article on that). We are running vSphere 6.5 with vProxy.

2) Even if not activated, a Block Based Backup should do that for you ?

3) Will an Incremental backup not take only modified Blocks ?

3) And what is the role of DDBoost in that “party” ?

Any explanation or answers welcomed because EMC dosen’t want to give me a clear explanation how things are really working here…

Related:

  • No Related Posts

VNX Family Monitoring & Reporting: Unable to run “manage-modules command”. Receive “! ERROR: Error encountered on startup”

Article Number: 482035 Article Version: 4 Article Type: Break Fix



VNX Family Monitoring & Reporting

The following or similar error is shown when executing any manage-modules command (for example: service status, list modules, etc.):

! ERROR: Error encountered on startup... com.watch4net.apg.module.storage.DataAccessException:java.lang.ArrayIndexOutOfBoundsException: 3 

This error is seen if there is a corruption of Module-Manager data files, which is often a result of disk space being full.

IMPORTANT! The following workaround was tested only on Linux OS. There is a possibility that the workaround may not be able to fix the corrupted Module-Manager files. A clean install of VNX M&R software is needed if the database is corrupted as a result of disk space being full, and the workaround is unable to repair Module-Manager. Historical data can be imported if a backup is available for the database prior to the database crash event.

If storage is full

  1. Expand the partition if the disk space is full.
  2. Check the storage utilization on /tmp folder as well, and add disk space, if necessary.

Workaround

  1. Install the same version of VNX M&R on a lab/dev machine. Install the VNX M&R software on the same directory as the target server if its VNX M&R is installed on another directory (other than /opt/VNX). This is important as the copied files will not work if the installation directory is different than the target.
  2. Copy all folders, sub-folders, and files from <VNX M&R install path>/Tools/Module-Manager/<version no.>/data folder and upload it to a temporary directory on target server.
  3. Remove files from <VNX M&R install path>/Tools/Module-Manager/<version no.>/data on the target server.
  4. Copy new files to <VNX M&R install path>/Tools/Module-Manager/<version no.>/data.
  5. Change owner and group if other than VNX user/group:
chown -R vnx:vnx <VNX M&R install path>/Tools/Module-Manager/<version no.>/data
  1. Test manage-modules command (for example, <VNX M&R install path>/bin/./manage-modules.sh service status all)

Proceed with repairing database if database files are corrupted.

Related:

  • No Related Posts