Existing we have one email server that are send/receive from messaging gateway.
We want to add exchange email server to run together with the existing email server.
Is there any guideline on this?
Checking on above
1) Any particular tips/tric to implement it? Will it distrupt any email flow?
2) Since the add-on create a folder at Outlook, would it ‘eat’ or own up temporary space from Exchange server?
Should customer make any new overall dis space sizing?
|Article Number: 499650||Article Version: 3||Article Type: Break Fix|
Avamar Plug-in for Exchange VSS
We observe that Exchange DAG backup change-ratio presents a huge change ratio if we perform a failover of a database in a DAG environment “moving one DB from one exchange node to another”
We see the % value spiking up from the usual daily value of ~0.5% going up to ~20%/~30% and depending on which DB is being failed-over we even see ~60% of change ratio.
The DB is exactly the same before and after the failover, the only difference is that after the failover it is being backed up from a different node of the DAG cluster.
This behavior gives the impression that the backup data is being sent twice to the avamar storage, the expectation is that it should only send the usual ~0.5 of data which reflects average the daily change ratio for all the DBs under this DAG cluster environment.
The result of this inconsistency is that the avamar capacity is being unnecessarily over-utilized.
This is an expected behavior.
The logical structure of the Exchange database is the same but Avamar uses the *physical* layout of the database when performing the de-duplication.
In DAG environments, Exchange performs log shipping from the active copy of the database to the passive copy and replays the transactions against this passive copy.
Each Exchange server is responsible for applying the transaction logs against its copy of the database and there is no guarantee that it will use the same physical database layout as the other servers in the DAG.
Here is an example to better explain this concept:
Since Avamar only looks at the shape of the pieces, it has no way to know that the data is the same logical data, therefore the data is being sent to the server again however.
The DB failover was performed in the DAG cluster.
As per above explanation we learned that this is the normal behavior of the product.
I’ve been spending a few hours reading docs and trying to get Exchange backups going without success.
From the Exchange server, I correctly configured a NMM Exchange Admin, I can validate an existing admin successfully.
Also when I run nsrnmmsv -P, I get successful message (no errors) -> Successfully listed supported save sets.
From the NetWorker console, I configured a client with Microsoft Exchange Server application enabled, I left most settings default.
From the Objects window, I selected all available:
APPLICATIONS:Microsoft Exchange 2010
APPLICATIONS:Microsoft Exchange 2010Database-Name1
Backup Objects: APPLICATIONS:Microsoft Exchange 2010
backup command: nsrnmmsv
Backup Type: Microsoft Exchange Server
Snapshot Management: false
Block based backup: true
Client direct: true
Parallel save streams: false
Data Domain interface: IP
Actually I can do a Filesystem backup successfully..
But when I try an Application: Exchange backup (to backup the Exchange database), I get the errors you see below.
I believe it has to do with my storage pool, I currently use local, adv_file media type..
After reading some posts on this forum, I can confirm that save -b POOL_NAME -a DIRECT_ACCESS=yes C:” is working well from the client computer.
Is it related to the fact that I didn’t configure a Remote user in the Device properties? If so, I don’t understand how to do it, please help.
107092:nsrnmmsv: Version information for C:Program FilesEMC NetWorkernsrbinnsrnmmsv.exe: Original file name: nsrnmmsv.exe Version: 9.1.0 (ntx64) Comments: Supporting Microsoft Volume Shadow Copy Service
127159:nsrnmmsv: Created temp path C:Program FilesEMC NetWorkernsrtmpnmm2018-08-16_15-00-12_5564-6044.
nsrnmmsv: multithreaded save set parallelism==6
145369:nsrnmmsv: Initialization success — Exchange shell successfully initialized. Required for Exchange.
133296 1534446013 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 26 Starting backup operation. 0
145369 1534446014 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 89 Initialization success — Exchange shell successfully initialized. Required for Exchange. 0
132558 1534446016 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 75 Initialization process successful. Required for getting required log range. 0
115938 1534446016 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 71 The maximum number of Consistency Check process threads is set to ‘%d’. 1 1 1 4
129273 1534446016 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 41 Consistency Check Throttling is disabled. 0
114457 1534446017 0 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 89 ‘%s’: adding to the file list with the minimum range as = %d and the maximum range as %d. 3 0 52 APPLICATIONS:Microsoft Exchange 2010Database-Name1 1 3 109 1 3 110
98519 1534446018 2 5 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR warning 47 Unable to setup direct save with server %s: %s. 2 12 33 Backup-Server.domain.local 49 294 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 0 10 4269132226 11 13 DIRECT_ACCESS 0 3 Yes
90018 1534446018 5 5 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 55 Cannot open a %s session with NetWorker server ‘%s’: %s 3 20 8 nsrnmmsv 12 33 Backup-Server.domain.local 49 294 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 0 10 4269132226 11 13 DIRECT_ACCESS 0 3 Yes
0 1534446018 1 5 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR notice 50 %s%s%s: %-*s%s%s%s %*s %2.2ld:%2.2ld:%2.2ld %6s %s 15 0 0 0 0 20 8 nsrnmmsv 1 1 0 23 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 0 7 level= 0 4 full 0 1 , 1 1 0 0 4 0 KB 30 1 0 30 1 0 30 1 0 36 1 0 0 5 files
110812 1534446018 5 0 0 6116 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 44 Backup of Saveset ‘%s’ failed with error %s. 2 51 66 APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles 24 240 Exchange-Server.domain.local:APPLICATIONS:Microsoft Exchange 2010Database-Name1DatabaseFiles: Unable to perform backup for save-set ID ‘4269132226’ with DIRECT_ACCESS=Yes, check device and pool configuration to enable Client direct.
98519 1534446018 2 5 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR warning 47 Unable to setup direct save with server %s: %s. 2 12 33 Backup-Server.domain.local 49 289 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 0 10 4252355010 11 13 DIRECT_ACCESS 0 3 Yes
90018 1534446018 5 5 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 55 Cannot open a %s session with NetWorker server ‘%s’: %s 3 20 8 nsrnmmsv 12 33 Backup-Server.domain.local 49 289 86929 125 %s:%s: Unable to perform backup for save-set ID ‘%s’ with %s=%s, check device and pool configuration to enable Client direct. 5 12 33 Exchange-Server.domain.local 51 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 0 10 4252355010 11 13 DIRECT_ACCESS 0 3 Yes
0 1534446018 1 5 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR notice 50 %s%s%s: %-*s%s%s%s %*s %2.2ld:%2.2ld:%2.2ld %6s %s 15 0 0 0 0 20 8 nsrnmmsv 1 1 0 23 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 0 7 level= 0 4 full 0 1 , 1 1 0 0 4 0 KB 30 1 0 30 1 0 30 1 0 36 1 0 0 5 files
110812 1534446018 5 0 0 5828 5564 0 Exchange-Server.domain.local nsrnmmsv NSR critical 44 Backup of Saveset ‘%s’ failed with error %s. 2 51 61 APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles 24 235 Exchange-Server.domain.local:APPLICATIONS:Microsoft Exchange 2010Database-Name1LogFiles: Unable to perform backup for save-set ID ‘4252355010’ with DIRECT_ACCESS=Yes, check device and pool configuration to enable Client direct.
140124 1534446020 0 0 0 6076 5564 0 Exchange-Server.domain.local nsrnmmsv NSR info 89 Performing the cleanup of ‘%s’ to avoid inconsistency in the indexes for child save sets. 1 51 52 APPLICATIONS:Microsoft Exchange 2010Database-Name1
139527 1534446027 3 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR error 51 The backup operation did not complete successfully. 0
102333 1534446027 3 0 0 6044 5564 0 Exchange-Server.domain.local nsrnmmsv NSR error 21 Exiting with failure. 0
2018-08-16 3:00:27 PM Removing files under temp path C:Program FilesEMC NetWorkernsrtmpnmm2018-08-16_15-00-12_5564-6044.
This is an environment with 20,000+ mailboxes and OWA is used to access the mailboxes (Outlook is almost not used). Now, this environment consists on 8 Exchange 2016 servers with mailbox and CAS roles ach of them and of course SourceOne extensions for OWA version 7.24.4058 are installed. on all Exchange servers and had been working good, or… That´s what it seems.
Issue: Exchange servers are experiencing high user processor at peak hours because the OWA App pool takes 90+% of processor, when this happens in one server, the issue replicates to other Exchange servers as well and the only workaround is restarting the affected Exchange servers.
Microsoft has been working on this for two weeks and now they wonder if ES1 extensions may be impacting, since ES1 and Exchange App pools are related (I ask myself if ES1 may be impacting too).
I know this intensive OWA usage is unusual but, Has anyone experienced this before? Has anybody idea of what to check from the SourceOne perspective on the Exchange server when this happens?
I´d really appreciate any tips or feedback.
Thanks in advance
We cannot send emails to messagelabs customers, our exchange queue shows a 421 4.4.2 Connection dropped due to ConnectionReset error. I confirmed our sending IP 188.8.131.52 does not have a negative reputation and I can telnet to cluster1.us.messagelabs.com and cluster4.us.messagelabs.com from the exchange server.
How can this be cleared up? Our client cannot communicate with their bank and one of their main customers….
This article will cover restore options for Exchange in NMM 9. The demo in the article is with NMM 184.108.40.206 and Exchange 2013 DAG. NMM 9 offers the choice of using ItemPoint to perform mailbox/item level recovery. ItemPoint is the preferred method of GLR in NMM 9. However traditional GLR (Granular Level Restore) using NMM GUI is available for Exchange 2013 and Exchange 2010. For Exchange 2016 ItemPoint is the only method of GLR .
If you choose to use ItemPoint for mailbox/mail item recovery, then traditional GLR from NMM GUI is not available. So you can either use ItemPoint or traditional GLR restore but not both on a given host. This choice is made during NMM 9 install. Additionally if there is a requirement to restore backups done with NMM 8, then during install of NMM 9, select the option ‘Restore of NMM 8.2.x and earlier backups(VSS workflow)’
Below is the screen shot of the NMM 9 install process that shows the selection to restore backups from previous version and selecting ‘Exchange Granular Recovery’ (ItemPoint) as the preferred method of GLR.
Note, when you select to ‘Restore of NMM 8.2.x and earlier backups(VSS workflow)’, the install process installs the binaries for NMM 8.2.4, that will be used for the restore. The binaries are stored in the default folder:
C:Program FilesEMC NetWorkernsrrpvnmm
RPVNMM stands for ‘Restore Previous Version of NMM’
There is also a selection added in the ‘EMC Networker’ Program Group, under ‘NetWorker Tools’ for the Restore GUI that will read older backups done with NMM 8.2.x. The program name is ‘Restore previous NMM release backups’
In addition, the install also contains the binary ‘nsrsnap_vss_ssrecover’ that allows for flat file restore of the NMM 8 backups, if required.
On a Exchange host where ItemPoint was chosen for GLR, there are 2 restore choices available. ‘Database Recovery’ and ‘Granular Level Recover’. Note there is no ‘RDB data recover’. If you are used to seeing this option with previous versions of NMM, then note that ItemPoint restore is not handled through RDB, hence the choice of ‘RDB data recover’ is not available.
Database Recovery provides the following options:
a. Overwrite the Database, if there is a need to recover the database to itself (overwriting it). Select the desired database and then click ‘Recover’. You will see the message window below advising to set the property ‘This database can be overwritten by a Restore’ of the database.
This can be done either using the PowerShell command as mentioned in the message window or using ECP / EAC GUI as below:
****Note this is a destructive action. Make sure this is the correct choice before you proceed. NMM does not let you set the ‘This database can be overwritten by a restore’ property from the NMM GUI, to ensure there is no accidental overwrites.
Once the property ‘This database can be overwritten by a restore’ is set, then NMM GUI allows you to proceed with restore. Click ‘recover…’ and then ‘Recover options…’ Under ‘Exchange’ tab there are certain choices to make that will determine how the ‘recover’ is done within Exchange
Below is a brief explanation of these choices:
‘Include Existing logs (Roll-forward Recovery)’. This option is useful if the database and logs were on separate volumes and the volume containing logs is still available, or even if both DB and logs are on the same volume and only the ‘edb’ file is corrupt and the logs are good, then you can do the restore of the backup and then perform a roll forward recovery using the logs on the disk. This will bring the database to the most recent state with minimum or no data loss.
‘Include only logs from this restore (Point-in-time recovery)’. Select this option when point in time restore is required, i.e the database will be recovered to the time of the last backup.
‘Put database online after restore’. By default the restore process will replay the logs and put the database online after restore. If this is not required, then click on this option again to deselect it and select ‘Do not replay the transaction logs’. If ‘Do not replay the transaction logs’ is selected then the logs are restored, but they will need to be manually replayed using ‘eseutil’
‘Deleted Database Target’, This is used if a flat file restore of database is required. This option bypasses VSS method of restore and simply restored the ‘edb’ and ‘logs’ as files to the target directory. Further processing is required to mount the database.
For Database Restore of a database that’s replicated in a DAG configuration, you first have to suspend the replication, otherwise the following error is seen in the Monitor tab:
The client name used NMMHOST2. The Exchange Server version used is Exchange 2013.
145369:nsrnmmrc: Initialization success — Exchange shell successfully initialized. Required for Exchange.
MailboxStore [DB01] is in replicated state, please suspend the replication on all DAG nodes and perform restore after that.
Also note that this restore can only be performed on the node that holds the active copy of the database.
Before trying GLR restore with ItemPoint, some preparatory steps have to be taken to meet all pre-requisites:
Microsoft does not recommend (or support) Outlook Installation on Exchange Server. Prior to NMM 9.2, ItemPoint could only be used on the Exchange server, which means outlook had to be installed on the Exchange server. However NMM 9.2 and greater allows ItemPoint GLR from a non-Exchange server (Proxy).
2. Ensure the user performing restore has send-as, receive-as permissions on the mailbox server to allow for browsing and restoring the mailboxes. Also make this user a member of the ‘Exchange Organization Management’ security group.
Also ItemPoint needs additional permission assignments to the user performing restore to allow for browsing and recovering mail items from any mailbox
Get-mailboxdatabase -identity db01 | add-adpermission -user nmm_svc -accessrights genericall
Get-mailbox | add-mailboxpermission -user nmm_svc -AccessRights FullAccess -InheritanceType All
For further information on this topic refer to ItemPoint documentation.
To perform ItemPoint GLR from a non-Exchange server ensure the following pre-requisites are met:
5. Grant remote access permissions to the user performing restore from this host. There are 2 ways to do this. Either add this user to a user group that has remote access privilege or add this user to the ‘remote access’ attribute of the DAG client resource.
Method 1: Adding the user performing the restore to the ‘Remote Access’ attribute of the client resource.
Method 2: Add the user to a user group that has ‘remote access’ privilege.
Once the above pre-requisites have been met, you are ready to perform GLR using itempoint.
To start the recovery, from NMM restore host, launch the NMM GUI as ‘administrator’. Then select the client from the drop down and then select ‘Exchange Recover Session’ => ‘Granular Level Recover’
Note this restore is being performed from a host that does not have Exchange server installed and as such NMM provides only the GLR option for restore. Since Exchange is not installed, Database Recovery or Restore to RDB cannot be performed.
From the browse window, select the desired client, browse time and database for restore and right click:
From the right click menu either choose ‘mount backup’ or ‘mount backup and run ItemPoint’. If you select ‘Mount backup’ then NMM will mount the backup and then you have to manually launch ItemPoint to restore from the mount points. ‘Mount backup and run ItemPoint’ is preferable as NMM GUI mounts the backup and then starts ItemPoint and auto fills the path for edb and log files:
Click ‘Finish’ and ItemPoint starts processing the log files and edb file:
Once the log and edb file is processed, ItemPoint GUI shows the content of the source database that was mounted. The next step is to open a target. This can be an Exchange server or a PST file. Here we open a target Exchange server, as we want to restore to a mailbox to the target server.
When you click ‘Open Target Exchange server’, you get the following choices. In ‘Select Target’ either choose to connect to a single mailbox or ‘All Mailboxes’. You would choose ‘All Mailboxes’ if you want to restore data to different mailboxes. Click ‘Next’
ItemPoint shows the number of mailboxes in the Target. Review and click ‘Close’
This output shows copy of mailbox ‘nmm_svc’ to the mailbox ‘blee’
If you need to restore a mailbox or mailbox items to a PST file, right click the mailbox / mail items and select ‘Export…’
(Only for Exchange server 2010 and Exchange Server 2013. With Exchange Server 2016 ItemPoint is the only choice for GLR)
If ItemPoint will not be used for GLR, then you can fall back on traditional GLR with NMM GUI. The recovery process here is very similar to the GLR in NMM 8, except that NMM uses ‘block based backup’ mechanism to mount the backup. In NMM 8, NwFS (Networker virtual file system) was used to perform the virtual mount of the backup.
Following are the pre-requisites for successful GLR using NMM GUI.
Once the above pre-requisites are met, you are ready to perform GLR restore with NMM.
4. Select the desired database. Then click ‘Recover…’
5. Click ‘Recover Options…’ if you want to modify the recovery behavior.
b. Under ‘Exchange’ tab below are the options. The defaults are fine for GLR restore.
6. Click start recover
7. Once the restore completes, the following message window is seen:
Review the monitor window for logging of the restore:
8. Click on the ‘recover’ tab to browse the GLR database and expand to the desired mailbox.
9. Select the mailbox or mailbox items
10. If recovering the mailbox/mailbox items back to the original mailbox choose ‘Recover..’ If recovering to another mailbox choose ‘Advanced Recover..’
Here I’m recovering to the original mailbox, so I choose ‘Recover..’. The ‘monitor’ window shows the messages for the restore.
When the recover completes, you can login to the mailbox to confirm the recover was successful. The recovered items are placed under a folder ‘Recovered Items …’ as seen below
This completes GLR using NMM GUI.
If there is a requirement to restore from backups done with NMM 8.2.x, and during install the choice ‘Restore of NMM 8.2.x and earlier backups(VSS workflow)’ was made, then NMM will install the software required to perform this restore.
Note this software is installed in “C:Program FilesEMC NetWorkernsrrpvnmm”.
NMM 9 GUI cannot be used to recover backups performed with NMM 8. To launch the NMM 8 GUI, from program group ‘EMC NetWorker -> NetWorker Tools -> ‘Restore previous NMM release backups’. Start this GUI as ‘administrator’
Check recovering Exchange backups in NMM 8 for details on recovery with NMM 8.