Change of MAC Address on Avamar Virtual Edition 7 servers

The server software will reject a license file if it's been tampered with. The license files will have to be regenerated with the new hardware information. Please get in touch with your account team for assistance with this.

Many thanks!

Have just send off the re-generated gsankeydata.xml files.

My pleasure!

Mmmmmm, sorry to report that the newly generated licenses didn't resolve the problem (invalid license messages in the dpnctl.log, and gsan & MCS not starting). There are lots of details reported in the log that we do not fully understand, and not sure what to do next.

When restarting the server (dpnctl start), we are required to chose if to restore a checkpoint, we chose to restore the last fully validated checkpoint. Is this complicating the issue perhaps? As the checkpoint was made with the previous license?

Happy to post the log, or if it looks too hard, we can just start again, with most of the data safely replicated to another server.

Thanks for the help.

You are right re getting the licensing process exactly right, thanks!

Another question if I may to ensure this doesn't happen again.

We have been evaluating Avamar using AVE 7 servers, and have successfully used replication between servers, configured using the Avamar Administrator GUI only (so I understand is policy-based replication and via the “MCS”).

We desire to employ root-to-root replication to create a simple DR facility, where the primary would be replicated to the secondary daily.

Although we see this method recommended for DR, the main EMC documentation for root-to-root replication is for System Migration (and the replication is handled manually).

After reviewing the available documentation (EMC Support and community based), we are not clear what the easiest replication method is/are, as there seem to be different ways…

1. MCS ("policy-based"?) – easiest, GUI based, client data only
2. EM ("cron-based"?)
3. manual (using the repl_cron.cfg file, so "cron-based" too?)

1. Questions..
   1. Is “Policy-based” the same as “plug-in-based” replication?
   2. As it seems we need to be able to specify the --fullcopy replicate command line option (in the repl_cron.cfg), only a cron-based option is applicable, correct?
   3. We read a Forum post that says “... issue can occur on 7.0 systems if both cron-based and plug-in-based replication are being used. If plug-in-based replication is in use, it is recommended to avoid cron-based replication entirely” – so we need to be careful we do never use policy-based replication on the primary server/s?
   4. We read forum posts about how to monitor the replicate process for progress, so does this mean it cannot be monitored via the Avamar Administrator GUI (eg. Activity page)?
   5. Is the EM the correct GUI monitor for cron-based replications, whether created using EM or manually editing the repl_cron.cfg file?

Thanks again for any help.

I'm glad to hear you were able to get the license issue sorted out.

In general, I would recommend against using root-to-root replication. Standard replication (also called root-to-REPLICATE replication) is much more flexible. The only scenario I'm aware of where root-to-root replication has an advantage over standard replication is in the case where the source system has been completely destroyed and is never coming back.

The main advantage of root-to-root is that you can run an Avamar Administrator Server (MCS) restore on the target, fail the source system's DNS name over to the target system and the clients will never notice the change.

Standard replication has a number of advantages over root-to-root replication:

• With root-to-root replication, the target cannot be used for operational backups (it can only accept new backups in a DR scenario). With standard replication, you can activate clients to the target grid and perform normal backups if you run into some kind of temporary issue with the source, for example.
• Root-to-root replication only permits unidirectional one-to-one replication. Replication can only run source => destination. With standard replication, it's possible to set up bi-directional replication, many-to-one replication and other, more exotic configurations depending on your needs.
• All the data required to perform disaster recovery for a replication source system is replicated to the target during standard replication anyway.

The main downside of standard replication is that failing over the clients in case of a disaster is a manual process.

Furthermore, I would not recommend trying to switch from root-to-REPLICATE to root-to-root replication. Changing replication modes has some serious potential pitfalls around client IDs on the replication target. Client ID issues can cause subtle "slow burn" problems on the system that are very difficult to diagnose and take a long time to correct.

Policy-based and plugin-based replication refer to the same thing. I would strongly recommend sticking with policy-based replication. It is much, much easier to manage.

Replication can be monitored through the Avamar Administrator GUI using the activity monitor, including both policy-based and cron-based replication.

Thanks for the info & advice, I understand better the limitations of root-to-root replication.

If all of this DR subject is nicely described in an EMC document, I am very happy to learn more there & not bother you, but I can’t the subject nicely shrink-wrapped sorry!

Ø All the data required to perform disaster recovery for a replication source system is replicated to the target during standard replication anyway.

Sorry but I don’t quite understand this. I see how clients & their data & logs are all nicely replicated to the REPLICATE domain, but I can’t see the Policy objects/configuration that would be required to resume normal backup operations. Without these, after the primary failure, surely we would need to:

1. Re-assign clients to secondary server, which would mean their folders/domains would have to already exist – is there a way to for these to be replicated (or export/import) ahead of time?

2. Copy the Policy configuration info (datasets etc.) to secondary server, eg. by...

a. Using “mccli group export” to export them to xml file, then use “mccli mcs import” on the target server to import them – this would need to be done periodically anyway to ensure its immediately available, yes?

b. We read “However, groups exported in this way do not contain a list of client members” – so this means clients will have to be manually assigned back to their correct Groups unless it was automated via “mccli group add-client” etc.

At this point, on the secondary server, the clients will appear in their own folders\domains, as well as in the REPLICATE folder, but with different CIDs.

When the primary server is up and running again, either in the same state it was in before, or…

1. If a fresh new server install, then it needs to be configured

2. replicate all the data back again

3. Policy configs imported

4. folders/domains re-created

5. client re-assignment process reversed

6. clients assigned to their correct Group/s

In the Avamar Administration guide, it describes “MCS configuration settings”, backing up & restoring “MCS data”, not sure if this is applicable to our DR scenario.

Thanks again for the help – I am only asking this in detail as after losing 2 AVE servers for a few days, I would like to know there’s a good dependable solution we as Avamar learners can make work!

Just to clarify, EMC normally performs replicate restore operations.

All the data required to perform disaster recovery for a replication source system is replicated to the target during standard replication anyway.

Everything needed to recover the primary server is replicated. Running backups to the replication target in the interim is a separate problem.

I see how clients & their data & logs are all nicely replicated to the REPLICATE domain, but I can’t see the Policy objects/configuration that would be required to resume normal backup operations.

The policies, datasets, etc. are stored in the MCS database. MCS flushes (backups of the MCS configuration and database) are replicated to the target during standard replication. Once the primary system has been returned to service, a "replicate restore" is run to migrate the GSAN accounts and MCS flushes back to the system, then the MCS is restored from flush. This restores the operating configuration of the primary system.

Without these, after the primary failure, surely we would need to: 1.       Re-assign clients to secondary server, which would mean their folders/domains would have to already exist – is there a way to for these to be replicated (or export/import) ahead of time? 2.       Copy the Policy configuration info (datasets etc.) to secondary server, eg. by... a.       Using “mccli group export” to export them to xml file, then use “mccli mcs import” on the target server to import them – this would need to be done periodically anyway to ensure its immediately available, yes? b.      We read “However, groups exported in this way do not contain a list of client members” – so this means clients will have to be manually assigned back to their correct Groups unless it was automated via “mccli group add-client” etc.

This is all true. However, I assume that the goal during a disaster that affects the primary system is to protect the clients temporarily until the primary system can be brought back into service. A disaster is an exceptional event, so "normal backup operations" may not be the best thing to strive for.

If the goal is just to make sure the clients are protected, a one-for-one duplicate of the source configuration seems like it would create a lot of unnecessary administrative overhead. Instead of trying to resume normal operations on the replication target, why not create a domain called "DR 2014-06-16" and activate the clients there temporarily? You could migrate the datasets from the source but you'd have to be sure to keep them in sync. Alternatively, you could configure some basic "DR datasets" directly on the target and use those.

This would be a little more tricky if you're a service provider or subject to regulatory requirements, etc. but for most customers, I suspect it would be easier to manage DR as I've described above.

When the primary server is up and running again, either in the same state it was in before, or… 1.       If a fresh new server install, then it needs to be configured 2.       replicate all the data back again 3.       Policy configs imported 4.       folders/domains re-created 5.       client re-assignment process reversed 6.       clients assigned to their correct Group/s

Once the source system is back up and running, the general process for bringing the system back into production is as follows:

1. Replicate restore of the GSAN User Accounting System information (this includes the domain structure, client CIDs, etc.) and MC_BACKUPS account containing the MCS data.
2. Restore MCS from the most recent flush to restore clients, groups, datasets, etc.
3. Re-activate clients to source grid
4. Replicate restore of backup data (Optional)

Many customers opt to skip the restore of the backup data completely because it can take considerable time. Some customers restore only the most recent backup because this allows the clients to use their cache files. Some customers restore everything because of regulatory requirements, etc.. It depends on the level of risk they're willing to accept.

Unless the source and target are configured for bi-directional replication, any backups created on the target would have to be manually replicated to the newly restored source since -- as you pointed out -- the clients have different CIDs.

My pleasure! I'm always happy to help.

Hey thanks for the info & advice Ian, some good options & thoughts there, and I think I see in the Avamar Administrator GUI where to do that REPLICATE-restore now too.

Thanks.