Data Guard migration with ZDM

In the previous post I showed how to setup cloning or migration using offline physical option with Zero Downtime Migration (ZDM). This time let’s take a look what happens with ONLINE PHYSICAL option.

Common migration requirement is to migrate with lowest amount of downtime and with all the latest data available and Data Guard is excellent tool for this in case all the pre-requisites match. I won’t go into all ZDM setup details like in the previous post, but I’ll look the configuration required what we need for Data Guard setup.

ZDM Configuration

I’ve setup new response file, zdm_online_physical.rsp from the template which you can see below.

#------------------------------------------#
TGT_DB_UNIQUE_NAME=TEST_phx17v
#------------------------------------------#
## Migration Method
## -----------------------------------------#
# Allowed values are
# ONLINE_PHYSICAL
# OFFLINE_PHYSICAL
#------------------------------------------#
DATA_TRANSFER_MEDIUM=OSS
#------------------------------------------#
# Platform Type
#------------------------------------------#
PLATFORM_TYPE=VMDB
#------------------------------------------#
# Database Backup Cloud Service Properties #
#------------------------------------------#
HOST=https://swiftobjectstorage.us-phoenix-1.oraclecloud.com/v1/yzzlzrqnfjpi
OPC_CONTAINER=zdmbackup_swift
#------------------------------------------#
# Target Database datapatch phase execution
#------------------------------------------#
TGT_SKIP_DATAPATCH=FALSE
#------------------------------------------#
# Shutdown source database after migration
# -----------------------------------------#
SHUTDOWN_SRC=FALSE
#------------------------------------------#
# ZDM UPLOAD LOGS to OSS Pre-Authenticated URL
# -----------------------------------------#
ZDM_LOG_OSS_PAR_URL=https://objectstorage.ca-toronto-1.oraclecloud.com/p/sdffdfdfd555trrrgff4s5QFt3N0Kq5R-mlURWVyh2Mhe487ARDvjIgn55_EO2/n/xyyqasrqnfjpi/b/zdm_logs/o/
#------------------------------------------#
ZDM_USE_DG_BROKER=TRUE
#------------------------------------------#

Don’t be fooled by seeing only few config items here. There are tons of other options related to ZDM setup, if you want to use existing backup, use existing Data Guard standby or restore using backup or active duplicate etc. It’s highly customizable so you are not limited to specific execution but have some room to do configuration how it’s required.

Few options I want to highlight, I’m not skipping datapatch execution on target after switchover, I’m keeping my source database up after switchover and I’m using DG broker for Data Guard (new in 21.3 version!).

For example, depending on your source and target system patch levels, you might need to skip datapatch via ZDM.

All the response file parameters can be found from here.

I’ve also configured ZDM logs to get uploaded to Object Storage bucket which requires a pre-authenticated request, PAR, configuration on Object Storage. Remember, the bucket needs to be accessible from the ZDM server, so likely placed in same region as Service Gateway doesn’t route traffic to another region.

You will also need to check both primary, and standby to be able to resolve themselves, so I’ve added their IPs and hostnames to /etc/hosts on both servers. Otherwise, the initial pre-check will fail with ZDM.

Running ZDM

This time I want to stop run ZDM up until Data Guard has been setup, but not do the actual switchover yet. For zdmcli, there’s a flag for it: -pauseafter

How to find which step I want to stop my execution on? You can use flag -listphases.

I wasn’t initially sure how listphases works, so I ran my command with flags -eval -listphases. Evaluation flag validates your execution and lists any errors, it also has different steps compared to normal execution, this resulted phases for -eval being shorter!

[zdmuser@bastion bin]$ ./zdmcli migrate database -sourcedb PROD_yyz1c4 -sourcenode prod -targetnode test -backupuser "oracleidentitycloudservice/tfg@tfg.com" -rsp /home/zdmuser/app/zdm/rhp/zdm/template/zdm_online_physical.rsp -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -srcauth zdmauth -srcarg1 user:opc -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -listphases
bastion.subnetpublic.vcndns.oraclevcn.com: Audit ID: 21
bastion: 2022-03-20T17:50:43.146Z : Processing response file ...
bastion: 2022-03-20T17:50:43.200Z : Processing response file ...
pause and resume capable phases for this operation: "
ZDM_GET_SRC_INFO
ZDM_GET_TGT_INFO
ZDM_PRECHECKS_SRC
ZDM_PRECHECKS_TGT
ZDM_SETUP_SRC
ZDM_SETUP_TGT
ZDM_PREUSERACTIONS
ZDM_PREUSERACTIONS_TGT
ZDM_OBC_INST_SRC
ZDM_OBC_INST_TGT
ZDM_VALIDATE_SRC
ZDM_VALIDATE_TGT
ZDM_BACKUP_FULL_SRC
ZDM_BACKUP_INCREMENTAL_SRC
ZDM_DISCOVER_SRC
ZDM_COPYFILES
ZDM_PREPARE_TGT
ZDM_SETUP_TDE_TGT
ZDM_CLONE_TGT
ZDM_FINALIZE_TGT
ZDM_CONFIGURE_DG_SRC
ZDM_SWITCHOVER_SRC
ZDM_SWITCHOVER_TGT
ZDM_POST_DATABASE_OPEN_TGT
ZDM_DATAPATCH_TGT
ZDM_POST_MIGRATE_TGT
ZDM_POSTUSERACTIONS
ZDM_POSTUSERACTIONS_TGT
ZDM_CLEANUP_SRC
ZDM_CLEANUP_TGT"

You can see from steps above, I want to pause after ZDM_CONFIGURE_DG_SRC step, so just before switchover.

./zdmcli migrate database -sourcedb PROD_yyz1c4 -sourcenode prod -targetnode test -backupuser "oracleidentitycloudservice/tfg@tfg.com" -rsp /home/zdmuser/app/zdm/rhp/zdm/template/zdm_online_physical.rsp -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -srcauth zdmauth -srcarg1 user:opc -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -pauseafter ZDM_CONFIGURE_DG_SRC

Above command will run all the steps before ZDM_SWITCHOVER_SRC and stop there.

Validating Data Guard and resuming ZDM

Once I’m running the job, I can query job status with zdmcli, and see which steps we have still to go. After database has been restored and Data Guard configured, it shows status PENDING for the remaining tasks.

[zdmuser@bastion bin]$ ./zdmcli query job -jobid 7
bastion.subnetpublic.vcndns.oraclevcn.com: Audit ID: 22
Job ID: 7
User: zdmuser
Client: bastion
Job Type: "MIGRATE"
Scheduled job execution start time: 2022-03-19T20:55:33Z. Equivalent local time: 2022-03-19 20:55:33
Current status: PAUSED
Current Phase: "ZDM_CONFIGURE_DG_SRC"
Result file path: "/home/zdmuser/app/base/chkbase/scheduled/job-7-2022-03-19-20:55:51.log"
Metrics file path: "/home/zdmuser/app/base/chkbase/scheduled/job-7-2022-03-19-20:55:51.json"
Job execution start time: 2022-03-19 20:55:51
Job execution end time: 2022-03-19 22:10:07
Job execution elapsed time: 54 minutes 16 seconds
ZDM_GET_SRC_INFO .............. COMPLETED
ZDM_GET_TGT_INFO .............. COMPLETED
ZDM_PRECHECKS_SRC ............. COMPLETED
ZDM_PRECHECKS_TGT ............. COMPLETED
ZDM_SETUP_SRC ................. COMPLETED
ZDM_SETUP_TGT ................. COMPLETED
ZDM_PREUSERACTIONS ............ COMPLETED
ZDM_PREUSERACTIONS_TGT ........ COMPLETED
ZDM_OBC_INST_SRC .............. COMPLETED
ZDM_OBC_INST_TGT .............. COMPLETED
ZDM_VALIDATE_SRC .............. COMPLETED
ZDM_VALIDATE_TGT .............. COMPLETED
ZDM_BACKUP_FULL_SRC ........... COMPLETED
ZDM_BACKUP_INCREMENTAL_SRC .... COMPLETED
ZDM_DISCOVER_SRC .............. COMPLETED
ZDM_COPYFILES ................. COMPLETED
ZDM_PREPARE_TGT ............... COMPLETED
ZDM_SETUP_TDE_TGT ............. COMPLETED
ZDM_CLONE_TGT ................. COMPLETED
ZDM_FINALIZE_TGT .............. COMPLETED
ZDM_CONFIGURE_DG_SRC .......... COMPLETED
ZDM_SWITCHOVER_SRC ............ PENDING
ZDM_SWITCHOVER_TGT ............ PENDING
ZDM_POST_DATABASE_OPEN_TGT .... PENDING
ZDM_DATAPATCH_TGT ............. PENDING
ZDM_POST_MIGRATE_TGT .......... PENDING
ZDM_POSTUSERACTIONS ........... PENDING
ZDM_POSTUSERACTIONS_TGT ....... PENDING
ZDM_CLEANUP_SRC ............... PENDING
ZDM_CLEANUP_TGT ............... PENDING

Pause After Phase: "ZDM_CONFIGURE_DG_SRC"

If I login to my standby host, I can start dgmgr and see if configuration has been done as expected.

Welcome to DGMGRL, type "help" for information.
Connected to "TEST_phx17v"
Connected as SYSDBA.
DGMGRL> show configuration;

Configuration - ZDM_prod_yyz1c4

  Protection Mode: MaxPerformance
  Members:
  prod_yyz1c4 - Primary database
    test_phx17v - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 29 seconds ago)

DGMGRL> show database 'prod_yyz1c4';

Database - prod_yyz1c4

  Role:               PRIMARY
  Intended State:     TRANSPORT-ON
  Instance(s):
    PROD

Database Status:
SUCCESS

DGMGRL> show database 'test_phx17v';

Database - test_phx17v

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          0 seconds (computed 1 second ago)
  Average Apply Rate: 7.00 KByte/s
  Real Time Query:    OFF
  Instance(s):
    TEST

Database Status:
SUCCESS

All good here! ZDM did setup Data Guard and broker before stopping on the step just before switchover.

Stopping ZDM here gives you an option to test functionality on your standby, potentially converting it to snapshot standby.

Finalizing Switchover

Now to finalize my migration, I will just resume my job with zdmcli and let it run until finish. Note that I could run up to another specific step with -pauseafter, which might be something you want to do.

[zdmuser@bastion bin]$ ./zdmcli resume job -jobid 7                                               bastion.subnetpublic.vcndns.oraclevcn.com: Audit ID: 23

And to verify all is proceeding as planned, few extracts from the log file.

bastion: 2022-03-21T18:49:29.189Z : Executing phase ZDM_SWITCHOVER_SRC
bastion: 2022-03-21T18:49:29.190Z : Switching database PROD_yyz1c4 on the source node prod to standby role ...
bastion: 2022-03-21T18:49:29.192Z : checking if source database is ready for switching role...
prod: 2022-03-21T18:49:44.046Z : Validating database PROD_yyz1c4 role is PRIMARY...
prod: 2022-03-21T18:49:44.554Z : Validating database PROD_yyz1c4 is in open mode...
prod: 2022-03-21T18:49:45.164Z : Waiting for PROD_yyz1c4 to catch up, this could take a few minutes...
prod: 2022-03-21T18:49:57.687Z : source database is ready for switching role...
bastion: 2022-03-21T18:49:57.705Z : checking if target database is ready for switching role...
test: 2022-03-21T18:50:08.962Z : Validating database TEST_phx17v role is STANDBY...
test: 2022-03-21T18:50:15.446Z : target database is ready for switching role...
prod: 2022-03-21T18:50:29.270Z : Executing Oracle Data Guard Broker switchover to database "TEST_phx17v" on database "PROD_yyz1c4" ...
prod: 2022-03-21T18:52:12.442Z : Oracle Data Guard Broker switchover to database "TEST_phx17v" executed successfully on database "PROD_yyz1c4"
prod: 2022-03-21T18:52:13.552Z : Updating PROD_yyz1c4 startup option
prod: 2022-03-21T18:52:15.564Z : Switchover actions in the source environment executed successfully
bastion: 2022-03-21T18:52:15.581Z : Execution of phase ZDM_SWITCHOVER_SRC completed
####################################################################
bastion: 2022-03-21T18:52:16.046Z : Executing phase ZDM_SWITCHOVER_TGT
bastion: 2022-03-21T18:52:16.047Z : Switching database TEST_phx17v on the target node test to primary role ...
test: 2022-03-21T18:52:35.117Z : Updating TEST_phx17v startup option
test: 2022-03-21T18:54:13.048Z : Warning: Updating primary database PROD_yyz1c4 with standby parameters failed, archive logs will not be shipped to standby of database PROD_yyz1c4
test: 2022-03-21T18:54:13.055Z : Switchover actions in the target environment executed successfully
bastion: 2022-03-21T18:54:13.076Z : Execution of phase ZDM_SWITCHOVER_TGT completed

bastion: 2022-03-21T18:55:53.214Z : Executing phase ZDM_DATAPATCH_TGT
bastion: 2022-03-21T18:55:53.216Z : Executing datapatch for database TEST_phx17v on the target node test ...
test: /u01/app/oracle/product/19.0.0.0/dbhome_1
test:
test: TEST
test:
test: trying datapatch run for TEST, attempt### 1 ###
test: datapatch completed successfully for database : TEST_phx17v
bastion: 2022-03-21T18:56:59.064Z : Execution of phase ZDM_DATAPATCH_TGT completed
####################################################################
bastion: 2022-03-21T18:56:59.366Z : Executing phase ZDM_POST_MIGRATE_TGT
test: 2022-03-21T18:57:32.227Z : Post database migration actions at the target executed successfully
bastion: 2022-03-21T18:57:32.242Z : Execution of phase ZDM_POST_MIGRATE_TGT completed
####################################################################
bastion: 2022-03-21T18:57:32.737Z : Executing phase ZDM_CLEANUP_SRC
bastion: 2022-03-21T18:57:50.490Z : Executing phase ZDM_CLEANUP_TGT
bastion: 2022-03-21T18:58:17.908Z : Execution of phase ZDM_CLEANUP_TGT completed
####################################################################

I took some rows off but you can see the switchover goes as planned, there is a warning about standby configuration failed which would need some investigation. Afterwards, ZDM is applying the datapatch and doing all the necessary cleanup.

You could also do the switchover manually but you would lose the cleanup features which are automated.

Summary

Using Data Guard is common for migrations when you need shorten the downtime windows and have all the pre-requisites with matching versions and platforms met. To have configuration done automatically shortens the setup time and removes again many of the manual steps.

Still, to run this in production environment, you will need to obtain authorization so ZDM can perform the source side database setup. If it’s a large database, perhaps you need to think what is the best way to set up the standby and play around with different ZDM options.

Remember, there are options to customize each step as well with your own scripts, running them either before or after each step. It gives you a lot of flexibility to meet all the requirements!

Next post, I’ll be looking offline migration to Autonomous and the new integrated CPAT tool.