Saturday, August 5, 2023

Oracle GoldenGate Setup for remote extract from 12.2 database using 21.3 Microservice Architecture

Objective: Oracle GoldenGate Setup for remote extract from 12.2 database using 21.3 Microservice Architecture


In continuation to the effort of db upgrade and replication.

We have completed...
1. TDE setup
2. Database Vault

In this note.. we will see how to 
1) Prepare the source db for GG extract
2) Setup new extract using existing deployment
3) start the extract

--next blog-- below
4) Initial load (which is rman backup->restore->upgrade)
5) setup replicat & start

Objective:
Setup replication between 12.2 source to target 19.3

Source:
DB Name: GGSRC03T
Schema to replicat: encvault_test
Complexity: TDE and Database Vault enabled are enabled for the schema
Server IP:192.168.194.10
Server Port: 1521

Target:
DB Name: GGTGT03T
Schema to replicat: encvault_test
Complexity: TDE and Database Vault enabled are enabled for the schema
Server IP:192.168.194.11
Server Port: 1521

1) Prepare the source db for GG extract

reference url: Oracle Document


## CGGNORTH DATABASE SETUP AT CDB LEVEL
ALTER SESSION SET CONTAINER=cdb$root;
ALTER SYSTEM SET ENABLE_GOLDENGATE_REPLICATION=TRUE;
ALTER SYSTEM SET STREAMS_POOL_SIZE=2G;
ALTER DATABASE FORCE LOGGING;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ARCHIVE LOG LIST;
CREATE TABLESPACE GG_DATA DATAFILE '+DATA' SIZE 100M AUTOEXTEND ON NEXT 100M;
CREATE USER c##ggadmin IDENTIFIED BY PASSWORD CONTAINER=ALL DEFAULT TABLESPACE GG_DATA TEMPORARY TABLESPACE TEMP;
GRANT ALTER SYSTEM TO c##ggadmin CONTAINER=ALL;
GRANT DBA TO c##ggadmin CONTAINER=ALL;
GRANT CREATE SESSION TO c##ggadmin CONTAINER=ALL;
GRANT ALTER ANY TABLE TO c##ggadmin CONTAINER=ALL;
GRANT RESOURCE TO c##ggadmin CONTAINER=ALL;
EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE(' c##ggadmin ',CONTAINER=>'ALL');


Ours is a standalone 12.2 setup. So we will skip CDB related changes.

Command:

ALTER SYSTEM SET ENABLE_GOLDENGATE_REPLICATION=TRUE;


Output:

SQL> create pfile='/home/oracle/dba/pfileGGSRC01T_Bkp.ora' from spfile;
File created.
SQL> ALTER SYSTEM SET ENABLE_GOLDENGATE_REPLICATION=TRUE;
System altered.

Command:

ALTER SYSTEM SET STREAMS_POOL_SIZE=2G;

Output:
SQL> alter system set sga_max_size=3G scope=spfile;
System altered.
SQL> alter system set sga_target=3G scope=spfile;
System altered.
SQL> alter system set pga_aggregate_target=500M scope=spfile;
System altered.
SQL> alter system set STREAMS_POOL_SIZE=750M scope=spfile;
System altered.
SQL>

Command:

ALTER DATABASE FORCE LOGGING;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

Output:
SQL> ALTER DATABASE FORCE LOGGING;
Database altered.
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
Database altered.


----------------------------------------------If you have database vault setup, modify the pfile for the below setting.
SQL> alter system set log_archive_dest_1='location=USE_DB_RECOVERY_FILE_DEST';
System altered.
SQL> alter system set db_recovery_file_dest_size=2G;
System altered.
SQL> sho parameter db_REco

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /oraarch
db_recovery_file_dest_size           big integer 2G
SQL> SQL>
SQL>
SQL> archive log list
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     13
Current log sequence           15
SQL>
-----------------------------------------------------------
>> need to bounce the db once for all.

SQL> sho parameter db_Rec
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /oraarch
db_recovery_file_dest_size           big integer 2G
db_recycle_cache_size                big integer 0
SQL> sho parameter log_archive_dest_1
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_1                   string      location=USE_DB_RECOVERY_FILE_
                                               DEST
..
log_archive_dest_18                  string
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_19                  string
SQL> sho parameter golden
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
enable_goldengate_replication        boolean     TRUE
resource_manage_goldengate           boolean     FALSE
SQL> sho parameter stream
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
streams_pool_size                    big integer 752M
SQL> alter database archivelog;
Database altered.
SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     1
Next log sequence to archive   2
Current log sequence           2
SQL> alter database open;
------------------------------------------------
*.db_recovery_file_dest_size=2G
*.db_recovery_file_dest='/oraarch'
*.log_archive_dest_1='location=USE_DB_RECOVERY_FILE_DEST'

-----------------------------------------------------------------

Command:

CREATE TABLESPACE GG_DATA DATAFILE '/oradata/GGSRC01T/gg_data_01.dbf' SIZE 100M AUTOEXTEND ON NEXT 100M;

output:

SQL> CREATE TABLESPACE GG_DATA DATAFILE '/oradata/GGSRC01T/gg_data_01.dbf' SIZE 100M AUTOEXTEND ON NEXT 100M;
Tablespace created.
SQL>

Create GGADMIN user using either dbv_admin or dbv_acctmgr user. I used dbv_owner

Command:

CREATE USER ggadmin IDENTIFIED BY "ggadmin" DEFAULT TABLESPACE GG_DATA TEMPORARY TABLESPACE TEMP;

Output:

SQL> conn dbv_owner
Enter password:
Connected.

SQL> CREATE USER ggadmin IDENTIFIED BY "ggadmin" DEFAULT TABLESPACE GG_DATA TEMPORARY TABLESPACE TEMP;
User created.
SQL> grant connect to ggadmin;
Grant succeeded.
SQL>
SQL> grant DV_GOLDENGATE_ADMIN to ggadmin; <<< pls grant this priv as well. Otherwise you will face an error later during GG registration.
Grant succeeded.


Command:

Other privileges to be granted as sys:
GRANT ALTER SYSTEM TO ggadmin;
GRANT DBA TO ggadmin;
GRANT CREATE SESSION TO ggadmin;
GRANT ALTER ANY TABLE TO ggadmin;
GRANT RESOURCE TO ggadmin;
EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('ggadmin');

Output:

SQL> GRANT ALTER SYSTEM TO ggadmin;
Grant succeeded.
SQL> GRANT DBA TO ggadmin;
Grant succeeded.
SQL> GRANT CREATE SESSION TO ggadmin;
Grant succeeded.
SQL> GRANT ALTER ANY TABLE TO ggadmin;
Grant succeeded.
SQL> GRANT RESOURCE TO ggadmin;
Grant succeeded.
SQL> EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('ggadmin');
PL/SQL procedure successfully completed.
SQL>
Post DB bounce  - verification:
SQL> select name,open_mode,force_logging,SUPPLEMENTAL_LOG_DATA_MIN from v$database;
NAME      OPEN_MODE            FORCE_LOGGING                           SUPPLEME
--------- -------------------- --------------------------------------- --------
GGSRC03T  READ WRITE           YES                                     YES

SQL>


>> source prepared.

2) Setup new extract using existing deployment

Add credential in preparation for extract:

ADD CREDENTIALS step:

[oracle@vcentos79-oracle-sa1 ~]$ export OGG_HOME=/u01/app/oracle/product/21.3.0/ogg_home_1
[oracle@vcentos79-oracle-sa1 ~]$ export PATH=$PATH:$OGG_HOME/bin

[oracle@vcentos79-oracle-sa1 ~]$ adminclient
Oracle GoldenGate Administration Client for Oracle
Version 21.3.0.0.0 OGGCORE_21.3.0.0.0_PLATFORMS_210728.1047
Copyright (C) 1995, 2021, Oracle and/or its affiliates. All rights reserved.
Oracle Linux 7, x64, 64bit (optimized) on Jul 28 2021 12:32:46
Operating system character set identified as UTF-8.

OGG (not connected) 2> connect http://127.0.0.1:9011 deployment oggdep01 user ggsca password ggsca
OGG (http://127.0.0.1:9011 oggdep01) 3>


Add credential:

alter credentialstore add user ggadmin@192.168.194.10:1521/ggsrc03t alias ggsrc03t  PASSWORD ggadmin;

OGG (http://127.0.0.1:9011 oggdep01) 14> alter credentialstore add user ggadmin@192.168.194.10:1521/ggsrc01t alias ggsrc01t  PASSWORD ggadmin;

2023-07-23T12:09:43Z  INFO    OGG-15114  Credential store altered.

OGG (http://127.0.0.1:9011 oggdep01) 15>

<<< but the above credential will fail with ORA-01017 invalid user/pswd. Better to update the password through cURL or web UI for the deployment.

After password update..

OGG (http://127.0.0.1:9011 oggdep01) 3> DBLOGIN USERIDALIAS ggsrc03t

Successfully logged into database.
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 4>


Add trandata, heartbeat & checkpoint tables:
ADD SCHEMATRANDATA encvault_test
ADD HEARTBEATTABLE
ADD CHECKPOINTTABLE ggadmin.ggs_checkpoint

Output:

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 4> ADD SCHEMATRANDATA encvault_test
2023-08-05T20:03:35Z  INFO    OGG-01788  SCHEMATRANDATA has been added on schema "encvault_test".
2023-08-05T20:03:35Z  INFO    OGG-01976  SCHEMATRANDATA for scheduling columns has been added on schema "encvault_test".
2023-08-05T20:03:35Z  INFO    OGG-10154  Schema level PREPARECSN set to mode NOWAIT on schema "encvault_test"

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 5> ADD HEARTBEATTABLE
2023-08-05T20:03:51Z  INFO    OGG-14101  Successfully added heartbeat table.

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 6> ADD CHECKPOINTTABLE ggadmin.ggs_checkpoint
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 7>

In the source db:

OWNER                         ,OBJECT_NAME                   ,CREATED             ,LAST_DDL_TIME       ,OBJECT_TYPE
------------------------------,------------------------------,--------------------,--------------------,-----------------------
.....
GGADMIN                       ,GG_HEARTBEAT_SEED             ,05/AUG/2023 21:03:48,05/AUG/2023 21:03:48,TABLE
GGADMIN                       ,GG_HEARTBEAT                  ,05/AUG/2023 21:03:48,05/AUG/2023 21:03:48,TABLE <<< HEARTBEAT table
GGADMIN                       ,GG_HEARTBEAT_HISTORY          ,05/AUG/2023 21:03:48,05/AUG/2023 21:03:48,TABLE
GGADMIN                       ,GG_LAG                        ,05/AUG/2023 21:03:48,05/AUG/2023 21:03:48,VIEW
GGADMIN                       ,GG_LAG_HISTORY                ,05/AUG/2023 21:03:48,05/AUG/2023 21:03:48,VIEW
GGADMIN                       ,GG_UPDATE_HB_TAB              ,05/AUG/2023 21:03:51,05/AUG/2023 21:03:51,PROCEDURE
GGADMIN                       ,GG_PURGE_HB_TAB               ,05/AUG/2023 21:03:51,05/AUG/2023 21:03:51,PROCEDURE
GGADMIN                       ,GG_UPDATE_HEARTBEATS          ,05/AUG/2023 21:03:51,05/AUG/2023 21:03:51,JOB
GGADMIN                       ,GG_PURGE_HEARTBEATS           ,05/AUG/2023 21:03:51,05/AUG/2023 21:03:51,JOB
GGADMIN                       ,GGS_CHECKPOINT                ,05/AUG/2023 21:03:57,05/AUG/2023 21:03:57,TABLE <<< CHECKPOINT table
GGADMIN                       ,SYS_C007358                   ,05/AUG/2023 21:03:57,05/AUG/2023 21:03:57,INDEX
GGADMIN                       ,GGS_CHECKPOINT_LOX            ,05/AUG/2023 21:03:57,05/AUG/2023 21:03:57,TABLE
GGADMIN                       ,SYS_C007364                   ,05/AUG/2023 21:03:57,05/AUG/2023 21:03:57,INDEX



Necessary tables are added.

Next step: Add Extract -> Add Extract Trail file -> Register the db

Adding Extract:

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 7> add extract rxt2, INTEGRATED TRANLOG, begin NOW
2023-08-05T20:05:42Z  INFO    OGG-08100  Integrated Extract added.
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 8>

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 8> info extract rxt2 detail
Extract    RXT2      Initialized  2023-08-05 21:05   Status STOPPED
Checkpoint Lag       00:00:00 (updated 00:00:13 ago)
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2023-08-05 21:05:42
                     SCN 0.0 (0)
Encryption Profile   LocalWallet
                     *** No TARGETEXTTRAILS specified yet ***

Integrated Extract outbound server first scn: Unavailable.
Integrated Extract outbound server filtering start scn: Unavailable.
  Extract Source                          Begin             End
  Not Available                           * Initialized *   2023-08-05 21:05

Current directory    /
Report file          /oggdata/oggdep01/var/lib/report/RXT2.rpt (does not yet exist)
Parameter file       /oggdata/oggdep01/etc/conf/ogg/rxt2.prm (does not yet exist)
Checkpoint file      /oggdata/oggdep01/var/lib/checkpt/RXT2.cpe
Process file         /oggdata/oggdep01/var/run/RXT2.pce
Error log            /oggdata/oggdep01/var/log/ggserr.log
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 9>


## >Add extract trail by modifying the param:

Create the file /oggdata/oggdep01/etc/conf/ogg/rxt2.prm with the below params:
EXTRACT rxt2
USERIDALIAS ggsrc03t DOMAIN OracleGoldenGate
EXTTRAIL src/r2
DDL INCLUDE MAPPED
TABLE encvault_test.*;
TABLE encvault_test.*;


OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 10> view params rxt2
EXTRACT rxt2
USERIDALIAS ggsrc03t DOMAIN OracleGoldenGate
EXTTRAIL src/r2
DDL INCLUDE MAPPED
TABLE encvault_test.*;
TABLE encvault_test.*;
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 11>

==========If you want you can put a limit for extract trail file. I didnt test it though, still it sets some limitation, see info command output below.

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc01t@GGSRC01T) 7> ADD EXTTRAIL src/r2 EXTRACT rxt2 MEGABYTES 1500
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc01t@GGSRC01T) 8>
=================================================================
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 11> info extract rxt2 detail
Extract    RXT2      Initialized  2023-08-05 21:05   Status STOPPED
Checkpoint Lag       00:00:00 (updated 00:04:23 ago)
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2023-08-05 21:05:42
                     SCN 0.0 (0)
Encryption Profile   LocalWallet
  Target Extract Trails:
  Trail Name                                       Seqno        RBA     Max MB Trail Type
  src/r2                                               0          0       1500 EXTTRAIL

Integrated Extract outbound server first scn: Unavailable.
Integrated Extract outbound server filtering start scn: Unavailable.
  Extract Source                          Begin             End
  Not Available                           * Initialized *   2023-08-05 21:05

Current directory    /
Report file          /oggdata/oggdep01/var/lib/report/RXT2.rpt (does not yet exist)
Parameter file       /oggdata/oggdep01/etc/conf/ogg/rxt2.prm
Checkpoint file      /oggdata/oggdep01/var/lib/checkpt/RXT2.cpe
Process file         /oggdata/oggdep01/var/run/RXT2.pce
Error log            /oggdata/oggdep01/var/log/ggserr.log
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 12>


Now register the DB:
To register the db, we have to disconnect our current session and reconnect
>> exit previous connection

[oracle@vcentos79-oracle-sa1 ~]$ adminclient
Oracle GoldenGate Administration Client for Oracle
Version 21.3.0.0.0 OGGCORE_21.3.0.0.0_PLATFORMS_210728.1047
Copyright (C) 1995, 2021, Oracle and/or its affiliates. All rights reserved.
Oracle Linux 7, x64, 64bit (optimized) on Jul 28 2021 12:32:46
Operating system character set identified as UTF-8.

OGG (not connected) 1> connect http://127.0.0.1:9011 deployment oggdep01 user ggsca password ggsca

OGG (http://127.0.0.1:9011 oggdep01) 2> DBLOGIN USERIDALIAS ggsrc03t
Successfully logged into database.

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 3> register extract rxt2 database

2023-08-05T20:12:31Z  ERROR   OGG-08221  Cannot register or unregister Extract group RXT2 because of the following SQL error: OCI Error ORA (status = 26723-ORA-26723: user "GGADMIN" requires the role "DV_STREAMS_ADMIN/DV_XSTREAM_ADMIN/DV_GOLDENGATE_ADMIN"
ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 581
ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 635
ORA-06512: at "SYS.DBMS_CAPTURE_ADM_INTERNAL", line 404
ORA-06512: at "SYS.DBMS_CAPTURE_ADM_IVK", line 176
ORA-06512: at "SYS.DBMS_CAPTURE_ADM", line 156
ORA-06512: at line 1
).
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 4>


To fix  OGG-08221 -- we need to grant additional privs to ggadmin user. I suppose this is occuring because of database vault. Irrespective of source or target, if Databse Vault is configured, we need to grant additional privileges [DV_GOLDENGATE_ADMIN].


SQL> conn dbv_owner
Enter password:
Connected.

SQL> grant DV_GOLDENGATE_ADMIN to ggadmin;
Grant succeeded.
SQL> exit


OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 4> register extract rxt2 database
2023-08-05T20:16:51Z  INFO    OGG-02003  Extract group RXT2 successfully registered with database at SCN 1443018.

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 5> info extract rxt2 detail
Extract    RXT2      Initialized  2023-08-05 21:05   Status STOPPED
Checkpoint Lag       00:00:00 (updated 00:14:58 ago)
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2023-08-05 21:05:42
                     SCN 0.0 (0)
Encryption Profile   LocalWallet
  Target Extract Trails:
  Trail Name                                       Seqno        RBA     Max MB Trail Type
  src/r2                                               0          0       1500 EXTTRAIL

Integrated Extract outbound server first scn: 0.1443018 (1443018)
Integrated Extract outbound server filtering start scn: 0.1443018 (1443018)
  Extract Source                          Begin             End
  Not Available                           * Initialized *   2023-08-05 21:05

Current directory    /
Report file          /oggdata/oggdep01/var/lib/report/RXT2.rpt (does not yet exist)
Parameter file       /oggdata/oggdep01/etc/conf/ogg/rxt2.prm
Checkpoint file      /oggdata/oggdep01/var/lib/checkpt/RXT2.cpe
Process file         /oggdata/oggdep01/var/run/RXT2.pce
Error log            /oggdata/oggdep01/var/log/ggserr.log

3) Start the extract: 

OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 6> start extract rxt2
2023-08-05T20:21:42Z  INFO    OGG-00975  Extract group RXT2 starting.
2023-08-05T20:21:42Z  INFO    OGG-15426  Extract group RXT2 started.


OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 7> info extract rxt2 detail
Extract    RXT2      Initialized  2023-08-05 21:05   Status RUNNING <<< Extract running, same can be verified using GG browser as well.
Checkpoint Lag       00:00:00 (updated 00:16:03 ago)
Process ID           739
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2023-08-05 21:05:42
                     SCN 0.0 (0)
Encryption Profile   LocalWallet
  Target Extract Trails:
  Trail Name                                       Seqno        RBA     Max MB Trail Type
  src/r2                                               0          0       1500 EXTTRAIL

Integrated Extract outbound server first scn: 0.1443018 (1443018)
Integrated Extract outbound server filtering start scn: 0.1443018 (1443018)
  Extract Source                          Begin             End
  Not Available                           * Initialized *   2023-08-05 21:05

Current directory    /
Report file          /oggdata/oggdep01/var/lib/report/RXT2.rpt
Parameter file       /oggdata/oggdep01/etc/conf/ogg/rxt2.prm
Checkpoint file      /oggdata/oggdep01/var/lib/checkpt/RXT2.cpe
Process file         /oggdata/oggdep01/var/run/RXT2.pce
Error log            /oggdata/oggdep01/var/log/ggserr.log
OGG (http://127.0.0.1:9011 oggdep01 as ggsrc03t@GGSRC03T) 8>




Thank you!

No comments:

Post a Comment

Flashback data archive steps

 Objective: Ways to track DML changes in a table Detailed objective: We should be able to track the changes in the table at any point in tim...