ODI 12C – Slowly Changing Dimensions

In this blog we will explore how we can setup Slowly Changing Dimension SCD2 in ODI 12c.

Slowly Changing Dimensions: Attributes that changes slowly over the period of time, rather than changing frequent. In order to generate the historical reports in DW we need to track the history of dimension.

Type 1: When there is no need to store historical data in the dimension table. So process overwrites the old data with the new data in the dimension table. This is normal incremental update.

Type 2: When there is need to stores the entire history of the attribute in the dimension table in the form of separate rows. Using dates & current flag you can identify which records is currently active. See below example.

scd2

Type 3: When there is need to store current as well as previous value of the attribute. In this scenario two separate columns required in table to store current as well as previous value. See below example.

scd3


Import the IKM: Oracle Slowly Changing Dimension

IKM

Step 1: Create the tables & reverse engineered the same in your model. I have created below tables in localhost database.

 

  CREATE TABLE "SRC_EMP_ASSIGNMENT"
   (	EMP_ID VARCHAR2(10 CHAR),
	ASSIGNMENT_ID VARCHAR2(150 CHAR)
   );
  CREATE TABLE "TRG_EMP_ASSIGNMENT"
   (  ROW_WID NUMBER,
      EMP_ID VARCHAR2(10 CHAR),
      ASSIGNMENT_ID VARCHAR2(150 CHAR),
      ASSIGNMENT_TYPE VARCHAR2(5 CHAR),
      CURRENT_FLAG CHAR(1 CHAR),
      EFF_START_DATE DATE,
      EFF_END_DATE DATE
   );
 CREATE SEQUENCE SEQ_TRG_EMP_ASSIGNMENT START WITH 1;

 

Step 2: Open the target data-store and change the OLAP type to “Slowly Changing Dimension“.

1

Step 3: Now you have to set the SCD behavior of columns for target SCD2 data-store.

  • Surrogate Key – Unique key for table which does not have any business meaning.
  • Natural Key – Unique Key from the source data.
  • Current Record Flag – Indicate whether the record is active or old.
  • Starting Timestamp – Indicate the starting date of that entry
  • Ending Timestamp – Indicate the ending date of that entry
  • Overwrite on Change – If you don’t want to enable on SCD2 on any column.
  • Add Row on Change – If you want to enable on SCD2 on any column.

In this example, for target table : TRG_EMP_ASSIGNMENT

  • Surrogate KeyROW_WID
  • Natural KeyEMP_ID
  • Current Record FlagCURRENT_FLAG
  • Starting TimestampEFF_START_DATE DATE
  • Ending TimestampEFF_END_DATE DATE
  • Overwrite on ChangeASSIGNMENT_TYPE (Don’t want to track history for this)
  • Add Row on Change ASSIGNMENT_ID

Open the data-store, attributes & select the SCD behavior for every column.

8

Step 4: Create Mapping: See below execute on hints for columns.

mappig

Executing Mapping 1st Time: Initial Run

Source Date:

src

Target: See the end dates of all the records, default 01-01-2400 EFF_END_DATE.

trg

Executing Mapping 1st Time: Incremental Run

Source Data:

src2

Target: 

trg2

In above screenshot you can notice that previous assignment for EMP_ID 100 has been updated with CURRENT_FLAG=0 and EFF_END_DATE updated. New assignment for EMP_ID 100 has been created with CURRENT_FLAG=1 and default 01-01-2400 EFF_END_DATE.

odiopr


Thanks!

Happy Learning! Your feedback would be appreciated!

ODI 12c | Version Control System Setup – Git

In this blog we will see step by step how we can setup the version control in ODI 12c using git. Git – open source version control system for tracking changes in source code during software development. I have used github.com/ to create a repository for my ODI 12c objects versioning.


Step 1: Created a repository using the github.com/

1

Copy the git url for you repository.

https://github.com/shobhit-singh/odi12c-repo.git

2

Step 2: Open the ODI Studio & change your “Version Control Management System” to git by default it was set to ODI.

3

4

Step 3: Setup the git, go to setting & edit connection.

5

Provide the git url, github repository. Change the authentication to SSL.

6

Accept the server certificate.

7

Step 3: Clone remote repository with local folder.

8

9

10

Step 4: Click on configure to setup the branch. I have selected the master branch & save the settings.

11

12

Adding ODI Object First Time in VCS:

I have created the Model Folder, Sub-Model Folder, Model & Data-store in ODI.

datastore

Right Click on the Model Folder & Add to VCS, as it was new object & going first time.

13

You can select object also if you want to move child objects also. Add comments & Click on OK.

14

15

You can check the ODI Objects in GitHub.

Check the objects

Re-Adding ODI Object in VCS:

I have changed the data-store ‘TEST_TABLE’ in ODI. I have added the column in data-store. Now I have create a new version for this object. See below steps to create new version.

16

Add comments & Click on OK.

17

18

Check the version in GitHub: Now this object has 2 version.

19

 


Thanks!

Happy Learning! Your feedback would be appreciated!

ODI 12c | Master & Work Repository Setup

ODI 12c | Master & Work Repository Setup

In this blog we will see step by step how we can setup the ODI 12c Master & Work Repository:


Prerequisite: Setup the repository schema first in database(Supported by ODI). I have created the schema in the Oracle database. Refer the blog for creating new schema in Oracle schema Oracle Database 11g | Create User

CREATE USER ODI_REPO IDENTIFIED BY welcome;
GRANT CONNECT, RESOURCE, DBA TO ODI_REPO;
GRANT CREATE SESSION TO ODI_REPO;
GRANT DBA TO ODI_REPO;
GRANT ALL PRIVILEGES TO ODI_REPO;

You can create separate schema for master repository as well as work repository, but in this blog we going to use only one schema for both master as well as work repository. Refer this blog for more understanding of ODI Repositories | Master & Work Repository

Checkout this blog for understanding of relationship between ODI Master & Work Repository.

Setup Master Repository

Open the ODI 12c Studio. Go to File -> New. Select “Create  New Master Repository“.

1

Step 1. Provide all the details for repository connection.  Like schema name, password, JDBC details, DBA credential details etc. See below screenshot.

jdbc:oracle:thin:@localhost:1522/orcl.168.56.1

2

After providing all the details, test the connection. Click next.

Step 2: Provide authentication mode using ODI authentication, provide supervisor password & note it down somewhere as it is required for login setup.

3

Click on finish after providing supervisor password. It will start the setup of the master repository. It will take few minutes to complete the setup of master repository.

4

5

master done

Here the master repository has been created. Now you have create login for the master repository: Follow below steps


Setup Master Repository Login

Step 1: Open the ODI 12c Studio. Go to File -> New. Select “Create  New ODI Repository Login“.

master login

Step 2. Provide the detail like login name (provide any name), supervisor name & supervisor password you provided at the time of repository setup.

For Database Connection: Provide details for master reposotory schema details. Select radio button master repository. Test the connection.

login2

Click on ‘Ok’. You are done with the master repository login setup. Now you have to login your master repository & create the work repository inside it.

In ODI 12c Studio. Click on connect repository.

connect

Below screen will come with all pre-populated data, you just have to click ‘Ok’. Now you are inside the master repository. Check the repository details ODI->’Repository Connection Information’

ml2

ml3

Here you have logged in ODI Master Repository. Notice here the designer navigator is by default disabled. Basically you cannot do development work in master repository, you have to create work repository for this master to start the development work.

Now you have to setup work repository under this master repository: Follow below steps


Setup Work Repository:

Checkout this blog for understanding of relationship between ODI Master & Work Repository.

Go to Topology Navigator, Repositories -> New Work Repository

work1

Notice here I have provided the same schema, you can create separate schema for work schema which is recommended.

work2

Provide the password for work repository& select development type. As you are going to develop ETL in this work repository. For productions & UAT generally we select ‘Execution‘ as work repository type where Designer navigator is by default disabled. It means you can develop object in Execution type work repository.

work3

Click on Finish.

work4

It will ask you to setup the login for work repo. Click on yes & provide the login name.

worklogin

worklogin2

workdone

Here the work repository setup &  login has been done & disconnect from master repository and login the work repository.

connect

Check the repository details ODI->’Repository Connection Information‘.

Notice here the designer navigator is not disabled. As this is as development type work repository, you can start the development work by creating new project.

Dataserver, Physical Schema, Logical Schema & Context | Oracle Data Integrator 11g

Model & Datastore| Creating and Reverse-Engineering| Oracle Data Integrator 11g

worklogin3


Thanks!

Happy Learning! Your feedback would be appreciated!

ODI 12C Installation & Setup | Windows 10 64-Bit

In this blog we will see how we can setup the ODI 12c on your Windows 10 – 64 Bit PC. First you have to download the latest ODI 12c using below link. I have downloaded the Oracle Data Integrator 12c (12.2.1.4.0) version.

Step 1. Lets begin the installation by  downloading & unzipping the file.

Oracle Data Integrator Download Link

Instal1

Step 2. Prerequisite: Check you java version & set the java variables.

2

java -version
set JAVA_HOME="C:\Program Files\Java\jdk1.8.0_221"
set path=C:\Program Files\Java\jdk1.8.0_221\bin;%path%

Step 3. Start the installation using below commands:

cd C:\Users\shobhit\Downloads\fmw_12.2.1.4.0_odi_Disk1_1of1
java -jar fmw_12.2.1.4.0_odi.jar

1

It will automatically start the installation.

Screenshot 23-02-2020 12_05_44

Installation steps are self explanatory you just have to check all the details like path & click on next.

3

4

Installation is completed. Go to windows start,  type ODI Studio and open the tool.

Step 3. Setup the Master Repository,Work Repository & Logins. Refer below blog to setup the master repository, work repository & logins.

Oracle Data Integrator 11g (ODI) Repositories | Master & Work Repository | Connection & Setup

tool

Repository configuration is done & now you can start building the project using ODI 12c.

If you encounter such issue while opening the ODI 12c Sudio. Please follow below steps.

issue

Delete all the temp files present in the below directory & try opening the ODI studio again.

C:\Users\<<Username>>\AppData\Roaming\odi\system12.2.1.4.0


Thanks!

Happy Learning! Your feedback would be appreciated!

 

Oracle Interval Partitioning

In the previous blog Oracle Range Partition we have explored how we can create range partition in oracle using date as partition key. Also we have seen one example where the range partition key is defined maximum upto ’01-APR-2019′ (Non-inclusive). But we are trying to insert ’20-OCT-2019′ date data, which is not getting mapped to any partition, that’s why ORA-14400 error occurred. See below error report provided by oracle.

SQL Error: ORA-14400: inserted partition key does not map to any partition
14400. 00000 – “inserted partition key does not map to any partition”

Here comes the Interval Partition, which is extension of  Oracle Range Partition in which, interval partitions are automatically created by the database when data is inserted into the partition.

Monthly Partition:

CREATE TABLE T_ACCOUNT
  (
    V_ACCOUNT_NUM  VARCHAR2(100 CHAR),
    V_ACCOUNT_NAME VARCHAR2(200 CHAR),
    ACC_DATE DATE
  )
  PARTITION BY RANGE
  (
    ACC_DATE
  )
  INTERVAL
  (
    NUMTOYMINTERVAL(1, 'MONTH')
  )
  (
    PARTITION P_DEC2018 VALUES LESS THAN (TO_DATE('01-JAN-2019','DD-MON-YYYY'))
  );

After that we have inserted the data of future months, i.e. SEP & OCT for which Partitions were not available. See below insert statements.

INSERT INTO "SAIL_IN_DEV"."T_ACCOUNT" (V_ACCOUNT_NUM, V_ACCOUNT_NAME, ACC_DATE) VALUES ('1234567890', 'ABC', TO_DATE('20-SEP-19', 'DD-MON-RR'));
INSERT INTO "SAIL_IN_DEV"."T_ACCOUNT" (V_ACCOUNT_NUM, V_ACCOUNT_NAME, ACC_DATE) VALUES ('1234567891', 'ABC', TO_DATE('20-OCT-19', 'DD-MON-RR'));
COMMIT;

After committing the data let see the automatic partitions created by Oracle :

2

Notice that a system generated partition named SYS_P601 & SYS_P402 has been created after inserting rows with a partition key greater than the provided partition.

Fetching the data from particular partition : In below example you will able to see only match partition data.

SELECT * FROM T_ACCOUNT partition (SYS_P601);

It would be really painful to look up the system generated partition name every time, if you have to query directly partition. Therefore, oracle provided a syntax to specify a partition is by using the partition for (DATE) clause in the query: See below example:

partition for (to_date(’15-OCT-2019′,’DD-MON-YYYY’));

SELECT * FROM T_ACCOUNT partition  for (to_date('01-SEP-2019','DD-MON-YYYY'));

The following restrictions apply on Interval Partition:

  • Only one partitioning key column allowed (NUMBER or DATE)
  • Minimum one partition must be defined when the table is created.
  • MAXVALUE partition cannot be defined.
  • NULL values are not allowed in the partition column.
  • Can’t be used at the subpartition level.
  • Not supported for index-organized tables.
  • You cannot create a domain index on an interval partitioned table.

 

Daily Partitions Example:

CREATE TABLE T_TRANSACTIONS
  (
    TRANSACTION_KEY NUMBER,
    AMOUNT          NUMBER,
    CUST_ID         NUMBER,
    TRANSACTION_DATE DATE
  )
  PARTITION BY RANGE
  (
    TRANSACTION_DATE
  )
  INTERVAL
  (
    NUMTODSINTERVAL(1, 'day')
  )
  (
    PARTITION P_01012019 VALUES LESS THAN (TO_DATE('01-JAN-2019','DD-MON-YYYY'))
  );

 


Thanks!

Happy Learning! Your feedback would be appreciated!

Oracle Range Partition

Range partitioning is useful when data has some sort of logical range. Example dates or numbers. See below example where monthly range partitions created based on date column in account table.

CREATE TABLE T_ACCOUNT
  (
    V_ACCOUNT_NUM  VARCHAR2(100 CHAR),
    V_ACCOUNT_NAME VARCHAR2(200 CHAR),
    ACC_DATE DATE
  )
  PARTITION BY RANGE
  (
    ACC_DATE
  )
  (
    PARTITION P_DEC2018 VALUES LESS THAN (TO_DATE('01-JAN-2019','DD-MON-YYYY')),
    PARTITION P_JAN2019 VALUES LESS THAN (TO_DATE('01-FEB-2019','DD-MON-YYYY')),
    PARTITION P_FEB2019 VALUES LESS THAN (TO_DATE('01-MAR-2019','DD-MON-YYYY')),
    PARTITION P_MAR2019 VALUES LESS THAN (TO_DATE('01-APR-2019','DD-MON-YYYY'))
  );

 

Fetching the data from particular partition : In below example you will able to see only match partition data.

SELECT * FROM T_ACCOUNT PARTITION(P_MAR2019);

 

Check partitions metadata in user_tab_partitions & subpartitions.

SELECT * FROM USER_TAB_PARTITIONS WHERE TABLE_NAME='T_ACCOUNT';

1

To check the Partition KEY of the table, use below query:

SELECT partition_name,
  column_name,
  high_value,
  partition_position
FROM USER_TAB_PARTITIONS PART ,
  USER_PART_KEY_COLUMNS PART_KEY
WHERE table_name ='T_ACCOUNT'
AND PART.table_name = PART_KEY.name;

In the above example any record having acc_date less than ’01-JAN-2019′ will go to P_DEC2018 partition. See below example:

INSERT
INTO "T_ACCOUNT"
  (
    V_ACCOUNT_NUM,
    V_ACCOUNT_NAME,
    ACC_DATE
  )
  VALUES
  (
    '1234567890',
    'ABC',
    TO_DATE('20-OCT-2010', 'DD-MON-YYYY')
  );
COMMIT;
SELECT * FROM T_ACCOUNT PARTITION (P_DEC2018) ;

In the this example any record having acc_date greater than ’31-MAR-2019′ will error out, as no partition exists for that date.

Example:

INSERT
INTO "T_ACCOUNT"
  (
    V_ACCOUNT_NUM,
    V_ACCOUNT_NAME,
    ACC_DATE
  )
  VALUES
  (
    '1234567890',
    'ABC',
    TO_DATE('20-OCT-2019', 'DD-MON-YYYY')
  );

Error report:
SQL Error: ORA-14400: inserted partition key does not map to any partition
14400. 00000 – “inserted partition key does not map to any partition”
*Cause: An attempt was made to insert a record into, a Range or Composite
Range object, with a concatenated partition key that is beyond
the concatenated partition bound list of the last partition -OR-
An attempt was made to insert a record into a List object with
a partition key that did not match the literal values specified
for any of the partitions.
*Action: Do not insert the key. Or, add a partition capable of accepting
the key, Or add values matching the key to a partition specification

1

Here in this example we can see the partition key is defined maximum upto 2019-04-01 (Non-inclusive) . But we are trying to insert 20-OCT-2019, which is not getting mapped to any partition, that’s why this error occured as date value which we are trying to insert is T_ACCOUNT not satisfying the partition key range criteria.

Solution: To fix it, you may have to add new partitions or add a Maxvalue partition.

ALTER TABLE T_ACCOUNT ADD partition P_OCT2019 VALUES less than (TO_DATE('01-NOV-2019', 'DD-MON-YYYY'));
ALTER TABLE T_ACCOUNT ADD partition P_MAXVALUE VALUES less than (MAXVALUE);

You can also go with : Oracle Interval Partitioning to resolve this issue, but this not preferable.

Drop Partition:

Once historical data is no longer required for business analysis, after retention period the whole partition can be dropped.

DROP PARTITION P_DEC2018;

Updating Key:

When you are trying to update the partition key column for any record.

UPDATE T_ACCOUNT SET ACC_DATE='20-JAN-2019' WHERE v_account_num='1234567890';

Error report:
SQL Error: ORA-14402: updating partition key column would cause a partition change
14402. 00000 – “updating partition key column would cause a partition change”
*Cause: An UPDATE statement attempted to change the value of a partition
key column causing migration of the row to another partition
*Action: Do not attempt to update a partition key column or make sure that
the new partition key is within the range containing the old
partition key.

You can’t move a row with an update: Row movement disabled, which is the default option.

Solution:  We can enable Row movement to partition table T_ACCOUNT. It allows rows to be moved across partitions.

ALTER TABLE T_ACCOUNT ENABLE ROW MOVEMENT;
UPDATE T_ACCOUNT SET ACC_DATE='20-JAN-2019' WHERE v_account_num='1234567890';
COMMIT;

 


Thanks!

Happy Learning! Your feedback would be appreciated!

Model & Datastore| Creating and Reverse-Engineering| Oracle Data Integrator 11g

Model & Datastore| Creating and Reverse-Engineering| Oracle Data Integrator 11g

Folks,

In this blog we will explore:

  • What is Model & how to create a new Model using Oracle or file technology.
  • What is Datastore & how to reverse-engineer a model to populate it with all new & existing datastores.
  • How to create manually data-store of a model.

Model:

Model is a group of tabular data structures stored in a data server defined in the topology.

  • Model is based on a Logical Schema which is defined in the topology. In a given Context, Logical Schema is mapped to a Physical Schema which contains physical data structure: such as tables, files etc.
  • Models only contain Metadata Information, i.e. description of the data structures, they do not contain actual data.

Datastores:

A datastore represents a data structure. It describe data in tabular form i.e rows & columns.  It can be a table, a flat file, XML file or any other data structure which is supported by ODI as technology.

Reverse-engineering:

Reverse-engineering is the process that populates the Model by retrieving metadata from the data server containing the data structures.


Creating and Reverse-Engineering –  Model & Data-store 

Refer this video for Technology – File System:

Refer this video for Technology – Oracle:

In next blog we will explore how to create a ODI Interface to load data from file to Oracle.


Thanks!

Happy Learning! Your feedback would be appreciated!