Oracle Interval Partitioning

In this blog Oracle Range Partition we have explored how we can create range partition in oracle using date as partition key. Also we have seen one example where the range partition key is defined maximum upto ’01-APR-2019′ (Non-inclusive). But we are trying to insert ’20-OCT-2019′ date data, which is not getting mapped to any partition, that’s why ORA-14400 error occurred. See below error report provided by oracle.

SQL Error: ORA-14400: inserted partition key does not map to any partition
14400. 00000 – “inserted partition key does not map to any partition”

Here comes the Interval Partition, which is extension of  Oracle Range Partition in which, interval partitions are automatically created by the database when data is inserted into the partition.

Monthly Partition:

CREATE TABLE T_ACCOUNT
  (
    V_ACCOUNT_NUM  VARCHAR2(100 CHAR),
    V_ACCOUNT_NAME VARCHAR2(200 CHAR),
    ACC_DATE DATE
  )
  PARTITION BY RANGE
  (
    ACC_DATE
  )
  INTERVAL
  (
    NUMTOYMINTERVAL(1, 'MONTH')
  )
  (
    PARTITION P_DEC2018 VALUES LESS THAN (TO_DATE('01-JAN-2019','DD-MON-YYYY'))
  );

After that we have inserted the data of future months, i.e. SEP & OCT for which Partitions were not available. See below insert statements.

INSERT INTO "SAIL_IN_DEV"."T_ACCOUNT" (V_ACCOUNT_NUM, V_ACCOUNT_NAME, ACC_DATE) VALUES ('1234567890', 'ABC', TO_DATE('20-SEP-19', 'DD-MON-RR'));
INSERT INTO "SAIL_IN_DEV"."T_ACCOUNT" (V_ACCOUNT_NUM, V_ACCOUNT_NAME, ACC_DATE) VALUES ('1234567891', 'ABC', TO_DATE('20-OCT-19', 'DD-MON-RR'));
COMMIT;

After committing the data let see the automatic partitions created by Oracle :

2

Notice that a system generated partition named SYS_P601 & SYS_P402 has been created after inserting rows with a partition key greater than the provided partition.

Fetching the data from particular partition : In below example you will able to see only match partition data.

SELECT * FROM T_ACCOUNT partition (SYS_P601);

It would be really painful to look up the system generated partition name every time, if you have to query directly partition. Therefore, oracle provided a syntax to specify a partition is by using the partition for (DATE) clause in the query: See below example:

partition for (to_date(’15-OCT-2019′,’DD-MON-YYYY’));

SELECT * FROM T_ACCOUNT partition  for (to_date('01-SEP-2019','DD-MON-YYYY'));

The following restrictions apply on Interval Partition:

  • Only one partitioning key column allowed (NUMBER or DATE)
  • Minimum one partition must be defined when the table is created.
  • MAXVALUE partition cannot be defined.
  • NULL values are not allowed in the partition column.
  • Can’t be used at the subpartition level.
  • Not supported for index-organized tables.
  • You cannot create a domain index on an interval partitioned table.

 

Daily Partitions Example:

CREATE TABLE T_TRANSACTIONS
  (
    TRANSACTION_KEY NUMBER,
    AMOUNT          NUMBER,
    CUST_ID         NUMBER,
    TRANSACTION_DATE DATE
  )
  PARTITION BY RANGE
  (
    TRANSACTION_DATE
  )
  INTERVAL
  (
    NUMTODSINTERVAL(1, 'day')
  )
  (
    PARTITION P_01012019 VALUES LESS THAN (TO_DATE('01-JAN-2019','DD-MON-YYYY'))
  );

 


Thanks!

Happy Learning! Your feedback would be appreciated!

Oracle Range Partition

Range partitioning is useful when data has some sort of logical range. Example dates or numbers. See below example where monthly range partitions created based on date column in account table.

CREATE TABLE T_ACCOUNT
  (
    V_ACCOUNT_NUM  VARCHAR2(100 CHAR),
    V_ACCOUNT_NAME VARCHAR2(200 CHAR),
    ACC_DATE DATE
  )
  PARTITION BY RANGE
  (
    ACC_DATE
  )
  (
    PARTITION P_DEC2018 VALUES LESS THAN (TO_DATE('01-JAN-2019','DD-MON-YYYY')),
    PARTITION P_JAN2019 VALUES LESS THAN (TO_DATE('01-FEB-2019','DD-MON-YYYY')),
    PARTITION P_FEB2019 VALUES LESS THAN (TO_DATE('01-MAR-2019','DD-MON-YYYY')),
    PARTITION P_MAR2019 VALUES LESS THAN (TO_DATE('01-APR-2019','DD-MON-YYYY'))
  );

 

Fetching the data from particular partition : In below example you will able to see only match partition data.

SELECT * FROM T_ACCOUNT PARTITION(P_MAR2019);

 

Check partitions metadata in user_tab_partitions & subpartitions.

SELECT * FROM USER_TAB_PARTITIONS WHERE TABLE_NAME='T_ACCOUNT';

1

To check the Partition KEY of the table, use below query:

SELECT partition_name,
  column_name,
  high_value,
  partition_position
FROM USER_TAB_PARTITIONS PART ,
  USER_PART_KEY_COLUMNS PART_KEY
WHERE table_name ='T_ACCOUNT'
AND PART.table_name = PART_KEY.name;

In the above example any record having acc_date less than ’01-JAN-2019′ will go to P_DEC2018 partition. See below example:

INSERT
INTO "T_ACCOUNT"
  (
    V_ACCOUNT_NUM,
    V_ACCOUNT_NAME,
    ACC_DATE
  )
  VALUES
  (
    '1234567890',
    'ABC',
    TO_DATE('20-OCT-2010', 'DD-MON-YYYY')
  );
COMMIT;
SELECT * FROM T_ACCOUNT PARTITION (P_DEC2018) ;

In the this example any record having acc_date greater than ’31-MAR-2019′ will error out, as no partition exists for that date.

Example:

INSERT
INTO "T_ACCOUNT"
  (
    V_ACCOUNT_NUM,
    V_ACCOUNT_NAME,
    ACC_DATE
  )
  VALUES
  (
    '1234567890',
    'ABC',
    TO_DATE('20-OCT-2019', 'DD-MON-YYYY')
  );

Error report:
SQL Error: ORA-14400: inserted partition key does not map to any partition
14400. 00000 – “inserted partition key does not map to any partition”
*Cause: An attempt was made to insert a record into, a Range or Composite
Range object, with a concatenated partition key that is beyond
the concatenated partition bound list of the last partition -OR-
An attempt was made to insert a record into a List object with
a partition key that did not match the literal values specified
for any of the partitions.
*Action: Do not insert the key. Or, add a partition capable of accepting
the key, Or add values matching the key to a partition specification

1

Here in this example we can see the partition key is defined maximum upto 2019-04-01 (Non-inclusive) . But we are trying to insert 20-OCT-2019, which is not getting mapped to any partition, that’s why this error occured as date value which we are trying to insert is T_ACCOUNT not satisfying the partition key range criteria.

Solution: To fix it, you may have to add new partitions or add a Maxvalue partition.

ALTER TABLE T_ACCOUNT ADD partition P_OCT2019 VALUES less than (TO_DATE('01-NOV-2019', 'DD-MON-YYYY'));
ALTER TABLE T_ACCOUNT ADD partition P_MAXVALUE VALUES less than (MAXVALUE);

You can also go with : Oracle Interval Partitioning to resolve this issue, but this not preferable.

Drop Partition:

Once historical data is no longer required for business analysis, after retention period the whole partition can be dropped.

DROP PARTITION P_DEC2018;

Updating Key:

When you are trying to update the partition key column for any record.

UPDATE T_ACCOUNT SET ACC_DATE='20-JAN-2019' WHERE v_account_num='1234567890';

Error report:
SQL Error: ORA-14402: updating partition key column would cause a partition change
14402. 00000 – “updating partition key column would cause a partition change”
*Cause: An UPDATE statement attempted to change the value of a partition
key column causing migration of the row to another partition
*Action: Do not attempt to update a partition key column or make sure that
the new partition key is within the range containing the old
partition key.

You can’t move a row with an update: Row movement disabled, which is the default option.

Solution:  We can enable Row movement to partition table T_ACCOUNT. It allows rows to be moved across partitions.

ALTER TABLE T_ACCOUNT ENABLE ROW MOVEMENT;
UPDATE T_ACCOUNT SET ACC_DATE='20-JAN-2019' WHERE v_account_num='1234567890';
COMMIT;

 


Thanks!

Happy Learning! Your feedback would be appreciated!

Model & Datastore| Creating and Reverse-Engineering| Oracle Data Integrator 11g

Model & Datastore| Creating and Reverse-Engineering| Oracle Data Integrator 11g

Folks,

In this blog we will explore:

  • What is Model & how to create a new Model using Oracle or file technology.
  • What is Datastore & how to reverse-engineer a model to populate it with all new & existing datastores.
  • How to create manually data-store of a model.

Model:

Model is a group of tabular data structures stored in a data server defined in the topology.

  • Model is based on a Logical Schema which is defined in the topology. In a given Context, Logical Schema is mapped to a Physical Schema which contains physical data structure: such as tables, files etc.
  • Models only contain Metadata Information, i.e. description of the data structures, they do not contain actual data.

Datastores:

A datastore represents a data structure. It describe data in tabular form i.e rows & columns.  It can be a table, a flat file, XML file or any other data structure which is supported by ODI as technology.

Reverse-engineering:

Reverse-engineering is the process that populates the Model by retrieving metadata from the data server containing the data structures.


Creating and Reverse-Engineering –  Model & Data-store 

Refer this video for Technology – File System:

Refer this video for Technology – Oracle:

In next blog we will explore how to create a ODI Interface to load data from file to Oracle.


Thanks!

Happy Learning! Your feedback would be appreciated!

Dataserver, Physical Schema, Logical Schema & Context | Oracle Data Integrator 11g

Dataserver, Physical Schema, Logical Schema & Context | Oracle Data Integrator 11g

Folks,

In this blog we will explore some areas of ODI Topology Navigaor.

  • We will explore Dataserver, Physical Schema, Logical Schema & Context.
  • Relationship between Physical, Logical Schema & Context.

Regarding setup & configuration of Dataserver, Physical Schema, Logical Schema in ODI using Oracle Technology please refer below video.


Let’s explore Topology Navigator in detail.

topology section
Topology Navigator

In Topology Navigator we have ‘Physical Architecture‘ section where we define the physical connection details of any Instance ( either Oracle DB, SQL Server,  IBM DB2, PostgreSQL, MySQL etc.) or any file system.

What is Dataserver?

A ODI object that defines the physical connection to any database instance or any file system. It basically store the host, username & password related details of an instance.

To define any physical connection in ‘Physical Architecture‘, you have to create a Data Server. Just click on your desired technology in Physical Architecture & click on ‘ New Data Server’.

NEWDATASERVER
Physical Architecture

After that you have to provide instance details. In case of any Database instance provide host, user/password & JDBC details.

Database server new
Data Server: DS_ORACLE_DEV

JDBC Details in case of oracle database as instance:

JDBC details
JDBC Details Data Sever: Localhost

In case of any file. Just provide the folder path from where you pick raw files in interface.

file1
Data Server File Type

JDBC Details in case of file:

file2
JDBC Details File Type

Now after adding the Data-server, next step is to add the ‘Physical Schema‘ in Data-server.

What is Physical Schema?

Physical Schema represents the actual connection to the data source (any database instance or any file system).

ODI Physical Schema corresponds to a pair of schema:

  • Main Schema (Data) – In which ODI look for the source and target objects for the interfaces.
  • Work Schema – Generally used by ODI for staging purpose, here ODI create temporary data objects associated to the sources and targets. Example C$, I$, E$ tables. It is always preferred to create & use separate work schema.

To define any physical schema, go to Data Server you created and then just click on new ‘New Physical Schema’.

PS.png
New Physical Schema

After that you have to provide Schema Name & Work Schema name using drop down. It will show you all the schema present in selected data-server instance.

PS02
Physical Schema Configuration
  • RAW_SCHEMA – schema where my source/target tables lies.
  • ODI_WORK_SCHEMA – Created for ODI staging purpose.
ps03.PNG
Physical Schema – RAW_SCHEMA present in Dataserver DS_ORACLE_DEV

Now after adding the Physical Schema, next step is to create Logical Schema & Context.

What is Logical Schema & Context?

  • Logical schema represents the logical name associated to that source/target objext.
  • One logical schema can be associated with multiple physical schema along with context, i.e. one logical schema is associated with different physical schema using different context.

Create Context:

To add any new context, go to Context Section and then just click on new ‘New Context’.

contextContext 02

Create Logical Schema:

To define any logical schema in ‘Logical Architecture‘ . Just click on your desired technology & click on ‘New Logical Schema’.

LA

LS02
Add New Logical Schema

After that you have to provide the logical schema name & Physical Schema mapping with context.

LS03
Logical Schema

So in Dev Context, Logical Schema LS_RAW_SCHEMA is pointing to Physical Schema DS_ORACLE_DEV.RAW_SCHEMA.

It may be possible that in Test Context, Logical Schema LS_RAW_SCHEMA is pointing to different Physical Schema DS_ORACLE_TEST.RAW_SCHEMA.

lS_Context.PNG

Refer this video clip for better understanding of Logical Schema & Context relationship.


Thanks!

Happy Learning! Your feedback would be appreciated!

 

Oracle Data Integrator | Master & Work Repositories Relationships

Oracle Data Integrator | Master & Work Repositories Relationships

Folks,

In this blog we will explore the relationship between ODI Master & Work Repository.

You can refer this blog  for better understanding of ODI Master & Work Repositories, how they differ from each other & how to setup new repository in ODI Studio using Oracle technology.


Master & Work Repositories Relationships

Master Repository is usually associated with multiple Work Repositories.

Each Work repository can associate with only one Master Repository.

Case 1)  When single Work Repository is attached to the Master Repository.

  • When we have to isolate any environment from the rest of the environments.
  • Usually case of Production/Pre-Production environment.

In below figure we can see that production/pre-prod master repository is isolated from other repositories. Its like one to one relationship. No information sharing.

 

Case 2)  When multiple Work Repository is attached to same Master Repository.

  • See below Figure 2, both work repositories are sharing same master repository.
  • Basically both are sharing the same Topology and Security data.
sharing
Figure 2

 

In real word production environment is isolated from rest of the environments.

practical.PNG


Thanks!

Happy Learning! Your feedback would be appreciated!

Oracle Data Integrator 11g (ODI) Repositories | Master & Work Repository | Connection & Setup

Folks,

In this blog we will learn about Oracle Data Integrator Repositories.

You can refer this blog for ODI 11g Installation & Setup on Windows OS.


Suppose you are just done with the ODI installation on your machine:

Now you are going to use the ODI Studio to develops, operate & administrate some ELT project work. But you need some space/location to start working & to save your all work. Basically you need ODI repository to store the information you are going to develop, configure or used by the ODI.

There are two types of ODI repositories: Master & Work Repository

  • Objects developed, configured, operated/used by the ODI stored in one of these two types of repositories.
  • Repositories are accessed in client/server mode by various components of the ODI architecture.
  • Master Repository is usually associated with multiple Work Repositories.
  • Each Work repository can associate with only one Master Repository.
  • Work repository can either be Development Work Repository or Execution Work repository. (We will discuss this point later in this blog)

repo1

Refer this YouTube video to setup Master & Work Repositories in ODI 11g using Oracle.


Master Repository:

Master Repositories are used to store:

  • ODI Connection Information: JDBC URLs, LDAP information, user names and passwords for source/target connections.
  • ODI Security Information: ODI users names and passwords, ODI users privileges and profiles information, if security is handled by ODI.
  • Version Information: When a new version of an object is created in the ODI, that information is saved in the Master Repository.

Information contained in the Master Repository is maintained with the Topology Navigator and the Security Navigator in ODI Studio.

  • Data contained in master repository is mostly static data and will be going through minimum changes, whenever any topology or security information added/updated.
  • ODI components access the Master Repository, whenever they need the topology and security information data stored in it.

Work Repository:

Work Repositories are used to store:

 

  • Project Information: Folders/Sub-folders, packages, interface, procedure, variables, sequences, functions,  knowledge module etc.
  • Models: Data Store structures and metadata, fields, columns, constraints etc.
  • Load Plans & Scenarios: Load plans information & scenarios.
  • Operations:  Execution details, scheduling information, and logs.

The contents of a Work Repository are managed by using Designer and Operator. They are also accessed by the agent at run time.

When a Work Repository is used only to store execution, operations logs information (typically for production environment), it is called an execution work repository. Designer Navigator is disabled in this execution repository.

Work repository consists of two sub-types: Development Work RepositoryExecution Work repository.

Development Work Repository:

  • Repository which contains all the objects which are used for developing the development project i.e. packages, interface, procedure, variables, sequences, load plans, data stores etc.
  • Designer Navigator & Operator Navigator both are available for Development Work Repository.
  • Usually, used for development projects.

2.PNG

Execution Work Repository:

  • Repository which stores only the scenarios (executable files) & execution information, no source code & designer information available here.
  • Usually, used for production and test environments to make sure that source code will not be modified in a live environment.
  • Operator Navigator is available, but Designer Navigator is restricted for Execution Work Repository.

3

Checkout this blog for understanding of relationship between ODI Master & Work Repository.

Refer this YouTube video to Install ODI 11g on Windows OS.


Thanks!

Happy Learning! Your feedback would be appreciated!