Migration From Oracle to PostgreSQL

Copyright©2016 NTT corp. All Rights Reserved.
Migration from Oracle to PostgreSQL
- The problems and the solutions -
Kazuki Uehara
NTT OSS Center
March 19, 2016
Copyright(c)2016 NTT Corp. All Rights Reserved.

2Copyright©2016 NTT corp. All Rights Reserved.
• Self Introduction
• Introduction of Today's topics
• Problems about Database migration.
• OSS Products that support the Database
migration
• Conclusion
Agenda

• Kazuki Uehara
• From Japan
• Hobby
• Travelling by bicycle
• Taking pictures
• Work
• technical support
• technical consulting
• functional verification and performance evaluation
• Products
• PostgreSQL
• pgpool-II、Slony-I
Who am I?

• Who we are?
• NTT（Nippon Telegraph and Telephone Corporation）
• National flagship carrier in Japan
• What NTT OSS Center is doing？
• Promotes the adoption of OSS by the group companies
• Total support
• support desk, Introduction support, Product maintenance
• R&D
• developing OSS and related tools with the communities
• Deals with about 60 OSS products.
About us
NTT group
subsidiary
about 900 companies
NTT
NTT OSS Center

• The world's most advanced OSS DBMS.
• Continued featuere/performance
improvement by the community.
We involve in PostgreSQL
600 systems
for the last seven years.
Number of adoptions of PostgreSQL
in the group companies
year

PostgreSQL vs Oracle
PostgreSQL 9.5 Oracle 12c
SQL ○ (ISO SQL 2011) ○ (ISO SQL 2011)
stored procedure ○ (PL/pgsql,Java,perl,...) ○ (PL/sql,Java,...)
trigger ○ ○
Online Backup ○ ○
partitionig ○ ○
Replication
○
(Synchronous/Asynchronous)
○
(Synchronous/Asynchronous)
HA Cluster ○ (pacemaker,pgpool-II,...) ○ (VCS, MSCS,...)
clustered systems with
shared disk storage
× ○ (RAC)
License BSD
Named User Plus License
/ Processor Lincense
License Fee ○ free of charge × Compensation
• No big difference.
• You can use PostgreSQL at many systems.

• Focus on Database migration technique from
Oracle to PostgreSQL.
• What is system migration?
• Migration is the process of transferring the system and
data to another environment.
• Three categories of system migration:
• rehost ・・・To replace only platform hardware
• rewrite ・・・To replace OS and programing languages
• rebuild ・・・Remake the entire system
• Database migration is needed in 'rewrite' or 'rebuild'.
Today's topics
1. Problems in the Database migration
2. How you can solve them

• Introduction
• Today's topic
• Problem about Database migration
• Why you need Database migration?
• The items to be considered for the Database migration
• Issues on Database migration
• OSS Products that support the DB migration
• Conclusion

• Your system is obsolete
• Support of HW expired.
• Maintenance costs rise due to aging of equipment.
• Your Database is expensive or poorly operating
• You want to cut the license cost
• You want to improve performance of middleware.
• You can downsize HW.
• You want to use new features.
• You found your commercial Database was over spec
• didn't use the proper functionalities of the commercial
product.
Why you need Database migration?

The items to be considered for the Database
migration
AP Server
1. Logical design of DB
• ER Diagram
2. Physical design of DB
• arrangement of the data
3. Data
• dump, restore
4. Operation procedures
• maintenance tool (backup,
batch)
5. SQL used in application
• the dedicated SQL of
commercial product.
DB Server
DB
（Oracle → PostgreSQL）
Logical
design
Physical
design
application
SQL a
SQL b
…
data
Client /
Web
DataBase
Administrator
Operationnal
procedures
entity
attribute
table
index
columnrelationship
…
…
1 2
5
3
4

1. It will require significant cost to estimate
for the migration of SQL.
• The migration cost go up when we spend too much
time on estimation.
• You can provide the rough estimate in order to reduce
cost.
2. It will require significant cost to modify
incompatible SQL.
• You have to find incompatible SQL from a lot of
sources. In addition, you have to consider for each how
should you modify.
What is the problems?
The risk of losing profits on migration exists.

• This is sample source for Oracle.
• If you use it on PostgreSQL, what should you modify?
• There are three places in this source that require to
modify.
What kind of SQL should you modify?
import java.sql.*;
import java.util.*;
public class test02 {
String sqlString = "DELETE mytbl";
public ResultSet testMethod() throws SQLException {
ResultSet rs = stmt.execute(sqlString);
ResultSet rs2= stmt.executeQuery("SELECT sysdate FROM dual");
return rs;
}
}

modify.
import java.sql.*;
import java.util.*;
return rs;
}
}
DELETE statement requires FROM clause in PostgreSQL.

modify.
import java.sql.*;
import java.util.*;
return rs;
}
}
PostgreSQL doesn't have sysdate.

modify.
import java.sql.*;
import java.util.*;
return rs;
}
}
PostgreSQL doesn't have sysdate.
It doesn't exist DUAL table in PostgreSQL

• Introduction
• Today's topic
• Problem about Database migration
• OSS Products that support the Database
migration
• db_syntax_diff
• orafce
• Case study
• Conclusion

• Two OSS products as solutions.
• db_syntax_diff
• This tool was made by us as migration supporting tool.
• Extracts incompatible SQL from application's source of
Oracle.
• Using this tool, anyone can be easily review source of
application.
• We can review in a relatively short time even for large
scale systems.
• orafce
• This is contrib module for PostgreSQL.
• It is an emulation tool for PostgreSQL to use
compatibility functions and operators with Oracle
RDBMS.
To solve the problem …

An overall outline of db_syntax_diff
ｓｒｃ
db_syntax_diff
XML Output file
• This file is XML format.
Outline of processing
• to do parsing using original parser.
• draw a comparison between the results of
parsing and the contents of dictionary file.
dictionary
file What is dictionary file?
• The list of incompatible 'SQL'.
• You can modify this file.
ｓｒｃｓｒｃ
Input files
• You can input single file or directory.
• C source（ProC）, Java source, JSP source, SQL file

• Install the required package using the yum.
• expand the files got from GitHub.
• set the environment variable.
How to use db_syntax_diff 1/4
# yum install perl perl-XML-SAX.noarch xalan-j2 perl-Parse-Yapp
perl-XML-NamespaceSupport.noarch perl-XML-LibXML.x86_64
$ tar -xvf db_syntax_diff.tar.gz
$ vi ~/.bash_profile
export CLASSPATH=/usr/share/java/xalan-j2.jar:$CLASSPATH
export CLASSPATH=/usr/share/java/xalan-j2-serializer.jar:$CLASSPATH
export PATH=$HOME/db_syntax_diff/src:$PATH
export PERL5LIB=$HOME/db_syntax_diff/src/lib
$ source ~/.bash_profile
※If you use xalan-java 2.7.1 later, you have to set CLASSPATH for serializer.jar.

$ db_syntax_diff.pl --help
db_syntax_diff version 2.0
The SQL analyzer for converting to PostgreSQL.
Usage:db_syntax_diff.pl [-e encodingname][-d definition-file]
[-i inputsourcedir[,suffix1[,suffix2]...] ]
[-o outfile][-f filterword][-m modename]
[-I includedir[,includedir1[,includedir2]...]][-h][-v [loglevel]]
[inputfilename]...
-e encodingname, --encoding=encodingname File encoding. The value which can be
specified is "utf8" and "shiftjis" and "eucjp". [default: eucjp]
-d definition-file, --define=definition-file Definition-file file name.
-i inputsourcedir, --input=inputsourcedir Input-file directry.
-o outfile, --output=outfile Output-file file name. [default: STDOUT]
-f filterword, --filter=filterword Pattern filterword. The value which can be specified is
"oracle8" and "oracle8i". [default: ALL]
-m modename, --mode=modename File type of source file. The value which can be
specified is "c" and "sql" and "cpp" and "java". [default: java]
-I includedir, --Include=includedir Add the directory includedir to the list of directories
to be searched for header files. [default: ./]
-h, --help Print usage and exit.
-v, --verbose Print progress of the practice to STDERR. The value which can be
specified is "1" and "3" and "5" and "7". [default: none]
inputfilename Input-file file name.

• I introduce the result as a simple example.
• target file : sample.java
• Run this command.
$ db_syntax_diff.pl -m java -e utf8 ~/tmp/sample.java -o ~/result.xml
import java.sql.*;
import java.util.*;
return rs;
}
}

• This is a part of result.
<?xml version="1.0" encoding="UTF-8"?>
<REPORT file_number="1" start_time="2016/2/25 19:12:06" finish_time="2016/2/25 19:12:07">
<METADATA>
<PARAMETER>-m java -e utf8 /home/uehara/tmp/sample.java -o /home/uehara/result.xml</PARAMETER>
</METADATA>
<FILE name="/home/uehara/tmp/sample.java" string_item_number="3" report_item_number="4" item_number="7">
<STRING_ITEM line="testMethod:sqlString:7">
<TARGET>DELETE mytbl</TARGET>
</STRING_ITEM>
・・・
<REPORT_ITEM id="SQL-107-008" type="SQL" level="LOW2">
<SOURCE>
<CLASS>test02</CLASS>
<METHOD>testMethod</METHOD>
<LINE>5</LINE>
<COLUMN>7</COLUMN>
<VARIABLE>sqlString</VARIABLE>
</SOURCE>
<STRUCT>!(?:[^¥w¥d_]|¥A)FROM(?:[^¥w¥d_]|¥z)</STRUCT>
<TARGET>DELETE mytbl</TARGET>
<MESSAGE>FROMの省略は未サポートです。</MESSAGE>
</REPORT_ITEM>
<REPORT_ITEM id="SQL-119-706" type="SQL" level="LOW1">
・・・
Which file?
What kind of incompatible SQL?
Simple report is output.
What type? How difficult?
What line/column?
What SQL statement?

dictionary
file
Specifications of the wrapper tool
ｓｒｃ
db_syntax_diff
Output files
• This file is XML format.
• It is aggregated in 4 patterns.
Outline of processing
• This tool operates db_syntax_diff.
• In addition, This tool makes CSV files
from XML file.
ｓｒｃｓｒｃ
Input files
• You can input single file or directory.
csv
XML
db_syntax_diff_wrapper
configuration
file

• result.csv
• file path of target file
• Number of line
• Number of columns
• ID (db_syntax_diff)
The results of wrapper tool
• The classification
• Difficulty of migration
• simple report
• target query
• Middle
• You can't simple migrate.
• High
• migration is very difficult.
• Low1
• It's only necessary to delete or replace.
• Low2
• You can't replace, but migration is easy.
Difficulty

• You can reduce the modification cost of SQL.
• The number of incompatible SQL decrease by using
orafce.
• Installation Instructions (Only 3 steps)
1. You can get a RPM file from PostgreSQL.org.
• http://yum.postgresql.org/9.5/redhat/rhel-7-
x86_64/repoview/orafce95.html
• Latest version(3.2.1) has been published.
2. Install orafce RPM.
3. After, you just run a query "CREATE EXTENSION
orafce".
How to use orafce
# rpm -ivh orafce95-3.2.1-1.rhel7.x86_64.rpm
Updating / installing...
1:orafce95-3.2.1-1.rhel7 ################################# [100%]

A simple example
-bash-4.2$ psql postgres
psql (9.5.1)
Type "help" for help.
postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
ERROR: type "varchar2" does not exist
LINE 1: CREATE TABLE bar(i int,v VARCHAR2(20));
^
postgres=#postgres=# CREATE EXTENSION orafce;
CREATE EXTENSION
postgres=#postgres=# CREATE TABLE bar(i int,v VARCHAR2(20));
CREATE TABLE
postgres=#postgres=# ¥d bar
Table "public.bar"
Column | Type | Modifiers
--------+--------------+-----------
i | integer |
v | varchar2(20) |
postgres=#

A simple example
psql (9.5.1)
^
CREATE EXTENSION
CREATE TABLE
Table "public.bar"
--------+--------------+-----------
i | integer |
v | varchar2(20) |
postgres=#
PostgreSQL doesn't have
data type 'VARCHAR2' .

A simple example
psql (9.5.1)
^
CREATE EXTENSION
CREATE TABLE
Table "public.bar"
--------+--------------+-----------
i | integer |
v | varchar2(20) |
postgres=#
PostgreSQL doesn't have
data type 'VARCHAR2' .
You can use
data type 'VARCHAR2'
in PostgreSQL.

IT infrastructureClient
SW
Web/AP/DB
Job Management/Backup
Other system
Server Enclosure
Case study 1/3
• Billing system
• A managing system of business contract information
• Migration from commercial product to RHEL / Apache /
mod_jk / JBoss EAP / PostgreSQL
intracompany
network
intracompany
network
target number of files Number of lines
SQL/DDL
about 740 files about 250KLJava
Pro*C
Scale of Database migration

• We judged that there are not big problem in
database migration by using db_syntax_diff.
• We extracted 10000 pieces of incompatible SQL.
• But difficulty of alomost the whole incompatible SQL
was Low1 or Low2.
Case study 2/3
# difficulty SQL/DDL Java Pro*C
1 Low1 4,186 1 11
2 Low2 5,361 798 94
3 Middle 20 0 0
4 High 0 0 0
total 9,567 799 105
tool's output
• Low1
• It's only necessary to
delete or replace.
• Low2
• You can't replace, but
migration is easy.
• Middle
• You can't simple migrate.
• High
• migration is very difficult.

• We can reduce the number of incompatible
SQL by orafce.
• In this case, it can reduce to 7000.
• Based on our own experience, PostgreSQL
can use 73% of Oracle's SQL by orafce
Case study 3/3

• The migration of SQL is the most difficult
process in Database migration.
1. It will require significant cost to estimate for the
migration of SQL.
2. It will require significant cost to modify incompatible
SQL.
• These issues can be solved by db_syntax_diff
and orafce.
• Try Database migration, don’t hesitate.
Conclusion
• If you have interest, please help the development of
db_syntax_diff.
• To start with translate manual into English.

Thank you!

Migration From Oracle to PostgreSQL

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Migration From Oracle to PostgreSQL (20)

More from PGConf APAC (17)

Recently uploaded (20)

Migration From Oracle to PostgreSQL