TAFJ-DB S e t u p

R22
 TAFJ-DB S e t u p

Amendment History:

Revisio
Date Amended Name Description
n

11 1st April 2011 TAFJ team Initial version

12 7st February 2012 H. Aubert R12GA review

13 16th January 2013 R. Vincent R13GA review

14 24th January 2014 R. Vincent R14GA review

15 15th April 2014 H. Aubert R14GA review

24th February
16 R. Vincent Update DBUpdate usage
2015

17 6th March 2015 H. Aubert R15 AMR review

18 24th March 2015 R. Vincent Add IndexTablespace option

19 15th March 2016 M.Kumar R16 AMR Review

20 22nd July 2016 R. Vincent Add –endwithfile option

21 11th August 2016 R. Vincent Added doc on TAFJ_WORK table

22 21st Sep 2016 M.Kumar Remove –Dsinglebytesep for DBUpdate option 3

23 26th July 2017 R. Vincent Add doc for running in SCHEMA mode

24 15th Sep 2017 R. Vincent Add doc for parallel insert

25 6th Oct. 2017 R. Vincent Add doc for Help with Large SCHEMA migrations

Page 2
TAFJ-DB S e t u p

Use Insert/Update only for DbUpdate instead of

26 11th Nov 2017 M. Kumar
Merge

27 22nd March 2019 Riswana F R19 AMR review

Prerequisite - TAFJ_VOC update for UD files on DB

28 23rd May 2019 Sakthi S
Conversion(DBUpdate)

Basheer Update DBUpdate section with convertblobfile and

29 6th Jul 2019
Ahamed dbconversioncheck sql scripts

Add docs for new parallel insert feature, and best

30 22nd Jan 2020 R. Vincent
practice for running large tables.

21 25th March 2020 R. Vincent R20 AMR review

Basheer Introduce script to convert datatype of RECID from

22 13th May 2020
Ahamed varchar to nvarchar

Inclusion of threads parameter for parallel insert

23 6th Aug 2020 Sakthi S
feature

DBUpdate Parallel Thread calculation and Memory

24 7th Sep 2020 Sakthi S
utilization

25 12th April 2021 H. Aubert R21 AMR review

26 1st March 2022 Riswana Add DBImmutability

H. Sanofar
27 23rd March 2022 R22 AMR Review
Nisha

Page 3
TAFJ-DB S e t u p

Copyri g h t
Copyright © Temenos Headquarters SA 2009-2022. All rights reserved.
This document contains proprietary information that is protected by copyright. No part of this document may
be reproduced, transmitted, or made available directly or indirectly to a third party without the express
written agreement of TEMENOS UK Limited. Receipt of this material directly TEMENOS UK Limited
constitutes its express permission to copy. Permission to use or copy this document expressly excludes
modifying it for any purpose, or using it to create a derivative therefrom.

Errat a and Com m e n t s

If you have any comments regarding this manual or wish to report any errors in the
documentation, please document them and send them to the address below:
Technology Department

Temenos Headquarters SA
2 Rue de l’Ecole-de-Chimie,
CH - 1205 Geneva,
Switzerland

Tel SB: +41 (0) 22 708 1150

Fax: +41 (0) 22 708 1160

Please include your name, company, address, and telephone and fax numbers, and email
address if applicable. [email protected]

Page 4
TAFJ-DB S e t u p

Table of Contents
Copyright................................................................................................................................................ 4
Errata and Comments............................................................................................................................ 4
DBImport................................................................................................................................................ 7
Introduction.......................................................................................................................... 7
Setting up TAFJ................................................................................................................... 7
Importing the Database from J4 files...................................................................................8
Importing the Database from another Database..................................................................8
Example of using –startwithfile and –endwithfile..............................................................9
DBImport Work Tables......................................................................................................10
DBImport Parallel Processing............................................................................................10
Error handling....................................................................................................................11
OTHERATTRIBUTES Field in TAFJ_VOC........................................................................11
Importing in SCHEMA mode..............................................................................................11
Help with Large SCHEMA Migrations................................................................................11
TAFJDBImport.default configuration file............................................................................12
For J4 or Database to Database Imports.......................................................................12
Specific to Database to Database Imports.....................................................................14
Specific to Read-Only Database....................................................................................14
DBImport Options:.............................................................................................................15
DB2 Special Considerations on zOS.................................................................................15
DBImport GUI Option:.......................................................................................................18
Syntax:.............................................................................................................................. 18
Screen Shots:....................................................................................................................19
DBUpdate............................................................................................................................................. 23
Overview........................................................................................................................... 23
TAFJ_Work Tables............................................................................................................23
DBUpdate Parallel Processing..........................................................................................23
Requirements.................................................................................................................... 24
Syntax............................................................................................................................... 25
Using convertblobfile sql script for DBUpdate....................................................................26

Page 5
TAFJ-DB S e t u p

Use dbconversioncheck sql script to verify DBUpdate.......................................................27

Only for SQL Server.......................................................................................................... 27
DBImmutabilty...................................................................................................................................... 28
Configuration..................................................................................................................... 28
Conversion........................................................................................................................ 28

Page 6
TAFJ-DB S e t u p

DBI m p o r t
Intr o d u c t i o n
TAFJ comes with its own tool for migrating a jBASE database (J4 files) to a DBMS. The
procedure is straightforward. An important point is to create your database with the correct
options. Once this is done, this is just a matter of answering a few questions and letting the
DBImport tool do the job.

Se t t i n g up TAFJ

!! WARNING !!

Do not run live with the database drivers in <TAFJ_HOME>/dbdrivers! They are there
merely as a convenience. You should use the correct drivers from your database
installation.

In order to run DBImport. you need the specific database drivers you are targeting to go in
<TAFJ_HOME>/ext. Please find these drivers in <TAFJ_HOME>/dbdrivers and copy them
to <TAFJ_HOME>/ext

i.e for oracle 11g database

<TAFJ_HOME>
├───dbdrivers
│ ├───db2_v10.1
│ ├───db2_v10_zOS
│ ├───h2_1.3.161
│ ├───h2_1.3.172
│ ├───oracle-11g
│ ├───oracle-12c
│ └───sqljdbc_1.2
├───ext
└───oracle-11g

Page 7
TAFJ-DB S e t u p

Imp o r t i n g th e Data b a s e fro m J4 file s

Before doing anything, make sure that the standard T24 file structure is available on your
disk.

To run T24, you will need to have the following structure somewhere available.

<xxx>.root
<xxx>.data
<xxx>.dict
<xxx>.run

VOC

The important directories are data, dict and the File VOC. During the Database import, you
will be prompted to specify where the VOC file is. Everything will be related to this file.

We will concentrate on the Oracle import below.

To install Oracle on a Windows ® machine, please check the Installing Oracle guide. You
must then create a database with Oracle using an Oracle tool such as dbca.

IMPORTANT: The database you will use must be created with the AL32UTF8 charset.
There is no way to modify it afterwards (See TAFJ Oracle Install document). Once your
database has been successfully created, DBImport will transfer the data from your T24
structure to Oracle.

Before starting with the import, you will need to create a TableSpace, a Schema (User) and
load some functions. All of these procedures are detailed in the TAFJ Oracle Install
document.

Imp o r t i n g th e Data b a s e fro m ano t h e r Data b a s e

Sometimes you might want to copy one database to another and maintain the state. For
example, an Oracle database can be exported to an H2 database. In the first screen of the
GUI or in console mode, there is a choice of which type of import to do. If “DATABASE” is
chosen, a database to database import/export will occur. Please note that the TAFJ_VOC
file is dropped each time a database to database import is done in the target database, so if
you have a TAFJ_VOC table existing and you want to keep it, please check the checkbox on
the screen. Generally this is necessary when you stop a DB to DB import in the middle and

Page 8
TAFJ-DB S e t u p

quit DBImport. If one wants to begin again or split the file processing (which is generally
faster), one needs to use the –startwithfile and –endwithfile option. Below is an example:

Exa m p l e of usi n g –start w i t h f i l e an d –end w i t h f i l e

On one console enter:

DBImport -GUI -startwithfile % -endwithfile FBNK.A

This takes all VOC items up through FBNK.A…On another console enter:

DBImport -GUI -startwithfile FBNK.A -endwithfile z

This takes all VOC items up through the end. Run through the wizard for the first console,
unchecking create views as this should be done separately after all of the tables have been
imported.

Let the first console get through processing F.STANDARD.SELECTION and F.PGM.FILE
before you run through the wizard of the second console. (These files are overwritten so you
don’t want the processes to clash). Once the first console is through processing
F.STANDARD.SELECTION and F.PGM.FILE, run the wizard for the second console,

Page 9
TAFJ-DB S e t u p

unchecking Create Views, and checking the checkbox “Keep VOC table”. Start the second
process.

DBI m p o r t Work Tabl e s

DBImport/DBUpdate have several tables which report the status of work. TAFJ_WORK
reflects the status of the table’s import (READY, DONE, RUNNING, ERROR, or SKIPPED).
The TAFJ_PROCESS corresponds to the status of an individual thread from thread pool one
and can contain multiple process IDs. A table in ERROR at the TAFJ_WORK level, might
have different statuses in the TAFJ_PROCESS table, but at least one of them should be in
error. Fix errors in the source, and simply re-run DBImport to reprocess errors.

DBI m p o r t Parall e l Pro c e s s i n g

DBImport has two modes of parallel processing. The first threadpool is handling all of the
tables and is controlled by the numThreads parameter. If a table is large, another thread
pool is started to parallel insert the data into the destination database.

Best Practice
Generally, it is best to separate the processing of very large tables (more than 5 million
records) into separate DBImport processes. Only one process will handle a single table if
multiple process are started. In the case of very large tables, a thread pool will be started to
parallel import, and you will want to ensure that the java process has enough memory.
Generally, it is best to give the process 5 Gig of memory.

How Parallel Insert Works

Parallel processing of tables begins when the number of records goes beyond a certain
threshold (100,000). If a table exceeds this threshold, the number of processors is
calculated and multiplied by a factor to arrive at a maximum number of threads for the
process. If several large tables consume all of the available threads, any other table
breaching the threshold will wait until more threads become available. This is why it is best
to process very large tables in a different DBImport process.

Identify the highest volume of the table and no of cores in the server (Before executing
DBImport/DBUpdate).
Default Runtime Properties
coreMultiplicationFactor: 20
recsPerParallelInsertThreadGoal: 10000
Desired Threads = Total Rows/recsPerParallelInsertThreadGoal
Maximum Threads = coreMultiplicationFactor * No of Cores
Note : Desired Threads < Maximum Threads
Example Default Properties
No of cores : 64, Table FBNK_STMT_VAL_ENTRY total rows : 16371506
16371506/10000 < 1280
1637.1 < 1280
Default its not satisfied, suggestion to increase the default value under TAFJ_HOME/bin-
tafj_conf.bat/tafj_conf.sh to match the condition.

Page 10
TAFJ-DB S e t u p

Set/export JAVA_OPTS="$JAVA_OPTS -DrecsPerParallelInsertThreadGoal=80000"

Error han d l i n g
Sometimes you might get an error, which will throw the table into ERROR status in the
TAFJ_WORK table. To reprocess the table, simply re-run DBImport again for that table once
the errors are fixed in the source database or J4 file. If CreateFiles option is enabled, the
table will be dropped and recreated. If not, the table will be truncated regardless of the
ClearFiles option. In the TAFJ_PROCESS table, any previous row in ERROR status will be
updated to INVALID, and a new row will be inserted for each thread in the new process.

OTHERATTRI B U T E S Fiel d in TAFJ_VOC

A field has been added to the TAFJ_VOC so that the VOC can be easily expanded in the
future. The current field definition is:

SCHEMA @AM DATABASE @AM DFFIELDDEF @AM RESERVED

Where SCHEMA is the schema in which the table exists, DATABASE is the database in
which the table exists (if NULL it is the local database), DFFIELDDEF is the field definition for
data event capture (If NULL no data capture event is defined for the table), and RESERVED
is RESERVED for future use.

All “old style” TAFJ_VOC entries where the ORCLEFILENAME contains

database.schema.TABLE_NAME are no longer valid with the OTHERATTRIBUTES field.

Imp o r t i n g in SCHE MA mo d e
Now DBImport supports importing into multiple schemas, however, there are several
prerequisites that are shown below.

1) The schemas must already exist and be created for the different mnemonics (ie.,
BNK, EU1, etc.)

2) For Oracle, running the following before importing :

a. GRANT SELECT ANY TABLE to <<DB User running DBImport>>

b. Failure to run the above will result in JOIN view failures.

3) The java stored functions or C# functions must be created in each schema (such that
the views can access them.)

4) The DB User running DBImport must have all access to tables in each company
schema otherwise views will fail.

To import into different schemas, check the checkbox “Create Schemas”. Dictionaries and
join views (that could span multiple schemas) are created in the default schema (DB User

Page 11
TAFJ-DB S e t u p

running DBImport) where tables and views on single tables are created in the appropriate
schema.

Hel p wit h Larg e SCHE MA Migr a t i o n s

With large schema migrations, DBImport offers a couple of commands to help. For all
databases the process is an OFFLINE process with the following steps:

1) Export from source schema, then import to target schema or run an ALTER
command (SQLServer) (NOTE: Alternatively with Oracle, very large files that are
partitioned can be moved with faster ALTER commands. See Oracle documentation
regarding this at: https://oracle-base.com/articles/misc/partitioning-an-existing-table-
using-exchange-partition). If your database is over 1 terabyte, one should consider
partitioning large tables.

2) Update TAFJ_VOC to reflect the new schema

3) Re-run view creation for the entire database.

Note: For both commands below, you must have the Source Database connection set in the
DBImport config file.

To create par files (Oracle) or ALTER scripts for each schema, run:

DBImport -createSchemaMigrationScripts database=<< one of ORACLE/MSSQL/DB2>>

inputFile=<<Input File>> outputDir=<<Output Directory>>

where <<Input File>> is the full path and filename to the file generated by T24 for schema
definition and <<Output Directory>> is the directory where the par files will be generated.

Once the export/import has been done, it is time to update the TAFJ_VOC. To do this run:

DBImport -updateVocForSchemas schemas=<<SCHEMA1,SCHEMA2,etc>>

This command updates the OTHERATTRIBUTES field of TAFJ_VOC based on what tables
are actually in the schemas. As well, it drops views and tables from the old schema.
Information is logged to the DBImport log at the INFO level.

TAFJDBI m p o r t . d e f a u l t con f i g u r a t i o n file

Page 12
TAFJ-DB S e t u p

Once your database, tablespace, and schema are created, and your TAFJ stored functions
are in place (they are needed for view creation), you will optionally need to edit the
TAFJDBImport.default configuration file to reflect what you just created. Values from this file
will be used as defaults when the DBImport process runs. The TAFJDBImport.default file is
in the directory conf/ of the tafj installation directory. The below lists the properties in this file.

For J4 or Data b a s e to Data b a s e Imp o r t s

“importType” specifies which type of import will be performed. This refers to the source of
the import. The value is either J4 (from j4 files) or DATABASE (another RDBMS database is
the source of the import). The default is J4, and in most cases J4 will be faster.

“URL” specifies “how” to connect. It is composed of a hostname (or IP address), a port and
a database name (what you specified in the step 3 of the Database configuration assistant).
If your database is installed locally, and the name you gave was tafjdb for an oracle
database, then the value should be: jdbc:oracle:thin:@localhost:1521:tafjdb

“Driver” tells what kind of database it is. Leave it to the default value unless you want
another database other than Oracle. The default is oracle.jdbc.driver.OracleDriver

“User” is the name of the database user you specified when you created your database.

“Password” is the database password you specified when you created your database.

Other parameters include:

“Layout” is Table types you want to create (XML, CLOB,STRING,MULTIPLE, TEXT, JSON).

TEXT is RECID VARCHAR, XMLRECORD VARCHAR

XML is RECID VARCHAR, XMLRECORD XMLTYPE

JSON is RECID VARCHAR, XMLRECORD BLOB (Only for Oracle databases)

CLOB IS RECID VARCHAR, XMLRECORD CLOB

STRING is RECID VARCHAR, STRRECORD VARCHAR, XMLRECORD CLOB (ie if

XMLRECORD > LENGTH of VARCHAR it is stored in the XMLRECORD CLOB
column)

MULTIPLE is creating relational columns with relational types. Multi-Value columns

stored with their multi-value markers

“File” = Files you want to import (comma separated list). Default is ALL

“ExcludeFile” = Files you don’t want to import (comma separated list). Default is NONE

“ExcludeFileButCreateFile” = Files you don’t want to insert but you do want to create
(comma separated list). Default is NONE

Page 13
TAFJ-DB S e t u p

“ExcludeFileNoXML” = Files you don’t want to import as XML files (comma separated list,
ignored if XML not specified as import type). This can be used with the % wildcard, ie.
F.JOB.LIST% would apply to F.JOB.LIST .1, F.JOB.LIST.2, F.JOB.LIST .3, etc. Default is
NONE

“UserDirectories” = Directory where to create UD type files. Must match what is in

tafj.properties key temn.tafj.runtime.directory.current

“CreateTables” = Create the tables?

“InsertRecords” = Insert the records?

“ClearTables” = When using the ClearTables option which is in the GUI and
console mode (if CreateTables is No and Insertrecords is Yes), the schema must already
exist AND the TAFJ_VOC must be created and filled with data already. Otherwise, do not
use it. It is really to be used to overwrite an existing database.

“CreateViews” = Create the views? To run T24 views are needed and are built
off the tables that are imported.

“CreateSchemas” = Create tables and views in separate schemas? Several prerequisites

must be met to run in this mode. See above.

“CommitFrequency” = How many rows to insert before commit is called. Defaults to

10,000 which is recommended.

“VocFile” = File that tells where J4 files are on disk. Data will be imported from
these J4 files.

“FileEncoding” = UTF-8 or ISO-8859-1

“CreateUDasTables” = User directories get converted to database tables during the

import

“InsertUDRecords” = Insert existing entries stored in directory structure into the database.

“UDFileEncoding” = Should match FileEncoding

“OracleUseXMLBinary” = Should oracle create XMLBinary datatype when Layout is

XML?

“IndexTableSpace” = If you have a separate tablespace for primary keys and indexes,
specify the name of it here. For MSSQL, this is the filegroup name that is not the primary.

There are other zOS DB2 specific parameters not covered. Please speak with a DBA
regarding these.

Page 14
TAFJ-DB S e t u p

Sp e c i f i c to Data b a s e to Data b a s e Imp or t s

“URLSourceDB” specifies “how” to connect TO THE SOURCE DATABASE. It is composed
of a hostname (or IP address), a port and a database name (what you specified in the step 3
of the Database configuration assistant). If your database is Oracle and installed locally, and
the name you gave was tafjdb, then the value should be,
jdbc:oracle:thin:@localhost:1521:tafjdb

“DriverSourceDB” tells what kind of SOURCE DATABASE driver it is. Use the default if it is
an Oracle database. The default value is oracle.jdbc.driver.OracleDriver

“UserSourceDB” is the name of the database user FROM THE SOURCE DATABASE you
specify when you connect to your source database.

“PasswordSourceDB” is the database password you use FROM THE SOURCE

DATABASE.

Sp e c i f i c to Read - Only Data b a s e

When the T24 read-only feature is enabled and the TAFJ_VOC has been updated to specify
a read-only database, the DBImport command can be used to set up the views in this read-
only database with a database to database import specifying views only. The below
properties are now deprecated in TAFJDBImport.default. Please see the TAFJ-Read Only
documentation.

“UrlRO” is the connection url to the read-only database

“DriverRO” is the database driver and must be identical to the transactional database

“UserRO” is the name of the database user

“PasswordRO” is the database password

DBI m p o r t Opti o n s :
DBImport Options:

-d Run DBImport in daemon mode. In this mode, DBImport will run in the background using
the TAFJDBImport.default file to receive its input. (Example usage: $nohup DBImport –d &)

–makeDDL DBImport has the ability to create a text file with the DDL (create table and other
database statements) without executing the commands on the database. In text mode,
specify the –makeDDL option when running DBImport (i.e. DBImport –makeDDL). The GUI
mode has this as a question in the wizard and hence does not have to be specified.

-startwithfile DBImport can begin creating files from a particular starting point (i.e.
useful if your DBImport program crashed for some reason and you do not want to start from

Page 15
TAFJ-DB S e t u p

scratch). PLEASE NOTE THAT THE FILENAME AFTER THE OPTION IS THE VOC (J4
FILENAME) AND NOT THE ORACLE FILENAME

-endwithfile DBImport can begin creating files from a particular starting file and stop
with a particular ending file (i.e. useful if your DBImport program crashed for some reason
and you don’t want to start from scratch as well as launching multiple DBImport processes if
the database isn’t doing to much IO and isn’t too busy). PLEASE NOTE THAT THE
FILENAME AFTER THE OPTION IS THE VOC (J4 FILENAME) AND NOT THE ORACLE
FILENAME

DB 2 Sp e c i a l Con s i d e r a t i o n s on zOS

Note: For DB2 zOS there are special parameters:

Special LOB creation DB2 can create the LOB tables automatically for you. This is the
default. If you do not want this and instead want them to be created by the DBImport
program, say « yes ». If you are running in DDL mode, the create LOB table statements will
appear in the DDL script.

STOGROUP is a parameter used in the CREATE TABLESPACE command for DB2.

Check with your DBA what this should be.

Once these modification are done, save the file and run the script DBImport(.bat) in the
<TAFJ_HOME/bin directory or DBImport –GUI for the GUI version of the tools.

The following section shows the typical dialog you will have for a full import

_____________________DBImport V4.0_____________________

Welcome to the Database Import program

This tool will import a T24 J4/RDBMS Database
in a other RDBMS using JDBC. It will also create the
necessary views and populate the database VOC
For J4 Files, only H4 and HR files are supported.
_______________________________________________________
Press <ENTER> to Continue ...

Please specify the type of the importation :

_________________________________________________________________________________
_______

Choose the type of import (source of import) ? J4/DATABASE (Default :

'J4')

Page 16
TAFJ-DB S e t u p

Please specify the actions to do :

_________________________________________________________________________________
_______

Do you want to create tables ? Y/N (Default : 'Yes')

Do you want to insert records ? Y/N (Default : 'Yes')

Do you want to insert UD records ? Y/N (Default : 'Yes')

Do you want to create views ? Y/N (Default : 'Yes')

Please specify the parameters for the source J4 :

_________________________________________________________________________________
_______
Please enter the VOC file path : (Default : '<full_path_to_VOC_File>')
D:\Temenos\BNK\bnk\bnk.run\VOC
________________________________________________________
J4 File Encoding :
1) UTF-8
2) ISO-8859-1
________________________________________________________
Please choose (1-2)(Default : '1')

________________________________________________________
UD File Encoding to read:
1) UTF-8
2) ISO-8859-1
3) Cp1252
4) IBM-1047
________________________________________________________
Please choose (1-4)(Default : '2')

Please specify the parameters for the target RDBMS :

_________________________________________________________________________________
_______

________________________________________________________
Available databases :
1) Oracle
2) MS-SQL Server
3) DB2
4) H2
________________________________________________________
Please choose the database (1-4)(Default : '1')
4
Please specify the connection URL, eg
(jdbc:h2:tcp://<host_or_ip>/<db_name>)
jdbc:h2:tcp://localhost/T24
User Id :
mbtafj
Password :
mbtafj
Connected to H2 with specified parameters.
____________________________________________________
Data DestinationTable Layout :
1) TEXT -> VARCHAR / VARCHAR

Page 17
TAFJ-DB S e t u p

2) STRING -> VARCHAR / VARCHAR / CLOB

________________________________________________________
Please choose the table type (1-2)(Default : '1')

Please enter where to create Tables TYPE=UD :

D:\data\temenos\src\Enterprise\dev\tafj\ReleaseDir\UD
Do you want to create UD data (outsite .run) as tables ? Y/N (Default :
'Yes')

Please specify the options to process :

_________________________________________________________________________________
________

Please enter the Tables to process (can be seperated by coma ',') :

(Default : 'All')

Please enter the Tables file to exclude (can be seperated by coma ',') :

Please enter the J4 files to not insert records for (can be seperated by
coma ',') :

Please enter the J4 files to exclude from XML type table (can be
seperated by coma ',') : (Default : 'F.JOB.LIST.*')

__________________________________________________________________________
Target Database : H2
URL : jdbc:h2:tcp://localhost/T24
Driver : org.h2.Driver
User ID : mbtafj
Password : mbtafj
Layout : TEXT -> VARCHAR / VARCHAR
Path UD : D:\data\temenos\src\Enterprise\dev\tafj\ReleaseDir\UD
UD Data as Table : Yes
mode zOS : No

Source J4
Voc File : D:\Temenos\BNK\bnk\bnk.run\VOC
J4 Encoding : UTF-8
UD Encoding : ISO-8859-1

To Do
Create tables : Yes
Inserting Records : Yes
Creating Views : Yes

Tables to
process : All
exclude :
not insert :
exclude from XML: F.JOB.LIST.*
__________________________________________________________________________
Press <ENTER> to Continue ...

The full import will take about 1 ½ hours, depending on your hardware.

Page 18
TAFJ-DB S e t u p

DBI m p o r t GUI Optio n:

Previously we saw the DBImport in normal text mode. We have another option for this
DBImport in GUI mode, which will get user inputs by few wizard slides and shows the
transaction in progress bar.

Syn t a x:
Run the DBImport.bat file under TAFJ_HOME/bin by using the below arguments:

DBImport.bat –cf tafj.properties –GUI

Here ‘-GUI’ is a flag to run DBImport in GUI mode.

Scr e e n Sh o t s :

For J4 File as Source

Page 19
TAFJ-DB S e t u p

For Database as Source

Page 20
TAFJ-DB S e t u p

Page 21
TAFJ-DB S e t u p

Note: DBImport will create user directories for those entries in the VOC table that are
directories (if “Import UD Table in RDBMS (as a Table)” is unchecked). The directory
specified must match that of the temn.tafj.runtime.directory.current property in your TAFJ
properties file. If it does not, these directories will not be opened correctly in the T24 code
base.

Page 22
TAFJ-DB S e t u p

DBU p d a t e
Overvi e w
The process to update a TAF-C format database is to load stored functions and create a
TAFJ_VOC table and views matching specific tables. This is also used to create read-only
database views in a separate instance from what is defined in a TAFC voc file.

TAFJ_Work Tabl e s
DBUpdate have several tables, which report the status of work. TAFJ_WORK reflects the
status of the table’s import (READY, DONE, RUNNING, ERROR, or SKIPPED). The
TAFJ_PROCESS corresponds to the status of an individual thread from thread pool one and
can contain multiple process IDs. A table in ERROR at the TAFJ_WORK level might have
different statuses in the TAFJ_PROCESS table, but at least one of them should be in error.
Fix errors in the source, and simply re-run DBUpdate to reprocess errors.

Page 23
TAFJ-DB S e t u p

DBU p d a t e Parall e l Proc e s s i n g

DBUpdate has two modes of parallel processing. The first threadpool is handling all of the
tables and is controlled by the numThreads parameter. If a table is large, another thread
pool is started to parallel insert the data into the destination database.

Best Practice
Generally, it is best to separate the processing of very large tables (more than 5 million
records) into separate DBUpdate processes. Only one process will handle a single table if
multiple process are started. In the case of very large tables, a thread pool will be started to
parallel import, and you will want to ensure that the java process has enough memory.
Generally, it is best to give the process 5 Gig of memory.

How Parallel Insert Works

Parallel processing of tables begins when the number of records goes beyond a certain
threshold (100,000). If a table exceeds this threshold, the number of processors is
calculated and multiplied by a factor to arrive at a maximum number of threads for the
process. If several large tables consume all of the available threads, any other table
breaching the threshold will wait until more threads become available. This is why it is best
to process very large tables in a different DBUpdate process.

Identify the highest volume of the table and no of cores in the server (Before executing
DBUpdate).
Default Runtime Properties
coreMultiplicationFactor: 20
recsPerParallelInsertThreadGoal: 10000
Desired Threads = Total Rows/recsPerParallelInsertThreadGoal
Maximum Threads = coreMultiplicationFactor * No of Cores
Note : Desired Threads < Maximum Threads

Example Default Properties

No of cores : 64, Table FBNK_STMT_VAL_ENTRY total rows : 16371506
16371506/10000 < 1280
1637.1 < 1280
Default its not satisfied, suggestion to increase the default value under TAFJ_HOME/bin-
tafj_conf.bat/tafj_conf.sh to match the condition and increase the memory accordingly 20G
Set/export "JAVA_OPTS=-Xss200M -Xms256M –Xmx20480M -Dfile.encoding=UTF-
8 -DrecsPerParallelInsertThreadGoal=80000 -DcoreMultiplicationFactor=5"
16371506/80000 < 5 * 64
204.64 < 320

Req u ir e m e n t s
First load stored functions. Please see the TAFJ-Oracle documentation sections for how to
load java stored functions

Page 24
TAFJ-DB S e t u p

Warning !!!!!

UD file VOC 1st value should be updated as DIRECTORY in TAFc database for proper
TAFJ_VOC update on DB Conversion.

For that VOC update please compile and execute the utility (SANITY.CHECK.UD.FILE)
which is present in <<TAFJ_HOME>>\dbscripts\basic on TAFc environment before taking
the database backup for DB conversion(DBUpdate)

For DBUpdate option 1 & 2

The option 1 (Update a TAFC database) and option 2 (Update an R13 release)
should make use of the files javaloadsingle and javafunctions_single_oracle.sql as TAFC
uses single byte seperators.

This has to be done before a DBUpdate as TAFJ views are dependent on them.

Next, it is necessary to put TAFJ in single byte mode. To this:

-> Change TAFJ_HOME\bin\tafj_conf

-> Uncomment: "JAVA_OPTS=%JAVA_OPTS% -Dsinglebytesep"

For DBUpdate option 3

The option 3 (Update a TAFC database and change data to double byte) should
make use of the files javaload and javafunctions_oracle.sql which will load the necessary
double byte separators.

This has to be done before a DBUpdate as TAFJ views are dependent on them.

Note : For option 3 there is no need to put TAFJ in single byte mode.

Please make sure to update the below properties in the properties files used

Always make sure to switch off merge as it causes connection stale with oracle DB
especially.

temn.tafj.jdbc.write.use.merge = false

temn.tafj.jdbc.write.use.merge.no.xml = false

Page 25
TAFJ-DB S e t u p

# Dict Tables are CLOB or BLOB ?

# When running on a Database having been create by the jBase Export tool or DBUpdate, there are BLOB

# When the Database has been created with DBImport, there are CLOB

# Default is CLOB

temn.tafj.jdbc.default.dict.type = BLOB

After this is done, run DBUpdate from <TAFJ_HOME>/bin.

# Define if the database has been converted in UTF8 or not.

# This property is only valid for the TAFC database RDBMS

temn.tafj.jdbc.jbase.utf8 = true

After this is done, run DBUpdate from <TAFJ_HOME>/bin.

Syn t a x
The DBUpdate syntax is the following:

DBUpdate –d –importType [number] -skipTAFJVOCcreation

-d --> run in daemon mode

-importType --> A number between 1-3:

1) Update a TAFC database

2) Update an R13 release

3) Update a TAFC database and change data to double byte

-skipTAFJVOCcreation --> used with Import type 3) above: to skip the VOC creation if
already done

Page 26
TAFJ-DB S e t u p

___________________DBUpdate V3.0___________________

Welcome in the Database Update program This tools will Update an T24 Oracle
database to
Make it compliant with TAFJ.
____________________________________________________
Press <ENTER> to Continue ...

________________________________________________________
Available databases :
1) Oracle
2) MS-SQL Server
3) DB2
4) H2
________________________________________________________
Please choose the database (1-6)(Default : '1')

Answer step by step to all question and let it work....

At the end, backup the database.
The database is ready to work with TAFJ.

Usi n g conv e r t b l o b f i l e sql scri p t for DBU p d a t e

We may face a situation where the table has huge amount of records (say, more than 10
million records). In this case, DBUpdate may take more time to update the table. To
overcome this, optionally, update could be run with convertblobfile.sql script present under
<TAFJ_Home>/dbscripts/<db>/

This script performs the operation of updating the single byte separators to double byte
separators. Load the script in database as like any other SQL script. The tables that are
converted with this script should be skipped from the traditional DBUpdate, to avoid
redundancy. To skip the tables during DBUpdate, in the properties file
TAFJDBImport.default, include the table names under the property ExcludeFile. The value
should be T24 file name, not the name of the table in database.

The script takes the table name, as present in database, as input. The script runs in two
phase – update the single byte separators to double byte separators and update
IS_RIGHTJUSTIFIED field, in TAFJ_VOC, of the table.

Advisable to have the entry for the table in TAFJ_VOC, as the script will update the
IS_RIGHTJUSTIFIED field in TAFJ_VOC. View should be (re)created for the table after the
script execution.

Note: This script does not involve TAFJ runtime and executed directly on the DB.

Page 27
TAFJ-DB S e t u p

Us e dbc o n v e r s i o n c h e c k sql scri p t to verify DBU p d a t e

Once DBUpdate is completed, the byte conversion process can be verified by running the sql
script dbconversioncheck.sql present under <TAFJ_Home>/dbscripts/<db>

This script creates a table BLOB_CONVERSION_FAIL in the target database, and runs
against all the T24 tables by checking each record for a single byte separator. When a single
byte separator is found, then the script will update the table BLOB_CONVERSION_FAIL with
the RECID and the table name for which the check failed.

If there are any entries found in the table BLOB_CONVERSION_FAIL, reach TAFJ support
for the analysis on the cause of conversion failure.

As with the earlier case, this script too does not involve TAFJ runtime and is executed
directly on the database.

!! NOTE !!

Only for SQL Serv e r

Once DBUpdate is run, mandatorily run the script dbrecidtypeconversion.sql available under
<TAFJHome>/dbscripts/ms-sql.

This script is to change the datatype of the column RECID in all tables from varchar to
nvarchar, thus by increasing the performance during SELECT

DBI m m u t a b i l t y

A new feature to verify a record that is read/write is not tampered. Create a file with
PROTECT=YES qualifier to enable this feature.

CREATE-FILE FILENAME TYPE=<type> PROTECT=YES

If it is set to YES,

an additional column called SIGNATURE is created in its table

ISPROTECTED column in TAFJ_VOC is set to Y.

Signature of each record is stored in the SIGNATURE column as below

SELECT RECID,ISPROTECTED FROM TAFJ_VOC tv

Page 28
TAFJ-DB S e t u p

RECID ISPROTECTED

F.TEST.PROTECTED Y

F.TEST.NOTPROTECTE N
D

When we read/write a record into a file, it validates against the signature of the record.

Confi g u r a t i o n
Set temn.db.immutablity.current.confid=<ConfigurationId> in Keystore.properties or as a
Java Property and also configure the other existing properties as required. Refer to TAFJ-
MessageIntegrity.doc for Keystore creation and configuration.

Conv e r s i o n
During DBImport, choose yes to Protect Tables Option to enable this feature for the
all/required files.

Using Option 4 in DBUpdate, enable this feature for the all/required files.

Page 29