Informix Data Streaming Overview

Informix Data Streaming Overview

• 1.
  Informix Change Data Streaming ExternalTriggers and Change Data Replication Brian Hughes
• 2.
  • Streaming Basics •Informix streaming technologies (overview) • Introducing a new streaming client for Change Data Capture • If you liked Smart Triggers – you might like this Agenda
• 3.
  What is InformixChange Data Streaming? • Streaming database changes(known as change streams or change data capture) is one or more mechanisms where data that is manipulated in a database generates events (triggers) which can be captured by either the database or an external application and processed • Easy to introduce into existing solutions • All applications still connect to the database (no topology changes) • Efficient means to push data to other parts of your infrastructure • Filtering/preprocessing done in the data • Only needed rows/changes can be subscribed to • No database polling
• 4.
  Informix Streaming technologies •Smart Triggers • Asynchronous Triggers • V-II Socket Streaming (unsupported, prototype) • CDC (change data capture)
• 5.
  Smart Triggers • Selectivetriggers on data changes in a table • Leverages Enterprise Replication components • Efficient asynchronous processing • Uses SQL style WHERE clause and projection list to define and filter what is triggered • Setup via the Admin API (JDBC and ODBC have language specific API’s wrapping this) • Uses the same large object TCP pipe as CDC to send the triggered records • Record are in JSON format with metadata • Transaction id, table, user, before/after image for updates • Useful if you are looking for specific events on a table • Scales to many watched tables with a few events occurring on each time • This is not a high throughput system (compared to CDC) or a replication system (not ensured all rows are processed)
• 6.
  Smart Trigger Topology ExternalDatabase MQTT Broker Java Application 2. Log records processed matching table triggered project list filters columns WHERE clause filters rows 3. Smart Trigger API processes JSON stream Processing4. User Application does any filtering/ if needed, then sends data on 1. Insert data into database ST API
• 7.
  Smart Triggers public classSmartTrigger implements IfmxSmartTriggerCallback { private final JsonParser p = new JsonParser(); private static final Logger logger = LoggerFactory.getLogger(SmartTrigger.class); public static void main(String[] args) throws SQLException, InterruptedException { try(IfxSmartTrigger trigger = new IfxSmartTrigger( "jdbc:informix-sqli://localhost:9088/bank:user=informix;password=informix");) { trigger.timeout(5).label("bank_alert"); // optional parameters trigger.addTrigger("account", "informix", "banktest", "SELECT * FROM account WHERE balance < 0", new SmartTrigger()); trigger.run(); //run forever } } @Override public void notify(String jsonString) { JsonObject json = p.parse(jsonString).getAsJsonObject(); if (json.has("ifx_isTimeout")) { logger.debug("[SmartTrigger] Server ping: No balance issues"); } else { json = json.get("rowdata").getAsJsonObject(); logger.debug("{}", json); logger.warn("[SmartTrigger] ALERT on account #{}. Balance ${}", json.get("id").getAsInt(), json.get("balance").getAsInt()); } } }
• 8.
  Asynchronous Triggers • Workssimilar to normal triggers but with a few important differences • Post commit trigger (data will already be committed before this trigger fires) • Can target UDRs as well as other tables • UDR with defined columns • UDR with a JSON column • Built on replication technology • Takes the project and filtering capabilities of replication (same we see with smart triggers) and processes the filtered rows into a stored procedure • Work is performed asynchronously (making it different from the usual pre-commit synchronous triggers of a normal DBMS) • Scales better, does not disrupt normal database processing • Different from normal replication or smart triggers, data triggers can route into a UDR. • Allows on-server processing of data
• 9.
  Asynchronous Post CommitTriggers (cont) • UDRs can be written in 3 languages • SPL • Java • C • Using a UDR you can process the records anyway you like and then you can store them back into a table or if you use C or Java UDRs you can ship the data outside the server • Example: Using a Java UDR with an MQTT library you can send triggered data to an MQTT broker
• 10.
  Async Post CommitTrigger Topology UDR External Database MQTT Broker 1. Insert data into database 2. Rows filtered, Sent to UDR 3. UDR processes records, Can send back to staging tables Can directly send to external application
• 11.
  V-II Socket Streaming •A custom C library that takes triggered data from a table and sends it to an MQTT broker • https://github.com/IBM-IoT/InformixSparkStreaming • Uses the Virtual Index Interface (V-II) feature in Informix • Only sends data to an MQTT broker • Requires manual compilation on target platforms • V-II is a synchronous trigger mechanism • Inserts wait until V-II trigger is completed, which waits on the UDR, which waits on MQTT to transmit • Can lead to slow/blocked inserts • Not Officially supported • Code is free to use and modify, but not part of the Informix product line
• 12.
  Informix Change DataCapture (CDC) • Processes Informix logical logs (log replay) • Data processed is sent to the client application over a large object API found in many drivers • ESQL/C example in $INFORMIXDIR/demo/cdc • Java API new in 4.50.xC2 • Supports basic data types as well as LOBs • Does not support UDT’s • Supports a large set of record types • Begin work, commit/rollback, truncate, insert/update/delete records • Can be used to stream entire database systems for heterogeneous replication • Fully feature replication can require a lot of work or a 3rd party product
• 13.
  CDC Topology External Database MQTTBroker Java Application 1. Insert data into database 2. Log records processed matching table triggered project list filters columns 3. CDC API processes byte stream, converts into usable object 4. User Application does any filtering/ Processing, then sends data on if needed. CDC API
• 14.
  • Based onwork done with Smart Triggers • Aims to simplify and reduce the work involved in using CDC • Working solutions started in under 30 lines of code • Needs JDBC 4.50.JC2 • Takes in a DataSource object as a constructor • Can wrap a JDBC URL in an IfxDataSource object • Must connect to syscdcv1 database • syscdcv1 database must exist • Run the $INFORMIXDIR/etc/syscdcv1.sql script Introducing the new Java API for CDC
• 15.
  CDC Java ExampleCode public class CDCExample { public static void main(String [] args) throws Exception { //You need a datasource for CDC to create connections from //Simple way is to wrap a URL IfxDataSource ds = new IfxDataSource("jdbc:informix-sqli://localhost:20290/syscdcv1:user=informix;password=informix"); //Create the builder with the datasource //Use this to configure what you want to capture IfxCDCEngine.Builder builder = new IfxCDCEngine.Builder(ds); //watch as many tables as you need to //Use format <database>:<owner>.<tablename> //provide the list of columns afterwards builder.watchTable("testdb:informix.cdcTable", "a", "b"); builder.timeout(5); //default 5 second timeout //Build the engine try(IfxCDCEngine engine = builder.build()) { //initialize the engine (creates the connections and begins listening for changes) engine.init(); IfmxStreamRecord record = null; while((record = engine.getRecord()) != null) { //Print out the basic record information System.out.println(record); //If it is an insert/update/delete, print the column data if(record instanceof IfxCDCOperationRecord) { System.out.println(((IfxCDCOperationRecord)record).getData()); } } } } }
• 16.
  Informix Streaming TechnologySummary CDC Smart Trigger Async Trigger V-II Socket Streaming Row Filtering No Yes Yes No Projection list Yes Yes Yes No Target for trigger TCP Pipe/External App TCP Pipe/External App Internal procedure MQTT Broker Commit type Whole transaction + rollback Post commit Post commit Pre commit Data Format Byte Stream JSON SPL declaration or JSON JSON/MQTT Message Data Replay ability Yes No On crash/error No Data Integrity Yes No Yes No Blob/Clob/Text/Byte support Yes No No No UDT Support No Yes Yes No Version Introduced 11.50 12.10.xC9 14.10.xC1 11.50 ? API support ESQL/C Java (4.50.xC2) ESQL/C JDBC ODBC SPL C-UDR Java-UDR None
• 17.
  • Make smarttriggers more resilient to failure • Better unify the APIs for smart triggers and CDC on the Java side • Smart triggers a few years old • Smart triggers could be updated to use the newer style CDC uses • Looking towards configuration based streaming • Configuration like JSON rather than use programming • Combined with REST API to setup triggers pushing to streaming systems ** Subject to change ** Future Direction
• 18.
  Questions?