![12 | Kafka Connect /Debezium - Stream MySQL events to Kafka
Debezium Connector – Add Connector to Kafka
Connect
For more configuration : http://debezium.io/docs/connectors/mysql/
More REST Endpoints : https://docs.confluent.io/current/connect/managing.html#using-the-rest-interface
List Available Connector plugins
$ curl -s http://kafka-connect:8083/connector-plugins
[
{
"class": "io.confluent.connect.jdbc.JdbcSinkConnector"
},
{
"class": "io.confluent.connect.jdbc.JdbcSourceConnector"
},
{
"class": "io.debezium.connector.mysql.MySqlConnector"
},
{
"class": "org.apache.kafka.connect.file.FileStreamSinkConnector"
},
{
"class": "org.apache.kafka.connect.file.FileStreamSourceConnector"
}
]
Add connector
$ curl -s -X POST -H "Content-Type: application/json" --data @connector-config.json http://kafka-connect:8083/conn
Remove connector
$ curl -X DELETE -H "Content-Type: application/json” http://kafka-connect:8083/connectors](https://image.slidesharecdn.com/kafkaconnect-debezium-180708202730/85/Kafka-Connect-debezium-12-320.jpg)
Kafka Connect - debezium
- 1. 1 | Kafka Connect /Debezium - Stream MySQL events to Kafka Kafka Connect - Debezium Stream MySQL events to Kafka
- 2. 2 | Kafka Connect /Debezium - Stream MySQL events to Kafka About me Kasun Don Software Engineer - London AWIN AG | Eichhornstraße 3 | 10785 Berlin Telephone +49 (0)30 5096910 | [email protected] | www.awin.com • Automation & DevOps enthusiastic • Hands on Big Data Engineering • Open Source Contributor
- 3. 3 | Kafka Connect /Debezium - Stream MySQL events to Kafka Why Streaming MySQL events (CDC) ? • Integrations with Legacy Applications Avoid dual writes when Integrating with legacy systems. • Smart Cache Invalidation Automatically invalidate entries in a cache as soon as the record(s) for entries change or are removed. • Monitoring Data Changes Immediately react to data changes committed by application/user. • Data Warehousing Atomic operation synchronizations for ETL-type solutions. • Event Sourcing (CQRS) Totally ordered collection of events to asynchronously update the read-only views while writes can be recorded as normal.
- 4. 4 | Kafka Connect /Debezium - Stream MySQL events to Kafka Apache Kafka Kafka is a distributed publish-subscribe messaging system that is designed to be fast, scalable, and durable. Producer Consumer Consumer Consumer Producer Producer Kafka
- 5. 5 | Kafka Connect /Debezium - Stream MySQL events to Kafka Kafka Connect Connectors – A logical process responsible for managing the copying of data between Kafka and another system. There are two types of connectors, • Source Connectors import data from another system • Sink Connectors export data from Kafka Workers – Unit of work that schedules connectors and tasks in a process. There are two main type of workers: standalone and distributed Tasks - Unit of process that handles assigned set of work load by connectors. Connector configuration allows set to maximum number of tasks can be run by a connector.
- 6. 6 | Kafka Connect /Debezium - Stream MySQL events to Kafka Kafka Connect - Overview Data Source Data Sink KafkaConnect KAFKA KafkaConnect
- 7. 7 | Kafka Connect /Debezium - Stream MySQL events to Kafka Kafka Connect – Configuration Common Connector Configuration • name - Unique name for the connector. Attempting to register again with the same name will fail. • connector.class - The Java class for the connector • tasks.max - The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this level of parallelism. Please note that connector configuration might vary, see specific connector documentation for more information. Distributed Mode - Worker Configuration bootstrap.servers - A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. group.id - A unique string that identifies the Connect cluster group this worker belongs to. config.storage.topic - The topic to store connector and task configuration data in. This must be the same for all workers with the same group.id. offset.storage.topic - The topic to store offset data for connectors in. This must be the same for all workers with the same group.id status.storage.topic - The name of the topic where connector and task configuration status updates are stored. For more distributed mode worker configuration : http://docs.confluent.io/current/connect/userguide.html#configuring-workers
- 8. 8 | Kafka Connect /Debezium - Stream MySQL events to Kafka Kafka Connect – Running A Instance It is recommended to run Kafka Connect on containerized environments such as Kubernetes, Mesos, Docker Swarm, or YARN. Kafka Connect distributed mode exposes port 8083 by default to serve management REST interface. Kafka Connect does not automatically handle restarting or scaling workers which means your existing clustering solutions can continue to be used transparently. – Confluent.io $ docker run -d > --name=kafka-connect > --net=host > -e CONNECT_BOOTSTRAP_SERVERS="kafka-broker:9092" > -e CONNECT_GROUP_ID="group_1" > -e CONNECT_CONFIG_STORAGE_TOPIC="kafka-connect-config" > -e CONNECT_OFFSET_STORAGE_TOPIC="kafka-connect-offset" > -e CONNECT_STATUS_STORAGE_TOPIC="kafka-connect-status" > -e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" > -e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" > -e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" > -e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" > -e CONNECT_LOG4J_LOGGERS="io.debezium.connector.mysql=INFO" > -v /opt/kafka-connect/jars:/etc/kafka-connect/jars > --restart always > confluentinc/cp-kafka-connect:3.3.0
- 9. 9 | Kafka Connect /Debezium - Stream MySQL events to Kafka Debezium Connector What is Debezium ? Debezium is an open source distributed platform for change data capture using MySQL row-level binary logs. Debezium built on top of Kafka Connect API Framework to support fault tolerance and high availability using Apache Kafka eco system. Debezium records in a transaction log all row-level changes committed to each database table. Supported Databases Debezium currently able to support following list of database software. • MySQL • MongoDB • PostgreSQL For more Information : http://debezium.io/docs/connectors/
- 10. 10 | Kafka Connect /Debezium - Stream MySQL events to Kafka Debezium Connector – MySQL Configuration Enable binary logs server-id = 1000001 log_bin = mysql-bin binlog_format = row binlog_row_image = full expire_logs_days = 5 or Enable GTIDs gtid_mode = on enforce_gtid_consistency = on MySQL user with sufficient privileges GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium' IDENTIFIED BY password'; Supported MySQL topologies • MySQL standalone • MySQL master and slave • Highly Available MySQL clusters • Multi-Master MySQL • Hosted MySQL eg: Amazon RDS and Amazon Aurora
- 11. 11 | Kafka Connect /Debezium - Stream MySQL events to Kafka Debezium Connector – MySQL Connector Configuration Example Configuration { "name": "example-connector", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "127.0.0.1", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "mysql-example", "database.whitelist": "db1", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.mysql-example" } } For more configuration : http://debezium.io/docs/connectors/mysql/
- 12. 12 | Kafka Connect /Debezium - Stream MySQL events to Kafka Debezium Connector – Add Connector to Kafka Connect For more configuration : http://debezium.io/docs/connectors/mysql/ More REST Endpoints : https://docs.confluent.io/current/connect/managing.html#using-the-rest-interface List Available Connector plugins $ curl -s http://kafka-connect:8083/connector-plugins [ { "class": "io.confluent.connect.jdbc.JdbcSinkConnector" }, { "class": "io.confluent.connect.jdbc.JdbcSourceConnector" }, { "class": "io.debezium.connector.mysql.MySqlConnector" }, { "class": "org.apache.kafka.connect.file.FileStreamSinkConnector" }, { "class": "org.apache.kafka.connect.file.FileStreamSourceConnector" } ] Add connector $ curl -s -X POST -H "Content-Type: application/json" --data @connector-config.json http://kafka-connect:8083/conn Remove connector $ curl -X DELETE -H "Content-Type: application/json” http://kafka-connect:8083/connectors
- 13. 13 | Kafka Connect /Debezium - Stream MySQL events to Kafka Debezium Connector – Sample CDC Event { "schema": {}, "payload": { "before": null, "after": { "id": 1004, "first_name": "Anne Marie", "last_name": "Kretchmar", "email": "[email protected]" }, "source": { "name": "mysql-server-1", "server_id": 223344, "ts_sec": 1465581, "gtid": null, "file": "mysql-bin.000003", "pos": 805, "row": 0, "snapshot": null }, "op": "d", "ts_ms": 1465581902461 } } { "schema": {}, "payload": { "before": { "id": 1004, "first_name": "Anne Marie", "last_name": "Kretchmar", "email": "[email protected]" }, "after": null, "source": { "name": "mysql-server-1", "server_id": 223344, "ts_sec": 1465889, "gtid": null, "file": "mysql-bin.000003", "pos": 806, "row": 0, "snapshot": null }, "op": "d", "ts_ms": 1465581902500 } } INSERT DELETE
- 14. 14 | Kafka Connect /Debezium - Stream MySQL events to Kafka Useful Links Kafka Connect – User Guide http://docs.confluent.io/2.0.0/connect/userguide. html Debezium – Interactive tutorial http://debezium.io/docs/tutorial/ Debezium – MySQL connector http://debezium.io/docs/connectors/mysql/ Kafka Connect – REST Endpoints http://docs.confluent.io/2.0.0/connect/userguide.html#rest- interface Debezium Support/User Group User :: https://gitter.im/debezium/user Dev :: https://gitter.im/debezium/dev Kafka Connect – Connectors https://www.confluent.io/product/connectors/
- 15. 15 | Kafka Connect /Debezium - Stream MySQL events to Kafka Q & A
- 16. 16 | Kafka Connect /Debezium - Stream MySQL events to Kafka Thank you http://linkedin.com/in/kasundon