Multi source replication pdf

MySQL Labs :
Multi Source Replication
Venkatesh Duggirala
MySQL Replication Team
1 | Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | MySQL User Camp| Bangalore, India, October 17th, 2014. |

Safe Harbour Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should not
be relied upon in making purchasing decisions. The development, release, and
timing of any features or functionality described for Oracle’s products remains at the
sole discretion of Oracle.

MySQL Labs features are not fit for production.
They are provided solely for testing purposes, to try the latest bug fixes and
generally to keep up with the development.
Please, do not use these binaries in production.
Instead, install them on a spare server.
If you are looking for production ready binaries, please visit MySQL Downloads.

Agenda
 Background: Why Use Replication?
 Multi-source Replication
– Introduction
– Use cases
– Internals
– Monitoring
 Reading More about Multi-source Replication

Background: What is Replication Used For?
 Redundancy: If master crashes, promote slave to master
Uh Oh! Whew!
C
B
A
C
B
CraAsh
C
B
A
B is the
new master

 Read scale-out
M S
More
reads?
More
slaves!
S
S
S
M
write clients
write clients read clients read clients

 On-line Backup and Reporting
M
S
 write clients
 business intelligent client apps
 reporting client apps
 big queries client apps


Different types of Replication servers setup
M
S
1


M
S
M
S1 S2
1 2


M
S
M
S1 S2
M1
M2
S
1 2 3


M
S
M
S1 S2
M1
M2
S
M1 M2
S
1 2 3 4


M
M
M
M
M S
M
S1 S2
M1
M2
S
M1 M2
S
1 2 3 4 5

Agenda
– Introduction
– Use cases
– Internals
– Monitoring

Before Multi-source replication :
● Time sharing replication, where a slave would connect to a master
for a particular time slice.
● Have a hierarchical replication, where a slave that is to receive
updates from the several masters is at the end of replication
hierarchy. (Like, M1->M2->M3->Slave)
● Using several instances of mysqlbinlog + GTIDs.
Multi-source replication feature aims to:
● Receive transactions (in the form of binary log events) from
several MySQL servers simultaneously via several
communication channels.
● Apply these received transactions (will not do any conflict
detection or resolution)

Multi-Source Replication: Introduction
 A Slave can have more than one master.
M1
M2
M3
S
M4

db1 M1

M1
M2
db1
db2

M1
M2
M3
M4
db1
db2
db3
db4

M1
M2
M3
db2
db3
S db1 db2 db3 db4
M4
db1
db4

Agenda
– Introduction
– Use cases
– Internals
– Monitoring

Multi-Source Replication: Use Cases
The main use cases of Multi-source replication are related to data aggregation.
M1
Business Intelligence
Data analytics
Backup
etc
M2 S
M3
Database 2
Database 3
Database 1, 2, 3
Database 1

Multi-Source Replication: Use Cases
The main use cases of Multi-source replication are related to data aggregation.
M
M S
M
Shard 2
Shard 3
Full table
Shard 1

Agenda
– Introduction
– Use cases
– Internals
– Monitoring

Multi-Source Replication: More details..
M1
M2
M3
S
M4
 Slave can have more than one
master.
– Receive transactions from
several MySQL servers
simultaneously.
– Apply transactions from
different masters
simultaneously.
– No conflict detection or
resolution.

Multi-Source Replication: What is a Channel?
 A channel is an an abstraction for a sender-receiver-applier path.
Insert...
Sender
thread
Insert...
B
binary log
relay log
Insert...
Insert...
A
binary log
Client
Network
Receiver
thread Applier
thread
Channel

Multi-Source Replication: How Many Channels?
 Number of channels on slave = Number of sources.
S
M1 binary log Network relay log1
binary log

Multi-Source Replication: Slave Appliers
 Each channel has its own single-threaded slave applier.
binary log S

Multi-Source Replication: Slave Appliers
 Each channel can also have its own multi-threaded slave applier.
binary log
M1 binary log Network
M2 S
M3
Network relay log2
Network rreellaayy lloogg33
binary log
binary log
relay log1

Adding / Configuring A Channel
 Each channel can be configured individually.
 To add or alter a channel configuration use:
CHANGE MASTER TO master_def …
FOR CHANNEL ”<channel_name>”
– Channel names are case insensitive
– Special chars are allowed
– maximum number of channels is 256

Replication Commands on a Channel
 FOR CHANNEL ”<channel_name>” clause is added for:
– start slave [thread_type] FOR CHANNEL ”<channel_name>"
– stop slave [thread_type] FOR CHANNEL ”<channel_name>”
– reset slave [all] FOR CHANNEL ”<channel_name>”
– flush relay logs FOR CHANNEL ”<channel_name>”
– show relay log events FOR CHANNEL ”<channel_name>”
 An extra argument added for functions like
– MASTER_POS_WAIT(<binlog_file>, <binlog_pos>, <time_out>,
<channel_name>);
– WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS(<gtid_set>, <time_out>,
<channel_name>);

Working With All Channels
 To make commands work on ALL CHANNELS
– START SLAVE [thread_type]
– STOP SLAVE [thread_type]
– SHOW SLAVE STATUS
– FLUSH RELAY LOGS
 ER_SLAVE_MULTIPLE_CHANNELS_CMD(1964)
– SHOW RELAYLOG EVENTS;
– CHANGE MASTER
– MASTER_POS_WAIT()
– WAIT_FOR_SQL_THREAD_AFTER_GTIDS()

Compatibility With Other Replication Features
Multi-Threaded Slaves Yes
Global Transaction Identifiers Yes
Semi-Synchronous Replication Yes*
Slave Repositories Yes*
Filters Yes*

Compatibility continued 2...
Multi threaded slaves:
slave_parallel_workers=k and M channels will have (k+1)*M applier threads running.
GTIDs:
A requirement as is the case with (normal master-slave replication), is that all the masters
and the slave should have GTID_MODE=ON.
Semi-Synchronous Replication:
all the masters and the slave should be in semi sync mode
Filters:
replication filters are global in the labs release, ie. the applier threads of all channels will
rely on a single global filter.

Compatibility continued 3...
Slave Repositories:
 Only TABLE repository is supported (Crash safe slave)
 --mysqld= --master_info_repository= TABLE;
 --mysqld= --relay_log_info_repository= TABLE;
 Convert FILE repository to TABLE repository dynamically
 STOP SLAVE;
 SET GLOBAL master_info_repository= 'TABLE';
 SET GLOBAL relay_log_info_repository= 'TABLE';

Simple Setup
✔ Start 3 servers

Simple Setup
13000 13001
M1 M2
13002
S
✔ Start 3 servers , M1 (13000), M2 (13001), S (13002)

Simple Setup
13000 13001
M1 M2
13002
S
✔ On Slave(S),
> CHANGE MASTER TO MASTER_PORT=13000,...,
FOR CHANNEL 'CHANNEL_M1';
✔CHANNEL_M1 is created
channel_m1

Simple Setup
master1-bin.000001
13000 13001
M1 M2
13002
S
✔ On Slave(S),
> START SLAVE FOR CHANNEL 'CHANNEL_M1'
✔ I/O thread for CHANNEL_M1 will be created
✔ M1's binary log (table t1 changes) will be retrieved by I/O thread
and written to slave-relay-bin-channel_m1.00001 relay log
✔ SQL thread will start reading it from relay log and will start applying
the data.
channel_m1
insert into t1(..)
insert into t1(..)
Slave-relay-bin-channel_
m1.000001

Simple Setup
master1-bin.000001
insert into t1(..)
13000 13001
M1 M2
13002
S
✔ On Slave(S),
the data.
FOR CHANNEL 'CHANNEL_M2'
✔ CHANNEL_M2 is created
channel_m1 channel_m2
insert into t1(..)
m1.000001

Simple Setup
master1-bin.000001
insert into t1(..)
master2-bin.000002
insert into t2(..)
13000 13001
M1 M2
13002
S
✔ On Slave(S),
the data.
the data.
insert into t1(..) insert into t2(..)
m1.000001
m2.000001

Simple Setup
master1-bin.000001
insert into t1(..)
insert into t1(..)
master2-bin.000002
insert into t2(..)
insert into t2(..)
13000 13001
M1 M2
13002
S
✔ On Slave(S),
the data.
the data.
insert into t1(..)
insert into t1(..)
insert into t2(..)
insert into t2(...)
m1.000001
m2.000001

Agenda
– Introduction
– Use cases
– Internals
– Monitoring

Replication Monitoring: P_S Replication Tables
Connection
Configuration
Connection
Status
Execution
Configuration
Execution
Status
Slave Status
Applier / Coordinator
Status
Workers
Status

Replication Monitoring: P_S Replication Tables
 We have six performance schema tables for replication
– replication_connection_configuration
– replication_connection_status
– replication_execute_configuration
– replication_execute_status
– replication_execute_status_by_coordinator
– replication_execute_status_by_worker
 Consistent semantics across tables. Lets explore one of them.
One row per CHANNEL
One row per CHANNEL per WORKER

45
Replication Monitoring: Connection Status
mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME : channel1
SOURCE_UUID : 7cff7406-23ca-11e3-ac3e-5c260a83b12b
THREAD_ID : 13
SERVICE_STATE : ON
RECEIVED_TRANSACTION_SET : 7cff7406-23ca-11e3-ac3e-5c260a83b12b:1-4
LAST_ERROR_NUMBER : 0
LAST_ERROR_MESSAGE :
LAST_ERROR_TIMESTAMP : 0000-00-00 00:00:00
*************************** 2. row ***************************
...
One row per CHANNEL

46
*************************** 1. row ***************************
THREAD_ID : 13
SERVICE_STATE : ON
*************************** 2. row ***************************
...
Receiver thread &
its service state

47
*************************** 1. row ***************************
THREAD_ID : 13
SERVICE_STATE : ON
*************************** 2. row ***************************
...
Set of transactions received
through this channel

48
*************************** 1. row ***************************
THREAD_ID : 13
SERVICE_STATE : ON
LAST_ERROR_MESSAGE : error connecting to master 'replssl@127.0.0.1:13000' ...
*************************** 2. row ***************************
...
Oops! There was an error in the
receiver thread on this channel

49
*************************** 1. row ***************************
THREAD_ID : NULL
SERVICE_STATE : OFF
LAST_ERROR_MESSAGE : error connecting to master 'replssl@127.0.0.1:13000' ...
*************************** 2. row ***************************
...
Receiver thread on this
channel stopped serving...

Agenda
– Introduction
– Use cases
– Internals
– Monitoring

Read More About Multi Source Replication
 Read a feature preview of Multi-source replication on Rith's blog:
http://on-mysql-replication.blogspot.in/2013/09/feature-preview-mysql-multi
-source-replication.html
 Read more about Multi-source replication on Rith's blog:
http://mysqlhighavailability.com/5-7-5-labs-multi-source-replication/
 Read more about replication P_S tables on Shiv's blogs:
http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-pe
rformance-schema-tables.html
http://mysqlhighavailability.com/global-variables-in-rpl-ps/
 Official MySQL documentation for replication P_S tables:
http://dev.mysql.com/doc/refman/5.7/en/performance-schema-replication-tables
.html

Multi source replication pdf

More Related Content

Similar to Multi source replication pdf (20)

More from Mysql User Camp (10)

Recently uploaded (20)

Multi source replication pdf