HTTP Plugin for MySQL!

HTTP Plugin for MySQL!
Ulf Wendel, MySQL/Oracle

The speaker says...
MySQL is the WebScaleSQL DBA darling:
• The power and popularity of SQL
• The – after all – deployment story
• Scale-up: supports latest commodity hardware
• Scale-out: asynchronous replication
• Scale-out: in-memory, synchronous (2PC) replication
• Scale-out: virtual synchronous, group based replication
• Scale-out: sharding through Fabric
Just, we forgot about the web developer!

shell> curl --user user:password -X PUT
-d $'{"msg": "Welcome, JavaScript user!"}'
http://new.mysql.com/doc/pres/http/key_message

The speaker says...
Imagine, you are under such pressure that you have to
revolutionize the way you work. Web developers do that.
Infrastructure costs socialized through Open Source: web
developers! Agile movement: web developers! Rusty
database vendors don't deliver: Not Only SQL created!
• Convenient APIs and relevant protocols
• Really easy sharding
• Schema flexibility and documents
• Using their latest tools: JavaScript and JSON
• No always failing short yet complicated ORM
• No lobby standard ECISANO 5.7 Rev. 16 – buy now

Backend: It's LAMP. Period.
Frontend: Dinosaur, it's 2014. Need a RESTful API.
Backend: No time. And, no to framework Sendso...

The speaker says...
Every other day two developers clash on the question of
databases...
MySQL cannot ignore the needs of web developers. MySQL
cannot hide itself behind the argument some third party
framework could be used – that failed badly for replication.
MySQL cannot hesitate and hope legacy code would stop
migrations. Web developers completed two revolutions.
Replacing MySQL would be a minor one. DBAs won't stop
them.
What if MySQL speaks HTTP, stores JSON, returns
JSON and preserves the richness of SQL?

Three HTTP APIs
All APIs return JSON
SQL
CRUD: flat JSON documents
mapped to tables
Nested JSON documents
stored in tables
Key-Document semantics

The speaker says...
Every other day, a proxy is desired to make MySQL more
accessible: HTTP – PHP|Perl|Python|... - MySQL. All the
proxies translate HTTP to SQL. SQL lets them offer any API.
What if MySQL had a built-in HTTP to SQL proxy?
Right from start, there could be a variety of APIs. Plain
SQL over HTTP is a given. Everybody knows SQL,
everybody understands the advantages of a declarative
language, only SQL can expose the full feature set of
MySQL. No attempt to dictate any whiteboard API:
plain SQL. Google created AngularJS for CRUD
applications. CRUD is a given. Nested JSON documents
are a must.

http://labs.mysql.com
Free (GPL), 80+ pages documentation, many examples

The speaker says...
The built-in MySQL HTTP APIs are real. You can get the
HTTP Plugin for MySQL from http://labs.mysql.com
(September 2014). As always, it's free and GPL. It comes
with a 80+ pages PDF document that has many examples
on using it. The HTTP Plugin is for the JavaScript users,
for those fighting corporate firewalls that allow no
other protocols but HTTP, those who love web services
and those who are tired of writing yet another
RESTful plain scripting language proxy for MySQL!
Next: Unboxing. Then a word on how it works, how it
serves not only data but you. And finally an outlook.

UNBOXING: a seasonal s carf ready MySQL 5.7 Server

The speaker says...
Autum is coming to the northern hemisphere but before the
nights get longer than the days, before we all start knitting
scarfs, it's time for world's first unboxing.
In the box of the initial release is a special version of the
MySQL Server. The server is based on the MySQL Server
preview (5.7.5-m15) made available for MySQL Connect
2014. But, it has some additional feature tweaks. For now,
you need this very server and no other.
Why that is the case will be explained in a minute.

UNBOXING: a high tech H TTP Plugin for MySQL

The speaker says...
MySQL stands for high tech: HTTP Plugin for MySQL!
The treasure box Sakila (the MySQL mascot) accidently
discovered at the beach contains the HTTP Plugin too.
One slide on installation to come, then we talk how it works.

UNBOXING: plain vanilla...
Install MySQL from the treasure box
Create a SQL user for the plugin
• Set password, grant access to SQL object
Install the plugin
• INSTALL PLUGIN myhttp SONAME 'libmyhttp.so'
Edit my.cnf
• Set default SQL user for plugin, HTTP basic auth, port, etc.
Restart MySQL
• If you edit first, then load – recall to use loose option prefix

The speaker says...
The installation procedure is not different from any
other plugin. Some stuff the plugin needs to work is to be
set up first. The HTTP plugin runs SQL and does that – by
default – as a certain user. Create that user. Personally, I
prefer loading the plugin in the next step before making
changes to the MySQL Server configuration file. This allows
me not to bother about a server that barks about unkown
settings. Once installed, the plugin begins listening to HTTP
requests (default 8080). If you have a server running on
that port, edit my.cnf first to set port, user etc., then start
MySQL. Recall to use the „loose“ prefix trick. Then install the
Plugin. The manual has the details... We get back to
settings in the usage section after: how it works (now).

How? Extending MySQL!
MySQL
Core
Deamon Plugin
Deamon Plugin
I_S Plugin
Full-Text Parser Plugin
Plugin Services
User-defined functions
Semisync Replication Plugin
Audit Plugin
Authentication Plugin
Password-Validation Plugin Protocol Trace Plugin
Storage Engine Plugin

The speaker says...
The MySQL manual spends 80 pages educating readers how
to extend MySQL: there are user-defined functions (UDF)
and there are plugins. SQL can be extended with UDFs. Nine
different types of plugins exist. Best known should be the
storage engine plugins (InnoDB, Memory, NDB, …, 3rd party)
and the authentication plugins (MySQL native, SHA256, …).
The HTTP Plugin is a daemon plugin. A daemon
plugins is a do-as-you-like library that is loaded into the
MySQL Server. Plugins have access to half a dozen plugin
service provided by the server (Memory, thread
management, ...). The HTTP Plugin is build atop of new
plugin services.

The Protocol deamon plugins
HTTP Client Memcache driver MySQL Connector
MySQL
Daemon Plugins
HTTP Plugin
Memcache Plugin
Core
HTTP
Protocol
(JSON)
Memcache
Protocol
(Binary)
MySQL Client/Server
Protocol
(Binary)

The speaker says...
Deamon plugins can start a network server and listen
to an arbitrary protocols to serve data. In 2009, the
year the equation NoSQL = Not Only SQL was proposed, the
first community plugin did exactly that. For rich queries
the customers used SQL, for fast but simple queries a
home-brewed daemon plugin using lower level APIs. MySQL
was at the forefront but few know it.
The MySQL Server comes with a free Memcache
Plugin for InnoDB since 5.6, so does MySQL Cluster.
The customers and community desire was to kill any read
access overhead: SQL parsing, network traffic. For write
they desired ACID, in-place update and predictable
performance under all circumstances.

The query language flexibility
HTTP Client Memcache driver MySQL Connector
MySQL
Daemon Plugins
HTTP Plugin
Memcache Plugin
Core
SQL
Key-Document Key-Value
SQL

The speaker says...
The HTTP Plugin Lab Release is not designed for maximum
speed. It targets the JavaScript user. Today, it is just a
demo of what APIs we could offer to the JavaScript
user. The Lab Release does nothing you cannot do
yourself for years already: all HTTP requests are
mapped to plain SQL internally. But it does it for you.
The day the APIs have been finalized we can boost the
performance.
We are also exposing the richness of SQL. There is no we
know the best ORM and you must love it. Did they ever
work? Use SQL or try the CRUD interface we sketched
or store nested JSON documents. At this point, all we
do is add a HTTP to SQL proxy to MySQL...

Your trouble vs. our trouble
HTTP Client
SQL, Key-Document
over HTTP
HTTP Plugin
MySQL
HTTP Client
Some service
over HTTP
Webserver
Script Language
MySQL Connector
MySQL

The speaker says...
This Lab Release is no rocket science. Effectively, we are
only announcing the intend to offer an HTTP
interface as a convenience feature. We move the proxy
you write today to make MySQL accessible through HTTP
into the server – as an optional plugin. We take the trouble
of thinking about security or performance.
There is one drawback, assuming the APIs work for you.
Our solution could be a little less scalable. A shard
nothing architecture, an echo of the Unix way of having
small independent programs, is often the best way to scale.
But then, the external proxy has more connections and may
be slower... Wait for it to stabilize before you stress
test – not worth checking today. Let's talk using and APIs.

http://new.mysql.com/sql/pres/SELECT%20'Hello World!'
http://new.mysql.com/crud/pres/http/hello
http://new.mysql.com/doc/pres/http/hello

The speaker says...
There are three APIs with some properties in common:
• they have JSON all over
• guess what: they all use HTTP, which sets limits
• the same (yet unsecure) security pattern
• they share the same restrictions
The APIs differ by:
• their query language
• their current interest in REST...
Common things first at the example of the SQL API.

The SQL endpoint and JSON
shell> curl --user basic_auth_user:basic_auth_passwd
--url "http://127.0.0.1:8080/sql/db/SELECT+1"
[
{
"meta":[
{"type":8,"catalog":"def","database":"","table":"",
"org_table":"","column":"1","org_column":"","charset":63,
"length":1,"flags":129,"decimals":0}
],
"data":[
["1"]
],
"status":[{"server_status":2,"warning_count":0}]
}
]

The speaker says...
The SQL endpoint is our Connector/JavaScript. What you
see is exactly what a standard MySQL Connector gets in
reply to the query. But the standard MySQL Connector is
using the binary MySQL Client/Server Protocol and all data
would look funny. Because the same information is
returned the HTTP client is not feature constrained.
That is a sweet lie as we will see later but you get the idea.
The HTTP client, the JavaScript user is not a second
class client who is limited to restricted APIs up-front.
All JSON is UTF-8. For now, the HTTP Plugins' web is UTF-8.
Binary data and funny german umlauts are encoded
using uxxxx notation, as json.org & friends ask for.

Dot zero: HTTP is stateless
No session concept for the 1.0
• Stateless helps with scaling and reduces complexity
• Some feature, e.g. SQL session variables, have limited value
ACID but autocommit
• Each web request is a transaction
• Difficult to isolate multi-statement/business transactions
--url "http://new.mysql.com/sql/db/SET+@no='session'"
{ "server_status": 2, "warning_count": 0, "affected_rows": 0,
"last_insert_id": 0 }

The speaker says...
The stateless nature of HTTP is no natural match for
SQL. ACID transactions are an integral part of SQL systems.
Transactions are units of work made of one or more
statements. The SQL endpoint lets you carry out one SQL
statement per request. Every new HTTP request is
independent of the previous one, thus every statement
forms its own transaction: it runs in autocommit mode.
You get ACID properties: atomicy, consistency, isolation
and durability. E.g. two concurrently executed web requests
are properly isolated from each other.
Sessions? Makes a lot sense to me! Not only for multi-statement/
business transactions.

Dot zero: user/security concept
HTTP Client
No SSL
Authentication
HTTP Plugin
Preconfigured user
MySQL
HTTP Client
SSL
Authentication
HTTP Plugin
Request user
MySQL
SQL grants

The speaker says...
The user/security concept is two-staged. First, a HTTP
client authenticates itself towards the HTTP Plugin. Then,
the HTTP Plugin executes SQL statements on behalf of the
HTTP client as a certain SQL user. If no SSL is used, the
HTTP Plugin logins in towards MySQL as a
preconfigured SQL user. The HTTP world and the SQL
world are decoupled. With secure connections (SSL) the
user name and password obtained from the HTTP
world can be passed to MySQL as login credentials.
A HTTP client is just another MySQL client. Use the SQL
permission system to restrict access!

HACKER Welcome!
MySQL morons using HTTP Basic Authentication...

The speaker says...
The initial dot zero Lab Release is using HTTP Basic
Authentication all over. All resources – SSL and Non-SSL –
are protected with basic auth.
HTTP Basic Authentication is fast to implement.
Everybody understands the examples because of their
simplicity. That is perfect for a /preview/, that is perfect
to kick of a discussion.
Unfortunately HTTP Basic Authentication is one of the
least secure choices one can make.

SSL: User/security concept
HTTP Client
SSL
OpenID, OAuth...
HTTP Plugin
MySQL
HTTP Client
SSL
Password exchange
HTTP Plugin
Authorization
MySQL

The speaker says...
How hard can it be to support any proper HTTP
authentication method: a day for OAuth, two weeks for
OpenID? The Lab Release uses HTTP Basic Auth as a
vehicle that combines two steps, which is confusing but was
fast to implement. For SSL connections, and only SSL
connections, users should first authenticate
themselves through some proper HTTP authentication
method. Once authenticated, they shall be able to
login as a MySQL user of their choice. We still need to
fiddle out the details and we lack something here as the
outlook will show.
Next common properties of all APIs: restrictions.

Dot zero: restrictions
No query cache support
• It's HTTP – use a standard HTTP caching!
• MySQL Support often suggests disabling the query cache
No commercial thread pool plugin support
• There are now HTTP network threads and SQL worker threads
• HTTP layer can do multiplexing
Not all MySQL pluggable auth methods supported
• Some pluggable auth methods have handshakes
• Handshake procedures require use of MySQL networking layers

The speaker says...
The preview has some restrictions. They all boil down to the
fact that building a proxy into MySQL is an all new
requirement. Various modules in MySQL have not been
designed for this new use case. How could they? It is
a new use case...
Do you mind being restricted to use MySQL SQL accounts
that have been created with MySQL's default
authentication plugin? Do you mind following a
MySQL Support advice of not using the query cache?
Recall, that you should deploy InnoDB with 80% of your
memory dedicated to the buffer cache. Disk access is rare.

The SQL endpoint
Unlimited: all SQL statements
• DDL, DML and utility statements
• Multi-resultset statements supported, e.g stored procedures
Unlimited: everything the server returns
• Meta data, query data, server status
• Freedom to build your own Connector/JavaScript
[{
"meta" : [ … ],
"data" : [ … ],
"status" : [ … ]
}]

The speaker says...
First class customers use the SQL endpoint. It has the
most to offer. There are no restrictions on which SQL
statements can be executed. Use the MySQL SQL permission
system to set any, if needed. No limited white-board API is
imposed on you.
All the information the server returns in reply to a query is
converted to JSON and sent to the client. A standard
MySQL Connectors using the MySQL Client/Server
Protocol gets the same information but in binary
form. Because the same data is delivered, clients can
develop their own Connectors, their own higher level
APIs and data structures.

Unlimited result set meta data
[{
"meta":[{
"type" : 4,
"catalog" : "def", "database" : "pres",
"table" : "sql_types", "org_table" : "sql_types",
"column" : "col_float", "org_column": "col_float",
"charset" : 63,
"length" : 12,
"flags" : 4097,
"decimals": 31
}],
"data" :[ … ],
"status":[ … ]
}]

The speaker says...
On the slide is the result set meta data for SELECT col_float
FROM sql_types. The full table definition of the example
table sql_types is given in the manual, however, col_float is
of the SQL type FLOAT.
For every column in the result set you get one object in the
meta array. Every object has the same members: there's the
database name, the table name, the column name, the
charset and so forth.
Why all this result set meta data?

Types: string or NULL
shell> php -r '
$m = new mysqli("host", "user", "password", "db", 3306);
$m->query("SELECT col_float FROM sql_types")->fetch_assoc());'
array(1) {
["col_float"]=>
string(6) "0.9999"
}
--url "http://…/sql/db/SELECT+col_float+FROM+sql_types"
[{
"meta":[ … ],
"data":[ ["0.9999"], … ],
"status":[ … ]
}]

The speaker says...
The SQL endpoint converts all SQL result data to
JSON strings or JSON NULL.
A conversion to string is the servers standard behaviour
for non-prepared statements in the MySQL
Client/Server Protocol too. This server implementation
detail still shows in PHP. Still today, PHP returns numbers as
strings. That made a lot sense in the last millennium. HTTP
GET and POST data are strings. PHPs' dynamic typing was a
relief for those doing web development in C before.
There's still a point for sending strings today.

No type mapping, no loss
How would the server know who consumes the data
• Which rules?
• Information loss allowed?
SQL type JSON candidate Information loss
SMALLINT Number Precision, range
FLOAT Number Precision, range
VARCHAR String Precision, charset
DATE String Type, range
BOOLEAN True, false
NULL Null

The speaker says...
The server does not know the type system of the
client. If, for example, SQL data types had been mapped to
JSON data types and no meta data had been included,
information had been lost.
Complicated? Choose between a Bugatti Veyron and a
Suzuki Maruti 800. The Suzuki has simpler controls. The
Suzuki costs less. The Suzuki is more fuel efficient. All
speaks for the Suzuki but we won't give it to you. The SQL
endpoint is as luxury and as complex as the Bugatti.
Here's why: it's easy to use a hammer to shrink the
Veyron to the size of the Suzuki. It's hard to make the
Maruti go 400 km/h. Even when dropping from the sky –
cW too high ;-).

JavaScript same origin policy
JSON with padding
• Optional escaping into string
• CORS on the radar
var ret = $.ajax({
url: ".../sql/db/SELECT%20'Hello world!'%20FROM%20DUAL",
type: "GET",
username: "basic_auth_user", password: "basic_auth_passwd",
dataType : "jsonp", jsonp: "jsonp",
success: function( json ) {
alert("First row: " + json[0].data[0]);
},
});

The speaker says...
Browserside JavaScript has two options to overcome the
same origin policy: JSON with padding (JSONP) and
Cross-Origin Resource Sharding (CORS). JSONP is there
for the Lab Release, CORS is missing. Add the GET
parameter jsonp=<my_callback_name> to request to get
JSONP replies. The result will be:
my_callback_name(<json>). The slide shows an example
for jQuery.
Optionally, you may request the <json> to be returned as a
string value, like so: my_callback_value("<json>"). If you
need the latter, add the GET parameter jsonp_escape to the
request URL.

HACKERs' heaven!
Popcorn for everbody. MySQL in the web...

The speaker says...
Ever noticed NoSQL systems offering HTTP interfaces?
Peace: it is an optional plugin...! You decide whether you
load it into MySQL or not.
The HTTP Plugin currently lacks settings for disabling
endpoints. Accidently exposing sensitive data, for
example, because of inappropriate SQL permissions,
is a security risk. We will talk about that at the end. The
SQL endpoint is the unlimited first class customer
API. Great power comes with great responsibility. Use it
with greatest care!
Next: the more restricted CRUD endpoint!

The CRUD endpoint
Flat JSON objects mapped to tables
• Always JSON objects towards the caller
Limited query language
• Data manipulation only: GET, PUT, DELETE
• Key-document semantics
• No data definition statements
• No administrative statements
Less network traffic
• No result set meta data

The speaker says...
The CRUD endpoint features simple key-document
semantics for unnested JSON documents. Only data
manipulation operations can be performed. Whereas the
SQL endpoint allows no other HTTP methods but GET, the
CRUD endpoint accepts GET, PUT and DELETE requests. In
that sense it shows more love for REST.
On the outside there are JSON documents (objects) only.
On the inside all requests are mapped to SQL in a
straightforward way. Let's see how.

The CRUD SQL mapping
Document = SQL table row
• Documents are flat, they are limited to scalar types
• Underlying tables must have a single column primary key
CREATE TABLE `simple` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`col_a` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
shell> curl
--user basic_auth_user:basic_auth_passwd
--url "http://.../crud/db/simple/1"
{"id":"1","col_a":"Hello"}

The speaker says...
It is that simple: SQL table rows are returned as
JSON objects. Plain SQL tables serve as JSON document
containers.
Each document is derived from only table only. Because
MySQL SQL tables support scalar types only, the
corresponding JSON douments contain scalar types only
such as JSON string, number, boolean or null. The CRUD
endpoint does not support JSON documents containing
arrays or objects.
Every mapped table must have a single column primary key.
The primary key is used for the key-document access logic.

GET = SELECT
shell> curl … --url "http://../crud/db/sql_types/1"
{
"id" : "1",
"col_char" : "CHAR(127)",
"col_null" : null,
"col_date" : "2014-08-21",
"col_decimal": "123.45",
"col_float" : "0.9999",
"col_bigint" : "9223372036854775807"
}
shell> curl … --url 'http://../crud/db/simple/'
{
"errno" : 2000,
"error" : "The request URL must include a primary key value"
}

The speaker says...
The initial version features key-document accesss only.
Therefore, all GET requests must include a primary key
value and follow the URL pattern
protocol://server:port/crud/database/table/pk_value.
Like the SQL endpoint, the CRUD endpoint returns all data
either as a string or null. All the APIs are far from finalized.
A possible future optimization would be a better mapping
between SQL and JSON data types. For example, boolean
values could be returned as true/false JSON constants.

Key-document only? No :-) !
Standard client, e.g. PHP
Memcache driver MySQL Connector
JS framework
Rich SQL
Tuneable K/V Fast K/V
MySQL
JS in browser
Rich SQL
HTTP Plugin Memcache Plugin Core

The speaker says...
MySQL 5.7.5 handles up to ~625.000 point select
SQL queries and ~1.100.000 comparable Memcache
accesses per second. This clearly shows the tuning
potential behind simplified key-value (document) style APIs.
Standard MySQL clients need two drivers to profit
from the rich query capabilities of SQL and the performance
of K/V. The CRUD endpoint is untuned. First, there needs
to be a proper API (not the current...), then one can
optimize. Once optimized HTTP clients can choose by
just changing the request URL slightly: sql/ vs. crud/
respectively doc/. Could not be better, could it?
Back to the CRUD endpoint basics: HTTP PUT

PUT = REPLACE
shell> curl … -X PUT -d '{"col_a":"Ahoy"}'
{
"affected_rows": 2,
"warning_count": 0
}
shell> curl … --url "http://.../crud/db/simple/1"
{
"id" : "1",
"col_a": "Ahoy"
}
shell> curl … -i -X PUT -d ''
HTTP/1.1 400 Bad Request
…

The speaker says...
HTTP PUT is mapped to SQL's REPLACE statement in the
CRUD endpoint of the initial Lab Release. PUT expects a
JSON object as input. The object must have one top
level member for each column in the underlying SQL
table that expects a value. If you fail to provide proper
input, the HTTP Plugin for MySQL barks at you: 400 Bad
Request.
No attempt is being made to hide the SQL reply to the
REPLACE statement. It is one of the many edges we failed
to cut off for the development announcement release.
Another edge is the lack of clever mapping like PUT =
INSERT, PATCH = UPDATE. Adding that is about as hard
for us as it is for you in your HTTP to MySQL proxy...

DELETE = DELETE, that's easy!
shell> curl -i … --url "http://.../crud/db/simple/1"
HTTP/1.1 200 OK
…
{"id":"1","col_a":"Ahoy"}
shell> curl -i -X DELETE … --url "http://.../crud/db/simple/1"
HTTP/1.1 200 OK
shell> curl -i … --url "http://.../crud/db/simple/1"
HTTP/1.1 404 Not Found
…
{"errno": 2000, "error":"Not found"}

The speaker says...
HTTP DELETE is translated to SQL DELETE. That's it.
There is some support for HTTP principles. For example,
HTTP methods are mapped appropriately to data
manipulation commands. Even HTTP headers are set: 200
OK, 400 Bad Request, 404 Not Found and so forth. This
alone does not make a REST API and ticking of the checklist
for the buzzword is the least (important).
Next: nested JSON documents – SQL completely hidden.

The DOCUMENT endpoint
Nested JSON objects mapped to tables
• Always JSON objects towards the caller
Limited query language
• Key-document semantics
• Data manipulation: GET, PUT, DELETE
• Data definition: Table creation and removal
• Optimistic locking
No sign of SQL
• No cryptic affected_rows ...
• Leave room for possible optimizations

The speaker says...
When it comes to HTTP details and access methods, CRUD
and DOCUMENT are very much the same. DOCUMENT is
just a tiny bit more.
If you consider the design space for possible APIs of a
HTTP Plugin that wants to speak JSON all over, you
may notice that the two extremes are plain SQL and a JSON
document store. In our initial Lab Release, the CRUD
endpoint sits in the between the two extremes. The
DOCUMENT endpoint is the „what if MySQL bend
itself towards a JSON document store“. Stay seated:
an API does not make MySQL a proper document store!

JSON goes into a BLOB
shell> curl -i -X PUT --user …
--url "http://.../doc/myhttp/new_table/"
HTTP/1.1 201 Created
…
{"info": "Table created"}
mysql> SHOW CREATE TABLE new_tableG
*************************** 1. row ***************************
Table: new_table
Create Table: CREATE TABLE `new_table` (
`_id` varchar(36) NOT NULL,
`_rev` bigint(20) unsigned NOT NULL DEFAULT '0',
`_extra` blob NOT NULL,
PRIMARY KEY (`_id`,`_rev`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

The speaker says...
Use HTTP PUT requests to create tables for storing
documents, to insert documents and to update them. There
are no restrictions on what kind of JSON objects the
endpoint handles. The API accepts every valid JSON
object. Internally, JSON documents are stored in a BLOB
column of a standard SQL table.
The primary key of the underlying SQL table is made of two
columns: the id or key that appears in request URLs and a
revision counter. The revision counter is managed by the
HTTP plugin to implement optimistic locking.

Optimistic locking
HTTP Client HTTP Client MySQL
GET ../key
{"_rev":1, …}
{"_rev":"1", …}
PUT ../key {"_rev":1, …} OK: _rev: 2
PUT ../key …1} Conflict!

The speaker says...
Arguing that HTTP is stateless and reflecting that fact in the
APIs is great if you are short on time. But then, without
session support, without multi-statement
transactions – all endpoints use autocommit – how can
two clients avoid and detect conflicting wirtes? One
can leave the task to the client, you the developer. One can
consider multi-versioning. This raises at least a garbage
collection problem because the server cannot know when
old revisions may be removed. Or, the DOCUMENT
endpoint supports optimistic locking – which just
works if the clients cooperate. Means: a client must not
modify the special JSON document object member _rev.

Solve conflicts by refetching
shell> curl --user … --url ".../first_message"
{"_id":"first_message","_rev":1,"msg": "No big news" }
shell> curl -X PUT
-d '{"_id": "first_message","_rev": 1,"msg": "Optimist?"}'
--user … --url ".../first_message"
{"info": "Document updated"}
shell> curl -X PUT
-d '{"_id":"first_message","_rev":1,"msg": "Optimist?"}'
--user … --url ".../first_message"
{"errno": 2000, "error": "Update failed. Your revision does not
match the current revision"}

The speaker says...
Here's an example of optimistic locking in action. Every
JSON document managed by the DOCUMENT has two
special members: _id and _rev. _id is the key of the
document. _rev is the revision number used for optimistic
locking. When a JSON document is written back to the
database, the database compares its revision number
with the revision stored in the database. Should they
be equal, the write is accepted and the database increments
the revision. If not, then someone must have updated the
document already and the write is rejected. In case of a
conflict, a client should fetch the latest version,
merge the changes to solve the conflict and try
again.

PUT = CREATE TABLE, REPLACE
shell> curl -i -X PUT
--user … --url "http://.../doc/db/another_table";
HTTP/1.1 201 Created
{"info": "Table created"}
shell> curl -i -X PUT -d '{"words": ["Hello", "world"]}'
--user … --url "http://.../doc/db/another_table/key"
HTTP/1.1 200 OK
{"info": "Document added"}
shell> curl -i -X PUT -d 'is this json?'
--user … --url "http://.../doc/myhttp/another_table/key"
{"errno":2000, "error":"Invalid JSON"}

The speaker says...
For the REST, let's do a walkthrough the HTTP methods
supported. In the initial development release, the
DOCUMENT endpoint has slightly more features than the
CRUD endpoint. The CRUD endpoint will certainly be
updated. Not only AngularJS framework lovers shall have
free choice between the two types of mapping from JSON to
SQL, both APIs shall offer comparable feature sets.
The DOCUMENT endpoint supports PUT for creating new
tables, inserting nested JSON documents into them
and updating documents. Nothing fancy but the
optimistic locking. Invalid JSON is rejected.

GET = SELECT
shell> curl --user …
--url "http://.../doc/db/another_table/key"
{ "_id": "key", "_rev": "1", "words": ["Hello", "world"] }
shell> curl --user … --url "http://.../doc/db/another_table/"
{
"another_table": [
{ "_id": "key","_rev": "1","words": ["Hello", "world"] }
]
}
shell> curl -i --user …
--url "http://.../doc/db/another_table"

The speaker says...
A HTTP GET request against the DOCUMENT endpoint either
returns one document identified by its key (id) or all
documents from a table.
Recall, that the access method is key-document only. For
now, if you need filtering and the like, switch to the
SQL endpoint. Change the request URL and use SQL for
filtering.

Filtering? Indexing? Coming...
See labs.mysql.com
• JSON UDFs, work in progress: more functions than listed below
• MySQL 5.7.4: Virtual columns, stay tuned on functional index
Search
• By value: JSON_SEARCH()
• By key: JSON_CONTAINS_KEY(), JSON_EXTRACT()
Modify
• By value: JSON_MERGE()
• By key: JSON_APPEND|REMOVE|REPLACE|SET()

The speaker says...
With MySQL 5.7.4 it is finally possible to work on
JSON using SQL. The release brings:
• updated SQL functions for working on JSON
• virtual columns
• … and stay tuned on function based index
The SQL functions let you search JSON and modify it. They
fail to close the gap between the JSON and SQL data
models but if you stay within the JSON model it works.
The HTTP Plugin means JSON all over... Also, indexing
JSON will boost search beyond key-document.

DELETE = DELETE
shell> curl -X DELETE --user …
--url "http://.../doc/myhttp/another_table/key"
{
"info": "Document removed"
}
shell> curl -X DELETE --user …
--url "http://.../doc/myhttp/another_table/"
{
"info": "Table dropped"
}
shell> curl -X DELETE -i --user …
--url "http://.../doc/myhttp/another_table/"
HTTP/1.1 404 Not Found

The speaker says...
To remove documents and tables issue HTTP DELETE
requests. That's it – very similar to the CRUD endpoint.

Dear JavaScript users, please check the manual.
Some examples already there, more coming.

Recap
HTTP Protocol
in addition to
MySQL Protocol
JSON all over
Key-Document and SQL
complementing each other
SQL support for JSON
getting 'real' with
MySQL 5.7.4+

Hmm...? What's Sakila doing...

Hidden treasures? Labs leaks?
Oh, no... back to the sea ?!

The speaker says...
A MySQL Lab Releases can be anything from almost
production ready to a development announcement.
For example, there have been several prior releases of the
SQL JSON functions, which means they should work
reasonably well. The day MySQL 5.7.4 was released a
colleguage was praised for contributing to the SQL standard
around computed columns, which means they should work
reasonably well...
The HTTP Plugin 1.0.0 is a yammy development
announcement release. It's all about discussing it
early. That's why we have 80+ pages documentation from
day one but edges in the APIs.

Leaving the safe harbour
Glancing over the ocean of possibilities

The speaker says...
More open talk: ideas.
If you are a developer using MySQL, you may stop here.
The presentation has explained what the HTTP Plugin is,
why it was created, what it has to offer and what the
current status is.
What's left is a deeper look inside, how it's done
inside MySQL. That's for MySQL experts who seeks to
understand MySQL internals. Understanding them is a
precondition to use them in a different context, tailored to
the own needs.

Module A
Module B
Module C
Module D
Plugin API

The speaker says...
The challenge in doing a HTTP Plugin is not the HTTP API
itself. How hard can it be to add something developers do
every day to MySQL? How hard can it be to add a Proxy
to MySQL?
Some say MySQL is not modular. Some say MySQL lacks
clearly seperated modules in the code. Is MySQL a ball
of woll with knotted fiber? Some say the code base shows
its age. Is MySQL in the autum of its life, does it need a
scarf? Well, RDBMS have undergone permanent refactoring
in the past 50 years and they are sill around. NoSQL has its
first wave yet to come. Think MongoDB and storage.
(Insider: Steward, I took that photo prior to your blog post.)

Module A
Module B
My new API :-)

The speaker says...
The problem is a different one. With every new feature
request someone comes and invents a new module. Some
say, the authors of the HTTP Plugin invent new non-existing
modules every other day.
There are two new plugin services inside the HTTP
Plugin. Let's discover their potential beyond the HTTP
Plugin.

Plugin Services
HTTP Client
SQL or any derived query method
HTTP Plugin
Output Formatter Web Server, ...
SQL Execution Service Authentication Service
MySQL

The speaker says...
The HTTP Plugin is build atop of two plugin service. The
first is an authentication service to log in a MySQL
internal SQL execution thread as a certain MySQL user.
Once authenticated and authorized, the MySQL internal SQL
execution thread uses the SQL execution service.
The results are returned as a binary string stream. The
stream is processed by an output formatter, which is
provided by the plugin calling the service. The plugins
output formatter can serialize query results into any
format. In case of the HTTP Plugin, its JSON. Could have
made it BSON but which JavaScript client wants that...

Proxy: SQL over Protocol X
Websocket Client
SQL
Websocket Plugin
JSON Serialization
MySQL
HTTP Client
SQL
HTTP Plugin
JSON Serialization
SQL Execution Service
Your Client
SQL
Your network Prot.
Your serialization

The speaker says...
The HTTP Plugin is a protocol and data serialization
format SQL proxy built-in to MySQL. It is one that is
tailored for the JavaScript user. This restricts the
protocol options to HTTP and Websocket.
In 2009 some MySQL high-end community users added an
alternative, faster communication channel to MySQL for key-value
style queries. Many expert users restrict their
developers to a subset of SQL for performance
reasons. The MySQL Client/Server Protocol has to support
all SQL features and all programming language
communities. Any ideas for optimized SQL centric
protocols tailored to your users?

Proxy: for data models
HTTP Client
JSON Document
HTTP Plugin
SQL Rewrite Schema Mapping Util: e.g Catalog
Relations
MySQL

The speaker says...
The HTTP Plugin is data model proxy. Closing the gaps
between data models can be done in many ways. In its
early days, when MySQL was still looking for its space in the
market, the storage engine API was used to offer MySQL as
SQL frontend for about anything. Here is a suggestion from
the Connectors team! In a perfect world, the
Connectors team understands developers and their
motivation. Working at this level is easier, more can
contribute. The RDBMS world tried data model
mappings in the past: it was never fast. But, the
systems where never fast to support new models either!
Getting it right is hard. Time to market counts.
Web developer = no mercy = they do revolutions...

Proxy: simplified shardig
HTTP Client
Key-document
HTTP Plugin
SQL Execution
Any shard
Target shard
301 Moved
Which shard? Fabric

The speaker says...
MySQL supports auto-everything sharding since a
decade. It comes with MySQL Cluster. MySQL Cluster
just learned active-active asynchronous replication with
automatic conflict solution between synchronous clusters in
different data centers! Problem is, the web user wants
InnoDB. So we build MySQL Fabric. Fabric manages
farms of MySQL servers and lets you shard data across
them. Problem is, it is not transparent and requires
special clients. Imagine you limit the scope to stateless
HTTP and key document. Stateless means redirect is no
pain. HTTP means you can use slim clients and 301.
Document means cross-shard joins can be avoided... Extra
hop? Pffft, look at MySQL Cluster.

Let's not loose focus: JS! V8!
HTTP Client
Improved security, more freedom
HTTP Plugin
app/ - v8 JavaScript sql/, crud/, docs/ - built-in
SQL Execution Service Authentication Service
MySQL

The speaker says...
Let's not loose focus! The HTTP Plugin is for the
developer not the DBA and database architect. That's why
MySQL 5.7 brings true multi-master replication – its ease of
use. Multi-master has scalability limits (~10...20 machines)
but how many web developers are happy with a single of
todays monster machines (6k Euro price point = 256GB
RAM, 7TB disk, …) and bother only about dev features? The
HTTP Plugin is specifically for the web and JavaScript
developer. What could be more desired than adding the v8
JavaScript engine to the plugin. This also enables scripts
to implement additional security rules. Rules that go
beyond what the MySQL user system has to offer!

A world of options
Make it happen, today!

The speaker says...
It is a world of options. No decisions have been made about
any of them. Looking forward to hear back or see you
implement your ideas for MySQL.

THE END
Contact: ulf.wendel@oracle.com

The speaker says...
Thank you for your attendance!
Upcoming shows:
International PHP Conference (Munich),
October 26 - 29, 2014
Your place? Just ask!

HTTP Plugin for MySQL!

More Related Content

Similar to HTTP Plugin for MySQL! (20)

More from Ulf Wendel (19)

Recently uploaded (20)

HTTP Plugin for MySQL!