Unstructure: Smashing the Boundaries of Data (SxSWi 2014)

Unstructure
:: smashing the boundaries of data ::
2014-03-07 - SxSWi Workshop
Ian Varley - @thefutureian

Hi. I’m Ian Varley.
I live in Austin, TX.
I work for Salesforce.com, doing data engineering.
(Note: this presentation is entirely my own work and opinions, and doesn’t imply anything about Salesforce’s products.)
→ @thefutureian, ianvarley.com

About me:
- BA in Philosophy
- MS in Software Engineering
- 15+ years database experience
Not really an authority on data structure, but
"You teach what you want to learn".

Why are we here?
To grok the structure of data,
and then smash it.

Some logistics:
- 150 minutes, no breaks.
(Feel free to get up, use the facilities, or leave if you're bored.)
- This will be dense.
(Lots to cover, so we'll move fast.)
- But! Do interrupt at any time with questions.
(If you’re lost, you’re not the only one.)

All materials are available:
- This presentation:
http://tiny.cc/unstructure-sxsw14-slides
- Live notes:
http://tiny.cc/unstructure-sxsw14-notes
- Code & samples:
https://github.com/ivarley/unstructure-sxsw14

There’s some setup required,
but we won’t need it right away.
Probably best if everyone starts trying to
download & install stuff now, and do it in the
background as I’m talking.

Setup, part 1: git
- Download & Install Git
http://git-scm.com/book/en/Getting-Started-Installing-Git
- Clone my repo:
$ cd ~
$ git clone https://github.com/ivarley/unstructure-sxsw14.git

Setup, part 2: heroku
- Download & Install Heroku Toolbelt
https://toolbelt.heroku.com/
- Create Heroku Account
https://id.heroku.com/signup

Setup, part 3: CouchDB
- Download & Install CouchDB
http://couchdb.apache.org/

Setup, part 4: miscellaneous
- Google Chrome
https://www.google.com/intl/en/chrome/browser/
- JSONView Plugin
http://goo.gl/K07fFs

Quick Survey:
- occupation: coders? designers? other?
- technical skill: low / medium / high
- know what a relational database is?
- know SQL?
- know what NoSQL means?
- have used a NoSQL database?
- have read Aristotle? :)

There’s a spectrum in an audience like this ...
The Hacker: Wants
hands on, doesn’t care
about theory
The Academic: Wants
heady concepts, not
comfortable with code.
The “Hackademic”:
Wants enough theory
to be grounded, and
enough hacking to
know when something
is bullshit.
What I’m aiming for:

Where we’re headed:
Chapter 1 - Warming Up
Chapter 2 - Hierarchy
Chapter 3 - Relation
Chapter 4 - Mutation
Chapter 5 - Conclusion

Any questions before we get started?

Talking about data can be a little …
dry.
So, we’re going to use an example that
most people can relate to easily.

Music.
Photo credit: Josh Haner/The New York Times

● It's a domain we all know about.
○ But, informally (not usually for work or study)
● Lots of meaty concepts to think about:
○ Recording, Performances, Compositions, Bands, Venues, etc ...
● There are lots of music sites with data APIs:
○ Do512, EchoNest, Songkick, Sched.org, MusicBrainz, 7Digital, etc.
● It's on everyone's mind during SxSW.
● Also, I'm a musician and I felt like it.
Why music?

Running Example:
● listen.up - Internet company for “all things music”.
● Duh, this is fake, there’s no “.up” TLD
● What do we do? Everything!
● Including:
● Recorded music catalogs, streaming, purchase...
● Live music performance, booking & tickets...
● Licensing, royalties, compositions, lyrics …
● Instruments, lessons, repairs, classifieds …
● Anything else you can think of.

Running Example:
● listen.up - Internet company for “all things music”.
● Duh, this is fake, there’s no “.up” TLD
● What do we do? Everything!
● Including:
● Recorded music catalogs, streaming, purchase...
● Live music performance, booking & tickets...
● Licensing, royalties, compositions, lyrics …
● Instruments, lessons, repairs, classifieds …
● Anything else you can think of.
Is this a good business model? No, but who cares!

You are my team of
professional ontologists,
you’re going to figure out
what music data is out there
in the world for us to store.
(I’ll pay you in stock. It’ll be worth a fortune, trust me.)

Normally, this is where we might
do a big group brainstorm.
But it turns out those don’t work.
In fact, they hurt more than they help.
(https://en.wikipedia.org/wiki/Brainstorming#Challenges_to_Effective_Brainstorming)

So, we’ll do 3 steps:
1. Take 60 seconds and type as many music-
related concepts as you can.
- concepts, not proper nouns (i.e. "band", "composer", "instrument";
not "Radiohead", "Beethoven", "guitar", etc.)
- make them singular ("band", not "bands")
- not sure if it’s music-related? put it anyway.
2. Dump them into a shared google doc.
3. I’ll lowercase, dedupe, and publish.

Go!I’ll add a few prompts in case you are getting stuck ...

So, we’ll do 3 steps:
1. Take 60 seconds and type as many music-
related concepts as you can.
- concepts, not proper nouns (i.e. not "Radiohead", but "band"; not
"Beethoven" but "composer", not "guitar" but "instrument", etc.)
- make them singular ("band", not "bands")
- not sure if it’s music-related? put it anyway.
2. Dump them into this google doc:
http://tiny.cc/unstructure-sxsw14-terms
3. I’ll lowercase, dedupe, and publish here.

Now, your job is to make some sense of this.
1. Break into groups of ~3 neighbors.
a. Introduce yourselves like civilized human beings.
2. Organize this list however you want to!
a. Group things together
b. Indent things
c. Draw lines in a drawing program
d. etc.

Go!We’ll take about 10 minutes for this.

Let’s discuss a few sample things people did.
(Goal is to "sample", not for everyone to share! We don’t have all day.)
● How did you sort or group the terms?
● Did you end up with a flat list, or hierarchy?
● Did anything not fit in?
● Any higher level organization of terms?
(Note: there’s no right answer here ... yet.)

Good job.
is off to a great start.
Now it’s time to get into the meat.
What is structure? What is data?

This is a story in three parts:
Hierarchy
Relation
Mutation

These are 3 successive viewpoints that
will gradually open our eyes to the
deep structure of data.

“Structure? Isn’t this workshop supposed
to be about unstructured data?”

Poppycock. You want to see some
real unstructured data?

Unstructure: Smashing the Boundaries of Data (SxSWi 2014)

What most people mean when they say
unstructured data is:
“flexibly structured data”
Or possibly:
“data we don’t know the structure of yet”
(We’ll get to both of those; hold your horses.)

There are lots of boundaries to
smash in the world of data.
But we have to learn to structure
before we can unstructure.
Image credit: Rodrigo Diaz Aravena

Our minds are very fluid, and can
connect concepts in subtle ways.
Our computers aren't. They need concrete
instructions to structure & connect data.

The next three chapters will be a deep
dive into how concepts are combined to
let us represent the world in computers.
This is usually called “modeling”.
Modeling gets a bad rap.

Models cut away the accidental
and leave the essential.
Model != diagram, drawing
Model == Skeleton, Essence, Abstraction

Most of the time, it’s good to remember:
“the map is not the territory”.
But for the next couple hours, for us, the
map is precisely the territory.
We are data cartographers.
- Alfred Korzybski

So let’s dive in!
photo credit: Michael Mundy

We all have a pretty good intuitive
understanding of “data”. What’s yours?

data = plural of datum
datum = Latin for “given”
so ... data is “givens”?
aka “facts”?

What’s the simplest fact? A bit:
1 / 0
on / off
yes / no
true / false
“Are the lights on in this room?”

When you want more complex facts, you
chunk together simpler ones.
In one dimension, that's a "list":
● Byte = list of bits → 01000010 = 66 = “B”
● Word = list of characters → [B,e,a,t,l,e,s]
● Phrase = list of words -> “The Beatles are a band”
● And so forth ...

Of course, a 1-dimensional list
is just one (very simplistic) way
to chunk things together.

Photo credit: http://thekingoflimbspart2.com/radiohead-setlists/radiohead-2012-setlists/radiohead-setlist-houston-texas-3032012/

So you just chunk datums together,
and you get information,
knowledge, wisdom … !?
Photo By Karora (Own work) [Public
domain], via Wikimedia Commons

No. It’s not quite that easy.
f∆ƒ˙73f˚=£ƒ••XMbritneysp3ars-giraffe
is a complex structure, but it lacks
something: “meaning”

Getting a little more haughty,
we could say that the “givens” of
structured data are really
pointers to concepts.
Without at least some concept,
it’s not data: it’s noise.

But that raises all kinds of questions.
What are concepts?
What can we say about them?
What kinds of concepts are there?
What’s the difference between a
concept and the thing it points at?
Who could answer such questions?

When the epistemological going gets tough,
the tough call ...
Aristotle, 384 –322 BC

Aristotle wrote a treatise called
The Praedicamenta
(The Categories)
It’s not clear exactly what he was
categorizing (he didn’t say), but the list
stands to this day as a pretty damn
sensible way to, well, categorize.

Aristotle’s 10 "categories":
substance Stuff, essence; matter, but also universal concepts
quantity How much? How many?
quality What kind? Of what nature?
relation More, less, double, half, stronger, weaker, etc.
place Where?
time When?
position Being situated on, in, next to, sitting, touching, etc.
having Possession, state like “clothed” or “armed”
causing What did it do, make happen?
being caused What happened to it, what did it undergo?

You don’t have to agree with Aristotle’s
categories (you’d be in good company).
(But you’re also unlikely to have a sudden inspiration about it
that hasn’t already been the subject of 12 papers and a dissertation.)

But having some version of Aristotle’s list
is hard to argue with, experientially. We
sort the world into a hierarchy of
concepts; everything in its right place.

Without concepts,
we don’t have data.
We just have noise.

Of course, you can’t just store a concept
on a disk, or send it over a network.
So, not long after we had computing
machines, folks set to work figuring out
how to map and store our concepts in
the unforgiving realm of silicon.

Let’s take a trip back in time.
Photo By NASA Ames Research Center (NASA-ARC) (NIX A-
28284) [Public domain], via Wikimedia Commons

The year is 1966.
NASA is working on Saturn V and Apollo rockets,
and they can’t figure out how to store this
ginormous bill of materials. They ask:
Could these new “computers” help?

IBM: “Hey NASA! We made an
system for you to manage
information!”
NASA: “Groovy! What’s it called?”
IBM: …
...
“Information Management System.”

Thus began the not-extremely-exciting era of
hierarchical databases.
● Data is stored in records, which can have sub-records.
● There's a single strict hierarchical arrangement.
● To access data, you need to know the hierarchy.

For example:
Show Band Time
House
Of Vans
Charlie
XCX
4:15
House
Of Vans
Pusha
T
5:00
Show Venue Date
House Of
Vans
The
Mohawk
3/13/14
Chaos in
Tejas
Iron And
Lace
3/14/14
Band Song Order
Charlie
XCX
You 1
Charlie
XCX
Super
Love
2
To get to the set list, you have to
navigate through the show, to the
band, to the song.
And, you only get to choose one
hierarchy to store things in.

That sounds arbitrarily restrictive.
Why did they make it like that?

To understand,
we have to
talk about
dimensions.

We already talked about zero dimensions:
And about one dimension:
point = bit = on/off = true/false
line = list = array

It’s pretty straightforward to see
how you’d represent these,
with a series of zeros and ones.
What about 2 dimensions?

It’s a plane! (aka table, grid, matrix, spreadsheet, etc)

You're certainly familiar with the
world's most ubiquitous 2-
dimensional data tool ...

X dimension
Y dimension
Excel!
Photo credit: http://decentralist.wordpress.com/2012/10/01/libreoffice-vs-openoffice-not-always-simple/

(Or, its successor,
Google Spreadsheets,
which is way better, IMHO.)
((Especially in combination with Google Forms, which make it extremely useful.))

Spreadsheets are totally flexible.
This is a blessing and a curse.

Or poorly ...
(Fun read: http://www.epmchannel.com/2013/02/22/is-excel-the-most-dangerous-piece-of-software-in-the-world/)

Or awesomely ...
Image credit: http://gadgetose.com/excel-stop-motion-music-video/

But you get the point. 2-dimensional data is
everywhere.

2-dimensional data
doesn’t always look
like a table or grid.
We just mean that it’s
“conceptually planar”: two axes,
each w/ a set coordinate system.
(Here, rows = “web results”, and
columns = “link name”, “url”,
“description”, and “image”. No link
has two URLs, for example.)

However! We are now faced with a choice,
because we can still only actually store things
in linear form (a single stream of bits).
So do we put rows inside columns,
or columns inside rows?

To linearize the two dimensions in
a table, I can either ...

<table>
<row>
<col>Creep</col>
<col>1993</col>
<col>Pablo Honey</col>
</row>
<row>
<col>No Surprises</col>
<col>1997</col>
<col>OK Computer</col>
</row>
<row>
<col>Lucky</col>
<col>1997</col>
</row>
<row>
<col>Karma Police</col>
...
</row>
</table>
Go row-wise ...

<table>
<col>
<row>Creep</row>
<row>No Surprises</row>
<row>Lucky</row>
<row>Karma Police</row>
<row>Fake Plastic
Trees</row>
</col>
<col>
<row>1993</row>
<row>1997</row>
<row>1997</row>
<row>1997</row>
<row>1995</row>
</col>
<col>
<row>Pablo Honey</row>
...
</col>
</table>
Or column wise ...

<table>
<row>
<col>Creep</col>
<col>1993</col>
</row>
<row>
<col>No Surprises</col>
<col>1997</col>
</row>
<row>
<col>Lucky</col>
<col>1997</col>
</row>
<row>
<col>Karma Police</col>
...
</row>
</table>
But I can’t have it both ways.
<table>
<col>
<row>Creep</row>
<row>No Surprises</row>
<row>Lucky</row>
<row>Karma Police</row>
<row>Fake Plastic
Trees</row>
</col>
<col>
<row>1993</row>
<row>1997</row>
<row>1997</row>
<row>1997</row>
<row>1995</row>
</col>
<col>
<row>Pablo Honey</row>
...
</col>
</table>
?
(Unless I store it twice.)

Because we read left-to-right and top-to-
bottom, most of our systems store tables that
way too (row-wise).
But it’s not mandatory, of course.

Whichever way you choose, you can’t
get around the fact that you have to
choose an ordering of dimensions.

That makes sense
for 2 dimensions.
But what about …
3+ dimensions?

Computer science has had the idea of
multi-dimensional arrays
since the beginning.
It's a straightforward extension to a table, conceptually. It's
very hard to visualize more than 3 unless you’re on dope*.
* - This is a verrrry funny joke because multidimensional arrays use locators called dope vectors. Ha ha ha hmm.

But you’re still making it linear when
you store it.

Imagine storing the venues, shows,
bands, and songs in one file.
That’s 4 dimensions.
(Each venue has many shows;
each show has many bands;
each band has many songs; etc.)

Venue: The Mohawk
Show: Vans Day Party, 3/13/14
Band: Eagulls (12:30 p.m.)
Song: Nerve Endings
Song: Tough Luck
Band: DJ Rashad (1:15 p.m.)
Song: Holiday
Song: I Can Feel It
Band: Kelela (2:15 p.m.)
Song: ...
Band: Charli XCX (3:15 p.m.)
Song: ...
Band: Dum Dum Girls (4:15 p.m.)
Song: ...
Band: Pusha T (5:00 p.m.)
Show:
Venue: ...
Show: ...
Band: ...
Song: ...
etc.
Song: Nerve Endings
Band: Eagulls
Venue: The Mohawk
Song: Nerve Endings
Band: Eagulls
Venue: The Mohawk
Show: Official Showcase, 3/14/14
Venue: The Mohawk
Song: Holiday
Band: DJ Rashad
Venue: The Mohawk
Band: Cattle Decapitation
Show: Chaos In Tejas, 3/10/14
Venue: Iron And Lace
Song: ...
Band ...
Show: ...
etc.
By venue → show → band → song By song → band → show → venue

No matter how many dimensions,
storing the data still requires that you
pick a single primary orientation.
So Big Blue’s design choice makes a
little more sense now, right?

BTW, If you've used an
ATM recently…
you're an IMS user.
(It’s not as obsolete as it sounds.)

How do you actually store the linearized data?
You put it in a format.

Creep•••••••••••••1993Pablo•Honey
No Surprises••••••1997OK•Computer
Lucky•••••••••••••1997OK•Computer
Karma Police••••••1997OK•Computer
Fake Plastic Trees1995The•Bends••
Fixed-width files were all the rage in the 1960s.
Row delimiter is a line break; column delimiter is a pre-set
agreement about how many characters are in each line.
This is wasteful, brittle, and hard to read.

"Creep","1993","Pablo Honey"
"No Surprises","1997","OK Computer"
"Lucky","1997","OK Computer"
"Karma Police","1997","OK Computer"
"Fake Plastic Trees","1995","The•Bends"••
Delimited files (e.g. CSV, comma separated values):
Row delimiter is still a line break; column delimiter is variable (a
comma, in this case). Optionally, also “qualifiers” (quotes, here).
This is a decent format (but, Microsoft’s version really screwed things up for everyone.)

<row>
<col>Creep</col>
<col>1993</col>
</row>
<row>
<col>No
Surprises</col>
<col>1997</col>
</row>
<row>
<col>Lucky</col>
<col>1997</col>
</row>
There’s also markup (e.g. HTML)
“Tags” (<tag></tag>) give you the
start and end of rows, and the
start and end of columns within
those rows.
SGML, HTML, XML, all follow this
approach.

Song: Creep
Year: 1993
Album: Pablo Honey
%
Song: No Surprises
Album: OK Computer
Year: 1997
%
Year: 1997
Song: Lucky
Album: OK Computer
...
Or even YAML (“YAML Ain’t Markup Language”)
(Like email headers.)
Column pointers (names) are
inline with the values; rows have
many lines, and are delimited by
another character (e.g. “%”).
This is obviously more flexible,
but still inherently hierarchical.

And there are a couple modern
technologies that are hierarchical all
the way down.

In practice, the only form of markup
people use for storing data is XML.
And the most prevalent subset of YAML
people use for storing data is JSON.
(Technically JSON isn’t a subset of YAML but you shut up.)

XML was early obvious choice, because
we were used to HTML, so we all "got" it.
<?xml version="1.0"?>
<venue name="The Mohawk">
<show title="Vans Day Party, 3/13/14">
<band name="Eagulls" time="12:30 p.m.">
<song order="1">Nerve Endings</song>
<song title="Touch Luck" order="2" />
</band>
<band name="DJ Rashad" time="1:15 p.m.">
<song title="Holiday" order="1" />
<song title="I Can Feel It" order="2" />
</band>
</show>
</venue>

But XML isn’t very human-friendly.
● It’s verbose
● The wrong things grab your eye
● It’s somewhat complicated to parse
● Distinction between attributes and tag contents
is confusing.
“XML is crap. Really. There are no excuses. XML is nasty to parse for humans, and it's a disaster to parse
even for computers. There's just no reason for that horrible crap to exist.” - Linus Torvalds, Yesterday
(2014-03-06), on Google+

Now JSON is winning ...
{
"venue": {
"name": "The Mohawk",
"show" : {
"title": "Vans Day Party, 3/13/14",
"bands": [{
"name": "Eagulls", "time": "12:30 pm",
"songs": [
{"title": "Nerve Endings", "order": 1},
{"title": "Touch Luck", "order": 2}
]},
{
"name": "DJ Rashad", "time": "1:15 pm",
"songs": [
{"title": "Holiday", "order": 1},
{"title": "I Can Feel It", "order": 2}
]}
]
}
}
}

JSON:
● Where XML looks like a mass of text, JSON
looks spacious (little clutter)
● Fast to parse, for humans and computers
● Self-describing, flexible format
● Extremely simple syntax (one page)

from http://www.json.org/

One more thing: so far, we’ve talked
about a certain kind of hierarchical
relationship: containment.
But there’s another kind worth
mentioning: generalization.

This kind of relationship is common in
programming (it’s called superclassing).
But it’s uncommon (at least, explicitly) in
database systems.
We’ll come back to it later.

Hands on!
You’re doing some corporate espionage for
listen.up. You notice that do512.com seems to
have a good way to organize data in their API.
What can you steal, er, learn?
Note: the guys at do512 are friends of mine and I am in no way encouraging anyone to perform any
actual corporate espionage, no matter how cool that sounds. Listen.up is a made up company, do512’
s json API is open, and this is an exercise for learning; no stealing anything. :)

Hands on!
● Make sure you’ve got the JSONView extension in Chrome
● Go to: http://2014.do512.com/events.json
● Explore the hierarchical data that comes up
● Try “-” to collapse all, click “+” signs to unfold sections
● Also try:
○ http://2014.do512.com/venues.json
○ http://2014.do512.com/artists.json

Hands on!
● What’s the hierarchical structure of this data?
● In the same way we talked about nesting bands, venues,
shows, etc, … what are the objects being nested here?
● Would you store it differently?
● Are there any superclass / subclass relationships?

So, hierarchy! Pretty great, right?

Actually, no; it’s quite problematic if you
use it as the method for storing data.
● Lots of stuff isn't naturally hierarchical.
● You can't change the organization without changing all the
code that accesses data.
● Above 3 dimensions, the number of possible access paths
goes up dramatically! (Exponentially, in fact.)
○ The academic literature of the 60s and 70s is full of
papers describing how to do this better or faster.

But, what are you gonna do? Computers
are just boring & hierarchical, so you’d
better learn to deal with it.

Meanwhile, outside the offices of IBM,
the revolution of the ‘60s was happening.

The desirability and feasibility of
hierarchy, especially a single universal
hierarchy, was seeming less and less
important.

Reconciliation seemed impossible.
And then ...

something wonderful happened
THE MONOLITH IN KUBRICK'S 2001: A SPACE ODYSSEY (1968)

Specifically,
the math nerds
beat the business jocks.

This is Dr. Edgar F. Codd (1923-2003)
He worked for IBM in the 1960s, and
couldn’t stand the thought of our rich,
interconnected world being subjugated to
storage in hierarchical databases.
So he came up with a radical theory.
image from wikipedia:
http://en.wikipedia.org/wiki/File:Edgar_F_Codd.jpg

Two intellectual ancestors:
Set Theory Graph Theory

Set Theory:
Sets are collections of
objects. You can precisely
describe operations on sets:
● Union
● Intersection
● Difference
● Cartesian Product

Graph Theory:
Graphs are collections of nodes, connected by
edges.
Not this:

Graph Theory:
Graphs are collections of nodes, connected by
edges.
This:
Think: a social network where
the nodes are people and the
edges are friend relationships.

Codd’s genius was combining these and
proposing a declarative (rather than
imperative) access model.
● The relational model is a graph of sets
● Relations (tables) are sets of tuples (rows).
● Some attributes (columns) are edges that let you
connect the sets in interesting ways.
● You never specify “how” to get to data, just “what”
data to get, based on sets.

When he proposed this as a
way to actually store data in 1969,
people thought he was from Mars.
Everyone said “It’ll never work, computers are too slow.”
His employer, IBM, said
“Thanks but no thanks; we’ll just keep selling IMS.”

But the haters didn’t bank on two things.

1: Moore’s Law
That graph is exponential,
not linear.
We can have plenty of CPU.

2: Programmer Time
As programmer time became
more valuable than computer
time, ease of representing the
problem domain became a
dominating factor.

Due to some tomfoolery at IBM,
Codd’s “Alpha” never made it.
But another group at IBM created a quasi-
relational version called SEQUEL, which
looked kind of like COBOL.
Then in 1979, Larry Ellison copied the design
to create Oracle and SQL (SEQUEL was
trademarked). And the rest is history.

For his part, Codd waged a
decades-long battle to get a more
true representation of the relational
model adopted.
But, it never was. SQL was king.

So what is the relational model, then?

Occupational Hazard:
Abstraction Vertigo
Concrete
Data
Metadata
The Structure
of Metadata
Band name: Radiohead
Year formed: 1985
Entity: Band
Attributes: Name, Year Formed
Concepts →
Entities, Attributes

Safety first.
If you feel dizzy, just ask a question.

Entity
Attribute
Relationship
There are 3 foundational concepts:

Entity
Attribute
Attribute
relationship
Entity
Attribute
Attribute

Show
Start time
Cover $
is at
Venue
Address
Age Req.

Vans Day Party
12pm
Free
is at
The Mohawk
123 Red
River
All Ages

Notice the subtle shift here.
Neither of those entities is “inside” the
other. They’re both “first class” entities,
and they’re in a relationship.
The relationships are described at the level of sets,
not ad hoc. Shows can be at venues, categorically.

Entities look pretty much exactly like 2-
dimensional tables, except for the
concept of a “key”.
That’s the attribute (or set of attributes)
that distinguishes this row from that row.

Show
Start time
Cover $
is at
Venue
Address
Age Req.
Show Start time Cover $
1234 8pm $5
5678 11pm $9
Venue Address Age Req.
The Mohawk 123 Red Riv. 21+
Beerland 456 Red Riv. All Ages
key
key

Keys describe an entity’s identity.
In practice, most systems today use surrogate
keys (i.e. IDs) to establish unambiguous identity.
eg: integers (123456), codes (X74-UUA2),
GUIDs

Show
Start time
Cover $
is at
Venue
Address
Age Req.
Show Is At Start time Cover $
1234 The Mohawk 8pm $5
5678 Stubb’s 11pm $9

Then, to get connected information out,
you traverse the relationships with
something called a JOIN.

Show Is At Start time Cover $
1234 The Mohawk 8pm $5
5678 Stubb’s 11pm $9
Show Is At Address Age Req. Start time Cover $
1234 The Mohawk 123 Red Riv. 21+ 8pm $5
+
=

Tuple is a fancy word for “row”.
(It comes from abstracting … septuple, octuple, N-tuple …)

Cardinality means, how many tuples of
one relation can match each tuple in
another relation?
(In English: a show is at one venue,
but a show can have many bands.)

Show
Start time
Cover $
played by
Band
Name
Genre
1234 8pm $5
5678 11pm $9
Band ID Name Genre
RDOHD Radiohead Rock
EGLS Eagulls Indie Rock
many to many?

Show
Start time
Cover $
played by
Band
Name
Genre
Show ID Start time Cover $
1234 8pm $5
5678 11pm $9
Band ID Name Genre
RDOHD Radiohead Rock
EGLS Eagulls Indie Rock
Show ID Band ID Start Time
1234 RDOHD 9:30pm
5678 EGLS 9pm
5678 RDOHD 11pm

Normalization is the idea that if
something repeats (i.e. has a cardinality
of more than 1), it should be expressed
as another entity, not as repeating data.

Show Date Venue Street City State
Austinist Party! - Day 1 3/13 Mohawk 123 Red River Austin TX
No:
Yes: Show Date Venue
Austinist Party! - Day 1 3/13 Mohawk
Venue Street City State
Mohawk 123 Red River Austin TX

Basically, normalization dictates that
the same fact is never repeated
in more than one place.

There’s nothing wrong with this data per se, but we all know
that the final 4 columns refer to the same thing.

Austinist Party! - Day 2 3/14 Mohawk 985 Congress Austin TX
There’s nothing wrong with this data per se, but we all know
that the final 4 columns refer to the same thing.
What would it mean if the street address were different in
one of these? Are there two Mohawks?

Normalization gives you consistency,
also known as relational integrity.
Certain kinds of problems (like that one)
just can’t happen, because there’s
literally only one place where you store
the address of the Mohawk.

Show Date Venue
Venue Street City State
Mohawk 123 Red River Austin TX

Normalization also means that the
attributes of an entity are non-repeating.
(i.e. there’s no repeating columns, or groups of columns)

Show Start time Cover $ Band #1 Band #2 Band #3
House of Vans 8pm $5 Eagulls Cyndi Lauper Radiohead
Chaos in Tejas 11am $10 Metalface CRUD Decapitation
House of Vans 8pm $5
Chaos in Tejas 11am $10
Show Band Order
House of Vans Eagulls 1
House of Vans Cyndi Lauper 2
House of Vans Radiohead 3
Chaos in Tejas Metalface 1
Chaos in Tejas CRUD 2
Chaos in Tejas Decapitation 3
No:
Yes:

This makes sense, because …
what if you had 4 bands? Or 400?

Do people break these rules?
All the time!
(We’ll get to valid reasons why they might want to do that.)

One last thing:
Schema is the structure of the database
itself. It is stored as ... data!
VenueID Name Address
123 Mohawk 123 Red River
456 Stubbs 456 Red River
BandID Name # Members
4321 Eagulls 4
8765 CRUD 17
Table Column Type
Venue VenueID ID
Venue Name String
Venue Address String
Band BandID ID
Band Name String
Band # Members Integer

The final piece of the relational database puzzle:
SQL (Structured Query Language)
which is about how you get stuff
out of this graph of sets.

We obviously don’t have time to really
learn SQL, but here’s the gist:

SELECT
S.cover, B.Band
FROM
Shows S INNER JOIN ShowBands B
ON B.Show = S.Show
WHERE
S.start_time > 8pm
AND B.band like ‘%face%’
ORDER BY
order ASC

Show Start time Cover
Chaos in Tejas 9pm $10
Show Band Order
SELECT
S.cover, B.Band
FROM
ON B.Show = S.Show
WHERE
S.start_time > 8pm
ORDER BY
order ASC

SELECT
S.cover, B.Band
FROM
ON B.Show = S.Show
WHERE
S.start_time > 8pm
ORDER BY
order ASC
Show Band Order

SELECT
S.cover, B.Band
FROM
ON B.Show = S.Show
WHERE
S.start_time > 8pm
ORDER BY
order ASC
Show Band Order
$10 Metalface

SELECT
S.cover, B.Band
FROM
ON B.Show = S.Show
WHERE
S.start_time > 8pm
ORDER BY
order ASC
Show Band Order
$10 Metalface 1

So that’s relational databases and SQL,
in a very small nutshell.

You can imagine,
relational databases
get very complicated
for non-trivial things.
But you now know
almost all there is to
know: relational
databases are graphs
of sets, navigated via
declarative language.
Source: http://wiki.musicbrainz.org/-/images/5/52/ngs.png

Also, remember generalization?
Theoretically, that’s just another type of
relationship between entities.

Band Genre
Radiohead Rock
Metalface Jazz
Entertainer Since
Radiohead 1985
Metalface 2005
Gonzo the Incredible 1968
Gob 2002
Penn & Teller 1990
Magician Style
Gonzo the Incredible Sorcery
Gob Fail
Penn & Teller Conjuring

I say theoretically because nobody
actually does this in practice, since
(unlike in object oriented programming)
doing this carries a performance and
complexity burden in databases.
(But, you could do it, theoretically.)

Hands on!
We’re going to spin up a relational
database using Heroku, create and
populate some tables, and show the data
on a web site.

Hands on!
About Heroku:
● Super simple hosted engine for running any code online
● Has built-in relational database capabilities with Postgres
● Uses a version control system called git
● Uses Amazon AWS to host the code
● It's owned by Salesforce.com (my company) but run
separately; I'm not an expert
● Also note that this example is written in Ruby and I’m
really not an expert on that. Hope it works!

Hands on!
Prerequisites:
1. You should already have git installed; do now if not.
2. You should have heroku toolbelt installed, and have
created a heroku login. Do now if you haven’t.
3. You should have already cloned my repo, but if not do it
now.
$ cd ~
$ git clone https://github.com/ivarley/unstructure-sxsw14

Hands on!
Steps:
1. Go to the directory you cloned the repo into:
$ cd unstructure-sxsw14
2. Create and publish the app:
$ heroku login
$ heroku create
$ git push heroku master
$ heroku open

Hands on!
Congrats! You just published a web app.
Not convinced? Edit some of the visible text in the file:
app/views/welcome/index.html.erb
and republish:
$ git commit -am 'made an edit'
$ git push heroku master
$ heroku open

Hands on!
Now let’s add some data!
Heroku automatically created a Postgres database for you.
$ heroku pg:psql
opens a database prompt. Create a table and insert data:
CREATE TABLE band (bandid INT PRIMARY KEY, name VARCHAR, genre VARCHAR);
INSERT INTO band (bandid, name, genre) VALUES (1, 'Radiohead', 'Rock');
SELECT * FROM band;

Hands on!
Add some more tables and data:
CREATE Table venue (venueid INT PRIMARY KEY, name VARCHAR, address VARCHAR);
CREATE TABLE show (venueid INT PRIMARY KEY, bandid INT, date VARCHAR);
INSERT INTO venue (venueid, name, address) VALUES (1, 'The Mohawk', '123 Red
River, Austin, TX');
INSERT INTO show (venueid, bandid, date) VALUES (1, 1, '2014-03-07');
And create a view (basically, a saved SQL statement):
CREATE VIEW shows AS SELECT b.name as band_name, v.name as venue_name, s.date
FROM band b INNER JOIN show s ON b.bandid = s.bandid INNER JOIN venue v ON s.
venueid = v.venueid;

Hands on!
Now you can SELECT data:
SELECT * FROM shows;
And you’ll get:
band_name | venue_name | date
-----------+------------+------------
Radiohead | The Mohawk | 2014-03-07
Note if you’re into nit-picking: I'm taking a shortcut here and saying that each record in "show" is a band/venue
combination, with a date. If "show" were a proper entity (for example, if the show had a name, a promoter, etc.) then
the proper “normalized” way to model it would be to create a show entity with an ID and a venue ID, show name,
promoter, etc; and then have intersection tables between bands and shows (like a band_show table). But this is fine
for now.

Hands on!
Leave the SQL prompt (type “q”) and reopen the app:
$ heroku open
And (hopefully) voila! You see the results of:
SELECT * FROM shows;
Feel free to mess around now--add more data, to see how it’ll
show up. If you’re really advanced, try changing the ruby code
to show different SQL statements, etc. We’ll take about 10
minutes to play around, ask questions, etc.

So, hierarchical databases were lame.
But relational databases are awesome!
SQL Rules!! Right … ?

Hrm. As awesome as they are, relational
databases have a few warts.
And some of them have only become
apparent recently.

● It’s really hard to work with recursive / graph relationships
● Results of SELECT queries are always flat tables, which
means you have to reassemble nested structures yourself
● They don’t play well with object oriented programs
● They don’t support inheritance & superclassing!
● Relational modeling tools are generally quite sucky.
● They don’t scale well to extremely large data sets,
because they promise things you can only do on a single
(non-distributed) system.
There’s a standard list of grievances:

These are all interesting points; if we had
a whole semester, we could spend
weeks on any one of them.

But I want to spend the rest of our time
on what is (IMO) the essential problem:
mutability.(i.e. change, being mutated)

Let’s zoom way out for a minute.
By NASA [Public domain], via Wikimedia Commons

We talked about hierarchies.
● Old school, classical
● Single top down view of reality
We talked about relations.
● Modern, networks, connections
● No single privileged access path or view

But both hierarchies and relations
tend towards a commitment to a
fixed representation of reality.
You’re modeling static concepts that exactly match the real
world. Doing this with high fidelity is the crowning
achievement of the relational model.

But reality doesn’t hold still.
It changes, all the time.
Any fixed way of representing the
world is doomed to become outdated.

Darwin knew what was up.
The nature of reality is to be dynamic, evolving.

A species seems like a fixed thing,
but that's just a label we attach.
They are fuzzy around the edges, and always changing.

Change is a fact of life at all levels
of the abstraction ladder…
our understanding of structure itself
the structure of our software apps
facts about things in the real world

And it’s only getting
faster.
We can either hide our
heads in the sand, or we can
figure out what to do about
it.

So let’s talk about 3 ways to
smash the boundaries of data,
in the mutation dimension:
Attribute Flexibility
Non-Destructive Mutability
Model Agility

This is an UPDATE statement in SQL:
UPDATE Band
SET name = 'Jaydiohead'
WHERE Band ID = 5678
and name = 'Radiohead'
Result:
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Jaydiohead

You know what you can’t do?
Undo.
Edits in traditional relational databases
are destructive. If you want to keep the
old version, you have to do it yourself.
Note for data nerds: yes, databases keep transaction logs so you can undo and redo edits as part
of transactions. But this is (a) implementation dependent, (b) not typically exposed to users in the
relational model, and (c) not guaranteed to persist beyond the transaction itself.

This means that everyone has to either:
● Add extra complexity to their data model, or
● Accept that changes are destructive.
Both are pretty crappy options. Most people just
do #2.

But, sometimes you can’t do that.
Consider compliance.
● Who changed my salary?
● Who deleted this opportunity from the
pipeline?
● Who took my stapler?
You might not care a bunch about that, but Sarbanes-Oxley
sure does.
And if you don’t know what that is, count yourself lucky and go back to making pretty things.

So this …
So how do people store history now?
Option #1: audit columns (partial solution)
Band ID Name
1234 Bjork
5678 Radiohead

Option #1: audit columns (partial solution)
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name Create Date Created By Modify Date Modified By Deleted?
1234 Bjork 3/6/14 2pm Ian Varley 3/6/14 3pm Jan Jones false
5678 Jaydiohead 3/5/14 1pm John Smith 3/7/14 1pm Ian Varley false
So this … becomes this.

So this …
Option #2: History tables
Band ID Name
1234 Bjork
5678 Radiohead

So this … becomes this
Option #2: History tables
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name Version Date By
1234 Bjork 1 3/6/14 Ian V
5678 Radiohead 1 3/6/14 Ian V
5678 Jaydiohead 2 3/7/14 Ian V
1234 Fjork 2 3/7/14 Ian V

Neither of these are ideal.
They ugly up your data model, which is
not just an inconvenience; it makes it
harder to see the “real” stuff.
It also violates the spirit of normalization: if something means
the same thing, don’t repeat it all over the place.

But what if … the database took care of
this for you?
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Name
1234 Bjork
5678 Radiohead

Some newer ones do:
● Google pioneered this with BigTable - time is a
privileged dimension, part of the model, stored
with every datum
● NoSQL Stores like HBase follow suit
● Salesforce offers “field history” out of the box,
as a meta-feature on any entity, expressed as
a history table
● Document stores can store older versions

Does this use a lot of space? Yes.
But guess what? We have a lot of space.

And, it turns out that for a lot of
computing applications, making it
immutable makes the problem way
easier to reason about.
(For more on that, see Pat Helland’s talk,
Immutability Changes Everything.)

Here’s the crazy part. If we do this
across the board, we end up with ...
data time travel.
“K-9, show me my accounts
receivable as of last May, and
compare it with today.”

Plain old “undo” is a subset of this. Ever
wondered why Google web products all
have “undo” and nobody else’s do?
(Full revision history is the fancy version. They have that too.)

My pronouncement:
Henceforth, all databases should
make time a privileged dimension,
and retain older versions of data in
a way that supports time travel.

Historically, databases have been
a little something of a schoolmarm.
Image: Public domain. From the 1894 Laughable Lyrics: A Fourth Book of
Nonsense Poems, Songs, Botany, Music, etc. by Edward Lear.

Some things are decided and enforced
directly at the database level:
- what attributes an entity can have
- how many attributes there are
- what type of data to store in each attribute
- (string, number, date, currency, etc.)
- other constraints (max & min values, etc)

But why is this the database's job?
It’s partially a historical accident.
Disk & memory used to be scarce and
highly optimized, so record formats had
to be prescriptive and fixed.

It’s also partially a mindset.
Centralizing decisions about structure
is very tempting; it’s easy to overestimate
your ability to “get it right” the first time,
or find the “one true model” for all.

But …
What if a database's job is really
to store whatever fields I give it?
What if I want to say, "Let me store any
additional facts I can think of about
bands, venues, shows"?

If your model imposes less, then there's
less to change when the world changes.
This analogous to the difference between
strongly typed languages (like C and Java)
and scripting languages (like Ruby and Python).
Sometimes you want to trade safety for flexibility.

As an example, Thom Yorke’s
booking agent might use
address book software.
But what if they had to update
the database schema for
each new kind of social media
service he decides to use?

You can picture just adding more
attributes, like key-value pairs in JSON:
{
id: "ARP5KQF1187B9B4DD1",
name: "Explosions in the Sky",
genres: [{name: "post rock"}],
years_active: [{start: 1999}],
artist_location: {
location: "Austin, TX, US",
city: "Austin",
region: "Texas",
country: "United States"
}
}

You can picture just adding more
attributes, like key-value pairs in JSON:
{
id: "ARP5KQF1187B9B4DD1",
name: "Explosions in the Sky",
genres: [{name: "post rock"}],
years_active: [{start: 1999}],
artist_location: {
location: "Austin, TX, US",
city: "Austin",
region: "Texas",
country: "United States"
},
familiarity: 0.687572,
favorite_color: "Blue"
}

“Tagging” is the simplest version of this.
Each tag is a boolean
(true/false), and there
can be any number of
them.

You can already model attribute flexibility
at a meta-level in a relational database...
Band ID Name
1234 Bjork
5678 Radiohead
Band ID Attribute Value
5678 City Austin
5678 State TX
5678 Fav. Color Blue
5678 Familiarity 0.687572
After all, it’s just another degree of cardinality, right?

But writing queries against it is a
massive PITA (trust me).
And, it’s the same meta problem: if you did this
for every entity in your model, your model would
be impossible to comprehend.

Attribute flexibility is “table stakes” for
new DBs:
● Google did it with BigTable - the columns for
a row are totally flexible at run time, and the
values are simple byte arrays
● Most other NoSQL stores offer this too
● Some services make it the backbone of
what they offer (e.g. keen.io - 1 entity, but
any set of attributes you want to send)

My second pronouncement:
Henceforth, most databases should
really think about offering attribute
flexibility, accepting writes and
reads of “columns” that haven’t
been declared in advance.

Aside: if columns aren’t predefined,
how do we know their data type?
● Strings, Numbers, Dates, etc?
● JSON has the right idea:
○ If it’s in quotes, it’s a string
○ If it’s not, it’s either:
■ a number →
■ true or false

And … while we’re at it, why not just
get rid of attributes altogether,
and say that entities store JSON blobs,
with nesting intact?
(We’ll come back to that …)

"Schemas usually remain relatively stable
over the lifetime of a database for most
applications."
- S Navathe, 1992
"No one will need more than 637 kB of
memory for a personal computer."
- B. Gates, 1979

It’s true, though. Database schemas do tend
to remain stable … because it sucks so bad
to change them!
● Honestly, databases have always been the
least “agile” part of software development.
● It's the final frontier of "BDUF" (big design
up front)

But, wait. Aren’t schemas stored as data?
Can’t you just change them?
ALTER TABLE Band ADD COLUMN favorite_color
STRING

In theory, that works. In practice, it doesn’t.
● You can change the schema, but you can only ever
have one schema at a time
● Some changes might require downtime
● For big tables, that could mean you’re offline for hours,
days, or even weeks.
● And, by the way, if you follow my first pronouncement
about data and time travel, what happens when you
change the schema?

What to do instead?
One option is to generalize: make your model
so generic, you never have to alter a table!

This is what “architecture astronauts” do, and
it doesn’t work, because you eventually end
up with this model:
(You can sometimes find a sweet spot, but more likely you’re just
pushing the essentialism and brittleness to another layer.)

What we really need is a system where
structure changes are also non-destructive.
(We’d need that anyway if we want to time
travel with the data, right?)

There’s actually an example of this!
http://couchdb.apache.org/

Apache CouchDB is a NoSQL database.
● The database is just a flat collection of JSON files
● There’s no schema! Put in whatever you want!
● You create views (using javascript) that “materialize”
certain access patterns across your documents.
It’s not perfect (scaling is tricky, etc.) but as an illustration
of these points, it’s spot on.

CouchDB hits all 3 of our mutability desires:
● All documents are versioned.
○ Non-destructive Mutation!
● Document JSON can have any structure
○ Attribute Flexibility!
● Schema-On-Read using views
○ Model Agility!

It also happens to be wicked easy to
get started with, so let’s do one more
hands-on exercise.

Hands on!
Let’s load some music data into CouchDB!
You should have already installed CouchDB, but
if not, do so now.

Hands on!
Earlier, we used the Do512 API. This time we’ll
use The Echo Nest, a huge music data
repository.
(Which was, incidentally, just bought by Spotify yesterday, March 6th, 2014! Too
bad we didn’t get on this listen.up thing a little sooner amiright?)
They require setting up an API key for access, so
as a shortcut I’ve done that part for you.

Hands on!
The file mutation/echonest-artists-austin-v1.json
is data on the top 100 “most familiar” artists with
a hometown of Austin, TX, from this API call:
http://developer.echonest.com/api/v4/artist/search?api_key={MY_API_KEY}
&format=json&results=100&start=0&bucket=familiarity&bucket=genre&bucket=artist_location&bucket=years_active&artist_l
ocation=austin
Full disclosure, I modified the result slightly, so it’d work immediately with CouchDB bulk load:
● removed the outer "response" wrapper from the API
● changed the name of the array from "artists" to "docs"
● changed all the "id" fields to "_id" so CouchDB would use them

Hands on!
Ensure CouchDB is installed and running:
http://127.0.0.1:5984/_utils/index.html
Create a new
database called
“listenup”

Hands on!
Upload a bunch of data:
$ cd ~/unstructure-sxsw14/mutation/
$ curl -H "Content-Type:application/json" -d @echonest-arists-austin_v1.
json -X POST http://127.0.0.1:5984/listenup/_bulk_docs
Reload the database web page:
http://127.0.0.1:5984/_utils/database.html?listenup

Hands on!
See what happens when you change a value.
Let’s pick a band at random:
http://127.0.0.1:5984/_utils/document.html?listenup/ARJXDMJ11FF10D84F9
Double click, and change to:
[{"name": "garage
soul"}]
Then click “Save Document”.

Hands on!
Now there are two versions of this document:
Non-destructive mutability, in the flesh!

Hands on!
Now add a field that never existed before:
http://127.0.0.1:5984/_utils/document.html?listenup/ARJXDMJ11FF10D84F9
Click “Add Field”.
Add “Influences” as the
Field name, and click
“Save Document”.
Double click the “null” next to “Influences”,
and change the value to
[{"name": "Michael McDonald"}]
Click “Save Document”.

Hands on!
Attribute flexibility!
Notice that so far,
we haven’t once
had to specify a
schema.

Hands on!
Now we query the data by writing code to
implement materialized views. This is actually a
little complicated ...

Hands on!
(Yes, I know this is actually about Riak, not CouchDB. Why do you hate laughing?)
Comic by John Muellerleile / http://thinkdifferent.ly/fault-tolerance.png

Hands on!
As a simple example, create a new document:
{
"_id": "_design/application",
"views": {
"genre-view": {
"map": "function(artist) {if(artist.name && artist.genres)
{artist.genres.forEach(function(genre) {emit(genre, artist.name);});}}"
}
}
}
Then visit: http://127.0.0.1:5984/listenup/_design/application/_view/genre-view

Hands on!
Try making your own view. Some examples are
here:
http://guide.couchdb.org/draft/cookbook.html

Chapter 5: Conclusion & Future Directions

Before we go off to our subsequent
debauchery, if you’ll indulge me, we
actually have a tiny bit more
smashing to do.

Now it’s time to go through the looking glass.
Image from Disney Wikia: http://disney.wikia.com/wiki/Alice

(Some of these thoughts are adapted from my previous
presentation, I’ve Always Wanted To Data Model)

This is a technical
book from the 1970s.
It’s a philosophy book.
It opened my eyes to
some of the real
underlying questions.

Of course not, not categorically.
Member Year Joined
Thom Yorke 1985
Jonny Greenwood, 1985
Ed O'Brien, 1985
Colin Greenwood 1985
Philip Selway 1985
Ian Varley 2014
Official Radiohead Band Members

Robin Hood’s Band Of Merry Men
Depends on what you mean.
Merry Man Merry-ness
Robin Hood High
Little John Medium
Much The Miller’s Son Medium
Friar Tuck High
Arthur a Bland Low
Maid Marian Unknown

My personal theory:
Data is “existential claims”.
The fact that this data, and not some other, is stored
implies that someone (or something) is making a claim
about the existential state of something. This claim
may or may not correspond to the actual existential
state of that something.

As cool as the boundary-smashing tools we
looked at today are, it’s also wise to
remember that we don’t really know much
of what’s going on. For example ...

Nobody actually knows
what an “entity” really is.

“Entity” is another word for Category,
in linguistics terms.
And an important property of linguistic
categories is that they are slippery.
See:
● Steven Pinker: The Stuff Of Thought
● Douglas Hofstadter: Surfaces & Essences
● George Lakoff: Women, Fire, and Dangerous Things

part: an abstract definition of a
connected set of physical
materials that serve some
purpose
part: one instance of a
manufactured item, which
either does or does not meet
quality standards
Images: (1) Atwood Hydraulic Surge Brake Actuator, http://www.pacifictrailers.com/Atwood-Hydraulic-Brake-Actuator-Parts-List-and-Schematic/; (2) Ford Motor Company flywheel magneto assembly line 1913, source unknown

And if you think you can “solve” the
problem, I’ve got some World Trade
Center insurance policies to sell you.

That said, there are a couple techniques
we could adopt that would help:
● First-class Sub- / Super-Typing
● First-class Scoping and Aliasing
(Not that there aren’t ways to do this in relational models, but
they’re unobvious and not widely used.)

Discrete models encourage
black & white thinking in a
gray world

Sometimes the deep structure is
actually ambiguous.
Image credit: By Chire (Own work) [Public domain or Public domain], via Wikimedia Commons

Our current tools are s#!t.
Boxes & lines aren’t how we actually
think, our spatial processing of
diagrams doesn’t map well to our
temporal, spatial, and causal
comprehension of data structure.

If we had the right tools, what would
they look like?

They’d have native support for ...
● My mutability requirements
○ Non-destructive mutation
○ Attribute Flexibility
○ Model Agility
● The 3 Ps:
○ Provenance, Provability, Probability

They’d have native support for ...
● Supertyping
● Extensible Meta-Metadata
● Semantic Zoom
● Prototype Generation
● Model Versioning and Diffing

Interesting direction: RAML
● Textual (YAML-based)
● Strong, simple syntax
● Generates useful models
● Communication focussed
Could there be something
like this for data models?

Thanks!
@thefutureian
ianvarley.com
(If you enjoyed this, please rate it a 5 … sxsw.com/rate)

Unstructure: Smashing the Boundaries of Data (SxSWi 2014)

More Related Content

Similar to Unstructure: Smashing the Boundaries of Data (SxSWi 2014) (20)

Recently uploaded (20)

Unstructure: Smashing the Boundaries of Data (SxSWi 2014)