Using Spring with NoSQL databases (SpringOne China 2012)

Using Spring with NoSQL databases
Chris Richardson,
Author of POJOs in Action, Founder of the original CloudFoundry.com
@crichardson chris.richardson@springsource.com http://plainoldobjects.com/

Presentation goal
NoSQL databases: what, why and
how
How Spring Data simpliﬁes the
development of NoSQL applications

About Chris

http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/

vmc push About-Chris

Developer Advocate

Signup at http://cloudfoundry.com

Agenda

• Why NoSQL?

• Overview of NoSQL databases

• Introduction to Spring Data

• Using Spring Data for Redis

• Using Spring Data for Mongo

• Deploying on Cloud Foundry

Relational databases are great...

• SQL • Well supported

• High-level • JDBC

• Sorting • Hibernate/JPA

• Aggregation • Spring

• ACID semantics • Well understood

• Developers

• Operators

... but they have limitations

• Object/relational impedance mismatch

• Complicated to map rich domain model to relational schema

• Difﬁcult to handle semi-structured data, e.g. varying attributes

• Schema changes

• Extremely difﬁcult/impossible to scale

• Poor performance for some use cases

Solution: Spend Money

OR
http://upload.wikimedia.org/wikipedia/commons/e/e5/Rising_Sun_Yacht.JPG

• Hire more DevOps
• Use application-level sharding
• Build your own middleware
• …

http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#

Solution: Use NoSQL
Benefits
• Higher performance
• Higher scalability
• Richer data-model
• Schema-less
Drawbacks
• Limited transactions
• Relaxed consistency
• Unconstrained data

Solution: Use NewSQL
• Relational databases with SQL and ACID transactions
AND
• New and improved architecture - designed for modern hardware
• Radically better scalability and performance

• NewSQL vendors: Gemﬁre/SQLFire, VoltDB, ...

Future = multi-paradigm data storage for
enterprise applications

IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg

Redis

• Advanced key-value store K1 V1
• Written in C
K2 V2
• Very fast, e.g. 100K reqs/sec

• Optional persistence ... ...
• Transactions with optimistic locking

• Master-slave replication

• Sharding using client-side consistent hashing

Using Redis (via CLI)
Datatypes: redis 127.0.0.1:6379> set foo 1
OK
•Strings redis 127.0.0.1:6379> get foo
•Hashes "1"
•Maps redis 127.0.0.1:6379> sadd myset a
•Lists (integer) 1
•Sets redis 127.0.0.1:6379> sadd myset b
(integer) 1
•Sorted sets redis 127.0.0.1:6379> smembers myset
1) "a"
2) "b"
redis 127.0.0.1:6379> srem myset a
(integer) 1
redis 127.0.0.1:6379> smembers myset
1) "b"

Redis use cases
• Replacement for Memcached • Handling tasks that overload an RDBMS
• Session state • Hit counts - INCR
• Cache of data retrieved from system of • Most recent N items - LPUSH and LTRIM
record (SOR)
• Randomly selecting an item –
• Replica of SOR for queries needing high- SRANDMEMBER
performance
• Queuing – Lists with LPOP, RPUSH, ….
• High score tables – Sorted sets and ZINCRBY
• …

MongoDB

• Document-oriented database • Geospatial queries

• JSON-style documents: objects, lists, • Grid FS provides ﬁle storage
primitives
• Very fast, asynchronous writes
• Schema-less
• Highly scalable and available
• Transaction = update of a single
document

• Rich query language for dynamic queries

Data model = Binary JSON documents
Server

Database: Food To Go
Collection: Restaurants
{
"name" : "TGI Fridays",
"type" : ”American",
"serviceArea" : [
"94619",
"94618"
], Sequence of
"openingHours" : [ bytes on disk
{ è fast i/o
"dayOfWeek" : "Wednesday",
"open" : 1730,
"close" : 2230
}
],
"_id" : ObjectId("4bddc2f49d1505567c6220a0")
}

MongoDB CLI
> r = {name: 'Ajanta'}
> db.restaurants.save(r)
> r
{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }
> r = db.restaurants.findOne({name:"Ajanta"})
{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }
> r.type= "Indian”
> db.restaurants.save(r)
> db.restaurants.update({name:"Ajanta"},
{$set: {name:"Ajanta Restaurant"},
$push: { menuItems: {name: "Chicken Vindaloo"}}})
> db.restaurants.find()
{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "menuItems" : [ { "name" :
"Chicken Vindaloo" } ], "name" : "Ajanta Restaurant", "type" : "Indian" }
> db.restaurants.remove(r.id)

MongoDB query by example
{
serviceArea:"94619",
Find a restaurant that
openingHours: { serves the 94619 zip
$elemMatch : { code and is open at 6pm
"dayOfWeek" : "Monday",
"open": {$lte: 1800}, on a Monday
"close": {$gte: 1800}
}
}
}

DBCursor cursor = collection.ﬁnd(qbeObject);
while (cursor.hasNext()) {
DBObject o = cursor.next();
…
}

Scaling MongoDB
Shard 1 Shard 2
Mongod Mongod
(replica) (replica)

Mongod Mongod Mongod Mongod
(master) (replica) (master) (replica)

Config Server

Mongos
mongod Mongos

mongod

Client
mongod

MongoDB use cases

• Use cases • Who is using it?

• High volume writes • Shutterﬂy, Foursquare

• Complex data • Bit.ly Intuit

• Semi-structured data • SourceForge, NY Times

• GILT Groupe, Evite,

• SugarCRM

Other NoSQL databases
Type Examples

Extensible columns/Column-oriented Hbase
e
SimpleDB, DynamoDB
r it
avo
Cassandra
r f
you
Graph Neo4j
t
t ou
Key-value
I lef Voldemort, Riak
if
Document So rry CouchDb

http://nosql-database.org/ lists 122+ NoSQL databases

Spring Data is here to help

For

NoSQL databases

http://www.springsource.org/spring-data

Spring Data sub-projects
• Relational • QueryDSL
• JPA • Big Data
• JDBC Extensions • Hadoop
• NoSQL • HDFS and M/R

• Redis • Hive

• Mongo • Pig

• HBase • Cascading

• Neo4j • Splunk

• Gemﬁre • Access

• Lucene • REST

What you get

• Template classes that hide the boilerplate code

• Auto-generated (generic) repositories for some NOSQL databases

• Java NoSQL mapping

• Cross Store Persistence

• Support in Roo

Redis challenges

• Connection management: need to get and reliably close connections

• Data mapping: application objects Redis binary/strings

• Multiple client libraries with gratuitously different APIs

Spring Data for Redis

• Low-level - RedisConnection(Factory) • Connection management

• Supports Jedis, Jredis, Rjc and Srp • Pluggable Java binary conversion

• Insulates client code from underlying • Support classes:
library
• Collections-backed by RedisTemplate
• High-level - RedisTemplate
• Atomic Counters
• Builds on RedisConnection(Factory)
• Support for Redis pub/sub

Low-level API = RedisConnection(Factory)

Using RedisConnectionFactory
public class LowLevelRedisTest {

@Autowired private RedisConnectionFactory redisConnectionFactory;

@Test
public void testLowLevel() { Library independent code J
RedisConnection con = null;
try {
con = redisConnectionFactory.getConnection();

byte[] key = "foo".getBytes(); Ugly byte arrays L
byte[] value = "bar".getBytes();
con.set(key, value);

byte[] retrievedValue = con.get(key);

Assert.assertArrayEquals(value, retrievedValue);

} ﬁnally {
if (con != null) { con.close(); }
Need to clean up L
}
}

Configuring RedisConnectionFactory
@Configuration
public class RedisConfiguration {

@Value("${databaseHostName}")
protected String databaseHostName;

@Bean
public RedisConnectionFactory jedisConnectionFactory() {
JedisConnectionFactory factory = new JedisConnectionFactory();
factory.setHostName(databaseHostName);
factory.setPort(6379);
factory.setUsePool(true);
return factory;
}

}

High-level API = RedisTemplate

• Builds on RedisConnection(Factory) • Maps Redis exceptions
DataAccessException
• Analogous to JdbcTemplate
• StringRedisTemplate
• Parameterized type
• Extends RedisTemplate<String,
• K - Key type
String>
• V – Value type
• Keys and values are Strings
• Handles Java Key/Value Redis byte[]

Using StringRedisTemplate
public class RedisTemplateTest {
@Autowired private StringRedisTemplate stringRedisTemplate; Returns KV type speciﬁc interface
@Test
public void testGetAndSet() {
stringRedisTemplate.opsForValue().set("foo", "bar");
assertEquals("bar", stringRedisTemplate.opsForValue().get("foo"));
}
@Test Converts between Strings and byte[]
public void testHashOps() {
stringRedisTemplate.opsForHash().put("myHash", "myKey", "value");
assertEquals("value",
stringRedisTemplate.opsForHash().get("myHash", "myKey"));
assertEquals(Collections.singleton("myKey"),
stringRedisTemplate.opsForHash().keys("myHash"));
assertEquals(Collections.singletonMap("myKey", "value"),
stringRedisTemplate.opsForHash().entries("myHash"));
}

Conﬁguring StringRedisTemplate
@Conﬁguration

@Bean
public RedisConnectionFactory jedisConnectionFactory() {
…
}

@Bean
public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory factory) {
StringRedisTemplate template = new StringRedisTemplate();
template.setConnectionFactory(factory);
return template;
}
}

RedisTemplate: Java objects binary data

• DefaultSerializer - defaults to JdkSerializationRedisSerializer

• KeySerializer

• ValueSerializer

• HashKeySerializer

• HashValueSerializer

StringRedisTemplate uses StringRedisSerializer

• Stores keys and values as Strings

Register serializers to override the default
behavior

Converted to JSON by RedisTemplate

Redis-backed Collections
The key

@Test
public void testRedisSet() {

Set<String> mySet =
new DefaultRedisSet<String>("mySet", stringRedisTemplate);

Assert.assertTrue(mySet.isEmpty());

mySet.add("a");

assertEquals(Collections.singleton("a"),
stringRedisTemplate.opsForSet().members("mySet"));

}

Redis Pub/Sub - Consumer
public class MyRedisSubscriber {
@Conﬁguration String message;
void handleMessage(String text) {
@Bean this.message = text;
public MyRedisSubscriber myListener() {
return new MyRedisSubscriber(); }
} }

@Bean
public RedisMessageListenerContainer
redisMessageListenerContainer(
RedisConnectionFactory redisConnectionFactory,
MyRedisSubscriber myListener) {
RedisMessageListenerContainer c =
new RedisMessageListenerContainer();
c.setConnectionFactory(redisConnectionFactory);
c.addMessageListener(new MessageListenerAdapter(myListener),
new ChannelTopic("myChannel"));
return c;
}

Redis Pub/Sub - Producer
public class RedisPubSubTest {

@Autowired private StringRedisTemplate stringRedisTemplate;

@Autowired private MyRedisSubscriber myListener;

@Test
public void testPubSub() throws InterruptedException {
…
stringRedisTemplate.convertAndSend("myChannel", "hello");
TimeUnit.SECONDS.sleep(4);
Assert.assertEquals("hello", myListener.message);
}
}

Redis caching support

KVs = <preﬁx + K, V>
Template needs to (de)serialize K and V Sorted set of all keys for clear()

MongoDB API usage patterns
• Create and store Mongo singleton • Queries

• Externalized server host, port etc. • Construct query object

• Inserts/Updates • mongo.getDatabase(…).getCollection(…)

• Map application POJO DBObject • Iterate through Cursor

• Map DBObject application POJO
• mongo.getDatabase(…).getCollection(…)

• Partial document updates

• Conﬁgure asynchronous vs. synchronous writes Higher-level than JDBC but still
repetitive, …

Spring Data - MongoDB

• MongoTemplate • Map-Reduce integration

• Query, Criteria, and Update DSLs • GridFS support

• Generic repositories

• Querydsl integration

• Cross-store persistence

• GeoSpatial integration

MongoTemplate
MongoTemplate
POJO ó DBObject
databaseName
Simplifies data access userId
mapping
Password
Translates exceptions defaultCollectionName

writeConcern
writeResultChecking

save()
insert()
remove() <<interface>>
updateFirst() MongoConvertor
findOne() write(Object, DBObject)
find()
read(Class, DBObject)
…

uses

Mongo MongoMapping
(Java Driver class) Converter

Example entity
public class Restaurant {
private String id; public class MenuItem {
private String name;
private String name;
private double price;
private List<MenuItem> menuItems;
public MenuItem(String name, double price) {
public Restaurant(String name) { this.name = name;
this.name = name; this.price = price;
}
…
} .....

...
}

Spring Data uses fields and non-
default constructors

Example data access code

@Repository
public class RestaurantRepository {

@Autowired
private MongoTemplate mongoTemplate;

public void add(Restaurant restaurant) {
mongoTemplate.save(restaurant);
}

public List<Restaurant> ﬁndRestaurantsByName(String restaurantName) {
return mongoTemplate.ﬁnd(
query(where("name").is(restaurantName)),
Restaurant.class);
}

A document in the restaurant collection
{
"_id" : ObjectId("4d977f55d3fe3119c904e026"),
"_class" : "net.chrisrichardson.mongodb.example.mongotemplate.Restaurant",
"name" : "Ajanta"
"menuItems" : [
{
"name" : "Tandoori Portobello Mushrooms",
"price" : 5.5
},
{
"name" : "Duck Curry Kerala",
"price" : 15
}
]
}

Spring MongoDB Example - Config 1
@Configuration
public class MongoExampleConfig extends AbstractMongoConfiguration {

private @Value("#{mongoDbProperties.databaseName}")
String mongoDbDatabase;

private @Value("#{mongoDbProperties.host}") Defines a
String mongoDbHost;
MongoTemplate
public Mongo mongo() throws Exception {
return new Mongo(mongoDbHost);
}

@Override <beans>
protected String getDatabaseName() { <context:annotation-config/>
return mongoDbDatabase;
} <context:component-scan base-package="net.chrisrichardson.mongodb.example"/>
... <util:properties id="mongoDbProperties" location="mongodb.properties"/>
} External Config
</beans>

mongodb.properties:

databaseName=demo1
host=192.168.253.150

Spring MongoDB Example - Conﬁg 2
<bean id="mongoTemplate"
class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg ref="mongoFactory"/>
</bean>

<mongo:db-factory id="mongoFactory"
host= "#{mongoDbProperties.host}"
dbname="#{mongoDbProperties.databaseName}" />

<util:properties
id="mongoDbProperties"
location="mongodb.properties"/>

Update example
@Repository
public class RestaurantRepository {
public void addMenuItem(String restaurantId,
MenuItem newMenuItem) {
mongoTemplate.updateFirst(
query(where("_id").is(new ObjectId(restaurantId))),
new Update().push("menuItems", newMenuItem),
Restaurant.class);

}

Atomic, in-place update of document

Geospatial example 1
case class FriendRecord(id : String,
name : String,
@Component
location : Point)
class MongoFriendService extends FriendService {

@Autowired
var mongoTemplate: MongoTemplate = _
Collection name
@PostConstruct
def createGeoIndex {
val dbo = new BasicDBObject
dbo.put("location", "2d")
mongoTemplate.getCollection("friendRecord").ensureIndex(dbo)
}

Create geospatial 2d index

Geospatial example 2 - ﬁnding nearby
@Component
class MongoFriendService extends FriendService {

override def findNearbyFriends(request: NearbyFriendsRequest) = {
val location = new Point(request.longitude, request.latitude)
val distance = new Distance(3, Metrics.MILES)
val query = NearQuery.near(location).maxDistance(distance)

val result = mongoTemplate.geoNear(query, classOf[FriendRecord])

val nearby = result.getContent.map(_.getContent)
FindNearbyFriendsResponse(nearby.map(f => FriendInfo(f.name, f.id)))
}

Callbacks – access driver API with exception
translation Exceptions are
translated

@Test
public void testDbCallback() {
Restaurant ajanta = makeAjantaRestaurant();
restaurantRepository.add(ajanta);
assertCollectionExists("restaurants2");
}

private Void assertCollectionExists(ﬁnal String collectionName) {
return mongoTemplate.execute(new DbCallback<Void>(){
@Override
public Void doInDB(DB db) {
Set<String> collectionNames = db.getCollectionNames();
Assert.assertTrue("Missing from " +
collectionNames,
collectionNames.contains(collectionName));
return null;
}});
}

Defining a Mongo Generic Repository
public class Person {
private ObjectId id;
private String firstname;
private String lastname;
… getters and setters
}

interface PersonRepository extends MongoRepository<Person, ObjectId> {
List<Person> findByLastname(String lastName);
}

Person p = new Person("John", "Doe");
personRepository.save(p);

Person p2 = personRepository.findOne(p.getId());

List<Person> johnDoes = personRepository.findByLastname("Doe");
assertEquals(1, johnDoes.size());

Mongo Repository conﬁguration
<bean>

<mongo:repositories
base-package="net.chrisrichardson.mongodb.example.mongorepository"
mongo-template-ref="mongoTemplate" />

</beans>

Scans classpath looking for subtypes of MongoRepository in the base package

Richer mapping
@Document(collection=”people”) Annotations define mapping: @Document, @Id, @Indexed,
public class Person { @PersistanceConstructor, @CompoundIndex, @DBRef,
@GeoSpatialIndexed, @Value
@Id
private ObjectId id; Map fields instead of properties no getters or setters
required

@Indexed
private String lastname; Non-default constructor
Index generation
@PersistenceConstructor
public Person(String firstname, String lastname) {
this.firstname = firstname;
this.lastname = lastname;
}
….
}

Richer mapping configuration
@Configuration
public class MongoExampleConfig extends AbstractMongoConfiguration {
private @Value("#{mongoDbProperties.databaseName}")
String mongoDbDatabase;
private @Value("#{mongoDbProperties.host}")
String mongoDbHost; Defines MongoTemplate bean
@Override
public Mongo mongo() throws Exception {
return new Mongo(mongoDbHost);
}
@Override Configures classpath scanning
public String getDatabaseName() {
return mongoDbDatabase;
}
@Override
public String getMappingBasePackage() {
return Person.class.getPackage().getName();
}
}

Support for the QueryDSL project
Generated from Type-safe
domain model class composable queries

QPerson person = QPerson.person;

Predicate predicate =
person.homeAddress.street1.eq("1 High Street")
.and(person.ﬁrstname.eq("John"))

List<Person> people = personRepository.ﬁndAll(predicate);

assertEquals(1, people.size());
assertPersonEquals(p, people.get(0));

Cross-store/polyglot persistence
Person person = new Person(…);
@Entity entityManager.persist(person);
public class Person {
// In Database Person p2 = entityManager.ﬁnd(…)
@Id private Long id;
private String lastname;

// In MongoDB
@RelatedDocument private Address address;

{ "_id" : ObjectId(”….."),
"_entity_id" : NumberLong(1),
"_entity_class" : "net.. Person", "_entity_ﬁeld_name" : "address",
"zip" : "94611", "street1" : "1 High Street", …}

Using Mongo and Redis with Cloud Foundry

• Create a Mongo or Redis service

• Bind the service to your application

• Access the service

• Via auto-reconﬁguration

• Using <cloud:*/> namespace

About <cloud:redis-connection-factory/>

<cloud:redis-connection-factory
id="redisConnectionFactory" Use when multiple
service-name="redis1" services are bound
/>

About <cloud:mongo-db-factory/>
Use when multiple services
are bound
<cloud:mongo-db-factory
id="mongoFactory"
service-name="mongo1"
>
<cloud:mongo-options
connections-per-host="..."
]
max-wait-time="..."
/>
</cloud:mongo-db-factory>

NoSQL and Caldecott
• Caldecott let’s you tunnel to a NoSQL service

• Use Redis CLI

• Explore database, adhoc operations

• ...

• Use Mongo CLI etc

• Explore database, adhoc operations

• Mongo dump/restore

• ...

Summary

• NoSQL databases sometimes offer a combination of:

• Higher scalability and performance

• Schema less, richer data models

• Spring Data simpliﬁes the development of NoSQL applications

• Cloud Foundry supports Mongo and Redis

@crichardson chris.richardson@springsource.com
http://plainoldobjects.com

Questions?
Sign up for CloudFoundry.com

Cloud Foundry 启动营
在www.cloudfoundry.com注册账号并成功上传应用程序，
即可于12月8日中午后凭账号ID和应用URL到签到处换取Cloud Foundry主题卫衣一件。

84

iPhone5 等你拿
第二天大会结束前，请不要提前离，将填写完整的意见反馈表投到签到处的抽奖箱内，
即可参与“iPhone5”抽奖活动。

85

Birds of a Feather 专家面对面
所有讲师都会在课程结束后，到紫兰厅与来宾讨论课程上的问题

86

Using Spring with NoSQL databases (SpringOne China 2012)

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Using Spring with NoSQL databases (SpringOne China 2012) (20)

More from Chris Richardson (20)

Recently uploaded (20)

Using Spring with NoSQL databases (SpringOne China 2012)