Vortex Tutorial -- Part I
6 likes2,525 views
The document outlines an introduction to the Vortex platform and its features for enabling efficient data sharing across various applications, including a micro-blogging application called Chirpit. It describes the architecture, security model, and data distribution service (DDS) principles, emphasizing data partitioning, content filtering, and the role of QoS policies. The presentation also illustrates the implementation aspects, showcasing a demo and hinting at future developments for integrating analytics and data persistence.
//
Define
the
publisher
and
link-‐it
up
to
the
“chirp:@user”
partition
val
pubQos
=
PublisherQos()
.withPolicy(Partition(Config.ChirpRootPartition
+
user))
implicit
val
pub
=
Publisher(pubQos)
//
Define
the
data
writer
and
make
it
reliable
and
for
the
time
being
//
Transient
Local…
Will
refined
this
in
the
Part
II
val
dwQos
=
DataWriterQos().withPolicies(
Reliability.Reliable,
Durability.TransientLocal)
//
Create
the
DataWriter
val
dw
=
DataWriter[ChirpAction](pub,
topic,
dwQos)
//
Chirp!!!
val
chirp
=
…
dw.write(chirp)](https://image.slidesharecdn.com/2014-141009120032-conversion-gate02/85/Vortex-Tutorial-Part-I-45-320.jpg)
//
Define
the
subscribe
and
link-‐it
up
to
the
partition
associated
to
//
the
user
followed
val
subQos
=
PublisherQos()
.withPolicy(Partition(followerList.map(Config.ChirpRootPartition
+
_))
implicit
val
sub
=
Subscriber(subQos)
//
Define
the
data
reader
and
make
it
reliable
and
for
the
time
being
//
Transient
Local…
Will
refined
this
in
the
Part
II
val
drQos
=
DataReaderQos().withPolicies(
Reliability.Reliable,
Durability.TransientLocal)
//
Create
the
DataWriter
val
dr
=
DataReader[ChirpAction](sub,
topic,
drQos)](https://image.slidesharecdn.com/2014-141009120032-conversion-gate02/85/Vortex-Tutorial-Part-I-46-320.jpg)
Vortex Tutorial -- Part I
- 1. Angelo Corsaro, PhD Chief Technology Officer [email protected] Vortex Tutorial Part I
- 2. CopyrightPrismTech,2014 Vortex enables seamless, ubiquitous, efficient and timely data sharing across mobile, embedded, desktop, cloud and web applications Vortex is based on the OMG DDS standard The Vortex Platform Vortex Device Tools Integration Vortex Cloud MaaS
- 5. CopyrightPrismTech,2014 Support for transport level security via TLS and DTLS Support for fine grained access control Access control plug-in compliant with DDS Security Vortex Security Model Arthur Dent Arthur Dent Ford Prerfect Zaphod Beeblebrox Marvin Trillian A(r,w), B(r) A(r,w), B(r,w), X(r) *(r,w) *(r) A(r,w), B(r,w), C(r,w) Ford Prerfect Zaphod Beeblebrox Trillian Marvin A B A,B X * * A,B,C Identity Access Rights Sessions are authenticated and communication is encrypted Only the Topic included as part of the access rights are visible and accessible
- 6. Use Cases
- 8. CopyrightPrismTech,2014 E-Health Hospital A Private Cloud Analytics Big Data Store Hospital B
- 10. CopyrightPrismTech,2014 SESAR PENS PENS: Pan European Network Service FDP: Flight Data Processor CWP: Controller Working Position IPv6 network SSM support FDP FDPFDP CWP Tower Device FDP FDPFDP Tower Device CWP FDP FDPFDP Tower Device CWP FDP FDPFDP CWP Tower Device EU Nation 4 EU Nation 1 EU Nation 2 EU Nation 3
- 11. Building ChirpIt
- 12. CopyrightPrismTech,2014 To explore the various features provided by the Vortex platform we will be designing and implementing a micro-blogging platform called ChirpIt. Specifically, we want to support the following features: ChirpIt users should be able to “chirp”, “re-chirp”, “like” and “dislike” trills as well as get detailed statistics The ChirpIt platform should provide information on trending topics — identified by hashtags — as well as trending users Third party services should be able to flexibly access slices of produced trills to perform their own trend analysis ChirpIt should scale to millions of users ChirpIt should be based on a Lambda Architecture ChirpIt Requirements
- 13. CopyrightPrismTech,2014 ChirpIt Architecture analytics data centre chirp emotions stats batch layer chirpit apps Cloud Messaging serving layer chirp emotions stats master dataset view view view chirp emotions stats 3rd party svcs Speed layer
- 15. CopyrightPrismTech,2014 DDS provides a Global Data Space abstraction that allows applications to autonomously, anonymously, securely and efficiently share data DDS’ Global Data Space is fully distributed (decentralised), highly efficient and scalable Data Distribution Service (DDS) DDS Global Data Space ... Data Writer Data Writer Data Writer Data Reader Data Reader Data Reader Data Reader Data Writer TopicA QoS TopicB QoS TopicC QoS TopicD QoS
- 16. CopyrightPrismTech,2014 DataWriters and DataReaders are automatically and dynamically matched by the DDS Discovery A rich set of QoS allows to control existential, temporal, and spatial properties of data Data Distribution Service (DDS) DDS Global Data Space ... Data Writer Data Writer Data Writer Data Reader Data Reader Data Reader Data Reader Data Writer TopicA QoS TopicB QoS TopicC QoS TopicD QoS
- 17. CopyrightPrismTech,2014 Elegant and High Level Data Sharing Abstraction Polyglot and platform independent • Java, Scala, C, C++, C#, JavaScript, CoffeeScript etc. • Android, Windows, Linux, VxWorks, etc. Peer-to-Peer by nature, Brokered when useful Key Highlights
- 18. CopyrightPrismTech,2014 Content and Temporal Filtering (both sender and receiver filtering supported) Queries 20+ QoS to control existential, temporal, and spatial properties of data Key Highlights
- 19. CopyrightPrismTech,2014 High Performance and Scalable • ~30 usec latency* • 7M msgs/sec node-to-node throughput* Key Highlights (*) Performance measured on Linux Host with i7 processor on a 1Gbps Ethernet network
- 20. Decomposing DDS
- 22. CopyrightPrismTech,2014 DDS data lives within a domain A domain is identified with a non negative integer, such as 1, 3, 31 The number 0 identifies the default domain A domain represent an impassable communication plane Domain DDS Domain
- 23. CopyrightPrismTech,2014 Partitions are the mechanism provided by DDS to organise information within a domain Access to partitions is controlled through QoS Policies Partitions are defined as strings: - “system:telemetry” - “system:log” - “data:row-2:col-3” Partitions addressed by name or regular expressions: - ”system:telemetry” - “data:row-2:col-*” Partitions Partitions
- 25. CopyrightPrismTech,2014 A Topic defines a domain-wide information’s class A Topic is defined by means of a (name, type, qos) tuple, where • name: identifies the topic within the domain • type: is the programming language type associated with the topic. Types are extensible and evolvable • qos: is a collection of policies that express the non-functional properties of this topic, e.g. reliability, persistence, etc. Topic Topic Type Name QoS ... TopicA QoS TopicB QoS TopicC QoS TopicD QoS
- 26. CopyrightPrismTech,2014 As explained in the previous slide a topic defines a class/type of information Topics can be defined as Singleton or can have multiple Instances Topic Instances are identified by means of the topic key A Topic Key is identified by a tuple of attributes -- like in databases Remarks: - A Singleton topic has a single domain-wide instance - A “regular” Topic can have as many instances as the number of different key values, e.g., if the key is an 8-bit character then the topic can have 256 different instances Topic and Instances
- 28. CopyrightPrismTech,2014 Chirp Actions union ChirpBody switch (ChirpActionKind) { case CHIRP_KIND: string<128> chirp; case RECHIRP_KIND: string<128> chirp; string user; case LIKE_KIND: string luser; case DISLIKE_KIND: string duser; }; ! struct ChirpAction { ChirpHeader header; ChirpBody body; }; #pragma keylist ChirpAction header.id.cid struct ChirpId { string uid; string cid; }; ! enum ChirpActionKind { CHIRP_KIND, RECHIRP_KIND, LIKE_KIND, DISLIKE_KIND }; struct ChirpHeader { ChirpId id; Location location; unsigned long long timestamp; ChirpActionKind kind; };
- 29. CopyrightPrismTech,2014 ChirpIt Statistics struct ChirpStats { ChirpId id; unsigned long rechirps; unsigned long likes; }; #pragma keylist ChirpStats id.cid struct UserStats { string userId; unsigned long followers; unsigned long chirps; unsigned long followed; }; #pragma keylist UserStats userId
- 30. CopyrightPrismTech,2014 User Actions struct enum UserActionKind { FOLLOW, UNFOLLOW }; struct UserAction { UserActionKind kind; string fromUser; string toUser; };
- 31. CopyrightPrismTech,2014 App Config typedef sequence<string> Partitions; ! struct AppConfig { string userId; Partitions ps; unsigned short history; }; #pragma keylist UserConfig userId
- 33. CopyrightPrismTech,2014 To make the system scale properly we need to ensure that the chirps, rechirps, etc., are distributed only to the interested parties As we have a single topic representing chirp actions, how can we ensure that information is received only by followers? DDS Provides two mechanisms: - Partitions: Allow to create information scopes - Content Filters: Allow to receive the data that satisfy certains predicates Scoping Information
- 34. CopyrightPrismTech,2014 ChirpIt Architecture ChirpAction @wolverine @drx chirp:wolverine ChirpAction @drx follows @wolverine
- 35. CopyrightPrismTech,2014 @drx ChirpAction ChirpIt Architecture data centre ChirpAction @wolverine chirp:wolverine ChirpAction @magneto chirp:magneto @drx follows @wolverine and @magneto
- 36. CopyrightPrismTech,2014 In summary partitions are used to scope information Each user will “join” a partition for each followed party Example: - If @drx follows @magneto, @wolverine, @cyclops and @mistique then he will receive ChirpActions from the partitions: • chirp:@magneto, chirp:@wolverine, chirp:@cyclops and chirp:@mistique - On the other hand, regardless of the people that follows @wolverine he will always and only write ChirpActions in the partitions • chirp:@wolverine Using Partitions to Scope Chirps
- 38. CopyrightPrismTech,2014 A DataWriter (DW) is a strongly typed entity used to produce samples for one or more instances of a Topic, with a given QoS Conceptually, the DataWriter QoS should be the same as the Topic QoS or more stringent However, DDS does enforce a specific relationship between the Topic and DataWriter QoS DataWriter DW Type Topic QoS
- 39. CopyrightPrismTech,2014 The DataWriter controls the life-cycle of Topic Instances and allows to: Define a new topic instance Write samples for a topic instance Dispose the topic instance DataWriter DW Type Topic QoS
- 41. CopyrightPrismTech,2014 A DataReader (DR) is a strongly typed entity used to access and/or consume samples for a Topic, with a given QoS Conceptually, the DataReader QoS should be the same as the Topic QoS or less stringent However, DDS does enforce a specific relationship between the Topic and DataReader QoS DataReader DR Type Topic QoS
- 43. CopyrightPrismTech,2014 DomainParticipant: Provides access to a data cloud -- called a domain in DDS Topic: Domain-wide definition of a kind of Information Publisher/Subscriber: Provide scope to data sharing through the concept of partitions DataReader/DataWriter: Allow to read/write data for a given topic in the partitions their Subscriber/Publisher are associated with. DDS Entities Domain (e.g. Domain 123) Domain Participant Topic Publisher DataWrter Subscriber DataReader Partition (e.g. “Telemetry”, “Shapes”, ) Topic Instances/Samples Ta Tb Tc Tx Ty T1 T1 T3
- 45. CopyrightPrismTech,2014 ChirpOut App // Domain Participant defined by dependency injection // Define the topic val topic = Topic[ChirpAction](Config.ChirpActionTopic) // Define the publisher and link-‐it up to the “chirp:@user” partition val pubQos = PublisherQos() .withPolicy(Partition(Config.ChirpRootPartition + user)) implicit val pub = Publisher(pubQos) // Define the data writer and make it reliable and for the time being // Transient Local… Will refined this in the Part II val dwQos = DataWriterQos().withPolicies( Reliability.Reliable, Durability.TransientLocal) // Create the DataWriter val dw = DataWriter[ChirpAction](pub, topic, dwQos) // Chirp!!! val chirp = … dw.write(chirp)
- 46. CopyrightPrismTech,2014 ChirpIn // Domain Participant defined by dependency injection // Define the topic val topic = Topic[ChirpAction](Config.ChirpActionTopic) // Define the subscribe and link-‐it up to the partition associated to // the user followed val subQos = PublisherQos() .withPolicy(Partition(followerList.map(Config.ChirpRootPartition + _)) implicit val sub = Subscriber(subQos) // Define the data reader and make it reliable and for the time being // Transient Local… Will refined this in the Part II val drQos = DataReaderQos().withPolicies( Reliability.Reliable, Durability.TransientLocal) // Create the DataWriter val dr = DataReader[ChirpAction](sub, topic, drQos)
- 47. CopyrightPrismTech,2014 ChirpIn // React to new Chirp dr listen { case DataAvailable(_) => { // take vs. read will become clear in Part II val samples = dr.take() // For the time being simply print-‐out the new chirps samples .filter(s => (s.getData != null)) .foreach(s => displayChirp) } }
- 48. CopyrightPrismTech,2014 ChirpOut App // Define the topic — assuming the topic is registered with the server…! topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)! // Define the data writer Qos! dwQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(userPartition))! ! // Use an Option monad to maintain the writer! chirpWriter = z_.None! ! // Create the writer when the runtime is connected! runtime.onconnect(() ->! dw = new dds.DataWriter(runtime, topic, dwQos)! chirpWriter = z_.Some(dw)! )! ! // …! // Write chirps! chirp = …! chirpWriter.map((dw) -> dw.write(chirp))!
- 49. CopyrightPrismTech,2014 ChirpIn App // Define the topic — assuming the topic is registered with the server…! topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)! // Define the data reader Qos! drQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(followedList))! ! // Use an Option monad to maintain the reader! chirpReader = z_.None! ! // Create the writer when the runtime is connected! runtime.onconnect(() ->! dr = new dds.DataReader(runtime, topic, dwQos)! // Attach a listener! dr.addListener((chirp) -> diplayChirp(chirp))! chirpReader = z_.Some(dr)! )
- 50. CopyrightPrismTech,2014 ChirpIt Thus Far… data centre chirp emotions stats chirp emotions stats chirp emotions stats chirp emotions stats
- 51. Live Demo
- 52. CopyrightPrismTech,2014 $ ./chirpOut @wolverin chirp>> @magneto will win!$ ./chirpOut @magneto @magneto>> @wolverine chirping while asleep… again! Download the package zip or tar.gz package available at: - https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.zip - https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.tar.gz Extract the package somewhere in your file-system and open a terminal at that location and run the app as shown below ChirpIt Demo $ ./chirpIn @magneto @wolverinse @mystique @wolverin>> @magneto will win! @magneto>> @wolverine chirping while asleep… again!
- 53. CopyrightPrismTech,2014 In Part II we extend our implementation to manage application configuration storage and analytics To do so, we will need to introduce additional features provided by the DDS and the Vortex Platform, such as Durability and Integration with Big-Data Stores What’s Next
- 54. CopyrightPrismTech,2014 In this presentation we have performed the first step toward implementing a micro-blogging platform The combination of Vortex Device and Vortex Cloud made it very easy to create and deploy our Internet-Scale, multi-device micro-blogging platform Some of the DDS QoS, such as Durability have already shown how historical data, i.e. chirps made while we were offline — can be provided by the platform The Part II will illustrate how the complete application fits together and what is the role of Vortex in the Lambda Architecture Summary