Paco Nathan, profile picture
Uploaded byPaco Nathan
PDF, PPTX4,895 views

Boulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading

The document discusses cluster computing with Apache Mesos and Cascading, focusing on the evolution of enterprise data workflows. It examines middleware for big data applications, including examples such as Cascading and Lingual, and highlights how functional programming principles have been integrated into data workflows for efficiency. Additionally, it outlines various use cases and deployments of Cascading in large-scale production environments, emphasizing the importance of declarative programming practices.

Download as PDF, PPTX
Boulder/Denver BigData, 2013-09-25: Cluster Computing with Apache Mesos and Cascading Paco Nathan @pacoid Chief Scientist, Mesosphere.io
Cluster Computing with Apache Mesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing Boulder, 2013-09-25
Enterprise Data Workflows middleware for Big Data applications is evolving, with commercial examples that include: Cascading, Lingual, Pattern, etc. Concurrent ParAccel Big Data Analytics Platform Actian Anaconda supporting IPython Notebook, Pandas,Augustus, etc. Continuum Analytics ETL data prep predictive model data sources end uses
Anatomy of an Enterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end uses ANSI SQL for ETL
Anatomy of an Enterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end usesJ2EE for business logic
Anatomy of an Enterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end uses SAS for predictive models
Anatomy of an Enterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end uses SAS for predictive modelsANSI SQL for ETL most of the licensing costs…
Anatomy of an Enterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end usesJ2EE for business logic most of the project costs…
ETL data prep predictive model data sources end uses Lingual: DW → ANSI SQL Pattern: SAS, R, etc. → PMML business logic in Java, Clojure, Scala, etc. sink taps for Memcached, HBase, MongoDB, etc. source taps for Cassandra, JDBC, Splunk, etc. Anatomy of an Enterprise app Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source a compiler sees it all… one connected DAG: • optimization • troubleshooting • exception handling • notifications cascading.org
a compiler sees it all… ETL data prep predictive model data sources end uses Lingual: DW → ANSI SQL Pattern: SAS, R, etc. → PMML business logic in Java, Clojure, Scala, etc. sink taps for Memcached, HBase, MongoDB, etc. source taps for Cassandra, JDBC, Splunk, etc. Anatomy of an Enterprise app Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source FlowDef flowDef = FlowDef.flowDef() .setName( "etl" ) .addSource( "example.employee", emplTap ) .addSource( "example.sales", salesTap ) .addSink( "results", resultsTap );   SQLPlanner sqlPlanner = new SQLPlanner() .setSql( sqlStatement );   flowDef.addAssemblyPlanner( sqlPlanner ); cascading.org
a compiler sees it all… ETL data prep predictive model data sources end uses Lingual: DW → ANSI SQL Pattern: SAS, R, etc. → PMML business logic in Java, Clojure, Scala, etc. sink taps for Memcached, HBase, MongoDB, etc. source taps for Cassandra, JDBC, Splunk, etc. Anatomy of an Enterprise app Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source FlowDef flowDef = FlowDef.flowDef() .setName( "classifier" ) .addSource( "input", inputTap ) .addSink( "classify", classifyTap );   PMMLPlanner pmmlPlanner = new PMMLPlanner() .setPMMLInput( new File( pmmlModel ) ) .retainOnlyActiveIncomingFields();   flowDef.addAssemblyPlanner( pmmlPlanner );
Cascading – functional programming Key insight: MapReduce is based on functional programming – back to LISP in 1970s. Apache Hadoop use cases are mostly about data pipelines, which are functional in nature. to ease staffing problems as “Main Street” Enterprise firms began to embrace Hadoop, Cascading was introduced in late 2007, as a new Java API to implement functional programming for large-scale data workflows: • leverages JVM and Java-based tools without any need to create new languages • allows programmers who have J2EE expertise to leverage the economics of Hadoop clusters Edgar Codd alluded to this (DSLs for structuring data) in his original paper about relational model
Cascading – functional programming • Twitter, eBay, LinkedIn, Nokia, YieldBot, uSwitch, etc., have invested in open source projects atop Cascading – used for their large-scale production deployments • new case studies for Cascading apps are mostly based on domain-specific languages (DSLs) in JVM languages which emphasize functional programming: Cascalog in Clojure (2010) Scalding in Scala (2012) github.com/nathanmarz/cascalog/wiki github.com/twitter/scalding/wiki Why Adopting the Declarative Programming PracticesWill ImproveYour Return fromTechnology Dan Woods, 2013-04-17 Forbes forbes.com/sites/danwoods/2013/04/17/why-adopting-the-declarative-programming- practices-will-improve-your-return-from-technology/
Functional Programming for Big Data WordCount with token scrubbing… Apache Hive: 52 lines HQL + 8 lines Python (UDF) compared to Scalding: 18 lines Scala/Cascading functional programming languages help reduce software engineering costs at scale, over time
Cascading – deployments • case studies: Climate Corp, Twitter, Etsy, Williams-Sonoma, uSwitch, Airbnb, Nokia, YieldBot, Square, Harvard, Factual, etc. • use cases: ETL, marketing funnel, anti-fraud, social media, retail pricing, search analytics, recommenders, eCRM, utility grids, telecom, genomics, climatology, agronomics, etc.
Workflow Abstraction – pattern language Cascading uses a “plumbing” metaphor in Java to define workflows out of familiar elements: Pipes, Taps, Tuple Flows, Filters, Joins, Traps, etc. Scrub token Document Collection Tokenize Word Count GroupBy token Count Stop Word List Regex token HashJoin Left RHS M R data is represented as flows of tuples operations in the flows bring functional programming aspects into Java A Pattern Language Christopher Alexander, et al. amazon.com/dp/0195019199
Workflow Abstraction – literate programming Cascading workflows generate their own visual documentation: flow diagrams in formal terms, flow diagrams leverage a methodology called literate programming provides intuitive, visual representations for apps – great for cross-team collaboration Scrub token Document Collection Tokenize Word Count GroupBy token Count Stop Word List Regex token HashJoin Left RHS M R Literate Programming Don Knuth literateprogramming.com
Workflow Abstraction – business process following the essence of literate programming, Cascading workflows provide statements of business process this recalls a sense of business process management for Enterprise apps (think BPM/BPEL for Big Data) Cascading creates a separation of concerns between business process and implementation details (Hadoop, etc.) this is especially apparent in large-scale Cascalog apps: “Specify what you require, not how to achieve it.” by virtue of the pattern language, the flow planner then determines how to translate business process into efficient, parallel jobs at scale
void map (String doc_id, String text): for each word w in segment(text): emit(w, "1"); void reduce (String word, Iterator group): int count = 0; for each pc in group: count += Int(pc); emit(word, String(count)); The Ubiquitous Word Count Definition: this simple program provides an excellent test case for parallel processing: • requires a minimal amount of code • demonstrates use of both symbolic and numeric values • shows a dependency graph of tuples as an abstraction • is not many steps away from useful search indexing • serves as a “HelloWorld” for Hadoop apps a distributed computing framework that runsWord Count efficiently in parallel at scale can handle much larger and more interesting compute problems count how often each word appears in a collection of text documents
Document Collection Word Count Tokenize GroupBy token Count R M 1 map 1 reduce 18 lines code gist.github.com/3900702 WordCount – conceptual flow diagram cascading.org/category/impatient
WordCount – Cascading app in Java String docPath = args[ 0 ]; String wcPath = args[ 1 ]; Properties properties = new Properties(); AppProps.setApplicationJarClass( properties, Main.class ); HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties ); // create source and sink taps Tap docTap = new Hfs( new TextDelimited( true, "t" ), docPath ); Tap wcTap = new Hfs( new TextDelimited( true, "t" ), wcPath ); // specify a regex to split "document" text lines into token stream Fields token = new Fields( "token" ); Fields text = new Fields( "text" ); RegexSplitGenerator splitter = new RegexSplitGenerator( token, "[ [](),.]" ); // only returns "token" Pipe docPipe = new Each( "token", text, splitter, Fields.RESULTS ); // determine the word counts Pipe wcPipe = new Pipe( "wc", docPipe ); wcPipe = new GroupBy( wcPipe, token ); wcPipe = new Every( wcPipe, Fields.ALL, new Count(), Fields.ALL ); // connect the taps, pipes, etc., into a flow FlowDef flowDef = FlowDef.flowDef().setName( "wc" ) .addSource( docPipe, docTap )  .addTailSink( wcPipe, wcTap ); // write a DOT file and run the flow Flow wcFlow = flowConnector.connect( flowDef ); wcFlow.writeDOT( "dot/wc.dot" ); wcFlow.complete(); Document Collection Word Count Tokenize GroupBy token Count R M
mapreduce Every('wc')[Count[decl:'count']] Hfs['TextDelimited[[UNKNOWN]->['token', 'count']]']['output/wc']'] GroupBy('wc')[by:['token']] Each('token')[RegexSplitGenerator[decl:'token'][args:1]] Hfs['TextDelimited[['doc_id', 'text']->[ALL]]']['data/rain.txt']'] [head] [tail] [{2}:'token', 'count'] [{1}:'token'] [{2}:'doc_id', 'text'] [{2}:'doc_id', 'text'] wc[{1}:'token'] [{1}:'token'] [{2}:'token', 'count'] [{2}:'token', 'count'] [{1}:'token'] [{1}:'token'] WordCount – generated flow diagram Document Collection Word Count Tokenize GroupBy token Count R M
(ns impatient.core   (:use [cascalog.api]         [cascalog.more-taps :only (hfs-delimited)])   (:require [clojure.string :as s]             [cascalog.ops :as c])   (:gen-class)) (defmapcatop split [line]   "reads in a line of string and splits it by regex"   (s/split line #"[[](),.)s]+")) (defn -main [in out & args]   (?<- (hfs-delimited out)        [?word ?count]        ((hfs-delimited in :skip-header? true) _ ?line)        (split ?line :> ?word)        (c/count ?count))) ; Paul Lam ; github.com/Quantisan/Impatient WordCount – Cascalog / Clojure Document Collection Word Count Tokenize GroupBy token Count R M
github.com/nathanmarz/cascalog/wiki • implements Datalog in Clojure, with predicates backed by Cascading – for a highly declarative language • run ad-hoc queries from the Clojure REPL – approx. 10:1 code reduction compared with SQL • composable subqueries, used for test-driven development (TDD) practices at scale • Leiningen build: simple, no surprises, in Clojure itself • more new deployments than other Cascading DSLs – Climate Corp is largest use case: 90% Clojure/Cascalog • has a learning curve, limited number of Clojure developers • aggregators are the magic, and those take effort to learn WordCount – Cascalog / Clojure Document Collection Word Count Tokenize GroupBy token Count R M
import com.twitter.scalding._   class WordCount(args : Args) extends Job(args) { Tsv(args("doc"), ('doc_id, 'text), skipHeader = true) .read .flatMap('text -> 'token) { text : String => text.split("[ [](),.]") } .groupBy('token) { _.size('count) } .write(Tsv(args("wc"), writeHeader = true)) } WordCount – Scalding / Scala Document Collection Word Count Tokenize GroupBy token Count R M
github.com/twitter/scalding/wiki • extends the Scala collections API so that distributed lists become “pipes” backed by Cascading • code is compact, easy to understand • nearly 1:1 between elements of conceptual flow diagram and function calls • extensive libraries are available for linear algebra, abstract algebra, machine learning – e.g., Matrix API, Algebird, etc. • significant investments by Twitter, Etsy, eBay, etc. • great for data services at scale • less learning curve than Cascalog WordCount – Scalding / Scala Document Collection Word Count Tokenize GroupBy token Count R M
A Thought Exercise Consider that when a company like Caterpillar moves into data science, they won’t be building the world’s next search engine or social network They will be optimizing supply chain, optimizing fuel costs, automating data feedback loops integrated into their equipment… Operations Research – crunching amazing amounts of data $50B company, in a $250B market segment Upcoming: tractors as drones – guided by complex, distributed data apps
Alternatively… climate.com
Two Avenues to the App Layer… scale ➞ complexity➞ Enterprise: must contend with complexity at scale everyday… incumbents extend current practices and infrastructure investments – using J2EE, ANSI SQL, SAS, etc. – to migrate workflows onto Apache Hadoop while leveraging existing staff Start-ups: crave complexity and scale to become viable… new ventures move into Enterprise space to compete using relatively lean staff, while leveraging sophisticated engineering practices, e.g., Cascalog and Scalding
Cluster Computing with Apache Mesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing Boulder, 2013-09-25
Hadoop Cluster source tap source tap sink tap trap tap customer profile DBsCustomer Prefs logs logs Logs Data Workflow Cache Customers Support Web App Reporting Analytics Cubes sink tap Modeling PMML Lingual – ANSI SQL • collab with Optiq – industry-proven code base • ANSI SQL parser/optimizer atop Cascading flow planner • JDBC driver to integrate into existing tools and app servers • relational catalog over a collection of unstructured data • SQL shell prompt to run queries • enable analysts without retraining on Hadoop, etc. • transparency for Support, Ops, Finance, et al. a language for queries – not a database, but ANSI SQL as a DSL for workflows
Lingual – CSV data in local file system cascading.org/lingual
Lingual – shell prompt, catalog cascading.org/lingual
Lingual – queries cascading.org/lingual
# load the JDBC package library(RJDBC)   # set up the driver drv <- JDBC("cascading.lingual.jdbc.Driver", "~/src/concur/lingual/lingual-local/build/libs/lingual-local-1.0.0-wip-dev-jdbc.jar")   # set up a database connection to a local repository connection <- dbConnect(drv, "jdbc:lingual:local;catalog=~/src/concur/lingual/lingual-examples/ tables;schema=EMPLOYEES")   # query the repository: in this case the MySQL sample database (CSV files) df <- dbGetQuery(connection, "SELECT * FROM EMPLOYEES.EMPLOYEES WHERE FIRST_NAME = 'Gina'") head(df)   # use R functions to summarize and visualize part of the data df$hire_age <- as.integer(as.Date(df$HIRE_DATE) - as.Date(df$BIRTH_DATE)) / 365.25 summary(df$hire_age) library(ggplot2) m <- ggplot(df, aes(x=hire_age)) m <- m + ggtitle("Age at hire, people named Gina") m + geom_histogram(binwidth=1, aes(y=..density.., fill=..count..)) + geom_density() Lingual – connecting Hadoop and R
> summary(df$hire_age) Min. 1st Qu. Median Mean 3rd Qu. Max. 20.86 27.89 31.70 31.61 35.01 43.92 Lingual – connecting Hadoop and R cascading.org/lingual
Hadoop Cluster source tap source tap sink tap trap tap customer profile DBsCustomer Prefs logs logs Logs Data Workflow Cache Customers Support Web App Reporting Analytics Cubes sink tap Modeling PMML Pattern – model scoring • migrate workloads: SAS,Teradata, etc., exporting predictive models as PMML • great open source tools – R, Weka, KNIME, Matlab, RapidMiner, etc. • integrate with other libraries – Matrix API, etc. • leverage PMML as another kind of DSL cascading.org/pattern
• established XML standard for predictive model markup • organized by Data Mining Group (DMG), since 1997 http://dmg.org/ • members: IBM, SAS, Visa, NASA, Equifax, Microstrategy, Microsoft, etc. • PMML concepts for metadata, ensembles, etc., translate directly into Cascading tuple flows “PMML is the leading standard for statistical and data mining models and supported by over 20 vendors and organizations.With PMML, it is easy to develop a model on one system using one application and deploy the model on another system using another application.” PMML – standard wikipedia.org/wiki/Predictive_Model_Markup_Language
PMML – vendor coverage
• Association Rules: AssociationModel element • Cluster Models: ClusteringModel element • Decision Trees: TreeModel element • Naïve Bayes Classifiers: NaiveBayesModel element • Neural Networks: NeuralNetwork element • Regression: RegressionModel and GeneralRegressionModel elements • Rulesets: RuleSetModel element • Sequences: SequenceModel element • SupportVector Machines: SupportVectorMachineModel element • Text Models: TextModel element • Time Series: TimeSeriesModel element PMML – model coverage ibm.com/developerworks/industry/library/ind-PMML2/
## train a RandomForest model   f <- as.formula("as.factor(label) ~ .") fit <- randomForest(f, data_train, ntree=50)   ## test the model on the holdout test set   print(fit$importance) print(fit)   predicted <- predict(fit, data) data$predicted <- predicted confuse <- table(pred = predicted, true = data[,1]) print(confuse)   ## export predicted labels to TSV   write.table(data, file=paste(dat_folder, "sample.tsv", sep="/"), quote=FALSE, sep="t", row.names=FALSE)   ## export RF model to PMML   saveXML(pmml(fit), file=paste(dat_folder, "sample.rf.xml", sep="/")) Pattern – create a model in R
public static void main( String[] args ) throws RuntimeException { String inputPath = args[ 0 ]; String classifyPath = args[ 1 ]; // set up the config properties Properties properties = new Properties(); AppProps.setApplicationJarClass( properties, Main.class ); HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties );   // create source and sink taps Tap inputTap = new Hfs( new TextDelimited( true, "t" ), inputPath ); Tap classifyTap = new Hfs( new TextDelimited( true, "t" ), classifyPath );   // handle command line options OptionParser optParser = new OptionParser(); optParser.accepts( "pmml" ).withRequiredArg();   OptionSet options = optParser.parse( args );   // connect the taps, pipes, etc., into a flow FlowDef flowDef = FlowDef.flowDef().setName( "classify" ) .addSource( "input", inputTap ) .addSink( "classify", classifyTap );   if( options.hasArgument( "pmml" ) ) { String pmmlPath = (String) options.valuesOf( "pmml" ).get( 0 ); PMMLPlanner pmmlPlanner = new PMMLPlanner() .setPMMLInput( new File( pmmlPath ) ) .retainOnlyActiveIncomingFields() .setDefaultPredictedField( new Fields( "predict", Double.class ) ); // default value if missing from the model flowDef.addAssemblyPlanner( pmmlPlanner ); }   // write a DOT file and run the flow Flow classifyFlow = flowConnector.connect( flowDef ); classifyFlow.writeDOT( "dot/classify.dot" ); classifyFlow.complete(); } Pattern – score a model, within an app
Cluster Computing with Apache Mesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing Boulder, 2013-09-25
Q3 1997: inflection point four independent teams were working toward horizontal scale-out of workflows based on commodity hardware this effort prepared the way for huge Internet successes in the 1997 holiday season… AMZN, EBAY, Inktomi (YHOO Search), then GOOG MapReduce and the Apache Hadoop open source stack emerged from this period
RDBMS Stakeholder SQL Query result sets Excel pivot tables PowerPoint slide decks Web App Customers transactions Product strategy Engineering requirements BI Analysts optimized code Circa 1996: pre- inflection point
RDBMS Stakeholder SQL Query result sets Excel pivot tables PowerPoint slide decks Web App Customers transactions Product strategy Engineering requirements BI Analysts optimized code Circa 1996: pre- inflection point “throw it over the wall”
RDBMS SQL Query result sets recommenders + classifiers Web Apps customer transactions Algorithmic Modeling Logs event history aggregation dashboards Product Engineering UX Stakeholder Customers DW ETL Middleware servletsmodels Circa 2001: post- big ecommerce successes
RDBMS SQL Query result sets recommenders + classifiers Web Apps customer transactions Algorithmic Modeling Logs event history aggregation dashboards Product Engineering UX Stakeholder Customers DW ETL Middleware servletsmodels Circa 2001: post- big ecommerce successes “data products”
Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile, etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere
Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile, etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere “optimize topologies”
Amazon “Early Amazon: Splitting the website” – Greg Linden glinden.blogspot.com/2006/02/early-amazon-splitting-website.html eBay “The eBay Architecture” – Randy Shoup, Dan Pritchett addsimplicity.com/adding_simplicity_an_engi/2006/11/you_scaled_your.html addsimplicity.com.nyud.net:8080/downloads/eBaySDForum2006-11-29.pdf Inktomi (YHOO Search) “Inktomi’s Wild Ride” – Erik Brewer (0:05:31 ff) youtu.be/E91oEn1bnXM Google “Underneath the Covers at Google” – Jeff Dean (0:06:54 ff) youtu.be/qsan-GQaeyk perspectives.mvdirona.com/2008/06/11/JeffDeanOnGoogleInfrastructure.aspx MIT Media Lab “Social Information Filtering for Music Recommendation” – Pattie Maes pubs.media.mit.edu/pubs/papers/32paper.ps ted.com/speakers/pattie_maes.html Primary Sources
Cluster Computing’s Dirty Little Secret many of us make a good living by leveraging high ROI apps based on clusters, and so execs agree to build out more data centers… clusters for Hadoop/HBase, for Storm, for MySQL, for Memcached, for Cassandra, for Nginx, etc. this becomes expensive! a single class of workloads on a given cluster is simpler to manage, but terrible for utilization… various notions of “cloud” help… Cloudera, Hortonworks, probably EMC soon: sell a notion of “Hadoop as OS” All your workloads are belong to us Google Data Center, Fox News ~2002
Three Laws, or more? meanwhile, architectures evolve toward much, much larger data… pistoncloud.com/ ... Rich Freitas, IBM Research Q: what disruptions in topologies+algorithms could this imply? given there’s no such thing as RAM anymore…
Three Laws, or more? meanwhile, architectures evolve toward much, much larger data… pistoncloud.com/ ... Rich Freitas, IBM Research regardless of how architectures change, death and taxes will endure: servers fail, data must move Q: what disruptions in topologies+algorithms could this imply? given there’s no such thing as RAM anymore…
The Modern Kernel: Top Linux Contributors…
Beyond Hadoop Hadoop – an open source solution for fault-tolerant parallel processing of batch jobs at scale, based on commodity hardware… however, other priorities have emerged for the analytics lifecycle: • apps require integration beyond Hadoop • multiple topologies, mixed workloads, multi-tenancy • higher utilization • lower latency • highly-available, long running services • more than “Just JVM” – e.g., Python growth keep in mind the priority for multi-disciplinary efforts, to break down even more silos – well beyond the de facto “priesthood” of data engineering
Beyond Hadoop Google has been doing data center computing for years, to address the complexities of large-scale data workflows: • leveraging the modern kernel: isolation in lieu of VMs • “most (>80%) jobs are batch jobs, but the majority of resources (55–80%) are allocated to service jobs” • mixed workloads, multi-tenancy • relatively high utilization rates • JVM? not so much… • reality: scheduling batch is simple; scheduling services is hard/expensive
“Return of the Borg” Return of the Borg: HowTwitter Rebuilt Google’s SecretWeapon Cade Metz wired.com/wiredenterprise/ 2013/03/google-borg-twitter-mesos The Datacenter as a Computer: An Introduction to the Design ofWarehouse-Scale Machines Luiz André Barroso, Urs Hölzle research.google.com/pubs/ pub35290.html 2011 GAFS Omega John Wilkes, et al. youtu.be/0ZFMlO98Jkc
“Return of the Borg” Omega: flexible, scalable schedulers for large compute clusters Malte Schwarzkopf,Andy Konwinski, Michael Abd-El-Malek, John Wilkes eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf
Mesos – definitions a common substrate for cluster computing heterogenous assets in your data center or cloud made available as a homogenous set of resources • top-level Apache project • scalability to 10,000s of nodes • obviates the need for virtual machines • isolation (pluggable) for CPU, RAM, I/O, FS, etc. • fault-tolerant replicated master using ZooKeeper • multi-resource scheduling (memory and CPU aware) • APIs in C++, Java, Python • web UI for inspecting cluster state • available for Linux, OpenSolaris, Mac OSX
Mesos – architecture R uby Kernel Apps servicesbatch Frameworks Python JVM C ++ Workloads distributed file system Chronos DFS distributed resources: CPU, RAM, I/O, FS, rack locality, etc. Cluster Storm Kafka JBoss Django RailsSharkImpalaScalding Marathon SparkHadoopMPI MySQL
Mesos – architecture given use of Mesos as a Data Center OS kernel… • Chronos provides complex scheduling capabilities, much like a distributed Unix “cron” • Marathon provides highly-available long-running services, much like a distributed Unix “init.d” • next time you need to build a distributed app, consider using these as building blocks a major lesson learned from Spark: • leveraging these kinds of building blocks, one can rebuild Hadoop 100x faster, in much less code
Mesos – data center OS stack HADOOP STORM CHRONOS RAILS JBOSS TELEMETRY Kernel OS Apps MESOS CAPACITY PLANNING GUISECURITYSMARTER SCHEDULING
Prior Practice: Dedicated Servers DATACENTER • low utilization rates • longer time to ramp up new services
Prior Practice: Virtualization DATACENTER PROVISIONED VMS • even more machines to manage • substantial performance decrease due to virtualization • VM licensing costs
Prior Practice: Static Partitioning DATACENTER STATIC PARTITIONING • even more machines to manage • substantial performance decrease due to virtualization • VM licensing costs • static partitioning limits elasticity
MESOS Mesos: One Large Pool Of Resources DATACENTER “We wanted people to be able to program for the data center just like they program for their laptop." Ben Hindman
What are the costs of Virtualization? benchmark type OpenVZ improvement mixed workloads 210%-300% LAMP (related) 38%-200% I/O throughput 200%-500% response time order magnitude more pronounced at higher loads
What are the costs of Single Tenancy? 0% 25% 50% 75% 100% RAILS CPU LOAD MEMCACHED CPU LOAD 0% 25% 50% 75% 100% HADOOP CPU LOAD 0% 25% 50% 75% 100% t t 0% 25% 50% 75% 100% Rails Memcached Hadoop COMBINED CPU LOAD (RAILS, MEMCACHED, HADOOP)
M Master Docker Registry index.docker.io Local Docker Registry ( optional ) M M S S S S S S marathon docker docker docker Mesos master servers Mesos slave servers Marathon can launch and monitor service containers from one or more Docker registries, using the Docker executor for Mesos S S S S S S … … … … … … … mesosphere.io/2013/09/26/docker-on-mesos/ Example: Docker on Mesos
Mesos Master Server init | + mesos-master | + marathon | Mesos Slave Server init | + docker | | | + lxc | | | + (user task, under container init system) | | | + mesos-slave | | | + /var/lib/mesos/executors/docker | | | | | + docker run … | | | The executor, monitored by the Mesos slave, delegates to the local Docker daemon for image discovery and management. The executor communicates with Marathon via the Mesos master and ensures that Docker enforces the specified resource limitations. mesosphere.io/2013/09/26/docker-on-mesos/ Example: Docker on Mesos
Mesos Master Server init | + mesos-master | + marathon | Mesos Slave Server init | + docker | | | + lxc | | | + (user task, under container init system) | | | + mesos-slave | | | + /var/lib/mesos/executors/docker | | | | | + docker run … | | | Docker Registry When a user requests a container… Mesos, LXC, and Docker are tied together for launch 2 1 3 4 5 6 7 8 Example: Docker on Mesos mesosphere.io/2013/09/26/docker-on-mesos/
Arguments for Data Center Computing rather than running several specialized clusters, each at relatively low utilization rates, instead run many mixed workloads obvious benefits are realized in terms of: • scalability, elasticity, fault tolerance, performance, utilization • reduced equipment cap­ex, Ops overhead, etc. • reduced licensing, eliminating need forVMs or potential vendor lock­in subtle benefits – arguably, more important for Enterprise IT: • reduced time for engineers to ramp­up new services at scale • reduced latency between batch and services, enabling new high­ROI use cases • enables Dev/Test apps to run safely on a Production cluster
Deployments
Opposite Ends of the Spectrum, One Substrate Built-in / bare metal Hypervisors Solaris Zones Linux CGroups
Opposite Ends of the Spectrum, One Substrate Request / Response Batch
Case Study: Twitter (bare metal / on premise) “Mesos is the cornerstone of our elastic compute infrastructure – it’s how we build all our new services and is critical forTwitter’s continued success at scale. It's one of the primary keys to our data center efficiency." Chris Fry, SVP Engineering blog.twitter.com/2013/mesos-graduates-from-apache-incubation • key services run in production: analytics, typeahead, ads • Twitter engineers rely on Mesos to build all new services • instead of thinking about static machines, engineers think about resources like CPU, memory and disk • allows services to scale and leverage a shared pool of servers across data centers efficiently • reduces the time between prototyping and launching
Case Study: Airbnb (fungible cloud infrastructure) “We think we might be pushing data science in the field of travel more so than anyone has ever done before… a smaller number of engineers can have higher impact through automation on Mesos." Mike Curtis,VP Engineering gigaom.com/2013/07/29/airbnb-is-engineering-itself-into-a-data-driven... • improves resource management and efficiency • helps advance engineering strategy of building small teams that can move fast • key to letting engineers make the most of AWS-based infrastructure beyond just Hadoop • allowed company to migrate off Elastic MapReduce • enables use of Hadoop along with Chronos, Spark, Storm, etc.
Media Coverage Play Framework Grid Deployment with Mesos James Ward, Flo Leibert, et al. Typesafe blog (2013-09-19) typesafe.com/blog/play-framework-grid... Mesosphere Launches Marathon Framework Adrian Bridgwater Dr. Dobbs (2013-09-18) drdobbs.com/open-source/mesosphere... New open source tech Marathon wants to make your data center run like Google’s Derrick Harris GigaOM (2013-09-04) gigaom.com/2013/09/04/... Running batch and long-running, highly available service jobs on the same cluster Ben Lorica O’Reilly (2013-09-01) strata.oreilly.com/2013/09/...
Resources Apache Mesos Project mesos.apache.org Mesosphere mesosphere.io Tutorial mesosphere.io/2013/08/01/... Documentation mesos.apache.org/documentation 2011 USENIX Research Paper usenix.org/legacy/event/nsdi11/tech/full_papers/Hindman_new.pdf Collected Notes/Archives goo.gl/jPtTP
Cluster Computing with Apache Mesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing SUMMARY… Boulder, 2013-09-25
Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile, etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony
Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile, etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 1. End Use Cases, the drivers
Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile, etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 2. A new kind of team process
Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile, etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 3. Abstraction layer as optimizing middleware, e.g., Cascading
Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile, etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 4. Data Center OS, e.g., Mesos
Enterprise DataWorkflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do monthly newsletter for updates, events, conference summaries, etc.: liber118.com/pxn/

More Related Content

PDF
ACM Bay Area Data Mining Workshop: Pattern, PMML, Hadoop
byPaco Nathan
 
PDF
Pattern: PMML for Cascading and Hadoop
byPaco Nathan
 
PDF
ScalaTo July 2019 - No more struggles with Apache Spark workloads in production
byChetan Khatri
 
PPTX
Phily JUG : Web Services APIs for Real-time Analytics w/ Storm and DropWizard
byBrian O'Neill
 
PDF
TensorFlow Extension (TFX) and Apache Beam
bymarkgrover
 
PDF
Fully Automated QA System For Large Scale Search And Recommendation Engines U...
bySpark Summit
 
PDF
Pattern -A scoring engine
byShivanna Madalabhavi
 
PDF
Webinar: ArangoDB 3.8 Preview - Analytics at Scale
byArangoDB Database
 
ACM Bay Area Data Mining Workshop: Pattern, PMML, Hadoop
byPaco Nathan
 
Pattern: PMML for Cascading and Hadoop
byPaco Nathan
 
ScalaTo July 2019 - No more struggles with Apache Spark workloads in production
byChetan Khatri
 
Phily JUG : Web Services APIs for Real-time Analytics w/ Storm and DropWizard
byBrian O'Neill
 
TensorFlow Extension (TFX) and Apache Beam
bymarkgrover
 
Fully Automated QA System For Large Scale Search And Recommendation Engines U...
bySpark Summit
 
Pattern -A scoring engine
byShivanna Madalabhavi
 
Webinar: ArangoDB 3.8 Preview - Analytics at Scale
byArangoDB Database
 

What's hot

PDF
GraphTech Ecosystem - part 2: Graph Analytics
byLinkurious
 
PPT
Data science and OSS
byKevin Crocker
 
PDF
Fossasia 2018-chetan-khatri
byChetan Khatri
 
PPTX
Spark - Philly JUG
byBrian O'Neill
 
PDF
An introduction to multi-model databases
byBerta Hermida Plaza
 
PDF
Reducing Development Time for Production-Grade Hadoop Applications
byCascading
 
PDF
Graph analytics in Linkurious Enterprise
byLinkurious
 
PDF
Practical Machine Learning Pipelines with MLlib
byDatabricks
 
PDF
What's new with Apache Spark?
byPaco Nathan
 
PDF
Guacamole Fiesta: What do avocados and databases have in common?
byArangoDB Database
 
PDF
Big Data - Lab A1 (SC 11 Tutorial)
byRobert Grossman
 
PDF
An Introduction to Spark with Scala
byChetan Khatri
 
PDF
towards_analytics_query_engine
byNantia Makrynioti
 
PDF
No more struggles with Apache Spark workloads in production
byChetan Khatri
 
PDF
PyConLT19-No_more_struggles_with_Apache_Spark_(PySpark)_workloads_in_production
byChetan Khatri
 
PDF
Apache Spark and the Emerging Technology Landscape for Big Data
byPaco Nathan
 
PPTX
From keyword-based search to language-agnostic semantic search
byCareerBuilder.com
 
PDF
Reproducible AI Using PyTorch and MLflow
byDatabricks
 
PDF
Data ops: Machine Learning in production
byStepan Pushkarev
 
PDF
Elegant and Scalable Code Querying with Code Property Graphs
byConnected Data World
 
GraphTech Ecosystem - part 2: Graph Analytics
byLinkurious
 
Data science and OSS
byKevin Crocker
 
Fossasia 2018-chetan-khatri
byChetan Khatri
 
Spark - Philly JUG
byBrian O'Neill
 
An introduction to multi-model databases
byBerta Hermida Plaza
 
Reducing Development Time for Production-Grade Hadoop Applications
byCascading
 
Graph analytics in Linkurious Enterprise
byLinkurious
 
Practical Machine Learning Pipelines with MLlib
byDatabricks
 
What's new with Apache Spark?
byPaco Nathan
 
Guacamole Fiesta: What do avocados and databases have in common?
byArangoDB Database
 
Big Data - Lab A1 (SC 11 Tutorial)
byRobert Grossman
 
An Introduction to Spark with Scala
byChetan Khatri
 
towards_analytics_query_engine
byNantia Makrynioti
 
No more struggles with Apache Spark workloads in production
byChetan Khatri
 
PyConLT19-No_more_struggles_with_Apache_Spark_(PySpark)_workloads_in_production
byChetan Khatri
 
Apache Spark and the Emerging Technology Landscape for Big Data
byPaco Nathan
 
From keyword-based search to language-agnostic semantic search
byCareerBuilder.com
 
Reproducible AI Using PyTorch and MLflow
byDatabricks
 
Data ops: Machine Learning in production
byStepan Pushkarev
 
Elegant and Scalable Code Querying with Code Property Graphs
byConnected Data World
 

Viewers also liked

PPTX
NLCMG - Performance is good, Understanding performance is better
bynlwebperf
 
PDF
IBM Big Data References
byRob Thomas
 
PDF
Blackwell Esteem AFSL
bysamueltay77
 
PPTX
Where can tell me who I am?
byseltzoid
 
PPS
ROR -Igal Assaf Paris sous la neige
byIgal Assaf
 
PPT
file handling1
bystudent
 
PDF
Top Thirty Global Media Owners 2014
byPerformics
 
PDF
How to unlock alcatel one touch fierce 7024w by unlock code
byrscooldesire
 
PPTX
3. Culture & Competition
bypaulwhite1983
 
PPS
Formula of failure and success
byUjjwal Panda
 
DOCX
Business is fun: Shopping in madrid
bysarasacristan
 
PDF
Ushul Firaq wal Adyaan wal Madzaahib Al Fikriyah (Ar)
bySayogyo Rahman Doko
 
PDF
1 mate5 ecuación-
byPapías Huiza Oyola
 
PDF
Spirax saco
byseoulfood
 
DOCX
Report..costing..
byNur Dalila Zamri
 
PDF
Pour les enfants du monde entier
bysatelite1
 
DOCX
Guia de estudio
bysaralorenaft
 
PPTX
P pt keys for good and happy life.
byRajasekhar Dasari
 
PDF
Prescrition co-owners- pdf
bysahib rox
 
PDF
Lecture 08: “two sides of the same coin”
byPatrick Mooney
 
NLCMG - Performance is good, Understanding performance is better
bynlwebperf
 
IBM Big Data References
byRob Thomas
 
Blackwell Esteem AFSL
bysamueltay77
 
Where can tell me who I am?
byseltzoid
 
ROR -Igal Assaf Paris sous la neige
byIgal Assaf
 
file handling1
bystudent
 
Top Thirty Global Media Owners 2014
byPerformics
 
How to unlock alcatel one touch fierce 7024w by unlock code
byrscooldesire
 
3. Culture & Competition
bypaulwhite1983
 
Formula of failure and success
byUjjwal Panda
 
Business is fun: Shopping in madrid
bysarasacristan
 
Ushul Firaq wal Adyaan wal Madzaahib Al Fikriyah (Ar)
bySayogyo Rahman Doko
 
1 mate5 ecuación-
byPapías Huiza Oyola
 
Spirax saco
byseoulfood
 
Report..costing..
byNur Dalila Zamri
 
Pour les enfants du monde entier
bysatelite1
 
Guia de estudio
bysaralorenaft
 
P pt keys for good and happy life.
byRajasekhar Dasari
 
Prescrition co-owners- pdf
bysahib rox
 
Lecture 08: “two sides of the same coin”
byPatrick Mooney
 

Similar to Boulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading

PDF
Functional programming
 for optimization problems 
in Big Data
byPaco Nathan
 
PDF
PDX Hadoop: Enterprise Data Workflows with Cascading and Mesos
byPaco Nathan
 
PDF
Accelerate Big Data Application Development with Cascading
byCascading
 
PDF
July Clojure Users Group Meeting: "Using Cascalog with Palo Alto Open Data"
byPaco Nathan
 
PDF
Using Cascalog to build an app with City of Palo Alto Open Data
byOSCON Byrum
 
PDF
OSCON 2013: Using Cascalog to build an app with City of Palo Alto Open Data
byPaco Nathan
 
PDF
Enterprise Data Workflows with Cascading and Windows Azure HDInsight
byPaco Nathan
 
PDF
Cascading concurrent yahoo lunch_nlearn
byCascading
 
PDF
Pattern - an open source project for migrating predictive models from SAS, et...
byDataWorks Summit
 
PDF
Elasticsearch + Cascading for Scalable Log Processing
byCascading
 
PDF
Intro to Cascading
byBen Speakmon
 
PDF
The Cascading (big) data application framework
byModern Data Stack France
 
PDF
The Cascading (big) data application framework - André Keple, Sr. Engineer, C...
byCascading
 
PDF
Augury and Omens Aside, Part 1:
 The Business Case for Apache Mesos
byPaco Nathan
 
PPTX
Introduction to Cascading
byCascading
 
PDF
Hadoop User Group EU 2014
bycwensel
 
PPTX
Cascading User Group Meet
byVinoth Kannan
 
PDF
Hadoop Summit: Pattern – an open source project for migrating predictive mode...
byPaco Nathan
 
PPTX
A brief history of "big data"
byNicola Ferraro
 
PDF
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
byHortonworks
 
Functional programming
 for optimization problems 
in Big Data
byPaco Nathan
 
PDX Hadoop: Enterprise Data Workflows with Cascading and Mesos
byPaco Nathan
 
Accelerate Big Data Application Development with Cascading
byCascading
 
July Clojure Users Group Meeting: "Using Cascalog with Palo Alto Open Data"
byPaco Nathan
 
Using Cascalog to build an app with City of Palo Alto Open Data
byOSCON Byrum
 
OSCON 2013: Using Cascalog to build an app with City of Palo Alto Open Data
byPaco Nathan
 
Enterprise Data Workflows with Cascading and Windows Azure HDInsight
byPaco Nathan
 
Cascading concurrent yahoo lunch_nlearn
byCascading
 
Pattern - an open source project for migrating predictive models from SAS, et...
byDataWorks Summit
 
Elasticsearch + Cascading for Scalable Log Processing
byCascading
 
Intro to Cascading
byBen Speakmon
 
The Cascading (big) data application framework
byModern Data Stack France
 
The Cascading (big) data application framework - André Keple, Sr. Engineer, C...
byCascading
 
Augury and Omens Aside, Part 1:
 The Business Case for Apache Mesos
byPaco Nathan
 
Introduction to Cascading
byCascading
 
Hadoop User Group EU 2014
bycwensel
 
Cascading User Group Meet
byVinoth Kannan
 
Hadoop Summit: Pattern – an open source project for migrating predictive mode...
byPaco Nathan
 
A brief history of "big data"
byNicola Ferraro
 
Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...
byHortonworks
 

More from Paco Nathan

PDF
Human in the loop: a design pattern for managing teams working with ML
byPaco Nathan
 
PDF
Human-in-the-loop: a design pattern for managing teams that leverage ML
byPaco Nathan
 
PDF
Human-in-a-loop: a design pattern for managing teams which leverage ML
byPaco Nathan
 
PDF
Humans in a loop: Jupyter notebooks as a front-end for AI
byPaco Nathan
 
PDF
Humans in the loop: AI in open source and industry
byPaco Nathan
 
PDF
Computable Content
byPaco Nathan
 
PDF
Computable Content: Lessons Learned
byPaco Nathan
 
PDF
SF Python Meetup: TextRank in Python
byPaco Nathan
 
PDF
Use of standards and related issues in predictive analytics
byPaco Nathan
 
PDF
Data Science in 2016: Moving Up
byPaco Nathan
 
PDF
Data Science Reinvents Learning?
byPaco Nathan
 
PDF
Jupyter for Education: Beyond Gutenberg and Erasmus
byPaco Nathan
 
PDF
GalvanizeU Seattle: Eleven Almost-Truisms About Data
byPaco Nathan
 
PDF
Microservices, containers, and machine learning
byPaco Nathan
 
PDF
GraphX: Graph analytics for insights about developer communities
byPaco Nathan
 
PDF
Graph Analytics in Spark
byPaco Nathan
 
PDF
QCon São Paulo: Real-Time Analytics with Spark Streaming
byPaco Nathan
 
PDF
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and More
byPaco Nathan
 
PDF
A New Year in Data Science: ML Unpaused
byPaco Nathan
 
PDF
Microservices, Containers, and Machine Learning
byPaco Nathan
 
Human in the loop: a design pattern for managing teams working with ML
byPaco Nathan
 
Human-in-the-loop: a design pattern for managing teams that leverage ML
byPaco Nathan
 
Human-in-a-loop: a design pattern for managing teams which leverage ML
byPaco Nathan
 
Humans in a loop: Jupyter notebooks as a front-end for AI
byPaco Nathan
 
Humans in the loop: AI in open source and industry
byPaco Nathan
 
Computable Content
byPaco Nathan
 
Computable Content: Lessons Learned
byPaco Nathan
 
SF Python Meetup: TextRank in Python
byPaco Nathan
 
Use of standards and related issues in predictive analytics
byPaco Nathan
 
Data Science in 2016: Moving Up
byPaco Nathan
 
Data Science Reinvents Learning?
byPaco Nathan
 
Jupyter for Education: Beyond Gutenberg and Erasmus
byPaco Nathan
 
GalvanizeU Seattle: Eleven Almost-Truisms About Data
byPaco Nathan
 
Microservices, containers, and machine learning
byPaco Nathan
 
GraphX: Graph analytics for insights about developer communities
byPaco Nathan
 
Graph Analytics in Spark
byPaco Nathan
 
QCon São Paulo: Real-Time Analytics with Spark Streaming
byPaco Nathan
 
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and More
byPaco Nathan
 
A New Year in Data Science: ML Unpaused
byPaco Nathan
 
Microservices, Containers, and Machine Learning
byPaco Nathan
 

Recently uploaded

PPTX
OutSystems ONE 2025-recap presentation.pptx
bymostafaothman27
 
PDF
MicroPython presentation - PyConFr Lyon 2025 - Amine Bendahmane
byAmine Bendahmane
 
PDF
Create, View and Edit Text Files - RHCSA (RH124).pdf
byLinuxCert Guru
 
PDF
Revolutionizing SQL Database Design for the Modern Era.pdf
bySQL DBM
 
PDF
Analyze and Store Logs - RHCSA (RH124).pdf
byLinuxCert Guru
 
PPTX
Expanding Your Toolbox: Beginners Guide to Controlling Kubernetes Logs
byEric D. Schabell
 
PDF
How Tridens DevSecOps Ensures Compliance, Security, and Agility
byTridens
 
PDF
A Complete Beginner's Guide to Internet of Things 2025.pdf
bySecuodsoft
 
PDF
Q3'25 Financial Results and Earnings Presentation
byDropbox
 
PDF
From Pixels to Insights: Getting Started with Imagery in FME
bySafe Software
 
PDF
The Complete Crypto Airdrop Playbook: Strategies, Scams & Success Stories | 3...
by3verseTV
 
PDF
The Smol Training Playbook: The Secrets to Building World-Class LLMs
byRazin Mustafiz
 
PDF
Virtualization and Container Technologies in Cloud Security - ISC2 Certificate
byVICTOR MAESTRE RAMIREZ
 
PDF
GDG Boise - Innovating with Google Cloud Artificial Intelligence
byJoey Brown
 
PDF
Wrapping up unowned UTF8 C strings: CStringView
byIgalia
 
PDF
"Visual Guide to DSA: Big O Notation, Data Structures, and Algorithms Fundame...
byNusrat Gulbarga
 
PPSX
AppSec Role Based Training OWASP Global AppSec USA 2025-11-06
byRobert Grupe, CSSLP CISSP PE PMP
 
PDF
PDF Security Intelligence Platform | Cybersecurity Project by Ashish Patil
byAshishPatil495087
 
PDF
Unleash AI power with the Dell Pro 14 Plus - Interactive PDF
byPrincipled Technologies
 
PPTX
Your First Deep Learning project With CNN (workshop).pptx
byMuhammad Fahad Bashir
 
OutSystems ONE 2025-recap presentation.pptx
bymostafaothman27
 
MicroPython presentation - PyConFr Lyon 2025 - Amine Bendahmane
byAmine Bendahmane
 
Create, View and Edit Text Files - RHCSA (RH124).pdf
byLinuxCert Guru
 
Revolutionizing SQL Database Design for the Modern Era.pdf
bySQL DBM
 
Analyze and Store Logs - RHCSA (RH124).pdf
byLinuxCert Guru
 
Expanding Your Toolbox: Beginners Guide to Controlling Kubernetes Logs
byEric D. Schabell
 
How Tridens DevSecOps Ensures Compliance, Security, and Agility
byTridens
 
A Complete Beginner's Guide to Internet of Things 2025.pdf
bySecuodsoft
 
Q3'25 Financial Results and Earnings Presentation
byDropbox
 
From Pixels to Insights: Getting Started with Imagery in FME
bySafe Software
 
The Complete Crypto Airdrop Playbook: Strategies, Scams & Success Stories | 3...
by3verseTV
 
The Smol Training Playbook: The Secrets to Building World-Class LLMs
byRazin Mustafiz
 
Virtualization and Container Technologies in Cloud Security - ISC2 Certificate
byVICTOR MAESTRE RAMIREZ
 
GDG Boise - Innovating with Google Cloud Artificial Intelligence
byJoey Brown
 
Wrapping up unowned UTF8 C strings: CStringView
byIgalia
 
"Visual Guide to DSA: Big O Notation, Data Structures, and Algorithms Fundame...
byNusrat Gulbarga
 
AppSec Role Based Training OWASP Global AppSec USA 2025-11-06
byRobert Grupe, CSSLP CISSP PE PMP
 
PDF Security Intelligence Platform | Cybersecurity Project by Ashish Patil
byAshishPatil495087
 
Unleash AI power with the Dell Pro 14 Plus - Interactive PDF
byPrincipled Technologies
 
Your First Deep Learning project With CNN (workshop).pptx
byMuhammad Fahad Bashir
 

Boulder/Denver BigData: Cluster Computing with Apache Mesos and Cascading

  • 1.
    Boulder/Denver BigData, 2013-09-25: ClusterComputing with Apache Mesos and Cascading Paco Nathan @pacoid Chief Scientist, Mesosphere.io
  • 2.
    Cluster Computing with ApacheMesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing Boulder, 2013-09-25
  • 3.
    Enterprise Data Workflows middlewarefor Big Data applications is evolving, with commercial examples that include: Cascading, Lingual, Pattern, etc. Concurrent ParAccel Big Data Analytics Platform Actian Anaconda supporting IPython Notebook, Pandas,Augustus, etc. Continuum Analytics ETL data prep predictive model data sources end uses
  • 4.
    Anatomy of anEnterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end uses ANSI SQL for ETL
  • 5.
    Anatomy of anEnterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end usesJ2EE for business logic
  • 6.
    Anatomy of anEnterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end uses SAS for predictive models
  • 7.
    Anatomy of anEnterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end uses SAS for predictive modelsANSI SQL for ETL most of the licensing costs…
  • 8.
    Anatomy of anEnterprise app definition of a typical Enterprise workflow which crosses through multiple departments, languages, and technologies… ETL data prep predictive model data sources end usesJ2EE for business logic most of the project costs…
  • 9.
    ETL data prep predictive model data sources end uses Lingual: DW → ANSISQL Pattern: SAS, R, etc. → PMML business logic in Java, Clojure, Scala, etc. sink taps for Memcached, HBase, MongoDB, etc. source taps for Cassandra, JDBC, Splunk, etc. Anatomy of an Enterprise app Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source a compiler sees it all… one connected DAG: • optimization • troubleshooting • exception handling • notifications cascading.org
  • 10.
    a compiler seesit all… ETL data prep predictive model data sources end uses Lingual: DW → ANSI SQL Pattern: SAS, R, etc. → PMML business logic in Java, Clojure, Scala, etc. sink taps for Memcached, HBase, MongoDB, etc. source taps for Cassandra, JDBC, Splunk, etc. Anatomy of an Enterprise app Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source FlowDef flowDef = FlowDef.flowDef() .setName( "etl" ) .addSource( "example.employee", emplTap ) .addSource( "example.sales", salesTap ) .addSink( "results", resultsTap );   SQLPlanner sqlPlanner = new SQLPlanner() .setSql( sqlStatement );   flowDef.addAssemblyPlanner( sqlPlanner ); cascading.org
  • 11.
    a compiler seesit all… ETL data prep predictive model data sources end uses Lingual: DW → ANSI SQL Pattern: SAS, R, etc. → PMML business logic in Java, Clojure, Scala, etc. sink taps for Memcached, HBase, MongoDB, etc. source taps for Cassandra, JDBC, Splunk, etc. Anatomy of an Enterprise app Cascading allows multiple departments to combine their workflow components into an integrated app – one among many, typically – based on 100% open source FlowDef flowDef = FlowDef.flowDef() .setName( "classifier" ) .addSource( "input", inputTap ) .addSink( "classify", classifyTap );   PMMLPlanner pmmlPlanner = new PMMLPlanner() .setPMMLInput( new File( pmmlModel ) ) .retainOnlyActiveIncomingFields();   flowDef.addAssemblyPlanner( pmmlPlanner );
  • 12.
    Cascading – functionalprogramming Key insight: MapReduce is based on functional programming – back to LISP in 1970s. Apache Hadoop use cases are mostly about data pipelines, which are functional in nature. to ease staffing problems as “Main Street” Enterprise firms began to embrace Hadoop, Cascading was introduced in late 2007, as a new Java API to implement functional programming for large-scale data workflows: • leverages JVM and Java-based tools without any need to create new languages • allows programmers who have J2EE expertise to leverage the economics of Hadoop clusters Edgar Codd alluded to this (DSLs for structuring data) in his original paper about relational model
  • 13.
    Cascading – functionalprogramming • Twitter, eBay, LinkedIn, Nokia, YieldBot, uSwitch, etc., have invested in open source projects atop Cascading – used for their large-scale production deployments • new case studies for Cascading apps are mostly based on domain-specific languages (DSLs) in JVM languages which emphasize functional programming: Cascalog in Clojure (2010) Scalding in Scala (2012) github.com/nathanmarz/cascalog/wiki github.com/twitter/scalding/wiki Why Adopting the Declarative Programming PracticesWill ImproveYour Return fromTechnology Dan Woods, 2013-04-17 Forbes forbes.com/sites/danwoods/2013/04/17/why-adopting-the-declarative-programming- practices-will-improve-your-return-from-technology/
  • 14.
    Functional Programming forBig Data WordCount with token scrubbing… Apache Hive: 52 lines HQL + 8 lines Python (UDF) compared to Scalding: 18 lines Scala/Cascading functional programming languages help reduce software engineering costs at scale, over time
  • 15.
    Cascading – deployments •case studies: Climate Corp, Twitter, Etsy, Williams-Sonoma, uSwitch, Airbnb, Nokia, YieldBot, Square, Harvard, Factual, etc. • use cases: ETL, marketing funnel, anti-fraud, social media, retail pricing, search analytics, recommenders, eCRM, utility grids, telecom, genomics, climatology, agronomics, etc.
  • 16.
    Workflow Abstraction –pattern language Cascading uses a “plumbing” metaphor in Java to define workflows out of familiar elements: Pipes, Taps, Tuple Flows, Filters, Joins, Traps, etc. Scrub token Document Collection Tokenize Word Count GroupBy token Count Stop Word List Regex token HashJoin Left RHS M R data is represented as flows of tuples operations in the flows bring functional programming aspects into Java A Pattern Language Christopher Alexander, et al. amazon.com/dp/0195019199
  • 17.
    Workflow Abstraction –literate programming Cascading workflows generate their own visual documentation: flow diagrams in formal terms, flow diagrams leverage a methodology called literate programming provides intuitive, visual representations for apps – great for cross-team collaboration Scrub token Document Collection Tokenize Word Count GroupBy token Count Stop Word List Regex token HashJoin Left RHS M R Literate Programming Don Knuth literateprogramming.com
  • 18.
    Workflow Abstraction –business process following the essence of literate programming, Cascading workflows provide statements of business process this recalls a sense of business process management for Enterprise apps (think BPM/BPEL for Big Data) Cascading creates a separation of concerns between business process and implementation details (Hadoop, etc.) this is especially apparent in large-scale Cascalog apps: “Specify what you require, not how to achieve it.” by virtue of the pattern language, the flow planner then determines how to translate business process into efficient, parallel jobs at scale
  • 19.
    void map (Stringdoc_id, String text): for each word w in segment(text): emit(w, "1"); void reduce (String word, Iterator group): int count = 0; for each pc in group: count += Int(pc); emit(word, String(count)); The Ubiquitous Word Count Definition: this simple program provides an excellent test case for parallel processing: • requires a minimal amount of code • demonstrates use of both symbolic and numeric values • shows a dependency graph of tuples as an abstraction • is not many steps away from useful search indexing • serves as a “HelloWorld” for Hadoop apps a distributed computing framework that runsWord Count efficiently in parallel at scale can handle much larger and more interesting compute problems count how often each word appears in a collection of text documents
  • 20.
    Document Collection Word Count Tokenize GroupBy token Count R M 1 map 1reduce 18 lines code gist.github.com/3900702 WordCount – conceptual flow diagram cascading.org/category/impatient
  • 21.
    WordCount – Cascadingapp in Java String docPath = args[ 0 ]; String wcPath = args[ 1 ]; Properties properties = new Properties(); AppProps.setApplicationJarClass( properties, Main.class ); HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties ); // create source and sink taps Tap docTap = new Hfs( new TextDelimited( true, "t" ), docPath ); Tap wcTap = new Hfs( new TextDelimited( true, "t" ), wcPath ); // specify a regex to split "document" text lines into token stream Fields token = new Fields( "token" ); Fields text = new Fields( "text" ); RegexSplitGenerator splitter = new RegexSplitGenerator( token, "[ [](),.]" ); // only returns "token" Pipe docPipe = new Each( "token", text, splitter, Fields.RESULTS ); // determine the word counts Pipe wcPipe = new Pipe( "wc", docPipe ); wcPipe = new GroupBy( wcPipe, token ); wcPipe = new Every( wcPipe, Fields.ALL, new Count(), Fields.ALL ); // connect the taps, pipes, etc., into a flow FlowDef flowDef = FlowDef.flowDef().setName( "wc" ) .addSource( docPipe, docTap )  .addTailSink( wcPipe, wcTap ); // write a DOT file and run the flow Flow wcFlow = flowConnector.connect( flowDef ); wcFlow.writeDOT( "dot/wc.dot" ); wcFlow.complete(); Document Collection Word Count Tokenize GroupBy token Count R M
  • 22.
    mapreduce Every('wc')[Count[decl:'count']] Hfs['TextDelimited[[UNKNOWN]->['token', 'count']]']['output/wc']'] GroupBy('wc')[by:['token']] Each('token')[RegexSplitGenerator[decl:'token'][args:1]] Hfs['TextDelimited[['doc_id', 'text']->[ALL]]']['data/rain.txt']'] [head] [tail] [{2}:'token','count'] [{1}:'token'] [{2}:'doc_id', 'text'] [{2}:'doc_id', 'text'] wc[{1}:'token'] [{1}:'token'] [{2}:'token', 'count'] [{2}:'token', 'count'] [{1}:'token'] [{1}:'token'] WordCount – generated flow diagram Document Collection Word Count Tokenize GroupBy token Count R M
  • 23.
    (ns impatient.core   (:use [cascalog.api]         [cascalog.more-taps:only (hfs-delimited)])   (:require [clojure.string :as s]             [cascalog.ops :as c])   (:gen-class)) (defmapcatop split [line]   "reads in a line of string and splits it by regex"   (s/split line #"[[](),.)s]+")) (defn -main [in out & args]   (?<- (hfs-delimited out)        [?word ?count]        ((hfs-delimited in :skip-header? true) _ ?line)        (split ?line :> ?word)        (c/count ?count))) ; Paul Lam ; github.com/Quantisan/Impatient WordCount – Cascalog / Clojure Document Collection Word Count Tokenize GroupBy token Count R M
  • 24.
    github.com/nathanmarz/cascalog/wiki • implements Datalogin Clojure, with predicates backed by Cascading – for a highly declarative language • run ad-hoc queries from the Clojure REPL – approx. 10:1 code reduction compared with SQL • composable subqueries, used for test-driven development (TDD) practices at scale • Leiningen build: simple, no surprises, in Clojure itself • more new deployments than other Cascading DSLs – Climate Corp is largest use case: 90% Clojure/Cascalog • has a learning curve, limited number of Clojure developers • aggregators are the magic, and those take effort to learn WordCount – Cascalog / Clojure Document Collection Word Count Tokenize GroupBy token Count R M
  • 25.
    import com.twitter.scalding._   class WordCount(args: Args) extends Job(args) { Tsv(args("doc"), ('doc_id, 'text), skipHeader = true) .read .flatMap('text -> 'token) { text : String => text.split("[ [](),.]") } .groupBy('token) { _.size('count) } .write(Tsv(args("wc"), writeHeader = true)) } WordCount – Scalding / Scala Document Collection Word Count Tokenize GroupBy token Count R M
  • 26.
    github.com/twitter/scalding/wiki • extends theScala collections API so that distributed lists become “pipes” backed by Cascading • code is compact, easy to understand • nearly 1:1 between elements of conceptual flow diagram and function calls • extensive libraries are available for linear algebra, abstract algebra, machine learning – e.g., Matrix API, Algebird, etc. • significant investments by Twitter, Etsy, eBay, etc. • great for data services at scale • less learning curve than Cascalog WordCount – Scalding / Scala Document Collection Word Count Tokenize GroupBy token Count R M
  • 27.
    A Thought Exercise Considerthat when a company like Caterpillar moves into data science, they won’t be building the world’s next search engine or social network They will be optimizing supply chain, optimizing fuel costs, automating data feedback loops integrated into their equipment… Operations Research – crunching amazing amounts of data $50B company, in a $250B market segment Upcoming: tractors as drones – guided by complex, distributed data apps
  • 28.
  • 29.
    Two Avenues tothe App Layer… scale ➞ complexity➞ Enterprise: must contend with complexity at scale everyday… incumbents extend current practices and infrastructure investments – using J2EE, ANSI SQL, SAS, etc. – to migrate workflows onto Apache Hadoop while leveraging existing staff Start-ups: crave complexity and scale to become viable… new ventures move into Enterprise space to compete using relatively lean staff, while leveraging sophisticated engineering practices, e.g., Cascalog and Scalding
  • 30.
    Cluster Computing with ApacheMesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing Boulder, 2013-09-25
  • 31.
    Hadoop Cluster source tap source tap sink tap trap tap customer profile DBsCustomer Prefs logs logs Logs Data Workflow Cache Customers Support Web App Reporting Analytics Cubes sink tap ModelingPMML Lingual – ANSI SQL • collab with Optiq – industry-proven code base • ANSI SQL parser/optimizer atop Cascading flow planner • JDBC driver to integrate into existing tools and app servers • relational catalog over a collection of unstructured data • SQL shell prompt to run queries • enable analysts without retraining on Hadoop, etc. • transparency for Support, Ops, Finance, et al. a language for queries – not a database, but ANSI SQL as a DSL for workflows
  • 32.
    Lingual – CSVdata in local file system cascading.org/lingual
  • 33.
    Lingual – shellprompt, catalog cascading.org/lingual
  • 34.
  • 35.
    # load theJDBC package library(RJDBC)   # set up the driver drv <- JDBC("cascading.lingual.jdbc.Driver", "~/src/concur/lingual/lingual-local/build/libs/lingual-local-1.0.0-wip-dev-jdbc.jar")   # set up a database connection to a local repository connection <- dbConnect(drv, "jdbc:lingual:local;catalog=~/src/concur/lingual/lingual-examples/ tables;schema=EMPLOYEES")   # query the repository: in this case the MySQL sample database (CSV files) df <- dbGetQuery(connection, "SELECT * FROM EMPLOYEES.EMPLOYEES WHERE FIRST_NAME = 'Gina'") head(df)   # use R functions to summarize and visualize part of the data df$hire_age <- as.integer(as.Date(df$HIRE_DATE) - as.Date(df$BIRTH_DATE)) / 365.25 summary(df$hire_age) library(ggplot2) m <- ggplot(df, aes(x=hire_age)) m <- m + ggtitle("Age at hire, people named Gina") m + geom_histogram(binwidth=1, aes(y=..density.., fill=..count..)) + geom_density() Lingual – connecting Hadoop and R
  • 36.
    > summary(df$hire_age) Min. 1stQu. Median Mean 3rd Qu. Max. 20.86 27.89 31.70 31.61 35.01 43.92 Lingual – connecting Hadoop and R cascading.org/lingual
  • 37.
    Hadoop Cluster source tap source tap sink tap trap tap customer profile DBsCustomer Prefs logs logs Logs Data Workflow Cache Customers Support Web App Reporting Analytics Cubes sink tap ModelingPMML Pattern – model scoring • migrate workloads: SAS,Teradata, etc., exporting predictive models as PMML • great open source tools – R, Weka, KNIME, Matlab, RapidMiner, etc. • integrate with other libraries – Matrix API, etc. • leverage PMML as another kind of DSL cascading.org/pattern
  • 38.
    • established XMLstandard for predictive model markup • organized by Data Mining Group (DMG), since 1997 http://dmg.org/ • members: IBM, SAS, Visa, NASA, Equifax, Microstrategy, Microsoft, etc. • PMML concepts for metadata, ensembles, etc., translate directly into Cascading tuple flows “PMML is the leading standard for statistical and data mining models and supported by over 20 vendors and organizations.With PMML, it is easy to develop a model on one system using one application and deploy the model on another system using another application.” PMML – standard wikipedia.org/wiki/Predictive_Model_Markup_Language
  • 39.
  • 40.
    • Association Rules:AssociationModel element • Cluster Models: ClusteringModel element • Decision Trees: TreeModel element • Naïve Bayes Classifiers: NaiveBayesModel element • Neural Networks: NeuralNetwork element • Regression: RegressionModel and GeneralRegressionModel elements • Rulesets: RuleSetModel element • Sequences: SequenceModel element • SupportVector Machines: SupportVectorMachineModel element • Text Models: TextModel element • Time Series: TimeSeriesModel element PMML – model coverage ibm.com/developerworks/industry/library/ind-PMML2/
  • 41.
    ## train aRandomForest model   f <- as.formula("as.factor(label) ~ .") fit <- randomForest(f, data_train, ntree=50)   ## test the model on the holdout test set   print(fit$importance) print(fit)   predicted <- predict(fit, data) data$predicted <- predicted confuse <- table(pred = predicted, true = data[,1]) print(confuse)   ## export predicted labels to TSV   write.table(data, file=paste(dat_folder, "sample.tsv", sep="/"), quote=FALSE, sep="t", row.names=FALSE)   ## export RF model to PMML   saveXML(pmml(fit), file=paste(dat_folder, "sample.rf.xml", sep="/")) Pattern – create a model in R
  • 42.
    public static voidmain( String[] args ) throws RuntimeException { String inputPath = args[ 0 ]; String classifyPath = args[ 1 ]; // set up the config properties Properties properties = new Properties(); AppProps.setApplicationJarClass( properties, Main.class ); HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties );   // create source and sink taps Tap inputTap = new Hfs( new TextDelimited( true, "t" ), inputPath ); Tap classifyTap = new Hfs( new TextDelimited( true, "t" ), classifyPath );   // handle command line options OptionParser optParser = new OptionParser(); optParser.accepts( "pmml" ).withRequiredArg();   OptionSet options = optParser.parse( args );   // connect the taps, pipes, etc., into a flow FlowDef flowDef = FlowDef.flowDef().setName( "classify" ) .addSource( "input", inputTap ) .addSink( "classify", classifyTap );   if( options.hasArgument( "pmml" ) ) { String pmmlPath = (String) options.valuesOf( "pmml" ).get( 0 ); PMMLPlanner pmmlPlanner = new PMMLPlanner() .setPMMLInput( new File( pmmlPath ) ) .retainOnlyActiveIncomingFields() .setDefaultPredictedField( new Fields( "predict", Double.class ) ); // default value if missing from the model flowDef.addAssemblyPlanner( pmmlPlanner ); }   // write a DOT file and run the flow Flow classifyFlow = flowConnector.connect( flowDef ); classifyFlow.writeDOT( "dot/classify.dot" ); classifyFlow.complete(); } Pattern – score a model, within an app
  • 43.
    Cluster Computing with ApacheMesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing Boulder, 2013-09-25
  • 44.
    Q3 1997: inflectionpoint four independent teams were working toward horizontal scale-out of workflows based on commodity hardware this effort prepared the way for huge Internet successes in the 1997 holiday season… AMZN, EBAY, Inktomi (YHOO Search), then GOOG MapReduce and the Apache Hadoop open source stack emerged from this period
  • 45.
    RDBMS Stakeholder SQL Query result sets Excelpivot tables PowerPoint slide decks Web App Customers transactions Product strategy Engineering requirements BI Analysts optimized code Circa 1996: pre- inflection point
  • 46.
    RDBMS Stakeholder SQL Query result sets Excelpivot tables PowerPoint slide decks Web App Customers transactions Product strategy Engineering requirements BI Analysts optimized code Circa 1996: pre- inflection point “throw it over the wall”
  • 47.
    RDBMS SQL Query result sets recommenders + classifiers WebApps customer transactions Algorithmic Modeling Logs event history aggregation dashboards Product Engineering UX Stakeholder Customers DW ETL Middleware servletsmodels Circa 2001: post- big ecommerce successes
  • 48.
    RDBMS SQL Query result sets recommenders + classifiers WebApps customer transactions Algorithmic Modeling Logs event history aggregation dashboards Product Engineering UX Stakeholder Customers DW ETL Middleware servletsmodels Circa 2001: post- big ecommerce successes “data products”
  • 49.
    Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile,etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere
  • 50.
    Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile,etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere “optimize topologies”
  • 51.
    Amazon “Early Amazon: Splittingthe website” – Greg Linden glinden.blogspot.com/2006/02/early-amazon-splitting-website.html eBay “The eBay Architecture” – Randy Shoup, Dan Pritchett addsimplicity.com/adding_simplicity_an_engi/2006/11/you_scaled_your.html addsimplicity.com.nyud.net:8080/downloads/eBaySDForum2006-11-29.pdf Inktomi (YHOO Search) “Inktomi’s Wild Ride” – Erik Brewer (0:05:31 ff) youtu.be/E91oEn1bnXM Google “Underneath the Covers at Google” – Jeff Dean (0:06:54 ff) youtu.be/qsan-GQaeyk perspectives.mvdirona.com/2008/06/11/JeffDeanOnGoogleInfrastructure.aspx MIT Media Lab “Social Information Filtering for Music Recommendation” – Pattie Maes pubs.media.mit.edu/pubs/papers/32paper.ps ted.com/speakers/pattie_maes.html Primary Sources
  • 52.
    Cluster Computing’s DirtyLittle Secret many of us make a good living by leveraging high ROI apps based on clusters, and so execs agree to build out more data centers… clusters for Hadoop/HBase, for Storm, for MySQL, for Memcached, for Cassandra, for Nginx, etc. this becomes expensive! a single class of workloads on a given cluster is simpler to manage, but terrible for utilization… various notions of “cloud” help… Cloudera, Hortonworks, probably EMC soon: sell a notion of “Hadoop as OS” All your workloads are belong to us Google Data Center, Fox News ~2002
  • 53.
    Three Laws, ormore? meanwhile, architectures evolve toward much, much larger data… pistoncloud.com/ ... Rich Freitas, IBM Research Q: what disruptions in topologies+algorithms could this imply? given there’s no such thing as RAM anymore…
  • 54.
    Three Laws, ormore? meanwhile, architectures evolve toward much, much larger data… pistoncloud.com/ ... Rich Freitas, IBM Research regardless of how architectures change, death and taxes will endure: servers fail, data must move Q: what disruptions in topologies+algorithms could this imply? given there’s no such thing as RAM anymore…
  • 55.
    The Modern Kernel:Top Linux Contributors…
  • 56.
    Beyond Hadoop Hadoop –an open source solution for fault-tolerant parallel processing of batch jobs at scale, based on commodity hardware… however, other priorities have emerged for the analytics lifecycle: • apps require integration beyond Hadoop • multiple topologies, mixed workloads, multi-tenancy • higher utilization • lower latency • highly-available, long running services • more than “Just JVM” – e.g., Python growth keep in mind the priority for multi-disciplinary efforts, to break down even more silos – well beyond the de facto “priesthood” of data engineering
  • 57.
    Beyond Hadoop Google hasbeen doing data center computing for years, to address the complexities of large-scale data workflows: • leveraging the modern kernel: isolation in lieu of VMs • “most (>80%) jobs are batch jobs, but the majority of resources (55–80%) are allocated to service jobs” • mixed workloads, multi-tenancy • relatively high utilization rates • JVM? not so much… • reality: scheduling batch is simple; scheduling services is hard/expensive
  • 58.
    “Return of theBorg” Return of the Borg: HowTwitter Rebuilt Google’s SecretWeapon Cade Metz wired.com/wiredenterprise/ 2013/03/google-borg-twitter-mesos The Datacenter as a Computer: An Introduction to the Design ofWarehouse-Scale Machines Luiz André Barroso, Urs Hölzle research.google.com/pubs/ pub35290.html 2011 GAFS Omega John Wilkes, et al. youtu.be/0ZFMlO98Jkc
  • 59.
    “Return of theBorg” Omega: flexible, scalable schedulers for large compute clusters Malte Schwarzkopf,Andy Konwinski, Michael Abd-El-Malek, John Wilkes eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf
  • 60.
    Mesos – definitions acommon substrate for cluster computing heterogenous assets in your data center or cloud made available as a homogenous set of resources • top-level Apache project • scalability to 10,000s of nodes • obviates the need for virtual machines • isolation (pluggable) for CPU, RAM, I/O, FS, etc. • fault-tolerant replicated master using ZooKeeper • multi-resource scheduling (memory and CPU aware) • APIs in C++, Java, Python • web UI for inspecting cluster state • available for Linux, OpenSolaris, Mac OSX
  • 61.
    Mesos – architecture R uby Kernel Apps servicesbatch Frameworks Python JVM C ++ Workloads distributedfile system Chronos DFS distributed resources: CPU, RAM, I/O, FS, rack locality, etc. Cluster Storm Kafka JBoss Django RailsSharkImpalaScalding Marathon SparkHadoopMPI MySQL
  • 62.
    Mesos – architecture givenuse of Mesos as a Data Center OS kernel… • Chronos provides complex scheduling capabilities, much like a distributed Unix “cron” • Marathon provides highly-available long-running services, much like a distributed Unix “init.d” • next time you need to build a distributed app, consider using these as building blocks a major lesson learned from Spark: • leveraging these kinds of building blocks, one can rebuild Hadoop 100x faster, in much less code
  • 63.
    Mesos – datacenter OS stack HADOOP STORM CHRONOS RAILS JBOSS TELEMETRY Kernel OS Apps MESOS CAPACITY PLANNING GUISECURITYSMARTER SCHEDULING
  • 64.
    Prior Practice: DedicatedServers DATACENTER • low utilization rates • longer time to ramp up new services
  • 65.
    Prior Practice: Virtualization DATACENTERPROVISIONED VMS • even more machines to manage • substantial performance decrease due to virtualization • VM licensing costs
  • 66.
    Prior Practice: StaticPartitioning DATACENTER STATIC PARTITIONING • even more machines to manage • substantial performance decrease due to virtualization • VM licensing costs • static partitioning limits elasticity
  • 67.
    MESOS Mesos: One LargePool Of Resources DATACENTER “We wanted people to be able to program for the data center just like they program for their laptop." Ben Hindman
  • 68.
    What are thecosts of Virtualization? benchmark type OpenVZ improvement mixed workloads 210%-300% LAMP (related) 38%-200% I/O throughput 200%-500% response time order magnitude more pronounced at higher loads
  • 69.
    What are thecosts of Single Tenancy? 0% 25% 50% 75% 100% RAILS CPU LOAD MEMCACHED CPU LOAD 0% 25% 50% 75% 100% HADOOP CPU LOAD 0% 25% 50% 75% 100% t t 0% 25% 50% 75% 100% Rails Memcached Hadoop COMBINED CPU LOAD (RAILS, MEMCACHED, HADOOP)
  • 70.
    M Master Docker Registry index.docker.io Local Docker Registry ( optional ) M M S S S S S S marathon docker docker docker Mesos masterservers Mesos slave servers Marathon can launch and monitor service containers from one or more Docker registries, using the Docker executor for Mesos S S S S S S … … … … … … … mesosphere.io/2013/09/26/docker-on-mesos/ Example: Docker on Mesos
  • 71.
    Mesos Master Server init | +mesos-master | + marathon | Mesos Slave Server init | + docker | | | + lxc | | | + (user task, under container init system) | | | + mesos-slave | | | + /var/lib/mesos/executors/docker | | | | | + docker run … | | | The executor, monitored by the Mesos slave, delegates to the local Docker daemon for image discovery and management. The executor communicates with Marathon via the Mesos master and ensures that Docker enforces the specified resource limitations. mesosphere.io/2013/09/26/docker-on-mesos/ Example: Docker on Mesos
  • 72.
    Mesos Master Server init | +mesos-master | + marathon | Mesos Slave Server init | + docker | | | + lxc | | | + (user task, under container init system) | | | + mesos-slave | | | + /var/lib/mesos/executors/docker | | | | | + docker run … | | | Docker Registry When a user requests a container… Mesos, LXC, and Docker are tied together for launch 2 1 3 4 5 6 7 8 Example: Docker on Mesos mesosphere.io/2013/09/26/docker-on-mesos/
  • 73.
    Arguments for DataCenter Computing rather than running several specialized clusters, each at relatively low utilization rates, instead run many mixed workloads obvious benefits are realized in terms of: • scalability, elasticity, fault tolerance, performance, utilization • reduced equipment cap­ex, Ops overhead, etc. • reduced licensing, eliminating need forVMs or potential vendor lock­in subtle benefits – arguably, more important for Enterprise IT: • reduced time for engineers to ramp­up new services at scale • reduced latency between batch and services, enabling new high­ROI use cases • enables Dev/Test apps to run safely on a Production cluster
  • 74.
  • 75.
    Opposite Ends ofthe Spectrum, One Substrate Built-in / bare metal Hypervisors Solaris Zones Linux CGroups
  • 76.
    Opposite Ends ofthe Spectrum, One Substrate Request / Response Batch
  • 77.
    Case Study: Twitter(bare metal / on premise) “Mesos is the cornerstone of our elastic compute infrastructure – it’s how we build all our new services and is critical forTwitter’s continued success at scale. It's one of the primary keys to our data center efficiency." Chris Fry, SVP Engineering blog.twitter.com/2013/mesos-graduates-from-apache-incubation • key services run in production: analytics, typeahead, ads • Twitter engineers rely on Mesos to build all new services • instead of thinking about static machines, engineers think about resources like CPU, memory and disk • allows services to scale and leverage a shared pool of servers across data centers efficiently • reduces the time between prototyping and launching
  • 78.
    Case Study: Airbnb(fungible cloud infrastructure) “We think we might be pushing data science in the field of travel more so than anyone has ever done before… a smaller number of engineers can have higher impact through automation on Mesos." Mike Curtis,VP Engineering gigaom.com/2013/07/29/airbnb-is-engineering-itself-into-a-data-driven... • improves resource management and efficiency • helps advance engineering strategy of building small teams that can move fast • key to letting engineers make the most of AWS-based infrastructure beyond just Hadoop • allowed company to migrate off Elastic MapReduce • enables use of Hadoop along with Chronos, Spark, Storm, etc.
  • 79.
    Media Coverage Play FrameworkGrid Deployment with Mesos James Ward, Flo Leibert, et al. Typesafe blog (2013-09-19) typesafe.com/blog/play-framework-grid... Mesosphere Launches Marathon Framework Adrian Bridgwater Dr. Dobbs (2013-09-18) drdobbs.com/open-source/mesosphere... New open source tech Marathon wants to make your data center run like Google’s Derrick Harris GigaOM (2013-09-04) gigaom.com/2013/09/04/... Running batch and long-running, highly available service jobs on the same cluster Ben Lorica O’Reilly (2013-09-01) strata.oreilly.com/2013/09/...
  • 80.
    Resources Apache Mesos Project mesos.apache.org Mesosphere mesosphere.io Tutorial mesosphere.io/2013/08/01/... Documentation mesos.apache.org/documentation 2011USENIX Research Paper usenix.org/legacy/event/nsdi11/tech/full_papers/Hindman_new.pdf Collected Notes/Archives goo.gl/jPtTP
  • 81.
    Cluster Computing with ApacheMesos and Cascading: 1. Enterprise Data Workflows 2. Lingual and Pattern Examples 3. An Evolution of Cluster Computing SUMMARY… Boulder, 2013-09-25
  • 82.
    Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile,etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony
  • 83.
    Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile,etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 1. End Use Cases, the drivers
  • 84.
    Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile,etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 2. A new kind of team process
  • 85.
    Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile,etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 3. Abstraction layer as optimizing middleware, e.g., Cascading
  • 86.
    Workflow RDBMS near timebatch services transactions, content social interactions Web Apps, Mobile,etc.History Data Products Customers RDBMS Log Events In-Memory Data Grid Hadoop, etc. Cluster Scheduler Prod Eng DW Use Cases Across Topologies s/w dev data science discovery + modeling Planner Ops dashboard metrics business process optimized capacitytaps Data Scientist App Dev Ops Domain Expert introduced capability existing SDLC Circa 2013: clusters everywhere – Four-Part Harmony 4. Data Center OS, e.g., Mesos
  • 87.
    Enterprise DataWorkflows withCascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do monthly newsletter for updates, events, conference summaries, etc.: liber118.com/pxn/