Scalding big ADta
2 likes2,176 views
1. Scalding is a library that provides a concise domain-specific language (DSL) for writing MapReduce jobs in Scala. It allows defining source and sink connectors, as well as data transformation operations like map, filter, groupBy, and join in a more readable way than raw MapReduce APIs. 2. Some use cases for Scalding include splitting or reusing data streams, handling exotic data sources like JDBC or HBase, performing joins, distributed caching, and building connected user profiles by bridging data from different sources. 3. For connecting user profiles, Scalding can be used to model the data as a graph with vertices for user interests and edges for bridging rules.
![Facts about Data Scientists
• Data Scientists do:
– Audience Modeling
identifying new user interests
[segments] and finding ways
to track them
– Audience Bridging
– Insights and Analytics
• They use IBM Netezza as
local warehouse
• They use R language](https://image.slidesharecdn.com/scaldingbigadta-141209034046-conversion-gate01/85/Scalding-big-ADta-15-320.jpg)
![Facts about Offline team
• The tasks we solve over hadoop:
– As a Storage to keep all logs we need
– As Profile DB to keep all users and their interests [segments]
– As MapReduce Engine to run jobs on transformations between data
– As a Warehouse to export data via hive
• We use Clouderra CDH 5.1.2
• Major language: Scala
• Pure MapReduce jobs & Scalding/Cascading
• All map reduce applications are wrapped by Oozie’s workflow(s)
• Developing nextgen paltform version based on Spark Streaming/Kafka](https://image.slidesharecdn.com/scaldingbigadta-141209034046-conversion-gate01/85/Scalding-big-ADta-17-320.jpg)
![Just one example (Java way)
public class WordCount {
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}](https://image.slidesharecdn.com/scaldingbigadta-141209034046-conversion-gate01/85/Scalding-big-ADta-20-320.jpg)
![Source
Just one example
(Scalding way)
class WordCountJob(args : Args) extends Job(args) {
TextLine( args("input") )
.flatMap('line -> 'word) { line : String => tokenize(line) }
.groupBy('word) { _.size }
.write( Tsv( args("output") ) )
// Split a piece of text into individual words.
def tokenize(text : String) : Array[String] = {
// Lowercase each word and remove punctuation.
text.toLowerCase.split("s+")
}
}
Sink
Transform
operations](https://image.slidesharecdn.com/scaldingbigadta-141209034046-conversion-gate01/85/Scalding-big-ADta-21-320.jpg)
![Bridging Profiles
General task definition:
• Build graph
• Vertexes – user’s interests
• Edges – bridging rules
[cookies, IP,…]
• Task – Identify
connected components](https://image.slidesharecdn.com/scaldingbigadta-141209034046-conversion-gate01/85/Scalding-big-ADta-28-320.jpg)
![Connected components
Let’s scalding it
/**
* The class represents just one iteration of searching connected component algorithm.
* Somewhere outside the Job code we have to run this job iteratively until N [~20] and should check number inside "count" file.
* If it is zero then we can stop running other iterations
*/
class ConnectedComponentsOneIterationJob(args : Args) extends Job(args) {
val vertexes = Tsv( args("vertexes"),('id,'gid)).read // by default gid is equal to id
val edges = Tsv( args("edges"), ('id_a,'id_b) ).read
val groups = vertexes.joinWithSmaller('id -> 'id_b, vertexes.joinWithSmaller('id -> 'id_a, edges).discard('id ).rename('gid ->'gid_a))
.discard('id )
.rename('gid ->'gid_b)
.filter('gid_a, 'gid_b) {gid : (String, String) => gid._1 != gid._2 }
.project ('gid_a, 'gid_b)
.mapTo(('gid_a, 'gid_b) -> ('gid_a, 'gid_b)) {gid : (String, String) => max(gid._1, gid._2) -> min(gid._1, gid._2) }
// if count=0 then we can stop running next iterations
groups.groupAll { _.size }.write(Tsv("count"))
val new_groups = groups.groupBy('gid_a) {_.min('gid_b)}.rename(('gid_a,'gid_b)->('source, 'target))
val new_vertexes = vertexes.joinWithSmaller('id -> 'source, new_groups, joiner = new LeftJoin )
.mapTo( ('id,'gid,'source,'target)->('id, 'gid)) { param:(String, String, String, String) =>
val (id, gid, source,target) = param
if (target != null) ( id , min( gid, target ) ) else ( id, gid )
}
new_vertexes.write( Tsv( args("new_vertexes") ) )
}](https://image.slidesharecdn.com/scaldingbigadta-141209034046-conversion-gate01/85/Scalding-big-ADta-29-320.jpg)
Scalding big ADta
- 1. Scalding Big ADta или обжигая горшки с рекламой Boris Trofimov @b0ris_1
- 2. Agenda • Two stories on how AD is served inside AD company • Awesome Scalding
- 3. The story about shoes or Big Brother is watching on you
- 4. We will answer this question in a few slides or be careful while buying shoes
- 5. How many web sites with Ad aboard do you open during the day?
- 6. Open any site with Ad
- 7. What can be simpler than loading web site via web browser, hah?
- 9. However it is deceptive judgment
- 10. The first second… Story Actors User Publisher (foxnews.com) Ad Server (Google’s Doubleclick) SSP (Ad Exchange) DSP (decides what ad to show) Advertiser (Nike) we are here
- 11. … 1 sec20 ms 100 ms 150 ms Publisher receives request Publisher sends response Content delivered to user 170 Site sends request to Ad Server 200 80 ms 280 SSP picks the winning bid and sends redirect url back to ad Server Every bidder/DSP receives info about user: • ssp_cookie_id • geo data • site url 300 SSP (Ad Exchange) receives ad request and opens RTB Auction 210 Ad Server receives ad request and redirects to Ad Exchange All bidders should send their decision (participate? & price) back 350 Ad Server shows page to user which redirects to the bidder’s server User’s web page asks Ad banner from CDN Showing ad & bidder’s 1x1 pixel (impression) 400 The first second… ~70% users have this cookie aboard >>1 independent companies take part in this auction
- 12. Under hood
- 13. Return info about new user’s interests with special markers (segments) that indicates the new fact about user, e.g user is man who has iphone and lives in NYC and has dog. Major format: <cookie_id – segment_id> Data Scientists Real time Offline Pixel Tracking Farm Warehouse Bidder Farm Auction requests SSP Ad Exchange Hourly Logs 3rd part data House holders data … Hadoop’s HDFS Updating user profiles HiveOozieMapReduce Partners HBASEScalding hbase keeps user profiles Update user’s profiles with new segments Data export Brand new feed about user interests 2 3 45 6 7 8 9 0 1 • Impressions • Clicks • Post-click Activities 5
- 14. Why do we need all this science? • Deep audience targeting • Case: customer would like to show ad for all men who live in NYC have iPhone and dog
- 15. Facts about Data Scientists • Data Scientists do: – Audience Modeling identifying new user interests [segments] and finding ways to track them – Audience Bridging – Insights and Analytics • They use IBM Netezza as local warehouse • They use R language
- 16. Facts about Realtime team • Scala, Java • Restful Services • Akka • In Memory Cache : Aerospike, Redis
- 17. Facts about Offline team • The tasks we solve over hadoop: – As a Storage to keep all logs we need – As Profile DB to keep all users and their interests [segments] – As MapReduce Engine to run jobs on transformations between data – As a Warehouse to export data via hive • We use Clouderra CDH 5.1.2 • Major language: Scala • Pure MapReduce jobs & Scalding/Cascading • All map reduce applications are wrapped by Oozie’s workflow(s) • Developing nextgen paltform version based on Spark Streaming/Kafka
- 19. hdfs Scalding in a nutshell • Concise DSL • Configurable Source(s) and sink(s) • Data transform operations: – map/flatMap – pivot/unpivot – project – groupBy/reduce/foldLeft hdfs
- 20. Just one example (Java way) public class WordCount { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); } }
- 21. Source Just one example (Scalding way) class WordCountJob(args : Args) extends Job(args) { TextLine( args("input") ) .flatMap('line -> 'word) { line : String => tokenize(line) } .groupBy('word) { _.size } .write( Tsv( args("output") ) ) // Split a piece of text into individual words. def tokenize(text : String) : Array[String] = { // Lowercase each word and remove punctuation. text.toLowerCase.split("s+") } } Sink Transform operations
- 22. Use Case 1 Split • Motivation: reuse calculated streams val common = Tsv("./file").map(...) val branch1 = common.map(..).write(Tsv("output")) val branch2 = common.groupby(..).write(Tsv("output"))
- 23. Use Case 2 Exotic Sources JDBC (out of the box) case object YourTableSource extends JDBCSource { override val tableName = "tableName" override val columns = List( varchar("col1", 64), date("col2"), tinyint("col3"), double("col4"), ) override def currentConfig = ConnectionSpec("www.gt.com", "username", "password", "mysql") } YourTableSource.read.map(...) ...
- 24. Use Case 2 Exotic Sources HBASE HBaseSource (https://github.com/ParallelAI/SpyGlass) • SCAN_ALL, • GET_LIST, • SCAN_RANGE HBaseRawSource (https://github.com/andry1/SpyGlass) • Advanced filtering via base64Scan val hbs3 = new HBaseSource( tableName, quorum, 'key, List("data"), List('data), sourceMode = SourceMode.SCAN_ALL) .read val scan = new Scan() scan.setCaching(caching) val activity_filters = new FilterList(MUST_PASS_ONE, { val scvf = new SingleColumnValueFilter(toBytes("family"), toBytes("column"), GREATER_OR_EQUAL, toBytes(value)) scvf.setFilterIfMissing(true) scvf.setLatestVersionOnly(true) val scvf2 = ... List(scvf, scvf2) }) scan.setFilter(activity_filters) new HBaseRawSource(tableName, quorum, families, base64Scan = convertScanToBase64(scan)).read. ...
- 25. Use Case 3 Join • Motivation: joining two streams by key • Different join strategies: – joinWithLarger – joinWithSmaller – joinWithTiny • Inner, Left, Right, strategies val pipe1 = Tsv("file1").read val pipe2 = Tsv("file2").read // small file val pipe3 = Tsv("file3").read // huge file val joinedPipe = pipe1.joinWithTiny('id1 -> 'id2, pipe2) val joinedPipe2 = pipe1.joinWithLarge('id1 -> 'id2, pipe3)
- 26. Use Case 4 Distributed Caching and Counters //somewhere outside Job definition val fl = DistributedCacheFile("/user/boris/zooKeeper.json") // next value can be passed through any Scalding's jobs via Args object for instance val fileName = fl.path ... class Job(val args:Args) { // once we receive fl.path we can read it like a ordinary file val fileName = args.get("fileName") lazy val data = readJSONFromFile(fileName) ... TSV(args.get("input")).read.map('line -> 'word ) { line => ... /* using data json object*/ ... } } // counter example Stat("jdbc.call.counter","myapp").incBy(1)
- 27. Use Case 5 Bridging Profiles Motivation: bridge information from different sources and build complete person profile imp Own company’s private cookie thanks to 1x1 pixel impression Bridging two ssp_cookies via private cookie ssp_cookie_Id1 ssp_cookie_Id2 Bridging via ip address
- 28. Bridging Profiles General task definition: • Build graph • Vertexes – user’s interests • Edges – bridging rules [cookies, IP,…] • Task – Identify connected components
- 29. Connected components Let’s scalding it /** * The class represents just one iteration of searching connected component algorithm. * Somewhere outside the Job code we have to run this job iteratively until N [~20] and should check number inside "count" file. * If it is zero then we can stop running other iterations */ class ConnectedComponentsOneIterationJob(args : Args) extends Job(args) { val vertexes = Tsv( args("vertexes"),('id,'gid)).read // by default gid is equal to id val edges = Tsv( args("edges"), ('id_a,'id_b) ).read val groups = vertexes.joinWithSmaller('id -> 'id_b, vertexes.joinWithSmaller('id -> 'id_a, edges).discard('id ).rename('gid ->'gid_a)) .discard('id ) .rename('gid ->'gid_b) .filter('gid_a, 'gid_b) {gid : (String, String) => gid._1 != gid._2 } .project ('gid_a, 'gid_b) .mapTo(('gid_a, 'gid_b) -> ('gid_a, 'gid_b)) {gid : (String, String) => max(gid._1, gid._2) -> min(gid._1, gid._2) } // if count=0 then we can stop running next iterations groups.groupAll { _.size }.write(Tsv("count")) val new_groups = groups.groupBy('gid_a) {_.min('gid_b)}.rename(('gid_a,'gid_b)->('source, 'target)) val new_vertexes = vertexes.joinWithSmaller('id -> 'source, new_groups, joiner = new LeftJoin ) .mapTo( ('id,'gid,'source,'target)->('id, 'gid)) { param:(String, String, String, String) => val (id, gid, source,target) = param if (target != null) ( id , min( gid, target ) ) else ( id, gid ) } new_vertexes.write( Tsv( args("new_vertexes") ) ) }
- 30. Other nice things • Typed pipes • Elegant and fast Matrix operations • Simple migration on Spark/Kafka • More sources: e.g. retrieve data from hive’s hcatalog
- 31. Useful Resources • http://www.adopsinsider.com/ad-serving/how-does-ad-serving- work/ • http://www.adopsinsider.com/ad-serving/diagramming-the-ssp- dsp-and-rtb-redirect-path/ • https://github.com/twitter/scalding • https://github.com/ParallelAI/SpyGlass • https://github.com/branky/cascading.hive
- 32. Thank you!