![df: Gentle Incline
“I want to put my projects on hold, and learn several new things simultaneously”
- No One Ever
• Attempts to provide an API on Spark that looks and feels like
pandas data frame
e.g. in pandas
df[“a”]
in df
df(“a”)
• Also intuitive for R programmers](https://image.slidesharecdn.com/df-big-data-conf-140910192455-phpapp01/85/df-Dataframe-on-Spark-13-320.jpg)
![Advantages
• Quite transparently runs on Spark: Distributed processing
• Is in Scala: No layering overhead
• Is in Scala: Can directly call cutting edge Spark libraries like
MLLib [pyspark wrappers usually a bit behind]
• Is an “internal DSL”: Advanced users can augment with
arbitrary Scala code. [python wrapper still possible]
• Is an “internal DSL”: Fast without resorting to code-generation
• Fully open sourced, Apache license](https://image.slidesharecdn.com/df-big-data-conf-140910192455-phpapp01/85/df-Dataframe-on-Spark-14-320.jpg)
![Real Life Examples
Snippets of data scientist code that was “converted” from Pandas to df larger data to make it scale to
Add a column with total
mppu[“total”] = mppu[“avg”] * mppu['c_line_srvc_cnt']
—>
mppu(“total”) = mppu(“avg”) * mppu(“c_line_srvc_cnt”)
Remove $ and , from numbers representing money
mppu[“de-comma”] = mppu[“dollar”].str.replace(‘$','')
mppu[“de-dollar”] = mppu[“de-comma”].str.replace(‘,’,’').astype(float)
—>
mppu(“de-dollar”) = mppu(“dollar”).map
{ x: String => x.replace("$", "").replace(",","").toDouble }](https://image.slidesharecdn.com/df-big-data-conf-140910192455-phpapp01/85/df-Dataframe-on-Spark-15-320.jpg)
df: Dataframe on Spark
- 2. df: Dataframe on Spark Mohit Jaggi Code Ninja and Troublemaker at
- 3. Agenda • About Ayasdi • df • Brief Demo (if time allows) • Conclusion
- 4. About Ayasdi
- 5. Traditional Analytics CODE ? Hypothesis
- 7. Ayasdi Solution UX Ayasdi Platform Distributed Computing Algorithmic Reach ETL
- 8. df
- 9. Day in a data scientist’s life • Get data • Need more/something else • Data wrangling • Rinse, repeat • Load into analysis software like Ayasdi Core • Actual data analysis, model-building etc
- 10. Data Wrangling Tools • grep, cut, wc -l, head, tail • Python Pandas • Most useful construct: pandas data frame ala Excel with CLI
- 11. Challenges • Applying data science techniques to data larger than single machine’s memory • Easy to procure cluster of small machines than one big machine • Processing takes too long
- 12. Solution: Distribute • Hadoop ecosystem: Spark is great • Learning curve, what is this RDD thing? where is my familiar data frame? • There is pyspark but to get the best out of Spark use Scala, another learning curve
- 13. df: Gentle Incline “I want to put my projects on hold, and learn several new things simultaneously” - No One Ever • Attempts to provide an API on Spark that looks and feels like pandas data frame e.g. in pandas df[“a”] in df df(“a”) • Also intuitive for R programmers
- 14. Advantages • Quite transparently runs on Spark: Distributed processing • Is in Scala: No layering overhead • Is in Scala: Can directly call cutting edge Spark libraries like MLLib [pyspark wrappers usually a bit behind] • Is an “internal DSL”: Advanced users can augment with arbitrary Scala code. [python wrapper still possible] • Is an “internal DSL”: Fast without resorting to code-generation • Fully open sourced, Apache license
- 15. Real Life Examples Snippets of data scientist code that was “converted” from Pandas to df larger data to make it scale to Add a column with total mppu[“total”] = mppu[“avg”] * mppu['c_line_srvc_cnt'] —> mppu(“total”) = mppu(“avg”) * mppu(“c_line_srvc_cnt”) Remove $ and , from numbers representing money mppu[“de-comma”] = mppu[“dollar”].str.replace(‘$','') mppu[“de-dollar”] = mppu[“de-comma”].str.replace(‘,’,’').astype(float) —> mppu(“de-dollar”) = mppu(“dollar”).map { x: String => x.replace("$", "").replace(",","").toDouble }
- 16. Demo
- 17. Future • pyspark wrapper • more data sources like SQL, parquet, HDF5 etc • charts and graphs • contributors welcome!
- 18. Conclusion
- 19. Summary • pandas is awesome • df scales to bigger data, looks and feels like pandas • fully open source https://github.com/AyasdiOpenSource/df • Check out our website. We are hiring! http://engineering.ayasdi.com/ http://www.ayasdi.com/careers/
- 20. Acknowledgements • Max Song for introducing me to Pandas • Jean-Ezra Young for insurance claims example • Ayasdi for open-sourcing this work • Hadoop and Spark communities for the awesome platform • Pandas team for the awesome tool