End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
3 likes1,777 views
This talk will go over how to build an end-to-end data processing system in Python, from data ingest, to data analytics, to machine learning, to user presentation. Developments in old and new tools have made this particularly possible today. The talk in particular will talk about Airflow for process workflows, PySpark for data processing, Python data science libraries for machine learning and advanced analytics, and building agile microservices in Python. System architects, software engineers, data scientists, and business leaders can all benefit from attending the talk. They should learn how to build more agile data processing systems and take away some ideas on how their data systems could be simpler and more powerful.
![Distributed Computing
# Read data as lines from a source
lines = spark.read.text(inpath).rdd.map(lambda r: r[0])
# Count the data
counts = lines.flatMap(lambda x: x.split(' '))
.map(lambda x: (x, 1))
.reduceByKey(add)
# Bring it locally
output = counts.collect()](https://image.slidesharecdn.com/donaldminer-pythonbigdata-170902025018/85/End-to-end-Big-Data-Projects-with-Python-StampedeCon-Big-Data-Conference-2017-17-320.jpg)
![# Spark job to build feature vectors
rows = myrdd.map(lambda r: r[0].split(‘,’))
out = rows.map(lambda row: (row[0], row))
.groupByKey().map(build_feature_vector) # outputs [(FV, label)]
# Bring data down locally and prepare it
localout = counts.collect()
X = [ row[0] for row in localout ] # feature is set of 40 aggregate properties
t = [ row[1] for row in localout ] # potential labels are types of devices
# Train a RF classifier on it
clf = sklearn.ensemble.RandomForestClassifier(n_estimators=250)
clf = clf.fit(X, y)
# save the model (maybe to s3 instead?)
pickle.dump(clf, open(‘/models/behavior.sklearn’, ‘w’))
--- training data sample of netflow ---
SOURCE IP DEST IP DATE STIME ETIME DATAIN DATAOUT
123.41.12.31, 123.41.155.32, 2017-02-01, 09:00, 09:59, 103KB, 959KB
123.41.59.99, 123.41.155.32, 2017-02-01, 09:00, 09:59, 44KB, 884KB
123.41.12.31, 123.41.155.32, 2017-02-01, 10:00, 10:59, 3KB, 9KB
123.41.59.99, 123.41.155.32, 2017-02-01, 10:00, 10:59, 4KB, 15KB](https://image.slidesharecdn.com/donaldminer-pythonbigdata-170902025018/85/End-to-end-Big-Data-Projects-with-Python-StampedeCon-Big-Data-Conference-2017-23-320.jpg)
![# http://56.120.177.55/predictip?ip=159.31.120.44
# http://56.120.177.55/predictfv -- for POST of feature vector
_MODEL = pickle.load(open(’/models/maintenance.sklearn’))
@app.route('/predictrepair')
def predicttypefrombehavior():
netflowlog = request.form[‘logcsv’]
fv = build_feature_vetor(netflowlog)
pr = _MODEL.predict(fv)
return json.dumps({ ‘query’ : fv,
‘prediction’ : pr })
# returns { ‘query’ : [9, 4, 123.1, …], ‘prediction’ : ‘HTTP PROXY’ }
--- training data sample of netflow ---
SOURCE IP DEST IP DATE STIME ETIME DATAIN DATAOUT
123.41.12.31, 123.41.155.32, 2017-02-01, 09:00, 09:59, 103KB, 959KB
123.41.59.99, 123.41.155.32, 2017-02-01, 09:00, 09:59, 44KB, 884KB
123.41.12.31, 123.41.155.32, 2017-02-01, 10:00, 10:59, 3KB, 9KB
123.41.59.99, 123.41.155.32, 2017-02-01, 10:00, 10:59, 4KB, 15KB](https://image.slidesharecdn.com/donaldminer-pythonbigdata-170902025018/85/End-to-end-Big-Data-Projects-with-Python-StampedeCon-Big-Data-Conference-2017-24-320.jpg)
More Related Content
What's hot (20)
Similar to End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017 (20)
More from StampedeCon (20)
Recently uploaded (20)
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
- 1. DONALD MINER End-to-end Big Data Projects with Python DONALD MINER StampedeCon Big Data July 25th, 2017
- 4. Real “Big Data” isn’t just about platforms anymore
- 5. Real “Big Data” isn’t just about platforms anymore Streaming Infrastructure Cloud Applications Mobile {
- 6. Real “Big Data” isn’t just about data processing anymore
- 7. Real “Big Data” isn’t just about data processing anymore Machine Learning Data Science NLP Deep Learning Visualization {
- 8. 2017 BIG DATA = & integrated and user facing & advanced analytics
- 9. Big Data in 2009 was so Java oriented. It was easier to use Java for everything or use a collection of random languages.
- 10. Python seemed to have everything we wanted, except for Big Data Some brave souls tried: Hadoopy, mrjob, Pig+Python
- 11. PySpark PySpark was the missing piece of the Big Data Python picture The first major Big Data platform with first-class Python support Thanks to PySpark, Python is now a viable and competitive option for end-to-end systems that utilize Big Data
- 12. What’s the big deal? Python has best in class functionality for all the other things we want to do with Big Data: Data manipulation, Machine Learning, Text, Applications, Visualization In 2017, we can build end-to-end Big Data systems entirely in Python: from ingest to user experience and everything between
- 13. The case for Python Succinct code that’s easy to read
- 14. The case for Python A language people know
- 15. The case for Python Interpreted, not compiled
- 16. The Python Big Data Architecture
- 17. Distributed Computing # Read data as lines from a source lines = spark.read.text(inpath).rdd.map(lambda r: r[0]) # Count the data counts = lines.flatMap(lambda x: x.split(' ')) .map(lambda x: (x, 1)) .reduceByKey(add) # Bring it locally output = counts.collect()
- 18. Machine Learning # Initialize Random Forest classifier clf = sklearn.ensemble.RandomForestClassifier(n_estimators=250) # Train Random Forest classifier clf = clf.fit(feature_vectors, labels) Why sklearn over MLLib?
- 19. Deep Learning # Load the ImageNet pretrained network model = VGG16(weights="imagenet") # Run the model on an image preds = model.predict(preprocess_input(image)) # Hotdog or not hotdog? print(decode_predictions(preds)) & Keras Also excited about pytorch!
- 20. Visualization Lots of visualization options in Python • Seaborn • Matplotlib • Bokeh • ggplot seaborn.swarmplot(x="measurement", y="value", hue="species", data=iris)
- 21. Integration # http://56.120.177.55/hello?name=Don @app.route('/hello') def say_hello(): name = request.args.get(‘name’) return json.dumps({ ‘query’ : name, ‘message’: ‘HELLO ‘ + name }) # returns { ‘query’ : ‘Don’, ‘message’ : ‘HELLO Don’ }
- 22. Workflows # Run this every day at 3:45AM mdag = DAG(’DRSpark', description=’DailyRun', schedule_interval=’45 3 * * *') sp1 = PythonOperator(task_id=‘sp1’, python_callable=runspark1, dag=mdag) sp2 = PythonOperator(task_id=’sp2’, python_callable=runspark2, dag=mdag) ou = PythonOperator(task_id=‘clean’, python_callable=cleanupresults, dag=mdag) sp1 >> ou # sp1 happens before ou sp2 >> ou # sp2 happens before ou, but doesn’t depend on sp1
- 23. # Spark job to build feature vectors rows = myrdd.map(lambda r: r[0].split(‘,’)) out = rows.map(lambda row: (row[0], row)) .groupByKey().map(build_feature_vector) # outputs [(FV, label)] # Bring data down locally and prepare it localout = counts.collect() X = [ row[0] for row in localout ] # feature is set of 40 aggregate properties t = [ row[1] for row in localout ] # potential labels are types of devices # Train a RF classifier on it clf = sklearn.ensemble.RandomForestClassifier(n_estimators=250) clf = clf.fit(X, y) # save the model (maybe to s3 instead?) pickle.dump(clf, open(‘/models/behavior.sklearn’, ‘w’)) --- training data sample of netflow --- SOURCE IP DEST IP DATE STIME ETIME DATAIN DATAOUT 123.41.12.31, 123.41.155.32, 2017-02-01, 09:00, 09:59, 103KB, 959KB 123.41.59.99, 123.41.155.32, 2017-02-01, 09:00, 09:59, 44KB, 884KB 123.41.12.31, 123.41.155.32, 2017-02-01, 10:00, 10:59, 3KB, 9KB 123.41.59.99, 123.41.155.32, 2017-02-01, 10:00, 10:59, 4KB, 15KB
- 24. # http://56.120.177.55/predictip?ip=159.31.120.44 # http://56.120.177.55/predictfv -- for POST of feature vector _MODEL = pickle.load(open(’/models/maintenance.sklearn’)) @app.route('/predictrepair') def predicttypefrombehavior(): netflowlog = request.form[‘logcsv’] fv = build_feature_vetor(netflowlog) pr = _MODEL.predict(fv) return json.dumps({ ‘query’ : fv, ‘prediction’ : pr }) # returns { ‘query’ : [9, 4, 123.1, …], ‘prediction’ : ‘HTTP PROXY’ } --- training data sample of netflow --- SOURCE IP DEST IP DATE STIME ETIME DATAIN DATAOUT 123.41.12.31, 123.41.155.32, 2017-02-01, 09:00, 09:59, 103KB, 959KB 123.41.59.99, 123.41.155.32, 2017-02-01, 09:00, 09:59, 44KB, 884KB 123.41.12.31, 123.41.155.32, 2017-02-01, 10:00, 10:59, 3KB, 9KB 123.41.59.99, 123.41.155.32, 2017-02-01, 10:00, 10:59, 4KB, 15KB
- 25. PYTHON! Viable option for Big Data Analytics with PySpark Tie it all together and integrate into the enterprise with the same language Leverage the benefits of Python for data analysis Get projects done faster