Processing Large Datasets for ADAS Applications using Apache Spark
0 likes576 views
Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis. Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them. Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy. This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.
![5
“…autonomous vehicles would have
to be driven hundreds of millions of
miles and sometimes hundreds of
billions of miles to demonstrate their
reliability in terms of fatalities and
injuries.” [1]
[1] Driving to Safety, Rand Corporation
https://www.rand.org/content/dam/rand/pubs/research_reports/RR1400/RR1478/RAND_RR1478.pdf](https://image.slidesharecdn.com/292arvindhosagrahara-210616155254/85/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-5-320.jpg)
![48
Scaling considerations
Storage
§ Cloud based storage allows
ingestion of larger datasets
Compute
§ Quicker training iterations both
locally and on the cloud
§ Faster inference pipeline
Cost
§ CapEx è OpEx
Workflows
§ Self-serve analytics optimized for
agility along a maturity framework
"Glue code and pipeline jungles are symptomatic of
integration issues that may have a root cause in
overly separated “research” and “engineering”
roles…
… engineers and researchers are embedded together
on the same teams (and indeed, are often the same
people) can help reduce this source of friction
significantly [1].
[1] The Hidden Technical Debt of Machine Learning Systems
https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf](https://image.slidesharecdn.com/292arvindhosagrahara-210616155254/85/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-48-320.jpg)
![54
Conclusions
§ Well-architected systems [1] accelerate development of automated
semantic segmentation against large datasets
§ Local development provides smooth workflows for the development and
refinement of deep learning models
§ Cloud-based scaling of compute and storage can be leveraged on
Databricks to enable self-service analytics
§ Techniques that are relevant for ADAS development are equally applicable
across other domains such as medical, geo-exploration, etc.
[1] AWS Well-Architected
https://aws.amazon.com/architecture/well-architected/](https://image.slidesharecdn.com/292arvindhosagrahara-210616155254/85/Processing-Large-Datasets-for-ADAS-Applications-using-Apache-Spark-54-320.jpg)
More Related Content
What's hot (20)
Similar to Processing Large Datasets for ADAS Applications using Apache Spark (20)
More from Databricks (20)
Recently uploaded (20)
Processing Large Datasets for ADAS Applications using Apache Spark
- 1. 1 © 2020 The MathWorks, Inc. Processing Large Datasets for ADAS applications using Spark Arvind Hosagrahara ([email protected]) Lead Solutions Architect May 2021
- 2. 2 Outline § The Big Picture – Problem Statement / Challenges § Architecture of the system (ETL, Training, Inference) § Deep-Dive into Spark Workflows – Cluster Management – Interactive exploration – Pushdown of analytics to Spark Clusters – Experiment and metadata management § Design considerations – Scaling, Accuracy and Interoperability – Security, Governance, Model Lifecycle management § Conclusions / Key Takeaways § Q&A
- 3. 3 The Big Picture – Automated Driving How can I analyze & simulate scenarios? How can I design & deploy algorithms? How can I integrate & test systems? Control Planning Perception
- 4. 4 Semantic Segmentation What? § Semantic segmentation is the classification of every pixel in an image/video Why? § Semantic segmentation: Higher quality perception outcomes, easier to analyze and validation of localization modules for AD. § Automation: Faster design iterations
- 5. 5 “…autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their reliability in terms of fatalities and injuries.” [1] [1] Driving to Safety, Rand Corporation https://www.rand.org/content/dam/rand/pubs/research_reports/RR1400/RR1478/RAND_RR1478.pdf
- 6. 6 Problem Statement How can we: 1. Accelerate development of semantic segmentation models from prototype to production in order to build better perception software 2. Scale the workflows, capabilities and throughput of engineering processes by leveraging automation, cloud storage, compute and other infrastructure
- 7. 7 Challenges Business Challenges § Prototype to production: establish system & software engineering processes § Verification and validation: connect data, embedded software and simulation § Workforce development: hire & integrate engineers with disparate background Technical Challenges § Handling large datasets and specialized formats § Leveraging the cloud to scale computational requirements § Enable self-serve analytics for the data scientist / engineer
- 8. 8 Demonstration / Results (Image-level features with batch normalization)
- 9. 9
- 10. 10 Demo / Results 3-D simulation data to train a semantic segmentation network and fine-tune it to real-world data using generative adversarial networks (GANs)
- 11. 11
- 13. 13 Architecture of the demo system Burst Data access Experiment Management / Governance / Security
- 14. 14 Burst Data access Experiment Management / Governance / Security Data access Experiment Management / Governance / Security Development Environment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest
- 15. 15 Data access Experiment Management / Governance / Security Development Environment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest
- 16. 16 A simplified view of the system architecture Experiment Management / Governance / Security Development Environment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest Data access
- 17. 17 The ETL (Extract Transform Load) Pipeline § The ETL (Extract Transform Load) Pipeline Experiment Management / Governance / Security Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Recorded and Simulated Data (Batch and Streaming Ingest) Ingest § §
- 18. 18 The Training Pipeline § Experiment Management / Governance / Security Development Environment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Cloud-based training system (Spark Clusters / GPU support / Fast networks) Burst Ingest Data access § The Training Pipeline §
- 19. 19 The Inference Pipeline § Experiment Management / Governance / Security Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Burst Ingest Data access § § The Inference Pipeline
- 20. 20 The Software Stack Experiment Management / Governance / Security Development Environment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest Data access
- 21. 21 The Software Stack Experiment Management / Governance / Security Development Environment (Desktop / Cloud) Storage (Large Datasets / Models / Metadata) Model Inference Engine for ETL and Performance Monitoring Monitoring / Visualization Front-ends Cloud-based training system (Spark Clusters / GPU support / Fast networks) Recorded and Simulated Data (Batch and Streaming Ingest) Burst Ingest Data access MATLAB® and Simulink® MATLAB Production Server™ MATLAB Webapp Server™ 3P Dashboards Amazon S3™ AWS Identity and Access Management (IAM) AWS™ MLflow™
- 23. 23 Workflows supporting DevOps Development Environment Operational Environment TEST O P E R A T E DEPLOY MONITOR DESIGN B U I L D P L A N F E E D B A C K Cluster Management JDBC based connectivity to data sources Databricks- connect and interactive exploration CI/CD integration Jobs and Spark-Submit Model Lifecycle Management Experiment Management Coder based acceleration Embedded Applications Inference engine Central result, config, data management
- 25. 25 Workflow: Cluster management and Data Engineering via REST Engineers Developers Scientists Data Analyst Compute Cluster (Cloud/On-Prem) jobsubmission.m MATLAB HTTP Client MATLAB Integration (REST 2.0 API) • Create Clusters • CRUD with DBFS • Create Jobs • Authentication /Tokens • Secrets • Notifications • Libraries • Workspace Databricks specific runtime image MATLAB Runtime init_script Dockerfile dockerized runtime analytics.jar
- 26. 26 Video 1 Outline § Planning capacity and performance § Cluster creation § Data and application transfer via the DBFS API
- 27. 27 Workflow: JDBC and Database toolbox (a) Engineers Developers Scientists Data Analyst Existing Compute Cluster (Cloud/On-Prem) DatabaseToolbox ODBC 2.6.x JDBC 2.6.x Driver SQL
- 28. 28 Workflow: JDBC and Database toolbox (b) Engineers Developers Scientists Data Analyst Existing Compute Cluster (Cloud/On-Prem) ODBC 2.6.x JDBC 2.6.x Driver SparkSession SQL Databases / Delta Tables
- 29. 29 Workflow: Databricks-connect Engineers Developers Scientists Data Analyst Existing Compute Cluster (Cloud/On-Prem) Databricks-connect Spark DAG Spark Master Spark API
- 30. 30 Video 2 Outline § Databricks-Connect workflows § Read/Write delta tables
- 32. 32 The Training Pipeline (Local Experimentation) § Optimized for GPU § High-level API (“write less and do more”) § Language abstractions to improve accuracy (eg: imageDataAugmenter) § Workflows for ease-of-use and debug – apps for model selection – monitor accuracy, loss and training performance – many quick-start examples
- 33. 33 Workflow: Push-down via MATLAB Compiler Engineers Developers Scientists Data Analyst Compute Cluster (Cloud/On-Prem) spark-submit job for New Clusters analytics.jar Spark DAG Spark Master analytics.m MATLAB Compiler/SDK MATLAB API for Spark Tall Arrays Libraries for Existing Clusters
- 34. 34 Video 3 Outline § SparkSession and RDD abstractions § Tall-based workflows § Deploytool and app based workflows – Compiler SDK for Java § mcc – Spark Submit § DBFS API to push application to Databricks
- 35. 35 Build and Test § Improve quality of code and models by testing on CI Servers (self-hosted or Cloud-hosted) § Package libraries for deployment § Adhere to agile development practices https://www.mathworks.com/solutions/continuous-integration.html
- 36. 36 The Training Pipeline (Cloud-based training) § Infrastructure defined as code § Leverage the latest hardware, optimized drivers and toolkits § APIs for command and control of Jobs, artifacts, notifications, execution, etc. § Capital Expenses (CapEx) è Operating Expenses (OpEx)
- 37. 37 Management of Experiments and tracking of performance
- 38. 38 Video 4 Outline § MLFlow Demo – Creation of Experiments / Runs – Logging of Parameters and Metrics – Integration with Experiment Management features
- 39. 39 Training Benchmarks and Performance VGG-16 based network on NVIDIA Quadro GV100
- 41. 41 The Inference Pipeline § Directed Acyclic Graphs (DAG) used to resolve changes to: – Training models, ETL changes – The training data – New data / changes to the cache § Trained Models can be deployed as cloud services § Cloud Reference Architectures available on github.com to integrate with existing systems
- 42. 42 Scaling Inference on the MATLAB Production Server § Scales gracefully § Optimal usage of available CPU and GPU resources § Can increase throughput as desired § PAYG models available on the cloud
- 43. 43 The ETL (Extract-Transform-Load) Pipeline
- 44. 44 The ETL Pipeline § Wide variety of Automotive and Engineering Formats (eg: ROSBag, MP4, MDF, BLF, PCAP) § RESTful endpoints or Client based access § Connectivity to Cloud Storage (S3) and distributed streaming systems (Kafka) § Horizontally scalable and designed to run 24x7x365 on headless cloud systems § Provides data proximity when executing on the cloud § Accelerated CUDA execution on NVIDIA® via GPU Coder™
- 45. 45 Putting it all together
- 46. 46 Build Graph (DAG) § Engine code § Trained models § New/Cached Data § Operates on S3 version metadata § Can be triggered manually or automatically (eg: via CI/CD pipelines)
- 48. 48 Scaling considerations Storage § Cloud based storage allows ingestion of larger datasets Compute § Quicker training iterations both locally and on the cloud § Faster inference pipeline Cost § CapEx è OpEx Workflows § Self-serve analytics optimized for agility along a maturity framework "Glue code and pipeline jungles are symptomatic of integration issues that may have a root cause in overly separated “research” and “engineering” roles… … engineers and researchers are embedded together on the same teams (and indeed, are often the same people) can help reduce this source of friction significantly [1]. [1] The Hidden Technical Debt of Machine Learning Systems https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
- 49. 49 Accuracy considerations § More Training Data § Simulation / Scene generation – Roadrunner Simulation
- 50. 50 Interop considerations § ONNX interop § Transfer learning techniques § Import/Export to – OpenDRIVE®, – FBX®, – glTF™, – OpenFlight, – OpenSceneGraph, – OBJ, and USD formats. § Automated driving simulators and game engines (eg: CARLA, VIRES VTD, NVIDIA DRIVE Sim®, Metamoto®, LGSVL, Baidu Apollo®, Unity®, and Unreal Engine®)
- 51. 51 Embedded Targeting § GPU Coder™ Support Package for NVIDIA GPUs § NVIDIA® DRIVE™ and Jetson hardware platforms
- 52. 52 Governance and Lifecycle management § Local Experimentation § Centralized Tracking
- 53. 53 Security Considerations § Cloud Storage Best Practices § ACL based permissions § IAM based identity and access
- 54. 54 Conclusions § Well-architected systems [1] accelerate development of automated semantic segmentation against large datasets § Local development provides smooth workflows for the development and refinement of deep learning models § Cloud-based scaling of compute and storage can be leveraged on Databricks to enable self-service analytics § Techniques that are relevant for ADAS development are equally applicable across other domains such as medical, geo-exploration, etc. [1] AWS Well-Architected https://aws.amazon.com/architecture/well-architected/
- 55. 55 Key Takeaways § Build upon proven, safety compliant, and DevOps-ready development tools § Leveraging best-in-class simulation integration platform and modeling tools leads to faster time-to-market outcomes for ADAS development § Domain specific tooling allows practitioners to write less and do more § If this is interesting, please contact us at [email protected] for more details or to get started.
- 57. 58