The evolution of array computing in Python

May 13, 20190 likes316 views

My PyData Amsterdam 2019 presentation. Have you ever wanted to run your NumPy based code on multiple cores, or on a distributed system, or on your GPU? Wouldn't it be nice to do this without changing your code? We will discuss how NumPy's array protocols work, and provide a practical guide on how to start using them. We will also discuss how array libraries in Python may evolve over the next few years.

The evolution of array
computing in Python
Ralf Gommers 
PyData Amsterdam 2019

whoami
Maintainer of NumPy & SciPy (2010 -- )
NumFOCUS board member (2012 -- 2018)
Director of Quansight Labs (2019 -- )
You can find me at:
https://github.com/rgommers
rgommers@quansight.com
2
Public benefit division of Quansight,
providing a home for a “PyData Core
Team” and growing the community

A very brief history of array computing in Python
3
Numeric
Numarray
1995
2003
2006
2008
2012
2015 -- today
Sparse
… ?

Do try this at home
6
$ conda create -n pydata-ams
$ conda activate pydata-ams
$ conda install numpy dask jupyterlab
$ # If you have a NVIDIA GPU. Needs CUDA installed.
$ # CuPy 6.0.0 will be out soon, then conda-installable.
$ pip install --pre cupy-cuda100 # or cupy-90 for CUDA 9
$ python -c "import numpy; print(numpy.__version__)"
1.16.3
$ python -c "import cupy; print(cupy.__version__)"
6.0.0rc1
$ python -c "import dask; print(dask.__version__)"
1.2.0
$ export NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1

The NumPy array protocols - goals
7
Separate NumPy API from NumPy “execution engine”
Allow other libraries (Dask, CuPy, PyTorch, …) to reuse
the NumPy API
Bigger picture: avoid or reduce ecosystem
fragmentation (we don’t want to see a reimplementation of SciPy for
PyTorch, SciPy for Tensorflow, etc.)

The NumPy array protocols - goals
Current state of N-dimensional arrays in Python
8

The NumPy array protocols - goals
What we’re aiming for
9

The NumPy array protocols - concept
10
Function body 
(the “implementation”, defines
semantics)
Function signature (the API)
Example for one function - this can be a ufunc or a regular function.

The NumPy array protocols - concept
11
Function body
Function signature (the API)
In short: use the NumPy API, bring your own implementation
Function body
Function signature
Function signature (the API)
if input arg has __array_function__:
execute other_function
Function body
Function signature
==

Using array protocols in your own code
Suitable for code that uses NumPy functions and ndarray
methods.
Try CuPy if you need more performance on large arrays,
and Dask if you want a distributed array.
Let’s play with this in a notebook!
12

Limits to these array protocols
13
Only functions can be overridden. And not even all functions -- only the
ones with an array_like parameter. Important exceptions:
np.array, np.asarray, np.linspace, np.concatenate

NumPy’s roadmap - what’s next?
Interoperability
Roll out __array_function__,
handle subclasses better,
new protocols?
Extensibility
Easier custom dtypes
Performance
Ufunc optimizations, 
more SIMD instructions,
...
np.random rewrite
Is about to be merged
Indexing
NEP 21 (oindex/vindex), for
more intuitive behavior.
Type annotations
PEP 484 / mypy compatible
annotations, see numpy-
stubs repo.
14

The array computing landscape
15
maturity
n-D arrays (a.k.a. tensors)
dataframes
CPU
GPU
NumPy
XND
CuPy
PyTorch
xtensor
CPU
GPU
Pandas
MXNet
DyND
cuDF
Apache Arrow
Tensorflow
Uarray
TensorLy
xframe
Xarray

XND
Recreates the foundations of NumPy as a number of
smaller libraries. Plus:
Variable length strings
Ragged arrays
Categorical type
Missing data support
Easy custom dtypes
Automatic multi-threading
JIT compilation (via Numba)
16

XND
17
Variable length strings:
Ragged arrays:

xtensor
A C++ library for n-D arrays. Plus:
Lazy evaluation
Performance - very fast
Can operate on NumPy arrays
Python, Julia and R bindings
JIT compilation (via Pythran)
Built on top: rray (NumPy-like arrays for R)
18

Uarray
A more general solution than the NumPy array protocols
for building APIs with multiple backends.
Override any object: functions, classes, ufuncs, dtypes,
context managers, and more
Uses multiple dispatch rather than protocols.
20

Talk given at GTC Fall 2021. The Python array API standard, which was first announced towards the end of 2020, is maturing and becoming available to Python end users. NumPy now has a reference implementation, PyTorch support is close to complete, and other libraries have started to implement support. In this talk we will discuss the current state of implementations, and look at a concrete use case of moving a scientific analysis workflow to using the API standard - thereby gaining access to GPU acceleration.

NumPy Roadmap presentation at NumFOCUS ForumRalf Gommers

Standardizing on a single N-dimensional array API for PythonRalf Gommers

PyData NYC whatsnew NumPy-SciPy 2019Ralf Gommers

This document discusses recent updates to NumPy and SciPy. Key updates include a complete overhaul of NumPy's random number generators and Fourier transform implementations. NumPy's __array_function__ protocol is now enabled by default, allowing other libraries to reuse the NumPy API. The NumPy array protocols were developed to separate the NumPy API from its execution engine. This avoids ecosystem fragmentation and allows the NumPy API to work with GPUs and distributed arrays via libraries like Dask. SciPy's FFT functions were reimplemented for increased speed and accuracy, and a new scipy.fft submodule was added, representing the first new SciPy submodule in a decade. Additional new global optimizers were also added to SciPy.

Scipy, numpy and friendsMichele Mattioni

The document introduces Scipy, Numpy and related tools for scientific computing in Python. It provides links to documentation and tutorials for Scipy and Numpy for numerical operations, Matplotlib for data visualization, and IPython for an interactive coding environment. It also includes short examples and explanations of Numpy arrays, plotting, data analysis workflows, and accessing help documentation.

Tom Peters, Software Engineer, Ufora at MLconf ATL 2016MLconf

Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way! In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.

Avi Pfeffer, Principal Scientist, Charles River Analytics at MLconf SEA - 5/2...MLconf

Practical Probabilistic Programming with Figaro: Probabilistic reasoning enables you to predict the future, infer the past, and learn from experience. Probabilistic programming enables users to build and reason with a wide variety of probabilistic models without machine learning expertise. In this talk, I will present Figaro, a mature probabilistic programming system with many applications. I will describe the main design principles of the language and show example applications. I will also discuss our current efforts to fully automate and optimize the inference process.

Get Your Hands Dirty with Intel® Distribution for Python*Intel® Software

This session reviews using Intel® Distribution for Python* and Intel® Data Analytics Acceleration Library (Intel® DAAL) to accelerate data analytics and machine learning algorithms. Build fast machine learning applications in Python with tools from Intel. Get an introduction to the Python API in Intel DAAL. Intel Distribution for Python is used as the foundation of all the session tasks. Implement some common algorithms, such as K-means, linear regression, multiclass support vector machine (SVM), and neural networks using Intel DAAL. This session uses real-world data collections available online, such as those from University of California, Irvine, Machine Learning Repository.

Scaling Python to CPUs and GPUsTravis Oliphant

NumbaTravis Oliphant

Hussein Mehanna, Engineering Director, ML Core - Facebook at MLconf ATL 2016MLconf

Applying Deep Learning at Facebook Scale: Facebook leverages Deep Learning for various applications including event prediction, machine translation, natural language understanding and computer vision at a very large scale. There are more than a billion users logging on to Facebook every daily generating thousands of posts per second and uploading more than a billion images and videos every day. This talk will explain how Facebook scaled Deep Learning inference for realtime applications with latency budgets in the milliseconds.

Array computing and the evolution of SciPy, NumPy, and PyDataTravis Oliphant

Python librariesVenkat Projects

Keynote at Converge 2019Travis Oliphant

PyCon Estonia 2019Travis Oliphant

Chris Fregly, Research Scientist, PipelineIO at MLconf ATL 2016MLconf

Standardizing arrays -- Microsoft PresentationTravis Oliphant

This document discusses standardizing N-dimensional arrays (tensors) in Python. It proposes creating a "uarray" interface that downstream libraries could use to work with different array implementations in a common way. This would include defining core concepts like shape, data type, and math operations for arrays. It also discusses collaborating with mathematicians on formalizing array operations and learning from NumPy's generalized ufunc approach. The goal is to enhance Python's array ecosystem and allow libraries to work across hardware backends through a shared interface rather than depending on a single implementation.

SciPy 2019: How to Accelerate an Existing Codebase with Numbastan_seibert

The document discusses a four step process for accelerating existing code with Numba: 1) Make an honest self-inventory of why speeding up code is needed, 2) Perform measurement of code through unit testing and profiling, 3) Refactor code following rules like paying attention to data types and writing code like Fortran, 4) Share accelerated code with others by packaging with Numba as a dependency. Key rules discussed include always using @jit(nopython=True), paying attention to supported data types, writing functions over classes, and targeting serial execution first before parallelism.

Rajat Monga, Engineering Director, TensorFlow, Google at MLconf 2016MLconf

This document provides an overview of TensorFlow, an open source machine learning framework. It discusses how machine learning systems can become complex with modeling complexity, heterogeneous systems, and distributed systems. It then summarizes key aspects of TensorFlow, including its architecture, platforms, languages, parallelism approaches, algorithms, and tooling. The document emphasizes that TensorFlow handles complexity so users can focus on their machine learning ideas.

Scientific Computing with Python Webinar --- August 28, 2009Enthought, Inc.

Pycon tw 2013show you

This document summarizes a report about PyCon Taiwan 2013 from the perspective of a data mining engineer. Key points include: maintaining Hadoop and BI tools; the conference had around 500 people mainly from academia; talks covered Python 3 differences, algorithms, data collection, and machine learning algorithms like boosting in scikit-learn. Python has many libraries useful for data scientists like NumPy, Pandas, Scikit-learn, NLTK, and Matplotlib. The conference showed the growing use of Python for statistical analysis and machine learning.

Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16MLconf

Smarter Search With Spark-Solr: Search gets smarter when you know more about your documents and their relationship to each other (think: PageRank) and the users (i.e. popularity), in addition to what you already know about their content (text search). It also gets smarter when you know more about your users (personalization) and both their affinity for certain kinds of content and their similarities to each other (collaborative filtering recommenders). Building all of these pieces typically requires a big mix of batch workloads to do log processing, as well as training machine-learned models to use during realtime querying, and are highly domain specific, but many techniques are fairly universal: we will discuss how Spark can interface with a Solr Cloud cluster to efficiently perform many of the pieces to this puzzle in one relatively self-contained package (no HDFS/S3, all data stored in Solr!), and introduce “spark-solr” – an open-source JVM library to facilitate this.

A Map of the PyData StackPeadar Coyle

Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...MLconf

Scripts that Scale with F# and mbrace.io: Nothing beats interactive scripting for productive data exploration and rapid prototyping: grab data, run code, and iterate based on feedback. However, that story starts to break down once you need to process large datasets or expensive computations. Your local machine becomes the bottleneck, and your are left with a slow and unresponsive environment. In this talk, we will demonstrate on live examples how you can have your cake and eat it, too, using mbrace.io, a free, open-source engine for scalable cloud programming. Using a simple programming model, you can keep working from your favorite scripting environment, and execute code interactively against a cluster on the Azure cloud. We will discuss the relevance of F# and mbrace in a data science and machine learning context, from parallelizing code and data processing in a functional style, to leveraging F# type providers to consume data or even run R packages.

PyTorch vs TensorFlow: The Force Is Strong With Which One? | Which One You Sh...Edureka!

( ** Deep Learning Training: https://www.edureka.co/ai-deep-learning-with-tensorflow ** ) This Edureka comparison PPT of "PyTorch vs TensorFlow" provides you with a detailed comparison between the top 2 Python Deep Learning Frameworks. Follow us to never miss an update in the future. Instagram: https://www.instagram.com/edureka_learning/ Facebook: https://www.facebook.com/edurekaIN/ Twitter: https://twitter.com/edurekain LinkedIn: https://www.linkedin.com/company/edureka

Erin LeDell, Machine Learning Scientist, H2O.ai at MLconf ATL 2016MLconf

Multi-algorithm Ensemble Learning at Scale: Software, Hardware and Algorithmic Approaches: Multi-algorithm ensemble machine learning methods are often used when the true prediction function is not easily approximated by a single algorithm. The Super Learner algorithm, also known as stacking, combines multiple, typically diverse, base learning algorithms into a single, powerful prediction function through a secondary learning process called metalearning. Although ensemble methods offer superior performance over their singleton counterparts, there is an implicit computational cost to ensembles, as it requires training and cross-validating multiple base learning algorithms. We will demonstrate a variety of software- and hardware-based approaches that lead to more scalable ensemble learning software, including a highly scalable implementation of stacking called “H2O Ensemble”, built on top of the open source, distributed machine learning platform, H2O. H2O Ensemble scales across multi-node clusters and allows the user to create ensembles of deep neural networks, Gradient Boosting Machines, Random Forest, and others. As for algorithm-based approaches, we will present two algorithmic modifications to the original stacking algorithm that further reduce computation time — Subsemble algorithm and the Online Super Learner algorithm. This talk will also include benchmarks of the implementations of these new stacking variants.

Introduction to Spark: Or how I learned to love 'big data' after all.Peadar Coyle

Five python libraries should know for machine learningNaveen Davis

Euro python2011 High Performance PythonIan Ozsvald

I ran this as a 4 hour tutorial at EuroPython 2011 to teach High Performance Python coding. Techniques covered include bottleneck analysis by profiling, bytecode analysis, converting to C using Cython and ShedSkin, use of the numerical numpy library and numexpr, multi-core and multi-machine parallelisation and using CUDA GPUs. Write-up with 49 page PDF report: http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/

carrow - Go bindings to Apache Arrow via C++-APIYoni Davidson

Apache Arrow is a cross-language development platform for in-memory data that specifies a standardized columnar memory format. It provides libraries and messaging for moving data between languages and services without serialization. The presenter discusses their motivation for creating Go bindings for Apache Arrow via C++ to share data between Go and Python programs using the same memory format. They explain several challenges of this approach, such as different memory managers in Go and C++, and solutions like generating wrapper code and handling memory with finalizers.

More Related Content

What's hot (20)

Scaling Python to CPUs and GPUsTravis Oliphant

NumbaTravis Oliphant

Hussein Mehanna, Engineering Director, ML Core - Facebook at MLconf ATL 2016MLconf

Array computing and the evolution of SciPy, NumPy, and PyDataTravis Oliphant

Python librariesVenkat Projects

Keynote at Converge 2019Travis Oliphant

PyCon Estonia 2019Travis Oliphant

Chris Fregly, Research Scientist, PipelineIO at MLconf ATL 2016MLconf

Standardizing arrays -- Microsoft PresentationTravis Oliphant

SciPy 2019: How to Accelerate an Existing Codebase with Numbastan_seibert

Rajat Monga, Engineering Director, TensorFlow, Google at MLconf 2016MLconf

Scientific Computing with Python Webinar --- August 28, 2009Enthought, Inc.

Pycon tw 2013show you

Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16MLconf

A Map of the PyData StackPeadar Coyle

Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...MLconf

PyTorch vs TensorFlow: The Force Is Strong With Which One? | Which One You Sh...Edureka!

Erin LeDell, Machine Learning Scientist, H2O.ai at MLconf ATL 2016MLconf

Introduction to Spark: Or how I learned to love 'big data' after all.Peadar Coyle

Five python libraries should know for machine learningNaveen Davis

Scaling Python to CPUs and GPUsTravis Oliphant

NumbaTravis Oliphant

Hussein Mehanna, Engineering Director, ML Core - Facebook at MLconf ATL 2016MLconf

Array computing and the evolution of SciPy, NumPy, and PyDataTravis Oliphant

Python librariesVenkat Projects

Keynote at Converge 2019Travis Oliphant

PyCon Estonia 2019Travis Oliphant

Chris Fregly, Research Scientist, PipelineIO at MLconf ATL 2016MLconf

Standardizing arrays -- Microsoft PresentationTravis Oliphant

SciPy 2019: How to Accelerate an Existing Codebase with Numbastan_seibert

Rajat Monga, Engineering Director, TensorFlow, Google at MLconf 2016MLconf

Scientific Computing with Python Webinar --- August 28, 2009Enthought, Inc.

Pycon tw 2013show you

Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16MLconf

A Map of the PyData StackPeadar Coyle

Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...MLconf

PyTorch vs TensorFlow: The Force Is Strong With Which One? | Which One You Sh...Edureka!

Erin LeDell, Machine Learning Scientist, H2O.ai at MLconf ATL 2016MLconf

Introduction to Spark: Or how I learned to love 'big data' after all.Peadar Coyle

Five python libraries should know for machine learningNaveen Davis

Similar to The evolution of array computing in Python (20)

Euro python2011 High Performance PythonIan Ozsvald

carrow - Go bindings to Apache Arrow via C++-APIYoni Davidson

Distributed Multi-GPU Computing with Dask, CuPy and RAPIDSPeterAndreasEntschev

The search for faster computing remains of great importance to the software community. Relatively inexpensive modern hardware, such as GPUs, allows users to run highly parallel code on thousands, or even millions of cores on distributed systems. Building efficient GPU software is not a trivial task, often requiring a significant amount of engineering hours to attain the best performance. Similarly, distributed computing systems are inherently complex. In recent years, several libraries were developed to solve such problems. However, they often target a single aspect of computing, such as GPU computing with libraries like CuPy, or distributed computing with Dask. Libraries like Dask and CuPy tend to provide great performance while abstracting away the complexity from non-experts, being great candidates for developers writing software for various different applications. Unfortunately, they are often difficult to be combined, at least efficiently. With the recent introduction of NumPy community standards and protocols, it has become much easier to integrate any libraries that share the already well-known NumPy API. Such changes allow libraries like Dask, known for its easy-to-use parallelization and distributed computing capabilities, to defer some of that work to other libraries such as CuPy, providing users the benefits from both distributed and GPU computing with little to no change in their existing software built using the NumPy API.

__array_function__ conceptual design & related conceptsRalf Gommers

Accelerating Apache Spark by Several Orders of Magnitude with GPUs and RAPIDS...Databricks

GPU acceleration has been at the heart of scientific computing and artificial intelligence for many years now. GPUs provide the computational power needed for the most demanding applications such as Deep Neural Networks, nuclear or weather simulation. Since the launch of RAPIDS in mid-2018, this vast computational resource has become available for Data Science workloads too. The RAPIDS toolkit, which is now available on the Databricks Unified Analytics Platform, is a GPU-accelerated drop-in replacement for utilities such as Pandas/NumPy/ScikitLearn/XGboost. Through its use of Dask wrappers the platform allows for true, large scale computation with minimal, if any, code changes. The goal of this talk is to discuss RAPIDS, its functionality, architecture as well as the way it integrates with Spark providing on many occasions several orders of magnitude acceleration versus its CPU-only counterparts.

TensorFlow meetup: Keras - Pytorch - TensorFlow.jsStijn Decubber

Hadoop 31-frequently-asked-interview-questionsAsad Masood Qazi

The document contains 31 questions and answers related to Hadoop concepts. It covers topics like common input formats in Hadoop, differences between TextInputFormat and KeyValueInputFormat, what are InputSplits and how they are created, how partitioning, shuffling and sorting occurs after the map phase, what is a combiner, functions of JobTracker and TaskTracker, how speculative execution works, using distributed cache and counters, setting number of mappers/reducers, writing custom partitioners, debugging Hadoop jobs, and failure handling processes for production Hadoop jobs.

Scale up and Scale Out Anaconda and PyDataTravis Oliphant

Getting started with Hadoop, Hive, and Elastic MapReduceobdit

1) The document provides an overview of tools for distributed computing including MapReduce, Hadoop, Hive, and Elastic MapReduce. 2) It discusses getting started with Elastic MapReduce using Python with mrjob or the AWS command line and challenges with getting started with Hive. 3) Potential pitfalls with EMR are also outlined such as JVM memory issues and problems with multiple small output files.

Querying Network Packet Captures with Spark and DrillVince Gonzalez

This document discusses using Apache Spark and Apache Drill to query network packet captures stored in Apache Hadoop. It describes capturing packets using tcpdump, preprocessing the packet data with Spark Streaming, indexing the data with Elasticsearch for querying, and demo queries on the packet data using SQL with Apache Drill. The goals were to store large packet captures in a scalable system and enable searching and basic analysis of the packets in near real-time. Challenges addressed include atomic file writing with tcpdump, processing PCAP files in Spark Streaming, and setting up indexing and querying of the packet data.

A Java Implementer's Guide to Better Apache Spark PerformanceTim Ellison

This document discusses techniques for improving the performance of Apache Spark applications. It describes optimizing the Java virtual machine by enhancing the just-in-time compiler, improving the object serializer, enabling faster I/O using technologies like RDMA networking and CAPI flash storage, and offloading tasks to graphics processors. The document provides examples of code style guidelines and specific Spark optimizations that further improve performance, such as leveraging hardware accelerators and tuning JVM heuristics.

Strata Beijing 2017: Jumpy, a python interface for nd4jAdam Gibson

GPUs should complement, not replace, the Hadoop ecosystem for big data workloads. Replacing the entire big data stack would be too costly. The presenter believes GPUs are best suited for accelerated computation and a few other use cases to gain an initial foothold in the market. Existing Python interfaces to machine learning frameworks rely too heavily on network communication and serialization, introducing significant overhead. Nd4j and Jumpy provide alternatives that use direct C++ interfaces and pointers for lower latency between Python and deep learning operations on CPU and GPU.

PyData Boston 2013Travis Oliphant

Making Python 100x Faster with Less Than 100 Lines of RustScyllaDB

Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks

With the rapid evolution of AI in recent years, we need to embrace advanced and emerging AI technologies to gain insights and make decisions based on massive amounts of data. Ray (https://github.com/ray-project/ray) is a fast and simple framework open-sourced by UC Berkeley RISELab particularly designed for easily building advanced AI applications in a distributed fashion.

Apache Spark Introduction @ University College LondonVitthal Gogate

Intro to Apache Spark by Marco VasquezMapR Technologies

Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonChristian Perone

This document provides an introduction to Apache Spark and collaborative filtering. It discusses big data and the limitations of MapReduce, then introduces Apache Spark including Resilient Distributed Datasets (RDDs), transformations, actions, and DataFrames. It also covers Spark Machine Learning (ML) libraries and algorithms such as classification, regression, clustering, and collaborative filtering.

NYAI - Scaling Machine Learning Applications by Braxton McKeeRizwan Habib

Scaling Machine Learning Systems - (Braxton McKee, CEO & Founder, Ufora) Braxton is the technical lead and founder of Ufora, a software company that develops Pyfora, an automatically parallel implementation of the Python programming language that enables data science and machine-learning at scale. Before founding Ufora with backing from Two Sigma Ventures and others, Braxton led the ten-person MBS/ABS Credit Modeling team at Ellington Management Group, a multi-billion dollar mortgage hedge fund. He holds a BS (Mathematics), MS (Mathematics), and M.B.A. from Yale University. Braxton will discuss scaling machine learning applications using the open-source platform Pyfora. He will describe both the general approach and also some specific engineering techniques employed in the implementation of Pyfora that make it possible to produce large-scale machine learning and data science programs directly from single-threaded Python code.

Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16MLconf

Euro python2011 High Performance PythonIan Ozsvald

carrow - Go bindings to Apache Arrow via C++-APIYoni Davidson

Distributed Multi-GPU Computing with Dask, CuPy and RAPIDSPeterAndreasEntschev

__array_function__ conceptual design & related conceptsRalf Gommers

Accelerating Apache Spark by Several Orders of Magnitude with GPUs and RAPIDS...Databricks

TensorFlow meetup: Keras - Pytorch - TensorFlow.jsStijn Decubber

Hadoop 31-frequently-asked-interview-questionsAsad Masood Qazi

Scale up and Scale Out Anaconda and PyDataTravis Oliphant

Getting started with Hadoop, Hive, and Elastic MapReduceobdit

Querying Network Packet Captures with Spark and DrillVince Gonzalez

A Java Implementer's Guide to Better Apache Spark PerformanceTim Ellison

Strata Beijing 2017: Jumpy, a python interface for nd4jAdam Gibson

PyData Boston 2013Travis Oliphant

Making Python 100x Faster with Less Than 100 Lines of RustScyllaDB

Running Emerging AI Applications on Big Data Platforms with Ray On Apache SparkDatabricks

Apache Spark Introduction @ University College LondonVitthal Gogate

Intro to Apache Spark by Marco VasquezMapR Technologies

Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonChristian Perone

NYAI - Scaling Machine Learning Applications by Braxton McKeeRizwan Habib

Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16MLconf

More from Ralf Gommers (8)

Reliable from-source builds (Qshare 28 Nov 2023).pdfRalf Gommers

Parallelism in a NumPy-based programRalf Gommers

The document discusses optimizing parallelism in NumPy-based programs. It provides examples of optimizing a main function from 50.1 ms to 2.83 ms using profiling and optimization. It discusses approaches for performant numerical code including vectorization and Python compilers. It also covers issues with oversubscription when using all CPU cores and parallel APIs in NumPy, SciPy, and scikit-learn. The document provides recommendations for tuning default parallel behavior and controlling parallelism in packages.

The road ahead for scientific computing with PythonRalf Gommers

1) The PyData ecosystem, including NumPy, SciPy, and scikit-learn, faces technical challenges related to fragmentation of array libraries, lack of parallelism, packaging constraints, and performance issues for non-vectorized algorithms. 2) There are also social challenges around sustainability of key projects due to limited funding and maintainers, tensions with proprietary involvement from large tech companies, and academia's role in supporting open-source scientific software. 3) NumPy is working to address these issues through efforts like the Array API standardization, improved extensibility and performance, and growing autonomous teams and diversity within the community.

Building SciPy kernels with PythranRalf Gommers

Pythran is a tool that can be used to accelerate SciPy kernels by transpiling pure Python and NumPy code into efficient C++. SciPy developers have started using Pythran for some computationally intensive kernels, finding it easier to write fast code with than alternatives like Cython or Numba. Initial integration into the SciPy build process has gone smoothly. Ongoing work includes porting more kernels to Pythran and exploring combining it with CuPy for fast CPU and GPU code generation.

Strengthening NumPy's foundations - growing beyond codeRalf Gommers

Inside NumPy: preparing for the next decadeRalf Gommers

Talk given at SciPy'19. Abstract: Over the past year, and for the first time since its creation, NumPy has been operating with dedicated funding. NumPy developers think it has invigorated the project and its community. But is that true, and how can we know? We will give an overview of the actions we've taken to improve the sustainability of the NumPy project and its community. We will draw some lessons from a first year of grant-funded activity, discuss key obstacles faced, attempt to quantify what we need to operate sustainably, and present a vision for the project and how we plan to realize it. Authors: Ralf Gommers, Matti Picus, Tyler Reddy, Stefan van der Walt and Charles Harris

NumFOCUS_Summit2018_Roadmaps_sessionRalf Gommers

SciPy 1.0 and Beyond - a Story of Community and CodeRalf Gommers

Reliable from-source builds (Qshare 28 Nov 2023).pdfRalf Gommers

Parallelism in a NumPy-based programRalf Gommers

The road ahead for scientific computing with PythonRalf Gommers

Building SciPy kernels with PythranRalf Gommers

Strengthening NumPy's foundations - growing beyond codeRalf Gommers

Inside NumPy: preparing for the next decadeRalf Gommers

NumFOCUS_Summit2018_Roadmaps_sessionRalf Gommers

SciPy 1.0 and Beyond - a Story of Community and CodeRalf Gommers

Recently uploaded (20)

Micro-Metrics Every Performance Engineer Should Validate Before Sign-OffTier1 app

When it comes to performance testing, most engineers instinctively gravitate toward the big-picture indicators—response time, memory usage, throughput. But what about the smaller, more subtle indicators that quietly shape your application’s performance and stability? we explored the hidden layer of performance diagnostics that too often gets overlooked: micro-metrics. These small but mighty data points can reveal early signs of trouble long before they manifest as outages or degradation in production. From garbage collection behavior and object creation rates to thread state transitions and blocked thread patterns, we unpacked the critical micro-metrics every performance engineer should assess before giving the green light to any release. This session went beyond the basics, offering hands-on demonstrations and JVM-level diagnostics that help identify performance blind spots traditional tests tend to miss. We showed how early detection of these subtle anomalies can drastically reduce post-deployment issues and production firefighting. Whether you're a performance testing veteran or new to JVM tuning, this session helped shift your validation strategies left—empowering you to detect and resolve risks earlier in the lifecycle.

Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentricNatan Silnitsky

At Wix, we revolutionized our platform by making integration events the backbone of our 4,000-microservice ecosystem. By abandoning traditional domain events for standardized Protobuf events through Kafka, we created a universal language powering our entire architecture. We'll share how our "single-aggregate services" approach—where every CUD operation triggers semantic events—transformed scalability and extensibility, driving efficient event choreography, data lake ingestion, and search indexing. We'll address our challenges: balancing consistency with modularity, managing event overhead, and solving consumer lag issues. Learn how event-based data prefetches dramatically improved performance while preserving the decoupling that makes our platform infinitely extensible. Key Takeaways: - How integration events enabled unprecedented scale and extensibility - Practical strategies for event-based data prefetching that supercharge performance - Solutions to common event-driven architecture challenges - When to break conventional architectural rules for specific contexts

How to purchase, license and subscribe to Microsoft Azure_PDF.pdfvictordsane

Microsoft Azure is a cloud platform that empowers businesses with scalable computing, data analytics, artificial intelligence, and cybersecurity capabilities. Arguably the biggest hurdle for most organizations is understanding how to get started. Microsoft Azure is a consumption-based cloud service. This means you pay for what you use. Unlike traditional software, Azure resources (e.g., VMs, databases, storage) are billed based on usage time, storage size, data transfer, or resource configurations. There are three primary Azure purchasing models: • Pay-As-You-Go (PAYG): Ideal for flexibility. Billed monthly based on actual usage. • Azure Reserved Instances (RI): Commit to 1- or 3-year terms for predictable workloads. This model offers up to 72% cost savings. • Enterprise Agreements (EA): Best suited for large organizations needing comprehensive Azure solutions and custom pricing. Licensing Azure: What You Need to Know Azure doesn’t follow the traditional “per seat” licensing model. Instead, you pay for: • Compute Hours (e.g., Virtual Machines) • Storage Used (e.g., Blob, File, Disk) • Database Transactions • Data Transfer (Outbound) Purchasing and subscribing to Microsoft Azure is more than a transactional step, it’s a strategic move. Get in touch with our team of licensing experts via [email protected] to further understand the purchasing paths, licensing options, and cost management tools, to optimize your investment.

Essentials of Resource Planning in a DownturnOnePlan Solutions

Software Engineering Process, Notation & Tools Introduction - Part 3Gaurav Sharma

Maintaining + Optimizing Database Health: Vendors, Orchestrations, Enrichment...BradBedford3

Join the Denver Marketo User Group, Captello and Integrate as we dive into the best practices, tools, and strategies for maintaining robust, high-performing databases. From managing vendors and automating orchestrations to enriching data for better insights, this session will unpack the key elements that keep your data ecosystem running smoothly—and smartly. We will hear from Steve Armenti, Twelfth, and Aaron Karpaty, Captello, and Frannie Danzinger, Integrate.

Generative Artificial Intelligence and its ApplicationsSandeepKS52

The exploration of generative AI begins with an overview of its fundamental concepts, highlighting how these technologies create new content and ideas by learning from existing data. Following this, the focus shifts to the processes involved in training and fine-tuning models, which are essential for enhancing their performance and ensuring they meet specific needs. Finally, the importance of responsible AI practices is emphasized, addressing ethical considerations and the impact of AI on society, which are crucial for developing systems that are not only effective but also beneficial and fair.

Key AI Technologies Used by Indian Artificial Intelligence CompaniesMypcot Infotech

Indian tech firms are rapidly adopting advanced tools like machine learning, natural language processing, and computer vision to drive innovation. These key AI technologies enable smarter automation, data analysis, and decision-making. Leading developments are shaping the future of digital transformation among top artificial intelligence companies in India. For more information please visit here https://www.mypcot.com/artificial-intelligence

FME for Climate Data: Turning Big Data into Actionable InsightsSafe Software

Regional and local governments aim to provide essential services for stormwater management systems. However, rapid urbanization and the increasing impacts of climate change are putting growing pressure on these governments to identify stormwater needs and develop effective plans. To address these challenges, GHD developed an FME solution to process over 20 years of rainfall data from rain gauges and USGS radar datasets. This solution extracts, organizes, and analyzes Next Generation Weather Radar (NEXRAD) big data, validates it with other data sources, and produces Intensity Duration Frequency (IDF) curves and future climate projections tailored to local needs. This presentation will showcase how FME can be leveraged to manage big data and prioritize infrastructure investments.

Best Inbound Call Tracking Software for Small BusinessesTheTelephony

The best inbound call tracking software for small businesses offers features like call recording, real-time analytics, lead attribution, and CRM integration. It helps track marketing campaign performance, improve customer service, and manage leads efficiently. Look for solutions with user-friendly dashboards, customizable reporting, and scalable pricing plans tailored for small teams. Choosing the right tool can significantly enhance communication and boost overall business growth.

Boost Student Engagement with Smart Attendance Software for SchoolsVisitu

Revolutionize Your Insurance Workflow with Claims Management SoftwareInsurance Tech Services

Agile Software Engineering MethodologiesGaurav Sharma

Build Smarter, Deliver Faster with Choreo - An AI Native Internal Developer P...WSO2

Enterprises must deliver intelligent, cloud native applications quickly—without compromising governance or scalability. This session explores how an internal developer platform increases productivity via AI for code and accelerates AI-native app delivery via code for AI. Learn practical techniques for embedding AI in the software lifecycle, automating governance with AI agents, and applying a cell-based architecture for modularity and scalability. Real-world examples and proven patterns will illustrate how to simplify delivery, enhance developer productivity, and drive measurable outcomes. Learn more: https://wso2.com/choreo

Leveraging Foundation Models to Infer IntentsKeheliya Gallaba

Invited Talk at RAISE 2025: Requirements engineering for AI-powered SoftwarE Workshop co-located with ICSE, the IEEE/ACM International Conference on Software Engineering. Abstract: Foundation Models (FMs) have shown remarkable capabilities in various natural language tasks. However, their ability to accurately capture stakeholder requirements remains a significant challenge for using FMs for software development. This paper introduces a novel approach that leverages an FM-powered multi-agent system called AlignMind to address this issue. By having a cognitive architecture that enhances FMs with Theory-of-Mind capabilities, our approach considers the mental states and perspectives of software makers. This allows our solution to iteratively clarify the beliefs, desires, and intentions of stakeholders, translating these into a set of refined requirements and a corresponding actionable natural language workflow in the often-overlooked requirements refinement phase of software engineering, which is crucial after initial elicitation. Through a multifaceted evaluation covering 150 diverse use cases, we demonstrate that our approach can accurately capture the intents and requirements of stakeholders, articulating them as both specifications and a step-by-step plan of action. Our findings suggest that the potential for significant improvements in the software development process justifies these investments. Our work lays the groundwork for future innovation in building intent-first development environments, where software makers can seamlessly collaborate with AIs to create software that truly meets their needs.

Topic 26 Security Testing Considerations.pptxmarutnand8

Artificial Intelligence Applications Across IndustriesSandeepKS52

Artificial Intelligence is a rapidly growing field that influences many aspects of modern life, including transportation, healthcare, and finance. Understanding the basics of AI provides insight into how machines can learn and make decisions, which is essential for grasping its applications in various industries. In the automotive sector, AI enhances vehicle safety and efficiency through advanced technologies like self-driving systems and predictive maintenance. Similarly, in healthcare, AI plays a crucial role in diagnosing diseases and personalizing treatment plans, while in financial services, it helps in fraud detection and risk management. By exploring these themes, a clearer picture of AI's transformative impact on society emerges, highlighting both its potential benefits and challenges.

Scalefusion Remote Access for Apple DevicesScalefusion

🔌Tried restarting. 🔁Then updating. 🔎Then Googled a fix. And then it crashed. Guess who has to fix it? You. And who’ll help you? - Scalefusion. Scalefusion steps in with real-time access, not just remote hope. Support for Apple devices that support you (and them) to do more. For more: https://scalefusion.com/remote-access-software-mac https://scalefusion.com/es/remote-access-software-mac https://scalefusion.com/fr/remote-access-software-mac https://scalefusion.com/pt-br/remote-access-software-mac https://scalefusion.com/nl/remote-access-software-mac https://scalefusion.com/de/remote-access-software-mac https://scalefusion.com/ru/remote-access-software-mac

Bonk coin airdrop_ Everything You Need to Know.pdfHerond Labs

The Bonk airdrop, one of the largest in Solana’s history, distributed 50% of its total supply to community members, significantly boosting its popularity and Solana’s network activity. Below is everything you need to know about the Bonk coin airdrop, including its history, eligibility, how to claim tokens, risks, and current status. https://blog.herond.org/bonk-coin-airdrop/

IBM Rational Unified Process For Software Engineering - IntroductionGaurav Sharma