Michael Stumpf, profile picture
Uploaded byMichael Stumpf
1,504 views

Statistical analysis of network data and evolution on GPUs: High-performance statistical computing

The document discusses the statistical analysis of network data and its evolution on GPUs, highlighting their advantages in high-performance computing due to lower energy requirements and flexibility across programming languages. It explores the complexities of network evolution models and how various methodologies, such as approximate Bayesian computation, can be employed to summarize and evaluate network structures. Challenges in using GPUs for computational statistics are addressed, including their original design for gaming applications, which may conflict with the needs of statistical analysis.

Download to read offline
Statistical analysis of network data and evolution on GPUs: High-performance statistical computing Thomas Thorne & Michael P.H. Stumpf Theoretical Systems Biology Group 25/01/2012 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf 1 of 22
Trends in High-Performance Computing CUDA OpenCL MPI OpenMPI PVM Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 2 of 22
(Hidden) Costs of Computing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
(Hidden) Costs of Computing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Run Time on single threads C/C++ (or Fortran) have better performance characteristics. But for specialized tasks other languages, e.g. Haskell, can show good characteristics. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
(Hidden) Costs of Computing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Run Time on single threads C/C++ (or Fortran) have better performance characteristics. But for specialized tasks other languages, e.g. Haskell, can show good characteristics. Energy Requirements Every Watt we use for computing we also have to extract with air conditioning. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
(Hidden) Costs of Computing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Run Time on single threads C/C++ (or Fortran) have better performance characteristics. But for specialized tasks other languages, e.g. Haskell, can show good characteristics. Energy Requirements Every Watt we use for computing we also have to extract with air conditioning. The role of GPUs • GPUs can be accessed from many different programming languages (e.g. PyCUDA). • GPUs have a comparatively small footprint and relatively modest energy requirements compared to clusters of CPUs. • GPUs were designed for consumer electronics: computer gamers have different needs from the HPC community. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
Evolving Networks ι δ δ ι δ γ δ Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf γ Network Evolution 4 of 22
Evolving Networks Model-Based Evolutionary Analysis • For sequence data we use models of nucleotide substitution in order to infer phylogenies in a likelihood or Bayesian framework. • None of these models — even the general time-reversible model — are particularly realistic; but by allowing for complicating factors e.g. rate variation we capture much of the variability observed across a phylogenetic panel. • Modes of network evolution will be even more complicated and exhibit high levels of contingency; moreover the structure and function of different parts of the network will be intricately linked. • Nevertheless we believe that modelling the processes underlying the evolution of networks can provide useful insights; in particular we can study how functionality is distributed across groups of genes. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 4 of 22
Network Evolution Models (a) Duplication attachment (b) Duplication attachment with complimentarity wj (c) Linear preferential wi (d) General scale-free attachment Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 5 of 22
ABC on Networks Summarizing Networks • Data are noisy and incomplete. • We can simulate models of network evolution, but this does not allow us to calculate likelihoods for all but very trivial models. • There is also no sufficient statistic that would allow us to summarize networks, so ABC approaches require some thought. • Many possible summary statistics of networks are expensive to calculate. Full likelihood: Wiuf et al., PNAS (2006). ABC: Ratman et al., PLoS Comp.Biol. (2008). Stumpf & Wiuf, J. Roy. Soc. Interface (2010). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 6 of 22
Graph Spectrum c a b c d e   0 1 1 1 0 a a e   d  1 0 1 1 0 b   A = 1 1 0 0 0  c    1 1 0 0 1 d   b 0 0 0 1 0 e Graph Spectra Given a graph G comprised of a set of nodes N and edges (i , j ) ∈ E with i , j ∈ N, the adjacency matrix, A, of the graph is dened by 1 if (i , j ) ∈ E , ai ,j = 0 otherwise. The eigenvalues, Ν, of this matrix provide one way of dening the graph spectrum. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 7 of 22
Spectral Distances A simple distance measure between graphs having adjacency matrices A and B, known as the edit distance, is to count the number of edges that are not shared by both graphs, D (A, B ) = (ai ,j − bi ,j )2 . i ,j Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 8 of 22
Spectral Distances A simple distance measure between graphs having adjacency matrices A and B, known as the edit distance, is to count the number of edges that are not shared by both graphs, D (A, B ) = (ai ,j − bi ,j )2 . i ,j However for unlabelled graphs we require some mapping h from i ∈ NA to i ∈ NB that minimizes the distance D (A, B ) Dh (A, B ) = (ai ,j − bh(i ),h(j ) )2 , i ,j Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 8 of 22
Spectral Distances A simple distance measure between graphs having adjacency matrices A and B, known as the edit distance, is to count the number of edges that are not shared by both graphs, D (A, B ) = (ai ,j − bi ,j )2 . i ,j However for unlabelled graphs we require some mapping h from i ∈ NA to i ∈ NB that minimizes the distance D (A, B ) Dh (A, B ) = (ai ,j − bh(i ),h(j ) )2 , i ,j Given a spectrum (which is relatively cheap to compute) we have (α) (β) 2 D (A, B ) = λl − λl l Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 8 of 22
ABC using Graph Spectra For an observed network, N, and a simulated network, Sθ , we use the distance between the spectra (N) (S) 2 D (N, Sθ ) = λl − λl , l in our ABC SMC procedure. Note that this distance is a close lower bound on the distance between the raw data; we therefore do not have to bother with summary statistics. Also, calculating graph spectra costs as much as calculating other O (N 3 ) statistics (such as all shortest paths, the network diameter or the within-reach distribution). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 9 of 22
Simulated Data δ 1.5 0.4 0.4 0.4 0.3 0.3 0.3 1.0 0.2 0.2 0.2 0.5 0.1 0.1 0.1 0.0 0.0 0.0 0.0 −0.1 0.1 0.3 0.5 0.0 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 α 0.4 0.8 0.8 0.8 0.3 0.6 0.6 0.6 0.2 0.4 0.4 0.4 0.1 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0 0.1 0.2 0.3 0.4 −0.2 0.2 0.6 1.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p 1.0 1.0 1.0 0.4 0.8 0.8 0.8 0.3 0.6 0.6 0.6 0.2 0.4 0.4 0.4 0.1 0.2 0.2 0.2 0.0 0.0 0.1 0.2 0.3 0.4 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0.0 0.2 0.4 0.6 0.8 1.0 m 1.0 1.0 1.0 1.2 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0 0.1 0.2 0.3 0.4 0.0 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 10 of 22
Protein Interaction Network Data Species Proteins Interactions Genome size Sampling fraction S.cerevisiae 5035 22118 6532 0.77 D. melanogaster 7506 22871 14076 0.53 H. pylori 715 1423 1589 0.45 E. coli 1888 7008 5416 0.35 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
Protein Interaction Network Data Species Proteins Interactions Genome size Sampling fraction S.cerevisiae 5035 22118 6532 0.77 D. melanogaster 7506 22871 14076 0.53 H. pylori 715 1423 1589 0.45 E. coli 1888 7008 5416 0.35 0.5 0.4 Model probability Organism 0.3 S.cerevisae D.melanogaster H.pylori E.coli 0.2 0.1 0.0 DA DAC LPA SF DACL DACR Model Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
Protein Interaction Network Data Species Proteins Interactions Genome size Sampling fraction S.cerevisiae 5035 22118 6532 0.77 D. melanogaster 7506 22871 14076 0.53 H. pylori 715 1423 1589 0.45 E. coli 1888 7008 5416 0.35 0.5 Model Selection • Inference here was based on all 0.4 the data, not summary Model probability 0.3 Organism S.cerevisae statistics. D.melanogaster H.pylori E.coli • Duplication models receive the 0.2 strongest support from the data. 0.1 • Several models receive support and no model is chosen 0.0 unambiguously. DA DAC LPA SF DACL DACR Model Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
Protein Interaction Network Data δ ι 15 8 6 10 DA 4 5 2 0 0 0.0 0.4 0.8 0.0 0.4 0.8 δ ι 15 8 6 10 DAC 4 5 2 S.cerevisiae 0 0 0.0 0.4 0.8 0.0 0.4 0.8 D. melanogaster δ ι p m H. pylori 1.0 10 10 4 0.8 E. coli 8 8 3 0.6 6 6 DACL 2 0.4 4 4 1 0.2 2 2 0.0 0 0 0 0.0 0.4 0.8 0.0 0.4 0.8 0.0 0.4 0.8 0 2 4 6 8 10 δ ι p m 1.0 4 5 8 0.8 4 3 6 0.6 3 DACR 2 4 0.4 2 1 2 0.2 1 0.0 0 0 0 0.0 0.4 0.8 0.0 0.4 0.8 0.0 0.4 0.8 0 2 4 6 8 10 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
GPUs in Computational Statistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
GPUs in Computational Statistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
GPUs in Computational Statistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). • Performance depends crucially on our ability to map tasks onto the hardware. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
GPUs in Computational Statistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). • Performance depends crucially on our ability to map tasks onto the hardware. Challenges • GPU hardware was initially conceived for different purposes — computer gamers need fewer random numbers than MCMC or SMC procedures. The address space accessible to single-precision numbers also suffices to kill zombies etc . Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
GPUs in Computational Statistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). • Performance depends crucially on our ability to map tasks onto the hardware. Challenges • GPU hardware was initially conceived for different purposes — computer gamers need fewer random numbers than MCMC or SMC procedures. The address space accessible to single-precision numbers also suffices to kill zombies etc . • Combining several GPUs requires additional work, using e.g. MPI. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
Alternatives to GPUs: FPGAs Field Programmable Gate Arrays are congurable electronic circuits that are partly (re)congurable. Here the hardware is adapted to the 2 D.B. Thomas, W. Luk, and M. Stumpf problem at hand and encapsulates the programme in its programmable logic arrays. Multiple instances in xc4vlx60 Performance (MSwap-Compare/s) 100000 Single Instance in Virtex-4 Quad Opteron Software 10000 1000 100 10 1 4 9 14 19 24 Graph Size Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 13 of 22 g. 9. A comparison of the raw swap-compare performance between the Virtex
Alternatives to GPUs: CPUs (and MPI. . . ) CPUs with multiple cores 40 are flexible, have large address spaces and a wealth of flexible numerical 30 routines. This makes implementation of processor time numerically demanding 20 CPU GPU tasks relatively straightforward. In particular there is less of an 10 incentive to consider how a problem is best implemented in software 100 500 1000 5000 10000 50000 size that takes advantage of hardware features. 6-core Xeon vs M2050 (448 cores) programmed in OpenCL Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 14 of 22
(Pseudo) Random Numbers The Mersenne-Twister is one of the standard random number generators for simulation. MT19937 has a period of 219937 − 1 ≈ 4 × 106001 . Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 15 of 22
(Pseudo) Random Numbers The Mersenne-Twister is one of the standard random number generators for simulation. MT19937 has a period of 219937 − 1 ≈ 4 × 106001 . But MT does not have cryptographic strength (once 624 iterates have been observed, all future states are predictable), unlike Blum-Blum-Shub, xn+1 = xn modM , where M = pq with p, q large prime numbers. But it is too slow for simulations. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 15 of 22
(Pseudo) Random Numbers The Mersenne-Twister is one of the standard random number generators for simulation. MT19937 has a period of 219937 − 1 ≈ 4 × 106001 . But MT does not have cryptographic strength (once 624 iterates have been observed, all future states are predictable), unlike Blum-Blum-Shub, xn+1 = xn modM , where M = pq with p, q large prime numbers. But it is too slow for simulations. Parallel Random Number Generation • Running RNGs in parallel does not produce reliable sets of random numbers. • We need algorithms produce large numbers of parallel streams of “good” random numbers. • We also need better algorithms for weighted sampling. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 15 of 22
What GPUs are Good At • GPUs are good for linear threads involving mathematical functions. • We should avoid loops and branches in the code. • In an optimal world we should aim for all threads to finish at the same time. Execution cycle Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 16 of 22
Generating RNGs in Parallel We assume that we have a function xn+1 = f (xn ) which generates a stream of (pseudo) random numbers x0 , x1 , x2 , . . . , xr , . . . , xs , . . . , xt , . . . , xz Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 17 of 22
Generating RNGs in Parallel We assume that we have a function xn+1 = f (xn ) which generates a stream of (pseudo) random numbers x0 , x1 , x2 , . . . , xr , . . . , xs , . . . , xt , . . . , xz Parallel Streams x0 , x1 , . . . x0 , x1 , . . . x0 , x1 , . . . Sub-Streams x0 , . . . , xr −1 xr , . . . , xs−1 xs , . . . , xt −1 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 17 of 22
Counter-Based RNGs We can represent RNGs as state-space processes, yn+1 = f (yn ) with f :Y −→ Y xn+1 = gk ,nmodJ (y n+1/J ) with g :Y × K × ZJ −→ X , where x are essentially behaving as x ∼ U[0,1] ; here K is the key space, J the number of random numbers generated from the internal state of the RNG. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 18 of 22
Counter-Based RNGs We can represent RNGs as state-space processes, yn+1 = f (yn ) with f :Y −→ Y xn+1 = gk ,nmodJ (y n+1/J ) with g :Y × K × ZJ −→ X , where x are essentially behaving as x ∼ U[0,1] ; here K is the key space, J the number of random numbers generated from the internal state of the RNG. Salmon et al.(SC11) propose to use a simple form for f (·) and gk ,j = hk ◦ bk Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 18 of 22
Counter-Based RNGs We can represent RNGs as state-space processes, yn+1 = f (yn ) with f :Y −→ Y xn+1 = gk ,nmodJ (y n+1/J ) with g :Y × K × ZJ −→ X , where x are essentially behaving as x ∼ U[0,1] ; here K is the key space, J the number of random numbers generated from the internal state of the RNG. Salmon et al.(SC11) propose to use a simple form for f (·) and gk ,j = hk ◦ bk For a sufficiently complex bijection bk we can use simple updates and leave the randomization to bk . Here cryptographic routines come in useful. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 18 of 22
RNG Performance on CPUs and GPUs Method Max. Min. Output Intel CPU Nvidia GPU AMD GPU input state size cpB GB/s cpB GB/s cpB GB/s Counter-based, Cryptographic AES(sw) (1+0)×16 11×16 1×16 31.2 0.4 – – – – AES(hw) (1+0)×16 11×16 1×16 1.7 7.2 – – – – Threefish (Threefry-4×64-72) (4+4)×8 0 4×8 7.3 1.7 51.8 15.3 302.8 4.5 Counter-based, Crush-resistant ARS-5(hw) (1+1)×16 0 1×16 0.7 17.8 – – – – ARS-7(hw) (1+1)×16 0 1×16 1.1 11.1 – – – – Threefry-2×64-13 (2+2)×8 0 2×8 2.0 6.3 13.6 58.1 25.6 52.5 Threefry-2×64-20 (2+2)×8 0 2×8 2.4 5.1 15.3 51.7 30.4 44.5 Threefry-4×64-12 (4+4)×8 0 4×8 1.1 11.2 9.4 84.1 15.2 90.0 Threefry-4×64-20 (4+4)×8 0 4×8 1.9 6.4 15.0 52.8 29.2 46.4 Threefry-4×32-12 (4+4)×4 0 4×4 2.2 5.6 9.5 83.0 12.8 106.2 Threefry-4×32-20 (4+4)×4 0 4×4 3.9 3.1 15.7 50.4 25.2 53.8 Philox2×64-6 (2+1)×8 0 2×8 2.1 5.9 8.8 90.0 37.2 36.4 Philox2×64-10 (2+1)×8 0 2×8 4.3 2.8 14.7 53.7 62.8 21.6 Philox4×64-7 (4+2)×8 0 4×8 2.0 6.0 8.6 92.4 36.4 37.2 Philox4×64-10 (4+2)×8 0 4×8 3.2 3.9 12.9 61.5 54.0 25.1 Philox4×32-7 (4+2)×4 0 4×4 2.4 5.0 3.9 201.6 12.0 113.1 Philox4×32-10 (4+2)×4 0 4×4 3.6 3.4 5.4 145.3 17.2 79.1 Conventional, Crush-resistant MRG32k3a 0 6×4 1000×4 3.8 3.2 – – – – MRG32k3a 0 6×4 4×4 20.3 0.6 – – – – MRGk5-93 0 5×4 1×4 7.6 1.6 9.2 85.5 – – Conventional, Crushable Mersenne Twister 0 312×8 1×8 2.0 6.1 43.3 18.3 – – XORWOW 0 6×4 1×4 1.6 7.7 5.8 136.7 16.8 81.1 Table 2: Memory and performance characteristics for a variety of counter-based and conventional PRNGs. Taken from Salmon et al.(see References). a counter type of width c×w bytes and a key type of width Maximum input is written as (c+k)×w, indicating k×w bytes. Minimum state and output size are c×w bytes. Counter-based PRNG performance is reported with the minimal number of rounds for Crush-resistance, and also with extra rounds for “safety margin.” Performance is shown in bold for recommended PRNGs that have the best platform-specific performance with 22 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 19 of
Computational Challenges of GPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
Computational Challenges of GPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
Computational Challenges of GPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. • Bandwidth is an issue — coordination between GPU and CPU is much more challenging in statistical applications than e.g. for texture mapping of graphics rendering tasks. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
Computational Challenges of GPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. • Bandwidth is an issue — coordination between GPU and CPU is much more challenging in statistical applications than e.g. for texture mapping of graphics rendering tasks. • Programming has to be much more hardware aware than for CPUs — more precisely, non-hardware adapted programming will be more obviously less efficient than for CPUs. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
Computational Challenges of GPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. • Bandwidth is an issue — coordination between GPU and CPU is much more challenging in statistical applications than e.g. for texture mapping of graphics rendering tasks. • Programming has to be much more hardware aware than for CPUs — more precisely, non-hardware adapted programming will be more obviously less efficient than for CPUs. • There are some differences from conventional ANSI standards for mathematics (e.g. rounding). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
References GPUs and ABC • Zhou, Liepe, Sheng, Stumpf, Barnes (2011) GPU accelerated biochemical network simulation. Bioinformatics 27:874-876. • Liepe, Barnes, Cule, Erguler, Kirk, Toni, Stumpf (2010) ABC-SysBioapproximate Bayesian computation in Python with GPU support. Bioinformatics 26:1797-1799. GPUs and Random Numbers • Salmon, Moraes, Dror, Shaw (2011) Parallel Random Numbers: As Easy as 1, 2, 3. in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, ACM, New York; http://doi.acm.org/10.1145/2063384.2063405. • Howes, Thomas (2007) Efficient Random Number Generation and Application Using CUDA in GPU Gems 3, Addison-Wesley Professional, Boston; http: //http.developer.nvidia.com/GPUGems3/gpugems3_ch37.html. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 21 of 22
Acknowledgements Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 22 of 22

More Related Content

PDF
Parallel Biological Sequence Comparison in GPU Platforms
byGanesan Narayanasamy
 
PDF
Predicting organic reaction outcomes with weisfeiler lehman network
byKazuki Fujikawa
 
PPTX
SociaLite: High-level Query Language for Big Data Analysis
byDataWorks Summit
 
PDF
Matching networks for one shot learning
byKazuki Fujikawa
 
PDF
Graph Convolutional Neural Networks
by신동 강
 
PDF
2012 05-10 kaiser
bySCEE Team
 
PDF
Ijetr042170
byEngineering Research Publication
 
PDF
NIPS2017 Few-shot Learning and Graph Convolution
byKazuki Fujikawa
 
Parallel Biological Sequence Comparison in GPU Platforms
byGanesan Narayanasamy
 
Predicting organic reaction outcomes with weisfeiler lehman network
byKazuki Fujikawa
 
SociaLite: High-level Query Language for Big Data Analysis
byDataWorks Summit
 
Matching networks for one shot learning
byKazuki Fujikawa
 
Graph Convolutional Neural Networks
by신동 강
 
2012 05-10 kaiser
bySCEE Team
 
Ijetr042170
byEngineering Research Publication
 
NIPS2017 Few-shot Learning and Graph Convolution
byKazuki Fujikawa
 

What's hot

PDF
Sucha_ICC_2012
bysucha
 
PPTX
Master Thesis Defense
byFilipo MĂłr
 
PDF
Common Design of Deep Learning Frameworks
byKenta Oono
 
PDF
Colfax-Winograd-Summary _final (1)
bySangamesh Ragate
 
PPT
The Concurrent Constraint Programming Research Programmes -- Redux
byPierre Schaus
 
PPTX
論文紹介 Fast imagetagging
byTakashi Abe
 
PPTX
ECE 565 FInal Project
byLakshmi Yasaswi Kamireddy
 
PDF
VAE-type Deep Generative Models
byKenta Oono
 
PPT
gSpan algorithm
bySadik Mussah
 
PDF
D I G I T A L I C A P P L I C A T I O N S J N T U M O D E L P A P E R{Www
byguest3f9c6b
 
PPT
Software tookits for machine learning and graphical models
bybutest
 
PPTX
Optimized feedforward network of cnn with xnor v5
byIndiana University Bloomington
 
PDF
Optimizedfeedforwardnetworkofcnnwithxnorv5 180321130759
byVandna Sambyal
 
PDF
ICML2013読み会 Large-Scale Learning with Less RAM via Randomization
byHidekazu Oiwa
 
PPT
EEDC Programming Models
byRoger Rafanell Mas
 
PDF
70
bysrimoorthi
 
PPT
The Concurrent Constraint Programming Research Programmes -- Redux (part2)
byPierre Schaus
 
PPT
Trends In Graph Data Management And Mining
bySrinath Srinivasa
 
PDF
Present v0.2
byd4k
 
Sucha_ICC_2012
bysucha
 
Master Thesis Defense
byFilipo MĂłr
 
Common Design of Deep Learning Frameworks
byKenta Oono
 
Colfax-Winograd-Summary _final (1)
bySangamesh Ragate
 
The Concurrent Constraint Programming Research Programmes -- Redux
byPierre Schaus
 
論文紹介 Fast imagetagging
byTakashi Abe
 
ECE 565 FInal Project
byLakshmi Yasaswi Kamireddy
 
VAE-type Deep Generative Models
byKenta Oono
 
gSpan algorithm
bySadik Mussah
 
D I G I T A L I C A P P L I C A T I O N S J N T U M O D E L P A P E R{Www
byguest3f9c6b
 
Software tookits for machine learning and graphical models
bybutest
 
Optimized feedforward network of cnn with xnor v5
byIndiana University Bloomington
 
Optimizedfeedforwardnetworkofcnnwithxnorv5 180321130759
byVandna Sambyal
 
ICML2013読み会 Large-Scale Learning with Less RAM via Randomization
byHidekazu Oiwa
 
EEDC Programming Models
byRoger Rafanell Mas
 
70
bysrimoorthi
 
The Concurrent Constraint Programming Research Programmes -- Redux (part2)
byPierre Schaus
 
Trends In Graph Data Management And Mining
bySrinath Srinivasa
 
Present v0.2
byd4k
 

Similar to Statistical analysis of network data and evolution on GPUs: High-performance statistical computing

PDF
Two numerical graph algorithms
byDavid Gleich
 
PDF
Gremlin: A Graph-Based Programming Language
byMarko Rodriguez
 
PDF
Gephi short introduction
bySĂŠbastien
 
PDF
Harvard_University_-_Linear_Al
byramiljayureta
 
PDF
Large Scale Graph Processing with Apache Giraph
bysscdotopen
 
PPTX
Chapter10.pptx jtuffuryrufhrhrurrufudurhrhr
by123go8
 
PPT
1_1_Graphs_121120.ppt for BS mathematics
byAbdulWahidGhanghro
 
PDF
Approximate Bayesian Computation on GPUs
byMichael Stumpf
 
PPTX
ANT COLONY OPTIMIZATION FOR IMAGE EDGE DETECTION
byHimanshu Srivastava
 
PDF
Sample thesis
bykmmanuel
 
PDF
Graph Based Clustering
bySSA KPI
 
PDF
IAP09 CUDA@MIT 6.963 - Lecture 07: CUDA Advanced #2 (Nicolas Pinto, MIT)
bynpinto
 
PDF
Undirected graphs
byNick Lee
 
PDF
A comparative study on multicast routing using dijkstra’s, prims and ant colo...
byIAEME Publication
 
PDF
A comparative study on multicast routing using dijkstra’s
byiaemedu
 
PDF
Dsouza Supernova 2008
byTerrorNova Guild
 
PDF
Massive MapReduce Matrix Computations & Multicore Graph Algorithms
byDavid Gleich
 
PDF
Lecture 7: Data-Intensive Computing for Text Analysis (Fall 2011)
byMatthew Lease
 
PDF
Harvard_University_-_Linear_Al
byRamil Jay Ureta
 
PDF
Areejit Samal Randomizing metabolic networks
byAreejit Samal
 
Two numerical graph algorithms
byDavid Gleich
 
Gremlin: A Graph-Based Programming Language
byMarko Rodriguez
 
Gephi short introduction
bySĂŠbastien
 
Harvard_University_-_Linear_Al
byramiljayureta
 
Large Scale Graph Processing with Apache Giraph
bysscdotopen
 
Chapter10.pptx jtuffuryrufhrhrurrufudurhrhr
by123go8
 
1_1_Graphs_121120.ppt for BS mathematics
byAbdulWahidGhanghro
 
Approximate Bayesian Computation on GPUs
byMichael Stumpf
 
ANT COLONY OPTIMIZATION FOR IMAGE EDGE DETECTION
byHimanshu Srivastava
 
Sample thesis
bykmmanuel
 
Graph Based Clustering
bySSA KPI
 
IAP09 CUDA@MIT 6.963 - Lecture 07: CUDA Advanced #2 (Nicolas Pinto, MIT)
bynpinto
 
Undirected graphs
byNick Lee
 
A comparative study on multicast routing using dijkstra’s, prims and ant colo...
byIAEME Publication
 
A comparative study on multicast routing using dijkstra’s
byiaemedu
 
Dsouza Supernova 2008
byTerrorNova Guild
 
Massive MapReduce Matrix Computations & Multicore Graph Algorithms
byDavid Gleich
 
Lecture 7: Data-Intensive Computing for Text Analysis (Fall 2011)
byMatthew Lease
 
Harvard_University_-_Linear_Al
byRamil Jay Ureta
 
Areejit Samal Randomizing metabolic networks
byAreejit Samal
 

More from Michael Stumpf

PDF
Gaining Confidence in Signalling and Regulatory Networks
byMichael Stumpf
 
PDF
Considerate Approaches to ABC Model Selection
byMichael Stumpf
 
PDF
Beyond Static Networks: A Bayesian Non-Parametric Approach
byMichael Stumpf
 
PDF
Noisy information transmission through molecular interaction networks
byMichael Stumpf
 
PDF
Multi-Scale Models in Immunobiology
byMichael Stumpf
 
PDF
Are Powerlaws Useful?
byMichael Stumpf
 
PDF
Time-Variable Networks in Candida Glabrata
byMichael Stumpf
 
Gaining Confidence in Signalling and Regulatory Networks
byMichael Stumpf
 
Considerate Approaches to ABC Model Selection
byMichael Stumpf
 
Beyond Static Networks: A Bayesian Non-Parametric Approach
byMichael Stumpf
 
Noisy information transmission through molecular interaction networks
byMichael Stumpf
 
Multi-Scale Models in Immunobiology
byMichael Stumpf
 
Are Powerlaws Useful?
byMichael Stumpf
 
Time-Variable Networks in Candida Glabrata
byMichael Stumpf
 

Recently uploaded

PDF
ODSC West 2025 OpenMetadata Workshop.pdf
byOpenMetadata
 
PDF
MicroPython presentation - PyConFr Lyon 2025 - Amine Bendahmane
byAmine Bendahmane
 
PDF
Session 7 Specialized AI Associate Series: UiPath Communications Mining Overview
byDianaGray10
 
PDF
Fighting CVEs in Embedded Linux with the Yocto Project and OTA Updates
byLeon Anavi
 
PDF
FME Realize in the Real World - The Power of Spatial Computing in Action
bySafe Software
 
PDF
n8n - Next Generation Workflow Automation with Open Source
byDaisuke Masuda
 
PDF
Session 8 Specialized AI Associate Series: Fundamentals of Model Training
byDianaGray10
 
PPTX
Softaken SQLite DB Converter for Seamless SQLite Database file Conversion on ...
byDemetrio Parrilla
 
PDF
API Days Australia - API Management in the AI Era
byNilesh Gule
 
PDF
2025 AI Adoption Report: Gen AI Fast-Tracks Into the Enterprise
byRazin Mustafiz
 
PDF
Spec Driven Development with Kiro: Ensuring Quality and Traceability in the A...
byHaim Michael
 
PDF
AI Demands More: Help Ensure Network Readiness With ThousandEyes
byThousandEyes
 
PPTX
Artificial Intelligence lecture slides.pptx
bymaimelabernard6
 
PDF
Top 11 Sites to Buy Verified GitHub Accounts in 2025
byBuy GitHub Accounts Looking for a verified GitHub account?
 
PDF
Unlocking Access: Enriching Public Services with Location Intelligence.pdf
byPrecisely
 
PDF
How Generative AI Helps US Healthcare Startups Save 40.pdf
byimoliviabennett
 
PDF
Transcript: Embedding sustainability: Tips for ebook and print production - T...
byBookNet Canada
 
DOCX
Formula 2 - APIC HBEL general calculation.docx
byMarkus Janssen
 
PDF
Ready, set, go: Pre-fall sales trends and data-driven insights - Tech Forum 2025
byBookNet Canada
 
PDF
Transcript: Ready, set, go: Pre-fall sales trends and data-driven insights - ...
byBookNet Canada
 
ODSC West 2025 OpenMetadata Workshop.pdf
byOpenMetadata
 
MicroPython presentation - PyConFr Lyon 2025 - Amine Bendahmane
byAmine Bendahmane
 
Session 7 Specialized AI Associate Series: UiPath Communications Mining Overview
byDianaGray10
 
Fighting CVEs in Embedded Linux with the Yocto Project and OTA Updates
byLeon Anavi
 
FME Realize in the Real World - The Power of Spatial Computing in Action
bySafe Software
 
n8n - Next Generation Workflow Automation with Open Source
byDaisuke Masuda
 
Session 8 Specialized AI Associate Series: Fundamentals of Model Training
byDianaGray10
 
Softaken SQLite DB Converter for Seamless SQLite Database file Conversion on ...
byDemetrio Parrilla
 
API Days Australia - API Management in the AI Era
byNilesh Gule
 
2025 AI Adoption Report: Gen AI Fast-Tracks Into the Enterprise
byRazin Mustafiz
 
Spec Driven Development with Kiro: Ensuring Quality and Traceability in the A...
byHaim Michael
 
AI Demands More: Help Ensure Network Readiness With ThousandEyes
byThousandEyes
 
Artificial Intelligence lecture slides.pptx
bymaimelabernard6
 
Top 11 Sites to Buy Verified GitHub Accounts in 2025
byBuy GitHub Accounts Looking for a verified GitHub account?
 
Unlocking Access: Enriching Public Services with Location Intelligence.pdf
byPrecisely
 
How Generative AI Helps US Healthcare Startups Save 40.pdf
byimoliviabennett
 
Transcript: Embedding sustainability: Tips for ebook and print production - T...
byBookNet Canada
 
Formula 2 - APIC HBEL general calculation.docx
byMarkus Janssen
 
Ready, set, go: Pre-fall sales trends and data-driven insights - Tech Forum 2025
byBookNet Canada
 
Transcript: Ready, set, go: Pre-fall sales trends and data-driven insights - ...
byBookNet Canada
 

Statistical analysis of network data and evolution on GPUs: High-performance statistical computing

  • 1.
    Statistical analysis ofnetwork data and evolution on GPUs: High-performance statistical computing Thomas Thorne & Michael P.H. Stumpf Theoretical Systems Biology Group 25/01/2012 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf 1 of 22
  • 2.
    Trends in High-PerformanceComputing CUDA OpenCL MPI OpenMPI PVM Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 2 of 22
  • 3.
    (Hidden) Costs ofComputing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
  • 4.
    (Hidden) Costs ofComputing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Run Time on single threads C/C++ (or Fortran) have better performance characteristics. But for specialized tasks other languages, e.g. Haskell, can show good characteristics. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
  • 5.
    (Hidden) Costs ofComputing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Run Time on single threads C/C++ (or Fortran) have better performance characteristics. But for specialized tasks other languages, e.g. Haskell, can show good characteristics. Energy Requirements Every Watt we use for computing we also have to extract with air conditioning. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
  • 6.
    (Hidden) Costs ofComputing Prototyping/Development Time is typically reduced for scripting languages such as R or Python. Run Time on single threads C/C++ (or Fortran) have better performance characteristics. But for specialized tasks other languages, e.g. Haskell, can show good characteristics. Energy Requirements Every Watt we use for computing we also have to extract with air conditioning. The role of GPUs • GPUs can be accessed from many different programming languages (e.g. PyCUDA). • GPUs have a comparatively small footprint and relatively modest energy requirements compared to clusters of CPUs. • GPUs were designed for consumer electronics: computer gamers have different needs from the HPC community. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Introduction 3 of 22
  • 7.
    Evolving Networks ι δ δ ι δ γ δ Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf γ Network Evolution 4 of 22
  • 8.
    Evolving Networks Model-Based EvolutionaryAnalysis • For sequence data we use models of nucleotide substitution in order to infer phylogenies in a likelihood or Bayesian framework. • None of these models — even the general time-reversible model — are particularly realistic; but by allowing for complicating factors e.g. rate variation we capture much of the variability observed across a phylogenetic panel. • Modes of network evolution will be even more complicated and exhibit high levels of contingency; moreover the structure and function of different parts of the network will be intricately linked. • Nevertheless we believe that modelling the processes underlying the evolution of networks can provide useful insights; in particular we can study how functionality is distributed across groups of genes. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 4 of 22
  • 9.
    Network Evolution Models (a) Duplication attachment (b) Duplication attachment with complimentarity wj (c) Linear preferential wi (d) General scale-free attachment Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 5 of 22
  • 10.
    ABC on Networks SummarizingNetworks • Data are noisy and incomplete. • We can simulate models of network evolution, but this does not allow us to calculate likelihoods for all but very trivial models. • There is also no sufficient statistic that would allow us to summarize networks, so ABC approaches require some thought. • Many possible summary statistics of networks are expensive to calculate. Full likelihood: Wiuf et al., PNAS (2006). ABC: Ratman et al., PLoS Comp.Biol. (2008). Stumpf & Wiuf, J. Roy. Soc. Interface (2010). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 6 of 22
  • 11.
    Graph Spectrum c a b c d e   0 1 1 1 0 a a e   d  1 0 1 1 0 b   A = 1 1 0 0 0  c    1 1 0 0 1 d   b 0 0 0 1 0 e Graph Spectra Given a graph G comprised of a set of nodes N and edges (i , j ) ∈ E with i , j ∈ N, the adjacency matrix, A, of the graph is dened by 1 if (i , j ) ∈ E , ai ,j = 0 otherwise. The eigenvalues, Ν, of this matrix provide one way of dening the graph spectrum. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 7 of 22
  • 12.
    Spectral Distances A simpledistance measure between graphs having adjacency matrices A and B, known as the edit distance, is to count the number of edges that are not shared by both graphs, D (A, B ) = (ai ,j − bi ,j )2 . i ,j Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 8 of 22
  • 13.
    Spectral Distances A simpledistance measure between graphs having adjacency matrices A and B, known as the edit distance, is to count the number of edges that are not shared by both graphs, D (A, B ) = (ai ,j − bi ,j )2 . i ,j However for unlabelled graphs we require some mapping h from i ∈ NA to i ∈ NB that minimizes the distance D (A, B ) Dh (A, B ) = (ai ,j − bh(i ),h(j ) )2 , i ,j Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 8 of 22
  • 14.
    Spectral Distances A simpledistance measure between graphs having adjacency matrices A and B, known as the edit distance, is to count the number of edges that are not shared by both graphs, D (A, B ) = (ai ,j − bi ,j )2 . i ,j However for unlabelled graphs we require some mapping h from i ∈ NA to i ∈ NB that minimizes the distance D (A, B ) Dh (A, B ) = (ai ,j − bh(i ),h(j ) )2 , i ,j Given a spectrum (which is relatively cheap to compute) we have (α) (β) 2 D (A, B ) = λl − λl l Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 8 of 22
  • 15.
    ABC using GraphSpectra For an observed network, N, and a simulated network, Sθ , we use the distance between the spectra (N) (S) 2 D (N, Sθ ) = λl − λl , l in our ABC SMC procedure. Note that this distance is a close lower bound on the distance between the raw data; we therefore do not have to bother with summary statistics. Also, calculating graph spectra costs as much as calculating other O (N 3 ) statistics (such as all shortest paths, the network diameter or the within-reach distribution). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 9 of 22
  • 16.
    Simulated Data δ 1.5 0.4 0.4 0.4 0.3 0.3 0.3 1.0 0.2 0.2 0.2 0.5 0.1 0.1 0.1 0.0 0.0 0.0 0.0 −0.1 0.1 0.3 0.5 0.0 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 α 0.4 0.8 0.8 0.8 0.3 0.6 0.6 0.6 0.2 0.4 0.4 0.4 0.1 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0 0.1 0.2 0.3 0.4 −0.2 0.2 0.6 1.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p 1.0 1.0 1.0 0.4 0.8 0.8 0.8 0.3 0.6 0.6 0.6 0.2 0.4 0.4 0.4 0.1 0.2 0.2 0.2 0.0 0.0 0.1 0.2 0.3 0.4 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0.0 0.2 0.4 0.6 0.8 1.0 m 1.0 1.0 1.0 1.2 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0 0.1 0.2 0.3 0.4 0.0 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 10 of 22
  • 17.
    Protein Interaction NetworkData Species Proteins Interactions Genome size Sampling fraction S.cerevisiae 5035 22118 6532 0.77 D. melanogaster 7506 22871 14076 0.53 H. pylori 715 1423 1589 0.45 E. coli 1888 7008 5416 0.35 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
  • 18.
    Protein Interaction NetworkData Species Proteins Interactions Genome size Sampling fraction S.cerevisiae 5035 22118 6532 0.77 D. melanogaster 7506 22871 14076 0.53 H. pylori 715 1423 1589 0.45 E. coli 1888 7008 5416 0.35 0.5 0.4 Model probability Organism 0.3 S.cerevisae D.melanogaster H.pylori E.coli 0.2 0.1 0.0 DA DAC LPA SF DACL DACR Model Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
  • 19.
    Protein Interaction NetworkData Species Proteins Interactions Genome size Sampling fraction S.cerevisiae 5035 22118 6532 0.77 D. melanogaster 7506 22871 14076 0.53 H. pylori 715 1423 1589 0.45 E. coli 1888 7008 5416 0.35 0.5 Model Selection • Inference here was based on all 0.4 the data, not summary Model probability 0.3 Organism S.cerevisae statistics. D.melanogaster H.pylori E.coli • Duplication models receive the 0.2 strongest support from the data. 0.1 • Several models receive support and no model is chosen 0.0 unambiguously. DA DAC LPA SF DACL DACR Model Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
  • 20.
    Protein Interaction NetworkData δ ι 15 8 6 10 DA 4 5 2 0 0 0.0 0.4 0.8 0.0 0.4 0.8 δ ι 15 8 6 10 DAC 4 5 2 S.cerevisiae 0 0 0.0 0.4 0.8 0.0 0.4 0.8 D. melanogaster δ ι p m H. pylori 1.0 10 10 4 0.8 E. coli 8 8 3 0.6 6 6 DACL 2 0.4 4 4 1 0.2 2 2 0.0 0 0 0 0.0 0.4 0.8 0.0 0.4 0.8 0.0 0.4 0.8 0 2 4 6 8 10 δ ι p m 1.0 4 5 8 0.8 4 3 6 0.6 3 DACR 2 4 0.4 2 1 2 0.2 1 0.0 0 0 0 0.0 0.4 0.8 0.0 0.4 0.8 0.0 0.4 0.8 0 2 4 6 8 10 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Network Evolution 11 of 22
  • 21.
    GPUs in ComputationalStatistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
  • 22.
    GPUs in ComputationalStatistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
  • 23.
    GPUs in ComputationalStatistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). • Performance depends crucially on our ability to map tasks onto the hardware. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
  • 24.
    GPUs in ComputationalStatistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). • Performance depends crucially on our ability to map tasks onto the hardware. Challenges • GPU hardware was initially conceived for different purposes — computer gamers need fewer random numbers than MCMC or SMC procedures. The address space accessible to single-precision numbers also suffices to kill zombies etc . Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
  • 25.
    GPUs in ComputationalStatistics Performance • With highly optimized libraries (e.g. BLAST/ATLAS) we can run numerically demanding jobs relatively straightforwardly. • Whenever poor-man’s paralellization is possible, GPUs offer considerable advantages over multi-core CPU systems (at comparable cost and energy requirements). • Performance depends crucially on our ability to map tasks onto the hardware. Challenges • GPU hardware was initially conceived for different purposes — computer gamers need fewer random numbers than MCMC or SMC procedures. The address space accessible to single-precision numbers also suffices to kill zombies etc . • Combining several GPUs requires additional work, using e.g. MPI. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 12 of 22
  • 26.
    Alternatives to GPUs:FPGAs Field Programmable Gate Arrays are congurable electronic circuits that are partly (re)congurable. Here the hardware is adapted to the 2 D.B. Thomas, W. Luk, and M. Stumpf problem at hand and encapsulates the programme in its programmable logic arrays. Multiple instances in xc4vlx60 Performance (MSwap-Compare/s) 100000 Single Instance in Virtex-4 Quad Opteron Software 10000 1000 100 10 1 4 9 14 19 24 Graph Size Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 13 of 22 g. 9. A comparison of the raw swap-compare performance between the Virtex
  • 27.
    Alternatives to GPUs:CPUs (and MPI. . . ) CPUs with multiple cores 40 are flexible, have large address spaces and a wealth of flexible numerical 30 routines. This makes implementation of processor time numerically demanding 20 CPU GPU tasks relatively straightforward. In particular there is less of an 10 incentive to consider how a problem is best implemented in software 100 500 1000 5000 10000 50000 size that takes advantage of hardware features. 6-core Xeon vs M2050 (448 cores) programmed in OpenCL Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf GPU and the Alternatives 14 of 22
  • 28.
    (Pseudo) Random Numbers TheMersenne-Twister is one of the standard random number generators for simulation. MT19937 has a period of 219937 − 1 ≈ 4 × 106001 . Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 15 of 22
  • 29.
    (Pseudo) Random Numbers TheMersenne-Twister is one of the standard random number generators for simulation. MT19937 has a period of 219937 − 1 ≈ 4 × 106001 . But MT does not have cryptographic strength (once 624 iterates have been observed, all future states are predictable), unlike Blum-Blum-Shub, xn+1 = xn modM , where M = pq with p, q large prime numbers. But it is too slow for simulations. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 15 of 22
  • 30.
    (Pseudo) Random Numbers TheMersenne-Twister is one of the standard random number generators for simulation. MT19937 has a period of 219937 − 1 ≈ 4 × 106001 . But MT does not have cryptographic strength (once 624 iterates have been observed, all future states are predictable), unlike Blum-Blum-Shub, xn+1 = xn modM , where M = pq with p, q large prime numbers. But it is too slow for simulations. Parallel Random Number Generation • Running RNGs in parallel does not produce reliable sets of random numbers. • We need algorithms produce large numbers of parallel streams of “good” random numbers. • We also need better algorithms for weighted sampling. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 15 of 22
  • 31.
    What GPUs areGood At • GPUs are good for linear threads involving mathematical functions. • We should avoid loops and branches in the code. • In an optimal world we should aim for all threads to finish at the same time. Execution cycle Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 16 of 22
  • 32.
    Generating RNGs inParallel We assume that we have a function xn+1 = f (xn ) which generates a stream of (pseudo) random numbers x0 , x1 , x2 , . . . , xr , . . . , xs , . . . , xt , . . . , xz Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 17 of 22
  • 33.
    Generating RNGs inParallel We assume that we have a function xn+1 = f (xn ) which generates a stream of (pseudo) random numbers x0 , x1 , x2 , . . . , xr , . . . , xs , . . . , xt , . . . , xz Parallel Streams x0 , x1 , . . . x0 , x1 , . . . x0 , x1 , . . . Sub-Streams x0 , . . . , xr −1 xr , . . . , xs−1 xs , . . . , xt −1 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 17 of 22
  • 34.
    Counter-Based RNGs We canrepresent RNGs as state-space processes, yn+1 = f (yn ) with f :Y −→ Y xn+1 = gk ,nmodJ (y n+1/J ) with g :Y × K × ZJ −→ X , where x are essentially behaving as x ∼ U[0,1] ; here K is the key space, J the number of random numbers generated from the internal state of the RNG. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 18 of 22
  • 35.
    Counter-Based RNGs We canrepresent RNGs as state-space processes, yn+1 = f (yn ) with f :Y −→ Y xn+1 = gk ,nmodJ (y n+1/J ) with g :Y × K × ZJ −→ X , where x are essentially behaving as x ∼ U[0,1] ; here K is the key space, J the number of random numbers generated from the internal state of the RNG. Salmon et al.(SC11) propose to use a simple form for f (·) and gk ,j = hk ◦ bk Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 18 of 22
  • 36.
    Counter-Based RNGs We canrepresent RNGs as state-space processes, yn+1 = f (yn ) with f :Y −→ Y xn+1 = gk ,nmodJ (y n+1/J ) with g :Y × K × ZJ −→ X , where x are essentially behaving as x ∼ U[0,1] ; here K is the key space, J the number of random numbers generated from the internal state of the RNG. Salmon et al.(SC11) propose to use a simple form for f (·) and gk ,j = hk ◦ bk For a sufficiently complex bijection bk we can use simple updates and leave the randomization to bk . Here cryptographic routines come in useful. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 18 of 22
  • 37.
    RNG Performance onCPUs and GPUs Method Max. Min. Output Intel CPU Nvidia GPU AMD GPU input state size cpB GB/s cpB GB/s cpB GB/s Counter-based, Cryptographic AES(sw) (1+0)×16 11×16 1×16 31.2 0.4 – – – – AES(hw) (1+0)×16 11×16 1×16 1.7 7.2 – – – – Threefish (Threefry-4×64-72) (4+4)×8 0 4×8 7.3 1.7 51.8 15.3 302.8 4.5 Counter-based, Crush-resistant ARS-5(hw) (1+1)×16 0 1×16 0.7 17.8 – – – – ARS-7(hw) (1+1)×16 0 1×16 1.1 11.1 – – – – Threefry-2×64-13 (2+2)×8 0 2×8 2.0 6.3 13.6 58.1 25.6 52.5 Threefry-2×64-20 (2+2)×8 0 2×8 2.4 5.1 15.3 51.7 30.4 44.5 Threefry-4×64-12 (4+4)×8 0 4×8 1.1 11.2 9.4 84.1 15.2 90.0 Threefry-4×64-20 (4+4)×8 0 4×8 1.9 6.4 15.0 52.8 29.2 46.4 Threefry-4×32-12 (4+4)×4 0 4×4 2.2 5.6 9.5 83.0 12.8 106.2 Threefry-4×32-20 (4+4)×4 0 4×4 3.9 3.1 15.7 50.4 25.2 53.8 Philox2×64-6 (2+1)×8 0 2×8 2.1 5.9 8.8 90.0 37.2 36.4 Philox2×64-10 (2+1)×8 0 2×8 4.3 2.8 14.7 53.7 62.8 21.6 Philox4×64-7 (4+2)×8 0 4×8 2.0 6.0 8.6 92.4 36.4 37.2 Philox4×64-10 (4+2)×8 0 4×8 3.2 3.9 12.9 61.5 54.0 25.1 Philox4×32-7 (4+2)×4 0 4×4 2.4 5.0 3.9 201.6 12.0 113.1 Philox4×32-10 (4+2)×4 0 4×4 3.6 3.4 5.4 145.3 17.2 79.1 Conventional, Crush-resistant MRG32k3a 0 6×4 1000×4 3.8 3.2 – – – – MRG32k3a 0 6×4 4×4 20.3 0.6 – – – – MRGk5-93 0 5×4 1×4 7.6 1.6 9.2 85.5 – – Conventional, Crushable Mersenne Twister 0 312×8 1×8 2.0 6.1 43.3 18.3 – – XORWOW 0 6×4 1×4 1.6 7.7 5.8 136.7 16.8 81.1 Table 2: Memory and performance characteristics for a variety of counter-based and conventional PRNGs. Taken from Salmon et al.(see References). a counter type of width c×w bytes and a key type of width Maximum input is written as (c+k)×w, indicating k×w bytes. Minimum state and output size are c×w bytes. Counter-based PRNG performance is reported with the minimal number of rounds for Crush-resistance, and also with extra rounds for “safety margin.” Performance is shown in bold for recommended PRNGs that have the best platform-specific performance with 22 Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Random Numbers on GPUs 19 of
  • 38.
    Computational Challenges ofGPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
  • 39.
    Computational Challenges ofGPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
  • 40.
    Computational Challenges ofGPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. • Bandwidth is an issue — coordination between GPU and CPU is much more challenging in statistical applications than e.g. for texture mapping of graphics rendering tasks. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
  • 41.
    Computational Challenges ofGPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. • Bandwidth is an issue — coordination between GPU and CPU is much more challenging in statistical applications than e.g. for texture mapping of graphics rendering tasks. • Programming has to be much more hardware aware than for CPUs — more precisely, non-hardware adapted programming will be more obviously less efficient than for CPUs. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
  • 42.
    Computational Challenges ofGPUs • Address spaces are a potential issue: using single precision we are prone to run out of numbers for challenging MCMC/SMC applications very quickly. • Memory is an issue. In particular, registers/memory close to the computing cores are precious. • Bandwidth is an issue — coordination between GPU and CPU is much more challenging in statistical applications than e.g. for texture mapping of graphics rendering tasks. • Programming has to be much more hardware aware than for CPUs — more precisely, non-hardware adapted programming will be more obviously less efficient than for CPUs. • There are some differences from conventional ANSI standards for mathematics (e.g. rounding). Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 20 of 22
  • 43.
    References GPUs and ABC •Zhou, Liepe, Sheng, Stumpf, Barnes (2011) GPU accelerated biochemical network simulation. Bioinformatics 27:874-876. • Liepe, Barnes, Cule, Erguler, Kirk, Toni, Stumpf (2010) ABC-SysBioapproximate Bayesian computation in Python with GPU support. Bioinformatics 26:1797-1799. GPUs and Random Numbers • Salmon, Moraes, Dror, Shaw (2011) Parallel Random Numbers: As Easy as 1, 2, 3. in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, ACM, New York; http://doi.acm.org/10.1145/2063384.2063405. • Howes, Thomas (2007) Efficient Random Number Generation and Application Using CUDA in GPU Gems 3, Addison-Wesley Professional, Boston; http: //http.developer.nvidia.com/GPUGems3/gpugems3_ch37.html. Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 21 of 22
  • 44.
    Acknowledgements Network Analysis on GPUs Thomas Thorne & Michael P.H. Stumpf Conclusions 22 of 22