Build and Run Containers With Lazy Pulling - Adoption status of containerd Stargz Snapshotter and eStargz

Build and Run Containers With Lazy Pulling - Adoption status of containerd Stargz Snapshotter and eStargz

• 1.
  Copyright(c)2021 NTT Corp.All Rights Reserved Build and Run Containers With Lazy Pulling Adop9on status of containerd Stargz Snapsho=er and eStargz Kohei Tokunaga, NTT Corpora9on FOSDEM 2021 (February 7)
• 2.
  Copyright(c)2021 NTT Corp.All Rights Reserved TL;DR l Pull is one of the time-consuming steps in the container lifecycle l Stargz Snapshotter, non-core subproject in containerd, is trying to solve it by lazy pulling • eStargz image based on Google stargz • Starndard compatibility, optimization and content verification l Collaboration in community • eStargz is usable with: containerd, Kubernetes, BuildKit, Kaniko, go-containerregistry, ko, nerdctl l On-going in 2021: Standardizing eStargz in OCI and improvements for stabilizing Stargz Snapshotter 0 5 10 15 20 25 30 35 40 45 estargz estargz-noopt legacy Start up Rme of python:3.7 (print “hello”) pull create run Host: EC2 Oregon (m5.2xlarge, Ubuntu 20.04) Registry: GitHub Container Registry (ghcr.io) Commit 7f45f74 (See detailed info in the later slides) [sec]
• 3.
  Copyright(c)2021 NTT Corp.All Rights Reserved Pull and OCI/Docker image
• 4.
  Copyright(c)2021 NTT Corp.All Rights Reserved Pull is 'me-consuming pulling packages accounts for 76% of container start Cme, but only 6.4% of that data is read [Harter et al. 2016] [Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribution with Lazy Docker Containers". 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA Caching images Minimizing image size Cold start is s'll slow Not all images are minimizable Language runCmes, frameworks, etc. Workarounds are known but not enough Node Registry Image Container pull run
• 5.
  Copyright(c)2021 NTT Corp.All Rights Reserved Problem on the current OCI/Docker image sha256:deadbeaf… sha256:1a3b5c… sha256:ffe63c… sha256:6ccde1… GET /v2/<image-name>/blobs/ bin/bash bin/ls etc/passwd etc/group usr/bin/apt layer = tarball (+compression) A container is a set of tarball layers A container can’t be started unNl the all layers become locally available l Need to scan the enNre blob even for extracNng single file entry • If the blob is gzip-compressed, it’s non-seekable anymore l No parallel extracNon • Need to scan the blob from the top, sequenNally
• 6.
  Copyright(c)2021 NTT Corp.All Rights Reserved eStargz and Stargz Snapsho/er
• 7.
  Copyright(c)2021 NTT Corp.All Rights Reserved l A plugin for containerd, developed in the non-core subproject l Allows containerd to lazily pull eStargz image from standard registry l eStargz comes with workload-based op7miza7on and content verifica7on Stargz Snapsho<er Standard-compatible lazy pulling with containerd Stargz Snapsho<er Kubelet, etc OCI runJmes Container Registry lazypull doesn’t download the en7re image on pull opera7on but fetches necessary chunks of contents on-demand https://github.com/containerd/stargz-snapshotter eStargz images
• 8.
  Copyright(c)2021 NTT Corp.All Rights Reserved The structure of eStargz l Seekable tar.gz and compaBble to RFC 1952 (gzip) = usable as a valid OCI/Docker image layer l Based on the stargz by Google CRFS (hIps://github.com/google/crfs) l eStargz comes with performance opBmizaBon and content verificaBon l Priori>zed files enables to prefetch and precache likely accessed files ⚠ eStargz is incompaBble to stargz: “footer” is changed to make eStargz compaBble to RFC 1952 eStargz bin/ls usr/bin/apt entrypoint.sh bin/bash Priori>zed files Prefetched by a single Range Request TOC(Table Of Contents) and footer l TOC contains files metadata, offset, etc… l footer contains offset of TOC ⚠ footer is incompaBble to stargz Files fetched on demand Can also be aggressively downloadeed in background gzip member per regular file/chunk It can be extracted per-file using HTTP Range Request stargz bin/ls usr/bin/apt entrypoint.sh bin/bash For more details: hIps://github.com/containerd/stargz-snapshoIer/blob/master/docs/stargz-estargz.md
• 9.
  Copyright(c)2021 NTT Corp.All Rights Reserved Workload-based Op/miza/on of eStargz proc container Input image Output image Convert & Op/mize Profile file access l Downloading each file/chunk on-demand costs extra overhead on each file access. l Leveraging eStargz, CLI converter command ctr-remote provides workload-based op/miza/on • Workload: entrypoint, envvar, etc… specified in Dockerfile (e.g. ENTRYPOINT) l ctr-remote analyzes which files are likely accessed during runRme • Runs provided image and profiles all file accesses • Regards accessed files are also likely accessed during runRme (= priori/zed files) • Stargz SnapshoTer will prefetch these files when mounts this image eStargz For more details: https://github.com/containerd/stargz-snapshotter/blob/master/docs/ctr-remote.md
• 10.
  Copyright(c)2021 NTT Corp.All Rights Reserved Content Verifica-on in eStargz chunkDigest chunkDigest chunkDigest containerd.io/snapshot/stargz/toc.digest file/chunk data file/chunk data file/chunk data Verified on resolve Verified on mount Verified on each fetch references by digest references by digest Manifest TOC (metadata of all files) l Chunks are lazily pulled from registry on-demand • so they cannot verified when mounJng the layer l Chunks are “lazily” verified • TOC (metadata file) records digests per chunk • Each chunk can be verified when it’s fetched to the node • TOC itself is verified when mounJng that layer using the digest wriOen in the manifest For more details: hOps://github.com/containerd/stargz-snapshoOer/blob/master/docs/verificaJon.md (the above figure is from this doc)
• 11.
  Copyright(c)2021 NTT Corp.All Rights Reserved Time to take for container startup l Measures the container startup ?me which includes: • Pulling an image from GitHub Container Registry • For language container, running “print hello world” program in the container • For server container, wai?ng for the readiness (un?l “up and running” message is printed) • This method is based on Hello Bench [Harter, et al. 2016] l Takes 95 percen?le of 100 opera?ons l Host: EC2 Oregon (m5.2xlarge, Ubuntu 20.04) l Registry: GitHub Container Registry (ghcr.io) l Target commit: 7f45f7438617728dd06bc9853a[5e42c1d3d5a3 [Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribu?on with Lazy Docker Containers". 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA
• 12.
  Copyright(c)2021 NTT Corp.All Rights Reserved Time to take for container startup 0 5 10 15 20 25 30 35 40 45 estargz estargz-noopt legacy python:3.7 (print “hello”) pull create run [sec] Waits for prefetch compleLon
• 13.
  Copyright(c)2021 NTT Corp.All Rights Reserved Time to take for container startup 0 10 20 30 40 50 60 estargz estargz-noopt legacy glassfish:4.1-jdk8 (runs unIl “Running GlassFish” is printed) pull create run [sec] Waits for prefetch compleIon
• 14.
  Copyright(c)2021 NTT Corp.All Rights Reserved Collabora'on in community
• 15.
  Copyright(c)2021 NTT Corp.All Rights Reserved eStargz on Kubernetes Nodes Stargz Snapsho3er External process gRPC API kubelet Lazy pull Container Registry eStargz Stargz Snapsho3er Stargz Snapsho3er l Lazy pulling can be enabled on Kubernetes using Stargz SnapshoCer, without patches • containerd is required as a CRI runHme l Stargz SnapshoCer needs to run on each node and containerd needs to be configured to recognize it l Real-world use-case at CERN for speeding up analysis pipeline [1] (13x faster pull for 5GB image) [1] Ricardo Rocha & Spyridon Trigazis, CERN. “Speeding Up Analysis Pipelines with Remote Container Images”. KubeCon+CloudNativeCon 2020 NA. https://sched.co/ekDj
• 16.
  Copyright(c)2021 NTT Corp.All Rights Reserved eStargz on containerd proc container Node Stargz Snapsho2er Lazy pull Container Registry eStargz Fetching files/chunks on demand Mounting rootfs as FUSE l Stargz Snapsho?er enables lazy pulling of eStargz on containerd • Implemented as a “remote snapsho?er” plugin l Mounts rooHs snapshots as FUSE and downloads accessed file contents on-demand l nerdctl (Docker-compaQble CLI for containerd; h?ps://github.com/AkihiroSuda/nerdctl) supports lazy pulling of eStargz on containerd Implemented as a “remote snapsho?er”
• 17.
  Copyright(c)2021 NTT Corp.All Rights Reserved FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev COPY ./hello.go /hello.go RUN go build -o hello /hello.go COPY and RUN without wai2ng for the pull comple2on Container Registry eStargz on BuildKit golang:1.15.3-esgz • /usr/local/go/bin/go • /usr/local/go/src/fmt/… etc... Fetch files/chunks on demand Build on node Lazy pull eStargz l BuildKit > v0.8.0 experimentally supports lazy pulling of eStargz base images during build • FROM instrucMon is skipped and chunks are lazily pulled on-demand during COPY/RUN/etc. l Can shorten the Mme of build e.g. on temporary (and fresh) CI instances with big base images. l More details at blog: hVps://medium.com/nVlabs/buildkit-lazypull-66c37690963f • speeding up building ”hello world” image from tens of seconds to a few seconds at the best
• 18.
  Copyright(c)2021 NTT Corp.All Rights Reserved Tools start to support eStargz crea0on (1/2) ctr-remote l Image converter developed in Stargz Snapshotter project l Converts image to eStargz l Comes with workload-based optimization hFps://github.com/containerd/stargz-snapshoFer/tree/master/cmd/ctr-remote nerdctl l Docker-compaKble CLI for containerd by Akihiro Suda, NTT l Converts image to eStargz l Comes with manual opKmizaKon (i.e. manually specifying prioriKzed files) hFps://github.com/AkihiroSuda/nerdctl kaniko l Container image builder by Google l Builds eStargz image (no opKmizaKon) l Base images need to be pre-converted to eStargz l GGCR_EXPERIMENT_ESTARGZ=1 is needed hFps://github.com/GoogleContainerTools/kaniko
• 19.
  Copyright(c)2021 NTT Corp.All Rights Reserved Tools start to support eStargz crea0on (2/2) go-containerregistry and crane CLI l Container registry client library and CLI by Google l Converts image to eStargz l Comes with manual opFmizaFon (i.e. manually specifying prioriFzed files) l GGCR_EXPERIMENT_ESTARGZ=1 is needed hPps://github.com/google/go-containerregistry ko l Build & Deployment tool of Go applicaFon on Kubernetes, by Google l Builds eStargz image (no opFmizaFon) l Base images need to be pre-converted to eStargz l GGCR_EXPERIMENT_ESTARGZ=1 is needed https://github.com/google/ko
• 20.
  Copyright(c)2021 NTT Corp.All Rights Reserved eStargz in 2021
• 21.
  Copyright(c)2021 NTT Corp.All Rights Reserved Updates will come in 2021 Standardizing eStargz l eStargz is proposed to OCI Image Spec l Discussion is on-going l Backward-compaHble extensions • OpHonal extension to applicaHon/vnd.oci.image.layer.v1.tar+gz • OpHonal annotaHon for content verificaHon hOps://github.com/opencontainers/image-spec/issues/815 Features and improvements for stabilizing Stargz Snapsho=er l Higher availability of Stargz Snapshotter (mounting images from multiple backend registries) l Improvements on memory consumption of Stargz Snapshotter l Speeding up image conversion l Static optimization of images l etc…
• 22.
  Copyright(c)2021 NTT Corp.All Rights Reserved Summary Feedbacks and sugges1ons are always welcome! h;ps://github.com/containerd/stargz-snapsho;er l Pull is one of the Gme-consuming steps in the container lifecycle l Stargz Snapsho:er, non-core subproject in containerd, is trying to solve it by lazy pulling • eStargz image based on Google stargz • Starndard compaGbility, opGmizaGon and content verificaGon l Collabora1on in community • eStargz on various plaMorms: Kubernetes, containerd and BuildKit • go-containerregistry, ko, kaniko and nerdctl start to support eStargz creaGon l On-going in 2021: Standardizing eStargz in OCI and improvements for stabilizing Stargz Snapsho;er