Kohei Tokunaga, profile picture
Uploaded byKohei Tokunaga
PDF, PPTX518 views

Faster Container Image Distribution on a Variety of Tools with Lazy Pulling

The document discusses advancements in container image distribution, focusing on 'lazy pulling' techniques that enhance container start times by reducing data pull time significantly. It introduces 'estargz' and 'nydus' as solutions offering compatibility with existing OCI image specifications while optimizing data integrity and retrieval processes. Real-world applications demonstrate these technologies speeding up processes such as analysis pipelines at CERN, indicating their effectiveness in improving container performance across multiple platforms.

Download as PDF, PPTX
Kohei Tokunaga, NTT Corporation Tao Peng, Ant Group Faster Container Image Distribution on a Variety of Tools with Lazy Pulling
Pull is time-consuming pulling packages accounts for 76% of container start time, but only 6.4% of that data is read [Harter et al. 2016] [Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribution with Lazy Docker Containers". 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA ● Large images speeds down cold start of containers ● Build takes long if base images are large ● Not all images are minimizable e.g. language runtimes, frameworks, ..
Problem on the current OCI image bin/bash bin/ls etc/passwd etc/group usr/bin/apt ・・・ Image is a set of tar (+compression) layers container can’t start before the all layers become locally available ● Non seekable • Need to scan the entire blob even for extracting single file entry ● No parallel extraction • Need to scan sequentially
OCI-alternative accelerated images and lazy pulling Lazy pulling: Starting up containers without waiting for the pull completion ● eStargz • Lazy pullable format with prefetch optimization + content verification • Proposed as a backward-compatible extension to OCI Image Spec ● Nydus • Lazy pullable format with prefetch, chunk dedup and e2e data integrity • Compatible with OCI Distribution Spec and Artifacts Spec • Proposed as OCI “v2” format (incompatible to current OCI Image Spec)
eStargz Kohei Tokunaga, NTT Corporation
eStargz: Standard-compatible lazy pulling ● 100% OCI-compatible • Lazy pullable from standard registries (ghcr.io, docker.io, …etc) • Even legacy (lazy-pulling-agnostic) runtime can run eStargz ● Usable with a variety of tools (tracker https://github.com/containerd/stargz-snapshotter/issues/258) • Kubernetes, k3s, containerd, CRI-O, Podman, BuildKit, Kaniko, ko, buildpacks.io, go-containerregistry… ● Performance optimization and content verification • Important files can be prefetched to avoid NW overhead • Each chunk comes with checksum Standard Container Registry eStargz-aware runtimes BuildKit Lazy pull Lazy pull legacy ctr-remote nerdctl Kaniko Image builders/convertes eStargz creates eStargz still runnable on eStargz-agnostic tools (e.g. Docker) etc… etc… legacy
eStargz image layer format eStargz bin/ls usr/bin/apt ・・・ ・・・ entrypoint.sh bin/bash Prioritized files Prefetched by a single Range Request TOC(Table Of Contents) and footer Files metadata, payload offset, etc… Files fetched on demand Downloaded in background as well gzip member per regular file/chunk can be extracted per-file using HTTP Range Request stargz bin/ls usr/bin/apt ・・・ ・・・ entrypoint.sh bin/bash For more details: https://github.com/containerd/stargz-snapshotter/blob/master/docs/stargz-estargz.md ● Discussed in Stargz Snapshotter of containerd: https://github.com/containerd/stargz-snapshotter ● Compatible to gzip = usable as a valid OCI/Docker image layer ● Based on stargz by CRFS (https://github.com/google/crfs) • eStargz comes with performance optimization and content verification
Benchmarking result [sec] Waits for prefetch completion during create ● Method based on Hello Bench [Harter, et al. 2016] ● Takes 95 percentile of 100 operations ● Host: EC2 Oregon (m5.2xlarge, Ubuntu 20.04) ● Registry: GitHub Container Registry (ghcr.io) ● Runtime: containerd ● Stargz Snapshotter commit: 7f45f7438617728dd06bc9853afb5e42c1d3d5a3
Benchmarking result ● Benchmarking spec is the same as the previous slide
Lazy pulling on Kubernetes Containerd Node Stargz Snapshotter external process gRPC API kubelet Lazy pull Container Registry eStargz CRI-O Node Stargz Store external process exposed on dir kubelet ● CRI runtimes + plugins (discussed later) enable lazy pulling on Kubernetes • Containerd + Stargz Snapshotter • CRI-O + Stargz Store ● Real-world use-case at CERN for speeding up analysis pipeline [1] (13x faster pull for 5GB image) [1] Ricardo Rocha & Spyridon Trigazis, CERN. “Speeding Up Analysis Pipelines with Remote Container Images”. KubeCon+CloudNativeCon 2020 NA. https://sched.co/ekDj $ kind create cluster --name estargz-demo --image ghcr.io/stargz-containers/estargz-kind-node:0.8.0 ● Lazy-pull-enabled KinD image is available on ghcr.io/stargz-containers repo ● k3s supports lazy pulling of eStargz (merged to the main, will be included in k3s v1.22) $ k3s server --snapshotter=stargz
proc Lazy pulling on containerd Stargz Snapshotter Fetches files/chunks on demand Mounts rootfs as FUSE ● Stargz Snapshotter plugin : https://github.com/containerd/stargz-snapshotter • Implements remote snapshotter plugin interface ● nerdctl (Docker-compatible CLI for containerd) supports lazy pulling • https://github.com/containerd/nerdctl $ nerdctl --snapshotter=stargz run ghcr.io/stargz-containers/python:3.9-esgz $ nerdctl --snapshotter=stargz compose -f docker-compose.stargz.yaml up Node Lazy pull Container Registry eStargz container
proc container Lazy pulling on Podman/CRI-O ● Stargz Store plugin : https://github.com/containerd/stargz-snapshotter • Developped in Stargz Snapshotter project • Implements Additional Layer Store plugin • Podman >= v3.3.0, CRI-O >= v1.22.0 Stargz Store Provides layers (mounted as FUSE) Podman, CRI-O Uses layers for rootfs $ podman run ghcr.io/stargz-containers/python:3.9-esgz Node Lazy pull Container Registry eStargz
Lazy pulling on BuildKit FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev COPY ./hello.go /hello.go RUN go build -o hello /hello.go COPY and RUN without waiting for the pull completion • /usr/local/go/bin/go • /usr/local/go/src/fmt/… etc... Fetch files/chunks on demand Build on node ● Experimentally supports lazy pulling of base images since v0.8.0 ● Can shorten the time of build that require pull • e.g. on temporary (and fresh) CI instances Lazy pull Container Registry eStargz golang:1.15.3-esgz More details at blog: https://medium.com/nttlabs/buildkit-lazypull-66c37690963f $ docker buildx create --use --name lazy-builder --driver docker-container --driver-opt image=moby/buildkit:master --buildkitd-flags '--oci-worker-snapshotter=stargz' $ docker buildx inspect --bootstrap lazy-builder $ docker buildx build .
Building eStargz BuildKit: https://github.com/moby/buildkit ● Building eStargz supported on the main branch, hope to come in v0.10.x ● eStargz is supported by Buildx or standalone BuildKit (buildctl + buildkitd) ● Docs: https://github.com/moby/buildkit/blob/master/docs/stargz-estargz.md#creating-stargzestargz-images $ docker buildx build -o type=registry,name=ktokunaga/hello:esgz,oci-mediatypes=true,compression=estargz . Kaniko: https://github.com/GoogleContainerTools/kaniko ● Image builder runnable in containers and Kubernetes ● Requires GGCR_EXPERIMENT_ESTARGZ=1 ● Base images need to be eStargz $ docker run --rm -e GGCR_EXPERIMENT_ESTARGZ=1 -v /tmp/context:/workspace -v ~/.docker/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.6.0 --destination "ghcr.io/ktock/sample:esgz"
Converting an image into eStargz ctr-remote: https://github.com/containerd/stargz-snapshotter/blob/v0.7.0/docs/ctr-remote.md ● CLI for containerd provided by Stargz Snapshotter project ● Supports prefetch optimization for eStargz $ ctr-remote image optimize --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz go-containerregistry and crane CLI : https://github.com/google/go-containerregistry ● Library and CLI to interact with registries ● requires GGCR_EXPERIMENT_ESTARGZ=1 ● downstream tools (e.g.Kaniko, ko and buildpacks.io) supports eStargz creation as well $ go install github.com/google/go-containerregistry/cmd/crane@latest $ GGCR_EXPERIMENT_ESTARGZ=1 crane optimize ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz nerdctl: https://github.com/containerd/nerdctl/blob/v0.11.1/docs/stargz.md ● Docker-compatible CLI for containerd ● Can be combined with nerdctl build command $ nerdctl image convert --estargz --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
Discussion toward OCIv1 extension ● eStargz is now widely usable on tools in the community ● Proposed to OCI Image Spec (discussion is on-going) ● Proposed as backward-compatible extensions • Extension for the current gzip layer ( application/vnd.oci.image.layer.v1.tar+gz ) • Additional annotation for content verification: org.opencontainers.image.toc.digest https://github.com/opencontainers/image-spec/issues/815
Nydus Image Service Tao Peng, Ant Group
Nydus: Improved Lazy Pulling and More ● Standard lazy pulling with prefetch policies • Lazy pullable from standard registries (cloud registries, docker registry, harbor, etc) • Semantical prefetch, hinted prefetch, background prefetch all data ● Chunk level deduplication with layered build cache • Speed up both image conversion and downloading ● End-to-end data integrity • Runtime data integrity check ● Reproducible Image Building • Build environment agnostic ● Compatible with OCI Distribution Spec and Artifacts Spec • Usable with most existing container registry implementations ● Rich container ecosystem integration • Kubernetes, docker, containerd, buildkit, harbor, dragonfly, runc, kata-containers etc ● Resource efficient and production ready • Large scale deployment at Ant and available via Alibaba Cloud services
FUSE/virtiofs Unified Architecture ● Native container runtime support • One translation layer (FUSE or virtiofs) for both runc and kata containers ● Shared uncompressed local cache • Download/uncompress once and use all the time ● Singleton mode with extremely low memory footprint • ~5MB per instance
Registry Acceleration File System ● Merkle tree metadata layer • self-verifiable ● Chunk-shared data layer • enabling chunk level deduplication
OCI Spec Compatibility ● Metadata/data layers are OCI manifest layers ● New media type in the image manifest spec ● Compatible with OCI artifacts/distribution spec ● Compatible registry GC functionality ● Widely working with container registries • cloud registries • docker registry • harbor • etc.
Lazy Pulling Benchmark
Lazy Pulling with Variety of Tools ● Nydus Snapshotter plugin • https://github.com/dragonflyoss/image-service/tree/master/contrib/nydus-sna pshotter ● Usable for variety of container management tools • kubernetes (kubelet) • crictl • nerdctl • ctr-remote
Manual Image Conversion -- nydusify ● nydusify • https://github.com/dragonflyoss/image-service/tree/master/contrib/nydusify • download image from remote registry, convert it to nydus image, and push nydus image to remote registry • nydusify convert --source <registry>/<repo>:<tag> --target <registry>/<repo>:<nydus-tag>
Manual Image Conversion -- buildkit ● buildkit • https://github.com/moby/buildkit/pull/2045 • build nydus image directly from dockerfile and push it to remote registry • buildctl build --frontend=dockerfile.v0 --local context=/build/dir --local dockerfile=/build/DockerfileDir --output type=nydus,name=<registry>/<repo>:<tag>
Automatic Conversion with Harbor ● https://github.com/goharbor/community/pull/167 ● New image conversion service in harbor ● Started image acceleration working group ● Image format agnostic • nydus • estargz • future more format
OCI Artifacts Manifest ● https://github.com/opencontainers/artifacts/pull/29 ● Map OCI image manifest to artifact manifest, to connect artifacts to images ● Beneficial to SBOM, Signatures, Nydus, Scan Results etc. ● Nydus image as an artifact type • cncf.nydus.v1-rc1
Community V2 Image Spec Requirements ● OCI community requirements on V2 Image format • https://hackmd.io/@cyphar/ociv2-brainstorm
Recap ● OCI images are large and slow ● Image lazy pulling is important for starting containers quickly ● Prefetching is a good friend to lazy pulling ● Ecosystem adoption is crucial for new image formats ● eStargz • Backward compatibility • Extension to the existing OCI image spec ● Nydus • Future looking • Proposal to the next generation OCI image spec
eStargz Project Information ● 100% OCI-compatible image format for lazy pulling ● Subproject of CNCF graduated containerd ● github： https://github.com/containerd/stargz-snapshotter ● slack： https://slack.containerd.io/ (#containerd-dev channel at CNCF slack) ● Pre-converted images are available on ghcr.io/stargz-containers
Nydus Project Information ● Image acceleration service with various improvements ● CNCF incubator dragonfly sub-project ● github： https://github.com/dragonflyoss/image-service ● slack： https://tinyurl.com/nydus-slack ● tutorial： https://tinyurl.com/nydus-tutorial
Thank You & Questions!

More Related Content

PDF
Kubernetes a comprehensive overview
byGabriel Carro
 
PDF
Rancher Rodeo
bySUSE
 
PPTX
Docker Networking Overview
bySreenivas Makam
 
PDF
[KubeCon EU 2022] Running containerd and k3s on macOS
byAkihiro Suda
 
PDF
NGINX Ingress Controller for Kubernetes
byNGINX, Inc.
 
PDF
Cloud-Native CI/CD on Kubernetes with Tekton Pipelines
byNikhil Thomas
 
PDF
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
byHenning Jacobs
 
PDF
The Patterns of Distributed Logging and Containers
bySATOSHI TAGOMORI
 
Kubernetes a comprehensive overview
byGabriel Carro
 
Rancher Rodeo
bySUSE
 
Docker Networking Overview
bySreenivas Makam
 
[KubeCon EU 2022] Running containerd and k3s on macOS
byAkihiro Suda
 
NGINX Ingress Controller for Kubernetes
byNGINX, Inc.
 
Cloud-Native CI/CD on Kubernetes with Tekton Pipelines
byNikhil Thomas
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
byHenning Jacobs
 
The Patterns of Distributed Logging and Containers
bySATOSHI TAGOMORI
 

What's hot

PDF
containerdの概要と最近の機能
byKohei Tokunaga
 
PDF
Dockerイメージの理解とコンテナのライフサイクル
byMasahito Zembutsu
 
PDF
P2P Container Image Distribution on IPFS With containerd and nerdctl
byKohei Tokunaga
 
PDF
Introduction and Deep Dive Into Containerd
byKohei Tokunaga
 
PDF
[KubeCon NA 2020] containerd: Rootless Containers 2020
byAkihiro Suda
 
PDF
Rootless Containers
byAkihiro Suda
 
PPTX
Dockerからcontainerdへの移行
byAkihiro Suda
 
PDF
Kubernetes ControllerをScale-Outさせる方法 / Kubernetes Meetup Tokyo #55
byPreferred Networks
 
PDF
BuildKitの概要と最近の機能
byKohei Tokunaga
 
PDF
DockerとKubernetesをかけめぐる
byKohei Tokunaga
 
PDF
Rootless Containers & Unresolved issues
byAkihiro Suda
 
PDF
Kubernetes
byerialc_w
 
PPTX
Containerd internals: building a core container runtime
byDocker, Inc.
 
PDF
Dockerライフサイクルの基礎　地雷を踏み抜けろ！
byMasahito Zembutsu
 
PPTX
OpenShift Introduction
byRed Hat Developers
 
PDF
Grafana LokiではじめるKubernetesロギングハンズオン（NTT Tech Conference #4 ハンズオン資料）
byNTT DATA Technology & Innovation
 
PPTX
Ceph アーキテクチャ概説
byEmma Haruka Iwao
 
PDF
[Container Runtime Meetup] runc & User Namespaces
byAkihiro Suda
 
PPTX
コンテナネットワーキング（CNI）最前線
byMotonori Shindo
 
PDF
仮想化環境におけるパケットフォワーディング
byTakuya ASADA
 
containerdの概要と最近の機能
byKohei Tokunaga
 
Dockerイメージの理解とコンテナのライフサイクル
byMasahito Zembutsu
 
P2P Container Image Distribution on IPFS With containerd and nerdctl
byKohei Tokunaga
 
Introduction and Deep Dive Into Containerd
byKohei Tokunaga
 
[KubeCon NA 2020] containerd: Rootless Containers 2020
byAkihiro Suda
 
Rootless Containers
byAkihiro Suda
 
Dockerからcontainerdへの移行
byAkihiro Suda
 
Kubernetes ControllerをScale-Outさせる方法 / Kubernetes Meetup Tokyo #55
byPreferred Networks
 
BuildKitの概要と最近の機能
byKohei Tokunaga
 
DockerとKubernetesをかけめぐる
byKohei Tokunaga
 
Rootless Containers & Unresolved issues
byAkihiro Suda
 
Kubernetes
byerialc_w
 
Containerd internals: building a core container runtime
byDocker, Inc.
 
Dockerライフサイクルの基礎　地雷を踏み抜けろ！
byMasahito Zembutsu
 
OpenShift Introduction
byRed Hat Developers
 
Grafana LokiではじめるKubernetesロギングハンズオン（NTT Tech Conference #4 ハンズオン資料）
byNTT DATA Technology & Innovation
 
Ceph アーキテクチャ概説
byEmma Haruka Iwao
 
[Container Runtime Meetup] runc & User Namespaces
byAkihiro Suda
 
コンテナネットワーキング（CNI）最前線
byMotonori Shindo
 
仮想化環境におけるパケットフォワーディング
byTakuya ASADA
 

Similar to Faster Container Image Distribution on a Variety of Tools with Lazy Pulling

PDF
[KubeCon EU 2020] containerd Deep Dive
byAkihiro Suda
 
PDF
[Paris Container Day 2021] nerdctl: yet another Docker & Docker Compose imple...
byAkihiro Suda
 
PDF
Starting up Containers Super Fast With Lazy Pulling of Images
byKohei Tokunaga
 
PDF
eStargzイメージとlazy pullingによる高速なコンテナ起動
byKohei Tokunaga
 
PDF
Stargz Snapshotter: イメージのpullを省略しcontainerdでコンテナを高速に起動する
byKohei Tokunaga
 
PDF
The overview of lazypull with containerd Remote Snapshotter & Stargz Snapshotter
byKohei Tokunaga
 
PDF
BuildKitでLazy Pullを有効にしてビルドを早くする話
byKohei Tokunaga
 
PPTX
containerd the universal container runtime
byDocker, Inc.
 
PDF
[KubeCon EU 2021] Introduction and Deep Dive Into Containerd
byAkihiro Suda
 
PDF
[Japan Community Day at KubeCon JP 2025] Overview of containerd
byKohei Tokunaga
 
PDF
Build and Run Containers With Lazy Pulling - Adoption status of containerd St...
byKohei Tokunaga
 
PDF
Startup Containers in Lightning Speed with Lazy Image Distribution
byKohei Tokunaga
 
PDF
20250617 [KubeCon JP 2025] containerd - Project Update and Deep Dive.pdf
byAkihiro Suda
 
PDF
[FOSDEM 2020] Lazy distribution of container images
byAkihiro Suda
 
PDF
Navigating container technology for enhanced security by Niklas Saari
byMetosin Oy
 
PDF
Container Runtimes and Tooling
byKublr
 
PDF
FILEgrain: Transport-Agnostic, Fine-Grained Content-Addressable Container Ima...
byAkihiro Suda
 
PPTX
Virtualization and Migration in Cloud - Edge Computing models using OpenStack...
bySai praveen Seva
 
PDF
Container Runtimes and Tooling, v2
byKublr
 
PPTX
The state of containerd
byDocker, Inc.
 
[KubeCon EU 2020] containerd Deep Dive
byAkihiro Suda
 
[Paris Container Day 2021] nerdctl: yet another Docker & Docker Compose imple...
byAkihiro Suda
 
Starting up Containers Super Fast With Lazy Pulling of Images
byKohei Tokunaga
 
eStargzイメージとlazy pullingによる高速なコンテナ起動
byKohei Tokunaga
 
Stargz Snapshotter: イメージのpullを省略しcontainerdでコンテナを高速に起動する
byKohei Tokunaga
 
The overview of lazypull with containerd Remote Snapshotter & Stargz Snapshotter
byKohei Tokunaga
 
BuildKitでLazy Pullを有効にしてビルドを早くする話
byKohei Tokunaga
 
containerd the universal container runtime
byDocker, Inc.
 
[KubeCon EU 2021] Introduction and Deep Dive Into Containerd
byAkihiro Suda
 
[Japan Community Day at KubeCon JP 2025] Overview of containerd
byKohei Tokunaga
 
Build and Run Containers With Lazy Pulling - Adoption status of containerd St...
byKohei Tokunaga
 
Startup Containers in Lightning Speed with Lazy Image Distribution
byKohei Tokunaga
 
20250617 [KubeCon JP 2025] containerd - Project Update and Deep Dive.pdf
byAkihiro Suda
 
[FOSDEM 2020] Lazy distribution of container images
byAkihiro Suda
 
Navigating container technology for enhanced security by Niklas Saari
byMetosin Oy
 
Container Runtimes and Tooling
byKublr
 
FILEgrain: Transport-Agnostic, Fine-Grained Content-Addressable Container Ima...
byAkihiro Suda
 
Virtualization and Migration in Cloud - Edge Computing models using OpenStack...
bySai praveen Seva
 
Container Runtimes and Tooling, v2
byKublr
 
The state of containerd
byDocker, Inc.
 

More from Kohei Tokunaga

PDF
Dockerからcontainerdへの移行
byKohei Tokunaga
 
PDF
今話題のいろいろなコンテナランタイムを比較してみた
byKohei Tokunaga
 
PDF
OCIv2？！軽量高速なイケてる次世代イメージ仕様の最新動向を抑えよう！
byKohei Tokunaga
 
PDF
コンテナ未経験新人が学ぶコンテナ技術入門
byKohei Tokunaga
 
PDF
OCIランタイムの筆頭「runc」を俯瞰する
byKohei Tokunaga
 
PDF
5分で振り返るKubeCon EU 2019：ランタイムとイメージの話題ダイジェスト
byKohei Tokunaga
 
PDF
Stargz Snapshotter: イメージのpullを省略してcontainerdでコンテナを高速に起動する
byKohei Tokunaga
 
Dockerからcontainerdへの移行
byKohei Tokunaga
 
今話題のいろいろなコンテナランタイムを比較してみた
byKohei Tokunaga
 
OCIv2？！軽量高速なイケてる次世代イメージ仕様の最新動向を抑えよう！
byKohei Tokunaga
 
コンテナ未経験新人が学ぶコンテナ技術入門
byKohei Tokunaga
 
OCIランタイムの筆頭「runc」を俯瞰する
byKohei Tokunaga
 
5分で振り返るKubeCon EU 2019：ランタイムとイメージの話題ダイジェスト
byKohei Tokunaga
 
Stargz Snapshotter: イメージのpullを省略してcontainerdでコンテナを高速に起動する
byKohei Tokunaga
 

Recently uploaded

PDF
Top Django Questions for Preparation .pdf
bysenseacademy2
 
PPTX
How John started to like TDD (instead of hating it) (Devcon, October'25)
byNacho Cougil
 
PDF
Google Developer Groups on Campus CSUEB: Intro to Google Agent Development Kit
byugoti
 
PDF
Hiring Dedicated Laravel Developers: Cost Breakdown
byShiv Technolabs Pvt. Ltd.
 
PDF
White-Label Development vs In-House Tech Teams: Key Differences
byShiv Technolabs Pvt. Ltd.
 
DOCX
From Text to Tune: How AI Music Generators Are Changing the Way We Create Songs
bycooperation2
 
PPTX
Intoduction_to_Maven best java tool for project management
byparasbramhankar02
 
PDF
How to Get Premium Capacity Power BI.pdf
byinfodobbsdataconsult
 
PDF
AI-Powered Alert Analysis with ClickHouse Cloud Databases.pdf
byAlkin Tezuysal
 
PDF
Build Vs. Buy: Cloud Cost Management and FinOps
byAmnic
 
PPTX
Lead Scoring Models in the Wild: What Works & What Doesn’t
bybbedford2
 
PDF
Cost to Develop a Careem Clone App in the UAE
byMobility Infotech
 
PPTX
PharmaAI techinical architecture and data flows
byiliyapav1999
 
PPTX
College Management System.pptx
bygehlotyash2005
 
PDF
The future Android App Development in 2025
byLoudOwls Solutions Private Limited
 
PDF
"SiyanoAV: Preventing Data Breaches in 2025"
byyogeshchauhanoffice
 
PDF
Top 10 Ways AI Can Improve SEO Strategies in 2025.pdf
bybpractseo
 
PDF
School-Hall-Pass-System-Modernizing-Student-Movement-Management.pdf
byinfoswipesolutionsin
 
PDF
Insurance-CRM-Software-The-Key-to-Digital-Transformation.pdf
byInsurance Tech Services
 
PPTX
EMR software | Best EMR software | EasyClinic
byEasy Clinic
 
Top Django Questions for Preparation .pdf
bysenseacademy2
 
How John started to like TDD (instead of hating it) (Devcon, October'25)
byNacho Cougil
 
Google Developer Groups on Campus CSUEB: Intro to Google Agent Development Kit
byugoti
 
Hiring Dedicated Laravel Developers: Cost Breakdown
byShiv Technolabs Pvt. Ltd.
 
White-Label Development vs In-House Tech Teams: Key Differences
byShiv Technolabs Pvt. Ltd.
 
From Text to Tune: How AI Music Generators Are Changing the Way We Create Songs
bycooperation2
 
Intoduction_to_Maven best java tool for project management
byparasbramhankar02
 
How to Get Premium Capacity Power BI.pdf
byinfodobbsdataconsult
 
AI-Powered Alert Analysis with ClickHouse Cloud Databases.pdf
byAlkin Tezuysal
 
Build Vs. Buy: Cloud Cost Management and FinOps
byAmnic
 
Lead Scoring Models in the Wild: What Works & What Doesn’t
bybbedford2
 
Cost to Develop a Careem Clone App in the UAE
byMobility Infotech
 
PharmaAI techinical architecture and data flows
byiliyapav1999
 
College Management System.pptx
bygehlotyash2005
 
The future Android App Development in 2025
byLoudOwls Solutions Private Limited
 
"SiyanoAV: Preventing Data Breaches in 2025"
byyogeshchauhanoffice
 
Top 10 Ways AI Can Improve SEO Strategies in 2025.pdf
bybpractseo
 
School-Hall-Pass-System-Modernizing-Student-Movement-Management.pdf
byinfoswipesolutionsin
 
Insurance-CRM-Software-The-Key-to-Digital-Transformation.pdf
byInsurance Tech Services
 
EMR software | Best EMR software | EasyClinic
byEasy Clinic
 

Faster Container Image Distribution on a Variety of Tools with Lazy Pulling

  • 1.
    Kohei Tokunaga, NTTCorporation Tao Peng, Ant Group Faster Container Image Distribution on a Variety of Tools with Lazy Pulling
  • 2.
    Pull is time-consuming pullingpackages accounts for 76% of container start time, but only 6.4% of that data is read [Harter et al. 2016] [Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribution with Lazy Docker Containers". 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA ● Large images speeds down cold start of containers ● Build takes long if base images are large ● Not all images are minimizable e.g. language runtimes, frameworks, ..
  • 3.
    Problem on thecurrent OCI image bin/bash bin/ls etc/passwd etc/group usr/bin/apt ・・・ Image is a set of tar (+compression) layers container can’t start before the all layers become locally available ● Non seekable • Need to scan the entire blob even for extracting single file entry ● No parallel extraction • Need to scan sequentially
  • 4.
    OCI-alternative accelerated imagesand lazy pulling Lazy pulling: Starting up containers without waiting for the pull completion ● eStargz • Lazy pullable format with prefetch optimization + content verification • Proposed as a backward-compatible extension to OCI Image Spec ● Nydus • Lazy pullable format with prefetch, chunk dedup and e2e data integrity • Compatible with OCI Distribution Spec and Artifacts Spec • Proposed as OCI “v2” format (incompatible to current OCI Image Spec)
  • 5.
  • 6.
    eStargz: Standard-compatible lazypulling ● 100% OCI-compatible • Lazy pullable from standard registries (ghcr.io, docker.io, …etc) • Even legacy (lazy-pulling-agnostic) runtime can run eStargz ● Usable with a variety of tools (tracker https://github.com/containerd/stargz-snapshotter/issues/258) • Kubernetes, k3s, containerd, CRI-O, Podman, BuildKit, Kaniko, ko, buildpacks.io, go-containerregistry… ● Performance optimization and content verification • Important files can be prefetched to avoid NW overhead • Each chunk comes with checksum Standard Container Registry eStargz-aware runtimes BuildKit Lazy pull Lazy pull legacy ctr-remote nerdctl Kaniko Image builders/convertes eStargz creates eStargz still runnable on eStargz-agnostic tools (e.g. Docker) etc… etc… legacy
  • 7.
    eStargz image layerformat eStargz bin/ls usr/bin/apt ・・・ ・・・ entrypoint.sh bin/bash Prioritized files Prefetched by a single Range Request TOC(Table Of Contents) and footer Files metadata, payload offset, etc… Files fetched on demand Downloaded in background as well gzip member per regular file/chunk can be extracted per-file using HTTP Range Request stargz bin/ls usr/bin/apt ・・・ ・・・ entrypoint.sh bin/bash For more details: https://github.com/containerd/stargz-snapshotter/blob/master/docs/stargz-estargz.md ● Discussed in Stargz Snapshotter of containerd: https://github.com/containerd/stargz-snapshotter ● Compatible to gzip = usable as a valid OCI/Docker image layer ● Based on stargz by CRFS (https://github.com/google/crfs) • eStargz comes with performance optimization and content verification
  • 8.
    Benchmarking result [sec] Waits forprefetch completion during create ● Method based on Hello Bench [Harter, et al. 2016] ● Takes 95 percentile of 100 operations ● Host: EC2 Oregon (m5.2xlarge, Ubuntu 20.04) ● Registry: GitHub Container Registry (ghcr.io) ● Runtime: containerd ● Stargz Snapshotter commit: 7f45f7438617728dd06bc9853afb5e42c1d3d5a3
  • 9.
    Benchmarking result ● Benchmarkingspec is the same as the previous slide
  • 10.
    Lazy pulling onKubernetes Containerd Node Stargz Snapshotter external process gRPC API kubelet Lazy pull Container Registry eStargz CRI-O Node Stargz Store external process exposed on dir kubelet ● CRI runtimes + plugins (discussed later) enable lazy pulling on Kubernetes • Containerd + Stargz Snapshotter • CRI-O + Stargz Store ● Real-world use-case at CERN for speeding up analysis pipeline [1] (13x faster pull for 5GB image) [1] Ricardo Rocha & Spyridon Trigazis, CERN. “Speeding Up Analysis Pipelines with Remote Container Images”. KubeCon+CloudNativeCon 2020 NA. https://sched.co/ekDj $ kind create cluster --name estargz-demo --image ghcr.io/stargz-containers/estargz-kind-node:0.8.0 ● Lazy-pull-enabled KinD image is available on ghcr.io/stargz-containers repo ● k3s supports lazy pulling of eStargz (merged to the main, will be included in k3s v1.22) $ k3s server --snapshotter=stargz
  • 11.
    proc Lazy pulling oncontainerd Stargz Snapshotter Fetches files/chunks on demand Mounts rootfs as FUSE ● Stargz Snapshotter plugin : https://github.com/containerd/stargz-snapshotter • Implements remote snapshotter plugin interface ● nerdctl (Docker-compatible CLI for containerd) supports lazy pulling • https://github.com/containerd/nerdctl $ nerdctl --snapshotter=stargz run ghcr.io/stargz-containers/python:3.9-esgz $ nerdctl --snapshotter=stargz compose -f docker-compose.stargz.yaml up Node Lazy pull Container Registry eStargz container
  • 12.
    proc container Lazy pulling onPodman/CRI-O ● Stargz Store plugin : https://github.com/containerd/stargz-snapshotter • Developped in Stargz Snapshotter project • Implements Additional Layer Store plugin • Podman >= v3.3.0, CRI-O >= v1.22.0 Stargz Store Provides layers (mounted as FUSE) Podman, CRI-O Uses layers for rootfs $ podman run ghcr.io/stargz-containers/python:3.9-esgz Node Lazy pull Container Registry eStargz
  • 13.
    Lazy pulling onBuildKit FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev COPY ./hello.go /hello.go RUN go build -o hello /hello.go COPY and RUN without waiting for the pull completion • /usr/local/go/bin/go • /usr/local/go/src/fmt/… etc... Fetch files/chunks on demand Build on node ● Experimentally supports lazy pulling of base images since v0.8.0 ● Can shorten the time of build that require pull • e.g. on temporary (and fresh) CI instances Lazy pull Container Registry eStargz golang:1.15.3-esgz More details at blog: https://medium.com/nttlabs/buildkit-lazypull-66c37690963f $ docker buildx create --use --name lazy-builder --driver docker-container --driver-opt image=moby/buildkit:master --buildkitd-flags '--oci-worker-snapshotter=stargz' $ docker buildx inspect --bootstrap lazy-builder $ docker buildx build .
  • 14.
    Building eStargz BuildKit: https://github.com/moby/buildkit ●Building eStargz supported on the main branch, hope to come in v0.10.x ● eStargz is supported by Buildx or standalone BuildKit (buildctl + buildkitd) ● Docs: https://github.com/moby/buildkit/blob/master/docs/stargz-estargz.md#creating-stargzestargz-images $ docker buildx build -o type=registry,name=ktokunaga/hello:esgz,oci-mediatypes=true,compression=estargz . Kaniko: https://github.com/GoogleContainerTools/kaniko ● Image builder runnable in containers and Kubernetes ● Requires GGCR_EXPERIMENT_ESTARGZ=1 ● Base images need to be eStargz $ docker run --rm -e GGCR_EXPERIMENT_ESTARGZ=1 -v /tmp/context:/workspace -v ~/.docker/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.6.0 --destination "ghcr.io/ktock/sample:esgz"
  • 15.
    Converting an imageinto eStargz ctr-remote: https://github.com/containerd/stargz-snapshotter/blob/v0.7.0/docs/ctr-remote.md ● CLI for containerd provided by Stargz Snapshotter project ● Supports prefetch optimization for eStargz $ ctr-remote image optimize --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz go-containerregistry and crane CLI : https://github.com/google/go-containerregistry ● Library and CLI to interact with registries ● requires GGCR_EXPERIMENT_ESTARGZ=1 ● downstream tools (e.g.Kaniko, ko and buildpacks.io) supports eStargz creation as well $ go install github.com/google/go-containerregistry/cmd/crane@latest $ GGCR_EXPERIMENT_ESTARGZ=1 crane optimize ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz nerdctl: https://github.com/containerd/nerdctl/blob/v0.11.1/docs/stargz.md ● Docker-compatible CLI for containerd ● Can be combined with nerdctl build command $ nerdctl image convert --estargz --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
  • 16.
    Discussion toward OCIv1extension ● eStargz is now widely usable on tools in the community ● Proposed to OCI Image Spec (discussion is on-going) ● Proposed as backward-compatible extensions • Extension for the current gzip layer ( application/vnd.oci.image.layer.v1.tar+gz ) • Additional annotation for content verification: org.opencontainers.image.toc.digest https://github.com/opencontainers/image-spec/issues/815
  • 17.
    Nydus Image Service TaoPeng, Ant Group
  • 18.
    Nydus: Improved LazyPulling and More ● Standard lazy pulling with prefetch policies • Lazy pullable from standard registries (cloud registries, docker registry, harbor, etc) • Semantical prefetch, hinted prefetch, background prefetch all data ● Chunk level deduplication with layered build cache • Speed up both image conversion and downloading ● End-to-end data integrity • Runtime data integrity check ● Reproducible Image Building • Build environment agnostic ● Compatible with OCI Distribution Spec and Artifacts Spec • Usable with most existing container registry implementations ● Rich container ecosystem integration • Kubernetes, docker, containerd, buildkit, harbor, dragonfly, runc, kata-containers etc ● Resource efficient and production ready • Large scale deployment at Ant and available via Alibaba Cloud services
  • 19.
    FUSE/virtiofs Unified Architecture ●Native container runtime support • One translation layer (FUSE or virtiofs) for both runc and kata containers ● Shared uncompressed local cache • Download/uncompress once and use all the time ● Singleton mode with extremely low memory footprint • ~5MB per instance
  • 20.
    Registry Acceleration FileSystem ● Merkle tree metadata layer • self-verifiable ● Chunk-shared data layer • enabling chunk level deduplication
  • 21.
    OCI Spec Compatibility ●Metadata/data layers are OCI manifest layers ● New media type in the image manifest spec ● Compatible with OCI artifacts/distribution spec ● Compatible registry GC functionality ● Widely working with container registries • cloud registries • docker registry • harbor • etc.
  • 22.
  • 23.
    Lazy Pulling withVariety of Tools ● Nydus Snapshotter plugin • https://github.com/dragonflyoss/image-service/tree/master/contrib/nydus-sna pshotter ● Usable for variety of container management tools • kubernetes (kubelet) • crictl • nerdctl • ctr-remote
  • 24.
    Manual Image Conversion-- nydusify ● nydusify • https://github.com/dragonflyoss/image-service/tree/master/contrib/nydusify • download image from remote registry, convert it to nydus image, and push nydus image to remote registry • nydusify convert --source <registry>/<repo>:<tag> --target <registry>/<repo>:<nydus-tag>
  • 25.
    Manual Image Conversion-- buildkit ● buildkit • https://github.com/moby/buildkit/pull/2045 • build nydus image directly from dockerfile and push it to remote registry • buildctl build --frontend=dockerfile.v0 --local context=/build/dir --local dockerfile=/build/DockerfileDir --output type=nydus,name=<registry>/<repo>:<tag>
  • 26.
    Automatic Conversion withHarbor ● https://github.com/goharbor/community/pull/167 ● New image conversion service in harbor ● Started image acceleration working group ● Image format agnostic • nydus • estargz • future more format
  • 27.
    OCI Artifacts Manifest ●https://github.com/opencontainers/artifacts/pull/29 ● Map OCI image manifest to artifact manifest, to connect artifacts to images ● Beneficial to SBOM, Signatures, Nydus, Scan Results etc. ● Nydus image as an artifact type • cncf.nydus.v1-rc1
  • 28.
    Community V2 ImageSpec Requirements ● OCI community requirements on V2 Image format • https://hackmd.io/@cyphar/ociv2-brainstorm
  • 29.
    Recap ● OCI imagesare large and slow ● Image lazy pulling is important for starting containers quickly ● Prefetching is a good friend to lazy pulling ● Ecosystem adoption is crucial for new image formats ● eStargz • Backward compatibility • Extension to the existing OCI image spec ● Nydus • Future looking • Proposal to the next generation OCI image spec
  • 30.
    eStargz Project Information ●100% OCI-compatible image format for lazy pulling ● Subproject of CNCF graduated containerd ● github： https://github.com/containerd/stargz-snapshotter ● slack： https://slack.containerd.io/ (#containerd-dev channel at CNCF slack) ● Pre-converted images are available on ghcr.io/stargz-containers
  • 31.
    Nydus Project Information ●Image acceleration service with various improvements ● CNCF incubator dragonfly sub-project ● github： https://github.com/dragonflyoss/image-service ● slack： https://tinyurl.com/nydus-slack ● tutorial： https://tinyurl.com/nydus-tutorial
  • 32.