Publishing Kubernetes packages on community infrastructure

[justaugustus]

Enhancement Description

• One-line enhancement description (can be used as a release note): Publishing Kubernetes packages on community infrastructure
• Kubernetes Enhancement Proposal: keps/sig-release/1731-publishing-packages/README.md
• Discussion Link: https://docs.google.com/spreadsheets/d/1Vgpb7HPg8WgoFtDuddaRD6j7jQLoqNK6vUu6Lb0ea28
• Primary contact (assignee): @justaugustus @saschagrunert
• Responsible SIGs: SIG Release
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): v1.28
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    Add KEP for publishing packages in provisional state #843
    KEP-1731 : Publishing Kubernetes packages on community infrastructure #3434
    KEP-1731: Extend the KEP with the OBS implementation #3750
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

---

Milestones and Tasks Checklist

Milestone 1.0—Code Deliverable

• Success looks like: debs and rpms are in multiple locations (Google infra and third-party hosting service(s)) @detiber
  • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.)
  • Describe proposed Flow (what are key actions that the user will take? How will they take them?)
  • Choose the third-party hosting service/Set up a package host of some sort @detiber, @leonardpahlke helping
    • Identified some options to explore
    • Outreach to reps of those options to see if they'll work with us on this WIP
  • Do a spike test to see if the third-party hosting service meets our needs @detiber, @leonardpahlke helping
  • Publish the debs/rpms to the third-party hosting service @detiber

Milestone 1.0—Documentation Deliverable

• Success looks like: Entire deb/rpms move to community infra plan is outlined in a KEP
  • Update this KEP: @RobertKielty (made a PR July 15, 2022)
  • Review the KEP @saschagrunert, @ameukam, @detiber
  • Approve the KEP
  • Get an understanding of the top-level script: what type of infra we're going to run, who pays for this, who can access to this, what type of account we have, tools available @RobertKielty
    - Script reviewed by Robert Kielty and discussed on SIG Release channel with Ben Elder. This KEP will drive outstanding infra/billing questions for the new implementation.
  • Clarify what "top-level script" means (linked in cell above) @RobertKielty
  • Finalize defining the user personas for this work, with group review and sign-off @detiber, @LappleApple

Milestone 1.0—Risk Mitigation

• Success looks like: All risk matters for this milestone are accounted for, plans in place to mitigate
  • Risk: A new GPG key will be issued incompatible with the existing GPG key managed by Google. This is a breaking change for the existing infrastructures using DEB/RPM packages as installation artifacts

Milestone 1.0—Questions resolved

• Resolve if this is needed: Give access to the sandbox env to release eng to work on it @ameukam, @onlydole helping
• Develop sandbox env (GCP project) and give access to Rel Eng to work on it @ameukam, @onlydole helping
• Needed more discussion: Meeting to specifically discuss if we are doing GPG or if we are owning the packages; then who owns new GPG keys and how and when we store them. @ameukam, @onlydole helping
• Generate new keys : Sign kubernetes system packages with GPG release#2627
• Decide on plan: Be able to issue new gcp trusted keys for entire ecosystem/whoever is relying on those packages to deploy/install/upgrade K8s project
• Find/line up people who can build packages for debian, CentOS to work on build tooling (@upodroid to help build Debian packages)

Milestone 2.0—Code deliverable

• Success looks like: implementation (source code) in our repositories, such that we can publish releases by ourselves without having to rely on Google people at all
  • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.
  • Proposed Flow (what are key actions that the user will take? How will they take them?)
  • Refine the current debs/rpms built by kubepkg
  • Resume discussing how we rethink / rewrite the build tools
  • Rebase the existing kubepkg work into something krel can use directly
  • Krel, the kubernetes release tool, should start including a package build step running the build scripts
  • Phase out publishing to the google controlled host, for having krel publish to our new host for 1-2 releases
  • Users migrate from the Google registry to the community registry, adding the signing key

Milestone 2.0—Documentation Deliverable

• Success looks like: we've published docs on how process works and when it's not working as expected, what to do (for release managers)
  • Expand documentation on the top-level script to drive knowledge transfer
  • Expand documentation on the package build scripts for Debian and RPM to drive knowledge transfer

Milestone 2.0—Questions resolved

• Success looks like: @saschagrunert and @justaugustus have provided context and knowledge about the package build scripts for Debian and rpms
  • Answer: what type of infra we're going to run
  • Answer: who pays for it
  • Answer: what type of account do we have
  • Answer: what tools do/will we have available

Milestone 3.0—Documentation deliverable

• Success looks like: We've updated the google build+publish+release script to download packages uploaded by the upstream release process

Milestone 4.0— Documentation Deliverable

• Success looks like: We've published docs for end users about old infra and new/migrated (more static)
  • Write email to community regarding the required migration to the community repository
  • Send email to community regarding the required migration to the community repository
  • SIG Release informs the community so we don't break people over just one milestone: "these are the keys we'll use to release 1.2X..."

---

Why is this needed?

• It's part of the effort for the community to fully own the infrastructure related to the Kubernetes project.
• Ensure all aspects of project releases can be staffed across companies
• We don’t want to be dependent on a Build Admin any longer (which tends to slow releases)
• We seek more control to eventually choose to build packages for prereleases, extend what packages are shipped, etc.

Why is it needed now?

• The existing infrastructure of this relies on a workstation inside Google offices—essentially one person. It’s not reliable or sustainable.

Who needs it? (User Personas WIP)

• Google Build Team as the existing team building the system packages.
• What does “done” look like/acceptance criteria?
• Automated builds of deb and rpm Kubernetes packages within community infrastructure

Related open issues

• #913

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.