Bitbucket Pipelines - Powered by Kubernetes
Download as PPTX, PDF1 like1,165 views
The document provides an overview of Bitbucket Pipelines, describing its integration with Kubernetes and AWS for running customer builds in a multi-tenanted environment. It includes technical details on deployment configurations, security best practices, and recommendations for optimizing cluster performance. Key tips are outlined for managing API and etcd instances, implementing network policies, and securing the kubelet.
Bitbucket Pipelines - Powered by Kubernetes
- 1. Bitbucket Pipelines Powered by Kubernetes Nathan Burrell - Senior Developer - Atlassian
- 7. Pipelines Use Cases Remote Code Execution Run any and all code, any docker image. Run our own Infrastructure Atlassian uses AWS exclusively so we have to run kubernetes cluster. Multi-tenanted We run all customer builds in the same cluster, with different customers pods on the same machine. “Short” lived batch jobs. Run pods that have lifetimes of minutes to hours.
- 13. apiVersion: v1 kind: Secret metadata: name: environment-variables type: Opaque data: username: c29tZXRoaW5nCg== password: c2VjdXJlCg==
- 14. pipelines: default: - step: image: name: private/image:latest username: $USERNAME Password: $PASSWORD script: ...
- 15. apiVersion: v1 kind: Secret metadata: name: build-image-docker- credentials type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: eyAiYXV0aHMiOiB7ICJodHRwczovL2l uZGV4LmRvY2tlci5pby92MS8iOiB7IC JhdXRoIjogIlpHOWphMlZ5T214dloyb HVDZz09IiB9IH0gfQo=
- 17. options: docker: true definitions: services: redis: redis:latest pipelines: default: - step: services: - redis image: my/image:latest script: ...
- 18. apiVersion: v1 kind: Pod spec: containers: - name: build image: my/image:latest - name: service-redis image: redis:latest - name: system-docker image: pipelines/docker:1.0 securityContext: - privileged: true
- 19. Pod agent clone builddocker serviceserviceservice volumes ● tmp ● scripts ● containers ● empty ● /dev/shm 4gb ram resource requirement privileged with a custom docker auth plugin
- 20. Deployments Deployment Set for Node Scaler.
- 21. Custom Node Scaler Autoscaling Group Node kubernetes-master node-scaler-pod 1. Query number of “step” pods currently in cluster 2. Set capacity as required
- 22. Daemon Sets Daemon Sets for Helpers.
- 23. https://docs.docker.com/registry/recipes/mirror/ Running an image cache. Node docker registry-pod Node docker registry-pod docker-registry S3 1. Try Pull from cache 2. Fallback to registry 3. Store public layers
- 24. https://github.com/fluent/fluentd-kubernetes-daemonset Collecting logs. Node fluentd splunk Scans the docker container directory and parses/enriches container logs Uploads logs
- 25. https://docs.datadoghq.com/integrations/kubernetes/ Collecting metrics. Node datadog-agent datadog docker Collect node metricsCollect container metrics Publish
- 27. The Boring Stuff Everything is on AWS Atlassian is going BIG on AWS for its cloud infrastructure Ansible + Cloudformation Is the secret sauce to having immutable Infrastructure Container Linux as a base Container Linux from CoreOS gives us a secure well tested base to build upon
- 28. Tip #1 Size your API instances correctly.
- 29. https://kubernetes.io/docs/admin/cluster-large Number of Nodes Instance Recommendation AWS gcloud 1-5 m3.medium n1-standard-1 6-11 m3.large n1-standard-2 11-100 m3.xlarge n1-standard-4
- 30. Various parts of the kubernetes codebase “ ~60mb of capacity per node
- 31. Tip #2 Size your etcd instances correctly.
- 32. https://coreos.com/etcd/docs/latest/op-guide/hardware.html Cluster Requirement Instance Recommendation AWS gcloud 100 Clients, 200 RQPS, 100MB Data m4.large n1-standard-1 500 Clients, 1000 RQPS, 500MB Data m4.xlarge n1-standard-4 1500 Clients, 10000 RQPS, 1GB Data m4.2xlarge n1-standard-8
- 33. Tip #3 Split etcd into role based clusters.
- 34. Control Plane VPC Worker Plane VPC Node flannel kubernetes-etcd flannel-etcd peering master
- 35. Tip #4 Use network policies.
- 36. https://github.com/projectcalico/canal/blob/master/InstallGuide.md Node flannel calico etcd master apiVersion: v1 kind: Namespace metadata: name: network-policy annotations: net.beta.kubernetes.io/network-policy: '{"ingress": {"isolation": "DefaultDeny"}}' Monitors for changes in network policies and applies new iptable rules
- 37. Tip #5 Hide default secrets.
- 38. apiVersion: v1 kind: Pod metadata: name: hide-secrets spec: volumes: - hidden containers: - name: potentially-malicious image: potentially/malicous:image volumeMounts: - name: hidden mountPath: /var/run/secrets/kubernetes.io/serviceaccount readOnly:true
- 39. Tip #6 Secure the kubelet.
- 40. https://kubernetes.io/docs/admin/kubelet/ Node kubelet: 10255 pod Disable the read only port. --read-only-port=0 (default: 10255) No credentials required
- 41. https://kubernetes.io/docs/admin/kubelet-authentication- Node kubelet: 10250 pod Enable authentication on kubelet port. --anonymous-auth=false (default: true) --authentication-token-webhook=true (default: false) No credentials required
- 42. Tip #7 KubeDNS Can Be Unreliable.
- 44. apiVersion: v1 kind: Pod metadata: name: hide-secrets spec: dnsPolicy: Default
- 46. Thank You Nathan Burrell - Senior Developer - Atlassian
Editor's Notes
- #29: One/Two of your masters will always consume more CPU/Memory due to scheduler/controller leader election. Consider splitting API servers into responsibility based instances.
- #30: One/Two of your masters will always consume more CPU/Memory due to scheduler/controller leader election. Consider splitting API servers into responsibility based instances.
- #31: One/Two of your masters will always consume more CPU/Memory due to scheduler/controller leader election. Consider splitting API servers into responsibility based instances.
- #32: Losing API instances is one thing losing etcd is a nightmare! Requires full cluster recreation once the backing store is lost.
- #33: Losing API instances is one thing losing etcd is a nightmare! Requires full cluster recreation once the backing store is lost.
- #34: Split your etcd instance into 2 clusters: One for the kubernetes API backing store One for the flannel backing store Only expose what you need to (principal of least privilege)
- #35: Split your etcd instance into 2 clusters: One for the kubernetes API backing store One for the flannel backing store Only expose what you need to (principal of least privilege)
- #36: Kubernetes by default uses flannel as its overlay network allowing for pod - pod communication regardless of host scheduling. Out of the box this allows any pod to talk to any other pod in the cluster (provided they know the ip addresses and ports of the services the pod exposes). Secure this with software defined rules using calico and network policies.
- #37: Kubernetes by default uses flannel as its overlay network allowing for pod - pod communication regardless of host scheduling. Out of the box this allows any pod to talk to any other pod in the cluster (provided they know the ip addresses and ports of the services the pod exposes). Secure this with software defined rules using calico and network policies.
- #38: Kubernetes is nice enough to mount into every container a default secret directory containing the certificate authority and token to allow you to authenticate with the api-servers as the namespace you are running in.
- #39: Kubernetes is nice enough to mount into every container a default secret directory containing the certificate authority and token to allow you to authenticate with the api-servers as the namespace you are running in.
- #40: By default the kubelet binds to two ports on a node that allow requests against the API also the default configurations are insecure.
- #41: By default the kubelet binds to two ports on a node that allow requests against the API also the default configurations are insecure.
- #42: By default the kubelet binds to two ports on a node that allow requests against the API also the default configurations are insecure.
- #43: KubeDNS performs well enough for low load and low volumes of requests but doesnt perform well under load.
- #44: KubeDNS performs well enough for low load and low volumes of requests but doesnt perform well under load.
- #45: KubeDNS performs well enough for low load and low volumes of requests but doesnt perform well under load.