From docker to kubernetes: running Apache Hadoop in a cloud native way
4 likes1,586 views
Creating containers for an application is easy (even if it’s a goold old distributed application like Apache Hadoop), just a few steps of packaging. The hard part isn't packaging: it's deploying How can we run the containers together? How to configure them? How do the services in the containers find and talk to each other? How do you deploy and manage clusters with hundred of nodes? Modern cloud native tools like Kubernetes or Consul/Nomad could help a lot but they could be used in different way. It this presentation I will demonstrate multiple solutions to manage containerized clusters with different cloud-native tools including kubernetes, and docker-swarm/compose. No matter which tools you use, the same questions of service discovery and configuration management arise. This talk will show the key elements needed to make that containerized cluster work. Tools: kubernetes, docker-swam, docker-compose, consul, consul-template, nomad together with: Hadoop, Yarn, Spark, Kafka, Zookeeper, Storm…. References: https://github.com/flokkr Speaker Marton Elek, Lead Software Engineer, Hortonworks
![version: "3"
services:
namenode:
image: flokkr/hadoop
hostname: namenode
command: ["hdfs","namenode"]
ports:
- 9870:9870
environment:
ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000"
HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000"
HDFS-SITE.XML_dfs.replication: "1"
datanode:
image: flokkr/hadoop
command: ["hdfs", "datanode"]
environment:
CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000"
HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000"
HDFS-SITE.XML_dfs.replication: "1"
LOG4J.PROPERTIES_log4j.rootLogger: "INFO, stdout"
LOG4J.PROPERTIES_log4j.appender.stdout: "org.apache.log4j.ConsoleAppender"
LOG4J.PROPERTIES_log4j.appender.stdout.layout: "org.apache.log4j.PatternLayout"
LOG4J.PROPERTIES_log4j.appender.stdout.layout.ConversionPattern: "%d{yyyy-MM-dd HH:mm:ss} %-5p](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-26-320.jpg)
![FROM frolvlad/alpine-oraclejdk8
ADD hadoop-3.2.0.tar.gz /opt
WORKDIR /opt/hadoop
CMD ["/opt/launcher.sh"]
DockerfileDockerfile](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-34-320.jpg)
![FROM frolvlad/alpine-oraclejdk8
ADD hadoop-3.2.0.tar.gz /opt
WORKDIR /opt/hadoop
CMD ["/opt/launcher.sh"]
DockerfileDockerfile
launcher.sh hdfs namenode
Final command executed by docker:](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-35-320.jpg)
![#!/usr/bin/env bash
if [ -n "$SLEEP_SECONDS" ]; then
echo "Sleeping for $SLEEP_SECONDS seconds"
sleep $SLEEP_SECONDS
fi
if [ -n "$ENSURE_NAMENODE_DIR" ]; then
if [ ! -d "$ENSURE_NAMENODE_DIR" ]; then
/opt/hadoop/bin/hdfs namenode -format -force $CLUSTERID_OPTS
fi
fi
"$@"](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-36-320.jpg)
![apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-105-320.jpg)
![apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]
Flexibility](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-106-320.jpg)
![apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
shareProcessNamespace: true
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]
- name: jmxpromo
image: flokkr/jmxpromo-sidecar](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-107-320.jpg)
![apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
shareProcessNamespace: true
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]
- name: jmxpromo
image: flokkr/jmxpromo-sidecar
Flexibility](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-108-320.jpg)
![apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-111-320.jpg)
![apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: ozone-hdfs-namenode
spec:
serviceName: ozone2-hdfs-namenode
replicas: 1
template:
metadata:
labels:
app: ozone
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "28942"
spec:
containers:
- name: hdfs-namenode
image: flokkr/ozone:2.1.0
args: ["hdfs","namenode"]](https://image.slidesharecdn.com/fromdockertokubernates-180625174754/85/From-docker-to-kubernetes-running-Apache-Hadoop-in-a-cloud-native-way-112-320.jpg)
More Related Content
What's hot (20)
Similar to From docker to kubernetes: running Apache Hadoop in a cloud native way (20)
More from DataWorks Summit (20)
Recently uploaded (20)
From docker to kubernetes: running Apache Hadoop in a cloud native way
- 1. k8s : h4pk8s : h4p From docker to kubernetes: Running Apache Hadoop in a cloud native way Marton Elek
- 5. RAFT library for Java
- 21. FROM frolvlad/alpine-oraclejdk8 ADD hadoop-3.2.0.tar.gz /opt WORKDIR /opt/hadoop DockerfileDockerfile
- 24. version: "3" services: service1: image: apache/imagename hostname: namenode ports: - 9870:9870 environment: CONFIGURATION1: value DFS_DIR: /dfs THREAD_NUMBER: 1
- 25. How to handle configuration?How to handle configuration? Create a simple launcher script to Create config file from environment variables Start the application
- 26. version: "3" services: namenode: image: flokkr/hadoop hostname: namenode command: ["hdfs","namenode"] ports: - 9870:9870 environment: ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name" CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000" HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000" HDFS-SITE.XML_dfs.replication: "1" datanode: image: flokkr/hadoop command: ["hdfs", "datanode"] environment: CORE-SITE.XML_fs.defaultFS: "hdfs://namenode:9000" HDFS-SITE.XML_dfs.namenode.rpc-address: "namenode:9000" HDFS-SITE.XML_dfs.replication: "1" LOG4J.PROPERTIES_log4j.rootLogger: "INFO, stdout" LOG4J.PROPERTIES_log4j.appender.stdout: "org.apache.log4j.ConsoleAppender" LOG4J.PROPERTIES_log4j.appender.stdout.layout: "org.apache.log4j.PatternLayout" LOG4J.PROPERTIES_log4j.appender.stdout.layout.ConversionPattern: "%d{yyyy-MM-dd HH:mm:ss} %-5p
- 32. Container is the unit of packaging.
- 33. Launcher script: the source of the power in containers
- 34. FROM frolvlad/alpine-oraclejdk8 ADD hadoop-3.2.0.tar.gz /opt WORKDIR /opt/hadoop CMD ["/opt/launcher.sh"] DockerfileDockerfile
- 35. FROM frolvlad/alpine-oraclejdk8 ADD hadoop-3.2.0.tar.gz /opt WORKDIR /opt/hadoop CMD ["/opt/launcher.sh"] DockerfileDockerfile launcher.sh hdfs namenode Final command executed by docker:
- 36. #!/usr/bin/env bash if [ -n "$SLEEP_SECONDS" ]; then echo "Sleeping for $SLEEP_SECONDS seconds" sleep $SLEEP_SECONDS fi if [ -n "$ENSURE_NAMENODE_DIR" ]; then if [ ! -d "$ENSURE_NAMENODE_DIR" ]; then /opt/hadoop/bin/hdfs namenode -format -force $CLUSTERID_OPTS fi fi "$@"
- 38. Launcher scriptLauncher script Create config files from ENV
- 39. Launcher scriptLauncher script Create config files from ENV Wait for the dependency (TCP check)
- 40. Launcher scriptLauncher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component
- 41. Launcher scriptLauncher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode, ...)
- 42. Launcher scriptLauncher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode, ...) Retrieve kerberos/SSL secrets
- 43. Launcher scriptLauncher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode, ...) Retrieve kerberos/SSL secrets Enable prometheus monitoring (Java agent)
- 44. Launcher scriptLauncher script Create config files from ENV Wait for the dependency (TCP check) Download additional optional component Prepare HDFS (format namenode, ...) Retrieve kerberos/SSL secrets Enable prometheus monitoring (Java agent) Show network traffic (Instrumentation with Java agent)
- 49. Hashicorp stackHashicorp stack "do it yourself""do it yourself"
- 61. Docker network
- 62. Host network
- 74. DIYDIY
- 76. Consul Yes (script) Restart host netw Nomad .nomad redeploy no yes host netw yes consul yes host
- 78. KubernetesKubernetes "out of the box""out of the box"
- 85. DaemonSet
- 86. ReplicaSet
- 90. Service Pod
- 91. StatefulSet
- 92. DaemonSet
- 96. Benefits ofBenefits of Hadoop + k8s?Hadoop + k8s?
- 97. Benefits ofBenefits of Hadoop + k8s?Hadoop + k8s? Ecosystem Flexibility
- 98. Example:Example: Monitor Hadoop with PrometheusMonitor Hadoop with Prometheus
- 102. Pod
- 103. Sidecar pattern Pod
- 104. Sidecar for monitoring over jmx
- 105. apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: serviceName: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone spec: containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"]
- 106. apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: serviceName: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone spec: containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"] Flexibility
- 107. apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: serviceName: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone spec: shareProcessNamespace: true containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"] - name: jmxpromo image: flokkr/jmxpromo-sidecar
- 108. apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: serviceName: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone spec: shareProcessNamespace: true containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"] - name: jmxpromo image: flokkr/jmxpromo-sidecar Flexibility
- 111. apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: serviceName: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone spec: containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"]
- 112. apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: ozone-hdfs-namenode spec: serviceName: ozone2-hdfs-namenode replicas: 1 template: metadata: labels: app: ozone annotations: prometheus.io/scrape: "true" prometheus.io/port: "28942" spec: containers: - name: hdfs-namenode image: flokkr/ozone:2.1.0 args: ["hdfs","namenode"]
- 114. Ecosystem
- 116. Is HadoopIs Hadoop cloud native?cloud native?
- 117. Is HadoopIs Hadoop cloud native?cloud native? Yes, with small modifications
- 118. Is HadoopIs Hadoop cloud native?cloud native? Yes, with small modifications DNS/IP handling. Shoud work without DNS with changing DNS
- 119. Is HadoopIs Hadoop cloud native?cloud native? Yes, with small modifications DNS/IP handling. Shoud work without DNS with changing DNS More flexible configuration loading
- 120. Is HadoopIs Hadoop cloud native?cloud native? Yes, with small modifications DNS/IP handling. Shoud work without DNS with changing DNS More flexible configuration loading Reverse proxy friendly UI
- 121. SummarySummary
- 122. SummarySummary Don't buy without checking the label
- 123. SummarySummary Don't buy without checking the label Containerization can help a lot to manage our Bigdata clusters
- 124. SummarySummary Don't buy without checking the label Hadoop is first class citizen of cloud-native/containerized environments* Containerization can help a lot to manage our Bigdata clusters
- 125. Q&AQ&A Márton Elek @anzix https://flokkr.github.io (bigdata + containers project) https://github.com/flokkr (source) [email protected]
- 126. Image creditsImage credits Yan Pritzker (CC) Carrie Cizauskas (CC)