Introduction
Previously, I had mastered the basics of Ansible as infrastructure configuration management tool as part of my self-learning DevOps journey. Today, I had to understand and try out what is a container orchestration tool. I started with Kubernetes as it was more popular than Docker Swarm but it has a very steep learning curve, way too difficult for me. Then, I switched to Docker Swarm and had made good progress though the results are not satisfactory because I failed to deploy full working services on the Worker node. In this tutorial, I will share how to initiate a Docker Swarm, deploy a stack to a swarm and some useful commands for troubleshooting purposes. While this tutorial is not complete, it does serves as a checkpoint for me to recap what I had learnt so far and to continue where I left off on another day.
Prerequisites
- Single server to act as the Manager node.
- Second server to act as the Worker node.
- Basic understanding of Docker containers and Docker Compose.
- Patience and tons of it because it is not as straightforward as Ansible’s concept of inventory and playbooks.
Step 1 – Open Ports for Manager and Worker Nodes
Manager and Worker nodes communicate with one another via ports 7946 (TCP/UDP) for container network discovery and 4789 (UDP) for overlay network traffic respectively. Mangers use an additional port 2377 (TCP) for cluster management communications.
# Swarm Managers
$ sudo iptables -A INPUT -p tcp --dport 2377 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p tcp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 4789 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
# Swarm Workers
$ sudo iptables -A INPUT -p tcp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 4789 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPTStep 2 – Initialize a Docker Swarm
To begin, we go to our Manager server and type docker swarm init –advertise-addr <IP> to create a swarm. Replace <IP> with the server IP address. A docker swarm join command containing an unique token will be generated for us to apply on Worker nodes.
$ docker swarm init --advertise-addr <IP>
Swarm initialized: current node (quhedtbhc1v10xuu5ggc5unsv) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-204js6i8sey05qkmcf6askfvwmpyxq3dd76qlrr4zgynholyok-4uqwmi8t2zsnrpcfnszdm7ukk <IP>:2377Next, we log in to the other servers and join them as Worker nodes with the docker swarm join command.
$ docker swarm join --token SWMTKN-1-204js6i8sey05qkmcf6askfvwmpyxq3dd76qlrr4zgynholyok-4uqwmi8t2zsnrpcfnszdm7ukk <IP>:2377
This node joined a swarm as a worker.Going back to the Manger node, we use docker node ls to check all servers associated with the swarm. It will list down all the host-name, status and availability information etc.
Step 3 – Deploy an App to Swarm
We use docker stack deploy to accept a file in Docker Compose format. It is important that we use version 3 of compose files, as docker stack deploy won’t support use of earlier versions. Do note that below is an incomplete compose file such that both “db” and “wordpress” services will get deployed to the Worker node as intended but the app, WordPress in this case, will fail to load in the browser.
version: '3.7'
services:
db:
image: mysql:8.0.19
networks:
- backend
environment:
- MYSQL_ROOT_PASSWORD=secret
volumes:
- dbdata:/var/lib/mysql
deploy:
restart_policy:
condition: on-failure
placement:
constraints: [node.role == worker]
wordpress:
image: wordpress:latest
depends_on:
- db
networks:
- backend
environment:
- WORDPRESS_DB_HOST=db:3306
- WORDPRESS_DB_USER=wpuser
- WORDPRESS_DB_PASSWORD=wpsecret
- WORDPRESS_DB_NAME=wordpress
volumes:
- wordpress_files:/var/www/html
ports:
- 8080:80
deploy:
#replicas: 2 # Only if required > 1 replica
update_config:
parallelism: 2
delay: 10s
restart_policy:
condition: on-failure
placement:
constraints: [node.role == worker]
volumes:
dbdata:
wordpress_files:
networks:
backend:There are some Docker Compose stuffs that will not work when deploying a stack to a swarm. For example, we cannot use container_name as it is not supported. As for restart: option, it is now moved to restart_policy under deploy: section. To deploy these services, we will use docker stack deploy –compose-file docker-stack.yml wordpress which will create both MySQL and WordPress containers at the Worker node because we had specifically state constraints: [node.role == worker]. If we do a docker ps at the Worker node after two to three minutes, we will see these two containers.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39ec0b5736d8 mysql:8.0.19 "docker-entrypoint.s…" 38 seconds ago Up 33 seconds 3306/tcp, 33060/tcp wordpress_db.1.4yglt1ghm8nd55tf4q9sj09g4
7b6b0c4ec09f wordpress:5.4.2-php7.4-apache "docker-entrypoint.s…" 39 seconds ago Up 31 seconds 80/tcp wordpress_wordpress.1.p78n4s3pu6nroq9m67v2r2u7rTroubleshooting
Sometimes, certain services may keep restarting due to some errors. We can always check the logs at the Manager node with docker stack ps wordpress –no-trunc.
$ docker stack ps wordpress --no-trunc
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
qied9o5xsstoeduq0bgwyex4m wordpress_wordpress.1 wordpress:latest@sha256:095e933a48619e3dfb11f51b11fbe2ebdd0e2f827f1a38c556c80f635c8baf72 devops1 Running Running 7 minutes ago
thbb91vb9ioxowlp73yutimqm wordpress_db.1 mysql:8.0.19@sha256:9643e9fbd6330d10686f8922292dcb20995e7b792c17d4e94ddf95255f1d5449 devops1 Running Running 14 minutes agoGoing into individual service logs will be docker service logs <ID> where <ID> is the container ID, for example, qied9o5xsstoeduq0bgwyex4m.
$ docker service logs qied9o5xsstoeduq0bgwyex4m
wordpress_wordpress.1.v4g2hb4g13gw@devops1 | MySQL Connection Error: (2002) php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution
wordpress_wordpress.1.v4g2hb4g13gw@devops1 | [23-Jul-2020 15:38:53 UTC] PHP Warning: mysqli::__construct(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in Standard input code on line 22
wordpress_wordpress.1.v4g2hb4g13gw@devops1 | [23-Jul-2020 15:38:53 UTC] PHP Warning: mysqli::__construct(): (HY000/2002): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in Standard input code on line 22
wordpress_wordpress.1.v4g2hb4g13gw@devops1 |
wordpress_wordpress.1.v4g2hb4g13gw@devops1 | MySQL Connection Error: (2002) php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution
wordpress_wordpress.1.v4g2hb4g13gw@devops1 |
wordpress_wordpress.1.v4g2hb4g13gw@devops1 | WARNING: unable to establish a database connection to 'db:3306'
wordpress_wordpress.1.v4g2hb4g13gw@devops1 | continuing anyways (which might have unexpected results)After we had fixed the errors from the service logs, we may want to force restart the service containers using docker service update –force <NAME> if Docker Swarm does not do it for us.
$ docker service update --force wordpress_db
wordpress_db
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
$ docker service update --force wordpress_wordpress
wordpress_wordpress
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service convergedConclusion
Docker Swarm is not as simple as I thought it would be like Ansible. There are some restrictions in the compose file due to the version 3 requirements and online help are still mostly based on version 2. Unfortunately, the WordPress installation page did not load on the browser as intended even though the MySQL and WordPress containers were successfully deployed to the Worker node. There is a voting app example using Docker Swarm at https://github.com/dockersamples/example-voting-app but I have yet to try it out as I was too focused and determined to correct the php_network_getaddresses: getaddrinfo failed: error with WordPress instead. Thus, this marks my milestone for learning a container orchestration tool.