Using Docker Swarm Container Orchestration to Manage Multiple Containers

Using Docker Swarm Container Orchestration to Manage Multiple Containers

Introduction

Previously, I had mastered the basics of Ansible as infrastructure configuration management tool as part of my self-learning DevOps journey. Today, I had to understand and try out what is a container orchestration tool. I started with Kubernetes as it was more popular than Docker Swarm but it has a very steep learning curve, way too difficult for me. Then, I switched to Docker Swarm and had made good progress though the results are not satisfactory because I failed to deploy full working services on the Worker node. In this tutorial, I will share how to initiate a Docker Swarm, deploy a stack to a swarm and some useful commands for troubleshooting purposes. While this tutorial is not complete, it does serves as a checkpoint for me to recap what I had learnt so far and to continue where I left off on another day.

Prerequisites

  • Single server to act as the Manager node.
  • Second server to act as the Worker node.
  • Basic understanding of Docker containers and Docker Compose.
  • Patience and tons of it because it is not as straightforward as Ansible’s concept of inventory and playbooks.

Step 1 – Open Ports for Manager and Worker Nodes

Manager and Worker nodes communicate with one another via ports 7946 (TCP/UDP) for container network discovery and 4789 (UDP) for overlay network traffic respectively. Mangers use an additional port 2377 (TCP) for cluster management communications.

# Swarm Managers
$ sudo iptables -A INPUT -p tcp --dport 2377 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p tcp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 4789 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Swarm Workers
$ sudo iptables -A INPUT -p tcp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 7946 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 4789 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

Step 2 – Initialize a Docker Swarm

To begin, we go to our Manager server and type docker swarm init –advertise-addr <IP> to create a swarm. Replace <IP> with the server IP address. A docker swarm join command containing an unique token will be generated for us to apply on Worker nodes.

$ docker swarm init --advertise-addr <IP>
Swarm initialized: current node (quhedtbhc1v10xuu5ggc5unsv) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-204js6i8sey05qkmcf6askfvwmpyxq3dd76qlrr4zgynholyok-4uqwmi8t2zsnrpcfnszdm7ukk <IP>:2377

Next, we log in to the other servers and join them as Worker nodes with the docker swarm join command.

$ docker swarm join --token SWMTKN-1-204js6i8sey05qkmcf6askfvwmpyxq3dd76qlrr4zgynholyok-4uqwmi8t2zsnrpcfnszdm7ukk <IP>:2377
This node joined a swarm as a worker.

Going back to the Manger node, we use docker node ls to check all servers associated with the swarm. It will list down all the host-name, status and availability information etc.

Step 3 – Deploy an App to Swarm

We use docker stack deploy to accept a file in Docker Compose format. It is important that we use version 3 of compose files, as docker stack deploy won’t support use of earlier versions. Do note that below is an incomplete compose file such that both “db” and “wordpress” services will get deployed to the Worker node as intended but the app, WordPress in this case, will fail to load in the browser.

version: '3.7'
services:
  db:
    image: mysql:8.0.19
    networks:
      - backend
    environment:
      - MYSQL_ROOT_PASSWORD=secret
    volumes: 
      - dbdata:/var/lib/mysql
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == worker]

  wordpress:
    image: wordpress:latest
    depends_on:
      - db
    networks:
      - backend
    environment:
      - WORDPRESS_DB_HOST=db:3306
      - WORDPRESS_DB_USER=wpuser
      - WORDPRESS_DB_PASSWORD=wpsecret
      - WORDPRESS_DB_NAME=wordpress
    volumes:
      - wordpress_files:/var/www/html
    ports:
      - 8080:80
    deploy:
      #replicas: 2 # Only if required > 1 replica
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == worker]

volumes:
  dbdata:
  wordpress_files:

networks:
  backend:

There are some Docker Compose stuffs that will not work when deploying a stack to a swarm. For example, we cannot use container_name as it is not supported. As for restart: option, it is now moved to restart_policy under deploy: section. To deploy these services, we will use docker stack deploy –compose-file docker-stack.yml wordpress which will create both MySQL and WordPress containers at the Worker node because we had specifically state constraints: [node.role == worker]. If we do a docker ps at the Worker node after two to three minutes, we will see these two containers.

$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                 NAMES
39ec0b5736d8        mysql:8.0.19                    "docker-entrypoint.s…"   38 seconds ago      Up 33 seconds       3306/tcp, 33060/tcp   wordpress_db.1.4yglt1ghm8nd55tf4q9sj09g4
7b6b0c4ec09f        wordpress:5.4.2-php7.4-apache   "docker-entrypoint.s…"   39 seconds ago      Up 31 seconds       80/tcp                wordpress_wordpress.1.p78n4s3pu6nroq9m67v2r2u7r

Troubleshooting

Sometimes, certain services may keep restarting due to some errors. We can always check the logs at the Manager node with docker stack ps wordpress –no-trunc.

$ docker stack ps wordpress --no-trunc
ID                          NAME                    IMAGE                                                                                                   NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
qied9o5xsstoeduq0bgwyex4m   wordpress_wordpress.1   wordpress:latest@sha256:095e933a48619e3dfb11f51b11fbe2ebdd0e2f827f1a38c556c80f635c8baf72                devops1             Running             Running 7 minutes ago
thbb91vb9ioxowlp73yutimqm   wordpress_db.1          mysql:8.0.19@sha256:9643e9fbd6330d10686f8922292dcb20995e7b792c17d4e94ddf95255f1d5449                    devops1             Running             Running 14 minutes ago

Going into individual service logs will be docker service logs <ID> where <ID> is the container ID, for example, qied9o5xsstoeduq0bgwyex4m.

$ docker service logs qied9o5xsstoeduq0bgwyex4m
wordpress_wordpress.1.v4g2hb4g13gw@devops1    | MySQL Connection Error: (2002) php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution
wordpress_wordpress.1.v4g2hb4g13gw@devops1    | [23-Jul-2020 15:38:53 UTC] PHP Warning:  mysqli::__construct(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in Standard input code on line 22
wordpress_wordpress.1.v4g2hb4g13gw@devops1    | [23-Jul-2020 15:38:53 UTC] PHP Warning:  mysqli::__construct(): (HY000/2002): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in Standard input code on line 22
wordpress_wordpress.1.v4g2hb4g13gw@devops1    |
wordpress_wordpress.1.v4g2hb4g13gw@devops1    | MySQL Connection Error: (2002) php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution
wordpress_wordpress.1.v4g2hb4g13gw@devops1    |
wordpress_wordpress.1.v4g2hb4g13gw@devops1    | WARNING: unable to establish a database connection to 'db:3306'
wordpress_wordpress.1.v4g2hb4g13gw@devops1    |   continuing anyways (which might have unexpected results)

After we had fixed the errors from the service logs, we may want to force restart the service containers using docker service update –force <NAME> if Docker Swarm does not do it for us.

$ docker service update --force wordpress_db
wordpress_db
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged
$ docker service update --force wordpress_wordpress
wordpress_wordpress
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

Conclusion

Docker Swarm is not as simple as I thought it would be like Ansible. There are some restrictions in the compose file due to the version 3 requirements and online help are still mostly based on version 2. Unfortunately, the WordPress installation page did not load on the browser as intended even though the MySQL and WordPress containers were successfully deployed to the Worker node. There is a voting app example using Docker Swarm at https://github.com/dockersamples/example-voting-app but I have yet to try it out as I was too focused and determined to correct the php_network_getaddresses: getaddrinfo failed: error with WordPress instead. Thus, this marks my milestone for learning a container orchestration tool.

Leave a Comment

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *