Supervising Docker

Lightning Talk @ Docker Meetup Barcelona

By Emmet O'Grady / @emmetogrady

Background

I'm building Nimble CI, a CI platform specializing in fast black-box testing of Docker containers.

Build pipelines are run in steps, each step is run in a Docker container.

Why Supervisor?

My containers stop after each job, I want to erase the stopped container and start a new one.


docker rm -f worker-container-01
docker run --name worker-container-01 worker-container-01
                    

Supervising (using Ansible)


## file ./ansible-playbooks/workers.yml

  - role: e.supervise
    name: worker-container-01
    autostart: true
    autorestart: true
    startsecs: 0
    startretries: 100
    command: docker run -it --rm \
             --name worker-container-01 \
             -a stdout -a stderr \
              worker-container-01
                    

The role


## file ./ansible-playbooks/roles/e.supervise/tasks/main.yml

  - name: installing config for {{ name }}
    template:
    src: app.conf.j2
    dest: /etc/supervisor/processes/{{ name }}.conf

  - name: reloading configuration for {{ name }}
    shell: >
    sudo /usr/bin/supervisorctl reread;
    sudo /usr/bin/supervisorctl add {{ name }};
    sudo /usr/bin/supervisorctl restart {{ name }};
                    

## file ./ansible-playbooks/roles/e.supervise/templates/app.conf.j2

  [program:{{ name }}]
  command={{ command }}
  stdout_logfile=/var/log/supervisor/{{ name }}.out.log
  stderr_logfile=/var/log/supervisor/{{ name }}.err.log

  autostart={{ autostart }}
  autorestart={{ autorestart }}
  startsecs={{ startsecs }}
  startretries={{ startretries }}
                    

But docker didn't like this, Supervisor tries sending signals to the container but docker doesn't care

:'(

Solution

Wrap the process in a wrapper script


## file ./ansible-playbooks/roles/e.supervise/templates/docker-wrapper.conf.j2
#!/bin/bash

trap "{ \
        echo Stopping container {{ name }}; \
        sudo docker rm -f {{ name }} || true; \
        exit 0; \
}" EXIT

echo "Killing any previous container"
sudo docker rm -f {{ name }} || true;
echo "Starting container"
{{ command }}
                    

This gives supervisor complete control over the container, you can use the supervisorctl utility to start/stop/restart the container.

Now when the container stops supervisor will wipe the container and spawn a new one, just what we wanted.