Docker Container Monitoring Using PRTG

Prerequisites

In this lesson we will walk you through all the measures necessary to be taken so PRTG is able to connect to a remote Docker socket.

As a prerequisite, it is necessary to have Docker installed on your system as well as an instance of PRTG with access to that host.

We assume you have at least a basic understanding of Docker and Linux. If you want to refresh your knowledge, we recommend looking at the lesson Docker Basics.

Explaining Linux would be far out of scope for this lesson, but it’s likely an answer to any Linux related question is out there on the internet. Anyway, if you read carefully, the listed commands should work with only minor adjustments.

Introduction

Monitoring your IT infrastructure has a lot of benefits, discovering bottlenecks and gaining insights for predictive measures being only the tip of the iceberg.

PRTG is a solid monitoring solution already present and actively used in a lot of IT departments. Because there are a lot of different monitoring solutions out there, this article is targeted to be compatible with the way PRTG handles Docker Container Monitoring.

PRTG requires the Docker Socket to be exposed to the network, which is not the case on a default setup. The reason for the port not being exposed by default is because of security reasons.

An exposed and unsecured port could lead to a major security issue! Whoever is able to connect to the docker socket could easily gain full control on the system – meaning root access.

Therefore it is really important to handle these configurations with care. The measurement we are going to take is to secure the remote access by using TLS certificates. You can read more about this in the Docker docs.

A guide on the PRTG Docker Container Sensor can be found here.

Certificate Generation

First of all we need to create a bunch of certificates. There are basically two options for doing this. 

• The first being to use your private company certificate authority. 
• The second is to create the certificates locally.

We are going to use the second option, which means all certificates are going to be self-signed, but that’s totally fine for the purpose of this lesson.

All instructions for the creation of the certificates can be found in the docker docs. To simplify this a little bit, we created a small script that executes all the commands for you.

All the steps below assume you are going to use the script. The script is non-interactive, meaning you do not have to enter anything during execution. The generated certificates won’t be password protected and are valid for 50 years.

Create a directory called .docker in your home directory. This directory is the default directory where the Docker CLI stores all its information.

$ mkdir -p ~/.docker
Code-Sprache: YAML (yaml)

Clone the script into the previously created directory.

$ git clone https://gist.github.com/6f6b9a85e136b37cd52983cb88596158.git ~/.docker/
Code-Sprache: YAML (yaml)

Change into the directory.

$ cd ~/.docker/
Code-Sprache: YAML (yaml)

Make the script executable.

$ chmod +x genCerts.sh
Code-Sprache: YAML (yaml)

Then we need to adjust a few things within the script.

$ nano genCerts.sh
Code-Sprache: YAML (yaml)

Adjust the HOST to match your hostname and the last IP of the HOSTS string to match your host ip address.

This is how it looks for my setup.

HOST="cybus.io"
HOSTS="DNS:$HOST,IP:127.0.0.1,IP:172.16.0.131"
Code-Sprache: YAML (yaml)

Now we are ready to execute the script.

$ sh genCerts.sh
Code-Sprache: YAML (yaml)

The output should be somewhat like this.

# Start

# Generate CA private and public keys
Generating RSA private key, 4096 bit long modulus (2 primes)
.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................++++
...............++++
e is 65537 (0x010001)

Create a server key
Generating RSA private key, 4096 bit long modulus (2 primes)
.++++
..........................................................................++++
e is 65537 (0x010001)

Create certificate signing request

Sign the public key with CA
Signature ok
subject=CN = cybus.io
Getting CA Private Key

Create a client key and certificate signing request
Generating RSA private key, 4096 bit long modulus (2 primes)
.................................................................................................................................++++
...............................................................................................................................................................................................................................................................................................................++++
e is 65537 (0x010001)

Make the key suitable for client authentication

Generate the signed certificate
Signature ok
subject=CN = client
Getting CA Private Key

Remove the two certificate signing requests and extensions config
removed 'client.csr'
removed 'server.csr'
removed 'extfile.cnf'
removed 'extfile-client.cnf'
Code-Sprache: YAML (yaml)

To verify all certificates have been generated successfully we inspect the content of the directory.

$ ls
Code-Sprache: YAML (yaml)

These files should be present. If there are more files than this, that’s no issue.

ca-key.pem  ca.pem  ca.srl  cert.pem  genCerts.sh  key.pem  server-cert.pem  server-key.pem
Code-Sprache: YAML (yaml)

The last step is to locate the full path to where the certificates live.

$ pwd
Code-Sprache: YAML (yaml)

This is the output in my case. Yours will look a little bit different.

/home/jan/.docker
Code-Sprache: YAML (yaml)

Docker Service Configuration

With all the necessary certificates in place, we have to assign them to the docker daemon. We can find the location of the configuration file by checking the status of the docker service.

$ sudo systemctl status docker.service
Code-Sprache: YAML (yaml)

As stated in the output, the configuration file is located at /lib/systemd/system/docker.service

● docker.service - Docker Application Container Engine     
       Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)     
       Active: active (running) since Mon 2022-05-02 10:26:56 EDT; 33s ago
  TriggeredBy: ● docker.socket       
       Docs: https://docs.docker.com   Main PID: 468 (dockerd)      
        Tasks: 9     
       Memory: 109.2M        
          CPU: 307ms     CGroup: /system.slice/docker.service          
               └─468 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Code-Sprache: YAML (yaml)

To adjust the configuration to our needs, we are going to open the configuration using sudo privileges. 

$ sudo nano /lib/systemd/system/docker.service
Code-Sprache: YAML (yaml)

Find the line starting with ExecStart=/usr/bin/dockerd -H fd:// and add the following content to it. Be sure to use the correct path for your setup.

-H tcp://0.0.0.0:2376 --tlsverify=true --tlscacert=/home/jan/.docker/ca.pem --tlscert=/home/jan/.docker/server-cert.pem --tlskey=/home/jan/.docker/server-key.pem
Code-Sprache: YAML (yaml)

For me the complete line looks like this. 

ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376 --tlsverify=true --tlscacert=/home/jan/.docker/ca.pem --tlscert=/home/jan/.docker/server-cert.pem --tlskey=/home/jan/.docker/server-key.pem --containerd=/run/containerd/containerd.sock
Code-Sprache: YAML (yaml)

Flush the changes and restart the docker service.

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
Code-Sprache: YAML (yaml)

Now we can verify our changes did take effect.

$ sudo systemctl status docker.service
Code-Sprache: YAML (yaml)

● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-05-03 04:56:12 EDT; 2min 32s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 678 (dockerd)
      Tasks: 9
     Memory: 40.8M
        CPU: 236ms
     CGroup: /system.slice/docker.service
            └─678 /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376 --tlsverify=true --tlscacert=/home/jan/.docker/ca.pem --tlscert=/home/jan/.docker/server-cert.pem --tlskey=/home/jan/.docker/server-key.pem --containerd=/run/containerd/containerd.sock
Code-Sprache: YAML (yaml)

Now we can use the Docker CLI to connect to the Docker Daemon using the specified port. The important part is to use –tlsverify=true as this tells the Docker CLI to use the generated certificates located in our home directory ( ~/.docker).

Remember to adjust the IP address in the second line with your individual one.

$ docker -H 127.0.0.1:2376 --tlsverify=true version
$ docker -H 172.16.0.131:2376 --tlsverify=true version
Code-Sprache: YAML (yaml)

This is the output of both commands on my system.

Client: Docker Engine - Community
Version:           20.10.14
API version:       1.41
Go version:        go1.16.15
Git commit:        a224086
Built:             Thu Mar 24 01:48:21 2022
OS/Arch:           linux/amd64
Context:           default
Experimental:      true

Server: Docker Engine - Community
Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Code-Sprache: YAML (yaml)

PRTG

The last and final step is to install the docker sensor inside of PRTG. This should be fairly easy to accomplish by following the provided instructions from https://www.paessler.com/manuals/prtg/docker_container_status_sensor.