Docker Logging and Monitoring

Log Rotation

The default behavior for container logs is to write JSON-formatted messages to the host filesystem without limit. This can lead to log files that are too large to store or reasonably use. This example daemon.json configuration file will enable reasonable log rotation.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "1m",
    "max-file": "3"
  }
}

Prometheus

Docker Prerequisites

First, configure dockerd(8) through daemon.json to advertise metrics.

{
  "metrics-addr": "0.0.0.0:9323",
  "experimental": true
}

Second, restart dockerd(8).

Third, determine what IP address dockerd(8) has bound to. Note that curl 127.0.0.1:9323/metrics is a red herring; check the output of ip addr show docker0.

Optionally, go back to step 1 and update the 'wildcard' IP address in daemon.json with the actual address that is bound.

Lastly, if you haven't already, initialize a Docker swarm.

docker swarm init

Prometheus Service

Prometheus needs to be configured to scrape dockerd(8) and publish it's database.

global:
  scrape_interval:     15s # Default is 60s
  evaluation_interval: 15s # Default is 60s
  # scrape_timeout default is 10s
  external_labels:
    monitor: 'my-monitor'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'docker'
    static_configs:
      - targets: ['172.17.0.1:9323']

Start one instance of Prometheus to the swarm.

docker service create --replicas=1 --name prometheus \
    --mount type=bind,src=/path/to/prometheus.yml,dst=/etc/prometheus/prometheus.yml \
    --publish 9090:9090/tcp \
    prom/prometheus

Verify that Prometheus has successfully connected to dockerd(8) at http://127.0.0.1:9090/targets.

CategoryRicottone