Differences between revisions 2 and 3
⇤ ← Revision 2 as of 2021-11-16 18:38:52 →
Size: 2030
Editor: DominicRicottone
Comment:
← Revision 3 as of 2021-11-19 17:18:29 → ⇥
Size: 2436
Editor: DominicRicottone
Comment:
Deletions are marked like this. Additions are marked like this.
Line 29: Line 29:


=== Name Resolution Prerequisite ===

For '''Prometheus''' monitoring, `dockerd(8)` will need to bind to an address and port.

If you aren't concerned with hardening your server, just use `0.0.0.0` and everything will work.

If all monitoring will be done locally, you can check the output of `ip addr show docker0` to find the exact intranet address that `dockerd(8)` has bound to.

When monitoring remote servers, the Prometheus client must connect to the same address that `dockerd(8)` is bound to. It is ''not possible'' to configure DNS for the client and bind to `127.0.0.1` on the server. The only workaround is for `dockerd(8)` to bind to `0.0.0.0`.
Line 41: Line 55:

Third, determine what IP address `dockerd(8)` has bound to. Note that `curl 127.0.0.1:9323/metrics` is a red herring; check the output of `ip addr show docker0`.

Optionally, go back to step 1 and update the 'wildcard' IP address in `daemon.json` with the actual address that is bound.
Line 73: Line 83:
      - targets: ['172.17.0.1:9323']       - targets: ['172.17.0.1:9323', 'example.com:9323']

Docker Logging and Monitoring

Contents

  1. Docker Logging and Monitoring
    1. Log Rotation
    2. Prometheus
      1. Name Resolution Prerequisite
      2. Docker Prerequisites
      3. Prometheus Service

Log Rotation

The default behavior for container logs is to write JSON-formatted messages to the host filesystem without limit. This can lead to log files that are too large to store or reasonably use. This example daemon.json configuration file will enable reasonable log rotation. 

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "1m",
    "max-file": "3"
  }
}

Prometheus

Name Resolution Prerequisite

For Prometheus monitoring, dockerd(8) will need to bind to an address and port.

If you aren't concerned with hardening your server, just use 0.0.0.0 and everything will work.

If all monitoring will be done locally, you can check the output of ip addr show docker0 to find the exact intranet address that dockerd(8) has bound to.

When monitoring remote servers, the Prometheus client must connect to the same address that dockerd(8) is bound to. It is not possible to configure DNS for the client and bind to 127.0.0.1 on the server. The only workaround is for dockerd(8) to bind to 0.0.0.0.

Docker Prerequisites

First, configure dockerd(8) through daemon.json to advertise metrics. 

{
  "metrics-addr": "0.0.0.0:9323",
  "experimental": true
}

Second, restart dockerd(8).

Lastly, if you haven't already, initialize a Docker swarm. 

docker swarm init

Prometheus Service

Prometheus needs to be configured to scrape dockerd(8) and publish it's database. 

global:
  scrape_interval:     15s # Default is 60s
  evaluation_interval: 15s # Default is 60s
  # scrape_timeout default is 10s
  external_labels:
    monitor: 'my-monitor'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'docker'
    static_configs:
      - targets: ['172.17.0.1:9323', 'example.com:9323']

Start one instance of Prometheus to the swarm. 

docker service create --replicas=1 --name prometheus \
    --mount type=bind,src=/path/to/prometheus.yml,dst=/etc/prometheus/prometheus.yml \
    --publish 9090:9090/tcp \
    prom/prometheus

Verify that Prometheus has successfully connected to dockerd(8) at http://127.0.0.1:9090/targets.

CategoryRicottone

Docker/LoggingAndMonitoring (last edited 2023-04-04 19:20:03 by DominicRicottone)