Collect logs with Promtail and Loki

Installation guide for Docker and system Logs

Collect any logs with Loki

Install Loki on Ubuntu server

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.5.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# move Loki to /usr/local/bin
$ sudo mv loki-linux-amd64 /usr/local/bin/loki
# make sure it is executable
$ sudo chmod a+x "/usr/local/bin/loki"

Install Loki plugin to collect docker logs

sudo docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Install Loki as system service

Create Loki config file:
sudo vim /usr/local/bin/config-loki.yml

Then insert this config:

auth_enabled: false

server:
  http_listen_port: 3100
  log_level: warn

ingester:
  wal:
    dir: "/tmp/wal"
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  alertmanager_url: http://localhost:9093

Create a system user "loki":
sudo useradd --system loki

Then create a systemd service file:
sudo vim /etc/systemd/system/loki.service

And paste this config:

[Unit]
Description=Loki service
After=network.target

[Service]
Type=simple
User=loki
ExecStart=/usr/local/bin/loki -config.file /usr/local/bin/config-loki.yml

[Install]
WantedBy=multi-user.target

At last start system service:

# create necessary folder for Loki
$ sudo mkdir /tmp/wal
# and grant Loki system user rights to it
$ sudo chown loki:loki /tmp/wal/
# enable Loki service to run at startup
$ sudo systemctl enable loki
# start Loki service right now
$ sudo systemctl start loki
# check that everything is correct
$ sudo systemctl status loki

Setup Docker service or container to enable loki plugin

logging:
  driver: loki
  options:
    loki-url: http://${LOG_ADDR}:${LOG_PORT}/loki/api/v1/push
        # default address is 127.0.0.1:3100

⚠️⚠️⚠️Docker-Swarm WARNING: in Swarm mode ${LOG_ADDR} should be set to real manager's node IP address and port should be accessible to worker servers. In other cases it can cause Docker-daemon hanging because of impossibility to send logs to Loki on 127.0.0.1:3100⚠️⚠️⚠️
or with docker command
sudo docker service update --force --log-driver=loki --log-opt loki-url="http://127.0.0.1:3100/loki/api/v1/push" --log-opt loki-retries=5 --log-opt loki-batch-size=400 service_name

Setup Promtail service to collecting system logs and sending them to Loki

Install Promtail service is almost the same.

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.5.0/promtail-linux-amd64.zip"
# extract the binary
$ unzip "promtail-linux-amd64.zip"
# move Promtail to /usr/local/bin
$ sudo mv promtail-linux-amd64 /usr/local/bin/promtail
# make sure it is executable
$ sudo chmod a+x "/usr/local/bin/promtail"

Open config file: sudo vim /usr/local/bin/config-promtail.yml

Promtail is capable to send it's metrics not only in local Loki instance and in remote Loki instance.

Then insert thi config and don't forget to specify a host labels:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
- job_name: nginx
  static_configs:
  - targets:
      - localhost
    labels:
      job: nginx
      host: ${HOST_NAME}
      __path__: /var/log/nginx/*log
- job_name: journald
  journal:
    labels:
      job: journald
      host: ${HOST_NAME}
    path: /var/log/journal

Then create a systemd service file:
sudo vim /etc/systemd/system/promtail.service

and insert:

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/promtail -config.file /usr/local/bin/config-promtail.yml

[Install]
WantedBy=multi-user.target

At last start system service:

# enable Promtail service to run at startup
$ sudo systemctl enable promtail
# start Promtail service right now
$ sudo systemctl start promtail
# check that everything is correct
$ sudo systemctl status promtail

Debugging

TODO