<!-- TITLE: Collect logs with Promtail and Loki --> <!-- SUBTITLE: Installation guide for Docker and system Logs --> # Collect any logs with Loki ## Install Loki on Ubuntu server ```sh $ curl -O -L "https://github.com/grafana/loki/releases/download/v2.5.0/loki-linux-amd64.zip" # extract the binary $ unzip "loki-linux-amd64.zip" # move Loki to /usr/local/bin $ sudo mv loki-linux-amd64 /usr/local/bin/loki # make sure it is executable $ sudo chmod a+x "/usr/local/bin/loki" ``` ## Install Loki plugin to collect docker logs `sudo docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions` ## Install Loki as system service Create Loki config file: `sudo vim /usr/local/bin/config-loki.yml` Then insert this config: ```yaml auth_enabled: false server: http_listen_port: 3100 log_level: warn ingester: wal: dir: "/tmp/wal" lifecycler: address: 127.0.0.1 ring: kvstore: store: inmemory replication_factor: 1 final_sleep: 0s chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m) max_transfer_retries: 0 # Chunk transfers disabled schema_config: configs: - from: 2020-10-24 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h storage_config: boltdb_shipper: active_index_directory: /tmp/loki/boltdb-shipper-active cache_location: /tmp/loki/boltdb-shipper-cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space shared_store: filesystem filesystem: directory: /tmp/loki/chunks compactor: working_directory: /tmp/loki/boltdb-shipper-compactor shared_store: filesystem limits_config: reject_old_samples: true reject_old_samples_max_age: 168h ingestion_rate_mb: 10 ingestion_burst_size_mb: 20 chunk_store_config: max_look_back_period: 0s table_manager: retention_deletes_enabled: false retention_period: 0s ruler: alertmanager_url: http://localhost:9093 ``` Create a system user "loki": `sudo useradd --system loki` Then create a systemd service file: `sudo vim /etc/systemd/system/loki.service` And paste this config: ```text [Unit] Description=Loki service After=network.target [Service] Type=simple User=loki ExecStart=/usr/local/bin/loki -config.file /usr/local/bin/config-loki.yml [Install] WantedBy=multi-user.target ``` At last start system service: ```sh # create necessary folder for Loki $ sudo mkdir /tmp/wal # and grant Loki system user rights to it $ sudo chown loki:loki /tmp/wal/ # enable Loki service to run at startup $ sudo systemctl enable loki # start Loki service right now $ sudo systemctl start loki # check that everything is correct $ sudo systemctl status loki ``` ## Setup Docker service or container to enable loki plugin ```yaml logging: driver: loki options: loki-url: http://${LOG_ADDR}:${LOG_PORT}/loki/api/v1/push # default address is 127.0.0.1:3100 ``` **⚠️⚠️⚠️Docker-Swarm WARNING: in Swarm mode ${LOG_ADDR} should be set to real manager's node IP address and port should be accessible to worker servers. In other cases it can cause Docker-daemon hanging because of impossibility to send logs to Loki on 127.0.0.1:3100⚠️⚠️⚠️** or with docker command `sudo docker service update --force --log-driver=loki --log-opt loki-url="http://127.0.0.1:3100/loki/api/v1/push" --log-opt loki-retries=5 --log-opt loki-batch-size=400 service_name` ## Setup Promtail service to collecting system logs and sending them to Loki Install Promtail service is almost the same. ```sh $ curl -O -L "https://github.com/grafana/loki/releases/download/v2.5.0/promtail-linux-amd64.zip" # extract the binary $ unzip "promtail-linux-amd64.zip" # move Promtail to /usr/local/bin $ sudo mv promtail-linux-amd64 /usr/local/bin/promtail # make sure it is executable $ sudo chmod a+x "/usr/local/bin/promtail" ``` Open config file: `sudo vim /usr/local/bin/config-promtail.yml` Promtail is capable to send it's metrics not only in local Loki instance and in remote Loki instance. Then insert thi config and don't forget to specify a host labels: ```text server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://127.0.0.1:3100/loki/api/v1/push scrape_configs: - job_name: system static_configs: - targets: - localhost labels: job: varlogs __path__: /var/log/*log - job_name: nginx static_configs: - targets: - localhost labels: job: nginx host: ${HOST_NAME} __path__: /var/log/nginx/*log - job_name: journald journal: labels: job: journald host: ${HOST_NAME} path: /var/log/journal ``` Then create a systemd service file: `sudo vim /etc/systemd/system/promtail.service` and insert: ```text [Unit] Description=Promtail service After=network.target [Service] Type=simple User=root ExecStart=/usr/local/bin/promtail -config.file /usr/local/bin/config-promtail.yml [Install] WantedBy=multi-user.target ``` At last start system service: ```sh # enable Promtail service to run at startup $ sudo systemctl enable promtail # start Promtail service right now $ sudo systemctl start promtail # check that everything is correct $ sudo systemctl status promtail ``` ## Debugging TODO