<!-- TITLE: Add Logs To Prometheus --> <!-- SUBTITLE: A quick summary of Add Logs To Prometheus --> # Add Fluentd logs to Prometheus metrics ## Preparation Install prometheus plugin on **logs server** for send logs to prometheus : `sudo td-agent-gem install fluent-plugin-prometheus` After plugin instalation need to configure td-agent (fluentd) config files to receive and send logs. In this instrution will be send nginx error and access logs. ## Client server Fluentd setup On a **client server side** only collecting and matching needed logs from services. All visualiazation creates on **logs server**. Sctrutcure of td-agent directory ```text /etc/td-agent/ ├── nginx.access.conf ├── nginx.error.conf ├── plugin ├── td-agent.conf ``` Configuration files **nginx.access** and **nginx.error** using for collect and filter logs from nginx service. Configuration file **td-agent.conf** using only for main send or receive logs. ### Structure of nginx configuration client files Default nginx.access.conf file should be look like this ```text # Count number of nginx access logs by tag <source> @type tail format nginx path /var/log/nginx/access.log pos_file /var/log/td-agent/log/nginx.access.pos format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwarder>[^\"]*)")?/ time_format %d/%b/%Y:%H:%M:%S %z tag nginx.access #Name of TAG. Used for matching logs on logging server </source> # Match nginx access logs and send it to log server <match nginx.access> @type copy <store> @type forward <server> name myserver1 host 10.10.2.11 port 24224 weight 60 </server> </store> </match> ``` Default nginx.error file will be look almoust the same ```text # Count number of nginx error logs by tag <source> @type tail format nginx path /var/log/nginx/error.log pos_file /var/log/td-agent/log/nginx.error.pos format /(?<time>[^ ]* [^ ]*) +(?<method>[^ ]*) +(?<path>[^ ]*) +(?<message>[^ ].*$)/ tag nginx.error #Name of TAG. Used for matching logs on logging server </source> # Match nginx error logs and sent it to log server <match nginx.error> @type copy <store> @type forward <server> name myserver1 host 10.10.2.11 port 24224 weight 60 </server> </store> </match> ``` ### Structure of td-agent client file Default td.agent.conf file ```text #Rsyslog configs <source> @type syslog port 5140 tag 10.10.2.12 </source> # Add nginx config files to td-agent file. @include nginx.error.conf @include nginx.access.conf ``` ## Logs server Fluentd setup Sctrutcure of td-agent directory ```text /etc/td-agent/ ├── nginx.access.conf ├── nginx.error.conf ├── plugin └── td-agent.conf ``` As on the **client server** *tg-agent.conf* using only for send logs to monitoring on prometheus format. Receiving and matching logs creating on *nginx.error.conf* and *nginx.error.conf* files ### Structure of nginx configuration log server files Default nginx.access.config file ```text # Filtering nginx access logs by tag and use prometheus plugin to send it to metrics <filter nginx.access> @type prometheus <metric> name nginx_access_records_total type counter desc The total number of incoming access <labels> tag ${tag} hostname ${hostname} </labels> </metric> </filter> ##Match which logs to receive and where to collect <match nginx.access> @type copy <store> @type file path /var/log/td-agent/nginx.access compress gzip <buffer> timekey 1d timekey_use_utc true timekey_wait 10m </buffer> </store> </match> ``` Default nginx.error.conf file almoust the same ```text # Filtering nginx error logs by tag and use prometheus plugin to send it to metrics <filter nginx.error> @type prometheus <metric> name nginx_error_records_total type counter desc The total number of incoming errors <labels> tag ${tag} hostname ${hostname} </labels> </metric> </filter> ##Match which logs to receive and where to collect <match nginx.error> @type copy <store> @type file path /var/log/td-agent/nginx.error compress gzip <buffer> timekey 1d timekey_use_utc true timekey_wait 10m` </buffer> </store> </match> ``` ### Structure of td-agent log server file ```text ## Address and port to receive logs <source> @type forward bind 0.0.0.0 port 24224 </source> # expose metrics in prometheus format <source> @type prometheus bind 0.0.0.0 port 24231 metrics_path /metrics </source> # Add nginx config files to td-agent file. @include nginx.access.conf @include nginx.error.conf ``` ## Check tg-agent config file After add all configs need to be sure that all configs are correct. So run this command (on both servers) : `sudo tg-agent -c /etc/td-agent/td-agent.cong` Result should be like this: ```text 2019-07-03 07:29:57 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf" 2019-07-03 07:29:57 +0000 [info]: using configuration file: <ROOT> <source> @type forward bind "0.0.0.0" port 24224 </source> <source> @type prometheus bind "0.0.0.0" port 24231 metrics_path "/metrics" </source> <filter nginx.access> @type prometheus <metric> name nginx_access_records_total type counter desc The total number of incoming access <labels> tag ${tag} hostname ${hostname} </labels> </metric> </filter> <match nginx.access> @type copy <store> @type "file" path "/var/log/td-agent/nginx.access" compress gzip <buffer> timekey 1d timekey_use_utc true timekey_wait 10m path "/var/log/td-agent/nginx.access" </buffer> </store> </match> <filter nginx.error> @type prometheus <metric> name nginx_error_records_total type counter desc The total number of incoming errors <labels> tag ${tag} hostname ${hostname} </labels> </metric> </filter> <match nginx.error> @type copy <store> @type "file" path "/var/log/td-agent/nginx.error" compress gzip <buffer> timekey 1d timekey_use_utc true timekey_wait 10m` path "/var/log/td-agent/nginx.error" </buffer> </store> </match> </ROOT> 2019-07-03 07:29:57 +0000 [info]: starting fluentd-1.4.2 pid=10594 ruby="2.4.6" 2019-07-03 07:29:57 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "-c", "/etc/td-agent/td-agent.conf", "--under-supervisor"] 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.1' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-kafka' version '0.9.4' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.4.0' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.10' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4' 2019-07-03 07:29:58 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3' 2019-07-03 07:29:58 +0000 [info]: gem 'fluentd' version '1.4.2' 2019-07-03 07:29:58 +0000 [info]: adding filter pattern="nginx.access" type="prometheus" 2019-07-03 07:29:58 +0000 [info]: adding match pattern="nginx.access" type="copy" 2019-07-03 07:29:58 +0000 [info]: adding filter pattern="nginx.error" type="prometheus" 2019-07-03 07:29:58 +0000 [info]: adding match pattern="nginx.error" type="copy" 2019-07-03 07:29:58 +0000 [info]: adding source type="forward" 2019-07-03 07:29:58 +0000 [info]: adding source type="prometheus" 2019-07-03 07:29:58 +0000 [info]: #0 starting fluentd worker pid=10601 ppid=10594 worker=0 2019-07-03 07:29:58 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0" 2019-07-03 07:29:58 +0000 [info]: #0 fluentd worker is now running worker=0 ``` If all configs works fine and no errors, after some time in localhost:24321/metrics should add created metrics: ```text # TYPE nginx_access_records_total counter # HELP nginx_access_records_total The total number of incoming access nginx_access_records_total{tag="nginx.access",hostname="ubuntu"} 236.0 # TYPE nginx_error_records_total counter # HELP nginx_error_records_total The total number of incoming errors nginx_error_records_total{tag="nginx.error",hostname="ubuntu"} 42.0 ``` After that, this metric could ne added to alert rules in Prometheus.