Collect Logs With Fluentd

A quick summary of Collect Logs With Fluentd

Collect any logs with Fluentd

Install Fluent on Ubuntu server 16.04

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh

Install Fluentd plugin to collect docker logs

sudo td-agent-gem install fluent-plugin-secure-forward

Setup Rsyslog to collect logs

Open Rsyslog config file and add strint in begging of the file

sudo vim /etc/rsyslog.conf

Add this line:

*.* @127.0.0.1:5140

Then restart service to seave changes

sudo systemctl restart rsyslog.service

sudo systemctl enable rsyslog.service

Setup Docker-compose file to enable fluentd plugin

logging:
  driver: fluentd
  options:
    fluentd-address: "${LOG_ADDR}:${LOG_PORT}"
    tag: name_of_tag.log

Setup Fluentd server for send logs

Prepare fluentd config file

sudo rm -fr /etc/td-agent/td-agent.conf

Insert castom configs to config file

`sudo vim /etc/td-agent/td-agent.conf`

And insert this config:

<source>
  @type forward
  port 42185 #Default port to collect logs from Docker containers. Should be the same in docker-compose.yml file
  bind 0.0.0.0
</source>

<source>
  type tail
  format nginx
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/log/nginx.access.pos
  <parse>
   @type nginx
  </parse>
  tag name_of_tag #Name of TAG. Used for matching logs on logging server
</source>

<source>
  type tail
  format nginx
  path /var/log/nginx/error.log
  pos_file /var/log/td-agent/log/nginx.error.pos
  <parse>
   @type nginx
  </parse>
  tag name_of_tag #Name of TAG. Used for matching logs on logging server
</source>

#This part use enctrypt connection with logging server for sent logs safely
#Port by default 24284
<match TAG_NAME.**>
  type secure_forward
  shared_key Kpymck35qgt3a2sZRRvasD5UHT8aMCsKQdAtZstz
  secure no
  self_hostname myserver #Server name
  <server>
    host prom.devopss.ru #Logging server address
  </server>
<secondary>
 @type file
 path /tmp/collectedm #Path where connection logs will be collected
</secondary>
</match>


#Rsyslog configs
<source>
  type syslog
  port 5140
  tag hostname.system
</source>

Setup logging server for receive logs

Install fluentd service is almost the same.
For receiveing logs fro other servers need to setup fluentd service.

Open config file: sudo vim /etc/td-agent/td-agent.conf

Then insert thi config:

# Other side of simple safety connection 
<source>
  type secure_forward
  shared_key Kpymck35qgt3a2sZRRvasD5UHT8aMCsKQdAtZstz #This part should be the same at both sides
  secure no
  self_hostname prom.devopss.ru #Name of server
  cert_auto_generate yes
</source>

#Match which logs to receive and where to collect
<match name_of_tag>
  @type copy
  <store>
   @type file
   path /var/log/td-agent/3dsense
   compress gzip
   <buffer>
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
   </buffer>
  </store>
</match>

#Match which logs to receive and where to collect
<match name_of_tag>
  @type copy
  <store>
   @type file
   path /var/log/td-agent/3dsense/nginx.access
   compress gzip
   <buffer>
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
   </buffer>
  </store>
</match>

#Match which logs to receive and where to collect
<match name_of_tag>
  @type copy
  <store>
   @type file
   path /var/log/td-agent/3dsense/nginx.error
   compress gzip
   <buffer>
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
   </buffer>
  </store>
</match>

@include 3dsense-prod.conf #There could be another config files.

Debugging

All logs state of td-agent located here:

sudo tail -f /var/log/td-agent/tg-agent.log

For read logs grom nginx and another services need to :

`sudo chmod -R og+rx /var/log/nginx/`