<!-- TITLE: Collect Logs With Fluentd -->
<!-- SUBTITLE: A quick summary of Collect Logs With Fluentd -->
# Collect any logs with Fluentd
## Install Fluent on Ubuntu server 16.04
`curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh`
## Install Fluentd plugin to collect docker logs
`sudo td-agent-gem install fluent-plugin-secure-forward`
## Setup Rsyslog to collect logs
Open Rsyslog config file and add strint in begging of the file
`sudo vim /etc/rsyslog.conf`
Add this line:
`*.* @127.0.0.1:5140`
Then restart service to seave changes
`sudo systemctl restart rsyslog.service`
`sudo systemctl enable rsyslog.service`
## Setup Docker-compose file to enable fluentd plugin
```yaml
logging:
driver: fluentd
options:
fluentd-address: "${LOG_ADDR}:${LOG_PORT}"
tag: name_of_tag.log
```
## Setup Fluentd server for send logs
Prepare fluentd config file
`sudo rm -fr /etc/td-agent/td-agent.conf`
Insert castom configs to config file
`sudo vim /etc/td-agent/td-agent.conf`
And insert this config:
```text
<source>
@type forward
port 42185 #Default port to collect logs from Docker containers. Should be the same in docker-compose.yml file
bind 0.0.0.0
</source>
<source>
type tail
format nginx
path /var/log/nginx/access.log
pos_file /var/log/td-agent/log/nginx.access.pos
<parse>
@type nginx
</parse>
tag name_of_tag #Name of TAG. Used for matching logs on logging server
</source>
<source>
type tail
format nginx
path /var/log/nginx/error.log
pos_file /var/log/td-agent/log/nginx.error.pos
<parse>
@type nginx
</parse>
tag name_of_tag #Name of TAG. Used for matching logs on logging server
</source>
#This part use enctrypt connection with logging server for sent logs safely
#Port by default 24284
<match TAG_NAME.**>
type secure_forward
shared_key Kpymck35qgt3a2sZRRvasD5UHT8aMCsKQdAtZstz
secure no
self_hostname myserver #Server name
<server>
host prom.devopss.ru #Logging server address
</server>
<secondary>
@type file
path /tmp/collectedm #Path where connection logs will be collected
</secondary>
</match>
#Rsyslog configs
<source>
type syslog
port 5140
tag hostname.system
</source>
```
## Setup logging server for receive logs
Install fluentd service is almost the same.
For receiveing logs fro other servers need to setup fluentd service.
Open config file: `sudo vim /etc/td-agent/td-agent.conf`
Then insert thi config:
```text
# Other side of simple safety connection
<source>
type secure_forward
shared_key Kpymck35qgt3a2sZRRvasD5UHT8aMCsKQdAtZstz #This part should be the same at both sides
secure no
self_hostname prom.devopss.ru #Name of server
cert_auto_generate yes
</source>
#Match which logs to receive and where to collect
<match name_of_tag>
@type copy
<store>
@type file
path /var/log/td-agent/3dsense
compress gzip
<buffer>
timekey 1d
timekey_use_utc true
timekey_wait 10m
</buffer>
</store>
</match>
#Match which logs to receive and where to collect
<match name_of_tag>
@type copy
<store>
@type file
path /var/log/td-agent/3dsense/nginx.access
compress gzip
<buffer>
timekey 1d
timekey_use_utc true
timekey_wait 10m
</buffer>
</store>
</match>
#Match which logs to receive and where to collect
<match name_of_tag>
@type copy
<store>
@type file
path /var/log/td-agent/3dsense/nginx.error
compress gzip
<buffer>
timekey 1d
timekey_use_utc true
timekey_wait 10m
</buffer>
</store>
</match>
@include 3dsense-prod.conf #There could be another config files.
```
## Debugging
All logs state of td-agent located here:
`sudo tail -f /var/log/td-agent/tg-agent.log`
For read logs grom nginx and another services need to :
`sudo chmod -R og+rx /var/log/nginx/`