Add New Server to Prometheus

1. Service structure of Prometheus monitoring
2. Add server Node_Exporter monitoring service
3. Add server to SSL Checker monitoring service
4. Add server to Blackbox monitoring service
5. Alert rules

Service structure of Prometheus monitoring

Prometheus

Current Prometheus configs server locates in /etc/prometheus/ directory.
Structure of Prometheus directory:

/etc/prometheus/
├── console_libraries - *list of libraries*
│   ├── menu.lib
│   └── prom.lib
├── consoles - *list of files for web*
│   ├── index.html.example
│   ├── node-cpu.html
│   ├── node-disk.html
│   ├── node.html
│   ├── node-overview.html
│   ├── prometheus.html
│   └── prometheus-overview.html
├── cpu.rules
├── hdd.rules
├── http.rules
├── main.rules - *list of rules (.rules)*
├── mem.rules
├── node_exporter.rules
├── prometheus.yml - *main config file*
└── ssl.rules

Example of prometheus config are show here

Alertnamager

/etc/alertmanager/
├── alertmanager.yml - *main alertmanager config*
└── template

Alertmanager using for send alert to services, for example to Slack.
Simple config and instalation of alertmanager show here

Blackbox exporter

/etc/blackbox_exporter/
├── blackbox.yml - *main service config*
└── targets
    ├── http_3ds.yml - *check rule for sites works over http*
    └── https_2xx.yml - *check rule for sites works over https*

Keywords are located in /etc/blackbox_exporter/blackbox.yml config file.
In /etc/blackbox_exporter/targets directory located web sites that should be checked.

Add server Node_Exporter monitoring service

Node exporter service should be installed on client server. It sends metrics to monitoring server over https.

All instalation proccess are show in this article
After instalation server IP address or domain name should be added to /etc/prometheus.yml config.
Then prometheus should be reloade by command: sudo systemctl restart prometheus.service

Add server to SSL_Checker monitoring service

SSL_Ckecker service checks ssl expiration period of ssl sertificate.
Service installing on monitoring server.
To add new server to SSL_Checker it need to add to /etc/prometheus/prometheus.yml config file.
Then prometheus should be reloade by command: sudo systemctl restart prometheus.service

Add server to Blackbox monitoring service

Blackbox exporter using for check sites over http/https by keyword/s.
In /etc/blackbox_exporter/blackbox.yml config file add new check rules and also new keyword.
In /etc/blackbox_exporter/target directory add list of web sites.

Alert rules

All alert riles are located in /etc/prometheus/ directory with .rules extention
List of main rules are show here
All new rule-files add to /etc/prometheus/prometheus.yml after prometheus service should be reloadet to accept new configuration.