Authentication and Encryption

Updated Nov 20, 2022 ·

Overview

Prometheus does not have built-in authentication mechanisms by default/ If authentication is not configured, Prometheus can freely scrape metrics from target nodes. However, this also means that unauthorized or rogue servers can scrape metrics from those nodes, potentially exposing sensitive information.

To mitigate this, it's important to implement security measures such as authentication and encryption to restrict access and secure the data flow.

Lab Environment

Pre-requisites

Generate the Certificate

This can be done through openssl:

sudo  openssl req -new \
-newkey rsa:2048 -x509 -nodes \
-days 365  \
-keyout node_exporter.key \
-out node_exporter.crt \
-subj "/C=SG/ST=Singapore/L=Singapore/O=MyOrg/CN=localhost" \
-addext "subjectAltName=DNS:localhost"

Enable TLS on Node Exporter

Create a config.yml file:
```
tls_server_config:
  cert_file: node_exporter.crt  
  key_file: node_exporter.key  
```
To test if it works, run:
```
node_exporter --web.config=/path/to/config.yml 
```
info
To make this work, make sure you have installed the node_exporter on the target nodes and havae moved the node_exporter binary to /usr/local/bin.

Next, create the directory in the target nodes and then move the files.

sudo mkdir /etc/node_exporter
mv node_exporter.crt node_exporter.key config.yml /etc/node_exporter

Set the permissions:

sudo chown -R node_exporter:node_exporter /etc/node_exporter

Update the systemd unit file of the Node Exporter service:

[Unit]
Description=Node Exporter
Wants=network.target
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.config=/etc/node_exporter/config.yml

[Install]
WantedBy=multi-user.target

Restart the exporter:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter
sudo systemctl status node_exporter

To test, run a curl command:
```
curl -k https://localhost:9100/metrics 
```

Enable TLS on Prometheus

Copy the node_exporter.crt to the Prometheus server. We can use SCP for this.

scp username:password@node:/etc/node_exporter/node_exporter.crt /etc/prometheus

Login to the Prometheus server and change the pwnership of the copied file.
```
sudo chown prometheus:prometheus node_exporter.crt
```

Update the Prometheus configuration file. Edit the /etc/prometheus/prometheus.yml

scrape_configs:
- job_name: "node_exporter"
  scheme: https
  tls_config: 
    ca_file: /etc/prometheus/node_exporter.crt
    insecure_skip_verify: true          
    # If selft-signed cert, set to true
    # If maanged cert, set to false 
    static_configs:
      - targets: ["<node1_ip>:9100", "<node2_ip>:9100"]

Restart Prometheus.

sudo systemctl start prometheus
sudo systemctl status prometheus

Enable Encryption on Node Exporter

Generate the hash of the password. This can done through a couple of ways:

a. Using apache2-utils or `httpd-tools'

Install the utility first.

sudo apt install -y apache2-utils

Generate the hash. When prompted, enter the password.

htpasswd -nbC 12 "" | tr -d ':\n'

b. Using your preferred language:

Install the library first:

pip install bcrypt

Copy the code below:

import bcrypt
import getpass

# Prompt user to enter a password
password = getpass.getpass("Enter your password: ")

salt = bcrypt.gensalt()
hashed_password = bcrypt.hashpw(password.encode('utf-8'), salt)

print(f"Hashed password: {hashed_password.decode('utf-8')}")

Update the /etc/node_exporter/config.yml on the target nodes.
```
tls_server_config:
  cert_file: node_exporter.crt  
  key_file: node_exporter.key  
basic_auth_users:
  prometheus: *****************
```
Note that this is the username and the hashed password. If you're using a different user, change the prometheus to your user.
```
prometheus: *****************
```

Restart Prometheus to apply the configuration:

sudo systemctl restart node_exporter
sudo systemctl status node_exporter

Enable Encryption on Prometheus Server

From the Prometheus Server, run a curl to the target nodes. It will now show Unauthorized.

$ curl http://node01:9100/metrics

Unauthorized 

To resolve this, we also need to update the Prometheus server:

Update the /etc/prometheus/prometheus.yml:

scrape_configs:
- job_name: "node_exporter"
  scheme: https
  basic_auth:
    username: prometheus 
    password: mypassword   ## in plain text.

Restart Prometheus to apply the configuration:

sudo systemctl restart prometheus
sudo systemctl status prometheus

Access the Prometheus console and go to Status > Targets. The targets should now show "up".

Overview​

Lab Environment​

Pre-requisites​

Generate the Certificate​

Enable TLS on Node Exporter​

Enable TLS on Prometheus​

Enable Encryption on Node Exporter​

Enable Encryption on Prometheus Server​