How to Add Linux Target Nodes to Prometheus Monitoring

In this tutorial, we will learn how to add a new Linux server (Target Node) to your existing Prometheus monitoring system. To monitor a new Linux server, Node Exporter must be installed and running to collect system metrics such as CPU, memory, and disk usage. After that, we will configure our main Prometheus server to scrape data from it.

Prerequisites

A running Prometheus Server.
A new Target Node (the server you want to monitor).
Root or sudo privileges on both servers.

Step 1 Install Node Exporter on the Target Node

To monitor the new Target Node, Node Exporter must be installed and running to collect system metrics such as CPU, memory, and disk usage.

On most RHEL-based distributions such as CentOS Stream, AlmaLinux, and Rocky Linux, install Node Exporter using:

Bash


dnf install -y prometheus-node-exporter

If the package is unavailable, enable the EPEL repository first:

Bash


dnf install epel-release -y

✅ Note: On some CentOS Stream or RHEL 9 systems, you may need to enable the CRB (CodeReady Builder) repository before installing certain packages from EPEL using the command: dnf config-manager --set-enabled crb

Then install Node Exporter again:

Bash


dnf install -y prometheus-node-exporter

Then install Node Exporter again:

Step 2 Verify Node Exporter is Running (Recommended)

Before moving to the firewall and Prometheus configuration, it is highly recommended to verify that Node Exporter is successfully generating metrics.

On your Target Node, run the following command:

Bash


curl http://localhost:9100/metrics

If Node Exporter is working correctly, you will immediately see a long list of system metrics printed on your screen.

Step 3 Open Port 9100 in the Firewall

Node Exporter runs on port 9100 by default. If you have firewalld enabled on your target node, you must allow traffic on this port so the main Prometheus server can reach it.

On your Target Node, run:

Bash


firewall-cmd --permanent --add-port=9100/tcp
firewall-cmd --reload

(You should see a success message after running each command).

Security Note: In production environments, it is strongly recommended to restrict access using firewall rules or security groups so that only your main Prometheus server's IP address can reach the Node Exporter endpoint on port 9100. This prevents unauthorized access to your system metrics.

Step 4 Configure the Prometheus Server

Now, log in to your Main Prometheus Server. We need to tell Prometheus to start fetching data from the new Target Node.

Open the Prometheus configuration file:

Bash


vi /etc/prometheus/prometheus.yml

Scroll down to the scrape_configs section. You can either add your new node to an existing job or create a brand new job for it.

Option A: Add to an existing job

YAML


scrape_configs:
  - job_name: 'node'
    static_configs:
      # Add your existing and new node IPs/hostnames here
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100']

Option B: Create a new job group (Recommended for production environments)

YAML


scrape_configs:
  - job_name: 'linux_servers'
    static_configs:
      - targets: ['node01.example.com:9100']

(Replace node01.example.com with your Target Node's actual IP address or hostname).

Save and exit the file.

Before applying the changes, it is a modern best practice to validate the YAML configuration to avoid any syntax errors:

Bash


promtool check config /etc/prometheus/prometheus.yml

If the check is successful, apply the changes without interrupting your existing monitoring processes by reloading the Prometheus service. If your service file doesn't support reload, the command will safely fall back to restarting it:

Bash


systemctl reload prometheus || systemctl restart prometheus

Step 5 Verify the Target in the Web UI

1. Open your web browser and go to your Prometheus Web UI (http://your-prometheus-ip:9090).

2. In the top menu, click on Status > Targets.

3. You should now see your new target node listed under the job name you configured (linux_servers). If the setup is correct, its state will show as UP.

Step 6Test with PromQL Queries

You can now start querying metrics from your new node directly from the Prometheus UI. Go to the main Graph page and try running these example queries:

To check the CPU load (5 minutes) for a specific job

Bash

node_load5{job="linux_servers"}

📢Pro Tip: Many sysadmins prefer to calculate the normalized CPU load across all cores using this advanced query: node_load5 / count(count(node_cpu_seconds_total{mode="idle"}) by (cpu))

To check running processes for a specific node

Bash

 node_procs_running{instance=~"node01.example.com:9100"}

Troubleshooting

If your target shows as DOWN in the Prometheus Web UI, check the following:

Bash

 systemctl status prometheus-node-exporter

Note: The service name may vary depending on your installation method. Common names include node_exporter or prometheus-node-exporter.

Verify port 9100 is reachable and listening

Bash

 ss -tulpn | grep 9100

Verify firewall rules allow access on the Target Node

Bash

 firewall-cmd --list-ports

Verify Prometheus configuration syntax on the Main Server

Bash

 promtool check config /etc/prometheus/prometheus.yml

Conclusion

You have successfully added a new Linux target node to your Prometheus monitoring environment. By configuring Node Exporter and Prometheus scrape targets, you can centrally monitor system performance, resource usage, and server health across multiple Linux machines. As your infrastructure grows, you can continue adding additional servers using the same process and later integrate Grafana for advanced dashboards and visualization.

What to Read Next

Ready to take your monitoring stack to the next level? Check out our related guides:

Ready to Deploy Your Monitoring Stack?

You have the knowledge, now get the hardware. Run Prometheus and Node Exporter seamlessly on our high-performance Bare Metal Dedicated Servers. Engineered for 24/7 uptime and real-time data processing.

Deploy Your Dedicated Server Today

Common Mistakes to Avoid

Node Exporter Service Not Running One of the most common mistakes is forgetting to start the node_exporter service on the Target Node. Even if Node Exporter is installed, Prometheus cannot collect metrics if the service is stopped.

Solution:
Verify the service status:
systemctl status node_exporter
If it is not running, start and enable it:
systemctl enable --now node_exporter

Port 9100 Blocked by Firewall Prometheus connects to Node Exporter through port 9100. If the firewall blocks this port, the target will appear as DOWN in the Prometheus Web UI.

Solution:
Allow port 9100 through the firewall:

firewall-cmd --permanent --add-port=9100/tcp
firewall-cmd --reload

Verify the port is allowed:
firewall-cmd --list-ports

Incorrect IP Address or Hostname in prometheus.yml A typo in the target IP address or hostname is another very common issue. Prometheus will fail to scrape metrics if the address is incorrect.

Solution:
Verify the target entry inside /etc/prometheus/prometheus.yml:

scrape_configs:
 - job_name: 'linux_servers'
   static_configs:
     - targets: ['192.168.1.10:9100']

Test connectivity from the Prometheus server:
curl http://192.168.1.10:9100/metrics

YAML Syntax Errors in Prometheus Configuration Prometheus configuration files use YAML formatting, where spacing and indentation are extremely important. A small indentation mistake can prevent Prometheus from starting.

Solution:
Always validate the configuration before reloading Prometheus:

promtool check config /etc/prometheus/prometheus.yml

If the syntax is correct, you will see:
SUCCESS: 0 rule files found

SELinux Blocking Prometheus Connections On RHEL-based systems such as CentOS Stream, AlmaLinux, and Rocky Linux, SELinux policies may block Prometheus from connecting to Node Exporter.

Solution:
Temporarily test SELinux:

setenforce 0

If the target starts working, configure the proper SELinux policy instead of permanently disabling SELinux.

Check current SELinux status:
getenforce