监控完全体 = Prometheus + Node Exporter + cadvisor + grafana Prometheus 作为轮转数据库,从各个服务器采集数据; node-exporter 作为数据采集器,接受 prometheus 采集请求,上报主机各项参数; vadvisor 作为容器数据采集器
监控完全体 = Prometheus + Node Exporter + cadvisor + grafana
Prometheus 作为轮转数据库,从各个服务器采集数据; node-exporter 作为数据采集器,接受 prometheus 采集请求,上报主机各项参数; vadvisor 作为容器数据采集器,接受 prometheus 采集请求,上报主机 docker 各项参数; grafana 作为看板,从 prometheus 查询数据做数据可视化。
准备配置文件 /data/docker/prometheus/prometheus.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 global: scrape_interval: 15s external_labels: monitor: 'codelab-monitor' scrape_configs: - job_name: 'prometheus' scrape_interval: 5s static_configs: - targets: ['localhost:9090' ] - job_name: 'vps2' static_configs: - targets: ['10.28.0.1:9100' ]
准备数据文件夹,并设置访问权限为 nobody
1 2 3 4 5 6 7 8 9 10 mkdir -p /data/docker/prometheus/data docker run --rm quay.io/prometheus/busybox cat /etc/passwd ··· nobody:x:65534:65534:nobody:/home:/bin/falsechown 65534:65534 -R /data/docker/prometheus/data
启动 prometheus
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 $ docker run -d \ --name=prometheus \ -p 9090:9090 \ --network=myDefault \ --restart always \ --volume /data/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \ --volume /data/docker/prometheus/data:/prometheus \ prom/prometheus:v2.40.4 \ --web.enable-lifecycle \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/prometheus \ --storage.tsdb.retention=365d $ docker run -d \ --name=prometheus \ -p 9090:9090 \ --network=myDefault \ --restart always \ --volume /data/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \ --volume /data/docker/prometheus/data:/prometheus \ prom/prometheus:v2.40.4 \ --web.enable-lifecycle \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/prometheus \ --storage.tsdb.retention=365d
部署 Node Exporter 采集主机数据 可采用编译安装或包管理器直接安装。
编译安装
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 $ curl -OL https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz $ tar -xzf node_exporter-1.5.0.linux-amd64.tar.gz $ cp node_exporter-1.5.0.linux-amd64/node_exporter /usr/local/bin/ $ curl -OL https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-arm64.tar.gz $ tar -xzf node_exporter-1.5.0.linux-arm64.tar.gz $ cp node_exporter-1.5.0.linux-arm64/node_exporter /usr/local/bin/ $ sudo useradd -rs /bin/false nodeusr $ echo "[Unit] Description=Node Exporter After=network.target [Service] User=nodeusr Group=nodeusr Type=simple ExecStart=/usr/local/bin/node_exporter [Install] WantedBy=multi-user.target" > /etc/systemd/system/node_exporter.service $ sudo systemctl daemon-reload $ sudo systemctl enable node_exporter && systemctl start node_exporter
Ubuntu 等直接安装:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 $ apt search node_exporter Sorting... Done Full Text Search... Done prometheus-node-exporter-collectors/focal 0+git20200110.fc91c86-1 all Supplemental textfile collector scripts for Prometheus node_exporter $ apt install -y prometheus-node-exporter-collectors $ systemctl enable prometheus-node-exporter.service && systemctl start prometheus-node-exporter.service Synchronizing state of prometheus-node-exporter.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable prometheus-node-exporter opkg update opkg install prometheus-node-exporter-lua \ prometheus-node-exporter-lua-nat_traffic \ prometheus-node-exporter-lua-netstat \ prometheus-node-exporter-lua-openwrt \ prometheus-node-exporter-lua-wifi \ prometheus-node-exporter-lua-wifi_stations curl localhost:9100/metricscat /etc/config/prometheus-node-exporter-lua config prometheus-node-exporter-lua 'main' option listen_ipv6 '0' option listen_port '9100' option listen_interface 'lan' /etc/init.d/prometheus-node-exporter-lua restart
重载 prometheus 配置文件
1 2 3 4 5 kill -HUP $(pidof prometheus) curl -X POST http://localhost:9090/-/reload
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 $ sudo docker run \ --restart always \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:ro \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --volume=/dev/disk/:/dev/disk:ro \ --publish=8080:8080 \ --detach=true \ --name=cadvisor \ --privileged \ --device=/dev/kmsg \ --net=myDefault \ songtianlun/cadvisor:v0.46.0 $ curl http://localhost:8080
部署 grafana 1 2 3 4 5 6 docker run -d \ --name=grafana \ -p 3000:3000 \ --network=myDefault \ --restart always \ grafana/grafana
默认帐号: admin/admin
看板模版在这里找: https://grafana.com/grafana/dashboards/
参考文献