Metrics
Pigsty INFRA module metric list
Pigsty has a battery-included, production-ready INFRA module, to provide ultimate observability.
Configuration | Administration | Playbook | Dashboard | Parameter
Each Pigsty deployment requires a set of infrastructure components to work properly. which including:
Component | Port | Domain | Description |
---|---|---|---|
Nginx | 80 | h.pigsty |
Web Service Portal (YUM/APT Repo) |
AlertManager | 9093 | a.pigsty |
Alert Aggregation and delivery |
Prometheus | 9090 | p.pigsty |
Monitoring Time Series Database |
Grafana | 3000 | g.pigsty |
Visualization Platform |
Loki | 3100 | - | Logging Collection Server |
PushGateway | 9091 | - | Collect One-Time Job Metrics |
BlackboxExporter | 9115 | - | Blackbox Probing |
Dnsmasq | 53 | - | DNS Server |
Chronyd | 123 | - | NTP Time Server |
PostgreSQL | 5432 | - | Pigsty CMDB & default database |
Ansible | - | - | Run playbooks |
Pigsty will set up these components for you on infra nodes. You can expose them to the outside world by configuring the infra_portal
parameter.
infra_portal: # domain names and upstream servers
home : { domain: h.pigsty }
grafana : { domain: g.pigsty ,endpoint: "${admin_ip}:3000" , websocket: true }
prometheus : { domain: p.pigsty ,endpoint: "${admin_ip}:9090" }
alertmanager : { domain: a.pigsty ,endpoint: "${admin_ip}:9093" }
blackbox : { endpoint: "${admin_ip}:9115" }
loki : { endpoint: "${admin_ip}:3100" }
#minio : { domain: sss.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }
To define an infra
cluster, use the hard-coded group name infra
in your inventory file.
You can use multiple nodes to deploy INFRA module, but at least one is required. You have to assign a unique infra_seq
to each node.
# Single infra node
infra: { hosts: { 10.10.10.10: { infra_seq: 1 } }}
# Two INFRA node
infra:
hosts:
10.10.10.10: { infra_seq: 1 }
10.10.10.11: { infra_seq: 2 }
Then you can init INFRA module with infra.yml
playbook.
Here are some administration tasks related to INFRA module:
./infra.yml # install infra/node module on `infra` group
./infra-rm.yml # remove infra module from `infra` group
./infra.yml -t repo # setup local yum/apt repo
./infra.yml -t repo_dir # create repo directory
./infra.yml -t repo_check # check repo exists
./infra.yml -t repo_prepare # use existing repo if exists
./infra.yml -t repo_build # build repo from upstream if not exists
./infra.yml -t repo_upstream # handle upstream repo files in /etc/yum.repos.d or /etc/apt/sources.list.d
./infra.yml -t repo_url_pkg # download packages from internet defined by repo_url_packages
./infra.yml -t repo_cache # make upstream yum/apt cache
./infra.yml -t repo_boot_pkg # install bootstrap pkg such as createrepo_c,yum-utils,... (or dpkg-dev in debian/ubuntu)
./infra.yml -t repo_pkg # download packages & dependencies from upstream repo
./infra.yml -t repo_create # create a local yum repo with createrepo_c & modifyrepo_c
./infra.yml -t repo_use # add newly built repo
./infra.yml -t repo_nginx # launch a nginx for repo if no nginx is serving
You can use the following subtasks to manage infra components:
./infra.yml -t infra_env : env_dir, env_pg, env_var
./infra.yml -t infra_pkg : infra_pkg, infra_pkg_pip
./infra.yml -t infra_user : setup infra os user group
./infra.yml -t infra_cert : issue cert for infra components
./infra.yml -t dns : dns_config, dns_record, dns_launch
./infra.yml -t nginx : nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_exporter
./infra.yml -t prometheus : prometheus_clean, prometheus_dir, prometheus_config, prometheus_launch, prometheus_reload
./infra.yml -t alertmanager : alertmanager_config, alertmanager_launch
./infra.yml -t pushgateway : pushgateway_config, pushgateway_launch
./infra.yml -t blackbox : blackbox_config, blackbox_launch
./infra.yml -t grafana : grafana_clean, grafana_config, grafana_plugin, grafana_launch, grafana_provision
./infra.yml -t loki : loki clean, loki_dir, loki_config, loki_launch
./infra.yml -t infra_register : register infra components to prometheus
./infra.yml -t nginx_index # render Nginx homepage
./infra.yml -t nginx_config,nginx_reload # render Nginx upstream server config
./infra.yml -t prometheus_conf,prometheus_reload # render Prometheus main config and reload
./infra.yml -t prometheus_rule,prometheus_reload # copy Prometheus rules & alert definition and reload
./infra.yml -t grafana_plugin # download Grafana plugins from the Internet
install.yml
: Install Pigsty on all nodes in one-passinfra.yml
: Init pigsty infrastructure on infra nodesinfra-rm.yml
: Remove infrastructure components from infra nodesinfra.yml
The playbook infra.yml
will init pigsty infrastructure on infra nodes.
It will also install NODE module on infra nodes too.
Here are available subtasks:
# ca : create self-signed CA on localhost files/pki
# - ca_dir : create CA directory
# - ca_private : generate ca private key: files/pki/ca/ca.key
# - ca_cert : signing ca cert: files/pki/ca/ca.crt
#
# id : generate node identity
#
# repo : bootstrap a local yum repo from internet or offline packages
# - repo_dir : create repo directory
# - repo_check : check repo exists
# - repo_prepare : use existing repo if exists
# - repo_build : build repo from upstream if not exists
# - repo_upstream : handle upstream repo files in /etc/yum.repos.d
# - repo_remove : remove existing repo file if repo_remove == true
# - repo_add : add upstream repo files to /etc/yum.repos.d
# - repo_url_pkg : download packages from internet defined by repo_url_packages
# - repo_cache : make upstream yum cache with yum makecache
# - repo_boot_pkg : install bootstrap pkg such as createrepo_c,yum-utils,...
# - repo_pkg : download packages & dependencies from upstream repo
# - repo_create : create a local yum repo with createrepo_c & modifyrepo_c
# - repo_use : add newly built repo into /etc/yum.repos.d
# - repo_nginx : launch a nginx for repo if no nginx is serving
#
# node/haproxy/docker/monitor : setup infra node as a common node (check node.yml)
# - node_name, node_hosts, node_resolv, node_firewall, node_ca, node_repo, node_pkg
# - node_feature, node_kernel, node_tune, node_sysctl, node_profile, node_ulimit
# - node_data, node_admin, node_timezone, node_ntp, node_crontab, node_vip
# - haproxy_install, haproxy_config, haproxy_launch, haproxy_reload
# - docker_install, docker_admin, docker_config, docker_launch, docker_image
# - haproxy_register, node_exporter, node_register, promtail
#
# infra : setup infra components
# - infra_env : env_dir, env_pg, env_var
# - infra_pkg : infra_pkg, infra_pkg_pip
# - infra_user : setup infra os user group
# - infra_cert : issue cert for infra components
# - dns : dns_config, dns_record, dns_launch
# - nginx : nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_exporter
# - prometheus : prometheus_clean, prometheus_dir, prometheus_config, prometheus_launch, prometheus_reload
# - alertmanager : alertmanager_config, alertmanager_launch
# - pushgateway : pushgateway_config, pushgateway_launch
# - blackbox : blackbox_config, blackbox_launch
# - grafana : grafana_clean, grafana_config, grafana_plugin, grafana_launch, grafana_provision
# - loki : loki clean, loki_dir, loki_config, loki_launch
# - infra_register : register infra components to prometheus
infra-rm.yml
The playbook infra-rm.yml
will remove infrastructure components from infra nodes
./infra-rm.yml # remove INFRA module
./infra-rm.yml -t service # stop INFRA services
./infra-rm.yml -t data # remove INFRA data
./infra-rm.yml -t package # uninstall INFRA packages
install.yml
The playbook install.yml
will install Pigsty on all node in one-pass.
Check Playbook: One-Pass Install for details.
Pigsty Home : Home dashboard for pigsty’s grafana
INFRA Overview : Overview of all infra components
Nginx Overview : Nginx metrics & logs
Grafana Overview: Grafana metrics & logs
Prometheus Overview: Prometheus metrics & logs
Loki Overview: Loki metrics & logs
Logs Instance: Logs for a single instance
Logs Overview: Overview of all logs
CMDB Overview: CMDB visualization
ETCD Overview: etcd metrics & logs
API Reference for INFRA
module:
META
: infra meta dataCA
: self-signed CAINFRA_ID
: Portals and identityREPO
: local yum/atp repoINFRA_PACKAGE
: packages to be installedNGINX
: nginx web serverDNS
: dnsmasq nameserverPROMETHEUS
: prometheus, alertmanager, pushgateway & blackbox_exporterGRAFANA
: Grafana, the visualization platformLOKI
: Loki, the logging serverParameter | Section | Type | Level | Comment |
---|---|---|---|---|
version |
META |
string | G | pigsty version string |
admin_ip |
META |
ip | G | admin node ip address |
region |
META |
enum | G | upstream mirror region: default,china,europe |
proxy_env |
META |
dict | G | global proxy env when downloading packages |
ca_method |
CA |
enum | G | create,recreate,copy, create by default |
ca_cn |
CA |
string | G | ca common name, fixed as pigsty-ca |
cert_validity |
CA |
interval | G | cert validity, 20 years by default |
infra_seq |
INFRA_ID |
int | I | infra node identity, REQUIRED |
infra_portal |
INFRA_ID |
dict | G | infra services exposed via portal |
repo_enabled |
REPO |
bool | G/I | create a yum/apt repo on this infra node? |
repo_home |
REPO |
path | G | repo home dir, /www by default |
repo_name |
REPO |
string | G | repo name, pigsty by default |
repo_endpoint |
REPO |
url | G | access point to this repo by domain or ip:port |
repo_remove |
REPO |
bool | G/A | remove existing upstream repo |
repo_modules |
REPO |
string | G/A | which repo modules are installed in repo_upstream |
repo_upstream |
REPO |
upstream[] | G | where to download upstream packages |
repo_packages |
REPO |
string[] | G | which packages to be included |
repo_url_packages |
REPO |
string[] | G | extra packages from url |
infra_packages |
INFRA_PACKAGE |
string[] | G | packages to be installed on infra nodes |
infra_packages_pip |
INFRA_PACKAGE |
string | G | pip installed packages for infra nodes |
nginx_enabled |
NGINX |
bool | G/I | enable nginx on this infra node? |
nginx_exporter_enabled |
NGINX |
bool | G/I | enable nginx_exporter on this infra node? |
nginx_sslmode |
NGINX |
enum | G | nginx ssl mode? disable,enable,enforce |
nginx_home |
NGINX |
path | G | nginx content dir, /www by default |
nginx_port |
NGINX |
port | G | nginx listen port, 80 by default |
nginx_ssl_port |
NGINX |
port | G | nginx ssl listen port, 443 by default |
nginx_navbar |
NGINX |
index[] | G | nginx index page navigation links |
dns_enabled |
DNS |
bool | G/I | setup dnsmasq on this infra node? |
dns_port |
DNS |
port | G | dns server listen port, 53 by default |
dns_records |
DNS |
string[] | G | dynamic dns records resolved by dnsmasq |
prometheus_enabled |
PROMETHEUS |
bool | G/I | enable prometheus on this infra node? |
prometheus_clean |
PROMETHEUS |
bool | G/A | clean prometheus data during init? |
prometheus_data |
PROMETHEUS |
path | G | prometheus data dir, /data/prometheus by default |
prometheus_sd_dir |
PROMETHEUS |
path | G | prometheus file service discovery directory |
prometheus_sd_interval |
PROMETHEUS |
interval | G | prometheus target refresh interval, 5s by default |
prometheus_scrape_interval |
PROMETHEUS |
interval | G | prometheus scrape & eval interval, 10s by default |
prometheus_scrape_timeout |
PROMETHEUS |
interval | G | prometheus global scrape timeout, 8s by default |
prometheus_options |
PROMETHEUS |
arg | G | prometheus extra server options |
pushgateway_enabled |
PROMETHEUS |
bool | G/I | setup pushgateway on this infra node? |
pushgateway_options |
PROMETHEUS |
arg | G | pushgateway extra server options |
blackbox_enabled |
PROMETHEUS |
bool | G/I | setup blackbox_exporter on this infra node? |
blackbox_options |
PROMETHEUS |
arg | G | blackbox_exporter extra server options |
alertmanager_enabled |
PROMETHEUS |
bool | G/I | setup alertmanager on this infra node? |
alertmanager_options |
PROMETHEUS |
arg | G | alertmanager extra server options |
exporter_metrics_path |
PROMETHEUS |
path | G | exporter metric path, /metrics by default |
exporter_install |
PROMETHEUS |
enum | G | how to install exporter? none,yum,binary |
exporter_repo_url |
PROMETHEUS |
url | G | exporter repo file url if install exporter via yum |
grafana_enabled |
GRAFANA |
bool | G/I | enable grafana on this infra node? |
grafana_clean |
GRAFANA |
bool | G/A | clean grafana data during init? |
grafana_admin_username |
GRAFANA |
username | G | grafana admin username, admin by default |
grafana_admin_password |
GRAFANA |
password | G | grafana admin password, pigsty by default |
grafana_plugin_cache |
GRAFANA |
path | G | path to grafana plugins cache tarball |
grafana_plugin_list |
GRAFANA |
string[] | G | grafana plugins to be downloaded with grafana-cli |
loki_enabled |
LOKI |
bool | G/I | enable loki on this infra node? |
loki_clean |
LOKI |
bool | G/A | whether remove existing loki data? |
loki_data |
LOKI |
path | G | loki data dir, /data/loki by default |
loki_retention |
LOKI |
interval | G | loki log retention period, 15d by default |
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.