This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

模块:INFRA

可独立使用的模块,为 Pigsty 提供 NTP,DNS,Prometheus & Grafana 可观测性技术栈等基础设施服务。

配置 | 管理 | 剧本 | 监控 | 参数


概览

每一套 Pigsty 部署都会提供一套基础架构组件,为纳管的节点与数据库集群提供服务,组件包括:

组件 端口 域名 描述
Nginx 80 h.pigsty Web服务门户(也用作yum/atp仓库)
AlertManager 9093 a.pigsty 告警聚合分发
Prometheus 9090 p.pigsty 时间序列数据库(收存监控指标)
Grafana 3000 g.pigsty 可视化平台
Loki 3100 - 日志收集服务器
PushGateway 9091 - 接受一次性的任务指标
BlackboxExporter 9115 - 黑盒监控探测
DNSMASQ 53 - DNS 服务器
Chronyd 123 - NTP 时间服务器
PostgreSQL 5432 - Pigsty CMDB 和默认数据库
Ansible - - 运行剧本

pigsty-arch.jpg

在 Pigsty 中,PGSQL 模块会使用到INFRA节点上的一些服务,具体来说包括:

  • 数据库集群/主机节点的域名,依赖INFRA节点的 DNSMASQ 解析
  • 在数据库节点软件上安装,需要用到INFRA节点上的Nginx托管的本地 yum/apt 软件源。
  • 数据库集群/节点的监控指标,会被INFRA节点的 Prometheus 收集抓取。
  • 数据库节点的日志会被 Promtail 收集,并发往 INFRA节点上的 Loki。
  • 用户会从 Infra/Admin 节点上使用 Ansible 或其他工具发起对数据库节点的管理
    • 执行集群创建,扩缩容,实例/集群回收
    • 创建业务用户、业务数据库、修改服务、HBA修改;
    • 执行日志采集、垃圾清理,备份,巡检等
  • 数据库节点默认会从INFRA/ADMIN节点上的 NTP 服务器同步时间
  • 如果没有专用集群,高可用组件 Patroni 会使用 INFRA 节点上的 etcd 作为高可用DCS。
  • 如果没有专用集群,备份组件 pgbackrest 会使用 INFRA 节点上的 minio 作为可选的集中备份仓库。

Nginx

Nginx是Pigsty所有WebUI类服务的访问入口,默认使用管理节点80端口。

有许多带有WebUI的基础设施组件通过Nginx对外暴露服务,例如Grafana,Prometheus,AlertManager,以及HAProxy流量管理页等,此外 yum/apt 仓库等静态文件资源也通过Nginx对外提供服务。

Nginx会根据 infra_portal 的内容,通过域名进行区分,将访问请求转发至对应的上游组件处理。如果您使用了其他的域名,或者公网域名,可以在这里进行相应修改:

infra_portal:  # domain names and upstream servers
  home         : { domain: h.pigsty }
  grafana      : { domain: g.pigsty ,endpoint: "${admin_ip}:3000" , websocket: true }
  prometheus   : { domain: p.pigsty ,endpoint: "${admin_ip}:9090" }
  alertmanager : { domain: a.pigsty ,endpoint: "${admin_ip}:9093" }
  blackbox     : { endpoint: "${admin_ip}:9115" }
  loki         : { endpoint: "${admin_ip}:3100" }
  #minio        : { domain: sss.pigsty  ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

Pigsty强烈建议使用域名访问Pigsty UI系统,而不是直接通过IP+端口的方式访问,基于以下几个理由:

  • 使用域名便于启用 HTTPS 流量加密,可以将访问收拢至Nginx,审计一切请求,并方便地集成认证机制。
  • 一些组件默认只监听 127.0.0.1 ,因此只能通过Nginx代理访问。
  • 域名更容易记忆,并提供了额外的配置灵活性。

如果您没有可用的互联网域名或本地DNS解析,您可以在 /etc/hosts (MacOS/Linux)或C:\Windows\System32\drivers\etc\hosts (Windows)中添加本地静态解析记录。

Nginx相关配置参数位于:配置:INFRA - NGINX


本地软件仓库

Pigsty会在安装时首先建立一个本地软件源,以加速后续软件安装。

该软件源由Nginx提供服务,默认位于为 /www/pigsty,可以访问 http://h.pigsty/pigsty 使用。

Pigsty的离线软件包即是将已经建立好的软件源目录(yum/apt)整个打成压缩包,当Pigsty尝试构建本地源时,如果发现本地源目录 /www/pigsty 已经存在,且带有 /www/pigsty/repo_complete 标记文件,则会认为本地源已经构建完成,从而跳过从原始上游下载软件的步骤,消除了对互联网访问的依赖。

Repo定义文件位于 /www/pigsty.repo,默认可以通过 http://${admin_ip}/pigsty.repo 获取

curl -L http://h.pigsty/pigsty.repo -o /etc/yum.repos.d/pigsty.repo

您也可以在没有Nginx的情况下直接使用文件本地源:

[pigsty-local]
name=Pigsty local $releasever - $basearch
baseurl=file:///www/pigsty/
enabled=1
gpgcheck=0

本地软件仓库相关配置参数位于:配置:INFRA - REPO


Prometheus

Prometheus是监控时序数据库,默认监听9090端口,可以直接通过IP:9090或域名http://p.pigsty访问。

Prometheus是监控用时序数据库,提供以下功能:

  • Prometheus默认通过本地静态文件服务发现获取监控对象,并为其关联身份信息。
  • Prometheus从Exporter拉取监控指标数据,进行预计算加工后存入自己的TSDB中。
  • Prometheus计算报警规则,将报警事件发往Alertmanager处理。

AlertManager是与Prometheus配套的告警平台,默认监听9093端口,可以直接通过IP:9093或域名 http://a.pigsty 访问。 Prometheus的告警事件会发送至AlertManager,但如果需要进一步处理,用户需要进一步对其进行配置,例如提供SMTP服务配置以发送告警邮件。

Prometheus、AlertManager,PushGateway,BlackboxExporter 的相关配置参数位于:配置:INFRA - PROMETHEUS


Grafana

Grafana是开源的可视化/监控平台,是Pigsty WebUI的核心,默认监听3000端口,可以直接通过IP:3000或域名http://g.pigsty访问。

Pigsty的监控系统基于Dashboard构建,通过URL进行连接与跳转。您可以快速地在监控中下钻上卷,快速定位故障与问题。

此外,Grafana还可以用作通用的低代码前后端平台,制作交互式可视化数据应用。因此,Pigsty使用的Grafana带有一些额外的可视化插件,例如ECharts面板。

Loki是用于日志收集的日志数据库,默认监听3100端口,节点上的Promtail向元节点上的Loki推送日志。

Grafana与Loki相关配置参数位于:配置:INFRA - GRAFANA配置:INFRA - Loki


Ansible

Pigsty默认会在元节点上安装Ansible,Ansible是一个流行的运维工具,采用声明式的配置风格与幂等的剧本设计,可以极大降低系统维护的复杂度。


DNSMASQ

DNSMASQ 提供环境内的DNS解析服务,其他模块的域名将会注册到 INFRA节点上的 DNSMASQ 服务中。

DNS记录默认放置于所有INFRA节点的 /etc/hosts.d/ 目录中。

DNSMASQ相关配置参数位于:配置:INFRA - DNS


Chronyd

NTP服务用于同步环境内所有节点的时间(可选)

NTP相关配置参数位于:配置:NODES - NTP


配置

要在节点上安装 INFRA 模块,首先需要在配置清单中的 infra 分组中将其加入,并分配实例号 infra_seq

# 配置单个 INFRA 节点
infra: { hosts: { 10.10.10.10: { infra_seq: 1 } }}

# 配置两个 INFRA 节点
infra:
  hosts:
    10.10.10.10: { infra_seq: 1 }
    10.10.10.11: { infra_seq: 2 }

然后,使用 infra.yml 剧本在节点上初始化 INFRA 模块即可。


管理

下面是与 INFRA 模块相关的一些管理任务:


安装卸载Infra模块

./infra.yml     # 在 infra 分组上安装 INFRA 模块
./infra-rm.yml  # 从 infra 分组上卸载 INFRA 模块

管理本地软件仓库

您可以使用以下剧本子任务,管理 Infra节点 上的本地yun源:

./infra.yml -t repo              #从互联网或离线包中创建本地软件源

./infra.yml -t repo_dir          # 创建本地软件源
./infra.yml -t repo_check        # 检查本地软件源是否已经存在?
./infra.yml -t repo_prepare      # 如果存在,直接使用已有的本地软件源
./infra.yml -t repo_build        # 如果不存在,从上游构建本地软件源
./infra.yml     -t repo_upstream     # 处理 /etc/yum.repos.d 中的上游仓库文件
./infra.yml     -t repo_remove       # 如果 repo_remove == true,则删除现有的仓库文件
./infra.yml     -t repo_add          # 将上游仓库文件添加到 /etc/yum.repos.d (或 /etc/apt/sources.list.d)
./infra.yml     -t repo_url_pkg      # 从由 repo_url_packages 定义的互联网下载包
./infra.yml     -t repo_cache        # 使用 yum makecache / apt update 创建上游软件源元数据缓存
./infra.yml     -t repo_boot_pkg     # 安装如 createrepo_c、yum-utils 等的引导包...(或 dpkg-)
./infra.yml     -t repo_pkg          # 从上游仓库下载包 & 依赖项
./infra.yml     -t repo_create       # 使用 createrepo_c & modifyrepo_c 创建本地软件源
./infra.yml     -t repo_use          # 将新建的仓库添加到 /etc/yum.repos.d | /etc/apt/sources.list.d 用起来
./infra.yml -t repo_nginx        # 如果没有 nginx 在服务,启动一个 nginx 作为 Web Server

其中最常用的命令为:

./infra.yml     -t repo_upstream     # 向 INFRA 节点添加 repo_upstream 中定义的上游软件源
./infra.yml     -t repo_pkg          # 从上游仓库下载包及其依赖项。
./infra.yml     -t repo_create       # 使用 createrepo_c & modifyrepo_c 创建/更新本地 yum 仓库

管理基础设施组件

您可以使用以下剧本子任务,管理 Infra节点 上的各个基础设施组件

./infra.yml -t infra           # 配置基础设施
./infra.yml -t infra_env       # 配置管理节点上的环境变量:env_dir, env_pg, env_var
./infra.yml -t infra_pkg       # 安装INFRA所需的软件包:infra_pkg_yum, infra_pkg_pip
./infra.yml -t infra_user      # 设置 infra 操作系统用户组
./infra.yml -t infra_cert      # 为 infra 组件颁发证书
./infra.yml -t dns             # 配置 DNSMasq:dns_config, dns_record, dns_launch
./infra.yml -t nginx           # 配置 Nginx:nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_exporter
./infra.yml -t prometheus      # 配置 Prometheus:prometheus_clean, prometheus_dir, prometheus_config, prometheus_launch, prometheus_reload
./infra.yml -t alertmanager    # 配置 AlertManager:alertmanager_config, alertmanager_launch
./infra.yml -t pushgateway     # 配置 PushGateway:pushgateway_config, pushgateway_launch
./infra.yml -t blackbox        # 配置 Blackbox Exporter: blackbox_launch
./infra.yml -t grafana         # 配置 Grafana:grafana_clean, grafana_config, grafana_plugin, grafana_launch, grafana_provision
./infra.yml -t loki            # 配置 Loki:loki_clean, loki_dir, loki_config, loki_launch
./infra.yml -t infra_register  # 将 infra 组件注册到 prometheus

其他常用的任务包括:

./infra.yml -t nginx_index                        # 重新渲染 Nginx 首页内容
./infra.yml -t nginx_config,nginx_reload          # 重新渲染 Nginx 网站门户配置,对外暴露新的上游服务。
./infra.yml -t prometheus_conf,prometheus_reload  # 重新生成 Prometheus 主配置文件,并重载配置
./infra.yml -t prometheus_rule,prometheus_reload  # 重新拷贝 Prometheus 规则 & 告警,并重载配置
./infra.yml -t grafana_plugin                     # 从互联网上下载 Grafana 插件,通常需要科学上网

剧本

Pigsty 提供了三个与 INFRA 模块相关的剧本:

  • infra.yml :在 infra 节点上初始化 pigsty 基础设施
  • infra-rm.yml:从 infra 节点移除基础设施组件
  • install.yml:在当前节点上一次性完整安装 Pigsty

infra.yml

INFRA模块剧本 infra.yml 用于在 Infra节点 上初始化 pigsty 基础设施

执行该剧本将完成以下任务

  • 配置元节点的目录与环境变量
  • 下载并建立一个本地软件源,加速后续安装。(若使用离线软件包,则跳过下载阶段)
  • 将当前元节点作为一个普通节点纳入 Pigsty 管理
  • 部署基础设施组件,包括 Prometheus, Grafana, Loki, Alertmanager, PushGateway,Blackbox Exporter 等

该剧本默认在 INFRA节点 上执行

  • Pigsty默认将使用当前执行此剧本的节点作为Pigsty的Infra节点与Admin节点。
  • Pigsty在配置过程中默认会将当前节点标记为Infra/Admin节点,并使用当前节点首要IP地址替换配置模板中的占位IP地址10.10.10.10
  • 该节点除了可以发起管理,部署有基础设施,与一个部署普通托管节点并无区别。
  • 单机安装时,ETCD 也会安装在此节点上,提供 DCS 服务

本剧本的一些注意事项

  • 本剧本为幂等剧本,重复执行会抹除元节点上的基础设施组件。
  • 当离线软件源 /www/pigsty/repo_complete 存在时,本剧本会跳过从互联网下载软件的任务。完整执行该剧本耗时约5-8分钟,视机器配置而异。
  • 不使用离线软件包而直接从互联网原始上游下载软件时,可能耗时10-20分钟,根据您的网络条件而异。

asciicast


infra-rm.yml

INFRA模块剧本 infra-rm.yml 用于从 Infra节点 上移除 pigsty 基础设施

常用子任务包括:

./infra-rm.yml               # 移除 INFRA 模块
./infra-rm.yml -t service    # 停止 INFRA 上的基础设施服务
./infra-rm.yml -t data       # 移除 INFRA 上的存留数据
./infra-rm.yml -t package    # 卸载 INFRA 上安装的软件包

install.yml

INFRA模块剧本 install.yml用于在所有节点上一次性完整安装 Pigsty

该剧本在 剧本:一次性安装 中有更详细的介绍。


监控

Pigsty Home : Pigsty 监控系统主页

Pigsty Home Dashboard

pigsty.jpg

INFRA Overview : Pigsty 基础设施自监控概览

INFRA Overview Dashboard

infra-overview.jpg

Nginx Overview : Nginx 监控指标与日志

Nginx Overview Dashboard

nginx-overview.jpg

Grafana Overview: Grafana 监控指标与日志

Grafana Overview Dashboard

grafana-overview.jpg

Prometheus Overview: Prometheus 监控指标与日志

Prometheus Overview Dashboard

prometheus-overview.jpg

Loki Overview: Loki 监控指标与日志

Loki Overview Dashboard

loki-overview.jpg

Logs Instance: 查阅单个节点上的日志信息

Logs Instance Dashboard

logs-instance.jpg

Logs Overview: 查阅全局日志信息

Logs Overview Dashboard

logs-overview.jpg

CMDB Overview: CMDB 可视化

CMDB Overview Dashboard

cmdb-overview.jpg

ETCD Overview: etcd 监控指标与日志

ETCD Overview Dashboard

etcd-overview.jpg


参数

INFRA 模块有下列10个参数组。

  • META:Pigsty元数据
  • CA:自签名公私钥基础设施/CA
  • INFRA_ID:基础设施门户,Nginx域名
  • REPO:本地软件源
  • INFRA_PACKAGE:基础设施软件包
  • NGINX:Nginx 网络服务器
  • DNS:DNSMASQ 域名服务器
  • PROMETHEUS:Prometheus 时序数据库全家桶
  • GRAFANA:Grafana 可观测性全家桶
  • LOKI:Loki 日志服务
完整参数列表
参数 参数组 类型 级别 说明
version META string G pigsty 版本字符串
admin_ip META ip G 管理节点 IP 地址
region META enum G 上游镜像区域:default,china,europe
proxy_env META dict G 下载包时使用的全局代理环境变量
ca_method CA enum G CA处理方式:create,recreate,copy,默认为没有则创建
ca_cn CA string G CA CN名称,固定为 pigsty-ca
cert_validity CA interval G 证书有效期,默认为 20 年
infra_seq INFRA_ID int I 基础设施节号,必选身份参数
infra_portal INFRA_ID dict G 通过Nginx门户暴露的基础设施服务列表
repo_enabled REPO bool G/I 在此基础设施节点上创建本地软件源?
repo_home REPO path G 软件仓库主目录,默认为`/www``
repo_name REPO string G 软件仓库名称,默认为 pigsty
repo_endpoint REPO url G 仓库的访问点:域名或 ip:port 格式
repo_remove REPO bool G/A 构建本地仓库时是否移除现有上游仓库源定义文件?
repo_modules REPO string G/A 启用的上游仓库模块列表,用逗号分隔
repo_upstream REPO upstream[] G 上游仓库源定义:从哪里下载上游包?
repo_packages REPO string[] G 从上游仓库下载哪些软件包?
repo_url_packages REPO string[] G 使用URL下载的额外软件包列表
infra_packages INFRA_PACKAGE string[] G 在基础设施节点上要安装的软件包
infra_packages_pip INFRA_PACKAGE string G 在基础设施节点上使用 pip 安装的包
nginx_enabled NGINX bool G/I 在此基础设施节点上启用 nginx?
nginx_exporter_enabled NGINX bool G/I 在此基础设施节点上启用 nginx_exporter?
nginx_sslmode NGINX enum G nginx SSL模式?disable,enable,enforce
nginx_home NGINX path G nginx 内容目录,默认为 /www,通常和仓库目录一致
nginx_port NGINX port G nginx 监听端口,默认为 80
nginx_ssl_port NGINX port G nginx SSL监听端口,默认为 443
nginx_navbar NGINX index[] G nginx 首页导航链接列表
dns_enabled DNS bool G/I 在此基础设施节点上设置dnsmasq?
dns_port DNS port G DNS 服务器监听端口,默认为 53
dns_records DNS string[] G 由 dnsmasq 解析的动态 DNS 记录
prometheus_enabled PROMETHEUS bool G/I 在此基础设施节点上启用 prometheus?
prometheus_clean PROMETHEUS bool G/A 初始化Prometheus的时候清除现有数据?
prometheus_data PROMETHEUS path G Prometheus 数据目录,默认为 `/data/prometheus``
prometheus_sd_interval PROMETHEUS interval G Prometheus 目标刷新间隔,默认为 5s
prometheus_scrape_interval PROMETHEUS interval G Prometheus 抓取 & 评估间隔,默认为 10s
prometheus_scrape_timeout PROMETHEUS interval G Prometheus 全局抓取超时,默认为 8s
prometheus_options PROMETHEUS arg G Prometheus 额外的命令行参数选项
pushgateway_enabled PROMETHEUS bool G/I 在此基础设施节点上设置 pushgateway?
pushgateway_options PROMETHEUS arg G pushgateway 额外的命令行参数选项
blackbox_enabled PROMETHEUS bool G/I 在此基础设施节点上设置 blackbox_exporter?
blackbox_options PROMETHEUS arg G blackbox_exporter 额外的命令行参数选项
alertmanager_enabled PROMETHEUS bool G/I 在此基础设施节点上设置 alertmanager?
alertmanager_options PROMETHEUS arg G alertmanager 额外的命令行参数选项
exporter_metrics_path PROMETHEUS path G exporter 指标路径,默认为 /metrics
exporter_install PROMETHEUS enum G 如何安装 exporter?none,yum,binary
exporter_repo_url PROMETHEUS url G 通过 yum 安装exporter时使用的yum仓库文件地址
grafana_enabled GRAFANA bool G/I 在此基础设施节点上启用 Grafana?
grafana_clean GRAFANA bool G/A 初始化Grafana期间清除数据?
grafana_admin_username GRAFANA username G Grafana 管理员用户名,默认为 `admin``
grafana_admin_password GRAFANA password G Grafana 管理员密码,默认为 `pigsty``
grafana_plugin_cache GRAFANA path G Grafana 插件缓存 tarball 的路径
grafana_plugin_list GRAFANA string[] G 使用 grafana-cli 下载的 Grafana 插件
loki_enabled LOKI bool G/I 在此基础设施节点上启用 loki?
loki_clean LOKI bool G/A 是否删除现有的 loki 数据?
loki_data LOKI path G loki 数据目录,默认为 `/data/loki``
loki_retention LOKI interval G loki 日志保留期,默认为 15d

1 - 指标列表

Pigsty INFRA 模块提供的完整监控指标列表与释义

INFRA 指标

INFRA 模块包含有 964 类可用监控指标。

Metric Name Type Labels Description
alertmanager_alerts gauge ins, instance, ip, job, cls, state How many alerts by state.
alertmanager_alerts_invalid_total counter version, ins, instance, ip, job, cls The total number of received alerts that were invalid.
alertmanager_alerts_received_total counter version, ins, instance, ip, status, job, cls The total number of received alerts.
alertmanager_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which alertmanager was built, and the goos and goarch for the build.
alertmanager_cluster_alive_messages_total counter ins, instance, ip, peer, job, cls Total number of received alive messages.
alertmanager_cluster_enabled gauge ins, instance, ip, job, cls Indicates whether the clustering is enabled or not.
alertmanager_cluster_failed_peers gauge ins, instance, ip, job, cls Number indicating the current number of failed peers in the cluster.
alertmanager_cluster_health_score gauge ins, instance, ip, job, cls Health score of the cluster. Lower values are better and zero means ’totally healthy’.
alertmanager_cluster_members gauge ins, instance, ip, job, cls Number indicating current number of members in cluster.
alertmanager_cluster_messages_pruned_total counter ins, instance, ip, job, cls Total number of cluster messages pruned.
alertmanager_cluster_messages_queued gauge ins, instance, ip, job, cls Number of cluster messages which are queued.
alertmanager_cluster_messages_received_size_total counter ins, instance, ip, msg_type, job, cls Total size of cluster messages received.
alertmanager_cluster_messages_received_total counter ins, instance, ip, msg_type, job, cls Total number of cluster messages received.
alertmanager_cluster_messages_sent_size_total counter ins, instance, ip, msg_type, job, cls Total size of cluster messages sent.
alertmanager_cluster_messages_sent_total counter ins, instance, ip, msg_type, job, cls Total number of cluster messages sent.
alertmanager_cluster_peer_info gauge ins, instance, ip, peer, job, cls A metric with a constant ‘1’ value labeled by peer name.
alertmanager_cluster_peers_joined_total counter ins, instance, ip, job, cls A counter of the number of peers that have joined.
alertmanager_cluster_peers_left_total counter ins, instance, ip, job, cls A counter of the number of peers that have left.
alertmanager_cluster_peers_update_total counter ins, instance, ip, job, cls A counter of the number of peers that have updated metadata.
alertmanager_cluster_reconnections_failed_total counter ins, instance, ip, job, cls A counter of the number of failed cluster peer reconnection attempts.
alertmanager_cluster_reconnections_total counter ins, instance, ip, job, cls A counter of the number of cluster peer reconnections.
alertmanager_cluster_refresh_join_failed_total counter ins, instance, ip, job, cls A counter of the number of failed cluster peer joined attempts via refresh.
alertmanager_cluster_refresh_join_total counter ins, instance, ip, job, cls A counter of the number of cluster peer joined via refresh.
alertmanager_config_hash gauge ins, instance, ip, job, cls Hash of the currently loaded alertmanager configuration.
alertmanager_config_last_reload_success_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last successful configuration reload.
alertmanager_config_last_reload_successful gauge ins, instance, ip, job, cls Whether the last configuration reload attempt was successful.
alertmanager_dispatcher_aggregation_groups gauge ins, instance, ip, job, cls Number of active aggregation groups
alertmanager_dispatcher_alert_processing_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_dispatcher_alert_processing_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_http_concurrency_limit_exceeded_total counter ins, instance, method, ip, job, cls Total number of times an HTTP request failed because the concurrency limit was reached.
alertmanager_http_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, handler N/A
alertmanager_http_request_duration_seconds_count Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_http_request_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_http_requests_in_flight gauge ins, instance, method, ip, job, cls Current number of HTTP requests being processed.
alertmanager_http_response_size_bytes_bucket Unknown ins, instance, method, ip, le, job, cls, handler N/A
alertmanager_http_response_size_bytes_count Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_http_response_size_bytes_sum Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_integrations gauge ins, instance, ip, job, cls Number of configured integrations.
alertmanager_marked_alerts gauge ins, instance, ip, job, cls, state How many alerts by state are currently marked in the Alertmanager regardless of their expiry.
alertmanager_nflog_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_gossip_messages_propagated_total counter ins, instance, ip, job, cls Number of received gossip messages that have been further gossiped.
alertmanager_nflog_maintenance_errors_total counter ins, instance, ip, job, cls How many maintenances were executed for the notification log that failed.
alertmanager_nflog_maintenance_total counter ins, instance, ip, job, cls How many maintenances were executed for the notification log.
alertmanager_nflog_queries_total counter ins, instance, ip, job, cls Number of notification log queries were received.
alertmanager_nflog_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
alertmanager_nflog_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_query_errors_total counter ins, instance, ip, job, cls Number notification log received queries that failed.
alertmanager_nflog_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last notification log snapshot in bytes.
alertmanager_notification_latency_seconds_bucket Unknown integration, ins, instance, ip, le, job, cls N/A
alertmanager_notification_latency_seconds_count Unknown integration, ins, instance, ip, job, cls N/A
alertmanager_notification_latency_seconds_sum Unknown integration, ins, instance, ip, job, cls N/A
alertmanager_notification_requests_failed_total counter integration, ins, instance, ip, job, cls The total number of failed notification requests.
alertmanager_notification_requests_total counter integration, ins, instance, ip, job, cls The total number of attempted notification requests.
alertmanager_notifications_failed_total counter integration, ins, instance, ip, reason, job, cls The total number of failed notifications.
alertmanager_notifications_total counter integration, ins, instance, ip, job, cls The total number of attempted notifications.
alertmanager_oversize_gossip_message_duration_seconds_bucket Unknown ins, instance, ip, le, key, job, cls N/A
alertmanager_oversize_gossip_message_duration_seconds_count Unknown ins, instance, ip, key, job, cls N/A
alertmanager_oversize_gossip_message_duration_seconds_sum Unknown ins, instance, ip, key, job, cls N/A
alertmanager_oversized_gossip_message_dropped_total counter ins, instance, ip, key, job, cls Number of oversized gossip messages that were dropped due to a full message queue.
alertmanager_oversized_gossip_message_failure_total counter ins, instance, ip, key, job, cls Number of oversized gossip message sends that failed.
alertmanager_oversized_gossip_message_sent_total counter ins, instance, ip, key, job, cls Number of oversized gossip message sent.
alertmanager_peer_position gauge ins, instance, ip, job, cls Position the Alertmanager instance believes it’s in. The position determines a peer’s behavior in the cluster.
alertmanager_receivers gauge ins, instance, ip, job, cls Number of configured receivers.
alertmanager_silences gauge ins, instance, ip, job, cls, state How many silences by state.
alertmanager_silences_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_gossip_messages_propagated_total counter ins, instance, ip, job, cls Number of received gossip messages that have been further gossiped.
alertmanager_silences_maintenance_errors_total counter ins, instance, ip, job, cls How many maintenances were executed for silences that failed.
alertmanager_silences_maintenance_total counter ins, instance, ip, job, cls How many maintenances were executed for silences.
alertmanager_silences_queries_total counter ins, instance, ip, job, cls How many silence queries were received.
alertmanager_silences_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
alertmanager_silences_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_query_errors_total counter ins, instance, ip, job, cls How many silence received queries did not succeed.
alertmanager_silences_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last silence snapshot in bytes.
blackbox_exporter_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which blackbox_exporter was built, and the goos and goarch for the build.
blackbox_exporter_config_last_reload_success_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last successful configuration reload.
blackbox_exporter_config_last_reload_successful gauge ins, instance, ip, job, cls Blackbox exporter config loaded successfully.
blackbox_module_unknown_total counter ins, instance, ip, job, cls Count of unknown modules requested by probes
cortex_distributor_ingester_clients gauge ins, instance, ip, job, cls The current number of ingester clients.
cortex_dns_failures_total Unknown ins, instance, ip, job, cls N/A
cortex_dns_lookups_total Unknown ins, instance, ip, job, cls N/A
cortex_frontend_query_range_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, status_code N/A
cortex_frontend_query_range_duration_seconds_count Unknown ins, instance, method, ip, job, cls, status_code N/A
cortex_frontend_query_range_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, status_code N/A
cortex_ingester_flush_queue_length gauge ins, instance, ip, job, cls The total number of series pending in the flush queue.
cortex_kv_request_duration_seconds_bucket Unknown ins, instance, role, ip, le, kv_name, type, operation, job, cls, status_code N/A
cortex_kv_request_duration_seconds_count Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
cortex_kv_request_duration_seconds_sum Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
cortex_member_consul_heartbeats_total Unknown ins, instance, ip, job, cls N/A
cortex_prometheus_notifications_alertmanagers_discovered gauge ins, instance, ip, user, job, cls The number of alertmanagers discovered and active.
cortex_prometheus_notifications_dropped_total Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_notifications_queue_capacity gauge ins, instance, ip, user, job, cls The capacity of the alert notifications queue.
cortex_prometheus_notifications_queue_length gauge ins, instance, ip, user, job, cls The number of alert notifications in the queue.
cortex_prometheus_rule_evaluation_duration_seconds summary ins, instance, ip, user, job, cls, quantile The duration for a rule to execute.
cortex_prometheus_rule_evaluation_duration_seconds_count Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_rule_evaluation_duration_seconds_sum Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_rule_group_duration_seconds summary ins, instance, ip, user, job, cls, quantile The duration of rule group evaluations.
cortex_prometheus_rule_group_duration_seconds_count Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_rule_group_duration_seconds_sum Unknown ins, instance, ip, user, job, cls N/A
cortex_query_frontend_connected_schedulers gauge ins, instance, ip, job, cls Number of schedulers this frontend is connected to.
cortex_query_frontend_queries_in_progress gauge ins, instance, ip, job, cls Number of queries in progress handled by this frontend.
cortex_query_frontend_retries_bucket Unknown ins, instance, ip, le, job, cls N/A
cortex_query_frontend_retries_count Unknown ins, instance, ip, job, cls N/A
cortex_query_frontend_retries_sum Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_connected_frontend_clients gauge ins, instance, ip, job, cls Number of query-frontend worker clients currently connected to the query-scheduler.
cortex_query_scheduler_connected_querier_clients gauge ins, instance, ip, job, cls Number of querier worker clients currently connected to the query-scheduler.
cortex_query_scheduler_inflight_requests summary ins, instance, ip, job, cls, quantile Number of inflight requests (either queued or processing) sampled at a regular interval. Quantile buckets keep track of inflight requests over the last 60s.
cortex_query_scheduler_inflight_requests_count Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_inflight_requests_sum Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_queue_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
cortex_query_scheduler_queue_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_queue_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_queue_length Unknown ins, instance, ip, user, job, cls N/A
cortex_query_scheduler_running gauge ins, instance, ip, job, cls Value will be 1 if the scheduler is in the ReplicationSet and actively receiving/processing requests
cortex_ring_member_heartbeats_total Unknown ins, instance, ip, job, cls N/A
cortex_ring_member_tokens_owned gauge ins, instance, ip, job, cls The number of tokens owned in the ring.
cortex_ring_member_tokens_to_own gauge ins, instance, ip, job, cls The number of tokens to own in the ring.
cortex_ring_members gauge ins, instance, ip, job, cls, state Number of members in the ring
cortex_ring_oldest_member_timestamp gauge ins, instance, ip, job, cls, state Timestamp of the oldest member in the ring.
cortex_ring_tokens_total gauge ins, instance, ip, job, cls Number of tokens in the ring
cortex_ruler_clients gauge ins, instance, ip, job, cls The current number of ruler clients in the pool.
cortex_ruler_config_last_reload_successful gauge ins, instance, ip, user, job, cls Boolean set to 1 whenever the last configuration reload attempt was successful.
cortex_ruler_config_last_reload_successful_seconds gauge ins, instance, ip, user, job, cls Timestamp of the last successful configuration reload.
cortex_ruler_config_updates_total Unknown ins, instance, ip, user, job, cls N/A
cortex_ruler_managers_total gauge ins, instance, ip, job, cls Total number of managers registered and running in the ruler
cortex_ruler_ring_check_errors_total Unknown ins, instance, ip, job, cls N/A
cortex_ruler_sync_rules_total Unknown ins, instance, ip, reason, job, cls N/A
deprecated_flags_inuse_total Unknown ins, instance, ip, job, cls N/A
go_cgo_go_to_c_calls_calls_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_mark_assist_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_mark_dedicated_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_mark_idle_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_pause_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_total_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_idle_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_scavenge_assist_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_scavenge_background_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_scavenge_total_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_total_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_user_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_gc_cycles_automatic_gc_cycles_total Unknown ins, instance, ip, job, cls N/A
go_gc_cycles_forced_gc_cycles_total Unknown ins, instance, ip, job, cls N/A
go_gc_cycles_total_gc_cycles_total Unknown ins, instance, ip, job, cls N/A
go_gc_duration_seconds summary ins, instance, ip, job, cls, quantile A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
go_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
go_gc_gogc_percent gauge ins, instance, ip, job, cls Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function.
go_gc_gomemlimit_bytes gauge ins, instance, ip, job, cls Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function.
go_gc_heap_allocs_by_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
go_gc_heap_allocs_by_size_bytes_count Unknown ins, instance, ip, job, cls N/A
go_gc_heap_allocs_by_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
go_gc_heap_allocs_bytes_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_allocs_objects_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_by_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
go_gc_heap_frees_by_size_bytes_count Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_by_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_bytes_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_objects_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_goal_bytes gauge ins, instance, ip, job, cls Heap size target for the end of the GC cycle.
go_gc_heap_live_bytes gauge ins, instance, ip, job, cls Heap memory occupied by live objects that were marked by the previous GC.
go_gc_heap_objects_objects gauge ins, instance, ip, job, cls Number of objects, live or unswept, occupying heap memory.
go_gc_heap_tiny_allocs_objects_total Unknown ins, instance, ip, job, cls N/A
go_gc_limiter_last_enabled_gc_cycle gauge ins, instance, ip, job, cls GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC’s CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled.
go_gc_pauses_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
go_gc_pauses_seconds_count Unknown ins, instance, ip, job, cls N/A
go_gc_pauses_seconds_sum Unknown ins, instance, ip, job, cls N/A
go_gc_scan_globals_bytes gauge ins, instance, ip, job, cls The total amount of global variable space that is scannable.
go_gc_scan_heap_bytes gauge ins, instance, ip, job, cls The total amount of heap space that is scannable.
go_gc_scan_stack_bytes gauge ins, instance, ip, job, cls The number of bytes of stack that were scanned last GC cycle.
go_gc_scan_total_bytes gauge ins, instance, ip, job, cls The total amount space that is scannable. Sum of all metrics in /gc/scan.
go_gc_stack_starting_size_bytes gauge ins, instance, ip, job, cls The stack size of new goroutines.
go_godebug_non_default_behavior_execerrdot_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_gocachehash_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_gocachetest_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_gocacheverify_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_http2client_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_http2server_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_installgoroot_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_jstmpllitinterp_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_multipartmaxheaders_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_multipartmaxparts_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_multipathtcp_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_panicnil_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_randautoseed_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_tarinsecurepath_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_tlsmaxrsasize_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_x509sha1_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_x509usefallbackroots_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_zipinsecurepath_events_total Unknown ins, instance, ip, job, cls N/A
go_goroutines gauge ins, instance, ip, job, cls Number of goroutines that currently exist.
go_info gauge version, ins, instance, ip, job, cls Information about the Go environment.
go_memory_classes_heap_free_bytes gauge ins, instance, ip, job, cls Memory that is completely free and eligible to be returned to the underlying system, but has not been. This metric is the runtime’s estimate of free address space that is backed by physical memory.
go_memory_classes_heap_objects_bytes gauge ins, instance, ip, job, cls Memory occupied by live objects and dead objects that have not yet been marked free by the garbage collector.
go_memory_classes_heap_released_bytes gauge ins, instance, ip, job, cls Memory that is completely free and has been returned to the underlying system. This metric is the runtime’s estimate of free address space that is still mapped into the process, but is not backed by physical memory.
go_memory_classes_heap_stacks_bytes gauge ins, instance, ip, job, cls Memory allocated from the heap that is reserved for stack space, whether or not it is currently in-use. Currently, this represents all stack memory for goroutines. It also includes all OS thread stacks in non-cgo programs. Note that stacks may be allocated differently in the future, and this may change.
go_memory_classes_heap_unused_bytes gauge ins, instance, ip, job, cls Memory that is reserved for heap objects but is not currently used to hold heap objects.
go_memory_classes_metadata_mcache_free_bytes gauge ins, instance, ip, job, cls Memory that is reserved for runtime mcache structures, but not in-use.
go_memory_classes_metadata_mcache_inuse_bytes gauge ins, instance, ip, job, cls Memory that is occupied by runtime mcache structures that are currently being used.
go_memory_classes_metadata_mspan_free_bytes gauge ins, instance, ip, job, cls Memory that is reserved for runtime mspan structures, but not in-use.
go_memory_classes_metadata_mspan_inuse_bytes gauge ins, instance, ip, job, cls Memory that is occupied by runtime mspan structures that are currently being used.
go_memory_classes_metadata_other_bytes gauge ins, instance, ip, job, cls Memory that is reserved for or used to hold runtime metadata.
go_memory_classes_os_stacks_bytes gauge ins, instance, ip, job, cls Stack memory allocated by the underlying operating system. In non-cgo programs this metric is currently zero. This may change in the future.In cgo programs this metric includes OS thread stacks allocated directly from the OS. Currently, this only accounts for one stack in c-shared and c-archive build modes, and other sources of stacks from the OS are not measured. This too may change in the future.
go_memory_classes_other_bytes gauge ins, instance, ip, job, cls Memory used by execution trace buffers, structures for debugging the runtime, finalizer and profiler specials, and more.
go_memory_classes_profiling_buckets_bytes gauge ins, instance, ip, job, cls Memory that is used by the stack trace hash map used for profiling.
go_memory_classes_total_bytes gauge ins, instance, ip, job, cls All memory mapped by the Go runtime into the current process as read-write. Note that this does not include memory mapped by code called via cgo or via the syscall package. Sum of all metrics in /memory/classes.
go_memstats_alloc_bytes counter ins, instance, ip, job, cls Total number of bytes allocated, even if freed.
go_memstats_alloc_bytes_total counter ins, instance, ip, job, cls Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter ins, instance, ip, job, cls Total number of frees.
go_memstats_gc_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge ins, instance, ip, job, cls Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge ins, instance, ip, job, cls Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge ins, instance, ip, job, cls Number of heap bytes that are in use.
go_memstats_heap_objects gauge ins, instance, ip, job, cls Number of allocated objects.
go_memstats_heap_released_bytes gauge ins, instance, ip, job, cls Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge ins, instance, ip, job, cls Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge ins, instance, ip, job, cls Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter ins, instance, ip, job, cls Total number of pointer lookups.
go_memstats_mallocs_total counter ins, instance, ip, job, cls Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge ins, instance, ip, job, cls Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge ins, instance, ip, job, cls Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge ins, instance, ip, job, cls Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge ins, instance, ip, job, cls Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge ins, instance, ip, job, cls Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge ins, instance, ip, job, cls Number of bytes obtained from system.
go_sched_gomaxprocs_threads gauge ins, instance, ip, job, cls The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously.
go_sched_goroutines_goroutines gauge ins, instance, ip, job, cls Count of live goroutines.
go_sched_latencies_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
go_sched_latencies_seconds_count Unknown ins, instance, ip, job, cls N/A
go_sched_latencies_seconds_sum Unknown ins, instance, ip, job, cls N/A
go_sql_stats_connections_blocked_seconds unknown ins, instance, db_name, ip, job, cls The total time blocked waiting for a new connection.
go_sql_stats_connections_closed_max_idle unknown ins, instance, db_name, ip, job, cls The total number of connections closed due to SetMaxIdleConns.
go_sql_stats_connections_closed_max_idle_time unknown ins, instance, db_name, ip, job, cls The total number of connections closed due to SetConnMaxIdleTime.
go_sql_stats_connections_closed_max_lifetime unknown ins, instance, db_name, ip, job, cls The total number of connections closed due to SetConnMaxLifetime.
go_sql_stats_connections_idle gauge ins, instance, db_name, ip, job, cls The number of idle connections.
go_sql_stats_connections_in_use gauge ins, instance, db_name, ip, job, cls The number of connections currently in use.
go_sql_stats_connections_max_open gauge ins, instance, db_name, ip, job, cls Maximum number of open connections to the database.
go_sql_stats_connections_open gauge ins, instance, db_name, ip, job, cls The number of established connections both in use and idle.
go_sql_stats_connections_waited_for unknown ins, instance, db_name, ip, job, cls The total number of connections waited for.
go_sync_mutex_wait_total_seconds_total Unknown ins, instance, ip, job, cls N/A
go_threads gauge ins, instance, ip, job, cls Number of OS threads created.
grafana_access_evaluation_count unknown ins, instance, ip, job, cls number of evaluation calls
grafana_access_evaluation_duration_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_access_evaluation_duration_count Unknown ins, instance, ip, job, cls N/A
grafana_access_evaluation_duration_sum Unknown ins, instance, ip, job, cls N/A
grafana_access_permissions_duration_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_access_permissions_duration_count Unknown ins, instance, ip, job, cls N/A
grafana_access_permissions_duration_sum Unknown ins, instance, ip, job, cls N/A
grafana_aggregator_discovery_aggregation_count_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_active_alerts gauge ins, instance, ip, job, cls amount of active alerts
grafana_alerting_active_configurations gauge ins, instance, ip, job, cls The number of active Alertmanager configurations.
grafana_alerting_alertmanager_config_match gauge ins, instance, ip, job, cls The total number of match
grafana_alerting_alertmanager_config_match_re gauge ins, instance, ip, job, cls The total number of matchRE
grafana_alerting_alertmanager_config_matchers gauge ins, instance, ip, job, cls The total number of matchers
grafana_alerting_alertmanager_config_object_matchers gauge ins, instance, ip, job, cls The total number of object_matchers
grafana_alerting_discovered_configurations gauge ins, instance, ip, job, cls The number of organizations we’ve discovered that require an Alertmanager configuration.
grafana_alerting_dispatcher_aggregation_groups gauge ins, instance, ip, job, cls Number of active aggregation groups
grafana_alerting_dispatcher_alert_processing_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_dispatcher_alert_processing_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_execution_time_milliseconds summary ins, instance, ip, job, cls, quantile summary of alert execution duration
grafana_alerting_execution_time_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_execution_time_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_gossip_messages_propagated_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_queries_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_nflog_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_query_errors_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last notification log snapshot in bytes.
grafana_alerting_notification_latency_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_notification_latency_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_notification_latency_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_alert_rules gauge ins, instance, ip, job, cls The number of alert rules that could be considered for evaluation at the next tick.
grafana_alerting_schedule_alert_rules_hash gauge ins, instance, ip, job, cls A hash of the alert rules that could be considered for evaluation at the next tick.
grafana_alerting_schedule_periodic_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_schedule_periodic_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_periodic_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_query_alert_rules_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_schedule_query_alert_rules_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_query_alert_rules_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_scheduler_behind_seconds gauge ins, instance, ip, job, cls The total number of seconds the scheduler is behind.
grafana_alerting_silences_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_gossip_messages_propagated_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_queries_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_silences_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_query_errors_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last silence snapshot in bytes.
grafana_alerting_state_calculation_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_state_calculation_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_state_calculation_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_state_history_writes_bytes_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_ticker_interval_seconds gauge ins, instance, ip, job, cls Interval at which the ticker is meant to tick.
grafana_alerting_ticker_last_consumed_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last consumed tick in seconds.
grafana_alerting_ticker_next_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the next tick in seconds before it is consumed.
grafana_api_admin_user_created_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_get_milliseconds summary ins, instance, ip, job, cls, quantile summary for dashboard get duration
grafana_api_dashboard_get_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_get_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_save_milliseconds summary ins, instance, ip, job, cls, quantile summary for dashboard save duration
grafana_api_dashboard_save_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_save_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_search_milliseconds summary ins, instance, ip, job, cls, quantile summary for dashboard search duration
grafana_api_dashboard_search_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_search_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_snapshot_create_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_snapshot_external_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_snapshot_get_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dataproxy_request_all_milliseconds summary ins, instance, ip, job, cls, quantile summary for dataproxy request duration
grafana_api_dataproxy_request_all_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dataproxy_request_all_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_login_oauth_total Unknown ins, instance, ip, job, cls N/A
grafana_api_login_post_total Unknown ins, instance, ip, job, cls N/A
grafana_api_login_saml_total Unknown ins, instance, ip, job, cls N/A
grafana_api_models_dashboard_insert_total Unknown ins, instance, ip, job, cls N/A
grafana_api_org_create_total Unknown ins, instance, ip, job, cls N/A
grafana_api_response_status_total Unknown ins, instance, ip, job, cls, code N/A
grafana_api_user_signup_completed_total Unknown ins, instance, ip, job, cls N/A
grafana_api_user_signup_invite_total Unknown ins, instance, ip, job, cls N/A
grafana_api_user_signup_started_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_audit_event_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_audit_requests_rejected_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_client_certificate_expiration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_apiserver_client_certificate_expiration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_client_certificate_expiration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_envelope_encryption_dek_cache_fill_percent gauge ins, instance, ip, job, cls [ALPHA] Percent of the cache slots currently occupied by cached DEKs.
grafana_apiserver_flowcontrol_seat_fair_frac gauge ins, instance, ip, job, cls [ALPHA] Fair fraction of server’s concurrency to allocate to each priority level that can use it
grafana_apiserver_storage_data_key_generation_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_apiserver_storage_data_key_generation_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_storage_data_key_generation_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_storage_data_key_generation_failures_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_storage_envelope_transformation_cache_misses_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_tls_handshake_errors_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_webhooks_x509_insecure_sha1_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_webhooks_x509_missing_san_total Unknown ins, instance, ip, job, cls N/A
grafana_authn_authn_failed_authentication_total Unknown ins, instance, ip, job, cls N/A
grafana_authn_authn_successful_authentication_total Unknown ins, instance, ip, client, job, cls N/A
grafana_authn_authn_successful_login_total Unknown ins, instance, ip, client, job, cls N/A
grafana_aws_cloudwatch_get_metric_data_total Unknown ins, instance, ip, job, cls N/A
grafana_aws_cloudwatch_get_metric_statistics_total Unknown ins, instance, ip, job, cls N/A
grafana_aws_cloudwatch_list_metrics_total Unknown ins, instance, ip, job, cls N/A
grafana_build_info gauge revision, version, ins, instance, edition, ip, goversion, job, cls, branch A metric with a constant ‘1’ value labeled by version, revision, branch, and goversion from which Grafana was built
grafana_build_timestamp gauge revision, version, ins, instance, edition, ip, goversion, job, cls, branch A metric exposing when the binary was built in epoch
grafana_cardinality_enforcement_unexpected_categorizations_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_idle gauge ins, instance, ip, job, cls The number of idle connections
grafana_database_conn_in_use gauge ins, instance, ip, job, cls The number of connections currently in use
grafana_database_conn_max_idle_closed_seconds unknown ins, instance, ip, job, cls The total number of connections closed due to SetConnMaxIdleTime
grafana_database_conn_max_idle_closed_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_max_lifetime_closed_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_max_open gauge ins, instance, ip, job, cls Maximum number of open connections to the database
grafana_database_conn_open gauge ins, instance, ip, job, cls The number of established connections both in use and idle
grafana_database_conn_wait_count_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_wait_duration_seconds unknown ins, instance, ip, job, cls The total time blocked waiting for a new connection
grafana_datasource_request_duration_seconds_bucket Unknown datasource, ins, instance, method, ip, le, datasource_type, job, cls, code N/A
grafana_datasource_request_duration_seconds_count Unknown datasource, ins, instance, method, ip, datasource_type, job, cls, code N/A
grafana_datasource_request_duration_seconds_sum Unknown datasource, ins, instance, method, ip, datasource_type, job, cls, code N/A
grafana_datasource_request_in_flight gauge datasource, ins, instance, ip, datasource_type, job, cls A gauge of outgoing data source requests currently being sent by Grafana
grafana_datasource_request_total Unknown datasource, ins, instance, method, ip, datasource_type, job, cls, code N/A
grafana_datasource_response_size_bytes_bucket Unknown datasource, ins, instance, ip, le, datasource_type, job, cls N/A
grafana_datasource_response_size_bytes_count Unknown datasource, ins, instance, ip, datasource_type, job, cls N/A
grafana_datasource_response_size_bytes_sum Unknown datasource, ins, instance, ip, datasource_type, job, cls N/A
grafana_db_datasource_query_by_id_total Unknown ins, instance, ip, job, cls N/A
grafana_disabled_metrics_total Unknown ins, instance, ip, job, cls N/A
grafana_emails_sent_failed unknown ins, instance, ip, job, cls Number of emails Grafana failed to send
grafana_emails_sent_total Unknown ins, instance, ip, job, cls N/A
grafana_encryption_cache_reads_total Unknown ins, instance, method, ip, hit, job, cls N/A
grafana_encryption_ops_total Unknown ins, instance, ip, success, operation, job, cls N/A
grafana_environment_info gauge version, ins, instance, ip, job, cls, commit A metric with a constant ‘1’ value labeled by environment information about the running instance.
grafana_feature_toggles_info gauge ins, instance, ip, job, cls info metric that exposes what feature toggles are enabled or not
grafana_frontend_boot_css_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_css_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_css_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_contentful_paint_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_first_contentful_paint_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_contentful_paint_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_paint_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_first_paint_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_paint_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_js_done_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_js_done_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_js_done_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_load_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_load_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_load_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_plugins_preload_ms_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_plugins_preload_ms_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_plugins_preload_ms_sum Unknown ins, instance, ip, job, cls N/A
grafana_hidden_metrics_total Unknown ins, instance, ip, job, cls N/A
grafana_http_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, status_code, handler N/A
grafana_http_request_duration_seconds_count Unknown ins, instance, method, ip, job, cls, status_code, handler N/A
grafana_http_request_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, status_code, handler N/A
grafana_http_request_in_flight gauge ins, instance, ip, job, cls A gauge of requests currently being served by Grafana.
grafana_idforwarding_idforwarding_failed_token_signing_total Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_from_cache_total Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_total Unknown ins, instance, ip, job, cls N/A
grafana_instance_start_total Unknown ins, instance, ip, job, cls N/A
grafana_ldap_users_sync_execution_time summary ins, instance, ip, job, cls, quantile summary for LDAP users sync execution duration
grafana_ldap_users_sync_execution_time_count Unknown ins, instance, ip, job, cls N/A
grafana_ldap_users_sync_execution_time_sum Unknown ins, instance, ip, job, cls N/A
grafana_live_client_command_duration_seconds summary ins, instance, method, ip, job, cls, quantile Client command duration summary.
grafana_live_client_command_duration_seconds_count Unknown ins, instance, method, ip, job, cls N/A
grafana_live_client_command_duration_seconds_sum Unknown ins, instance, method, ip, job, cls N/A
grafana_live_client_num_reply_errors unknown ins, instance, method, ip, job, cls, code Number of errors in replies sent to clients.
grafana_live_client_num_server_disconnects unknown ins, instance, ip, job, cls, code Number of server initiated disconnects.
grafana_live_client_recover unknown ins, instance, ip, recovered, job, cls Count of recover operations.
grafana_live_node_action_count unknown action, ins, instance, ip, job, cls Number of node actions called.
grafana_live_node_build gauge version, ins, instance, ip, job, cls Node build info.
grafana_live_node_messages_received_count unknown ins, instance, ip, type, job, cls Number of messages received.
grafana_live_node_messages_sent_count unknown ins, instance, ip, type, job, cls Number of messages sent.
grafana_live_node_num_channels gauge ins, instance, ip, job, cls Number of channels with one or more subscribers.
grafana_live_node_num_clients gauge ins, instance, ip, job, cls Number of clients connected.
grafana_live_node_num_nodes gauge ins, instance, ip, job, cls Number of nodes in cluster.
grafana_live_node_num_subscriptions gauge ins, instance, ip, job, cls Number of subscriptions.
grafana_live_node_num_users gauge ins, instance, ip, job, cls Number of unique users connected.
grafana_live_transport_connect_count unknown ins, instance, ip, transport, job, cls Number of connections to specific transport.
grafana_live_transport_messages_sent unknown ins, instance, ip, transport, job, cls Number of messages sent over specific transport.
grafana_loki_plugin_parse_response_duration_seconds_bucket Unknown endpoint, ins, instance, ip, le, status, job, cls N/A
grafana_loki_plugin_parse_response_duration_seconds_count Unknown endpoint, ins, instance, ip, status, job, cls N/A
grafana_loki_plugin_parse_response_duration_seconds_sum Unknown endpoint, ins, instance, ip, status, job, cls N/A
grafana_page_response_status_total Unknown ins, instance, ip, job, cls, code N/A
grafana_plugin_build_info gauge version, signature_status, ins, instance, plugin_type, ip, plugin_id, job, cls A metric with a constant ‘1’ value labeled by pluginId, pluginType and version from which Grafana plugin was built
grafana_plugin_request_duration_milliseconds_bucket Unknown endpoint, ins, instance, target, ip, le, plugin_id, job, cls N/A
grafana_plugin_request_duration_milliseconds_count Unknown endpoint, ins, instance, target, ip, plugin_id, job, cls N/A
grafana_plugin_request_duration_milliseconds_sum Unknown endpoint, ins, instance, target, ip, plugin_id, job, cls N/A
grafana_plugin_request_duration_seconds_bucket Unknown endpoint, ins, instance, target, ip, le, status, plugin_id, source, job, cls N/A
grafana_plugin_request_duration_seconds_count Unknown endpoint, ins, instance, target, ip, status, plugin_id, source, job, cls N/A
grafana_plugin_request_duration_seconds_sum Unknown endpoint, ins, instance, target, ip, status, plugin_id, source, job, cls N/A
grafana_plugin_request_size_bytes_bucket Unknown endpoint, ins, instance, target, ip, le, plugin_id, source, job, cls N/A
grafana_plugin_request_size_bytes_count Unknown endpoint, ins, instance, target, ip, plugin_id, source, job, cls N/A
grafana_plugin_request_size_bytes_sum Unknown endpoint, ins, instance, target, ip, plugin_id, source, job, cls N/A
grafana_plugin_request_total Unknown endpoint, ins, instance, target, ip, status, plugin_id, job, cls N/A
grafana_process_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
grafana_process_max_fds gauge ins, instance, ip, job, cls Maximum number of open file descriptors.
grafana_process_open_fds gauge ins, instance, ip, job, cls Number of open file descriptors.
grafana_process_resident_memory_bytes gauge ins, instance, ip, job, cls Resident memory size in bytes.
grafana_process_start_time_seconds gauge ins, instance, ip, job, cls Start time of the process since unix epoch in seconds.
grafana_process_virtual_memory_bytes gauge ins, instance, ip, job, cls Virtual memory size in bytes.
grafana_process_virtual_memory_max_bytes gauge ins, instance, ip, job, cls Maximum amount of virtual memory available in bytes.
grafana_prometheus_plugin_backend_request_count unknown endpoint, ins, instance, ip, status, errorSource, job, cls The total amount of prometheus backend plugin requests
grafana_proxy_response_status_total Unknown ins, instance, ip, job, cls, code N/A
grafana_public_dashboard_request_count unknown ins, instance, ip, job, cls counter for public dashboards requests
grafana_registered_metrics_total Unknown ins, instance, ip, stability_level, deprecated_version, job, cls N/A
grafana_rendering_queue_size gauge ins, instance, ip, job, cls size of rendering queue
grafana_search_dashboard_search_failures_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_search_dashboard_search_failures_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_search_dashboard_search_failures_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_search_dashboard_search_successes_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_search_dashboard_search_successes_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_search_dashboard_search_successes_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_stat_active_users gauge ins, instance, ip, job, cls number of active users
grafana_stat_total_orgs gauge ins, instance, ip, job, cls total amount of orgs
grafana_stat_total_playlists gauge ins, instance, ip, job, cls total amount of playlists
grafana_stat_total_service_account_tokens gauge ins, instance, ip, job, cls total amount of service account tokens
grafana_stat_total_service_accounts gauge ins, instance, ip, job, cls total amount of service accounts
grafana_stat_total_service_accounts_role_none gauge ins, instance, ip, job, cls total amount of service accounts with no role
grafana_stat_total_teams gauge ins, instance, ip, job, cls total amount of teams
grafana_stat_total_users gauge ins, instance, ip, job, cls total amount of users
grafana_stat_totals_active_admins gauge ins, instance, ip, job, cls total amount of active admins
grafana_stat_totals_active_editors gauge ins, instance, ip, job, cls total amount of active editors
grafana_stat_totals_active_viewers gauge ins, instance, ip, job, cls total amount of active viewers
grafana_stat_totals_admins gauge ins, instance, ip, job, cls total amount of admins
grafana_stat_totals_alert_rules gauge ins, instance, ip, job, cls total amount of alert rules in the database
grafana_stat_totals_annotations gauge ins, instance, ip, job, cls total amount of annotations in the database
grafana_stat_totals_correlations gauge ins, instance, ip, job, cls total amount of correlations
grafana_stat_totals_dashboard gauge ins, instance, ip, job, cls total amount of dashboards
grafana_stat_totals_dashboard_versions gauge ins, instance, ip, job, cls total amount of dashboard versions in the database
grafana_stat_totals_data_keys gauge ins, instance, ip, job, cls, active total amount of data keys in the database
grafana_stat_totals_datasource gauge ins, instance, ip, plugin_id, job, cls total number of defined datasources, labeled by pluginId
grafana_stat_totals_editors gauge ins, instance, ip, job, cls total amount of editors
grafana_stat_totals_folder gauge ins, instance, ip, job, cls total amount of folders
grafana_stat_totals_library_panels gauge ins, instance, ip, job, cls total amount of library panels in the database
grafana_stat_totals_library_variables gauge ins, instance, ip, job, cls total amount of library variables in the database
grafana_stat_totals_public_dashboard gauge ins, instance, ip, job, cls total amount of public dashboards
grafana_stat_totals_rule_groups gauge ins, instance, ip, job, cls total amount of alert rule groups in the database
grafana_stat_totals_viewers gauge ins, instance, ip, job, cls total amount of viewers
infra_up Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_baggage_restrictions_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_baggage_truncations_total Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_baggage_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_finished_spans_total Unknown ins, instance, ip, sampled, job, cls N/A
jaeger_tracer_reporter_queue_length gauge ins, instance, ip, job, cls Current number of spans in the reporter queue
jaeger_tracer_reporter_spans_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_sampler_queries_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_sampler_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_span_context_decoding_errors_total Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_started_spans_total Unknown ins, instance, ip, sampled, job, cls N/A
jaeger_tracer_throttled_debug_spans_total Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_throttler_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_traces_total Unknown ins, instance, ip, sampled, job, cls, state N/A
kv_request_duration_seconds_bucket Unknown ins, instance, role, ip, le, kv_name, type, operation, job, cls, status_code N/A
kv_request_duration_seconds_count Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
kv_request_duration_seconds_sum Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
legacy_grafana_alerting_ticker_interval_seconds gauge ins, instance, ip, job, cls Interval at which the ticker is meant to tick.
legacy_grafana_alerting_ticker_last_consumed_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last consumed tick in seconds.
legacy_grafana_alerting_ticker_next_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the next tick in seconds before it is consumed.
logql_query_duration_seconds_bucket Unknown ins, instance, query_type, ip, le, job, cls N/A
logql_query_duration_seconds_count Unknown ins, instance, query_type, ip, job, cls N/A
logql_query_duration_seconds_sum Unknown ins, instance, query_type, ip, job, cls N/A
loki_azure_blob_egress_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_boltdb_shipper_apply_retention_last_successful_run_timestamp_seconds gauge ins, instance, ip, job, cls Unix timestamp of the last successful retention run
loki_boltdb_shipper_compact_tables_operation_duration_seconds gauge ins, instance, ip, job, cls Time (in seconds) spent in compacting all the tables
loki_boltdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds gauge ins, instance, ip, job, cls Unix timestamp of the last successful compaction run
loki_boltdb_shipper_compact_tables_operation_total Unknown ins, instance, ip, status, job, cls N/A
loki_boltdb_shipper_compactor_running gauge ins, instance, ip, job, cls Value will be 1 if compactor is currently running on this instance
loki_boltdb_shipper_open_existing_file_failures_total Unknown ins, instance, ip, component, job, cls N/A
loki_boltdb_shipper_query_time_table_download_duration_seconds unknown ins, instance, ip, component, job, cls, table Time (in seconds) spent in downloading of files per table at query time
loki_boltdb_shipper_request_duration_seconds_bucket Unknown ins, instance, ip, le, component, operation, job, cls, status_code N/A
loki_boltdb_shipper_request_duration_seconds_count Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_boltdb_shipper_request_duration_seconds_sum Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_boltdb_shipper_tables_download_operation_duration_seconds gauge ins, instance, ip, component, job, cls Time (in seconds) spent in downloading updated files for all the tables
loki_boltdb_shipper_tables_sync_operation_total Unknown ins, instance, ip, status, component, job, cls N/A
loki_boltdb_shipper_tables_upload_operation_total Unknown ins, instance, ip, status, component, job, cls N/A
loki_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which loki was built, and the goos and goarch for the build.
loki_bytes_per_line_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_bytes_per_line_count Unknown ins, instance, ip, job, cls N/A
loki_bytes_per_line_sum Unknown ins, instance, ip, job, cls N/A
loki_cache_corrupt_chunks_total Unknown ins, instance, ip, job, cls N/A
loki_cache_fetched_keys unknown ins, instance, ip, job, cls Total count of keys requested from cache.
loki_cache_hits unknown ins, instance, ip, job, cls Total count of keys found in cache.
loki_cache_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, status_code N/A
loki_cache_request_duration_seconds_count Unknown ins, instance, method, ip, job, cls, status_code N/A
loki_cache_request_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, status_code N/A
loki_cache_value_size_bytes_bucket Unknown ins, instance, method, ip, le, job, cls N/A
loki_cache_value_size_bytes_count Unknown ins, instance, method, ip, job, cls N/A
loki_cache_value_size_bytes_sum Unknown ins, instance, method, ip, job, cls N/A
loki_chunk_fetcher_cache_dequeued_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_fetcher_cache_enqueued_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_fetcher_cache_skipped_buffer_full_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_fetcher_fetched_size_bytes_bucket Unknown ins, instance, ip, le, source, job, cls N/A
loki_chunk_fetcher_fetched_size_bytes_count Unknown ins, instance, ip, source, job, cls N/A
loki_chunk_fetcher_fetched_size_bytes_sum Unknown ins, instance, ip, source, job, cls N/A
loki_chunk_store_chunks_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_chunks_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_chunks_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_deduped_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_deduped_chunks_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_fetched_chunk_bytes_total Unknown ins, instance, ip, user, job, cls N/A
loki_chunk_store_fetched_chunks_total Unknown ins, instance, ip, user, job, cls N/A
loki_chunk_store_index_entries_per_chunk_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_index_entries_per_chunk_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_index_entries_per_chunk_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_index_lookups_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_index_lookups_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_index_lookups_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_post_intersection_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_series_post_intersection_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_post_intersection_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_pre_intersection_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_series_pre_intersection_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_pre_intersection_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_stored_chunk_bytes_total Unknown ins, instance, ip, user, job, cls N/A
loki_chunk_store_stored_chunks_total Unknown ins, instance, ip, user, job, cls N/A
loki_consul_request_duration_seconds_bucket Unknown ins, instance, ip, le, kv_name, operation, job, cls, status_code N/A
loki_consul_request_duration_seconds_count Unknown ins, instance, ip, kv_name, operation, job, cls, status_code N/A
loki_consul_request_duration_seconds_sum Unknown ins, instance, ip, kv_name, operation, job, cls, status_code N/A
loki_delete_request_lookups_failed_total Unknown ins, instance, ip, job, cls N/A
loki_delete_request_lookups_total Unknown ins, instance, ip, job, cls N/A
loki_discarded_bytes_total Unknown ins, instance, ip, reason, job, cls, tenant N/A
loki_discarded_samples_total Unknown ins, instance, ip, reason, job, cls, tenant N/A
loki_distributor_bytes_received_total Unknown ins, instance, retention_hours, ip, job, cls, tenant N/A
loki_distributor_ingester_appends_total Unknown ins, instance, ip, ingester, job, cls N/A
loki_distributor_lines_received_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_distributor_replication_factor gauge ins, instance, ip, job, cls The configured replication factor.
loki_distributor_structured_metadata_bytes_received_total Unknown ins, instance, retention_hours, ip, job, cls, tenant N/A
loki_experimental_features_in_use_total Unknown ins, instance, ip, job, cls N/A
loki_index_chunk_refs_total Unknown ins, instance, ip, status, job, cls N/A
loki_index_request_duration_seconds_bucket Unknown ins, instance, ip, le, component, operation, job, cls, status_code N/A
loki_index_request_duration_seconds_count Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_index_request_duration_seconds_sum Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_inflight_requests gauge ins, instance, method, ip, route, job, cls Current number of inflight requests.
loki_ingester_autoforget_unhealthy_ingesters_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_blocks_per_chunk_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_blocks_per_chunk_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_blocks_per_chunk_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_creations_failed_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_creations_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_deletions_failed_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_deletions_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_duration_seconds summary ins, instance, ip, job, cls, quantile Time taken to create a checkpoint.
loki_ingester_checkpoint_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_logged_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_age_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_age_seconds_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_age_seconds_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_bounds_hours_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_bounds_hours_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_bounds_hours_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_compression_ratio_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_compression_ratio_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_compression_ratio_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_encode_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_encode_time_seconds_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_encode_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_entries_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_entries_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_entries_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_size_bytes_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_stored_bytes_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_chunk_utilization_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_utilization_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_utilization_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunks_created_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunks_flushed_total Unknown ins, instance, ip, reason, job, cls N/A
loki_ingester_chunks_stored_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_client_request_duration_seconds_bucket Unknown ins, instance, ip, le, operation, job, cls, status_code N/A
loki_ingester_client_request_duration_seconds_count Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_ingester_client_request_duration_seconds_sum Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_ingester_limiter_enabled gauge ins, instance, ip, job, cls Whether the ingester’s limiter is enabled
loki_ingester_memory_chunks gauge ins, instance, ip, job, cls The total number of chunks in memory.
loki_ingester_memory_streams gauge ins, instance, ip, job, cls, tenant The total number of streams in memory per tenant.
loki_ingester_memory_streams_labels_bytes gauge ins, instance, ip, job, cls Total bytes of labels of the streams in memory.
loki_ingester_received_chunks unknown ins, instance, ip, job, cls The total number of chunks received by this ingester whilst joining.
loki_ingester_samples_per_chunk_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_samples_per_chunk_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_samples_per_chunk_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_sent_chunks unknown ins, instance, ip, job, cls The total number of chunks sent by this ingester whilst leaving.
loki_ingester_shutdown_marker gauge ins, instance, ip, job, cls 1 if prepare shutdown has been called, 0 otherwise
loki_ingester_streams_created_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_streams_removed_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_wal_bytes_in_use gauge ins, instance, ip, job, cls Total number of bytes in use by the WAL recovery process.
loki_ingester_wal_disk_full_failures_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_duplicate_entries_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_logged_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_records_logged_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_chunks_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_entries_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_streams_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_replay_active gauge ins, instance, ip, job, cls Whether the WAL is replaying
loki_ingester_wal_replay_duration_seconds gauge ins, instance, ip, job, cls Time taken to replay the checkpoint and the WAL.
loki_ingester_wal_replay_flushing gauge ins, instance, ip, job, cls Whether the wal replay is in a flushing phase due to backpressure
loki_internal_log_messages_total Unknown ins, instance, ip, level, job, cls N/A
loki_kv_request_duration_seconds_bucket Unknown ins, instance, role, ip, le, kv_name, type, operation, job, cls, status_code N/A
loki_kv_request_duration_seconds_count Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
loki_kv_request_duration_seconds_sum Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
loki_log_flushes_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_log_flushes_count Unknown ins, instance, ip, job, cls N/A
loki_log_flushes_sum Unknown ins, instance, ip, job, cls N/A
loki_log_messages_total Unknown ins, instance, ip, level, job, cls N/A
loki_logql_querystats_bytes_processed_per_seconds_bucket Unknown ins, instance, range, ip, le, sharded, type, job, cls, status_code, latency_type N/A
loki_logql_querystats_bytes_processed_per_seconds_count Unknown ins, instance, range, ip, sharded, type, job, cls, status_code, latency_type N/A
loki_logql_querystats_bytes_processed_per_seconds_sum Unknown ins, instance, range, ip, sharded, type, job, cls, status_code, latency_type N/A
loki_logql_querystats_chunk_download_latency_seconds_bucket Unknown ins, instance, range, ip, le, type, job, cls, status_code N/A
loki_logql_querystats_chunk_download_latency_seconds_count Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_chunk_download_latency_seconds_sum Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_downloaded_chunk_total Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_duplicates_total Unknown ins, instance, ip, job, cls N/A
loki_logql_querystats_ingester_sent_lines_total Unknown ins, instance, ip, job, cls N/A
loki_logql_querystats_latency_seconds_bucket Unknown ins, instance, range, ip, le, type, job, cls, status_code N/A
loki_logql_querystats_latency_seconds_count Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_latency_seconds_sum Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_panic_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_corruptions_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_encode_errors_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_gets_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_hits_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_puts_total Unknown ins, instance, ip, job, cls N/A
loki_querier_query_frontend_clients gauge ins, instance, ip, job, cls The current number of clients connected to query-frontend.
loki_querier_query_frontend_request_duration_seconds_bucket Unknown ins, instance, ip, le, operation, job, cls, status_code N/A
loki_querier_query_frontend_request_duration_seconds_count Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_querier_query_frontend_request_duration_seconds_sum Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_querier_tail_active gauge ins, instance, ip, job, cls Number of active tailers
loki_querier_tail_active_streams gauge ins, instance, ip, job, cls Number of active streams being tailed
loki_querier_tail_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_querier_worker_concurrency gauge ins, instance, ip, job, cls Number of concurrent querier workers
loki_querier_worker_inflight_queries gauge ins, instance, ip, job, cls Number of queries being processed by the querier workers
loki_query_frontend_log_result_cache_hit_total Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_log_result_cache_miss_total Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_partitions_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_query_frontend_partitions_count Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_partitions_sum Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_shard_factor_bucket Unknown ins, instance, ip, le, mapper, job, cls N/A
loki_query_frontend_shard_factor_count Unknown ins, instance, ip, mapper, job, cls N/A
loki_query_frontend_shard_factor_sum Unknown ins, instance, ip, mapper, job, cls N/A
loki_query_scheduler_enqueue_count Unknown ins, instance, ip, level, user, job, cls N/A
loki_rate_store_expired_streams_total Unknown ins, instance, ip, job, cls N/A
loki_rate_store_max_stream_rate_bytes gauge ins, instance, ip, job, cls The maximum stream rate for any stream reported by ingesters during a sync operation. Sharded Streams are combined.
loki_rate_store_max_stream_shards gauge ins, instance, ip, job, cls The number of shards for a single stream reported by ingesters during a sync operation.
loki_rate_store_max_unique_stream_rate_bytes gauge ins, instance, ip, job, cls The maximum stream rate for any stream reported by ingesters during a sync operation. Sharded Streams are considered separate.
loki_rate_store_stream_rate_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_rate_store_stream_rate_bytes_count Unknown ins, instance, ip, job, cls N/A
loki_rate_store_stream_rate_bytes_sum Unknown ins, instance, ip, job, cls N/A
loki_rate_store_stream_shards_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_rate_store_stream_shards_count Unknown ins, instance, ip, job, cls N/A
loki_rate_store_stream_shards_sum Unknown ins, instance, ip, job, cls N/A
loki_rate_store_streams gauge ins, instance, ip, job, cls The number of unique streams reported by all ingesters. Sharded streams are combined
loki_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, ws, route, job, cls, status_code N/A
loki_request_duration_seconds_count Unknown ins, instance, method, ip, ws, route, job, cls, status_code N/A
loki_request_duration_seconds_sum Unknown ins, instance, method, ip, ws, route, job, cls, status_code N/A
loki_request_message_bytes_bucket Unknown ins, instance, method, ip, le, route, job, cls N/A
loki_request_message_bytes_count Unknown ins, instance, method, ip, route, job, cls N/A
loki_request_message_bytes_sum Unknown ins, instance, method, ip, route, job, cls N/A
loki_response_message_bytes_bucket Unknown ins, instance, method, ip, le, route, job, cls N/A
loki_response_message_bytes_count Unknown ins, instance, method, ip, route, job, cls N/A
loki_response_message_bytes_sum Unknown ins, instance, method, ip, route, job, cls N/A
loki_results_cache_version_comparisons_total Unknown ins, instance, ip, job, cls N/A
loki_store_chunks_downloaded_total Unknown ins, instance, ip, status, job, cls N/A
loki_store_chunks_per_batch_bucket Unknown ins, instance, ip, le, status, job, cls N/A
loki_store_chunks_per_batch_count Unknown ins, instance, ip, status, job, cls N/A
loki_store_chunks_per_batch_sum Unknown ins, instance, ip, status, job, cls N/A
loki_store_series_total Unknown ins, instance, ip, status, job, cls N/A
loki_stream_sharding_count unknown ins, instance, ip, job, cls Total number of times the distributor has sharded streams
loki_tcp_connections gauge ins, instance, ip, protocol, job, cls Current number of accepted TCP connections.
loki_tcp_connections_limit gauge ins, instance, ip, protocol, job, cls The max number of TCP connections that can be accepted (0 means no limit).
net_conntrack_dialer_conn_attempted_total counter ins, instance, ip, dialer_name, job, cls Total number of connections attempted by the given dialer a given name.
net_conntrack_dialer_conn_closed_total counter ins, instance, ip, dialer_name, job, cls Total number of connections closed which originated from the dialer of a given name.
net_conntrack_dialer_conn_established_total counter ins, instance, ip, dialer_name, job, cls Total number of connections successfully established by the given dialer a given name.
net_conntrack_dialer_conn_failed_total counter ins, instance, ip, dialer_name, reason, job, cls Total number of connections failed to dial by the dialer a given name.
net_conntrack_listener_conn_accepted_total counter ins, instance, ip, listener_name, job, cls Total number of connections opened to the listener of a given name.
net_conntrack_listener_conn_closed_total counter ins, instance, ip, listener_name, job, cls Total number of connections closed that were made to the listener of a given name.
nginx_connections_accepted counter ins, instance, ip, job, cls Accepted client connections
nginx_connections_active gauge ins, instance, ip, job, cls Active client connections
nginx_connections_handled counter ins, instance, ip, job, cls Handled client connections
nginx_connections_reading gauge ins, instance, ip, job, cls Connections where NGINX is reading the request header
nginx_connections_waiting gauge ins, instance, ip, job, cls Idle client connections
nginx_connections_writing gauge ins, instance, ip, job, cls Connections where NGINX is writing the response back to the client
nginx_exporter_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which nginx_exporter was built, and the goos and goarch for the build.
nginx_http_requests_total counter ins, instance, ip, job, cls Total http requests
nginx_up gauge ins, instance, ip, job, cls Status of the last metric scrape
plugins_active_instances gauge ins, instance, ip, job, cls The number of active plugin instances
plugins_datasource_instances_total Unknown ins, instance, ip, job, cls N/A
process_cpu_seconds_total counter ins, instance, ip, job, cls Total user and system CPU time spent in seconds.
process_max_fds gauge ins, instance, ip, job, cls Maximum number of open file descriptors.
process_open_fds gauge ins, instance, ip, job, cls Number of open file descriptors.
process_resident_memory_bytes gauge ins, instance, ip, job, cls Resident memory size in bytes.
process_start_time_seconds gauge ins, instance, ip, job, cls Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge ins, instance, ip, job, cls Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge ins, instance, ip, job, cls Maximum amount of virtual memory available in bytes.
prometheus_api_remote_read_queries gauge ins, instance, ip, job, cls The current number of remote read queries being executed or waiting.
prometheus_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which prometheus was built, and the goos and goarch for the build.
prometheus_config_last_reload_success_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last successful configuration reload.
prometheus_config_last_reload_successful gauge ins, instance, ip, job, cls Whether the last configuration reload attempt was successful.
prometheus_engine_queries gauge ins, instance, ip, job, cls The current number of queries being executed or waiting.
prometheus_engine_queries_concurrent_max gauge ins, instance, ip, job, cls The max number of concurrent queries.
prometheus_engine_query_duration_seconds summary ins, instance, ip, job, cls, quantile, slice Query timings
prometheus_engine_query_duration_seconds_count Unknown ins, instance, ip, job, cls, slice N/A
prometheus_engine_query_duration_seconds_sum Unknown ins, instance, ip, job, cls, slice N/A
prometheus_engine_query_log_enabled gauge ins, instance, ip, job, cls State of the query log.
prometheus_engine_query_log_failures_total counter ins, instance, ip, job, cls The number of query log failures.
prometheus_engine_query_samples_total counter ins, instance, ip, job, cls The total number of samples loaded by all queries.
prometheus_http_request_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls, handler N/A
prometheus_http_request_duration_seconds_count Unknown ins, instance, ip, job, cls, handler N/A
prometheus_http_request_duration_seconds_sum Unknown ins, instance, ip, job, cls, handler N/A
prometheus_http_requests_total counter ins, instance, ip, job, cls, code, handler Counter of HTTP requests.
prometheus_http_response_size_bytes_bucket Unknown ins, instance, ip, le, job, cls, handler N/A
prometheus_http_response_size_bytes_count Unknown ins, instance, ip, job, cls, handler N/A
prometheus_http_response_size_bytes_sum Unknown ins, instance, ip, job, cls, handler N/A
prometheus_notifications_alertmanagers_discovered gauge ins, instance, ip, job, cls The number of alertmanagers discovered and active.
prometheus_notifications_dropped_total counter ins, instance, ip, job, cls Total number of alerts dropped due to errors when sending to Alertmanager.
prometheus_notifications_errors_total counter ins, instance, ip, alertmanager, job, cls Total number of errors sending alert notifications.
prometheus_notifications_latency_seconds summary ins, instance, ip, alertmanager, job, cls, quantile Latency quantiles for sending alert notifications.
prometheus_notifications_latency_seconds_count Unknown ins, instance, ip, alertmanager, job, cls N/A
prometheus_notifications_latency_seconds_sum Unknown ins, instance, ip, alertmanager, job, cls N/A
prometheus_notifications_queue_capacity gauge ins, instance, ip, job, cls The capacity of the alert notifications queue.
prometheus_notifications_queue_length gauge ins, instance, ip, job, cls The number of alert notifications in the queue.
prometheus_notifications_sent_total counter ins, instance, ip, alertmanager, job, cls Total number of alerts sent.
prometheus_ready gauge ins, instance, ip, job, cls Whether Prometheus startup was fully completed and the server is ready for normal operation.
prometheus_remote_storage_exemplars_in_total counter ins, instance, ip, job, cls Exemplars in to remote storage, compare to exemplars out for queue managers.
prometheus_remote_storage_highest_timestamp_in_seconds gauge ins, instance, ip, job, cls Highest timestamp that has come into the remote storage via the Appender interface, in seconds since epoch.
prometheus_remote_storage_histograms_in_total counter ins, instance, ip, job, cls HistogramSamples in to remote storage, compare to histograms out for queue managers.
prometheus_remote_storage_samples_in_total counter ins, instance, ip, job, cls Samples in to remote storage, compare to samples out for queue managers.
prometheus_remote_storage_string_interner_zero_reference_releases_total counter ins, instance, ip, job, cls The number of times release has been called for strings that are not interned.
prometheus_rule_evaluation_duration_seconds summary ins, instance, ip, job, cls, quantile The duration for a rule to execute.
prometheus_rule_evaluation_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_rule_evaluation_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_rule_evaluation_failures_total counter ins, instance, ip, job, cls, rule_group The total number of rule evaluation failures.
prometheus_rule_evaluations_total counter ins, instance, ip, job, cls, rule_group The total number of rule evaluations.
prometheus_rule_group_duration_seconds summary ins, instance, ip, job, cls, quantile The duration of rule group evaluations.
prometheus_rule_group_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_rule_group_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_rule_group_interval_seconds gauge ins, instance, ip, job, cls, rule_group The interval of a rule group.
prometheus_rule_group_iterations_missed_total counter ins, instance, ip, job, cls, rule_group The total number of rule group evaluations missed due to slow rule group evaluation.
prometheus_rule_group_iterations_total counter ins, instance, ip, job, cls, rule_group The total number of scheduled rule group evaluations, whether executed or missed.
prometheus_rule_group_last_duration_seconds gauge ins, instance, ip, job, cls, rule_group The duration of the last rule group evaluation.
prometheus_rule_group_last_evaluation_samples gauge ins, instance, ip, job, cls, rule_group The number of samples returned during the last rule group evaluation.
prometheus_rule_group_last_evaluation_timestamp_seconds gauge ins, instance, ip, job, cls, rule_group The timestamp of the last rule group evaluation in seconds.
prometheus_rule_group_rules gauge ins, instance, ip, job, cls, rule_group The number of rules.
prometheus_sd_azure_cache_hit_total counter ins, instance, ip, job, cls Number of cache hit during refresh.
prometheus_sd_azure_failures_total counter ins, instance, ip, job, cls Number of Azure service discovery refresh failures.
prometheus_sd_consul_rpc_duration_seconds summary endpoint, ins, instance, ip, job, cls, call, quantile The duration of a Consul RPC call in seconds.
prometheus_sd_consul_rpc_duration_seconds_count Unknown endpoint, ins, instance, ip, job, cls, call N/A
prometheus_sd_consul_rpc_duration_seconds_sum Unknown endpoint, ins, instance, ip, job, cls, call N/A
prometheus_sd_consul_rpc_failures_total counter ins, instance, ip, job, cls The number of Consul RPC call failures.
prometheus_sd_discovered_targets gauge ins, instance, ip, config, job, cls Current number of discovered targets.
prometheus_sd_dns_lookup_failures_total counter ins, instance, ip, job, cls The number of DNS-SD lookup failures.
prometheus_sd_dns_lookups_total counter ins, instance, ip, job, cls The number of DNS-SD lookups.
prometheus_sd_failed_configs gauge ins, instance, ip, job, cls Current number of service discovery configurations that failed to load.
prometheus_sd_file_mtime_seconds gauge ins, instance, ip, filename, job, cls Timestamp (mtime) of files read by FileSD. Timestamp is set at read time.
prometheus_sd_file_read_errors_total counter ins, instance, ip, job, cls The number of File-SD read errors.
prometheus_sd_file_scan_duration_seconds summary ins, instance, ip, job, cls, quantile The duration of the File-SD scan in seconds.
prometheus_sd_file_scan_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_sd_file_scan_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_sd_file_watcher_errors_total counter ins, instance, ip, job, cls The number of File-SD errors caused by filesystem watch failures.
prometheus_sd_http_failures_total counter ins, instance, ip, job, cls Number of HTTP service discovery refresh failures.
prometheus_sd_kubernetes_events_total counter event, ins, instance, role, ip, job, cls The number of Kubernetes events handled.
prometheus_sd_kuma_fetch_duration_seconds summary ins, instance, ip, job, cls, quantile The duration of a Kuma MADS fetch call.
prometheus_sd_kuma_fetch_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_sd_kuma_fetch_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_sd_kuma_fetch_failures_total counter ins, instance, ip, job, cls The number of Kuma MADS fetch call failures.
prometheus_sd_kuma_fetch_skipped_updates_total counter ins, instance, ip, job, cls The number of Kuma MADS fetch calls that result in no updates to the targets.
prometheus_sd_linode_failures_total counter ins, instance, ip, job, cls Number of Linode service discovery refresh failures.
prometheus_sd_nomad_failures_total counter ins, instance, ip, job, cls Number of nomad service discovery refresh failures.
prometheus_sd_received_updates_total counter ins, instance, ip, job, cls Total number of update events received from the SD providers.
prometheus_sd_updates_total counter ins, instance, ip, job, cls Total number of update events sent to the SD consumers.
prometheus_target_interval_length_seconds summary ins, instance, interval, ip, job, cls, quantile Actual intervals between scrapes.
prometheus_target_interval_length_seconds_count Unknown ins, instance, interval, ip, job, cls N/A
prometheus_target_interval_length_seconds_sum Unknown ins, instance, interval, ip, job, cls N/A
prometheus_target_metadata_cache_bytes gauge ins, instance, ip, scrape_job, job, cls The number of bytes that are currently used for storing metric metadata in the cache
prometheus_target_metadata_cache_entries gauge ins, instance, ip, scrape_job, job, cls Total number of metric metadata entries in the cache
prometheus_target_scrape_pool_exceeded_label_limits_total counter ins, instance, ip, job, cls Total number of times scrape pools hit the label limits, during sync or config reload.
prometheus_target_scrape_pool_exceeded_target_limit_total counter ins, instance, ip, job, cls Total number of times scrape pools hit the target limit, during sync or config reload.
prometheus_target_scrape_pool_reloads_failed_total counter ins, instance, ip, job, cls Total number of failed scrape pool reloads.
prometheus_target_scrape_pool_reloads_total counter ins, instance, ip, job, cls Total number of scrape pool reloads.
prometheus_target_scrape_pool_sync_total counter ins, instance, ip, scrape_job, job, cls Total number of syncs that were executed on a scrape pool.
prometheus_target_scrape_pool_target_limit gauge ins, instance, ip, scrape_job, job, cls Maximum number of targets allowed in this scrape pool.
prometheus_target_scrape_pool_targets gauge ins, instance, ip, scrape_job, job, cls Current number of targets in this scrape pool.
prometheus_target_scrape_pools_failed_total counter ins, instance, ip, job, cls Total number of scrape pool creations that failed.
prometheus_target_scrape_pools_total counter ins, instance, ip, job, cls Total number of scrape pool creation attempts.
prometheus_target_scrapes_cache_flush_forced_total counter ins, instance, ip, job, cls How many times a scrape cache was flushed due to getting big while scrapes are failing.
prometheus_target_scrapes_exceeded_body_size_limit_total counter ins, instance, ip, job, cls Total number of scrapes that hit the body size limit
prometheus_target_scrapes_exceeded_native_histogram_bucket_limit_total counter ins, instance, ip, job, cls Total number of scrapes that hit the native histogram bucket limit and were rejected.
prometheus_target_scrapes_exceeded_sample_limit_total counter ins, instance, ip, job, cls Total number of scrapes that hit the sample limit and were rejected.
prometheus_target_scrapes_exemplar_out_of_order_total counter ins, instance, ip, job, cls Total number of exemplar rejected due to not being out of the expected order.
prometheus_target_scrapes_sample_duplicate_timestamp_total counter ins, instance, ip, job, cls Total number of samples rejected due to duplicate timestamps but different values.
prometheus_target_scrapes_sample_out_of_bounds_total counter ins, instance, ip, job, cls Total number of samples rejected due to timestamp falling outside of the time bounds.
prometheus_target_scrapes_sample_out_of_order_total counter ins, instance, ip, job, cls Total number of samples rejected due to not being out of the expected order.
prometheus_target_sync_failed_total counter ins, instance, ip, scrape_job, job, cls Total number of target sync failures.
prometheus_target_sync_length_seconds summary ins, instance, ip, scrape_job, job, cls, quantile Actual interval to sync the scrape pool.
prometheus_target_sync_length_seconds_count Unknown ins, instance, ip, scrape_job, job, cls N/A
prometheus_target_sync_length_seconds_sum Unknown ins, instance, ip, scrape_job, job, cls N/A
prometheus_template_text_expansion_failures_total counter ins, instance, ip, job, cls The total number of template text expansion failures.
prometheus_template_text_expansions_total counter ins, instance, ip, job, cls The total number of template text expansions.
prometheus_treecache_watcher_goroutines gauge ins, instance, ip, job, cls The current number of watcher goroutines.
prometheus_treecache_zookeeper_failures_total counter ins, instance, ip, job, cls The total number of ZooKeeper failures.
prometheus_tsdb_blocks_loaded gauge ins, instance, ip, job, cls Number of currently loaded data blocks
prometheus_tsdb_checkpoint_creations_failed_total counter ins, instance, ip, job, cls Total number of checkpoint creations that failed.
prometheus_tsdb_checkpoint_creations_total counter ins, instance, ip, job, cls Total number of checkpoint creations attempted.
prometheus_tsdb_checkpoint_deletions_failed_total counter ins, instance, ip, job, cls Total number of checkpoint deletions that failed.
prometheus_tsdb_checkpoint_deletions_total counter ins, instance, ip, job, cls Total number of checkpoint deletions attempted.
prometheus_tsdb_clean_start gauge ins, instance, ip, job, cls -1: lockfile is disabled. 0: a lockfile from a previous execution was replaced. 1: lockfile creation was clean
prometheus_tsdb_compaction_chunk_range_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_chunk_range_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_range_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_samples_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_chunk_samples_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_samples_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_chunk_size_bytes_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_populating_block gauge ins, instance, ip, job, cls Set to 1 when a block is currently being written to the disk.
prometheus_tsdb_compactions_failed_total counter ins, instance, ip, job, cls Total number of compactions that failed for the partition.
prometheus_tsdb_compactions_skipped_total counter ins, instance, ip, job, cls Total number of skipped compactions due to disabled auto compaction.
prometheus_tsdb_compactions_total counter ins, instance, ip, job, cls Total number of compactions that were executed for the partition.
prometheus_tsdb_compactions_triggered_total counter ins, instance, ip, job, cls Total number of triggered compactions for the partition.
prometheus_tsdb_data_replay_duration_seconds gauge ins, instance, ip, job, cls Time taken to replay the data on disk.
prometheus_tsdb_exemplar_exemplars_appended_total counter ins, instance, ip, job, cls Total number of appended exemplars.
prometheus_tsdb_exemplar_exemplars_in_storage gauge ins, instance, ip, job, cls Number of exemplars currently in circular storage.
prometheus_tsdb_exemplar_last_exemplars_timestamp_seconds gauge ins, instance, ip, job, cls The timestamp of the oldest exemplar stored in circular storage. Useful to check for what timerange the current exemplar buffer limit allows. This usually means the last timestampfor all exemplars for a typical setup. This is not true though if one of the series timestamp is in future compared to rest series.
prometheus_tsdb_exemplar_max_exemplars gauge ins, instance, ip, job, cls Total number of exemplars the exemplar storage can store, resizeable.
prometheus_tsdb_exemplar_out_of_order_exemplars_total counter ins, instance, ip, job, cls Total number of out of order exemplar ingestion failed attempts.
prometheus_tsdb_exemplar_series_with_exemplars_in_storage gauge ins, instance, ip, job, cls Number of series with exemplars currently in circular storage.
prometheus_tsdb_head_active_appenders gauge ins, instance, ip, job, cls Number of currently active appender transactions
prometheus_tsdb_head_chunks gauge ins, instance, ip, job, cls Total number of chunks in the head block.
prometheus_tsdb_head_chunks_created_total counter ins, instance, ip, job, cls Total number of chunks created in the head
prometheus_tsdb_head_chunks_removed_total counter ins, instance, ip, job, cls Total number of chunks removed in the head
prometheus_tsdb_head_chunks_storage_size_bytes gauge ins, instance, ip, job, cls Size of the chunks_head directory.
prometheus_tsdb_head_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_head_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_head_max_time gauge ins, instance, ip, job, cls Maximum timestamp of the head block. The unit is decided by the library consumer.
prometheus_tsdb_head_max_time_seconds gauge ins, instance, ip, job, cls Maximum timestamp of the head block.
prometheus_tsdb_head_min_time gauge ins, instance, ip, job, cls Minimum time bound of the head block. The unit is decided by the library consumer.
prometheus_tsdb_head_min_time_seconds gauge ins, instance, ip, job, cls Minimum time bound of the head block.
prometheus_tsdb_head_out_of_order_samples_appended_total counter ins, instance, ip, job, cls Total number of appended out of order samples.
prometheus_tsdb_head_samples_appended_total counter ins, instance, ip, type, job, cls Total number of appended samples.
prometheus_tsdb_head_series gauge ins, instance, ip, job, cls Total number of series in the head block.
prometheus_tsdb_head_series_created_total counter ins, instance, ip, job, cls Total number of series created in the head
prometheus_tsdb_head_series_not_found_total counter ins, instance, ip, job, cls Total number of requests for series that were not found.
prometheus_tsdb_head_series_removed_total counter ins, instance, ip, job, cls Total number of series removed in the head
prometheus_tsdb_head_truncations_failed_total counter ins, instance, ip, job, cls Total number of head truncations that failed.
prometheus_tsdb_head_truncations_total counter ins, instance, ip, job, cls Total number of head truncations attempted.
prometheus_tsdb_isolation_high_watermark gauge ins, instance, ip, job, cls The highest TSDB append ID that has been given out.
prometheus_tsdb_isolation_low_watermark gauge ins, instance, ip, job, cls The lowest TSDB append ID that is still referenced.
prometheus_tsdb_lowest_timestamp gauge ins, instance, ip, job, cls Lowest timestamp value stored in the database. The unit is decided by the library consumer.
prometheus_tsdb_lowest_timestamp_seconds gauge ins, instance, ip, job, cls Lowest timestamp value stored in the database.
prometheus_tsdb_mmap_chunk_corruptions_total counter ins, instance, ip, job, cls Total number of memory-mapped chunk corruptions.
prometheus_tsdb_mmap_chunks_total counter ins, instance, ip, job, cls Total number of chunks that were memory-mapped.
prometheus_tsdb_out_of_bound_samples_total counter ins, instance, ip, type, job, cls Total number of out of bound samples ingestion failed attempts with out of order support disabled.
prometheus_tsdb_out_of_order_samples_total counter ins, instance, ip, type, job, cls Total number of out of order samples ingestion failed attempts due to out of order being disabled.
prometheus_tsdb_reloads_failures_total counter ins, instance, ip, job, cls Number of times the database failed to reloadBlocks block data from disk.
prometheus_tsdb_reloads_total counter ins, instance, ip, job, cls Number of times the database reloaded block data from disk.
prometheus_tsdb_retention_limit_bytes gauge ins, instance, ip, job, cls Max number of bytes to be retained in the tsdb blocks, configured 0 means disabled
prometheus_tsdb_retention_limit_seconds gauge ins, instance, ip, job, cls How long to retain samples in storage.
prometheus_tsdb_size_retentions_total counter ins, instance, ip, job, cls The number of times that blocks were deleted because the maximum number of bytes was exceeded.
prometheus_tsdb_snapshot_replay_error_total counter ins, instance, ip, job, cls Total number snapshot replays that failed.
prometheus_tsdb_storage_blocks_bytes gauge ins, instance, ip, job, cls The number of bytes that are currently used for local storage by all blocks.
prometheus_tsdb_symbol_table_size_bytes gauge ins, instance, ip, job, cls Size of symbol table in memory for loaded blocks
prometheus_tsdb_time_retentions_total counter ins, instance, ip, job, cls The number of times that blocks were deleted because the maximum time limit was exceeded.
prometheus_tsdb_tombstone_cleanup_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_tombstone_cleanup_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_tombstone_cleanup_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_too_old_samples_total counter ins, instance, ip, type, job, cls Total number of out of order samples ingestion failed attempts with out of support enabled, but sample outside of time window.
prometheus_tsdb_vertical_compactions_total counter ins, instance, ip, job, cls Total number of compactions done on overlapping blocks.
prometheus_tsdb_wal_completed_pages_total counter ins, instance, ip, job, cls Total number of completed pages.
prometheus_tsdb_wal_corruptions_total counter ins, instance, ip, job, cls Total number of WAL corruptions.
prometheus_tsdb_wal_fsync_duration_seconds summary ins, instance, ip, job, cls, quantile Duration of write log fsync.
prometheus_tsdb_wal_fsync_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_fsync_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_page_flushes_total counter ins, instance, ip, job, cls Total number of page flushes.
prometheus_tsdb_wal_segment_current gauge ins, instance, ip, job, cls Write log segment index that TSDB is currently writing to.
prometheus_tsdb_wal_storage_size_bytes gauge ins, instance, ip, job, cls Size of the write log directory.
prometheus_tsdb_wal_truncate_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_truncate_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_truncations_failed_total counter ins, instance, ip, job, cls Total number of write log truncations that failed.
prometheus_tsdb_wal_truncations_total counter ins, instance, ip, job, cls Total number of write log truncations attempted.
prometheus_tsdb_wal_writes_failed_total counter ins, instance, ip, job, cls Total number of write log writes that failed.
prometheus_web_federation_errors_total counter ins, instance, ip, job, cls Total number of errors that occurred while sending federation responses.
prometheus_web_federation_warnings_total counter ins, instance, ip, job, cls Total number of warnings that occurred while sending federation responses.
promhttp_metric_handler_requests_in_flight gauge ins, instance, ip, job, cls Current number of scrapes being served.
promhttp_metric_handler_requests_total counter ins, instance, ip, job, cls, code Total number of scrapes by HTTP status code.
pushgateway_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which pushgateway was built, and the goos and goarch for the build.
pushgateway_http_requests_total counter ins, instance, method, ip, job, cls, code, handler Total HTTP requests processed by the Pushgateway, excluding scrapes.
querier_cache_added_new_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_added_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_entries gauge ins, instance, ip, job, cache, cls The total number of entries
querier_cache_evicted_total Unknown ins, instance, ip, job, reason, cache, cls N/A
querier_cache_gets_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_memory_bytes gauge ins, instance, ip, job, cache, cls The current cache size in bytes
querier_cache_misses_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_stale_gets_total Unknown ins, instance, ip, job, cache, cls N/A
ring_member_heartbeats_total Unknown ins, instance, ip, job, cls N/A
ring_member_tokens_owned gauge ins, instance, ip, job, cls The number of tokens owned in the ring.
ring_member_tokens_to_own gauge ins, instance, ip, job, cls The number of tokens to own in the ring.
scrape_duration_seconds Unknown ins, instance, ip, job, cls N/A
scrape_samples_post_metric_relabeling Unknown ins, instance, ip, job, cls N/A
scrape_samples_scraped Unknown ins, instance, ip, job, cls N/A
scrape_series_added Unknown ins, instance, ip, job, cls N/A
up Unknown ins, instance, ip, job, cls N/A

PING 指标

PING 任务包含有 54 类可用监控指标,由 blackbox_epxorter 提供。

Metric Name Type Labels Description
agent_up Unknown ins, ip, job, instance, cls N/A
probe_dns_lookup_time_seconds gauge ins, ip, job, instance, cls Returns the time taken for probe dns lookup in seconds
probe_duration_seconds gauge ins, ip, job, instance, cls Returns how long the probe took to complete in seconds
probe_icmp_duration_seconds gauge ins, ip, job, phase, instance, cls Duration of icmp request by phase
probe_icmp_reply_hop_limit gauge ins, ip, job, instance, cls Replied packet hop limit (TTL for ipv4)
probe_ip_addr_hash gauge ins, ip, job, instance, cls Specifies the hash of IP address. It’s useful to detect if the IP address changes.
probe_ip_protocol gauge ins, ip, job, instance, cls Specifies whether probe ip protocol is IP4 or IP6
probe_success gauge ins, ip, job, instance, cls Displays whether or not the probe was a success
scrape_duration_seconds Unknown ins, ip, job, instance, cls N/A
scrape_samples_post_metric_relabeling Unknown ins, ip, job, instance, cls N/A
scrape_samples_scraped Unknown ins, ip, job, instance, cls N/A
scrape_series_added Unknown ins, ip, job, instance, cls N/A
up Unknown ins, ip, job, instance, cls N/A

PUSH 指标

PushGateway 提供 44 类监控指标。

Metric Name Type Labels Description
agent_up Unknown job, cls, instance, ins, ip N/A
go_gc_duration_seconds summary job, cls, instance, ins, quantile, ip A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown job, cls, instance, ins, ip N/A
go_gc_duration_seconds_sum Unknown job, cls, instance, ins, ip N/A
go_goroutines gauge job, cls, instance, ins, ip Number of goroutines that currently exist.
go_info gauge job, cls, instance, ins, ip, version Information about the Go environment.
go_memstats_alloc_bytes counter job, cls, instance, ins, ip Total number of bytes allocated, even if freed.
go_memstats_alloc_bytes_total counter job, cls, instance, ins, ip Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter job, cls, instance, ins, ip Total number of frees.
go_memstats_gc_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge job, cls, instance, ins, ip Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge job, cls, instance, ins, ip Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge job, cls, instance, ins, ip Number of heap bytes that are in use.
go_memstats_heap_objects gauge job, cls, instance, ins, ip Number of allocated objects.
go_memstats_heap_released_bytes gauge job, cls, instance, ins, ip Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge job, cls, instance, ins, ip Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge job, cls, instance, ins, ip Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter job, cls, instance, ins, ip Total number of pointer lookups.
go_memstats_mallocs_total counter job, cls, instance, ins, ip Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge job, cls, instance, ins, ip Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge job, cls, instance, ins, ip Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge job, cls, instance, ins, ip Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge job, cls, instance, ins, ip Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge job, cls, instance, ins, ip Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge job, cls, instance, ins, ip Number of bytes obtained from system.
go_threads gauge job, cls, instance, ins, ip Number of OS threads created.
process_cpu_seconds_total counter job, cls, instance, ins, ip Total user and system CPU time spent in seconds.
process_max_fds gauge job, cls, instance, ins, ip Maximum number of open file descriptors.
process_open_fds gauge job, cls, instance, ins, ip Number of open file descriptors.
process_resident_memory_bytes gauge job, cls, instance, ins, ip Resident memory size in bytes.
process_start_time_seconds gauge job, cls, instance, ins, ip Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge job, cls, instance, ins, ip Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge job, cls, instance, ins, ip Maximum amount of virtual memory available in bytes.
pushgateway_build_info gauge job, goversion, cls, branch, instance, tags, revision, goarch, ins, ip, version, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which pushgateway was built, and the goos and goarch for the build.
pushgateway_http_requests_total counter job, cls, method, code, handler, instance, ins, ip Total HTTP requests processed by the Pushgateway, excluding scrapes.
scrape_duration_seconds Unknown job, cls, instance, ins, ip N/A
scrape_samples_post_metric_relabeling Unknown job, cls, instance, ins, ip N/A
scrape_samples_scraped Unknown job, cls, instance, ins, ip N/A
scrape_series_added Unknown job, cls, instance, ins, ip N/A
up Unknown job, cls, instance, ins, ip N/A

2 - 常见问题

Pigsty INFRA 基础设施模块常见问题答疑

INFRA模块中包含了哪些组件?

  • Ansible 用于自动化、部署和管理;
  • Nginx 用于公开对外暴露各种 WebUI 服务,并为提供一个本地软件源
  • 自签名 CA 用于 SSL/TLS 证书;
  • Prometheus 用于收集存储监控指标;
  • Grafana 用于监控/可视化;
  • Loki 用于收集存储查询日志;
  • AlertManager 用于告警聚合;
  • Chronyd 用于 NTP 时间同步;
  • DNSMasq 用于 DNS 注册和解析;
  • ETCD 作为 PGSQL HA 的 DCS;(可以部署到别的专用节点集群上)
  • 在管理节点上的 PostgreSQL 作为 CMDB;(可选)
  • Docker 用于无状态的应用程序和工具(可选)。

如何重新向 Prometheus 注册监控目标?

如果你不小心删除了基础设施节点上 Prometheus 的目标目录(/etc/prometheus/target),你可以使用以下命令再次向 Prometheus 注册监控目标:

./infra.yml -t register_prometheus  # 在 infra 节点上向 prometheus 注册所有 infra 目标
./node.yml  -t register_prometheus  # 在 infra 节点上向 prometheus 注册所有 node  目标
./etcd.yml  -t register_prometheus  # 在 infra 节点上向 prometheus 注册所有 etcd  目标
./minio.yml -t register_prometheus  # 在 infra 节点上向 prometheus 注册所有 minio 目标
./pgsql.yml -t register_prometheus  # 在 infra 节点上向 prometheus 注册所有 pgsql 目标

如何重新向 Grafana 注册 PostgreSQL 数据源?

pg_databases 中定义的 PGSQL 数据库默认会被注册为 Grafana 数据源(以供 PGCAT 应用使用)。

如果你不小心删除了在 Grafana 中注册的 postgres 数据源,你可以使用以下命令再次注册它们:

# 将所有(在 pg_databases 中定义的) pgsql 数据库注册为 grafana 数据源
./pgsql.yml -t register_grafana

如何重新向 Nginx 注册节点的 Haproxy 管控界面?

如果你不小心删除了 /etc/nginx/conf.d/haproxy 中的已注册 haproxy 代理设置,你可以使用以下命令再次恢复它们:

./node.yml -t register_nginx     # 在 infra 节点上向 nginx 注册所有 haproxy 管理页面的代理设置

如何恢复 DNSMASQ 中的域名注册记录?

PGSQL 集群/实例域名默认注册到 infra 节点的 /etc/hosts.d/<name>。你可以使用以下命令再次恢复它们:

./pgsql.yml -t pg_dns    # 在 infra 节点上向 dnsmasq 注册 pg 的 DNS 名称

如何使用Nginx对外暴露新的上游服务?

尽管您可以直接通过 IP:Port 的方式访问服务,但我们依然建议收敛访问入口,使用域名并统一从 Nginx 代理访问各类带有 Web 界面的服务。 这样有利于统一收口访问,减少暴露的端口,便于进行访问控制与审计。

如果你希望通过 Nginx 门户公开新的 WebUI 服务,你可以将服务定义添加到 infra_portal 参数中。 例如,下面是 Pigsty 官方 Demo 使用的 Infra 门户配置,对外暴露了几种额外的服务:

infra_portal:
  home         : { domain: home.pigsty.cc }
  grafana      : { domain: demo.pigsty.cc ,endpoint: "${admin_ip}:3000" ,websocket: true }
  prometheus   : { domain: p.pigsty.cc ,endpoint: "${admin_ip}:9090" }
  alertmanager : { domain: a.pigsty.cc ,endpoint: "${admin_ip}:9093" }
  blackbox     : { endpoint: "${admin_ip}:9115" }
  loki         : { endpoint: "${admin_ip}:3100" }
  # 新增的 Web 门户
  minio        : { domain: sss.pigsty  ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }
  postgrest    : { domain: api.pigsty.cc  ,endpoint: "127.0.0.1:8884"   }
  pgadmin      : { domain: adm.pigsty.cc  ,endpoint: "127.0.0.1:8885"   }
  pgweb        : { domain: cli.pigsty.cc  ,endpoint: "127.0.0.1:8886"   }
  bytebase     : { domain: ddl.pigsty.cc  ,endpoint: "127.0.0.1:8887"   }
  gitea        : { domain: git.pigsty.cc  ,endpoint: "127.0.0.1:8889"   }
  wiki         : { domain: wiki.pigsty.cc ,endpoint: "127.0.0.1:9002"   }
  noco         : { domain: noco.pigsty.cc ,endpoint: "127.0.0.1:9003"   }
  supa         : { domain: supa.pigsty.cc ,endpoint: "127.0.0.1:8000", websocket: true }

完成 Nginx 上游服务定义后,使用以下配置与命令,向 Nginx 注册新的服务。

./infra.yml -t nginx_config           # 重新生成 Nginx 配置文件
./infra.yml -t nginx_launch           # 更新并应用 Nginx 配置。

# 您也可以使用 Ansible 手工重载 Nginx 配置
ansible infra -b -a 'nginx -s reload'  # 重载Nginx配置

如果你希望通过 HTTPS 访问,你必须删除 files/pki/csr/pigsty.csrfiles/pki/nginx/pigsty.{key,crt} 以强制重新生成 Nginx SSL/TLS 证书以包括新上游的域名。 如果您希望使用权威机构签发的 SSL 证书,而不是 Pigsty 自签名 CA 颁发的证书,可以将其放置于 /etc/nginx/conf.d/cert/ 目录中并修改相应配置:/etc/nginx/conf.d/<name>.conf


如何手动向节点添加上游仓库的Repo文件?

Pigsty 有一个内置的包装脚本 bin/repo-add,它将调用 ansible 剧本 node.yml 来将 repo 文件添加到相应的节点。

bin/repo-add <selector> [modules]
bin/repo-add 10.10.10.10           # 为节点 10.10.10.10 添加 node 源
bin/repo-add infra   node,infra    # 为 infra 分组添加 node 和 infra 源
bin/repo-add infra   node,local    # 为 infra 分组添加节点仓库和本地pigsty源
bin/repo-add pg-test node,pgsql    # 为 pg-test 分组添加 node 和 pgsql 源