Prometheus Targets

Prometheus runs on akira-mgmt-01-staging and listens only on 127.0.0.1:9090. Targets are scraped over Tailscale Magic DNS.

Job	Target	Labels
prometheus	localhost:9090	host=akira-mgmt-01-staging
node_exporter	akira-bastion-01-staging:9100	host=akira-bastion-01-staging, role=bastion
node_exporter	akira-db-01-staging:9100	host=akira-db-01-staging, role=db
node_exporter	akira-mgmt-01-staging:9100	host=akira-mgmt-01-staging, role=management
akira_api	akira-mgmt-01-staging:8000	service=akira-api
redis	akira-cache-01-staging:9121	role=cache, service=redis

Port allocation akira-mgmt-01-staging

Port	Service	Exposure
80, 443	Caddy	Public reverse proxy
3000	Next.js frontend	mgmt.akira-staging.asheep.it via Caddy
3001	Grafana	grafana.akira-staging.asheep.it via Caddy
3100	Loki	Internal, accessed via Grafana datasource
3200	Tempo	Localhost, accessed via Grafana datasource
4317	Tempo OTLP gRPC	Localhost and Docker network for backend export
4318	Tempo OTLP HTTP	Localhost, browser ingest via Caddy /otlp/*
8000	FastAPI	Internal now, future api.akira-staging.asheep.it route
9090	Prometheus	Internal, accessed via Grafana datasource
9093	Alertmanager	alerts.akira-staging.asheep.it via Caddy
9100	node_exporter	Tailscale interface only
9121	redis_exporter	Cache node, Tailscale interface only

To add a target, create infra/roles/prometheus/templates/scrape.d/<service>.yml.j2, add its explicit include to infra/roles/prometheus/templates/prometheus.yml.j2, and apply the prometheus role to akira-mgmt-01-staging. See docs/runbooks/observability-scrape-targets.md for the full workflow.

Verify scraping from akira-mgmt-01-staging:

curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, host: .labels.host, health: .health}'
curl -s 'localhost:9090/api/v1/query?query=up' | jq '.data.result[] | {host: .metric.host, role: .metric.role, value: .value[1]}'

If a target is down, first check Tailscale and the local service:

tailscale status
systemctl is-active node_exporter
curl -s localhost:9100/metrics | head