Prometheus

Understand Prometheus for Kubernetes metrics collection, service discovery, alerting rules, recording rules, and production observability.

Last reviewed: June 2026

·Kubernetes 1.28+kubectl 1.28+Helm 3+Managed Kubernetes: EKS, AKS, GKE
✅
Quick Understanding: Prometheus collects and stores time-series metrics from Kubernetes and applications.

🔍 What Is It?

Prometheus is a monitoring system widely used in Kubernetes.

It scrapes metrics endpoints, stores time-series data, evaluates alerting rules, and supports PromQL queries.

🎯 Why Do We Need It?

Production Kubernetes needs visibility into Pods, nodes, deployments, ingress, databases, queues, and business metrics.

Common developer mistakes:

  • →Treating Kubernetes YAML as copy-paste configuration instead of production architecture
  • →Ignoring requests, limits, health probes, security context, and rollout behavior
  • →Debugging only the Pod while forgetting Services, DNS, NetworkPolicies, nodes, and admission controls
  • →Manually changing live objects without updating Git or infrastructure automation

🏗️ What Happens in Real Projects?

A platform team deploys Prometheus Operator. Application teams expose /metrics endpoints. Alerts fire for high error rate, Pod crash loops, HPA maxed out, node pressure, and API latency.

⚙️ How It Works

Prometheus discovers targets through Kubernetes metadata or ServiceMonitor/PodMonitor resources. It scrapes metrics at intervals, stores samples, evaluates rules, and sends alerts to Alertmanager.

💻 Example

Prometheus alert idea — A production alert for API error rate.

text
Alert: HighApiErrorRate
Expression:
  rate(http_requests_total{status=~"5.."}[5m])
  /
  rate(http_requests_total[5m]) > 0.05

Response:
  notify on-call
  include dashboard and runbook
  inspect rollout and dependency health

🤔 Common Confusions

❓ Is Prometheus for logs?

No. Prometheus is for metrics. Logs need a logging system such as Loki, Elasticsearch, or cloud logging.

❓ Does Prometheus automatically know business metrics?

No. Applications must expose meaningful custom metrics.

❓ Is every alert useful?

No. Alerts should be actionable and tied to user impact or urgent risk.

🚨 Common Production Issues

**Too many noisy alerts** causes alert fatigue.

**No application metrics** hides business failures.

**High cardinality labels** overload Prometheus.

**No retention or remote storage plan** loses investigation data.

✅ Best Practices

Expose RED/USE metrics for services and infrastructure

Keep labels controlled to avoid high cardinality

Attach runbooks to alerts

Use recording rules for expensive queries

Review alerts after incidents

🏛️ Architect's Note

Senior Architect's Perspective

Prometheus is valuable when metrics describe user impact and system saturation. Random dashboards are not observability.

💬 Frequently Asked Questions

Interview: How would you explain Prometheus in a real Kubernetes project?

Prometheus should be explained through the workload it supports, the Kubernetes object relationships, the failure modes, the security controls, and how it is observed and rolled back in production.

Interview: What production mistake do developers commonly make with Prometheus?

The common mistake is making the object work once, then skipping ownership, resource sizing, rollout safety, monitoring, least privilege, and recovery behavior.

🎯 Key Takeaways

1Prometheus stores time-series metrics
2PromQL powers queries and alerts
3Application metrics are essential
4Alert quality matters more than alert quantity

🎯 Practice this topic in interviews

↗ Where this topic fits

Sources

Sources / References

Use these Kubernetes references to verify the concept in official docs and continue into the most relevant SplashRide pages.

Related SplashRide pages