How it works
Applications expose metrics over HTTP, usually through a client library or an exporter that translates another system's statistics. Prometheus finds targets through static configuration or service discovery, including Kubernetes. It scrapes the targets at regular intervals. Each metric name and its complete label set identify a time series.
The ingestion path appends samples to the local time-series database. The head block holds recent samples, and a write-ahead log supports crash recovery. The database stores older data in immutable blocks and compacts them. Blocks contain compressed sample chunks and an index that finds series by labels.
PromQL selects series by labels and calculates rates, aggregates, and results over time windows. An instant query evaluates an expression at one time. A range query evaluates it repeatedly over a time interval. Queries use the server's own data unless an additional distributed query layer provides access to other data.
A rule evaluator runs recording rules and alerting rules at regular intervals. Recording rules save costly or frequently reused expressions as new series. Alerting rules produce alerts that Prometheus sends to Alertmanager. Alertmanager groups alerts, removes duplicates, routes alerts, and applies silences and inhibition.
Optional remote write sends samples to an external backend for longer retention or central queries. Federation lets one Prometheus server scrape selected series from another, often for aggregate views across clusters. Neither method gives the local Prometheus server automatic database sharding and replication.
What it gives you in a design
- Pull-based collection and target health
- Prometheus controls scrape frequency and records each scrape's success through the up metric. This shows whether Prometheus can reach targets. The design must explicitly provide network access from the scraper to the targets.
- Dimensional metrics
- Labels support separate views by service, region, status class, and route template. Each distinct label combination creates another series. Label design directly affects memory, storage, and query cost.
- PromQL aggregation and rates
- PromQL supports counter rates, ratios, joins, and calculations over time windows. For counters, apply rate before you aggregate across instances. This lets PromQL detect resets separately for each source.
- Latency distributions
- Histograms support latency distributions and quantile estimates that you can aggregate. Classic histograms expose cumulative bucket series. You generally cannot combine quantiles from client summaries into a valid percentile for the whole fleet.
- Recording and alerting rules
- Recording rules reduce repeated query work. Alerting rules express operational conditions through metrics. A for duration requires a condition to stay active across evaluations. This helps exclude brief spikes.
- Exporter ecosystem
- Exporters expose infrastructure and third-party service metrics without changes to their internal implementations. The monitor system's failure and capacity model must include exporter availability and collection cost.
Scaling, replication and failure
First, control active series, label churn, scrape frequency, and costly queries to increase scale. Estimate ingestion as samples per scrape divided by the scrape interval, summed across targets. Classic histogram buckets and label combinations can multiply that volume substantially. Recording rules reduce repeated query work but also create more stored series.
Divide scrape work by cluster, region, team, or target group when one server is insufficient. Standalone Prometheus does not automatically distribute storage and queries across servers. Systems such as Thanos or Grafana Mimir provide broader query and storage capabilities, with different architectures and operational costs.
For high availability, run independent Prometheus replicas that scrape the same targets. They do not synchronously replicate local databases or use consensus. Their samples and rule evaluations can differ. A compatible query layer or remote backend can remove duplicate replica data. Alertmanager can remove equivalent duplicate alerts when the alert identity excludes replica-specific labels.
Scrape failures cause observation gaps. A target restart does not recover measurements that Prometheus never collected. The WAL supports recovery of locally ingested data, not replay of missed scrapes. Remote write can buffer samples and retry during an outage, but it does not provide an unlimited durable queue. Long failures can exceed the recovery window. Monitor lag and check the behavior of the version you use.
Limits and defaults to know
Many of these values are configurable. Check the current documentation before you use an exact value.
| Default scrape interval | The global scrape_interval defaults to 1 minute. You can configure it globally and per scrape job. |
|---|---|
| Default scrape timeout | scrape_timeout defaults to 10 seconds and cannot exceed the scrape interval. You can configure it. |
| Default local retention | Local storage defaults to 15 days of retention if you configure neither a time limit nor a size limit. You can configure retention. Prometheus itself does not replicate local storage. |
| Default query timeout | The query.timeout flag defaults to 2 minutes. You can configure it. This limit does not guarantee that a large query will finish within that time. |
| Default lookback delta | The query lookback defaults to 5 minutes. You can configure it. Instant selectors use the latest eligible sample within this window. A staleness marker can make a series disappear sooner. |
Use it when
- You need operational metrics and alerts for request rate, errors, latency, saturation, and service availability.
- Targets can expose scrape endpoints, and the monitor network can reach them.
- You need label-based aggregation across workloads that change, such as Kubernetes pods.
- You can limit label cardinality and accept gaps in monitor data during collection failures.
- You want local metrics and alerts per cluster or region, with an optional separate backend for long-term metrics.
Avoid it when
- You need an exact ledger that supports replay for billing, payments, or other business transactions.
- Your main data consists of logs, trace spans, or arbitrary event payloads, not numeric time series.
- You need a separate metric series for every user, request, or transaction at large scale.
- You require automatic distributed storage, replication, and global queries from standalone Prometheus without additional components.
What interviewers ask about it
When Prometheus is on your whiteboard, expect these follow-up questions. Each answer gives the core of a strong answer.
- How do you estimate Prometheus capacity?
- Estimate active series from metric families and actual label combinations. Calculate samples per second from the scrape frequency. Include series churn, query concurrency, recording rules, retention, and histogram buckets. Test memory and disk needs with representative workloads. Do not use one limit for all servers.
- What happens when a scrape fails?
- Prometheus records up as 0 for the failed target. It receives no application samples from that target for that scrape. A later counter scrape can capture cumulative growth if the counter survived. However, Prometheus loses intermediate gauge values and event times.
- How do you give the monitor system high availability?
- Use independent Prometheus replicas and a highly available Alertmanager deployment. Monitor the monitor system itself. Normalize alert labels specific to each replica for deduplication. Use a compatible backend or query layer if users need a deduplicated view of both replicas.
- How do you distinguish an unhealthy service from missing telemetry?
- Create separate alerts for application symptoms, scrape failures, and missing expected series. Use expressions such as absent or absent_over_time for missing series. If service discovery removes a target, the target may disappear instead of retaining up equal to 0. Compare with an expected inventory or another independent signal when necessary.
- How do you calculate p99 latency for the whole fleet?
- For classic histograms, calculate counter rates per bucket. Aggregate across instances and preserve the le label. Apply histogram_quantile. The result is an estimate. Bucket boundaries determine its accuracy. An average of instance-level p99 values is not a valid replacement.
- When do you use federation instead of remote write?
- Use federation to let another Prometheus scrape selected raw or preaggregated series, often for a hierarchical overview. Use remote write to send samples to a compatible backend for central storage or queries. Do not assume that either path provides an unlimited buffer or automatic deduplication.
Common mistakes
- You put user IDs, request IDs, raw URLs, or unlimited error messages in labels. Instead, use dimensions with limits and links to logs or traces.
- You treat HA replicas as a database with synchronous replication. Or you sum both replicas and count the same workload twice.
- You average percentiles or apply rate after you aggregate counters across instances.
- You assume that no alert means a healthy service. You do not check scrape health, missing series, rule evaluation, and notification delivery.
Alternatives
- Grafana Mimir
- Choose it when Prometheus-compatible metrics need a shared backend with horizontal scale, multi-tenancy, and long retention. It adds distributed system components and operational complexity.
- Thanos
- Choose it when independent Prometheus deployments need one query view, removal of duplicate replica data, and historical data in object storage.
- Managed metrics services
- Choose a managed service when less storage and monitor administration matters more than full infrastructure control. Evaluate ingestion prices, cardinality limits, query behavior, and data-residency requirements.
Where it is used
Prometheus is a good fit for these interview questions. Each line tells you why.
- Design a metrics monitoring system like DatadogPrometheus provides a concrete reference for scrape-based ingestion, label cardinality, time-series storage, query evaluation, and alerts that account for missing data.
- Design a control plane for a distributed databasePrometheus can monitor fleet health, resource saturation, replication lag, and reconciliation failures. The durable store in the control plane remains the authoritative source for lifecycle state.
- Design an on-call escalation system like PagerDutyPrometheus and Alertmanager can supply grouped alerts. The escalation service handles durable escalation deadlines, acknowledgments, and provider delivery.
Common questions
Does Prometheus use push or pull?
Prometheus normally uses a pull model to collect data. It scrapes HTTP endpoints on a schedule. Pushgateway supports limited cases, such as service-level batch job results. It does not replace scrapes in general. It requires explicit lifecycle management for metrics that it retains.
What is the difference between Prometheus and Grafana?
Prometheus collects and stores metrics, runs PromQL queries, and evaluates alert rules. Grafana provides dashboards and visual displays across Prometheus and other data sources. Grafana can also evaluate alerts. Grafana is not the storage engine for Prometheus metrics.
Does Prometheus support long-term storage?
You can configure local retention. Prometheus can retain data longer if disk capacity permits. However, its local database has no cluster or replicas. Longer retention does not remove those limits. Designs that need durable shared history commonly add a remote backend or a system that uses object storage, such as Thanos.