Saharsh Tibrewala
2 November 2026 · 6 minutes · cognee, open source, python, prometheus

Prometheus rules have no id, so I gave them one

Ask a Prometheus server for its rules twice, a few seconds apart, and you get two different answers, even if nobody touched a rule file. Each rule carries the time it was last evaluated, how long that took, whether it is firing, and the alerts it has active right now. For a dashboard that is the point. For an agent's memory it is noise, and expensive noise. If those fields go into the stored text, every content hash changes on every sync, and every rule is processed again each time.

A real alerting rule from the Prometheus demo server. Name, type, query, duration, labels and annotations are kept. State, health, alerts, evaluation time and last evaluation are dropped

That was the first thing to solve in the Prometheus connector I wrote for Cognee during Mergetober. The second was that rules have no id at all.

Why rules belong in memory

Alerting rules are where a team writes down what it cares about: which expression means trouble, how long it has to hold before anyone is woken up, how bad it is, and where the runbook lives. That knowledge usually sits in YAML files that only the person who wrote them reads.

I proposed the connector in cognee#5424 on October 6, was assigned, and opened cognee-community#397 on October 15. At the end I ran its example file with a real LLM, against a local server with my own rules, and asked which alerts page someone. It named the two rules labelled severity=page, what each one checks and their runbook URLs, and left out the one labelled severity=ticket.

Keep what the rule is

The connector makes one call, GET /api/v1/rules, and turns each alerting or recording rule into a document. With include_metadata=True it also reads GET /api/v1/metadata and adds one document per metric, with its type, unit and help text.

Each row is rebuilt from the fields that define the rule: name, group, file, interval, expression, for, keep_firing_for, labels and annotations. State, health, active alerts, evaluation time and last evaluation are left out. The request also sets exclude_alerts=true, so active alerts never leave the server, and the row builder drops them anyway for a server that ignores the parameter.

Against the public demo server, prometheus.demo.prometheus.io, two reads 15 seconds apart gave identical rows.

Make an id

Rules have no id, so the connector makes one: a hash of the file, the group, the type, the name and the labels. The name alone is not enough, and the demo server shows why.

The same alert name twice in one group: one with severity warning at under 40% free, one with severity critical at under 20% free. The row id is a hash of file, group, type, name and labels

NodeFilesystemSpaceFillingUp appears twice in the same group of the same file. One is a warning when a filesystem has under 40% free and will fill within 24 hours. The other is critical at under 20% and 4 hours. Keyed on the name, the second would overwrite the first, and memory would hold one of them. With the labels in the key they stay two documents. On that server five alert names repeat this way in the same group.

If two rules match on all of those fields, they get a counter, in the order the server lists them. My first version kept that counter on the source object, so running one source object twice gave every rule a new id and ingested all of them again. The counter is now made fresh on each run, with a test that runs one object twice.

Read everything, every time

The rule list is one request, so a cursor would save nothing. Every run reads the whole list and loads it with dlt's replace, and Cognee's orphan cleanup forgets whatever was in the last load and is not in this one. A removed rule drops out. An unchanged one keeps its content hash and its document id.

I checked this live, through cognee.add(), against a local Prometheus 3.15.0 with my own rule files.

Live runs: 5 rules gave 5 documents, a re-sync kept all 5 ids, changing one rule replaced 1 and kept 4, removing a rule forgot 1, an unreachable server raised an error and kept the documents, a wrong password stopped on 401

The row that matters most is near the bottom. With the server unreachable, the sync raised an error and the documents stayed. That is what the rest of the connector exists to protect.

Success that would forget

Under replace, a short answer that loads is the same as deleting everything it left out. The hard part is that Prometheus, and the systems that speak its API, can give a short answer with "status": "success".

Three answers that say success and would forget rules: an empty list of groups, a success with a warning from Thanos, and metric metadata that is empty or short after a restart

An empty list of rule groups is what a server returns when its rule files failed to load, or when a filter matches nothing. It is also the honest answer from a server with no rules. So the connector refuses an empty list only when an earlier sync had rules, and allow_empty=True lets one sync through when you really did remove them all. Through cognee.add(), the empty sync failed with all 5 documents kept.

Thanos Querier answers success with only the groups from the rulers it can reach, plus a warning when one is down. Any response that carries warnings now stops the run.

Metric metadata took three tries. Right after a restart, Prometheus answers /api/v1/metadata with an empty object, before its first scrapes, and that used to wipe the metrics table. My first fix refused every empty answer. That was too strict: a server with no scrape targets always answers empty, so it could never sync with metadata on. The second refused an empty answer only when an earlier load had metrics. That still forgot metrics, because Prometheus fills metadata in target by target after a restart, and a half filled answer is not empty. The connector now refuses metadata with under half the metrics of the last sync. allow_empty_metrics gets past that, and it is separate from allow_empty, so forgetting rules on purpose never takes the metrics with it.

One more came from the review of my Vercel connector. The empty check used to count rule groups before the rule_type and rule_groups filters ran. A server that ignores those filters answered with groups, none matched after filtering, and the empty table forgot every rule without an error. The check now counts what is left after filtering.

Small bugs that real servers found

  • dlt builds query parameters with Python's str.format, so a rule group name containing {{ reached the server as { and matched nothing. Braces are escaped now.
  • A plain string for rule_groups was split into letters, the same mistake the Vercel review found in project_ids. It is rejected now.
  • The server returned 1036800.5 for a for: of 12 days and 500 ms, which printed as 1.0368e+06s. Durations now print in whole milliseconds.
  • Default table names come from the server address, so two servers synced into one dataset keep their own documents. prom-eu.example.com and prom.eu.example.com used to fold into the same name and replace each other's rows. The name now ends in a hash of the exact address.
  • A NUL byte in a rule name or annotation would make Postgres refuse every load. Row text loses control characters now.

Testing it by breaking it

There are 137 tests. They run offline against a fake transport that serves responses recorded from a real Prometheus 3.15.0, on Python 3.10 to 3.14 and on dlt 1.9.0.

On the Vercel PR, the reviewer took lines out of my fixes and the tests stayed green, so passing tests alone did not tell me much. Here I broke the connector on purpose 37 ways: runtime state in the row, each response check, auth, headers, both filters, the duplicate counter, labels in the id, exclude_alerts, and the fixes from my review passes. Each break fails at least one test.

Mimir, Thanos and VictoriaMetrics serve a compatible rules API. I did not test the connector against them, and the README says so.

Where it stands

Pull request 397 on GitHub: open, 8 commits, +8,452 lines, no reviews yet

The PR is open and has no review yet. Everything from the Vercel review that applies here is already in it: the retry budget, error types that keep URLs and their query strings out of messages, keyword only arguments, sync stats, and the remembered list of groups that allow_empty forgot.

The Vercel story is in My Vercel connector passed its tests and could still forget your data, and everything I opened for Mergetober has the rest.

more writing

all the writing