Dgraph Data Loss in Production: from a Silent OOM to a Neo4j Rebuild

There is a moment in every database incident when you realize the backup is not a backup. For me it was 07:48 on a Saturday morning, staring at Dgraph health returning max_assigned: 40003, knowing the volume below it held 2.7 million keys — and having zero queries return a single predicate.

This is the story of how a lead-scoring graph lost 6,500 companies in less than an hour, why the "restore" that ran on top of it produced data nobody could read, and how the recovery actually went through a database we had migrated away from months earlier.

The topology that set the trap

The production stack runs leads.contextual-intelligence.org: a Fastify app on host networking, PostgreSQL, and Dgraph (dgraph/standalone:v24.0.0) for the company/signal graph that powers /api/graph/score.

Because the app container uses network_mode: host, there is no Docker DNS. Dependencies are pinned in the compose file via extra_hosts:

network_mode: host
extra_hosts:
  - "dgraph:10.30.0.3" # the dgraph container, static IP on leads-internal
  - "postgres:10.30.0.2"

So DGRAPH_URL=http://dgraph:8080 in the app env resolves through those hosts. It works. It also means the operators have to remember that the "real" dgraph port is 18080 → 8080 on the host, and that attacking the wrong port on the wrong host produces very confusing "the database is empty" readings.

The graph had grown to roughly 6,500 companies and 21,000 signals, built up by a nightly ingestion pipeline over months.

The trigger: five concurrent full-graph scans

The trigger was boring and my fault. A load-test suite had a case that fired five concurrent GET /api/graph/score requests. Each request scans and scores the entire company graph in memory. On a healthy box that's a second or two. Five at once, on a graph that had just more than doubled in size, was not.

At 06:50:53 UTC the host's OOM killer shot down the leads-dgraph container:

ExitCode: 255, OOMKilled: true

The container had no memory ceiling (mem_limit unset), so instead of the kernel recycling dgraph — the correct outcome — it let dgraph balloon and then killed whoever the kernel happened to pick. It got dgraph. The scoring endpoint went from 200 to 500:

{ "ok": false, "error": { "code": "SCORING_FAILED", "message": "fetch failed" } }

A restart brought it back, slowly: 6,456 companies, 18–20s scans. Degraded, but alive. The graph data was still on disk.

The deploy that made it "empty"

Then a routine deploy ran. The deploy workflow does:

docker compose -f docker-compose.prod.yml up -d --force-recreate leads

That command bumps the app. But compose also recreates a service's depends_on dependencies when their config changed — and in the very same commit I had added a mem_limit to the dgraph service. So the deploy recreated Dgraph too, SIGKILLing the recovered containers mid-recovery.

From then on, every graph query returned zero companies. GET /api/graph/score came back 200 OK with an empty summary:

{ "ok": true, "data": { "summary": { "total": 0, "hot": 0, "warm": 0, "cold": 0 } } }

Zero. Not an error — an empty 200. Which is the worst possible failure mode, because it does not page anyone and every health check stays green.

I will come back to this: the deploy infrastructure had been allowed to destroy stateful infrastructure. The only reason nobody had hit it before is that previous deploys had not changed the dgraph service's config. A one-line memory ceiling change became a data-loss event.

The backup that restored into a ghost town

We had backups. Every night, 03:00 UTC, a cron wrote a Dgraph binary backup to /backups and kept seven days. On paper everything was fine.

The restore went like this:

rm -rf /data/*
dgraph restore --location /backups --postings /data --force_zero=false

It wrote 73 MB of postings, reported 2,715,574 keys, and finished with "Restore version: 6080539". dgraph debug --postings /data/p1 confirmed the keys were physically there:

{d} ns: 0x0 attr: Signal.hasCompany uid: 270471 ts: 1

The Dgraph timestamp lease had to be re-aligned by hand because of --force_zero=false:

curl "localhost:6080/assign?what=timestamps&num=6080600"
# {"startId":"40006","endId":"6120605"}

Reads moved to start_ts: 6120607 — comfortably above the restore version. Alpha was healthy. And every meaningful query still returned nothing:

q(func: type(Company))        → 0
q(func: has(Company.description), first: 2)  → []

Individual uid lookups worked. Predicate and type index lookups returned empty. This is the shape of a restore that materialized postings the running cluster cannot serve — a community-format dgraph restore (an Enterprise feature on this image) that "succeeds" while producing data no query can reach. We burned hours proving this with dgraph debug before stopping.

The real source of truth was a "dead" database

While we were chasing the restore, someone noticed the other server still ran leads-neo4j — the Neo4j instance the graph had been migrated away from months earlier. It had been left running, and post-migration ingestion had added companies on top, but the original 5,267 companies were still there.

Neo4j listened on a bridge-only address with no published port. To reach it I chained two SSH tunnels through my workstation:

# hop 1: my machine → the neo4j host, forward the container's bridge IP
ssh -N -L 17687:172.21.0.2:7687 weiss@178.254.2.90 &
# hop 2: my machine → the dgraph host, reverse-bind the same port
ssh -N -R 17687:localhost:17687 weiss@195.90.216.159 &

Now the dgraph host's loopback localhost:17687 reached Neo4j through my laptop. Add a forward for dgraph itself:

ssh -N -L 18080:localhost:18080 weiss@195.90.216.159 &

This is undignified but it works, and it is how I can run the migration tool from a checkout on my laptop against both databases.

The rebuild: schema, seed, migrate, verify

The repo already had the two halves of the recovery path:

1. reimport-dgraph.ts applies the schema, truncates, and seeds the owning company + products. The tenant is literally default:

   DGRAPH_URL=http://localhost:18080 SKIP_INGEST=true \
   TENANT_NAME=default COMPANY_NAME="Siemens Healthineers" \
     npx tsx packages/server/src/scripts/reimport-dgraph.ts
   

2. neo4j-to-dgraph-export.ts reads every company, product, application and signal from Neo4j and writes them through the application's own DgraphRepository (which writes prefixed predicates like Signal.hasCompany, exactly what the GraphQL layer reads):

   NEO4J_URI=bolt://localhost:17687 NEO4J_USER=neo4j NEO4J_PASSWORD=... \
   TENANT_ID=default DGRAPH_URL=http://localhost:18080 \
     npx tsx packages/server/src/scripts/neo4j-to-dgraph-export.ts
   

Thirty-five minutes and 5,267 companies / 10 products / 20,958 signals later, the endpoint was healthy again:

graph/score: 200 in 1.8s
summary: { total: 5240, hot: 38, warm: 150, cold: 5052 }

Recovery complete. But the part that mattered was what we changed so it cannot quietly happen again.

The guards (write them down)

1. Stateful services are pets — app deploys never touch them

The core mistake was letting a deploy recreate Dgraph because its config changed in the app repo's compose file. The deploy now excludes dependencies:

docker compose up -d --force-recreate --no-deps leads

--no-deps means an app deploy can never recreate dgraph or postgres, regardless of how their compose stanza changes. Stateful migrations go through the reimport profile, explicitly, on purpose.

2. A runaway query must recycle the container, not the host

mem_limit: 8g
memswap_limit: 10g
stop_grace_period: 60s

The ceiling means a burst of scans grows dgraph until it recycles the container — the right victim — instead of ballooning to take the host down. stop_grace_period gives zero+alpha time to checkpoint on SIGTERM rather than being SIGKILLed mid-write.

3. A backup that cannot be restored is not a backup

We replaced the binary backup with the RDF export — plain N-Quads plus schema — which loads back with a plain /mutate on any Dgraph:

curl -X POST localhost:8080/admin -H "Content-Type: application/graphql" \
  -d 'mutation { export(input: {format: "rdf", destination: "/backups"}) { response { code } } }'

(The old POST /admin/export REST endpoint returns 404 on Dgraph v24 — the export moved to GraphQL.) And we make the backup prove itself: the cron fails loudly unless the export contains at least a few thousand Company predicates. An unverifiable backup is treated as no backup, so a silent empty-graph export fails the same way a full disk would.

4. An empty graph must not look healthy

/api/health stayed green through the entire incident because it never looks at graph data. The production smoke suite now has a tripwire that scores the graph and asserts summary.total > 1000 every four hours. If the graph is empty, someone gets an email — not a surprise at the next load test.

5. Load tests are production pushes

The five concurrent scans that started this are now three, matching the 8 GB ceiling, and the "budget" assertions are documented as ceilings rather than a stale 1.5 s number from when the graph was half the size. Blasting five full-graph scans at prod is a load test only if you intend to run it as one.

The hard-won rules

• 200 OK with empty data is a worse failure than 500. A 500 pages someone. Empty-alive stays green. If you have a cache-addicted, aggregate-reading endpoint, put a minimum-row assertion behind it.
• Deploy tooling that can recreate a database is a foot-gun waiting for a config change. --no-deps, and keep stateful compose stanzas out of the app deploy path.
• "Restore succeeded" and "data is readable" are different facts. Verify with type(X) / has(predicate) counts on the server that will serve them, not with a dgraph debug key listing.
• The database you migrated away from is often your best backup. It had grown stale, but 5,267 companies of lineage were still correct. Don't tear down old infrastructure as "cleanup" until you have a proven migration out — and prove it by restoring, not by a log line.
• Write the runbook before you need it. The rebuild procedure (tunnels, reimport, neo4j export, verification) is now documented in the repo. The recovery that took hours this time is a scripted half hour next time.

We lost roughly the post-migration delta — the ~1,200 adapter-sourced companies above the Neo4j archive. The nightly pipeline re-added them over the following weeks, which is the correct outcome for a growth source. But the lineage — the scored graph we actually depended on — came back from a database we had officially stopped using, and that is the part worth remembering.