Backups You Can't Restore Are Theatre: the Dgraph EE-format vs RDF trap

There is a specific, expensive way to discover your backup does not work: at 07:45 on the morning after something expensive went wrong. I found out the hard way that Dgraph's restore on the community image can report success, write 73 MB of postings, and still leave your database completely unqueryable. This is the postmortem of that discovery and the rule we now apply to every backup we own.

This is a companion to the full incident writeup (Dgraph Data Loss in Production); here I focus on the backup half of the lesson.

The backup that looked fine

Every night a cron wrote a Dgraph binary backup to /backups and kept seven days. On paper the bookstore was fully stocked:

backups/dgraph.20260912.010002.567/
  r6080539-g1.backup    # 58 MB
backups/manifest.json

A manifest. A recent timestamp. A size that said "there is real data here." This is what most people's backup monitoring stops at: something was written.

The restore that half-worked

When the production graph came back empty, I ran the documented restore:

rm -rf /data/*
dgraph restore --location /backups --postings /data --force_zero=false

It printed Restore version: 6080539, wrote 73 MB of postings, and exited cleanly. Memory pressure slightly eased. Then:

docker run --rm -v <volume>:/data --entrypoint dgraph \
  dgraph/standalone:v24.0.0 debug --postings /data/p1
# 2,715,574 keys — plenty of rows

Confirmed: Signal.hasCompany uid 270471, thousands of entries, all physically present on disk. So the cluster had data. And every query still came back empty:

q(func: type(Company))            → 0
q(func: has(Company.description), first: 2) → []

Individual uid() lookups resolved. Predicate and type-index lookups returned nothing. The restored posting lists were not the shape the running alpha could serve.

The short version of why: dgraph backup writes the Enterprise binary format, and restoring it is an Enterprise operation. On the community image, dgraph restore runs the code path, writes bytes to the posting directory, and hands you back — effectively — a ghost. There is no such thing as dgraph restore failing loudly here; it succeeds into a cluster that cannot serve the data. This is the distinction that cost us hours: "restore reported success" and "the data is readable" are two different facts, and the first does not imply the second.

What actually loads: the RDF export

The export that is restorable by construction is plain RDF (N-Quads plus schema). It is a text format any Dgraph — Enterprise or not — can load with a plain /mutate. On Dgraph v24 the export went through the GraphQL admin API (the old POST /admin/export REST endpoint is a 404 in v24):

curl -X POST localhost:8080/admin -H "Content-Type: application/graphql" \
  -d 'mutation { export(input: {format: "rdf", destination: "/backups"}) { response { code message } } }'

That queues an async task that lands as dgraph.r<ts>.../g01.rdf.gz plus a .gql_schema.gz. To restore:

head -c 1000 <(zcat g01.rdf.gz)      # schema block is on top
zcat g01.rdf.gz | sed '1,/^$/d' > data.rdf
curl -X POST localhost:8080/mutate?commitNow=true \
  -H "Content-Type: application/rdf" --data-binary @data.rdf

Restorable by construction, on any image, tested against an empty cluster.

The rule: a backup must prove itself

We replaced "the cron wrote a file" with "the backup fails loudly unless it could actually restore data." The new nightly job:

1. Triggers the RDF GraphQL export, not the binary Enterprise backup.
2. Verifies content — counts Company predicates and enforces a minimum size. If the export is empty or tiny, the job exit 1 and pages someone. An unverifiable backup is treated as no backup, with the same consequences as a full disk.
3. Randomly restores into a throwaway Dgraph on a schedule, and only then marks the backup "good."

Point 2 is the one most teams skip. A backup whose contents are never checked is not just worth little — it gives you false confidence at exactly the moment you are about to do something destructive. The size threshold caught nothing on the night of the incident truthfully, because the binary backup had real bytes; only a restore test would have caught it. So: verification must be at least a content check, and ideally a real restore into a scratch instance.

Checklist for your own database

• Pick a backup format the restore path you have can actually read. If "restore" is Enterprise-only, you have no restore path on the free build until you prove otherwise — export to a portable text format.
• Verify contents, not just file presence. Minimum row/predicate counts, not "file exists and is N MB."
• Test-restore on a schedule. A scratch container + a cron is hours cheaper than the next incident.
• SUCCESS from a restore tool is not the goal. The goal is type(X) returning rows from the server that will actually serve them.
• The dead database you migrated from counts as a backup too (see the companion post). Its restore is a migration you already know how to run.

Binary backup formats are seductive: they are "the real thing," compressed, single-file. But a backup's only job is to be restorable, and "restorable" is a property that must be demonstrated, not assumed. The RDF export and a restore test cost an hour to set up and save the next all-nighter.