Daily file reference

What is inside each daily file, how the daily deltas work, and how to apply them. How to fetch the files with your API key is on Data feeds and downloads.

What each day holds

Files are published into one directory per day, named for the UTC date the run started. Generation starts at midnight US Eastern (04:00 UTC, 05:00 during standard time) and the datasets publish one at a time over the following hours.

A day holds up to five datasets, each in two formats, plus a manifest.json describing them. Basic covers the first two datasets; Pro covers all five. Compare the plans.

Dataset	One row per	Key to upsert on	Plan
domains	registrable domain	domain	Basic
dns	name and record type, merged across vantage points	name, record_type	Basic
dns_vantage	name, record type and vantage point	name, record_type, vantage	Pro
certificates	certificate	cert_id	Pro
address_history	domain and address	domain, ip	Pro

A key appears at most once in a file. Every row carries the record's complete current state, not just the fields that changed.

File names

Each file is named {kind}.{dataset}.{format}.zst. The kind is delta for a daily file and snapshot for a full one, and the format is csv or json.

2026-09-17/manifest.json
2026-09-17/delta.domains.csv.zst
2026-09-17/delta.domains.json.zst
2026-09-17/delta.dns.csv.zst
2026-09-17/delta.dns.json.zst
2026-09-17/delta.dns_vantage.csv.zst
...

The CSV and JSON file of a dataset for a day are cut from the same data. They hold the same rows, with the same values, in the same order, so pick one format and ignore the other.

The manifest: what a day contains and what each file covers

manifest.json lists every file published for the day, sorted by name. It is the source of truth for what a day contains.

{
  "day": "2026-09-17",
  "generated": "2026-09-17T05:41:09Z",
  "files": [
    {
      "day": "2026-09-17",
      "name": "delta.domains.csv.zst",
      "dataset": "domains",
      "kind": "delta",
      "format": "csv",
      "since": "2026-09-16T00:00:00Z",
      "as_of": "2026-09-17T04:00:02Z",
      "rows": 61204877,
      "bytes": 1597224401,
      "sha256": "0f1c...e9a4"
    }
  ]
}

Field	Type	Meaning
day	string	The day directory, YYYY-MM-DD.
name	string	The file name within the day.
dataset	string	One of the five datasets.
kind	string	delta or snapshot.
format	string	csv or json.
since	timestamp | null	The lower bound this delta covers; see Deltas. null for a snapshot.
as_of	timestamp	When the data for this file was read. Everything recorded before this moment is in the file.
rows	integer	Data rows in the file, the same for both formats. The CSV header is not counted.
bytes	integer	Size of the compressed file.
sha256	string	SHA-256 of the compressed file, 64 lowercase hex characters.

A file listed in the manifest is complete. Files are written elsewhere and moved into the day only after their row counts and compression have been verified.

The manifest grows during the run. Each dataset is added when it publishes. A day where nothing has been published yet has no manifest at all.

A missing dataset is not an empty one. If a dataset could not be generated, it is left out of that day's manifest and its next file covers the gap: its since reaches back to cover the missed days. A dataset can also be added to a day's manifest later, including after that UTC day has ended, when it is generated again.

Trust sha256, not the size. A file can be regenerated, which changes its bytes and its hash. While that happens its entries briefly leave the manifest, so a manifest can for a moment list fewer files, or none at all.

A CSV file can have more lines than rows plus one, because a quoted value can contain a line break. Count records with a CSV parser, not with line counts.

Deltas: a lower bound and no upper bound

A delta holds every record that changed on or after its since up to the moment the file was generated.

since is 00:00 UTC of the day this dataset's previous file was published for. On a normal day that is yesterday. If a day was missed, it stays at the last published day, so the next file is larger and nothing falls between two files.

Because there is no upper bound, consecutive files overlap. The 2026-09-17 file covers from 2026-09-16 00:00, and the 2026-09-16 file already carried changes up to its own as_of that morning. Records that changed in that overlap appear in both files. Applying rows as upserts makes this harmless.

Dataset	A row is in the delta when
domains	Anything about the domain was updated on or after since: registration data, address, certificate or zone sighting.
dns	The answer for the name and record type changed at any vantage point on or after since.
dns_vantage	The answer at that vantage point changed on or after since.
certificates	We recorded the certificate on or after since, for the first time or again.
address_history	The address appeared in a new or changed answer for the domain on or after since.

Late arrivals still ship. Some records reach us after the fact and carry an older timestamp, such as a zone file that is a day old when we process it. A delta also includes every record we received on or after since, so these arrive in the next file even though their dates are earlier than since.

Historical backfill does not. When we import older data in bulk, it reaches you through the full snapshot rather than as a sudden oversized daily file.

Applying a delta

Upsert every row on its dataset's key, replacing the whole row. Apply each dataset's files in day order. Replaying an older file after a newer one would move rows backwards.

• Check for a gap before applying. Keep, for each dataset, the day of the last file you applied. If a file's since is later than 00:00 UTC of that day, a file between the two is missing: apply the missing one first.
• Nothing is ever deleted by a delta. A record that stops appearing keeps its last values. A DNS record type that stops answering, for example, remains with the values it last had.
• Address history: current moves between rows. A delta carries the address a domain moved to, with current set to true. It does not re-send the address the domain moved away from. When you apply a row with current true, set current to false on every other address you hold for that domain.
• Confirmation times refresh through the snapshot. A delta is driven by changes. When a name is re-resolved and its answer has not changed, its last_confirmed or last_seen moves forward in our data but does not by itself put the row in a delta for domains, DNS or certificates; address history is keyed on the sighting itself, so it does move. The full snapshot carries the current confirmation time for every record.

An example for the DuckDB command-line client, applying one day's domains delta to a local table keyed on domain. Verify the file first, against the manifest's sha256 and with zstd -t: DuckDB reads a truncated .zst without reporting an error.

.bail on
BEGIN;
DROP TABLE IF EXISTS d;
CREATE TEMP TABLE d AS
  SELECT * FROM read_csv('delta.domains.csv.zst', header = true, all_varchar = true,
                         allow_quoted_nulls = false);
CREATE TABLE IF NOT EXISTS domains AS SELECT * FROM d LIMIT 0;
DELETE FROM domains WHERE domain IN (SELECT d.domain FROM d);
INSERT INTO domains SELECT * FROM d;
COMMIT;

Each line guards against a failure we reproduced. Without .bail on the client carries on past an error, and COMMIT keeps the DELETE of a file that did not load. With domain unqualified in the subquery, a file that loads without a domain column makes the DELETE remove every row. Without allow_quoted_nulls = false, empty values load as NULL instead of empty strings. DROP TABLE IF EXISTS d lets one session apply several days in turn.

The full snapshot

The whole corpus for each dataset is published as snapshot.{dataset} files, with the same columns and encoding as the deltas. Only the latest is kept on the download page. A snapshot's manifest entries carry kind snapshot and a null since. Seed a new system from it, then apply the daily deltas published after its as_of.

Retention and the download allowance

Files stay downloadable here for 14 days, counting today. Every file we publish is archived permanently. Those are two different numbers and it is worth keeping them apart: 14 days is how long a file stays on the download page, not how long the data lives. A consumer that falls further behind than that has missed deltas and needs to re-seed: take the latest full snapshot from the download page and apply the deltas published after its as_of.

The archive is what makes the 14-day certificate window in the database a working window rather than a limit. Certificate detail ages out of the API after 14 days; the daily file that recorded it does not age out at all.

Each file can be downloaded once per day, shared between the API and the buttons in your account. How to automate the download within that allowance is covered in Automating the download.

Encoding rules shared by every file

Topic	Rule
Compression	Zstandard. DuckDB, polars, pandas and ClickHouse read .zst directly; otherwise stream through zstd -dc.
CSV	A header row naming the columns in the order given below. Every string value and every header name is double-quoted, with a literal " written as "". Integers and booleans are not quoted. Lines end with LF.
JSON	One object per line, keys in column order. Standard JSON escapes apply, and / is written as \/.
Arrays	A real JSON array in the JSON file. In the CSV file the cell holds the same JSON text, quoted as a CSV string.
Timestamps	YYYY-MM-DDTHH:MM:SSZ in UTC, whole seconds.
Unknown values	An empty string, "", in both formats. Never null, never a placeholder date, never :: or a hash of zeros. An empty value is an answer about what we hold: we hold no value for that field.
Integers and booleans	Never empty. Booleans are true or false.
Addresses	IPv4 as a dotted quad, never as ::ffff:-mapped IPv6. IPv6 in its standard compressed text form.
Hex	Lowercase.
Row order	Identical in the CSV and JSON file of the same dataset and day. Otherwise unspecified.
Empty files	A delta with no rows is still published: a CSV file with only the header row and an empty JSON file.

Basic: domains

One row per registrable domain, keyed on domain. Where the domain lookup API publishes the same fact, the column carries the API field's name.

Column	Type	Meaning
domain	string	The registrable domain, resolved through the Public Suffix List and lowercased.
tld	string	The final label, with no leading dot. A .co.uk domain carries uk.
registered	timestamp	The earliest creation date in the registry (WHOIS and RDAP) data we hold for the domain. Empty where we hold none, which can happen while registrar and expires are filled. Never inferred from certificate dates or zone files. A domain that was deleted and registered again keeps the earliest creation date we hold.
expires	timestamp	The latest expiry date in the registry data we hold.
registrar	string	A sponsoring registrar named in the registry data we hold. After a transfer it may not be the current one.
ip	string	An address from the most recent new or changed A or AAAA answer our resolvers recorded for the domain or any name under it. One address, even where the answer holds several; which one is arbitrary, and it can change without the answer changing. Empty where our resolvers have recorded no address for the domain.
first_cert_seen	timestamp	The earliest time our Certificate Transparency collection recorded a certificate for the domain. It is not when the domain's first certificate was issued, so it can be later than the certificate's own date, and it is empty where every certificate we hold for the domain predates our record of it.
first_zone_seen	timestamp	When the domain first appeared as a new name in a registry zone file we hold. Our zone history begins in March 2025, so a domain already listed then and never deleted has none. It is an upper bound on the domain's first registration. For a domain that was deleted and registered again it can predate the current registration by months.
last_update	timestamp	The newest timestamp we hold for the domain from any source: a DNS answer, a certificate, a zone file or a registry record. For registry data it is when the record was fetched, which can predate our other collection.

Basic: DNS

One row per name and record type, merged across our vantage points, keyed on name and record_type.

DNS does not expire on a clock. We hold the current answer for as long as the name still resolves, however many years that runs, and we keep the record of what changed for 18 months. A name that stops answering entirely leaves the live data; the daily files that recorded it keep it permanently. There is no fixed window to design around.

Column	Type	Meaning
name	string	The fully qualified name.
record_type	string	A, AAAA, NS, MX, TXT and the rest.
values	array of strings	Every value in the current answer at any vantage point, de-duplicated. The order carries no meaning.
ttl	integer	TTL in seconds, the largest across vantage points, as the resolver returned it when the answer last changed. It is not refreshed by re-resolutions that return the same answer, and a resolver's cached TTL can be lower than the authoritative one.
last_confirmed	timestamp	The most recent time any vantage point resolved this name and type and either got its current answer back unchanged or saw it change.

Unchanged re-resolutions have been recorded since 2026-09-16. A name not resolved again since then carries the time its answer last changed; each name is picked up as the sweep reaches it.

Pro: DNS by vantage point

One row per name, record type and vantage point, keyed on all three. A name can answer differently by geography, and the merged Basic file hides that.

Column	Type	Meaning
name	string	The fully qualified name.
record_type	string	Record type.
vantage	string	An identifier for the resolver node that saw this answer, such as fra, lax, nyc or dal. Several nodes can share a region, and nrd is the lane that resolves newly registered domains. Treat it as an opaque label: nodes are added over time.
values	array of strings	The answer this vantage point holds, unmerged.
ttl	integer	TTL in seconds as this vantage point's resolver returned it when the answer last changed. Re-resolutions that return the same answer do not refresh it.
last_seen	timestamp	The most recent time this vantage point resolved the name and type and either got the same answer back or saw it change.
last_changed	timestamp	When the answer at this vantage point last changed.
changes	integer	How many times the answer has changed at this vantage point. A high count on an A record marks rotation; on NS, a migration.

Before 2026-09-17, most heavily on 2026-08-27 and 2026-08-28, a fault on our side reset some rows without a real change: last_changed moved to the time of the check and changes went back to 0. The next real change corrects last_changed. It does not correct changes, which stays lower than the true count.

Pro: certificates

One row per certificate, keyed on cert_id. The Pro API returns the same fields for a single certificate.

Certificates cover a rolling 14 days in the database, and every daily file is kept permanently. A certificate we have not recorded for 14 days leaves the database, and first_observed and ct_logs_seen count only the records still held there. The daily certificates file for the day we recorded it keeps the full row regardless, so a date outside the window is a file to read rather than a gap in what we observed.

Column	Type	Meaning
cert_id	string	Our certificate key, 32 lowercase hex characters. It is the same value as a domain record's ssl_cert, which the API writes in uppercase, so join on it case-insensitively.
cert_sha256	string	SHA-256 of the DER-encoded certificate. Empty where we have seen only the precertificate.
serial	string	Serial number, hex.
spki_sha256	string	SHA-256 of the SubjectPublicKeyInfo. Certificates sharing it were issued for the same key pair. Empty where we have not derived it.
issuer	string	Full issuer distinguished name, for example CN=YE2,O=Let's Encrypt,C=US.
issuer_org	string	Issuing CA organization on its own, for grouping.
subject_cn	string	Subject common name. Empty on certificates that carry names only in the SAN extension, and where we did not derive it, which happens only on certificates first recorded before 03:00 UTC on 2026-09-07.
key_alg	string	Public key algorithm, such as RSA or ECDSA.
key_bits	integer	Key size in bits. 0 where we did not derive it, which happens only on certificates first recorded before 03:00 UTC on 2026-09-07.
signature_alg	string	Signature algorithm, such as SHA256-RSA.
sans	array of strings	Every name the certificate covers. This is the column that links co-tenants behind one CDN certificate.
san_count	integer	Length of sans.
wildcard	boolean	True where any SAN begins with *..
ja4x	string	JA4X fingerprint of the certificate's structure. Certificates minted by the same tooling share it.
first_observed	timestamp	When we first recorded the certificate from a CT log, within the 14 days we keep.
last_observed	timestamp	When we last recorded it.
ct_logs_seen	integer	How many CT logs we have seen it in, within the 14 days we keep.

Pro: address history

One row per domain and address, keyed on domain and ip. It covers addresses our resolvers saw for the domain and for any name under it, including the ones it has moved away from. An address known to us only from a third-party dataset is not included.

Column	Type	Meaning
domain	string	The registrable domain.
ip	string	An address the domain, or a name under it, resolved to.
first_seen	timestamp	The first time this address appeared in an answer for the domain. For some addresses that also appear in a third-party hosting dataset we loaded, it is 2026-08-05, the date of that dataset, rather than our own first answer.
last_seen	timestamp	The last time this address appeared in a new or changed answer. Re-resolutions that return the same answer do not move it; the DNS files carry confirmation times.
observed	integer	How many new or changed answers included this address. For an address that also appears in the third-party hosting dataset, the count includes that sighting once.
current	boolean	True for the address in the domains file's ip column; which one is arbitrary when several were recorded together. When you apply a row with current true, clear it on the domain's other addresses; see Applying a delta.

Sample rows

The same two domains rows in each format, as the generator writes them. The second domain has no registration data, so those cells are empty.

"domain","tld","registered","expires","registrar","ip","first_cert_seen","first_zone_seen","last_update"
"example-shop.com","com","2019-04-02T11:15:00Z","2027-04-02T11:15:00Z","GoDaddy.com, LLC","203.0.113.42","2026-09-06T06:22:11Z","","2026-09-16T04:11:50Z"
"newly-parked.xyz","xyz","","","","","2026-09-16T01:19:30Z","2026-09-15T22:07:48Z","2026-09-16T04:12:44Z"

{"domain":"example-shop.com","tld":"com","registered":"2019-04-02T11:15:00Z","expires":"2027-04-02T11:15:00Z","registrar":"GoDaddy.com, LLC","ip":"203.0.113.42","first_cert_seen":"2026-09-06T06:22:11Z","first_zone_seen":"","last_update":"2026-09-16T04:11:50Z"}
{"domain":"newly-parked.xyz","tld":"xyz","registered":"","expires":"","registrar":"","ip":"","first_cert_seen":"2026-09-16T01:19:30Z","first_zone_seen":"2026-09-15T22:07:48Z","last_update":"2026-09-16T04:12:44Z"}

A DNS-by-vantage row with an array value. In the CSV file the array is JSON text inside a quoted cell, so its quotes are doubled.

"name","record_type","vantage","values","ttl","last_seen","last_changed","changes"
"example-shop.com","TXT","fra","[""v=spf1 ip4:203.0.113.0\/24 -all""]",300,"2026-09-16T05:02:11Z","2026-08-30T17:40:02Z",1

{"name":"example-shop.com","record_type":"TXT","vantage":"fra","values":["v=spf1 ip4:203.0.113.0\/24 -all"],"ttl":300,"last_seen":"2026-09-16T05:02:11Z","last_changed":"2026-08-30T17:40:02Z","changes":1}

Where to go next

• Data feeds and downloads: fetching files with your API key, the download allowance, a sync script, and the live newly registered domains feed.
• Public API: the domain lookup whose field names the files reuse.
• Pro API: certificate, SPKI and reverse IP pivots.
• Pricing: which datasets each plan includes.

Last updated September 17, 2026. Questions the docs do not answer: get in touch.

↑ Top