Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,515 words · 28 segments analyzed
When I last wrote about our DNS propagation pipeline in 2022, the Zone Builder received about 250 DNS record changes per second. Today it receives about 700, nearly three times as many, plus about 100 other events, such as zone setting changes.
However, the number of events was never the real problem.
The problem was the kind of work some of those events needed, and how that work grows with the size of a zone.
Some zones now hold more than 20 million records, and we expect them to keep growing toward hundreds of millions.Each event starts a clock that measures how long it takes us to turn that change into data our authoritative DNS servers can use.
We call that work a build.
A build reads the current database state, converts it into the format used at the edge, and writes any changes to Quicksilver.In 2022, per-record builds were 150 times faster than full builds on average. On a test zone with one million records, per-record builds were 4,250 times faster. We ended that post with the obvious next step: do the same thing for zone settings.Why doesn't the Zone Builder just rebuild the setting that changed?It should, and that is where we ended up.
However, proving when it was safe took most of the work. A record event identifies what changed.
A zone event can come from many Cloudflare products, may not say why the zone changed, and can alter data shared by several zones with the same name. Choosing too much work wastes capacity.
Choosing too little can leave stale or incorrect DNS data at the edge.We built a targeted build planner that chooses the smallest build we can prove is safe, and a new scheduler that combines work for the same zone.
Zone setting changes no longer rebuild zones with tens of millions of records.
Together, they cut peak memory by 85%, and the extra headroom let us safely shorten waits the pipeline has always needed for resilience, halving the time from a DNS change to its data being written to Quicksilver.How DNS changes reach the edgeA DNS zone is the collection of records and settings for a domain. Cloudflare uses a multi-stage pipeline to turn a change to that zone into data available across our global network.A customer changes a DNS record through the DNS Records API or dashboard, or another Cloudflare service changes a zone setting.The change is stored in a database.The database change or another system produces a Kafka event.The DNS Kafka Aggregator combines events from several Kafka topics and sends them to the Zone Builder.The Zone Builder reads the required zone data and converts it into keys understood by our authoritative DNS servers.The Zone Builder compares those keys with Quicksilver, our distributed key-value store. It writes only what changed, and Quicksilver propagates those changes across Cloudflare's network.Diagram of the DNS propagation pipeline: the DNS Records API, zone settings, and internal services write to databases or Kafka topics, the DNS Kafka Aggregator combines them, and the DNS Zone Builder writes to Quicksilver.This is a simplified view.
The Zone Builder does not copy database rows directly to the edge.
It combines records with zone settings, applies DNS serving rules, and organizes the result so our authoritative DNS servers can read it quickly.Why full builds stopped scalingThe original Zone Builder had one useful property: it did not need to know why a zone changed. It rebuilt the desired state from the current database state.Read the current zone and its settings.Read every DNS record in the zone.Convert those records into Quicksilver keys.Compare the new keys with those already stored.Remove stale keys, write changed keys, and update zone metadata.The final write contains only changed keys. However, a full build still reads, encodes, and compares every record, and holds the whole zone in memory while it does so. That is cheap for a small zone. For our largest zones, with tens of millions of records, the zone does not fit in memory at all. I explain how we handle that below.Zone settings made targeting harder.
Universal SSL is one example. Enabling or disabling it updates metadata and can change Certification Authority Authorization (CAA) records at the zone apex, the bare domain, such as example.com.
CAA records tell certificate authorities whether they may issue a certificate for the domain.A full build only needed to know that the zone changed. A targeted build must answer three questions:Which DNS names can change?Can shared metadata or the list of published zones change?What previous state must be removed from Quicksilver?Answering those questions safely meant solving five problems, which the rest of this post covers in order:Events had to say what changed.Each change had to map to the smallest build that is still correct.Builds had to account for data shared by every zone with the same name.The database could change between the event and the build.Combining work could not lose a Kafka message.Making events say what changedWe changed the event producer to include changed zone fields and settings, with previous and new values when available. We then traced each setting through the code that generates DNS output and classified what it could affect.Not every event was that specific. Some database functions only update a timestamp to say, in effect, "something about this zone changed." Sometimes that is harmless bookkeeping. Sometimes it means data changed elsewhere and must be read again.Those cases look identical in the event. Until we can identify the caller and its purpose, a timestamp-only update still causes a full build. The same rule applies to missing details, unknown events, or any change that may affect records anywhere in the zone. The planner never turns uncertainty into less work.Choosing the smallest safe buildQuicksilver output has two parts that matter here.
Record keys contain the DNS answers for one zone and name. A smaller shared metadata key tells our authoritative DNS servers how to use that zone, including where to find special records such as nameservers and wildcards.
Some events change only record keys; others must update both.With that distinction, the planner can choose one of five build types:Resources only: Rebuild the records at specific DNS names, which we call resources, without changing zone metadata.
Ordinary A, AAAA, TXT, and CNAME changes usually take this path.Resources and metadata: Rebuild specific names and shared metadata. Nameserver and wildcard changes use this path.Apex and metadata: Rebuild the zone apex and metadata.
Settings such as Universal SSL can use this path.Purge: Remove a zone that should no longer be published.Full zone: Rebuild every record when smaller work cannot be proven safe.The rule is simple: every key a targeted build writes must be byte-for-byte identical to what a full build would write. If we cannot prove that, we do a full build.Removing old state works as it did for per-record builds in 2022.
Record events include deleted records and the previous version of updated ones.
The Zone Builder then scans Quicksilver for the existing keys under each name it rebuilds, removing any that the new build did not write.Full builds remain necessary when a change can affect records anywhere in the zone or when a zone may need to be published for the first time.There is one deliberate exception. A full build holds every record from the database and every existing Quicksilver key for the zone in memory at once, and our largest zones do not fit. Above a fixed record-count threshold, the Zone Builder never escalates to a full build. It rebuilds the apex, the metadata, and every name identified by the queued events instead. We accept this tradeoff because a full build of those zones cannot finish, and, while it retries, it holds back every other change on its Kafka partition. Many of the cases that would otherwise need a full build involve several zones sharing one name, and zone holds reduce that risk by stopping the same name from being added to another account.
For the rare cases that remain, such as enabling a zone on another network, we can manually rebuild individual names in the zone.When one name means more than one zoneQuicksilver stores each zone's record keys separately.
It also stores one metadata key shared by every Cloudflare zone with that domain name.
This shared key acts like a directory: it tells our authoritative DNS servers which internal zone IDs they may use when answering a query.Why can one name have several zone IDs?
During customer onboarding, offboarding, and other lifecycle changes, multiple Cloudflare zone objects can briefly exist for the same public domain. We use a priority policy to choose a small, bounded set to publish.Consider four internal zones for example.com: Zone 101, Zone 202, Zone 303, and Zone 404. The metadata key currently lists Zones 101, 202, and 303.
Zone 404 exists, but it is not selected.
The numbers identify separate Cloudflare zone objects; customers still see one domain name.A TXT record change at www.example.com in Zone 202 affects only Zone 202's record keys, so the metadata key does not change.
Nameserver and wildcard changes are different because the metadata key records information about them.