Bot Detection Without JavaScript: What My Blog Measured

On my blog, network and request-header rules moved 277 of 372 browser-User-Agent requests out of the Browsers category: 74.5%. That gives me a much more useful account of the traffic arriving at my Cloudflare Worker. It has not established how many people read the site. Over the same two complete UTC days, the remaining 95 Browser HTML observations still differ from 14 Cloudflare Web Analytics page loads. The useful result is knowing which requests the rules separate, why they separate them, and where the evidence stops. Network evidence catches requests that pass the header checks. In that window, 60 cloud-classified requests carried the navigation headers the browser rule requires. A reason for each classification makes the counter explainable. It also exposed mistakes: our HTML-acceptance check mishandled valid headers, and new rows were missing their network-provenance marker. Both have since been repaired. Client identity and readership need different evidence. Nine stored signature verifications identify signers, including crawlers and deliberate tests. They do not count people asking an assistant to read. The remaining disagreement is a measured problem. Neither a smaller Browser count nor agreement with a script counter establishes audience accuracy. Comparing edge page views with a script counter Comparing two counters exposed the problem with my first-party analytics: I had treated browser User-Agents as evidence of readers. The counters measure different events, so their disagreement is a starting point for investigation. It cannot, by itself, tell me which requests were automation. The initial alarm came from this comparison, saved on September 3: Source Page events / loads Client or visit metric D1 browser-UA class, seven UTC days ending September 2 1,209 578 daily client identifiers Cloudflare Web Analytics, its rolling seven-day dashboard window 113 52 visits The 578-versus-52 difference looked like eleven times as many readers. But a daily client identifier is not a visit, the time windows were not identical, and the script dashboard included /stats. Even the more comparable 1,209-versus-113 page totals needed those qualifications. The original query record preserves the comparison as it was made. There was stronger evidence inside the requests. On September 2, 100 of 113 daily clients loaded one page; 156 of 164 browser-UA page observations carried no referrer. Those facts alone would not establish automation. One client classified as mobile, however, fetched 31 distinct pages in the same timestamp second. My note that night was: i am seeing daily clients as 113 for today, and it seems unbelievable to me, like which articles are they reading, where are they coming from and so on... i just published a new article and its not even coming up in the Top pages by views section... like whats going on... Like sometimes when i publish the post i wanna see for this particular post how many readers have arrived and through which sources, its impossible to figure out. But still the most bizarre is the numbers, who is all reading these articles, it seems insane, which i appreciate but I don't want to gaslight ourselves, like something is not adding up A useful comparison needs four decisions made before calculating the ratio: Choose the same host and explicit start-inclusive, end-exclusive UTC window. Compare page events with page loads. Report daily identifiers and visits separately. Record which routes, response types, bots, owners, and test requests each system excludes. Keep sampling information with the result. Inspect the discrepant requests and test the possible collection differences. Cloudflare documents script blockers and browser or network loss as reasons its beacon can miss page loads. Browser caching and differing eligibility also need checking against the edge counter. A script can run in an automated browser. There is no universal ratio that separates these causes. Cloudflare Web Analytics FAQ, checked September 6, 2026. The comparison reveals questions we can investigate. The earlier version of this article's under-two validation threshold was unsupported, and I have removed it. The request rules in the Cloudflare Worker The Worker classifies recorded request characteristics. It can apply those rules deterministically without establishing who controlled the client. This section is for someone implementing the classifier; the measured results below can be read without the implementation detail. The edge counter schedules a D1 observation after an eligible successful page GET. It excludes prefetches, /stats, API routes, and non-page responses. It includes HTML and negotiated Markdown page responses; direct .md requests are outside this counter. That collection boundary comes from the eligibility code. Four sources of evidence feed the classification: Network metadata. Cloudflare supplies the client's autonomous system number (ASN). This describes the network reaching the edge. A client cannot change it by editing an HTTP header, but can reach the site through another network or proxy. It does not establish the operator's purpose. Fetch Metadata. Sec-Fetch-Mode: navigate and Sec-Fetch-Dest: document describe a page navigation. Sec-Fetch-Site can be none, same-origin, same-site, or cross-site; the classifier records it but does not require one value. Header definitions. Accept and Accept-Language. The rule checks whether HTML is acceptable and whether a language header is present. These are request characteristics, not evidence of attention. User-Agent. Named crawler and assistant rules match the client's declaration. The same string also supplies the browser-version claim used below. A client can imitate it. D1 stores selected headers, derived flags, names, and classification reasons. It does not retain the full User-Agent. Existing Cloudflare operational logs contain additional request detail. Storing a derived flag is useful, but does not preserve enough information to replay every future parser change. Rule one: classify known hosting networks before browser shape. After signature and named-client rules, a browser-UA request on the curated hosting list becomes cloud-browser, even if it passes the navigation-header check. In the original 72-hour log sample ending September 3, 430 of 844 successful page GETs were navigation-shaped traffic on hosting networks. The largest cluster was 374 requests attributed to one Google Cloud client claiming Chrome Mobile 114. Header presence could not separate that cluster. Original measurement and coverage. A cloud browser may be carrying out useful work for a person. I want that access recorded and visible. The network label describes how it reached the site, and moving it out of Browsers does not remove its observation. The curated network list excludes several shared service and consumer-VPN networks. Broad hosting lists can include networks that carry legitimate browsing. Curating the list accepts the opposite cost: some automation will arrive through networks outside it. Rule two: check missing Fetch Metadata against the claimed browser version. The implementation checks Chromium 76+, Firefox 90+, and Safari/iOS 16.4+. If a request claims one of those engines but lacks Sec-Fetch-Mode, it becomes http-client with reason no-fetch-metadata. Older or unreadable claims receive legacy-browser. A request with Fetch Metadata that fails the navigation combination receives not-navigation-shaped. Classifier code. Those version thresholds follow browser support data, checked September 6. They establish an expected browser capability; they do not authenticate each request or establish a zero false-positive rate for every embedded client. We do not require Sec-Fetch-User: the Safari compatibility investigation found support for other Fetch Metadata headers without it. The follow-up audit also found a defect in our own Accept check. The original extractRequestMetadata() returned acceptsHtml: 1 for Accept: text/html;q=0 and 0 for Accept: text/*. The first explicitly excludes HTML, while the second admits it. The substring check ignored HTTP quality and media-range rules. RFC 9110, Accept semantics; recorded reproduction. The September 6 repair now evaluates quality, specificity, and matching representation parameters. In twelve selected local ingestion cases, seven incorrect acceptance results became zero; this is a regression check, not a production error-rate estimate. Executed experiment and artifacts. Stored booleans cannot tell us how many historical requests the defect affected. What open-source classifiers contributed The useful prior art supplied specific pieces of this design: declared-bot detection, network classification, and retained reasons. Our reading did not establish that the combination is novel, or that every other counter misses the same traffic. Project What the inspected source contributes Boundary `isbot` User-Agent matching for clients that identify themselves Its stated scope excludes programs disguising themselves as users Plausible UA checks, explicit ingestion outcomes, and consumption of an upstream IP classification Reading that code does not reveal or reproduce the upstream classifier GoatCounter Server-side bot classification and retention of bot reasons Its categories and collection rules differ from ours Anubis A combination of browser headers that adjusts a classification weight The rule file warns that automated scrapers can bypass it These sources were checked on September 6; revisions and source boundaries are in the pinned review. isbot remains useful for the declared clients it aims to recognize. Anubis makes access-control decisions, whereas this counter aims to retain and explain access. The part I adopted from GoatCounter was a reason attached to each classification. A lower total tells me little; a table of requests and the rules that classified them tells me what changed. Agent names, signatures, and request purpose