Telling Bots from People: A Practical Guide to Visitor Classification

Contents
What the access log records
A 2026 reality check
Verify crawler identity
Behavioral signals
What JavaScript can add
From signals to a decision
Privacy and false positives
A practical workflow
Sources and further reading

Your server does not receive visitors. It receives requests. A person may produce one page view and a dozen asset requests; a crawler may fetch thousands of pages; an AI assistant may fetch one page because a person asked a question. The useful question is not "is this a human?" but "what kind of client is this, and what should I do with it?"

Access logs are the right starting point. They are complete in a way that JavaScript analytics is not: every request that reaches the server is there, including crawlers that never execute a script. But logs are also thin. They record the address, time, path, status, referer and declared user-agent. They do not record intent.

The short version. The user-agent is a claim. The IP is a location. The cookie is a maybe. Reliable classification comes from independent signals that agree: verified network identity, request behavior, session continuity and, where it earns its cost, a small first-party JavaScript beacon.

This guide starts with what an access log can prove, then adds the signals a browser can collect. It keeps three ideas in view: verified crawlers are not people, headless browsers can run JavaScript, and "unknown" is a valid answer.

Examples use documentation address ranges (203.0.113.0/24, 198.51.100.0/24 and 192.0.2.0/24), so no real visitor is exposed.

What the access log records

Start with one line from an Apache combined log:

203.0.113.42 - - [18/Jul/2026:14:22:07 +0300] "GET /catalog/pumps/ HTTP/1.1" 200 18432 "https://example.com/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The address is from a documentation range. The user-agent claims Googlebot. That combination is the whole problem in one line.

One log line, six useful fields
Field Example What it tells you Where it misleads
Client address 203.0.113.42 Network owner (its ASN), country, datacenter or residential range NAT, VPNs and proxies hide the person; a CDN hides the real client
Timestamp 18/Jul/2026:14:22:07 +0300 Rhythm, time zone, working hours Intent; a crawler can run on a human schedule
Request line GET /catalog/pumps/ HTTP/1.1 Method, path and protocol Whether a person wanted the page
Status and bytes 200 18432 Success, error, payload size Engagement; a 200 can be a challenge page
Referer https://example.com/ Navigation context Truth; often absent or spoofed
User-Agent Mozilla/5.0 ... The client's declared identity The actual client; anyone can type any string

Two practical details matter more than they look. First, if the site is behind a CDN or reverse proxy, the logged address is the proxy, not the visitor. Log the forwarded client address and the immediate peer, and trust the forwarded header only when it comes from your own proxy. Second, the standard combined format omits request duration, TLS details and most headers. Add what you will actually use, but do not log cookies, authorization headers or full query strings that may contain personal data.

On Debian or Ubuntu, a dedicated Apache configuration file keeps the format out of the virtual host:

/etc/apache2/conf-available/visitor-log.conf

sudo vi /etc/apache2/conf-available/visitor-log.conf

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D" combined_timing

Then refer to combined_timing in your existing CustomLog directive. Enable the fragment and reload Apache:

sudo a2enconf visitor-log

sudo systemctl reload apache2

%a is the client address and %D is request time in microseconds. With a proxy, enable mod_remoteip and list only the proxies you control. Otherwise an attacker can forge X-Forwarded-For and choose the identity your log records. The Apache documentation covers the available fields in mod_log_config.

On nginx the same caveat applies to $remote_addr: configure real_ip with a trusted proxy list, and consider logging the immediate peer and the forwarded address in separate fields.

What the log cannot see is just as important. It does not know whether JavaScript ran, whether a form was filled by a person, whether the page was read, or whether the visitor came back tomorrow. A request to a page is not a page view. That is why log-based classification and browser-based classification complement each other rather than compete.

The log does record things a browser usually fetches: CSS, JavaScript, images, fonts and the favicon. Their presence or absence is evidence. It is not proof. Keep that distinction through the rest of the guide.

A 2026 reality check

In July 2026, Valentina Melanina published a three-day audit of a real B2B equipment shop: 16,705 requests, 39% of them automated, and every line checked against the source IP. The percentages are a useful calibration, not a universal law.

Claimed identity versus verified identity, three days in July 2026
Claimed token Requests Verified What the audit found
Googlebot 896 268 About 70% came from outside Google's networks
Bytespider 1,073 0 Rented AWS addresses in Singapore, not ByteDance
ChatGPT-User 265 237 71 IPs, 23 sessions; live user-triggered fetches
Bingbot 467 467 Fully verified in this sample

The numbers are not the whole story. Two Google Cloud addresses each claimed fifteen different bot identities; one Cloudflare address claimed eleven. A single server is not fifteen companies; it is a scanner or an AI-access checker rotating masks. The 237 verified ChatGPT-User requests came from 71 IPs and formed 23 sessions: people who asked ChatGPT a question and caused a live fetch. Web analytics did not see them. Separately, 57 requests went to service paths such as /.git, /.ssh, /.env, /secrets.json and /actuator, all under AI-bot names. They were scanners wearing a fashionable mask.

The audit's method is reproducible: group by user-agent, list the IPs, verify each IP against the vendor's current ranges or forward-confirmed reverse DNS, then read the surviving requests as crawler traffic. The full write-up is at melanina.ru.

Two lessons matter. First, a famous name is a mask, not an identity. Second, AI traffic is not one thing. GPTBot is a training crawler. OAI-SearchBot is an index crawler. ChatGPT-User is a fetch triggered by a person's question. The last one is human-initiated but machine-executed: not a human page view, and not ordinary bot noise either. A classification scheme that only has "bot" and "human" will mislabel it.

For a broader baseline, a 2026 study of more than 4.6 million requests found that automated traffic is roughly half of all web requests. The exact share depends on the site, the week and what you count. The method matters more than the headline.

Verify the name, not just the user-agent

Known-name filtering is a useful first pass. It is cheap, it catches honest automation, and it gives you a dictionary to work from. It is not identity verification. Anyone can send any string in User-Agent; MDN's own guide to browser detection recommends feature detection over UA sniffing for the same reason.

Use two independent checks:

  1. Published IP ranges. The vendor publishes JSON or CIDR blocks (address ranges) for its crawlers. Match the log address against the current list.
  2. Forward-confirmed reverse DNS (FCrDNS). Reverse-lookup the address, check that the hostname belongs to the vendor's domain, then forward-lookup the hostname and check that it returns the same address.

Reverse DNS alone is not enough. Whoever controls an address can set its PTR record to a famous name. The forward lookup and the domain check are what make the claim testable. For Google, the hostname must end in googlebot.com, google.com or googleusercontent.com, depending on the crawler type.

host 66.249.66.1

1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com

crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The first lookup gives a name; the second proves the name points back. A fake PTR can pass the first test but not the second unless the attacker also controls forward DNS for a domain they do not own, which is why the domain check matters.

What vendors published as of October 2026
Operator Verification method What to check
Google Five IP files by crawler type + FCrDNS googlebot.com for common crawlers; googleusercontent.com or google-proxy names for user-triggered fetchers
Microsoft Bing IP list + Verify Bingbot tool + FCrDNS Reverse names end in search.msn.com; refresh the list daily
OpenAI One IP file per agent GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot; no published rDNS method
Anthropic One combined IP file The list does not distinguish ClaudeBot, Claude-User and Claude-SearchBot
Perplexity Two IP files PerplexityBot and Perplexity-User
Apple IP list + FCrDNS Reverse names end in *.applebot.apple.com
Common Crawl IP list + FCrDNS crawl.commoncrawl.org; IPv6 reverse DNS is not supported
Amazon Three IP pages Amazonbot, Amzn-SearchBot, Amzn-User
Meta No published verification method As of October 2026, a meta-externalagent claim cannot be verified from Meta's page
Yandex FCrDNS Reverse names end in spider.yandex.com; use any current IP list the vendor publishes

The lists change. Fetch them from the vendor on the day you use them; do not copy CIDR blocks from an article, including this one. Google's current instructions and files are linked from Verify requests from Google; OpenAI's per-agent files and semantics are in its crawler documentation; Anthropic publishes one combined list; Perplexity documents both files in its bot guide.

Two names never appear in a log: Google-Extended and Applebot-Extended are robots.txt control tokens, not crawlers. If a dashboard claims visits from them, the dashboard is wrong.

The ai.robots.txt project is a useful dictionary of AI crawler names and a ready-made robots.txt policy. Use it to find tokens, then verify the IP; the list itself is not proof.

Cloudflare's Verified Bots program and the emerging Web Bot Auth signatures are a third route: an edge provider or a cryptographic signature attests to the client. They are useful when you are already behind that provider, but they are not universal.

What the declared name actually means
Declared token What it is Count as a visitor?
Googlebot, Bingbot, YandexBot Search engine crawlers No
GPTBot, ClaudeBot, CCBot, Bytespider Training crawlers No
OAI-SearchBot, PerplexityBot AI search index crawlers No
ChatGPT-User, Perplexity-User, Claude-User User-triggered fetch Human-initiated, but not a page view
curl, python-requests, Go-http-client, Scrapy, axios, HeadlessChrome Declared automation Likely no, but verify
Uptime monitors, link previews, SEO crawlers Automated services No

A verified crawler is still a bot. Being legitimate does not make it a person. Keep verified_crawler separate from human in your schema, and keep unverified separate from fake. A failed check means "I could not confirm this", not "I proved this is fraud".

Behavior that leaves a trace

Identity checks catch bots that declare themselves. Behavior catches bots that lie. The log gives you a sequence of requests; the sequence is often more informative than any single line.

Signals you can compute from a standard access log
Signal What it looks like Strength
Asset ratio Bot: HTML only, or the same fixed assets. Human: CSS, JS, images and fonts, unevenly. Medium
Session rhythm Bot: regular gaps, low variation. Human: bursts, pauses and backtracking. Medium
Navigation shape Bot: sitemap order, sequential IDs, deep crawls. Human: search or social entry, reading, backtracking. Medium
HTTP methods Bot: HEAD, OPTIONS or TRACE; POST to HTML pages. Human: GET and POST on real forms. High for scanners
Error probes Bot: /.env, /.git, wp-login.php, /actuator. Human: an occasional broken link. High for scanners
Multi-persona IP Bot: more than 3–5 user-agents from one IP in a short window. Human: usually one or two. High
Cookie continuity Bot: cookies absent or random on every request. Human: a stable session cookie across pages. Medium
Referer Bot: absent, self-referencing or fake. Human: search, social or internal navigation. Low to medium
Time pattern Bot: flat around the clock. Human: local working hours. Low alone
Network type Bot: cloud or hosting ASN. Human: residential or mobile, with VPN exceptions. Medium
Favicon and robots.txt Bot: often skips the favicon, often requests robots.txt. Human: usually the reverse. Low

Asset ratio and rendering

A browser fetches HTML, then CSS, JavaScript, fonts and images. A simple HTTP client often fetches only HTML. A high HTML-to-asset ratio is therefore a weak bot signal. But Googlebot may fetch assets when it renders a page, and a human may block them. The 2026 Shy Guys study of 4.6 million requests found that bots often skip the favicon; a favicon request is weak evidence of a browser, not proof of a person.

Compute the ratio per session, not per IP, when you can: static_requests / html_requests. A session with 40 HTML requests and no CSS, JavaScript or images is not a reader. A session with one HTML request and no assets might be a prefetch or a text browser.

Rhythm

Bots are often regular; humans are bursty. For each session, take the gaps between requests and compute the coefficient of variation: CV = stdev(gaps) / mean(gaps). A CV near zero over 20 or more requests is machine-like. A CV above 1 is common for humans. But a polite crawler with random delays can look human, and an auto-refresh dashboard can look regular. Use rhythm with volume and path shape, not alone.

"Requests per minute" is a poor standalone feature: a crawler can be slow, and a human can open thirty tabs. The pattern of gaps is more informative than the average.

Navigation shape

Scrapers enumerate. They follow sitemap order, sequential product IDs, category pages and next-page links. Humans arrive from search or social, read, backtrack, open a new tab and leave. A session that starts at a deep URL and visits two hundred pages in a perfect sequence is not a reader.

Error probes are the sharpest version of this signal. /.env, /.git, /wp-login.php and /actuator are not content. A cluster of them is a scanner signature even when the user-agent says "Chrome".

Two quick checks before you write a parser. In the combined format, the first field is the address and the ninth is the status:

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -10

Example output:

1842 66.249.66.1 913 203.0.113.42 407 198.51.100.7

Counts are not classification. A high count can be Googlebot doing its job or a scraper pretending to be Googlebot.

awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -nr | head -10

Example output:

57 /.env 31 /.git/config 18 /wp-login.php 12 /actuator/env

These paths are scanner signatures. They are high-confidence even when the rest of the request looks like a browser.

Multi-persona IPs

Group by IP and count distinct user-agent strings in a short window. More than three is suspicious; the 2026 audit found two cloud IPs claiming fifteen identities each. A shared corporate proxy, a CDN or a browser-update rollout can produce several legitimate user-agents, so require volume and a short window. This is one of the cheapest high-value features in the log.

Cookie and session continuity

A real browser usually keeps a first-party session cookie across pages. A scraper often discards cookies or gets a new session on every request. But privacy browsers, cookie banners and stateless APIs break this. Check whether the same cookie returns across a sequence, not whether a cookie exists at all.

To compute per-IP features instead of eyeballing counts, save this script as features.py. It reads a combined log, ignores malformed lines, and prints the top addresses by request count with static ratio, error ratio and distinct user-agents:

features.py

vi features.py

#!/usr/bin/env python3 """Print per-IP request, static, error, and user-agent counts.""" from __future__ import annotations import re import sys from collections import Counter, defaultdict LINE = re.compile( r'^(?P<ip>\S+) \S+ \S+ \[[^\]]+\] ' r'"(?P<method>\S+) (?P<path>\S+) [^"]+" ' r'(?P<status>\d{3}) \S+ "[^"]*" "(?P<agent>[^"]*)"' ) STATIC = (".css", ".js", ".png", ".jpg", ".jpeg", ".gif", ".svg", ".ico", ".woff2") def main(path: str) -> None: """Read a combined log and print the top IPs by request count.""" requests: Counter[str] = Counter() static: Counter[str] = Counter() errors: Counter[str] = Counter() agents: defaultdict[str, set[str]] = defaultdict(set) with open(path, encoding="utf-8", errors="replace") as log: for line in log: match = LINE.match(line) if not match: continue row = match.groupdict() ip = row["ip"] requests[ip] += 1 agents[ip].add(row["agent"]) if row["path"].lower().endswith(STATIC): static[ip] += 1 if row["status"].startswith(("4", "5")): errors[ip] += 1 for ip, count in requests.most_common(20): print( f"{ip:16} req={count:5d} " f"static={static[ip] / count:.2f} " f"errors={errors[ip] / count:.2f} " f"uas={len(agents[ip])}" ) if __name__ == "__main__": main(sys.argv[1] if len(sys.argv) > 1 else "access.log")

Run it against the log:

python3 features.py access.log

Illustrative output:

203.0.113.42 req= 913 static=0.04 errors=0.01 uas=1 66.249.66.1 req= 1842 static=0.02 errors=0.00 uas=1 198.51.100.7 req= 407 static=0.00 errors=0.38 uas=15

One address is a high-volume crawler with almost no assets; another has fifteen user-agents and a 38% error rate. The script does not decide. It gives you features for the scoring step.

What JavaScript can add

Logs see the request; JavaScript sees the client. That is the trade. JavaScript can observe the environment and interaction, but only for clients that execute it. Most crawlers do not; a headless browser does. A missing beacon is strong evidence of a non-JavaScript client. A present beacon is weak evidence of a human.

Three layers are worth collecting, in this order:

  1. Existence and timing. Did the page run, when, for how long, and was it visible?
  2. Environment and consistency. Does the browser's self-description agree with the request headers and with itself?
  3. Interaction. Did a pointer, scroll, key or focus event happen?

Start with a small first-party beacon. Do not start with canvas fingerprinting: it is harder to justify, easier to break, and less useful than knowing whether the page executed at all.

A minimal beacon

Server-render a random visit ID into the page. It is a join key, not a secret:

<meta name="visit-id" content="a7f3c9e2">

Then send three small events to a first-party endpoint: load, alive after five seconds, and hide when the page becomes hidden. This snippet deliberately records whether an interaction happened, not where the pointer went or what was typed:

<script> (() => { const visitId = document.querySelector('meta[name="visit-id"]')?.content ?? ""; const endpoint = "/api/visit"; const started = performance.now(); const storageWorks = () => { try { localStorage.setItem("__probe", "1"); localStorage.removeItem("__probe"); return true; } catch { return false; } }; const base = () => ({ visit_id: visitId, path: location.pathname, referrer: document.referrer, screen: `${screen.width}x${screen.height}`, viewport: `${innerWidth}x${innerHeight}`, language: navigator.language, languages: navigator.languages?.join(",") ?? "", timezone: Intl.DateTimeFormat().resolvedOptions().timeZone, webdriver: navigator.webdriver === true, cores: navigator.hardwareConcurrency ?? 0, memory: navigator.deviceMemory ?? 0, touch: navigator.maxTouchPoints ?? 0, cookies: navigator.cookieEnabled, storage: storageWorks(), visibility: document.visibilityState, }); const send = (event, extra = {}) => { const body = JSON.stringify({ ...base(), event, ...extra }); if (navigator.sendBeacon) { navigator.sendBeacon(endpoint, new Blob([body], { type: "application/json" })); } else { fetch(endpoint, { method: "POST", body, keepalive: true, headers: { "Content-Type": "application/json" }, }); } }; const onLoad = () => send("load", { load_ms: Math.round(performance.now()) }); if (document.readyState === "complete") onLoad(); else addEventListener("load", onLoad, { once: true }); let interacted = false; for (const name of ["pointermove", "scroll", "keydown"]) { addEventListener(name, () => { interacted = true; }, { once: true, passive: true }); } setTimeout(() => send("alive", { interacted, elapsed_ms: Math.round(performance.now() - started), }), 5000); addEventListener("visibilitychange", () => { if (document.visibilityState === "hidden") { send("hide", { interacted, elapsed_ms: Math.round(performance.now() - started), }); } }); })(); </script>

What the fields suggest

JavaScript signals and their failure modes
Signal What it suggests How it fails
No beacon after 10 seconds No JavaScript, a blocked script, or a simple HTTP client The user has JavaScript disabled; a bot runs JavaScript
webdriver: true An automation framework left the flag on The flag can be patched; legitimate automation also sets it
Empty languages or plugins A headless or stripped environment Privacy browsers reduce these; the values can be spoofed
touch: 0 with a mobile user-agent An inconsistent claim Desktop mode, tablets and hybrid devices exist
Timezone differs from IP country A proxy or a mismatched environment VPNs, travel and expats
No interaction after 5 seconds A scripted load The user opened the page in a background tab; a bot can synthesize events
storage: false Blocked storage or a restricted environment Privacy settings and iframe restrictions
Fast load and immediate hide A prefetch or a bot Real users bounce too
Canvas or WebGL hash Device or automation identity Spoofable, noisy and privacy-heavy
isTrusted: false A synthetic event CDP can produce trusted events; it is not a wall

navigator.webdriver is a standard property that indicates a user agent is controlled by automation. It is one useful example, not a verdict: MDN documents it as a signal, and modern automation can suppress it. The same applies to missing plugins, empty languages and "headless-looking" screens. F5's 2015 PhantomJS article is a catalog of environment gaps; the specific tells have changed, but the principle has not. Treat each one as a weak feature.

Fingerprinting goes further: canvas, WebGL, audio, fonts, TLS handshake, header order. It can be powerful, but it is probabilistic, brittle and privacy-invasive. The W3C's fingerprinting guidance treats it as a privacy risk; the ICO's PECR guidance treats storage and device access as something that generally needs consent. For ordinary analytics, don't. For fraud or security, document the purpose, minimize the data, retain it briefly and get legal review. A fingerprint is not a person; it is a similarity score.

Transport fingerprints (JA3/JA4) are a strong mismatch signal: a request that claims Chrome but negotiates TLS like a Python script is not Chrome. They require access at the TLS terminator or a CDN that exposes the fingerprint; they are not in a standard access log. Add them only if you control that layer.

Challenges are a step, not a foundation. A JavaScript proof-of-work or a CAPTCHA raises the cost, but CAPTCHAs are bypassable and hurt accessibility. Use them for ambiguous sessions, after cheaper checks, and always keep an allow path for verified crawlers you want. Azion's overview of bot detection techniques describes the same order: reputation, known-good verification, fingerprint, behavior, then a challenge or block.

This is an arms race, and it is asymmetric. Browserless's 2026 automation guide openly describes stealth routes that remove navigator.webdriver and inject entropy into canvas, WebGL and audio, residential proxies that repair IP reputation, and CAPTCHA solving. If your site is a high-value target, assume a determined client can pass every client-side check. The goal is not an unbeatable wall; it is to make the cheap attack fail and to route the expensive one to a challenge or a human review. See Browserless on bot detection.

Modern AI agents complicate the browser layer further. Some drive real Chrome, use residential connections and suppress navigator.webdriver; they can pass every single check above. The cside 2026 guide argues that behavior is the most reliable tell: deterministic mouse paths, DOM burst reads, Chrome DevTools Protocol (CDP) artifacts, no scroll before a query. It also argues for intent classification rather than a binary block, because a consumer agent may be trying to buy something. That is the right framing here too: score the session, label the intent, and choose an action per route. See cside's AI-agent guide.

From signals to a decision

Scoring combines weak evidence. The weights below are an example, not a standard. Calibrate them on your own labeled sample; the structure matters more than the numbers.

An illustrative scoring model
Signal Example weight Why
Verified crawler (IP list or FCrDNS) Override to verified_crawler Hard positive identity; do not add points to a known bot
Known bot name, verification failed +40 Impersonation or an unverifiable claim
Service-path probe (/.env, /.git, /actuator) Override to scanner The intent is not content
More than 3 user-agents from one IP in 10 minutes +25 Multi-persona scanner
Datacenter or hosting ASN +15 Not conclusive: VPNs, cloud desktops and CDNs exist
HTML-only session, 10+ requests +15 No assets fetched
Inter-arrival CV < 0.3 over 20+ requests +15 Machine rhythm
4xx rate > 30% on 10+ requests +15 Probing behavior
No cookies across pages +10 Stateless client
JS beacon with an interaction event and 5s dwell -30 Weak human evidence
Consistent environment, cookies and assets -20 Coherent browser session

Categories, not a binary

The order matters. Overrides first, then additive signals, then the residual categories. A verified crawler should never be "saved" by a JavaScript beacon, and a scanner should not be rescued by a browser user-agent. Conversely, a human with JavaScript disabled should not be classified as a bot just because the beacon is missing.

Different decisions need different thresholds
Use case Action What not to do
Analytics Exclude verified crawlers, scanners and likely automated traffic; keep uncertain; report human-initiated agents separately. Do not present "visitors" as people when the number includes crawlers.
Rate limiting Apply lower limits to automated categories; give verified search and AI search crawlers the limits you intend. Do not rate-limit a verified crawler into 429s you did not intend.
Access control Challenge ambiguous sessions; block only high-confidence scanners or malicious behavior. Do not block uncertain sessions or whole cloud ASNs.
AI visibility Allow the AI crawlers you want; check that they receive 200, not 403 or 429. Do not block every AI token by reflex; user-triggered agents may bring customers.

Evaluate the classifier, not just the score. Label 200 to 500 sessions by hand, stratified across score bands. Compute precision and recall per category, not one overall accuracy number. Track false positives: accessibility tools, corporate proxies, privacy browsers, text browsers, link previews and uptime monitors are the usual victims. Recalibrate when your traffic or the bots change.

Keep an uncertain bucket. It is not a failure; it is honesty. A classifier that never says "unknown" is overclaiming, and a hard block on an uncertain session is how false positives become support tickets.

Privacy, false positives, and policy

Logs and beacons process personal data. An IP address can be personal data; a cookie or device fingerprint can be an online identifier. In the EU and UK, PECR/ePrivacy requires consent for storing or accessing information on a user's device, with limited exceptions, and GDPR applies to the processing. The ICO's cookies and similar technologies guidance is a starting point, not legal advice.

Minimize what you collect:

False positives are not edge cases:

Design for these. A hard block based on "no JavaScript" will hit some humans. A block based on a datacenter ASN will hit some customers. Challenge first, block only high-confidence bad behavior, and give people a way to report a false positive.

Policy is a separate decision from detection:

A practical workflow

Start with one week of logs and one question: how many requests are not people? A workable sequence:

  1. Fix the log format. Make sure the real client IP survives the CDN or proxy. Include user-agent, referer, status, bytes and time; add duration if you can.
  2. Build a verified-bot dictionary. For each token: vendor, purpose, IP-list URL, rDNS suffix and last-checked date. Refresh the lists daily or weekly.
  3. Sessionize. Use a 30-minute inactivity window. Prefer a first-party cookie; fall back to IP plus user-agent plus time when cookies are absent.
  4. Enrich. Add ASN, network type, rDNS, coarse geo, user-agent family and path class.
  5. Compute features. Asset ratio, error rate, distinct user-agents, rhythm, depth, entry path and cookie continuity.
  6. Label a sample. Start with verified crawlers and obvious scanners; then label ambiguous sessions without looking at the score.
  7. Score and set thresholds. Separate analytics from access control. Keep an uncertain bucket and an appeal path.
  8. Add the beacon. Start on a sample of pages. Join by visit ID or by IP, user-agent and time. Measure what it adds before rolling it out everywhere.
  9. Review and recalibrate. Weekly for the first month, then monthly or quarterly. Track false positives and blocked crawlers.

A weekly report template that keeps the categories honest:

category requests share verified search crawler ... ... verified AI crawler ... ... declared automation ... ... scanner ... ... likely automated ... ... uncertain ... ... human-initiated agent ... ... likely human ... ...

Fill it from the same classifier you use for decisions. Do not maintain a second spreadsheet with a different definition of "human".

If you do only one thing, verify the IP before you trust the name. If you do two, add the service-path probe check. Both are cheap, and together they remove a large share of the noise.

Sources and further reading

The articles below informed this guide. Vendor lists and verification methods change; check the current documentation before relying on a range or a token.

Logs, behavior and the 2026 picture

JavaScript and fingerprinting

Verification and implementation

The Stack Overflow, Melanina, vc.ru, WorkOS, Azion, Web Scraper and cside links are the sources supplied for this article; the rest are the primary vendor or standards documents used to check the claims.

Article author: Arthur Isaev

Related articles
Web
Bots vs people
SEO
Choose Hosting for Website in Russia
JavaScript
Python

Search this site

Channel @aofeed Chat @aofeedchat

Contacts and cooperation:
I recommend our hosting beget.com
Write to info@aredel.com if you:
1. Want to write an article for our site or translate an article into your native language.
2. Want to place thematically relevant ads on the site.
3. Ads on my site pass maximum censorship. If you see an ad block unsuitable for school-age children, shocking or misleading - please contact us by e-mail
4. Found a mistake, inaccuracy, bug, etc. on the site. ... .......
5. Articles can be shared on social media by clicking a network icon: