Bot Traffic at the Edge: Filtering Scrapers Without Blocking Real Buyers
Filter bots at the edge using layered signals: known bad signatures, rate limits on sensitive paths like login and checkout, and challenges only when behavior looks automated. Keep the default for ordinary browsing permissive, watch false positives daily, and never let a rule block payment or search crawlers you actually need.
Scrapers, credential stuffers, and inventory hoarders all arrive as traffic that looks busy and buys nothing. Edge filtering separates them from shoppers if you tune it by behavior instead of blocking blindly.
Filter bots at the edge using layered signals: known bad signatures, rate limits on sensitive paths like login and checkout, and challenges only when behavior looks automated. Keep the default for ordinary browsing permissive, watch false positives daily, and never let a rule block payment or search crawlers you actually need. This guide covers where to put the pressure, what to measure, and how to avoid the classic mistake of blocking revenue along with abuse.
Put strict rules on sensitive paths, loose rules elsewhere
A scraper reading your public catalog is a nuisance. A botnet testing stolen passwords on your login page is an emergency. Treat paths differently: strict rate limits and challenges on login, account creation, gift card balance checks, and checkout endpoints, while product and content pages stay easy to reach. Attackers concentrate where accounts and money move, so that is where friction belongs.
Rate limits should count failures as well as requests. A human mistypes a password twice. A credential stuffing run fails hundreds of times per minute from rotating addresses. Limits keyed on failed attempts per account, per address range, and per device fingerprint catch the pattern without punishing a shopper who simply forgot a password.
Use behavior signals before hard blocks
Instant blocks feel decisive and quietly cost sales when they misfire on corporate networks, VPNs, and shared mobile carrier addresses. A challenge first approach works better: suspicious sessions get a lightweight check, proven automation gets blocked, and clean sessions sail through. Reserve hard blocks for signatures you have verified, such as known scraper user agents hitting at machine speed.
Keep an allowlist for the bots you depend on: search engine crawlers, your uptime monitor, payment webhooks, and partner feeds. Every filtering rollout should start by confirming those still work, because blocking a search crawler or a payment callback hurts more in a day than a month of scraping. Verify with server logs, not assumptions.
Measure false positives like a revenue metric
Every bot program needs a false positive budget and someone who reads the challenge and block logs weekly. Sort blocked sessions by path and user agent, sample them, and ask a blunt question: could any of these have been a buyer? Rising challenge rates on checkout pages, or blocks clustered on one carrier or country, are the usual early warnings.
Tie bot metrics to business metrics before and after every rule change: conversion rate, checkout completion, and search visibility. A rule that cuts bot traffic in half and conversion by two percent is a bad trade wearing a security costume. The goal was never zero bots. The goal is cheap abuse and uninterrupted shopping.
Related reading
Readers working through this topic often find these earlier guides useful as well: How Edge Personalization Works Without Slowing the Storefront Down, Edge Observability: Seeing Storefront Errors Where They Actually Happen, and Image and Font Delivery at the Edge: The Quiet Speed Win.
Frequently asked questions
How much of a typical store's traffic is bots?
Estimates vary widely by store and season, but automated traffic is commonly a third or more of requests. The share that matters is what hits login, checkout, and inventory endpoints, not the raw total.
Will bot filtering hurt my search rankings?
Only if you block or challenge verified search crawlers. Allowlist them by reverse DNS verification rather than user agent alone, and check crawl stats after any rule change. Done carefully, filtering has no ranking cost.
What is the first rule I should add?
Rate limit failed logins and account creation, and add a challenge to gift card and discount code guessing endpoints. Those paths carry the highest abuse value and the lowest legitimate traffic, so friction there is nearly free.
How do I know a rule blocked a real customer?
Watch support tickets mentioning access problems, checkout completion dips after rule changes, and blocked session logs sampled by hand. If you cannot inspect what you blocked, you are flying blind and should fix logging first.