Building a web scraper is easy. requests.get(url) and BeautifulSoup(html, 'html.parser') will get you started in 5 minutes. But building a web scraper that runs reliably every day without getting hit by Cloudflare blocks, CAPTCHAs, or IP bans? That requires an arsenal of anti-bot evasion techniques.
In this post, we cover the exact techniques used to scrape high-security e-commerce and social media sites without triggering alarm bells.
1. Fixing Your Headers
By default, the Python requests library sends a User-Agent that literally says python-requests/2.28.1. This is an instant block on any modern website. You must mimic a real browser.
But changing the User-Agent isn't enough. Anti-bot systems look at the order and presence of other headers (like Accept-Language, Sec-Fetch-Dest, etc).
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1"
}
response = requests.get("https://example.com", headers=headers)
2. Session Management & Cookies
If you make 10 requests to a site and don't return the cookies they gave you on the first request, the site knows you aren't a real browser. Use requests.Session() to automatically handle cookies across multiple requests.
session = requests.Session()
session.headers.update(headers)
# First request gets the cookies
session.get("https://example.com/login")
# Second request sends those cookies back naturally
session.post("https://example.com/login", data={"user": "admin", "pass": "123"})
3. Humanizing Request Delays
A loop that makes 100 requests in 2 seconds is obviously a bot. Introduce random delays between requests. Use a normal distribution rather than a flat random to mimic human pacing.
import time
import random
def human_delay():
# Base delay of 2 seconds, plus a random float between 0 and 2.5
delay = 2.0 + random.uniform(0.0, 2.5)
time.sleep(delay)
4. Rotating Residential Proxies
If you need to make thousands of requests, pacing won't save you from an IP rate-limit. You need proxies. However, Datacenter proxies (like AWS or DigitalOcean IPs) are often blacklisted by default. You need Residential Proxies—IP addresses assigned by ISPs to real homes.
Services like BrightData or Smartproxy provide rotating proxy endpoints.
proxies = {
"http": "http://username:password@us.smartproxy.com:10000",
"https": "http://username:password@us.smartproxy.com:10000"
}
response = requests.get("https://example.com", headers=headers, proxies=proxies)
5. Beating Cloudflare with Undetected Chromedriver
Some sites use advanced JavaScript challenges (like Cloudflare Turnstile). Pure HTTP requests will fail because they can't execute the JS. You must use a headless browser.
Standard Selenium is easily detected by its WebDriver flags. Instead, use undetected-chromedriver.
import undetected_chromedriver as uc
options = uc.ChromeOptions()
options.add_argument("--headless=new")
driver = uc.Chrome(options=options)
driver.get("https://cloudflare-protected-site.com")
print(driver.page_source)
driver.quit()
Conclusion
Web scraping is an arms race. By using proper headers, session management, human-like delays, and avoiding standard WebDriver flags, you can build resilient scrapers that gather the data you need without waking up the anti-bot sentinels.
— Ankit Kumar