How to Fix Ahrefs Site Audit "Broken Page" 404 Errors from Cloudflare
Updated with a verified-bot fix, current Ahrefs IP sources, and 403/522 troubleshooting.
Running an Ahrefs Site Audit only to find it flooded with “Page has links to broken page” errors? If these 404 errors all point to URLs containing /cdn-cgi/l/email-protection, you’ve found the culprit: a conflict with Cloudflare’s Email Address Obfuscation feature.
The crawler flags these dynamically generated links as internal broken pages, skewing your audit results and making it difficult to identify genuine issues on your site.
This guide shows how to configure Cloudflare so the Ahrefs Site Audit runs without these false errors, which IP addresses Ahrefs crawls from, and what to do when Cloudflare returns 403 or 522 errors to Ahrefs.
What is /cdn-cgi/l/email-protection?
/cdn-cgi/l/email-protection is the link Cloudflare puts in place of email addresses when Email Address Obfuscation is on. Cloudflare rewrites every mailto: link and plain-text address on the page into /cdn-cgi/l/email-protection#<encoded string>, then decodes it back into the real address in the browser with a small script (email-decode.min.js). Visitors never notice. Crawlers that don’t run that script, including the Ahrefs Site Audit, follow the raw link and get a 404.
Every page with an email address on it shows up in Ahrefs as linking to a broken page, even though nothing is broken for real visitors.
How to find these 404s in Ahrefs Site Audit
- Open your Site Audit project and go to Overview.
- Open Internal pages > 4XX page to see every URL that returned a 4xx code. The No. of inlinks column shows which pages link to each one.
- Open Outgoing links > Page has links to broken page to see the pages that contain the broken links.
- Filter the URL by
/cdn-cgi/to separate the Cloudflare false positives from real broken links.
If most of the 404s are /cdn-cgi/l/email-protection URLs, use one of the fixes below and re-crawl to confirm they’re gone.
Why editing robots.txt won’t fix your Site Audit?
You might be tempted to disallow the /cdn-cgi/ directory in your robots.txt file. That won’t solve the problem. The crawler still discovers the links on your pages and, because it can’t crawl them, keeps reporting them. The fix has to happen in Cloudflare.
Method 1: Turn off obfuscation for verified Ahrefs bots (recommended)
Both AhrefsBot and AhrefsSiteAudit are verified bots in Cloudflare. Cloudflare checks a verified bot’s identity itself, so a rule based on it can’t be faked with a copied user-agent and doesn’t need an IP list you have to keep up to date.
-
In your Cloudflare dashboard, select your domain and go to Rules > Create rule > Configuration Rule.
-
Name it, e.g.
Ahrefs - disable email obfuscation. -
Under “When incoming requests match…”, select Edit expression and paste:
(cf.client.bot and http.user_agent contains "Ahrefs")cf.client.botis the Known Bots field. It is only true for bots Cloudflare has verified, so a bad bot that fakes the Ahrefs user-agent won’t match. -
Under “Then…”, set Email Obfuscation to Off.
-
Click Deploy.
If the Known Bots field isn’t available in your rule builder, use Method 2.
Method 2: Turn off obfuscation for Ahrefs IP addresses
This method matches requests by Ahrefs’ published IP ranges. It’s just as secure, but the list changes over time, so you’ll need to refresh it now and then.
Note: You need access to account-level Lists in Cloudflare. If your plan or permissions don’t include them, use Method 1.
Step 1: Create an IP list
- In the Cloudflare dashboard, open your account (not the domain), go to Settings > Lists and click Create new list.
- Name it
ahrefs_crawlersand choose IP as the content type. - Add the current Ahrefs IP ranges (see Ahrefs crawler IP addresses below) and save.
Step 2: Create the Configuration Rule
- Go back to your domain, then Rules > Create rule > Configuration Rule.
- Name the rule, e.g.
Ahrefs IP list - disable email obfuscation. - For the condition, select the IP Source Address field, the is in list operator, and your
ahrefs_crawlerslist. - Under “Then…”, set Email Obfuscation to Off.
- Click Deploy.
Why not match on the user-agent alone?
A user-agent is a piece of text any bot can copy. A rule like http.user_agent contains "AhrefsSiteAudit" works, but a scraper that fakes that string would get your email addresses unprotected. Both methods above confirm the request really comes from Ahrefs.
Ahrefs crawler IP addresses
Ahrefs crawls only from published IP ranges, and AhrefsBot and AhrefsSiteAudit use the same ones. The lists change, so always take them from Ahrefs directly:
- IP ranges (CIDR): api.ahrefs.com/v3/public/crawler-ip-ranges
- Individual IPs: api.ahrefs.com/v3/public/crawler-ips
To get the ranges as a plain list you can paste into a Cloudflare IP list:
curl -s https://api.ahrefs.com/v3/public/crawler-ip-ranges | jq -r '.prefixes[].ipv4Prefix'How to check if an IP really belongs to Ahrefs
Run a reverse DNS lookup. Ahrefs states that the hostname always ends in ahrefs.com or ahrefs.net:
$ host 54.36.148.10
10.148.36.54.in-addr.arpa domain name pointer hydrogen010-ext2.ahrefs.net.These hostnames don’t resolve forward, so combine the reverse DNS check with the published IP list instead of doing a full forward-confirmed lookup.
Ahrefs user-agents
| Bot | User-agent |
|---|---|
| AhrefsBot | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) |
| AhrefsSiteAudit (desktop) | Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit) |
| AhrefsSiteAudit (mobile) | Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version> Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit) |
The mobile Chrome version changes regularly. Check the Ahrefs robot page for current versions, and match with contains rather than an exact string.
Does Cloudflare block Ahrefs?
Not by default. Both Ahrefs bots are Cloudflare verified bots, and Cloudflare’s default bot settings let verified bots through. When Ahrefs does get blocked, it’s almost always because of your own configuration:
- Custom WAF rules or IP access rules, especially ones blocking hosting networks. Ahrefs crawls largely from OVH data centers.
- Country blocks
- Rate limiting rules
- I’m Under Attack mode or other challenge pages
- A firewall on your origin server that drops Ahrefs IPs
To let Ahrefs through your WAF, add a custom rule with the action Skip and the expression:
(cf.client.bot and http.user_agent contains "Ahrefs")Bot Fight Mode can’t be bypassed with WAF rules. If you suspect it, turn it off temporarily and re-run the crawl.
Common Ahrefs errors behind Cloudflare
| Ahrefs shows | Usual cause | Fix |
|---|---|---|
404 on /cdn-cgi/l/email-protection |
Email Address Obfuscation | Method 1 or 2 above |
| 403 Forbidden | Cloudflare WAF rule, IP access rule, or origin firewall | WAF Skip rule for verified Ahrefs bots; allow Ahrefs IPs on the origin |
| 522 | Cloudflare connected but the origin didn’t respond: the origin firewall drops Ahrefs IPs, or the server is overloaded | Allow Ahrefs IPs on the origin; lower the crawl speed in Site Audit settings |
| Timeouts, “robots.txt took too long” | Rate limiting, challenge pages, or a slow origin | Exclude verified bots from rate limiting; lower the crawl speed |
522 is a Cloudflare error code. It always means the problem is between Cloudflare and your server, not with Ahrefs.
Get accurate SEO audits
With Method 1 or Method 2 in place, the false “Page has links to broken page” errors disappear from your Ahrefs Site Audit, and your email addresses stay protected from everything else. Method 1 is the simplest to maintain. Use Method 2 if the Known Bots field isn’t available to you.
Q&A
Q: What is /cdn-cgi/l/email-protection? A: It’s the link Cloudflare substitutes for email addresses when Email Address Obfuscation is on. A script decodes it back into the real address in the browser. Crawlers that don’t run the script, like the Ahrefs Site Audit, see a 404.
Q: Can I fix it by adding /cdn-cgi/ to robots.txt?
A: No. Ahrefs still finds the links on your pages and reports them as broken. You need a Cloudflare Configuration Rule that turns off Email Obfuscation for Ahrefs.
Q: Does Cloudflare block Ahrefs? A: Not by default, because AhrefsBot and AhrefsSiteAudit are Cloudflare verified bots. Blocks usually come from your own WAF rules, IP or country blocks, rate limiting, challenge modes, or an origin firewall.
Q: What IP addresses does Ahrefs use? A: Ahrefs publishes its crawler IPs at api.ahrefs.com/v3/public/crawler-ip-ranges and api.ahrefs.com/v3/public/crawler-ips. Reverse DNS for those IPs always ends in ahrefs.com or ahrefs.net.