I Built an SEO Crawler with Claude

The problem I had to solve

Beginners in AI has roughly 1,681 published posts and pages. The site has grown fast enough that I’ve lost track of which links inside articles still work, which pages are getting linked to from anywhere on the site, and which Yoast SEO titles are over 60 characters (the length Bing flags as too long). Microsoft Bing is sending me about 66% of my Google-Bing-combined search traffic in 2026, and Bing’s Webmaster Tools keeps pointing at problems I can’t see all of at once.

I needed a single audit that would tell me, in one report:

• Which internal links on the site are broken?
• Which pages have zero inbound links from anywhere (orphan pages)?
• Which post titles, meta descriptions, and H1s are out of spec?
• Which images are missing alt text?

I had three obvious options. I worked through them in order.

Option 1: Have Claude crawl the site itself

The naive approach: tell Claude to start at the homepage, follow every link, and report what it finds. This is the kind of task you can describe to Claude Code in one sentence.

I ruled it out fast. Crawling a 1,681-page site by having an AI fetch and parse each page is wasteful in two ways. First, the token cost — every page Claude reads burns input tokens, and at 1,681 pages with my average page size, this would run into thousands of API calls per audit. Second, it’s the wrong tool for the job. Pattern-matching internal links and extracting title lengths is structured work. You don’t need a large language model for it. You need a script.

Lesson I’ve learned the hard way running this site: use AI for the parts of the job that need language understanding (writing, summarizing, judgment). Use code for the parts that don’t (counting, parsing, fetching). Don’t pay AI prices to do work a regex can do.

Option 2: Screaming Frog SEO Spider

Screaming Frog is the industry standard for SEO crawling — 15-plus years of polish, used at agencies everywhere. It has a free version that crawls up to 500 URLs. Above 500, it’s $259 per year.

I downloaded it. I let Claude help me try the headless command-line mode. The free version returned FATAL - Could not locate licence file — CLI automation requires the paid tier. I could have run the GUI version manually, but the 500-URL cap meant it would cover less than a third of my site. The bigger problems — orphan pages and broken links that exist on the long tail of less-visited content — would be the first things to fall outside the cap.

I also looked at the open-source alternative LibreCrawl. It’s a credible free option for sites you don’t own, but it doesn’t solve the harder issue: a generic web crawler has to fetch every page over HTTP and parse the rendered HTML, which is slow and rate-limit-fragile against WP.com Atomic hosting (more on that below).

$259/year isn’t a lot of money for a business asset. I would have paid it if I had to. But before paying, I wanted to check whether I had to.

Option 3: Build it with Claude

Here’s the insight I almost missed. Beginners in AI runs on WordPress. WordPress has a REST API. I have admin credentials. That means I can pull every post’s stored content directly from the database via authenticated HTTP requests — faster, cleaner, and rate-limit-friendly compared to crawling 1,681 rendered pages over HTTP one at a time.

For a site you own with admin access, a generic web crawler is the wrong abstraction. The right abstraction is: pull the structured data from the API, parse it once, build the indexes you need.

I asked Claude to write a Python script that does six things, in this order:

1. Fetch the universe of URLs from sitemap_index.xml — the full list of pages that should exist.
2. Pull every post and page via /wp-json/wp/v2/posts and /wp-json/wp/v2/pages, including Yoast meta data.
3. Parse each piece of content for title length, meta description length, H1 counts, word count, all internal links, all images, and alt text.
4. Build the link graph — for every URL, who links to it? Find URLs in the sitemap with zero inbound links (orphans). Flag links to URLs that don’t exist (broken).
5. HEAD-check the broken candidates against the live site to confirm they really return 404.
6. Write 14 CSVs mirroring Screaming Frog’s output naming, plus a summary.

The script is about 400 lines. Claude wrote the first draft, I reviewed it, we iterated through three small bugs together (parser edge cases, parallelization safety), and it ran clean. Build time: roughly 10 minutes of conversation.

What the crawler actually found

First full crawl — 1,681 items, 180.8 seconds. Total time from “let’s build this” to “here are the results”: about 30 minutes.

The numbers that surprised me:

• 1,258 Yoast SEO titles over 60 characters. Bing Webmaster Tools had told me 356. The crawler found nearly four times that, because Yoast appends “ – Beginners in AI” (19 characters) to every post’s title to form the SEO title, and Bing only reports the ones it has actually surfaced as errors.
• 423 orphan pages. Pages in my sitemap that nothing else on the site links to. These are also the most likely candidates for Google’s “Crawled but not indexed” bucket — nothing on the site is telling Google or Bing that these pages matter.
• 679 internal links pointing to a 3xx redirect instead of the final URL. Each one is a small SEO leak. Adds up.
• 15 truly broken internal 404 links, after I caught a false-positive class. The first pass said 37; slow-mode re-verification dropped that to 15. The other 22 were WP.com Atomic edge-throttle responses (HTTP 429) faking 404s, not real broken targets.

From audit to fix in the same session

Having the CSV outputs in hand, I had Claude help me execute the highest-leverage fix immediately: closing all 15 broken internal links. We did it in three phases, all in the same conversation.

The categories were:

• Phase 1. Seven of the broken targets had existing destinations — the link just pointed at the wrong slug. We rewrote 14 source-post references to the right targets (for example, /comprehensive-ai-models-overview → /ai-models-2026/, our actual AI models hub).
• Phase 2. Five of the broken targets were glossary entries that other posts kept trying to link to, but didn’t exist yet. Those weren’t really “broken links” — they were missing destinations. So Claude wrote the five missing glossary entries and we shipped them: What Is Extended Thinking?, What Is Hallucination in AI?, What Is a Small Language Model?, What Is the Claude API?, and What Are Sub-agents?. Now every other post on the site has those as valid link targets going forward.
• Phase 3. Three surgical content edits — one removed an orphan sentence, one rewrote an internal link to an external one, one cleaned up a malformed nested anchor tag that had been corrupted in a prior bulk operation.

By the end of the same session, the crawl had been built, run, verified, and turned into a clean fix-cycle. Re-checking the 14 affected source posts: zero broken patterns remaining.

Two lessons that surprised me

One-third of broken links were just missing glossary entries. Five out of fifteen broken targets pointed to URLs that should have existed. Other posts were trying to link to them; they just hadn’t been written yet. That changed how I think about “broken links.” They’re sometimes a signal that I’m one post short of a complete cluster — not a signal that I made a typo. Creating the missing posts is higher-leverage than rewriting links away from them, because every other post in the site now benefits from having those targets to point to.

WP.com Atomic rate-limiting silently corrupts crawl data. My first pass found 37 broken internal links. The slow-mode re-verifier dropped that to 15. The other 22 were HTTP 429 responses from my own hosting’s edge throttle — pages that exist and load fine, but were too “burst-checked” in the parallel HEAD-check pass. If I had trusted the first-pass output, I’d have “fixed” 22 links that weren’t actually broken. Lesson: anytime you run a fast crawl against your own site, run a slow re-verify on the suspect findings before acting on them.

How AI subscriptions actually pay for themselves

The conventional wisdom for the past decade was: pay $30 here for a writing tool, $50 there for an SEO tool, $99 there for project management, $80 there for an analytics dashboard. Stack up enough SaaS subscriptions and your monthly tooling bill is bigger than your rent.

A Claude Pro subscription is around $20 per month — $240 a year. That’s roughly the same as one of the dedicated SaaS tools I would have paid for to do just one job. Here’s the actual math on this single project:

• Screaming Frog SEO Spider, paid tier: $259/year
• Claude Pro subscription: $240/year (and I would be paying for it anyway)
• Net difference: $19 — and I now have a crawler that does things Screaming Frog can’t, that I own, that I can extend.

That’s one tool. Now multiply the pattern across the others I could build for myself with Claude: a custom analytics dashboard, a content audit tool, a competitor-pricing monitor, a workflow that emails me when one of my own posts breaks. Each one of those, as a third-party subscription, would run $20 to $100 per month. As a script Claude wrote for me in 30 minutes, the marginal cost is zero.

The shift in framing: instead of “what app should I subscribe to that does X?” the question becomes “can Claude help me build X in 30 minutes?” For a surprising number of X’s, the answer is yes — especially for the small, single-purpose tools that SaaS companies have built entire businesses around.

You don’t need to be a developer to do this. You need to be clear about what you want, describe it carefully to Claude, and verify the result does what you asked for. That’s a skill any motivated beginner can develop in a few weeks. The AI subscription stops being an expense category and starts being the lever that retires the rest of them.

What you can take from this

Three things, in order from broadest to most specific.

If you own the system you’re auditing, you have leverage generic tools don’t. Screaming Frog has to treat every site as a black box and crawl over HTTP. You don’t. You probably have database access, API access, or admin credentials. Use them. This applies to SEO audits, but also analytics, customer-data analysis, internal-link enrichment, and a dozen other tasks where the “industry-standard tool” is doing more work than necessary because it assumes you’re an outsider.

Match the tool to the job. The naive option — have Claude do everything — would have cost more tokens and produced worse results. The right pattern was: use Claude to write the script, use the script to do the heavy lifting, use Claude again to interpret results and write the fixes. Each part of the job goes to the tool that’s best at it.

Build small, verify small, iterate fast. The crawler took 30 minutes total. The fix cycle took another 30 minutes. Neither of those would have been possible if I’d tried to build “the perfect SEO platform” up front. The right size for the first version is the smallest version that answers the specific question you asked.

Frequently asked questions

How long would this take a non-developer?
Longer, but not by as much as you’d think. If you’ve never written a Python script, plan on a couple of hours to understand what Claude is producing and verify it does what you want. If you’re a developer who’s never used the WordPress REST API, plan on about an hour. The “30 minutes” figure here counted only the back-and-forth time with Claude — not the years of background that let me know what to ask for.

What about Google Search Console and Bing Webmaster Tools? Don’t they do this?
Partially. GSC and Bing WMT surface a curated list of issues they’ve seen during crawls — usually the worst ones. They don’t give you the full link graph, the full orphan list, or the full metadesc-length distribution. The custom crawler complements them; it doesn’t replace them.

What if I don’t use WordPress?
The pattern works for anything with a structured API. Shopify, Ghost, Webflow, and Notion all have admin APIs. Static-site generators (Hugo, Eleventy, Astro) give you direct access to the source files, which is even better. The specific code changes; the approach doesn’t.

Should I still buy Screaming Frog?
It depends on your needs. Screaming Frog’s killer feature is the Google Search Console overlay — you can see which orphan URLs are actually getting impressions vs. zero traffic. That requires either Screaming Frog Pro or building your own GSC integration. For most content-site owners with admin access, the custom approach gets you 80% of the value for free. The remaining 20% is what the $259 buys.

What was the actual prompt you used with Claude?
I described the problem in plain English: “I need to audit my whole WordPress site for broken internal links, orphan pages, and out-of-spec titles and meta descriptions. Don’t crawl the site over HTTP — pull everything via WP REST API where possible. Output CSVs that match Screaming Frog’s tab naming conventions.” That was enough. Claude wrote the first draft, I asked for changes, and we iterated.

Is the code open source?
It’s specific to my site’s structure and credentials, so I’m not publishing it as a generic library. But if you’re a Beginners in AI reader and you want to do something similar, the structure is straightforward enough that Claude can write you your own version in an hour. Describe your site and what you want to know about it; let Claude write the rest.

Update: Round 2 — Learning From What’s Already on GitHub

A few weeks after the first version was running, I wanted the crawler to do more — and I was honest about it: I had no idea what the “more” should be.

So I had Claude survey the actual open-source SEO audit projects on GitHub — the ones with hundreds of stars, maintained by real teams — and report back on which features they have that mine didn’t. The shortlist that came back was the kind of thing that’s easy to miss when you build alone:

• External link health-check — not just internal links. My Tool Directory had broken external URLs the crawler had never even tested.
• Canonical-tag audit — catches when Yoast emits a canonical pointing somewhere weird (common after slug renames).
• Meta-robots / X-Robots-Tag audit — catches accidental noindex pages that quietly disappear from Bing and Google.
• Sitemap reconciliation — finds URLs in the sitemap that don’t actually exist, plus pages that exist but are missing from the sitemap.
• Duplicate title / meta-description detection — one of the easiest fixes nobody catches.

I added all five. The script grew from 473 lines to about 630, the CSV output expanded from 14 files to 20, and the next run surfaced findings I’d never have thought to check. The external-link check alone caught broken citations to fake Wikipedia URLs (AI-hallucinated from old content), dead newsletter post URLs from a Beehiiv slug-format change, and Grokipedia URLs using the wrong path pattern. None of those would have shown up in a normal internal-link audit.

A few engineering details I borrowed from the open-source projects: dedupe external URLs before checking (one dead domain referenced 50 times across the site generates one HEAD request, not 50); use a real browser-like User-Agent so Cloudflare doesn’t bot-challenge the request; and fall back from HEAD to GET when a vendor rejects HEAD methods.

The point I want to make: I didn’t use any of those open-source projects directly. I used them as a feature checklist. The actual implementation is still mine — around 600 lines of Python that does exactly what I need and nothing else. But the “what I need” question got way smarter once I’d looked at what other people had already solved.

If you’re building something similar, these are the projects worth studying as feature inventories:

• seonaut — Go-based SEO crawler. Strong on the broad audit checklist (canonicals, robots, hreflang, sitemap reconciliation).
• gflare-tk — Python-based crawler with a wide feature surface. Where I borrowed the page-depth and anchor-text analysis ideas.
• viasite/site-audit-seo — Node-based, very practical. Good thin-content detection patterns.
• lychee — Rust-based broken-link checker, very fast. The model for my external-link audit.
• seo-audits-toolkit — broader audit framework with JSON-LD validation and Open Graph checks.

The lesson: the open-source world has already done most of the thinking on what an SEO audit should cover. You don’t have to use their code — but you should absolutely use their feature lists.

Sources

• Screaming Frog SEO Spider — pricing and free-tier limits
• LibreCrawl on GitHub — the open-source alternative I evaluated
• seonaut — Go SEO crawler I used as a feature reference for Round 2
• gflare-tk — Python crawler with page-depth and anchor-text analysis
• viasite/site-audit-seo — Node-based practical auditor
• lychee — fast Rust-based broken-link checker
• seo-audits-toolkit — JSON-LD validation and OG-tag checks
• WordPress REST API documentation
• Anthropic documentation — how I worked with Claude on the build

You might also like

• Claude AI Review — the AI I used to build this.
• What is Claude Code? — the terminal coding tool I used for the script.
• How to Use Claude AI — the beginner guide.
• Claude Code Hooks and Plugins Guide — extending Claude Code further.
• What Is the Claude API? — one of the glossary entries created in this fix cycle.
• Every AI Model Worth Knowing in 2026.
• AI Glossary — every term in plain English.