Can I vibe code Screaming Frog?
The SEO spider: crawl any site, find every technical issue.
A site crawler is a beautiful agent project: fetch pages, parse HTML, follow internal links politely, and report what you find — broken links, missing titles, duplicate metas, thin pages, redirect chains. For auditing your own sites, a vibe-coded crawler does the core job and exports exactly the report you want. Screaming Frog's two decades of edge cases (JavaScript rendering, massive-site memory management, every CMS's quirks) earn its keep for agency work across strange client sites — but your own site doesn't have strange quirks you can't fix.
Build an SEO site auditor for my own domains. Given a start URL: crawl same-domain pages politely (respect robots.txt, 2 concurrent, delay between requests, cap at 2,000 pages), parsing each for title, meta description, H1s, canonical, status, internal/external links, images without alt, and word count. Report page grouped by issue: broken links (with sources), redirect chains, missing/duplicate/too-long titles and descriptions, multiple H1s, thin pages (<200 words), and pages in the crawl but not the sitemap (and vice versa). Store crawls in Postgres to diff two crawls over time. CSV export per issue. Next.js with the crawler as a background job.
A polite crawler for your own domains producing an issues report: broken links, redirect chains, missing/duplicate titles and descriptions, missing alt text, thin content, and orphan pages vs your sitemap.
- JavaScript rendering for client-side-heavy sites
- Memory-managed crawls of million-page sites
- Twenty years of parser edge cases across every CMS
- Integrations pulling GSC/analytics into the crawl view
Agencies crawl other people's weird sites all day — the edge-case armor is the product. Your own site is a known quantity.
A prompt is the first move, not the whole game. Course 01 teaches you to take a prompt like this one to a shipped, working app — reviewing, correcting, and steering the agent the whole way.
Start Course 01 →