Why a 15-year-old desktop app is still the default

Ask an SEO agency what it opens first on a new client site and the answer is usually the same green frog. Screaming Frog's SEO Spider launched in November 2010 at £99, runs on your own computer rather than in someone's cloud, and charges one flat price with no feature tiers: £199 / $279 / €245 per user per year.

What changed in the last eighteen months is less visible. Version 22 (June 2025) added semantic similarity analysis built on vector embeddings. Version 23 (October 2025) rebuilt the PageSpeed reports around Google's new "Insight" audits. Version 24, released on 19 May 2026, ships a built-in MCP server, so Claude, LM Studio and other AI chat assistants can start a crawl and read the results in plain English. The current build is 24.3 from 29 June 2026, checked on the release history page on 14 October 2026.

The MCP server: crawling by chat

The headline v24 feature works like this: you switch on the MCP server in the SEO Spider's configuration, point an AI client at it, and ask something like "crawl this site and tell me which templates are missing canonicals." Screaming Frog's announcement describes it as running "crawls, analyse, export and manipulate data using the SEO Spider and node.js within Claude, LM Studio and other AI chat assistants."

The launch post is candid about the limits. It is a paid-licence feature only, it requires database storage mode, and "not every feature and function is currently supported." A staff reply under the post confirms list mode, where you paste URLs instead of following links, does not work through MCP yet. Rich Voller's agency guide adds that crawl speed, user agent, JavaScript rendering and custom extraction cannot be set from the chat either; the workaround is loading a saved configuration profile.

The cost nobody puts on the banner: the assistant's tokens are yours. Voller warns against pulling the full content of 20,000 URLs into one conversation, and he is right. The same bring-your-own-key rule covers the older AI features. Custom prompts can run against every crawled page using OpenAI, Gemini, Anthropic or a local Ollama model, but OpenAI needs "a funded OpenAI account and an OpenAI API key. This is different to a ChatGPT subscription." Gemini's free tier now allows "just 5 requests per minute", per the configuration guide - useless on a 10,000-page crawl. v24 at least shows token usage per provider.

What a crawl actually finds

Strip away the AI layer and the SEO Spider does what it always has. It requests every URL it can reach, the way a search engine bot would, and sorts what it finds into more than 300 issues, warnings and opportunities: broken links and server errors, redirect chains and loops, missing or duplicated titles and meta descriptions, canonical and hreflang mistakes, and robots directives wherever they hide. It renders JavaScript with a bundled Chromium, so React and Vue sites are crawled as Google sees them rather than as empty shells.

The most useful small addition in v24 is the "Pages With Uncrawlable Internal Outlinks" filter. It flags links Google says it may not follow: a span or div carrying an href, an onclick handler, a javascript: link. Single-page apps produce these by the hundred; the tool flags them, a developer still fixes them.

The deeper features are where the licence earns its money. Custom extraction with XPath, CSS Path or regex pulls any on-page value, every product price or author name, into its own column. Structured data is validated against Schema.org and Google's rich result rules, accessibility is checked with the AXE engine against WCAG. Analytics, Search Console and PageSpeed Insights connect over their APIs, and link metrics come in from Ahrefs, Majestic or Moz. Since v23 the Ahrefs connection works on any paid Ahrefs plan instead of enterprise only, though you still need that subscription.

Scheduled crawls now tell you what changed

Scheduling has been in the paid version for years, but until v24 a scheduled crawl just left a file behind. Now a scheduled task can automatically compare the last two crawls, and the crawl-complete email carries the issue tables plus everything that changed since the previous run, with zipped exports attached if you ask for them. For an agency watching 30 client sites, a weekly manual diff becomes an inbox rule.

The catch is physical. The schedule runs on your machine. If the laptop is asleep at 3am, so is the crawl.

The free version stops at 500 URLs, and sooner in practice

The free download crawls up to 500 URLs and covers the basics: broken links, errors and redirects, titles and meta data, meta robots, hreflang, exact duplicates, XML sitemap generation and site visualisations. For a 200-page brochure site that is a genuine free audit.

Everything else on Screaming Frog's own comparison table is paid. That includes crawl configuration, saving crawls, JavaScript rendering, scheduling, structured data validation, custom extraction, the Google integrations and every AI feature. No configuration means you cannot exclude a URL pattern, so on a shop with faceted filters the 500 URLs are often spent on ?colour= variations before the crawler reaches the pages you cared about. No saving means closing the window loses the crawl.

Pricing and the licence small print

One product, one price, with a discount only for buying five or more licences at once. Checked on the pricing page on 14 October 2026, per licence per year:

The pound price has not moved since January 2023, when it rose from £149. The dollar price went from $259 to $279 in a May 2025 currency adjustment. Without a VAT number, tax may be added at checkout. There are no coupon codes and no Black Friday sale; the FAQ says so outright.

A licence is a person, not a seat pool. Each key belongs to one named individual, who may install it on several of their own devices but not on a shared machine. A five-person team buys five licences, and reassigning one when someone leaves is limited to once a year. There is no cap on how many websites that person crawls.

An expired licence locks your archive. When it lapses the app returns to the free version and "previously saved crawls cannot be opened." Export anything you need to keep while the licence is live.

Renewal is manual by default. Auto-renew is off unless you chose it at purchase, buying a new key early does not extend the old one, and renewing after a price rise gets you the new price. Refunds are full within 14 days of purchase and nothing pro rata after that.

Get Screaming Frog SEO Spider

Where a desktop crawler runs out

"Unlimited" on the pricing page carries an asterisk: the crawl ceiling "is dependent on allocated memory and storage." The default database storage mode writes to disk and wants an SSD. Memory mode is "generally better suited for crawling websites under 500k URLs", and the guide recommends allocating at least 2 GB less than your machine's total RAM. On a 16 GB laptop, a million-URL ecommerce crawl is an overnight job that ties up the machine.

There is no shared workspace either. Crawls are files, and handing one to a colleague means sending a file to someone who also holds a licence. Server log analysis is not in the SEO Spider at all. It is a separate product, the Log File Analyser, at £99 / $139 / €129 a year, and we found no bundle price for the pair.

Alternatives worth pricing

If the site is past a few hundred thousand URLs, or you need crawl data and server logs in one place, look at OnCrawl or Botify. Both run in the cloud, keep shared history and read your logs, at many times the price. If you want a cheap audit that runs in a browser and emails you weekly, Seobility handles small sites without anything to install. If you already pay for Semrush or Ahrefs, their cloud site audits cover the routine checks with far less control over what gets crawled and extracted. And SEO PowerSuite is the other desktop option, with rank tracking and backlink tools alongside its auditor.

Verdict

Buy the SEO Spider if you audit sites for a living, alone or in a small team, and you want control: custom extraction, JavaScript rendering, staging-versus-live comparisons, and now an AI assistant that can drive the crawl. At £199 / $279 per person per year it is the cheapest professional crawler on the market, and Aleyda Solis's line on the product page, "My 'go to' tool for initial SEO audits and quick validations", is how most of the industry uses it.

Skip it if your site runs to millions of URLs and the whole team needs one shared crawl history; that is Botify or OnCrawl territory. Skip the licence, not the tool, if your site is under 500 pages and you only need broken links and metadata, because the free version does that. And if you are buying it for the AI features, budget for your own API tokens on top of the £199.

Try Screaming Frog SEO Spider