Screaming Frog – The complete guide to SEO audits for professionals

In the world of SEO, everyone knows the little green frog. For many, it’s the go-to tool for a quick check of page titles and hunting down 404 errors. But if that’s all you’re using it for, you’re essentially using a supercomputer as a simple calculator. The true power of this tool lies deep beneath the surface, in features that most people don’t know exist or are hesitant to touch.

This article isn’t another basic tour of the interface. It’s a professional guide that shows you how to configure the tool to audit any website (from tiny to enormous), perform advanced diagnostics, and find critical issues that your competitors are missing.

Who is this guide for? It’s for SEO specialists, marketers, and site owners who understand the basics but are ready to level up their technical analysis skills. Let’s begin.

The fundamentals for an ironclad configuration

A successful audit is built on a proper setup. Spending five minutes on these settings will save you hours of frustration and yield far more accurate data.

Memory vs. Database

This is the most critical decision for crawl stability. Memory Storage is the default mode, which is fast and perfect for sites with up to ~500,000 URLs. For giant websites, Database Storage is an absolute necessity, as it writes the crawl data to your hard drive to prevent crashing.

How to set it – Navigate to Configuration > System > Storage Mode to choose your option. Even when using Database mode, allocating more RAM via System > Memory Allocation will keep the user interface running smoothly.

JavaScript rendering as a mandatory setting

Today, most websites rely heavily on JavaScript to load content and links. Scanning in the default Text Only mode is like auditing with your eyes closed—the crawler simply won’t see what Google sees.

How to set it – Go to Configuration > Spider > Rendering and change the mode to JavaScript.

Pro tip – In the same tab, increase the AJAX Timeout from 5 to 10 or 15 seconds. This gives the crawler enough time to fully render complex, resource-heavy pages.

API integrations for audit superpowers

This is where you transform your technical audit into a strategic analysis. Connecting Screaming Frog to your Google tools allows you to overlay technical data with real-world performance metrics.

How to set it up – Go to Configuration > API Access and connect to Google Analytics 4, Google Search Console, and PageSpeed Insights.

What you gain – You can see traffic data (GA4), search performance (GSC), and Core Web Vitals (PSI) for every single URL, all in one interface. This lets you instantly spot high-traffic pages with technical errors or pages with high impressions but low click-through rates, pointing to a title tag or meta description issue.

Saving your configuration

You don’t need to repeat these steps every time you launch the application.

How to do it – Once you’re happy with your settings, go to File > Configuration > Save As.... Then, select File > Configuration > Always Use to make it your default startup configuration.

The advanced technical audit for finding hidden problems

With a solid foundation in place, we can move on to the analysis. The key is to focus on interpreting the data to find less obvious problems.

Response codes

This report tells a story that goes far beyond 404s. It’s not just about finding “Page Not Found” errors; it’s about understanding how your server is responding to requests. Use the built-in reports to find redirect chains (Reports > Redirects > Redirect Chains), which slow down users and waste crawl budget, and look for patterns in server errors (5xx) that might indicate an underlying server health issue.

Indexability audit

Your goal is to have full control over what Google can and cannot index. This section is your control panel. Use the Indexability and Indexability Status columns to quickly find pages that are mistakenly blocked by noindex tags or your robots.txt file. A single incorrect Disallow rule can make an entire section of your site invisible to Google.

Content analysis

Use the Content tab to find sitewide issues that impact quality. The Duplicates report is your best tool for identifying pages with identical or near-identical content, which can lead to keyword cannibalization. Additionally, sorting the main Internal report by Word Count is a simple but effective way to find “thin content” pages that offer little value to users.

Diagnostic techniques that separate the amateurs from the experts

This is what will truly elevate your audits. These three advanced techniques will give you a significant competitive advantage.

Finding orphan pages with the built-in Crawl Analysis

Forget about manually exporting and comparing spreadsheets. Screaming Frog has a powerful, automated module for finding orphan pages by cross-referencing your crawl with hard data from Google.

• Step 1 – Configure your data sources. Ensure your Google Analytics and Google Search Console APIs are connected. Then go to Crawl Analysis > Configure. In the Orphan Pages tab, check the boxes to find orphans from GA, GSC, and your XML Sitemaps.
• Step 2 – Run the analysis. After your main crawl is complete, simply click Crawl Analysis > Start.
• Step 3 – Interpret the results. The tool will populate the Orphan URLs tab and report. This shows you every URL that exists in Google’s world (or your sitemap) but has no internal links pointing to it. You can then decide whether these pages need to be integrated into your site architecture, redirected, or removed.

Custom extraction for scraping any data you want

This is arguably the most powerful and underused feature in the entire tool. It lets you scrape any piece of information from a page’s HTML, even if it’s not a standard SEO element.

• How it works – Go to Configuration > Custom > Extraction. You can use XPath, CSS Selectors, or Regex to define what data you want to pull.
• Example for Schema – Use it to check which pages have (or are missing) FAQPage or Article schema.
• Example for E-commerce – Use it to scrape product prices, stock status (“Out of Stock”), or review counts from thousands of pages at once.
• Example for Content Audits – Use it to extract publication dates to find outdated content or check if author names are present on all blog posts.

Architecture visualizations to see your site like a robot

Tables of data are useful, but sometimes you need to see the bigger picture. Visualizations provide an immediate understanding of your site’s structure.

How to generate them – After a crawl, go to Visualisations > Force-Directed Crawl Diagram. This instantly reveals your site’s architecture, showing you content hubs, pages buried deep within the site structure, and disconnected sections that require better internal linking.