Screaming Frog review: Overview of the SEO crawler

Screaming Frog is an advanced SEO analytics tool that offers in-depth analysis of websites. It enables monitoring of crucial SEO metrics such as titles, metadata, and server response statuses, and is particularly adept at identifying broken links and other technical errors. This capability makes Screaming Frog essential for optimizing websites for search engine visibility.

The tool is widely used across various sectors, including e-commerce and IT, where SEO professionals, webmasters, and digital marketers rely on it to enhance the visibility and performance of their websites. Screaming Frog provides detailed insights for auditing links, images, CSS, and JavaScript, empowering users to pinpoint and resolve technical SEO issues, thereby improving the overall structure and visibility of websites.

The upcoming sections will delve into the key features of Screaming Frog, illustrating how it can be leveraged to bolster SEO performance for various types of websites.

Key features of Screaming Frog

Screaming Frog is a versatile SEO analytics tool that offers comprehensive features for optimizing websites for search engines. Here's a breakdown of its key functionalities:

Comprehensive analysis of content and technical SEO aspects

Screaming Frog enables detailed examination of each web page, assessing HTTP statuses, analyzing blog posts, and on-page SEO text, and identifying duplicate content. These capabilities are crucial for enhancing SEO and content management, contributing to better search engine rankings. Additionally, the tool identifies and tracks broken links, which can significantly improve both user experience and SEO performance.

Integration with Google Analytics and Google Search Console

By integrating with Google Analytics and Google Search Console, Screaming Frog enhances its analytics capabilities. To locate the Google Analytics tracking code, including the Google Analytics ID, navigate to the Admin section of your Google Analytics account and select Tracking Info under the Property column. This integration provides deeper insights into user behavior and page performance, enabling the formulation of more precise SEO strategies based on comprehensive data.

Data export

Screaming Frog supports exporting data in various formats, including CSV and Google Sheets, facilitating the analysis and manipulation of data outside the platform. This feature is particularly useful for analytical research, allowing SEO specialists and analysts to effectively pinpoint and address site issues. Exporting data also facilitates data analysis for improving SEO campaigns by transforming raw data into actionable insights.

Metadata analysis

A pivotal feature of Screaming Frog is its ability to analyze site metadata thoroughly. The tool examines page titles, meta descriptions, keywords, and other critical elements, helping to identify and resolve issues such as duplicate or missing metadata. Optimizing metadata is essential for improving a site’s visibility in search results, making this feature invaluable for search engine optimization efforts.

Custom settings

Screaming Frog allows for extensive customization of the crawling process. With settings that enable data extraction via CSS Path, XPath, or regular expressions, the tool offers a flexible and robust solution for conducting technical SEO audits.

Screaming Frog pricing

Screaming Frog offers both free and paid versions, catering to different user needs and project scales.

Free Version

The Screaming Frog starter edition is designed for smaller projects or those new to the tool. It allows analysis of up to 500 URLs and includes basic SEO features such as link checks, header and metadata analysis, and sitemap creation. This version is perfect for those looking to explore Screaming Frog's capabilities without financial commitment.

Paid Version

For more extensive requirements, the Screaming Frog professional version is available at $259 per year. It supports unlimited URL analysis and encompasses all the features of the free version plus additional advanced tools for comprehensive SEO audits. These tools include JavaScript rendering, advanced duplicate content analysis, and integration with Google Analytics and Google Search Console. It also supports checks for AMP and structured data, making it well-suited for in-depth SEO projects and ongoing performance assessments.

Screaming Frog interface

The main interface of Screaming Frog is intuitively designed, featuring a top panel that houses key functionalities. Hovering over each menu item reveals a dropdown with the tool’s primary features. Notably, Screaming Frog offers both dark and light themes, allowing users to customize the visual experience to their preferences. This includes adjusting the color of specific interface elements. The following sections will detail the main tabs of the interface and explore their specific functionalities.

File

The “File” tab in Screaming Frog centralizes file management and scraping configurations. It offers functionalities to open and save projects, adjust settings, and schedule tasks for automated operations. With options like “Open Recent” and “Crawl Recent”, users can quickly revisit and reanalyze previous projects. The “Scheduling” feature enhances productivity by automating regular SEO audits, streamlining the process for consistent data monitoring and updates.

Mode

The “Mode” tab in Screaming Frog allows users to select from three distinct scanning modes, each tailored to specific website analysis needs:

• Spider mode: this is the default mode for comprehensive website analysis through internal links. Users begin by entering the desired URL into the program's address bar, enabling a full scan of the site to assess SEO elements and structural integrity.
• List mode: designed for targeted scans of specific URLs, this mode is ideal for in-depth analysis of particular pages. Users can input URLs manually, upload them from a file, or import them via a sitemap, making it useful for focused evaluations of individual page features.
• SERP mode: unlike the other modes, SERP Mode does not involve scanning a website. Instead, it allows users to upload and adjust metadata to preview how it might appear in search engine results. This mode supports batch processing of data, simplifying the optimization of metadata for better SERP visibility.

Configuration

The “Configuration” tab in Screaming Frog offers extensive settings to tailor the website crawling process to specific needs. Users can adjust a wide array of parameters such as handling rules for different content types, adherence to or exclusion of robots.txt directives, management of URL redirects, and settings for CDN usage. This level of customization allows users to specify precisely which elements of a site to analyze, exclude irrelevant content, and address particular SEO challenges like JavaScript rendering and URL parameter handling.

Beyond basic configurations, this tab also includes advanced settings for HTTP headers and custom configurations, as well as API integration capabilities, enhancing the tool's ability to interact with external services and applications. By utilizing different segments and profiles, users can set up unique crawling scenarios tailored to diverse analytical needs, enabling more targeted and effective SEO strategies.

Bulk export

The next tab facilitates the export of a wide range of data obtained from site analyses into versatile formats like Excel. This feature is invaluable for SEO specialists and analysts, enabling them to conduct in-depth analyses and devise targeted improvements. Exporting this data is also crucial for analyzing organic search traffic and backlinks, helping to identify areas for improvement in SEO strategy.

Through this tab, users can selectively export data according to various parameters such as links, server responses, JavaScript configurations, AMP settings, canonical links, and images. This capability proves particularly useful for pinpointing site issues, like incorrect redirects indicated by 3xx server responses, and subsequently crafting tasks for technical teams to rectify these to direct 200 response codes.

Reports

This tab is designed to facilitate the creation and exportation of various SEO analysis reports. This includes detailed reports on crawl results, redirect chains, non-indexable tags, and issues related to pagination. This feature allows users to systematically review and manage SEO aspects of a website, providing actionable insights to optimize site performance and search engine ranking.

Sitemaps

The “Sitemaps” tab in Screaming Frog provides tools for generating XML sitemaps, which include pages and images. Generating a sitemap is crucial for enhancing a site's indexing by search engines. Users can select the type of sitemap they wish to generate and then configure necessary parameters like update frequency and URL priority in the subsequent settings window. Once generated, the sitemap can be uploaded to the site’s root directory, facilitating improved search engine indexing and contributing to enhanced SEO performance.

Visualizations

This tab offers tools that help visualize a website’s structure, enhancing the analysis process. It includes various graphical representations, such as the Crawl Tree Graph and Directory Tree Graph, which illustrate the architecture of the site and its sections. These visual tools enable users to clearly identify and analyze structural errors and issues, aiding in the optimization process and providing a better understanding of the site’s performance.

License and Help

The final tabs, “License” and “Help”, serve specific purposes.

• The “License” tab is designated for managing license keys.
• The “Help” tab offers extensive support resources, including user guides, FAQs, and direct access to technical support.

You can also activate or update the program using this tab.

Setting up a proxy in Screaming Frog

Using proxy servers is crucial for bypassing IP restrictions and blocks that can occur due to frequent site requests. This is particularly critical when dealing with large websites or conducting analyses in various geographical locations to simulate requests from specific countries or regions.

• Residential proxies are optimal for mimicking real user requests, and aiding in SEO analysis and data gathering under realistic conditions.
• Data center proxies offer faster access, but they are more susceptible to being blocked due to their common usage.

Setting up proxies in the program is straightforward and is not influenced by the type of proxy used. Here’s a step-by-step guide to adding proxies:

1. Navigate to the “File” tab, select “Settings”, and then choose “Proxy”.

   

2. Check the “Use Proxy Server” box and input the IP address and port.

   

3. If using private proxies, check the “Use Proxy Credentials” box and enter your username and password. Confirm the settings by clicking “OK and Restart”.

   

To optimize the effectiveness of the tool, it's crucial to meticulously set up crawling parameters. Adjusting the crawling speed, choosing appropriate User Agents, and utilizing proxies to mimic requests from various regions are key strategies. Additionally, regularly employing data export features in formats like CSV or Google Sheets and prioritizing the creation of sitemaps will significantly enhance SEO optimization efforts.