Screaming Frog SEO Spider – Audit Your Site for Free!

Screaming Frog SEO Spider - Audit your site for free!

Are you looking for a free tool that can accurately analyse your website and detect its weak points?

Website auditing is something that every serious online business owner should perform.

Sometimes, we spend a lot of time writing quality content and looking for the right images, but having the correct title and metadata, checking for broken links or discovering duplicate content is just as important.

Screaming Frog SEO Spider is a powerful website crawler that offers both free and paid features, that can help you improve your website’s visibility to search engines.

Screaming Frog SEO Spider - Audit your site for free


What is Screaming Frog SEO Spider?

Screaming Frog SEO Spider is a bot that crawls and analyses your websites in order to look for problems that need to be rectified.

It is available for Windows, Ubuntu and Mac users.

The tool was created by Screaming Frog, a UK based company that was founded in 2011.

The Screaming Frog SEO Spider had great success among the SEO community, in fact, in 2011, this was downloaded over 100k time in 1 year.

Both a free and a paid version are available to users, with single or multiple licenses renewable yearly.


Free vs Paid version

Users can download and use Screaming Frog SEO Spider website crawler for free, without the need of having to upgrade in the future.

However, there are some features that are only available to paid users. I have gathered the list of features as shown on Screaming Frog‘s website.

Here is what the Free version has to offer:

  • Find Broken Links, Errors & Redirects
  • Analyse Page Titles & Meta Data
  • Review Meta Robots & Directives
  • Audit hreflang Attributes
  • Discover Duplicate Pages
  • Generate XML Sitemaps
  • Site Visualisations
  • Crawl Limit – 500 URLs

The Paid version offers all of the above features plus the ones listed below:

  • Crawl Limit – Unlimited
  • Scheduling
  • Crawl Configuration
  • Save Crawls & Re-Upload
  • Custom Source Code Search
  • Custom Extraction
  • Google Analytics Integration
  • Search Console Integration
  • Link Metrics Integration
  • Rendering (JavaScript)
  • Custom robots.txt
  • AMP Crawling & Validation
  • Structured Data & Validation
  • Store & View Raw & Rendered HTML
  • Free Technical Support

Licences are per user and need to be renewed yearly at £149/year. However, a discount can be obtained if 5 or more licences are needed:

  • 1 to 4: £149 per licence, per year
  • 5 to 9: £139 per licence, per year
  • 10 to 19: £129 per licence, per year
  • 20 or more: £119 per licence, per year

Screaming Frog SEO Spider installation

Before you proceed with the installation of Screaming Frog SEO Spider website crawler, make sure that your machine has got at least 1gb of RAM.

Installation is pretty straight forward, in fact, upon clicking “Download”, the website should already detect what operating system you are using and show you the correct installer to download:

Screaming Frog SEO Spider Download

Screaming Frog SEO Spider installer

If the website detects the wrong operating system, then click on “Using a different operating system” as shown above and select the correct one.

Once the installer is downloaded, follow the instructions below:

Installation on Windows

  • Find the installer that you have just downloaded. Usually, this would be in your “Downloads” folder
  • Double click on the installer
  • Allow the app to start the installation process
  • Choose the installation type and continue –> The installer will do the rest
  • Once the script has finished running, search for Screaming Frog SEO Spider among your apps
  • Open the app and accept the license agreement

 

Installation on Mac

  • Go to your “Downloads” folder
  • Double-click on the file
  • Click on the Screaming Frog icon on the left and drag it onto the Applications folder on the right
  • Close the window
  • Go to Finder, then go to Devices and click on the eject icon next to ScreamingFrogSeoSpider
  • Run the SEO Spider by going to the Applications folder and double clicking on the app launcher 

Installation on Ubuntu

  • Go to your “Downloads” folder
  • Double click on the SEO Spider .deb file
  • Click on “Install” and enter your password
  • Accept the license for ttf-mscorefonts-install to be run
  • Once the installation is complete, run the app by double clicking on the app icon

Tool configuration

The tool configuration feature is only offered to paid users.

There are various tweaks that can be done to the app, such as:

  • Preventing images from being crawled
  • Limiting the lenght of URLs to be crawled
  • Setting the rendering mode for the crawl
  • Ignoring “no index” URLs
  • Avoiding response timeout by increasing the waiting time, which is useful for slow websites
  • Extracting structured data
  • Validating Schema.org
  • Validating structured data against Google
  • Ignoring robots.txt

Please refer to SEO Spider Configuration Guide to see the complete list with instructions.


Auditing your website

Open Screaming Frog SEO Spider and enter the URL you wish to crawl in the top bar, next to the tool’s logo and click on “Start”:

Audit website seo spider

The audit might take a while, depending on the amount of URLs that are being crawled.

Remember that, as a free user, you will only be able to crawl up to 500 URLs at the time.

A report containing the crawled website’s data will be shown under the relevant tabs:

  • Internal: displays all data crawled from all other tabs except the external, hreflang and custom tabs
  • External: This includes information about the external URLs present in the crawled website. This is information such as the status code, content type and inlinks (number of links pointing to the external URL)
  • Protocol: It shows details of the hypertext transfer protocol data, including status code, indexability and meta robots
  • Response Codes: A list of response codes received when crawling the URLs present on the website
  • URL: It has information related to the URLs that have been crawled, for example indexability and length
  • Page Titles: It includes data related to page titles
  • Meta Description: It includes data related to meta descriptions
  • Meta Keywords: This shows data such as meta keywords and their length
  • H1: Data related to the H1 tag, such as the text used and its lenght
  • H2: Same as the H1 tag
  • Images: This tab shows data related to images, for example their type and size
  • Canonicals: This displays information on canonical link elements and HTTP canonicals discovered during a crawl
  • Pagination: It includes information on rel=”next” and rel=”prev” HTML link elements. These are used to indicate the relationship between component URLs in a paginated series.
  • Directives: Showing data for meta robots, canonicals and rel=“next” and rel=“prev” link
  • Hreflang: This includes details of hreflang annotations crawled by the tool. The Hreflang attribute tells Google what language you are using
  • Ajax: This shows data for websites that support Ajax. It includes Pretty and Ugly URLs
  • AMP: Show the Accellerated Mobile Pages discovered during the crawl. AMP is a framework that allows browsers and apps to load a site quicker on mobile phones
  • Structured Data: Error, warnings and other information regarding structured data
  • Sitemaps: Information related to XML Sitemaps for each crawled URL
  • Custom: This shows data related to the custom search and custom extraction features
  • Analytics: When the SEO Spider is linked to a Google Analytics account, data such as number of users, bounce rates, page views and so on, will be shown
  • Search Console: Linking Google Search Console to the SEO Spider, will allow you to view data such as clicks, impressions, CTR and Position
  • Link Metrics: This includes lots of useful data, such as word count, text ratio and link score

One-day training workshop

Screaming Frog SEO Spider is pretty intuitive, but if you wish to get fully trained by an expert, then you can choose to attend a one-day training workshop in London.

This training session is aimed at beginners and intermediate level users and will give an understanding of what SEO is, how SEO works and how to use the SEO Spider.

The workshop costs £599 + VAT and currently runs from 9:30am to 4:30pm.

Go to the training section of Screaming Frog SEO Spider to gain a better understanding of what you will be doing during the workshop and click on “Book Now” if you wish to attend.

Screaming frog workshop trainer


Conclusion: Is Screaming Frog SEO Spider worth it?

Screaming Frog SEO Spider helps you find issues on your website and gives you loads of data you can work on.

If you are just starting out, then the free version will already give you enough work to do.

However, I recommend you eventually upgrade to the paid version to unlock its full potentials.

If you wish to take a look at this tool prior to downloading it, then watch the video below and get a taste of what can be done with this powerful SEO Spider:


Are you going to try Screaming Frog SEO Spider? If so, feel free to leave a comment below and let me know what you thought about this tool 🙂

Author: Cristina

Hey guys, I am Cristina Pucci, the owner of TheOnlineEntrepreneurSecret.com. I am here to help you start your online business and avoid scams! Feel free to email me for any questions at cristina@theonlineentrepreneursecret.com :)

14 thoughts on “Screaming Frog SEO Spider – Audit Your Site for Free!

  1. This a great and simple to follow guide on how to use screaming fox seo spider.

    You laid it all out clearly and showed series of examples that helped emphasize your various points. This guide will be very useful for me. I have heard about this free tool before but I haven’t tasted its benefit this much. 

    This is the most comprehensive review I have ever read on  screaming frog which is an eye opener for me.

    This is the type of information some so-called professional bloggers will love to hide from public knowledge. I’m so happy I found your blog

    1. Thank you Ola 🙂

      f you get to try Screaming Frog SEO Spider, please do let me know what you think!

      Cristina

  2. The screaming frog SEO spider sounds like a very relevant tool to help bloggers access the quality of their webpage. I’m going to try the SEO spider because I believe it’s a product that can help the overall usability of my website. If I join the paid service, can I earn by referring others?  Expecting to hear your response soon.

    1. Hi Louis,

      No, I am afraid they do not have an affiliate program at the moment.

      This is stated in their FAQ section.

      Many thanks,

      Cristina

  3. Thanks for writing this screaming frog seo spider.i find this article so full of informative on how to install and use it and so other things 

    I must say this is my first time of reading or hearing anything about screaming frog seo spider maybe is because I am a beginner and am just writing my first content for my first website.but either way I will like to make use of screaming flog seo spider in other for my work to be as professional as possible 

  4. This is a very good feature in every website and most importantly, it is easy to install. Great job by the developers but my question is ” Does the screaming frog bot help in checking for plagiarism on the website”? In my opinion, the bot should also be able to check for plagiarism. 

  5. This is a very great post and a wonderful writeup too. I have always wanted something like this, a way to detect faults in my websites, what I need to do and fix for them to get better. I guess Screaming Frog SEO Spider according to your review will really do justice to any loopholes found provided I go fully for the paid version. Thanks for sharing this. Very helpful. 

  6. This is a nice product every website owner need to integrate on their websites. It is good the company has the free and the paid version. I would have to try the free version. first, based on the results I got that would determine if I would have to switch to the paid version. What are the options for people outside London and England as regarding attending the one-day training? Is there an alternative to this? The training cost looks high for a few hours of training. 

    1. Hi Tolu,

      It looks like for now, the workshop is only available in London.

      I personally don’t think it is necessary for you to attend the workshop. They have a support team, so you can easily contact them and ask about things you need clarifications on 🙂

      Cristina

  7. dear Cristina this review is fantastic. I have been looking for a tool that will help me analyse how SEO compliant my website is. And i am happy to come across your article. Indeed I have not heard of Screaming Frog, but you description is perfect. I will definitely try is out. Thank you for sharing. 

  8. Hello Cristina,

    I have read your article on screaming frog seo spider, really very helpful and informative article and truly appreciated. thank you so much for this helpful article. i would like to follow the instructions and share this with my friends also. need more article on this topic. best regards for you.

Leave a Reply

Your e-mail address will not be published. Required fields are marked *