Are you looking for a free tool that can accurately analyse your website and detect its weak points?
Website auditing is something that every serious online business owner should perform.
Sometimes, we spend a lot of time writing quality content and looking for the right images, but having the correct title and metadata, checking for broken links or discovering duplicate content is just as important.
Screaming Frog SEO Spider is a powerful website crawler that offers both free and paid features, that can help you improve your website’s visibility to search engines.
What is Screaming Frog SEO Spider?
Screaming Frog SEO Spider is a bot that crawls and analyses your websites in order to look for problems that need to be rectified.
It is available for Windows, Ubuntu and Mac users.
The tool was created by Screaming Frog, a UK based company that was founded in 2011.
The Screaming Frog SEO Spider had great success among the SEO community, in fact, in 2011, this was downloaded over 100k time in 1 year.
Both a free and a paid version are available to users, with single or multiple licenses renewable yearly.
Free vs Paid version
Users can download and use Screaming Frog SEO Spider website crawler for free, without the need of having to upgrade in the future.
However, there are some features that are only available to paid users. I have gathered the list of features as shown on Screaming Frog‘s website.
Here is what the Free version has to offer:
- Find Broken Links, Errors & Redirects
- Analyse Page Titles & Meta Data
- Review Meta Robots & Directives
- Audit hreflang Attributes
- Discover Duplicate Pages
- Generate XML Sitemaps
- Site Visualisations
- Crawl Limit – 500 URLs
The Paid version offers all of the above features plus the ones listed below:
- Crawl Limit – Unlimited
- Crawl Configuration
- Save Crawls & Re-Upload
- Custom Source Code Search
- Custom Extraction
- Google Analytics Integration
- Search Console Integration
- Link Metrics Integration
- Custom robots.txt
- AMP Crawling & Validation
- Structured Data & Validation
- Store & View Raw & Rendered HTML
- Free Technical Support
Licences are per user and need to be renewed yearly at £149/year. However, a discount can be obtained if 5 or more licences are needed:
- 1 to 4: £149 per licence, per year
- 5 to 9: £139 per licence, per year
- 10 to 19: £129 per licence, per year
- 20 or more: £119 per licence, per year
Screaming Frog SEO Spider installation
Before you proceed with the installation of Screaming Frog SEO Spider website crawler, make sure that your machine has got at least 1gb of RAM.
Installation is pretty straight forward, in fact, upon clicking “Download”, the website should already detect what operating system you are using and show you the correct installer to download:
If the website detects the wrong operating system, then click on “Using a different operating system” as shown above and select the correct one.
Once the installer is downloaded, follow the instructions below:
Installation on Windows
- Find the installer that you have just downloaded. Usually, this would be in your “Downloads” folder
- Double click on the installer
- Allow the app to start the installation process
- Choose the installation type and continue –> The installer will do the rest
- Once the script has finished running, search for Screaming Frog SEO Spider among your apps
- Open the app and accept the license agreement
Installation on Mac
- Go to your “Downloads” folder
- Double-click on the file
- Click on the Screaming Frog icon on the left and drag it onto the Applications folder on the right
- Close the window
- Go to Finder, then go to Devices and click on the eject icon next to ScreamingFrogSeoSpider
- Run the SEO Spider by going to the Applications folder and double clicking on the app launcher
Installation on Ubuntu
- Go to your “Downloads” folder
- Double click on the SEO Spider .deb file
- Click on “Install” and enter your password
- Accept the license for ttf-mscorefonts-install to be run
- Once the installation is complete, run the app by double clicking on the app icon
The tool configuration feature is only offered to paid users.
There are various tweaks that can be done to the app, such as:
- Preventing images from being crawled
- Limiting the lenght of URLs to be crawled
- Setting the rendering mode for the crawl
- Ignoring “no index” URLs
- Avoiding response timeout by increasing the waiting time, which is useful for slow websites
- Extracting structured data
- Validating Schema.org
- Validating structured data against Google
- Ignoring robots.txt
Please refer to SEO Spider Configuration Guide to see the complete list with instructions.
Auditing your website
Open Screaming Frog SEO Spider and enter the URL you wish to crawl in the top bar, next to the tool’s logo and click on “Start”:
The audit might take a while, depending on the amount of URLs that are being crawled.
Remember that, as a free user, you will only be able to crawl up to 500 URLs at the time.
A report containing the crawled website’s data will be shown under the relevant tabs:
- Internal: displays all data crawled from all other tabs except the external, hreflang and custom tabs
- External: This includes information about the external URLs present in the crawled website. This is information such as the status code, content type and inlinks (number of links pointing to the external URL)
- Protocol: It shows details of the hypertext transfer protocol data, including status code, indexability and meta robots
- Response Codes: A list of response codes received when crawling the URLs present on the website
- URL: It has information related to the URLs that have been crawled, for example indexability and length
- Page Titles: It includes data related to page titles
- Meta Description: It includes data related to meta descriptions
- Meta Keywords: This shows data such as meta keywords and their length
- H1: Data related to the H1 tag, such as the text used and its lenght
- H2: Same as the H1 tag
- Images: This tab shows data related to images, for example their type and size
- Canonicals: This displays information on canonical link elements and HTTP canonicals discovered during a crawl
- Pagination: It includes information on rel=”next” and rel=”prev” HTML link elements. These are used to indicate the relationship between component URLs in a paginated series.
- Directives: Showing data for meta robots, canonicals and rel=“next” and rel=“prev” link
- Hreflang: This includes details of hreflang annotations crawled by the tool. The Hreflang attribute tells Google what language you are using
- Ajax: This shows data for websites that support Ajax. It includes Pretty and Ugly URLs
- AMP: Show the Accellerated Mobile Pages discovered during the crawl. AMP is a framework that allows browsers and apps to load a site quicker on mobile phones
- Structured Data: Error, warnings and other information regarding structured data
- Sitemaps: Information related to XML Sitemaps for each crawled URL
- Custom: This shows data related to the custom search and custom extraction features
- Analytics: When the SEO Spider is linked to a Google Analytics account, data such as number of users, bounce rates, page views and so on, will be shown
- Search Console: Linking Google Search Console to the SEO Spider, will allow you to view data such as clicks, impressions, CTR and Position
- Link Metrics: This includes lots of useful data, such as word count, text ratio and link score
One-day training workshop
Screaming Frog SEO Spider is pretty intuitive, but if you wish to get fully trained by an expert, then you can choose to attend a one-day training workshop in London.
This training session is aimed at beginners and intermediate level users and will give an understanding of what SEO is, how SEO works and how to use the SEO Spider.
The workshop costs £599 + VAT and currently runs from 9:30am to 4:30pm.
Go to the training section of Screaming Frog SEO Spider to gain a better understanding of what you will be doing during the workshop and click on “Book Now” if you wish to attend.
Conclusion: Is Screaming Frog SEO Spider worth it?
Screaming Frog SEO Spider helps you find issues on your website and gives you loads of data you can work on.
If you are just starting out, then the free version will already give you enough work to do.
However, I recommend you eventually upgrade to the paid version to unlock its full potentials.
If you wish to take a look at this tool prior to downloading it, then watch the video below and get a taste of what can be done with this powerful SEO Spider:
Are you going to try Screaming Frog SEO Spider? If so, feel free to leave a comment below and let me know what you thought about this tool 🙂