Server log analysis for SEO

The reports in Google Search Console are useful, but they only show Google’s interpretation of the data. If you want to see the uncensored truth about how Googlebot (and other bots) sees and crawls your site, you have to go to the source, raw server logs. This is the only place you’ll find every single bot request, every server response it encounters, and every wasted byte of your crawl budget.

This 5-step guide will show you exactly how to conduct such an analysis to make SEO decisions based on hard data, not guesswork.

Step 1. Get access to raw server logs

Server logs are text files that record every single request made to the server. Before you can begin your analysis, you need to get these files.

• Where to find them? The location depends on your server configuration, but you will most commonly find them in these paths:
• How to download them?

You need the raw access logs (access.log), not the error logs (error.log). These files can be very large (several gigabytes for popular sites), so ensure you have enough disk space.

Choose and configure an analysis tool

Manually reading millions of lines in a text file is impossible. You need specialized software to process this data and present it in a readable format.

• Screaming Frog SEO Log File Analyser (Paid, with a free version) – this is the industry standard. It’s relatively inexpensive and incredibly powerful. The free version allows you to analyze up to 1,000 log lines, which is enough to get familiar with the tool.
• Other options:

Example of script in Python

Here’s the Python code. You can save this as a .py file e.g., seo_log_analyzer.py

How to use the script

Prerequisites:

• Python – ensure Python 3.x is installed on your system.
• Log file – obtain your access.log file(s) from your web server (Apache, Nginx). You can usually find these in /var/log/apache2/ or /var/log/nginx/ via SSH/FTP, or download them from your hosting panel’s “Raw Access Logs” section.
• Placement – place your access.log file(s) in the same directory as your Python script, or update the LOG_FILE_PATH variable to point to its exact location.

Save the Script – save the code above as a Python file (e.g., seo_log_analyzer.py).

Prepare your log file:

• If your log file is compressed (e.g., access.log.gz), you’ll need to decompress it first. You can use tools like gunzip on Linux/macOS or 7-Zip on Windows.
• If you have multiple log files (e.g., access.log.1, access.log.2), you can concatenate them into one large file for a comprehensive analysis using cat access.log.* > combined_access.log (Linux/macOS) or manually combine them. Then, update LOG_FILE_PATH to point to this combined file.

Configure LOG_FILE_PATH:

• Open seo_log_analyzer.py in a text editor.
• Locate the LOG_FILE_PATH variable.
• Change 'access.log' to the actual name/path of your log file. For example: LOG_FILE_PATH = 'my_website_access_logs.log' or LOG_FILE_PATH = '/path/to/your/logs/access.log'.

Remove dummy data block:

• The if __name__ == "__main__" – block at the bottom of the script contains dummy data creation for testing. For analyzing your real logs, you must remove or comment out the lines that create the dummy_log_data and write it to LOG_FILE_PATH. Keep only the analysis_results = analyze_log_file(LOG_FILE_PATH) and print_report(analysis_results) lines.
• The cleanup os.remove(LOG_FILE_PATH) at the end should also be removed if you’re working with your actual log files, as you don’t want to delete them.

Run the script:

• Open your terminal or command prompt.
• Navigate to the directory where you saved seo_log_analyzer.py and your log file.
• Execute the script using: python seo_log_analyzer.py

Interpreting the report & taking action

The script will output a detailed report directly to your console, similar to this:

Starting analysis of file: access.log...
Found 8 potential Googlebot requests. Starting IP verification (this may take some time for large files)...
Successfully verified 7 requests from authentic Googlebot IPs.

========================================
        SEO LOG ANALYSIS REPORT        
========================================
Total 'Googlebot' User-Agent hits: 8
Number of **verified** Googlebot hits: 7
----------------------------------------

## Googlebot Response Status Codes Breakdown:
  - Code 200: 5 times
  - Code 404: 1 times
  - Code 503: 1 times

Status code category summary:
  - Category 2xx: 5 times
  - Category 4xx: 1 times
  - Category 5xx: 1 times
----------------------------------------

## TOP 20 Most Crawled URLs by Googlebot:
 1. /home-page (2 times)
 2. /products/new-model (2 times)
 3. /old-article (1 times)
 4. /assets/style.css (1 times)
 5. /api/data (1 times)
----------------------------------------

Actionable Insights:
  - High 4xx/5xx codes suggest broken internal links or server issues. Investigate these URLs.
  - If Googlebot frequently crawls low-value URLs (e.g., filtered results, old content), consider using robots.txt or canonical tags to manage crawl budget.
  - Pages with low crawl frequency but high importance might need improved internal linking or sitemap updates.
========================================

Configuration in Screaming Frog SEO Log File Analyser:

1. Launch the program and create a new project.
2. Drag and drop your .log or .gz file(s) into the program window.
3. The tool will automatically start processing the data. In the “User Agents” tab, you’ll see a list of all bots that have visited your site.

Identify and Verify Googlebot’s Activity

Not every request with a “Googlebot” User-Agent actually comes from Google. Malicious bots often spoof their user agent to bypass security measures. That’s why verification is critical.

In the Screaming Frog Log File Analyser, navigate to the “Bots” tab. The tool automatically performs a reverse DNS lookup to verify if the request’s IP address truly belongs to Google. You will see a breakdown of “Verified Bots” and “Spoofed Bots.” For your analysis, only consider the verified bots.

Analyze key data – What is Googlebot actually doing?

This is the core of the entire process. You must now interpret the data to understand the bot’s behavior. Focus on the following reports and metrics:

• Most Crawled URLs (URLs -> All URLs):
• Server Response Codes (Response Codes):
• Crawl Waste:
• Crawl Frequency (Events):

Take concrete actions based on your analysis

The analysis itself is worthless without implementation. Here is a table of common problems and their solutions:

Server log analysis is one of the most powerful techniques in a technical SEO’s toolkit. It allows you to stop guessing and start acting based on hard, undeniable evidence. It shows you where you’re losing money, where Google is encountering problems, and which elements of your site require immediate attention. Dedicate one day to this analysis, and you’ll get a concrete to-do list that will yield far better results than months of “creative” marketing.