Log file analysis

Owned by Brandon Hendry
Nov 07, 2024
2 min read

What is a log file analysis?  

Log file analysis is a way to understand how search engines interact with & crawl a website, offering insights into areas that could improve search engine optimization (SEO) and overall website performance. Here’s a breakdown of what it is and why it’s a valuable research method: 

 

  1. Overview 

Every time a search engine, like Google, visits your site, it leaves a record in a log file. This file logs details about each visit, such as which pages were visited, when, and how often. By analyzing these log files, we can uncover valuable information about how search engines are “crawling” the site & where they may have issues navigating our site.  

  

  1. Why is this important?  

  • Crawl Efficiency: Not all website pages get equal attention from search engines. Analyzing log files helps us understand which pages are being crawled frequently and which ones are being ignored. This insight helps us prioritize critical pages and ensure search engines are focused on the most important content. 

  • Detecting Errors: Errors, like broken links or “404 not found” pages, can hurt SEO if executed incorrectly. Log file analysis shows where these issues are occurring, allowing teams to address them before they impact search rankings. For Nurse, the largest area where 404s occur is our job board with expired job postings.  

  • Page Performance: Slow-loading pages can lead to lower rankings. By examining log files, we can identify pages with performance issues that may deter both search engines and users.  

   

How to review Log Files with Google Search Console

Navigate to Settings below “Security & Manual Actions”

Open the Crawl Stats report.

Here you can see the Total Crawl Requests, Total Download Size, and the Average Response Time.

  • In the Total Crawl Requests report, look for any abnormality in the regular cycle of how many crawl requests were received over the period shown. If you see a large increase or large decrease, this may be cause for further investigation and requesting log files from Engineering.

  • In the Total Download Size report, you can see the total download size of all crawled pages during this time frame. This metric normally closely follows the trend line of the crawl requests. As you can see in the following report, there is a large decrease in both download size and crawl requests, indicating a larger issue with crawling.

image-20241107-211800.png

  • In the Average Response Time report, this displays the average response time from the site to the Googlebot.

    • Ideal response times are around 100 ms

    • A score closer to 1000 ms could mean that Googlebot isn’t able to crawl as much of a site as it ideally would. (John Mueller, Google)