Updated October 2026. I first wrote this guide in 2016. The process still works. I’ve updated the tools (Screaming Frog, Google Search Console and GA4 all work differently now) and added what Google says about the lastmod date.
On larger websites, auditing the XML sitemap often uncovers lost traffic and gives search engines a more accurate, trustworthy list of your pages. Fixing sitemap issues can mean thousands of extra visits and a real revenue impact. It’s an easy exercise for anyone comfortable with a crawler and a spreadsheet, and the process below works on almost any site.
Why an Accurate XML Sitemap Matters
Years ago, Duane Forrester (then with Bing Webmaster Tools) put it bluntly in a Moz Whiteboard Friday:
…if you start showing me 301s in here, rel=canonicals, 404 errors, all of that, I’m going to start distrusting your sitemap and I’m just not going to bother with it anymore.
Google says something similar about the lastmod date in its sitemap documentation: it uses lastmod only if it’s consistently accurate. If every URL shows today’s date every time the sitemap is generated, Google learns to ignore it.
Your sitemap should only list pages you want indexed: pages that return a 200 status, have a self-referencing canonical and aren’t set to noindex. An audit usually turns up URLs like these that are holding your site back:
- 3XX redirected URLs (301, 302, etc.)
- 4XX client errors (404, 410, etc.)
- 5XX server errors (500, 502, etc.)
- URLs with a “noindex” meta robots tag or X-Robots-Tag
- URLs that canonicalize to a different URL
- URLs blocked by robots.txt
What You’ll Need
- Screaming Frog (or a similar crawler)
- Google Sheets or Microsoft Excel
- Google Search Console and Google Analytics 4
How to Audit an XML Sitemap
Step 1: Check Search Console First
In Google Search Console, go to Indexing > Sitemaps. Check that your sitemap was read successfully and click into it to see how many of its URLs are indexed. Then open the Page Indexing report and filter it to that sitemap. Any reason listed under “Why pages aren’t indexed” (redirects, noindex, duplicates and so on) is a sitemap URL that probably shouldn’t be there.
Step 2: Crawl Your XML Sitemap with Screaming Frog
Switch Screaming Frog to List mode, click “Upload” and choose “Download XML Sitemap” (or “Download XML Sitemap Index” if you have more than one). Paste in your sitemap URL and Screaming Frog will pull in every URL and crawl them.
Before you start, connect the Google Analytics 4 and Search Console APIs under Configuration > API Access. That pulls traffic and click data for each URL into the crawl, which saves you a step later. On very large sites it can slow the crawl down, so skip it if speed matters more.
If you have thousands of URLs, go get a cup of coffee or work on something else while it runs.
Tip: Screaming Frog can also find the opposite problem. Run a normal crawl of your site with your sitemap added (under Configuration > Spider > Crawl), then use Crawl Analysis. It flags indexable pages missing from your sitemap and sitemap URLs your crawl couldn’t find through links (orphaned pages).
Step 3: Export the Crawl Data
Export the Internal (All) report. It has far more columns than you need. Keep these:
- Address (URL)
- Status Code
- Indexability and Indexability Status
- Meta Robots and X-Robots-Tag
- Canonical Link Element
- Redirect URL
- GA4 sessions and revenue, and Search Console clicks (if you connected the APIs)
I like Google Sheets for this, but Excel handles sitemaps with tens of thousands of URLs better.
Step 4: Label the Issues
Add an “Issue” column and label each problem URL: redirect, 404, noindex, canonicalized, blocked and so on. Here’s a simple example.
Step 5: Add Traffic and Revenue Data
If you didn’t connect the APIs in Step 2, export organic landing page data from GA4 (sessions and revenue or key events) and clicks from the Search Console Performance report. I like to use 12 months of data. Pull them into new tabs and use XLOOKUP or VLOOKUP to match the numbers to each URL.
Sort by traffic so the URLs with the most to lose are at the top. Then look for opportunities like these:
- Restore pages that should return a 200 status but are redirecting or erroring out
- Fix canonical tags that point to the wrong URL
- Fix accidental noindex tags
- Replace redirected URLs with their final destination
- Remove URLs that shouldn’t be in the sitemap at all
- Fix lastmod dates that don’t reflect real content changes
On a free 30-minute call, we’ll talk through your site and goals.
Bonus Step: Total the Impact by Issue Type
It helps to see the traffic and revenue tied to each issue type at a glance, whether you’re reporting to a client, upper management or just prioritizing your own fixes. Create a new tab like this:
Here are the Google Sheets formulas, assuming your crawl data is in a tab called “All Sitemap URLs” with the issue labels in column B, sessions in column G and revenue in column H.
Number of URLs
=COUNTIF('All Sitemap URLs'!$B:$B,$A2)
Organic Sessions at Risk
=SUMIF('All Sitemap URLs'!$B:$B,$A2,'All Sitemap URLs'!G:G)
Organic Revenue at Risk
=SUMIF('All Sitemap URLs'!$B:$B,$A2,'All Sitemap URLs'!H:H)
Taking Action on Your Data
Knowing what’s wrong with your sitemap doesn’t change anything until someone fixes it. Most development teams are buried in company-wide priorities, which is where the summary tab helps. It shows the traffic and revenue tied to each issue, so you can make the case for getting it fixed.
SEOs like perfection, but development work gets prioritized by business impact. The more clearly you show the impact on traffic and revenue, the faster your sitemap issues get fixed.
If your sitemap is generated by a plugin like Yoast or by your eCommerce platform, many of these fixes happen at the source: fix the redirect, noindex or canonical on the page, and the sitemap updates itself.
Questions? Contact me, and I’ll be happy to answer them. And if you’d like help, a sitemap review is part of every technical SEO audit I do.
