Updated October 2026. I first wrote this post in 2016. Google Search Console has changed a lot since then, so this version walks through the current Page Indexing report and adds a few newer causes I see on client sites.
Getting (and keeping) your pages indexed in Google can be a challenge, especially for new sites and sites with technical SEO or content quality issues. Sometimes it’s a quick fix. Other times you have to dig deeper to find the real cause.
Indexing matters for AI search too. Google’s AI Overviews and AI Mode pull from pages in Google’s index, and Microsoft Copilot leans on Bing’s. If a page isn’t indexed, it’s very unlikely to be cited.
How to Check If Google Has Indexed Your Pages
- Use the URL Inspection tool in Google Search Console. Paste in any URL from your site. It tells you whether the page is indexed, which canonical Google chose, when it last crawled the page and whether anything blocked it. This is the most reliable check for a single page.
- Check the Page Indexing report. In Search Console, go to Indexing > Pages. It shows how many of your pages are indexed and groups the rest by the reason they aren’t. I walk through the most common reasons below.
- Check your Sitemaps report. Under Indexing > Sitemaps, click into a sitemap to see how many of its URLs are indexed. If a big share of your sitemap URLs aren’t indexed, that’s a sign of a bigger problem.
- Use a site: search for a quick look. Search site:kernmedia.com (with your domain) to see a sample of indexed pages. It’s not a complete or exact count, so use it as a spot check, not a measurement.
Reading the Page Indexing Report
These are the statuses I see most often on client sites, and what they usually mean.
- Crawled – currently not indexed: Google saw the page and decided not to index it (at least for now). This usually points to thin, duplicate or low-value content. Improve the page, merge it with a stronger one, or remove it.
- Discovered – currently not indexed: Google knows the URL exists but hasn’t crawled it yet. On large sites this often means Google is spending its time on other URLs (see index bloat), or your server is slow.
- Duplicate without user-selected canonical: Google thinks the page duplicates another page and you haven’t told it which one to use. Add a canonical tag.
- Duplicate, Google chose different canonical than user: You named a canonical, but Google picked another page. Check for near-identical content or conflicting signals (like internal links and sitemaps pointing to the “wrong” version).
- Alternate page with proper canonical tag: Usually fine. Google is respecting your canonical tag.
- Excluded by ‘noindex’ tag: The page tells Google not to index it. Fine if it’s on purpose. A problem if it isn’t.
- Blocked by robots.txt: Google isn’t allowed to crawl the page.
- Not found (404), Soft 404, Server error (5xx) and Page with redirect: The URL isn’t returning a working page.
On a free 30-minute call, we’ll talk through your site and goals.
10 Common Reasons Google Isn’t Indexing Your Pages
1. The Page Doesn’t Return a 200 (OK) Response
If a page doesn’t return a 200 (OK) status code, don’t expect Google to index it (or keep it indexed). Pages can get redirected by accident or start returning 404 or 500 errors after a CMS change, a server issue or a simple mistake. Check the URL in the URL Inspection tool, crawl your site with Screaming Frog, or run it through a status checker like HTTPStatus.io.
2. Blocked by Robots.txt
Your robots.txt file tells Google which parts of your site it can crawl. If a page is missing from Google, this is one of the first places to check. Keep in mind that robots.txt blocks crawling, not indexing, so a blocked page can still show up in results without a description. See my guide on how to write a robots.txt file for more.
3. A “Noindex” Meta Robots Tag
A noindex tag in the <head> of a page tells Google not to index it, and Google follows it. It can show up as noindex, noindex, follow or noindex, nofollow. I see accidental noindex tags most often after a site launch, when a “discourage search engines” setting was left on from the staging site.
<meta name="robots" content="noindex, follow" />
View the page source to check. If your site uses JavaScript to build the <head>, use the URL Inspection tool’s rendered HTML instead, since the tag might be added (or removed) after the page loads.
4. A “Noindex” X-Robots-Tag
An X-Robots-Tag does the same job as a meta robots tag, but it’s sent in the HTTP header. It’s most common on PDFs and other files that don’t have a <head>. It’s rarely added by accident, but server or CDN rules can apply it more broadly than intended. The URL Inspection tool or your browser’s developer tools (Network tab) will show it.
X-Robots-Tag: noindex
5. Internal Duplicate Content
If a page is very similar to another page on your site, Google may index only one of them. This is common with product variants, filtered category URLs, tag pages and location pages that only swap out a city name. Crawl your site with Screaming Frog (its near-duplicate report helps here) or Siteliner, then consolidate, canonicalize or rewrite the duplicates.
6. External Duplicate Content
Content copied from (or onto) other websites is a sign of low value to Google. The most common example I see is eCommerce sites using manufacturer product descriptions that dozens of other retailers also use. Big brands with strong authority and lots of reviews can still rank with them. Smaller sites often can’t.
To check, put a sentence from your page in quotes and search Google, or run your URLs through Copyscape.
7. The Page Doesn’t Add Enough Value
It’s also possible that a page (or your whole site) could suck so bad that it doesn’t add enough value to Google’s index. Thin affiliate pages, auto-generated pages and mass-produced AI content with nothing new to offer are common examples. Google’s guidance on helpful content is a good gut check: does the page offer something that isn’t already available on other sites?
8. Your Site Is New and Unproven
New sites don’t get indexed and ranked overnight. Google needs links, mentions and other signals before it crawls and indexes a site deeply. Submit an XML sitemap, link to your key pages from your homepage and navigation, and work on earning mentions and links from relevant sites.
9. Slow or Unreliable Servers
If your server is slow or throws errors, Google crawls less. On large sites, that can leave a lot of pages stuck in “Discovered – currently not indexed.” Check the Crawl Stats report (under Settings in Search Console) for average response time and server errors.
10. Orphaned Pages
If no other page on your site links to a page, Google may never find it (or may decide it isn’t important). These are called orphaned pages. Crawl your site with Screaming Frog and compare the crawl against your XML sitemap and analytics data. Any URL in your sitemap that the crawl didn’t find is probably orphaned.
Bonus: Content That Depends on JavaScript
Google can render JavaScript, but it’s an extra step and doesn’t always go smoothly. If your main content, links or product details only appear after JavaScript runs, Google may index a thinner version of the page. And most AI crawlers (like GPTBot, ClaudeBot and PerplexityBot) don’t run JavaScript at all, so they may see a nearly empty page. Compare your page’s raw HTML to the rendered version in the URL Inspection tool to check.
Don’t Forget Bing
Bing powers Microsoft Copilot and feeds several other AI tools, so check Bing Webmaster Tools too. It has its own URL Inspection and indexing reports, and it supports IndexNow, which lets you notify Bing right away when pages are added or changed.
When in Doubt, Ask for Help
For some people, this stuff is simply too technical, and it’s best to get help from an SEO specialist…like me 🙂 Figure out how valuable your time is before you spend late nights on it. And remember that indexing is only the first step. Once your pages are indexed, content quality, links and brand signals decide how well they rank in Google and how often AI tools cite them.
If you’re stuck, request a free consultation.
