How to fix robots.txt, noindex and sitemap problems

robots.txt

robots.txt lives at the root of the domain (https://example.com/robots.txt). A safe default that allows everything and points at your sitemap:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

• Look for Disallow: / under User-agent: *. It blocks the whole site for every crawler that does not have a group of its own. Remove it unless the site really should be invisible.
• Do not block CSS, JavaScript or images that pages need to render. Search engines render pages much like a browser, and a page they cannot render properly is judged on what is left.
• robots.txt controls crawling, not indexing. A blocked page can still appear in search results, without a description, if other sites link to it. To keep a page out of results, use noindex and let it be crawled.

If you have no robots.txt, crawlers treat that as permission to fetch everything, but it is still worth adding one so you can point to your sitemap. Make sure /robots.txt answers with plain text: on single-page apps the app often answers every address, including this one, with its HTML page. Put the file in the static or public folder so the server returns it directly.

Where to edit it: on a static host, it is a file in the site's root folder. WordPress generates one; SEO plugins such as Yoast SEO and Rank Math have an editor for it. Shopify uses a robots.txt.liquid theme template. Wix has a Robots.txt Editor in its SEO dashboard. Squarespace manages the file for you: you cannot edit it, but its crawler settings change what it says. Each subdomain needs its own robots.txt.

noindex

A page is kept out of search results by either of these:

<meta name="robots" content="noindex">
X-Robots-Tag: noindex

The first is a tag in the page's <head>; search the page source for noindex. The second is an HTTP response header; check it with curl -sI https://example.com/. Common sources of a noindex nobody meant to keep:

• WordPress's Settings → Reading → Search engine visibility box ("Discourage search engines from indexing this site"), left ticked after launch. It also switches off WordPress's built-in sitemap.
• A per-page or per-post-type setting in an SEO plugin.
• Staging configuration copied to production, or a host or CDN rule meant for preview deployments.

Remove it from every page you want found, then use URL Inspection in Google Search Console to request indexing of the most important ones.

Do not block and noindex the same page

If robots.txt blocks a page, crawlers never fetch it, so they never see its noindex tag, and the page can stay in results. Pick one: noindex (and allow crawling) to keep a page out of results, or a robots.txt rule to stop crawling of areas that do not matter, such as internal search results.

XML sitemap

• List only pages you want indexed: the canonical address of each page, answering 200. No redirects, no 404s, no noindex pages.
• Use full addresses on your live domain, with the same https and www choice as the rest of the site.
• Stay within the limits: 50,000 addresses or 50 MB uncompressed per file. Larger sites use a sitemap index that lists several sitemaps.
• Keep lastmod honest. Set it to the date the page last changed in a meaningful way, not the date the sitemap was rebuilt. Google only uses lastmod when it is consistently accurate.

Most platforms generate the sitemap for you: WordPress at /wp-sitemap.xml (or /sitemap_index.xml with Yoast SEO), and Shopify, Squarespace and Wix at /sitemap.xml. Reference it from robots.txt with a Sitemap: line and submit it under Sitemaps in Google Search Console.

Check it worked

Run WEBCHKR again. The Crawling & Indexing check confirms that robots.txt exists and is not an HTML page (an app answering every address), and it fails outright if your home page carries a noindex in a robots meta tag or an X-Robots-Tag header. It does not judge the rules inside robots.txt, so read them yourself for a stray Disallow: /. The XML Sitemap check looks for the sitemaps listed in robots.txt, then in the usual locations, and reads the start of the file it finds to confirm it is a sitemap rather than an HTML page. It does not validate every entry; Search Console's Sitemaps report does.