Pre-submission URL checks are the cheapest part of any indexing workflow. Every URL that is blocked, redirected or broken will fail no matter which indexer you use, so finding them before submission saves money and keeps your reports clean.
1. noindex in HTML or in the headers
A page with noindex asks Google not to index it, and Google respects that. The directive can sit in two places: the meta robots tag in the HTML, or the X-Robots-Tag HTTP header. The header is easy to miss because it is invisible in the browser and in view-source. Google also notes that noindex only works if the page is not blocked by robots.txt; otherwise the crawler never sees the rule.
Paste a list of URLs; the checker reads both the meta robots tag and the response header.
2. Redirects and redirect chains
If the URL you submit redirects, Google indexes the destination, not the URL you paid to promote. Google follows up to 10 redirect hops by default; a 301 is a strong signal that the target should be processed, a 302 a weak one (HTTP status codes in Google Search). For backlinks, a redirect to the homepage usually means the page you bought is gone.
Shows each hop and the final URL, so you can submit the destination or drop the link.
3. 404 and 410 responses
Google doesn't index URLs that return a 4xx status code, and already indexed URLs that start returning 4xx are removed from the index. Dead pages are the single largest group in most backlink lists: in one audit of 15,389 bought links, a third were dead after a year.
Validates status codes for a whole list using a Googlebot user agent.
4. 5xx server errors
Server errors make Google slow its crawling, and URLs that keep returning them are eventually dropped. A page that answers 503 only under load can look fine when you open it and still fail for the crawler. Check from outside, more than once if the server is unstable.
Bulk check of server responses, including Retry-After headers.
A simple order of work
- Export the URL list. If it is a site, the XML sitemap URL extractor turns a sitemap into a plain list.
- Run the status checks first: 404/410 and 5xx remove the obviously broken URLs.
- Trace redirects and replace redirected URLs with their destinations, or drop them.
- Check noindex and robots on what is left.
- Submit only the clean list, then check the index status of each URL later.
A shorter list with fewer failures, and an indexing report that reflects real indexing problems rather than pages that could never be indexed.
What these checks don't tell you
- Whether Google will choose to index a clean page. Open and indexable is not the same as indexed.
- Whether the content is good enough to keep in the index. That is a quality question, not a technical one.
- Whether JavaScript adds a noindex after the page loads. Raw-HTML checks can miss it.
FAQ
Are these checks free?
Yes. The noindex, redirect, 404 and 5xx checkers on SpeedyIndex are free to use.
Should I submit a URL that redirects?
Usually not. Submit the final destination, or drop the URL if the redirect goes to an unrelated page such as the homepage.
Why check X-Robots-Tag separately?
It is sent in the HTTP response header, so it does not appear in the page source. A page can look clean and still be blocked.
Sources
- Google Search Central: Block indexing with noindex
- Google Search Central: HTTP status codes and network errors