Google Said NoMore Support For Robots.txt Noindex

Google officially announced that robots.txt noindex, GoogleBot will no longer hear a Robots.txt directive related to indexing. Google said we’re retiring all code that handles unsupported and unpublished rules (such as Noindex) on September 1, 2019.

Publishers relying on the robots.txt Noindex directive have time until September 1, 2019, to remove it, and begin using an alternatives Google providing to publishers.

If you are just started learning about search engine optimization (SEO). Then it sounds like unfamiliar terms to you but it’s worth knowing about it. Google has in the past supported this robots.txt but this will be no longer the case.

Robots.txt Noindex

Robots.txt Noindex was an unofficial directive. Used to help in Crawling pages, But GoogleBot mostly used to obey that directive, from now it will be no longer in use of indexing purposes.

Today We’re saying goodbye to undocumented and unsupported rule in robots.txt. If you were relying on these rules, learn about your alternative options.

Alternatives To Control Crawling?

Google’s official blog post listed five ways to control indexing. Google never want publishers to keep away from alternatives, so they can easily control their webpage crawling. Here are five alternatives to Google talking about.

1. Noindex in robots meta tags: Supported both in the HTTP response headers and in HTML.
2. 404 and 410 HTTP status codes: Both status codes mean that the page does not exist, this will drop pages from indexing.
3. Password protection: Unless markup is used to indicate subscription or paywalled content, hiding a page behind a login will generally remove it from Google’s index.
4. Disallow in robots.txt: You can use the disallow directive in the robot.txt file to instruct the search engine not to index your site.
5. Search Console Remove URL tool: You can use this tool to remove URL temporarily from the search engine.

Read the official Google announcement here