.Google's John Mueller addressed a concern concerning why Google.com indexes web pages that are forbidden from crawling by robots.txt and why the it's risk-free to disregard the relevant Browse Console reports regarding those crawls.Crawler Visitor Traffic To Query Guideline URLs.The person inquiring the question chronicled that bots were producing hyperlinks to non-existent inquiry parameter URLs (? q= xyz) to webpages with noindex meta tags that are additionally shut out in robots.txt. What motivated the inquiry is actually that Google is creeping the links to those webpages, acquiring shut out through robots.txt (without watching a noindex robotics meta tag) after that obtaining turned up in Google Look Console as "Indexed, though shut out by robots.txt.".The person talked to the adhering to question:." However right here's the major question: why would Google mark webpages when they can not also observe the web content? What's the benefit in that?".Google's John Mueller confirmed that if they can not crawl the page they can not see the noindex meta tag. He additionally creates a fascinating reference of the site: search operator, urging to disregard the end results due to the fact that the "ordinary" customers won't see those results.He wrote:." Yes, you're right: if our team can't creep the web page, our company can not see the noindex. That stated, if our company can not creep the pages, at that point there is actually certainly not a great deal for us to mark. Therefore while you could view some of those web pages with a targeted website:- question, the normal individual will not find them, so I would not bother it. Noindex is additionally great (without robots.txt disallow), it only indicates the URLs will certainly find yourself being actually crawled (and also wind up in the Browse Console file for crawled/not recorded-- neither of these conditions trigger problems to the rest of the internet site). The important part is actually that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the constraints being used the Website: hunt advanced hunt operator for diagnostic main reasons. One of those factors is because it is actually certainly not linked to the normal hunt mark, it is actually a distinct trait entirely.Google's John Mueller talked about the web site hunt operator in 2021:." The short response is actually that an internet site: inquiry is actually certainly not meant to be complete, nor used for diagnostics reasons.A site question is a details kind of hunt that restricts the end results to a certain website. It is actually essentially just words website, a digestive tract, and afterwards the web site's domain.This query confines the end results to a details site. It's certainly not meant to become a detailed selection of all the pages coming from that website.".2. Noindex tag without utilizing a robots.txt is actually fine for these kinds of situations where a robot is actually connecting to non-existent pages that are getting found by Googlebot.3. URLs along with the noindex tag are going to generate a "crawled/not indexed" entry in Explore Console which those will not possess an adverse impact on the rest of the site.Read through the concern and answer on LinkedIn:.Why would certainly Google.com mark pages when they can't also see the web content?Included Image by Shutterstock/Krakenimages. com.