.Google.com's John Mueller responded to a question about why Google.com marks pages that are actually disallowed from crawling by robots.txt and also why the it is actually safe to neglect the relevant Search Console files regarding those crawls.Bot Visitor Traffic To Inquiry Criterion URLs.The person talking to the concern recorded that crawlers were producing hyperlinks to non-existent query criterion URLs (? q= xyz) to pages along with noindex meta tags that are actually also shut out in robots.txt. What cued the question is actually that Google is actually creeping the web links to those web pages, getting blocked out through robots.txt (without noticing a noindex robotics meta tag) at that point getting reported in Google.com Search Console as "Indexed, though blocked through robots.txt.".The individual asked the following inquiry:." However right here's the major concern: why would Google.com mark webpages when they can not even observe the information? What is actually the conveniences because?".Google's John Mueller validated that if they can not creep the page they can't find the noindex meta tag. He also creates an appealing mention of the site: search operator, urging to disregard the end results considering that the "ordinary" users won't see those outcomes.He created:." Yes, you are actually right: if our experts can not crawl the page, we can not see the noindex. That claimed, if we can not creep the webpages, then there's not a whole lot for us to index. Therefore while you might view some of those web pages with a targeted website:- concern, the common customer will not observe all of them, so I definitely would not bother it. Noindex is also great (without robots.txt disallow), it simply means the URLs will find yourself being actually crawled (and end up in the Explore Console report for crawled/not indexed-- neither of these conditions lead to concerns to the remainder of the site). The important part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the constraints being used the Website: search progressed hunt driver for analysis main reasons. Among those reasons is actually considering that it's not linked to the frequent search mark, it's a different point entirely.Google.com's John Mueller talked about the internet site hunt operator in 2021:." The quick solution is actually that a website: inquiry is not meant to be full, nor utilized for diagnostics objectives.A site query is a particular type of search that confines the end results to a certain internet site. It is actually essentially only words web site, a digestive tract, and then the website's domain name.This concern limits the outcomes to a certain website. It's not suggested to become a thorough collection of all the pages coming from that site.".2. Noindex tag without using a robots.txt is fine for these type of situations where a bot is connecting to non-existent web pages that are acquiring uncovered through Googlebot.3. URLs with the noindex tag are going to produce a "crawled/not recorded" entry in Browse Console and that those won't possess an unfavorable impact on the remainder of the internet site.Read through the inquiry as well as address on LinkedIn:.Why would certainly Google mark pages when they can not also find the information?Included Image through Shutterstock/Krakenimages. com.