Robots.txt can be a file that exists on the root directory of every website and may be used to instruct engines like google on which directories/documents of the web site they can crawl and include things like within their index. Webmaster applications can provide you with loads of useful information https://technik.forum