You should definitely Have a very clear robots.txt file so engines like google determine what to have a look at. Use XML sitemaps to tell them about all of your significant internet pages. Look for mistakes like Meta Tags that say NOINDEX or Erroneous Guidance inside the robots.txt file that can preserve search bots away.Got any queries to talk to?