Robots.txt is a great way of controlling what the search engine’s index about your website. This way you can control that only important information that is beneficial to visitors is index and your SEO is well worth the effort. At Website Designers R Us, our favorite Robots.txt structure looks something like User-agent: * Allow: / […]