🐄 John 🐄 on Twitter: "@robdwoods @jroakes @jennyhalasz @jakebohall We never supported crawl-delay in robots.txt -- the numbers given there tend to be unrealistic for the most part." / Twitter
Robots.txt help | How to Create a robots.txt For Your Website