Create a robots.txt file with allow and disallow rules per user-agent. Add your sitemap, set a crawl delay and download the finished file — free and private.
Need an XML sitemap too? Try Sitemap Generator →