User-agent: * # disallow all files in these directories Disallow: /generator/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /cron/ User-Agent: * Crawl-Delay: 60 User-Agent: googlebot Crawl-Delay: 2 Disallow: /generator/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /cron/