User-agent: * Allow: / Disallow: /Admin/ Disallow: /bin/ Disallow: /aspnet_client/ User-agent: twiceler disallow: / User-agent: ia_archiver Disallow: /