# All robots will spider the domain User-agent: * Disallow: # Disallow directory site User-agent: * Disallow: site # Disallow directory tools User-agent: * Disallow: tools