A websites robots.txt file tells search engines what they should and should not index. When the robots.txt file lists a path or file as ‘Disallow,’ then the search engine will not index that location and it will not show up in searches.
The robots.txt for the Bush White House website was nearly 2,400 lines and included things like:
Disallow: /omb/search
Disallow: /expectmore/search
Disallow: /911/heroes/text
On and on for nearly 2,400 lines.
Obama’s robots.txt in its entirety:
User-agent: *
Disallow: /includes/
Thats it, thats the whole thing.
Twitter Updates
Written by lame
Topics: politics