In development, testing and staging mode robot indexing should be turned off, but not in production. So at least two different robots.txt are needed.