Dignity nobility top quality condition highness standing course caste position superiority precedence manner distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot as to which web pages are not to generally be crawled. To be a search engine https://www.youtube.com/watch?v=MxDLv74CqK0