Agent
User-Agent or bot are a piece of software that performs tasks on behalf of a user or another program. Agents can be designed to perform a wide variety of tasks, but they’re usually designed to automate routine processes or gather information. Search Engine use them to “crawl” web looking for new url to discover each day. They almost act like a crazy user online, looking on and on for url.
Bots are User-Agent are a fundamental assets of Search Engine. They help them to discover new content and stay up to date.
All search engine use bots for 2 actions:
- Gather HTML code of the page discovered in order to index it
- Clic on every link they can discover in the page to discover new page and so on and on. Almost like a crazy web user who wants to discover new content and save it on a his directory. This is called crawl and it is use to determine crawl budget.
Every single Search Engine has a name for is bot: Google bot, Bingbot, Yandexbot and so on. For a same Search Engine they can use as many bots as they have search system, like images, video and others.