Mediabids Bot Filtering

Back

Mediabids Bot Filtering - Utilizing User Agents

What is a User Agent?

A user agent is like an ID card for web browsers and apps. When you visit a website, your browser sends a user agent string to the site, which tells the site what kind of device and browser you're using. For example, it might say "I'm Chrome on a Windows PC" or "I'm Safari on an iPhone."

Why Filter Bots?

Bots are automated programs that can visit websites just like humans do. Some bots are helpful, like search engine bots that index web pages. However, others can be harmful, such as bots that try to steal data or overload a website with too many requests. Filtering out unwanted bots helps keep a website secure and running smoothly.

For Mediabids digital campaigns based on click-thrus even helpful bots, such as search engine bots, can skew an ad’s response and as such, all bots should be targeted and filtered before redirecting a request to the client website.

To do this, Mediabids reads the user-agent from the request and in addition to looking for common bot tokens in the user agent, it also matches it against a maintained list of known bot agents. This list is a publicly available and maintained list of bot related user agents.

The complete list is available here.

Since new bots come online daily, often not following standard protocols and identifying themselves as a bot, using a cumulative and well-maintained list of bot user agents is a way of ensuring we redirect only legitimate traffic to a client’s website. While no process is 100% effective due to the ever-changing landscape of bots, this process helps us greatly reduce invalid/unwanted traffic.

How Does the Feature Work?

1) Identify User Agents: The feature looks at the user agent string sent by each visitor to the website.

  • Look for known bot tokens: It looks at the user agent and looks for common tokens such as ‘bot’.
  • Compare with Known Bots: It compares the user agent string against a list of known bot user agents. This list is regularly updated to include new bots.

2) Allow or Block Access:

  • If the user agent contains a known bot token or matches a known bot agent, the feature blocks the bot from being redirected to the client site.
  • If the user agent does not match any known bots, the visitor is allowed to access the client’s website.

Benefits:

  • Accurate Analytics: Filtering out bots ensures that website traffic data reflects real human visitors, providing more accurate insights.
  • Improved Security: By blocking bots, the website is less likely to be attacked or overloaded.
  • Better Performance: With fewer bots, the website can run faster and more efficiently for real users.

Conclusion

By identifying and blocking known bots utilizing a well-maintained list of user agents, Mediabids greatly reduces the number of invalid redirects to a client’s site. While no method is 100% accurate, this methodology helps ensure traffic driven by Mediabids remains as clean and as precise as reasonably expected.