If we failed to handle bad 🤖 bots, your business /blog is at risk

How do we protect our site crawling from bad bots

May 19, 2021 | by mishipay

What is a crawler?

A crawler, A.K.A, a bot, is one of the most important factors in a website. Once you create a website, you generally tend to send it to your friends, family, stakeholders and related circles. Due to this, related people get familiarized with your website. As same some crawlers are market giants, so they help in familiarizing your website over the world by introducing it to various search engines.

If no crawlers crawl your website, or if you have manually blocked crawlers from crawling your website, then you may not get any impressions from other sources except direct traffic and impression.

How does it work?

Every search engine has its own bot to crawl websites. (Some of them share bots.) Each bot first comes to your robots.txt file before entering the site. Based on the rules, it is determined if the bot has permission to crawl or not. If the crawler is permitted, then it will enter your website and start crawling. If not, then it will move on to the next site.

Webmasters have 100% control of the crawlers. In this, we can mention the specific directory that we want to be crawled. For example, if you have a copyrighted content on a promotional section on your website, it is better to ban the crawler from crawling that particular section.

Here is how it is done:

/* To block only Google products */
User-agent: Googlebot
Disallow: /promo

/* To block all the crawlers */
User-agent: *
Disallow: /promo

This lets the bot crawl your website but not these sections.

How are Mishipay handling crawlers?

We, at Mishipay, are very clear as to what should be allowed to crawl and what not. Therefore, we have allowed a major bot and have blocked a few bad bots as producing.

What if we failed to protect from bad bots?

It’s simple: if we fail to handle these bad bots, your business /blog is at risk, and indeed, people from outside will run or monitor your site. Do you truly want that?

If our opponents or the opposing party will impose 100s of requests on our domain URL at the same time, the website will be overloaded and the site will immediately go down. Furthermore, this kind of traffic / requests is so detrimental to SEO that it will result in a 100% bounce rate and a session duration of 0 seconds. Once the Google bot regains these stats, it drives our site back to the bottom of the list in terms of authority, scores, and so on.

How Mishipay Handled Bad Bot Traffic?

To be honest, we just got bad bot traffic once, and it was 120+ in real-time with a single location and from many pages and posts, which is really dangerous to the analytics data and server, so we took urgent steps to fix it. How did we keep it under control?

Initially, we modified our robots.txt file to make just a few of them; however, all bots are still banned.

Get the traffic source and blocked list from the .htaccess file, as seen below.

/* To restrict access to a specific bot */
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^http://.*trafficreviews.club/ [NC]
RewriteRule ^(.*)$ – [F,L]

/* If we don’t know the source? then block common bad bots */
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(agent1|Cheesebot|Catall Spider).*$ [NC]
RewriteRule .* – [F,L]

Because the bad bots will come under either agent1 of cheesebot or as a spider so we can eliminate them.

  1. In addition, we added a filter in our GA (Google Analytics) to prevent analytics data abuse.
  2. We have the Cloud Flare service, so we added a portal for bot traffic and set up threshold values to detect bad traffic and bad requests.
  3. As previously said, we have blocked bad bot traffic and requests.

I hope this is useful for your company or website as well.

How to check allowed or disallowed Bots?

Is there a way to tell whether you have approved this particular bot or not? Since we want to ban some bots but, unfortunately, it’s allowed; similarly, we want to allow some bots but it’s blocked by another rule; so, to be transparent, you can check the status on this page. Add your server’s or site’s IP address here so that it can address you the status, and then rework your robots.txt file depending on that status.

Familiar Web crawlers and their user Agent name

Some bots are exclusive

Some of them are exceptional bots, which do not respect your robots.txt file. What are?

Final words

As we all know, bots are just rule followers, not rule breakers, so become mindful of how to handle bots on your web and handle them properly. Get your business to a good start.