Most of you might have not even heard about these terminology. So, lets first get on to understand what is crawl budget and proceed further on various of crawl budget and crawl rate.
What is Crawl Rate?
The rate or frequency at which bots visit your website for crawling is known as Crawl rate.
What's the definition of crawl budget in SEO?
Crawl budget is the number of times a search engine bot known as spider crawls your web page in a given or allotted time. The phrase 'Crawl budget' is coined by the SEO network.
For example, if a spider hits your web page 51 times per day then your sites crawl budget is 1530 per month.
You can see your sites crawl budget in your website search console and scroll to the crawl stats section under crawl and notice your websites average crawl budget that is average number of pages crawled per day.
Why should you care about crawl budget?
Although crawling is mentioned as not a ranking factor but crawl budget has its importance in SEO. Don't you want your website been crawled immediately after the content of the site is updated; and get the updated pages or newly added ones indexed? This is why you need to care about crawl budget. Because if you wont pay necessary attention to the crawl budget then bots won't crawl your site or page that is to be crawled, there won't be chance for the most relevant visitors to be brought to the site.
What is the crawl budget for my website?
Google serves you with the best and most appropriate information about your sites crawl budget via Google search console. The value marked in the above image is your crawl budget per day, which means 32 pages of your site are being crawled per day. So for 30days if this crawl budget remains same then it would turn to 30X32=960 pages per month are being crawled in your website.
How do you optimize your crawl budget?
Optimizing crawl budget is saving on wasting. Fixing the issues that dont waste crawling leads to improved and useful crawling and thus improved indexation. Most common issues that many websites see include but not limited to and if fixed would improve crawl budget are:
- URLs with parameters like color, size, etc.,
- Duplicate content like pages with tags, search pages in the website
- Poor content quality
- Broken links
- Avoid including non-indexable or non-pages like redirection pages in the sitemap.
- Make sure your site don't have pages with poor load time or those that never load
- Don't maintain too many pages that are not indexable
- Maintaining good internal linking structure.
How do you increase your website's crawl budget?
Crawl budget although not mentioned directly., has a good noticeable impact on the page authority. So to improve page authority work on gaining trust worthy and genuine external links and the rest play of crawling will be taken by the bots. Its proved from Yauhen Khutarniuk’s experiment that there is a very close correlation between external links and the number of times spiders visit your website. Check the below image of his experiment.
How does Google’s crawl budget work?
Google has a simple crawling structure that's put in the form of the below image.
There are few components of a page that Google considers for Crawling, few of which are listed below:
- Depth of page
- Page rank of the website
- Internal page rank
- What type of document is it; PDF or HTML or TXT or anything else. PDF type has good crawl rate.
- Internal links quantity
- If included in the sitemap
- Importance and / or quality of the Anchor's
- Content quality
- Websites home page importance
Why limit Crawling and How to limit Crawling?
Over crawling is not suggested as it has a direct on your servers response. So always maintain and limit your sites crawling to the best. With that said here is how you can limit Google's crawling of your site.
- On the Google's Search Console, go to the site that you are about to work on.
- Click on the gear icon and look for site settings option and the click on it.
- Go to crawl rate section and chose the option you want to maintain a desired crawl rate.
This newly custom set crawl rate is valid for 90 days only. For sites with a "calculated as optimal" crawl rate then you have to raise a special request by filling a form available at https://www.google.com/webmasters/tools/googlebot-report.
No comments:
Post a Comment
Your thoughts and suggestions are valuable for me.