Google and its indexing budget

Google the leading Search engine has two closely inter-related concepts those of its crawl budget and its indexing budget, these determine how many pages of a site Google is prepared to crawl in any given cycle and more importantly how many it is prepared to index.  For small website or blogs it is not any issue as all domains are given at least a few 100 pages which for most sites are fine.  However larger sites such as Article Alley and our own Brasskangaroo site this presents an issue as a site grows then unless the crawl and index budget also increases then only a % of the site will be indexed at any one time.

What determines the crawl and indexing budget for a site?

Google seems to use factors such as incoming links and the length of an article plus user metrics to determine which pages justify being included. So it is important to use your robots file and make use of the noindex tag to exclude pages that are of low importance for example blog category of tag pages are great examples, they are useful for a human user but don’t add value from a search point of view and consume valueable crawl budget.

How to increase a websites crawl budget?

The biggest factor appears to be your sites backlink profile, a large number of on topic and strong links ie ones with a high Moz or ahrefs score means that site enjoys a higher crawl budget, just how high that is depends on how large a site is, clearly if a strong site has a limited number of pages then everything will be indexed, if it has millions of pages and a high Domain Authority score then a high proportion of a sites content will be indexed but not necessarily all content.

If a site is crawl budget limited should I still publish content there?

The answer is clearly yes but just publishing is not enough, you need to build internal links, which can be done by developing followers or comments if the site offers thoses types of features, building external links to your content from your own blog or social media sites such as Reddit or Twitter also seems to help but is not as effective as building internal site links.