Make your site easier to crawl for search engines
Do away with unnecessary resources and URLs with the crawl settings
Available in Yoast SEO Free and Yoast SEO Premium
Make your site green and search engines happy
We’re on a mission to reduce website crawl load — making it a more straightforward process for crawlers. As a result, we’re reducing the strain on your website and the environment. With the crawl settings in Yoast SEO, these two things come together. By selecting which resources and URLs are automatically added by WordPress to discard, you make your site easier to crawl and increase the chance that Google will crawl the URLs you want.
Control how search engines crawl your site
- Makes your site more sustainable
- Block crawling useless URLs and assets
- Prevent spam attacks on your site
Why do you need the crawl optimization settings in Yoast SEO?
Getting every possible URL on your site crawled, indexed, and ranked by a search engine is nearly impossible. Every crawler has a limit to what it can do at any given time, so it’s best to focus on getting your most important URLs crawled. Unfortunately, a CMS like WordPress comes with a lot of baggage: URLs that it automatically adds to your site, unnecessary scripts, useless metadata, and much more. What if you could throw this all out?
Unlock crawl settings in Yoast SEO Premium
Get this feature and much more for your WordPress site with the Yoast SEO Premium plugin!
Making it easier to get your essential URLs crawled
Crawling processes are very resource-intensive, and with a never-ending stream of new content to discover, search engines like Google have to be selective in what they can do. This means that search engines cannot come by all the time. So if they come, you better offer the most important URLs to them — the ones you want them to crawl! The crawl settings let you turn off certain types of URLs, scripts, and metadata that WordPress automatically adds to enhance crawling.
Making your site more sustainable
Crawling costs search engines a tremendous amount of energy. Search engines like Google operate tens of data centers, helping them to crawl, index, and rank information from around the world. All of these use incredible power to function, but the same goes for your site. The server you host your site on uses power, and every request to your site costs energy. So, to make the whole process more power-efficient, use the crawl settings to turn off stuff that has no reason to be crawled. Together, we can make your site — and the web — a lot greener!
Crawl settings help clean up your site
You can see the crawl settings as a way to easily clean up your site of unwanted overhead. For instance, you’ll also find options to clean up the internal site search of your site to prevent SEO spam attacks. And a way to clean up your permalinks by filtering campaign tracking and unregistered URL parameters, which leads to fewer duplicates found by search engines.
Let Yoast SEO help you win the search rankings
Discover all the features of the Yoast SEO for WordPress plugin.