Crawl directives archives

One of the most important aspects of SEO is ensuring your content is crawled and indexed by search engines. In fact, without your content in the index, your website won’t be able to generate traffic from search at all. But indexing isn’t always a fast process. There are a lot of factors affecting how fast …

This guide discusses what hreflang is, what it is for and gives in-depth information on how to implement it for your multilingual websites.

Traditionally, you will use a robots.txt file on your server to manage what pages, folders, subdomains, or other content search engines can crawl. But did you know there’s also such a thing as the X-Robots-Tag HTTP header? Here, we’ll discuss the possibilities and how this might be a better option for your blog. Quick recap: …

Indexing is an important part of what a search engine does. Without indexing, all the pages Googlebot crawls don’t have a place to live — and the ranking systems don’t have the input they need to do their work. If Google can’t index your site it can’t appear in the search results. The basics of …

How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. …

We’ve said it way back when, but we’ll repeat it: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result, their site shows up in the search engines anyway. Do you know why it keeps amazing us? Because …

With the meta robots tag you can control what search engine spiders do on your site. Learn what it does right here!

Links are an important part of SEO. Without links, Google (or other search engines) may not discover your pages, or might not think that they’re important. Sometimes, though, you might want Google not to follow a link. Or you might want to tell them a particular is sponsored, or added to your page by a …

Some posts and pages should not show up in search results. To make sure they don’t show up, you should tell search engines to exclude them. You do this with a meta robots noindex tag. Setting a page to noindex makes sure search engines never show it in their results. Here, we’ll explain how easy it …

It can happen to anyone: You’re working on your site, fiddling on some posts here and there, and hit update when you’re done. After a while, you check back on how a post is doing and, to your dismay, it disappeared completely from the search engines! It turns out you’ve accidentally set a post or …