Google search advocate John Mueller has always downplayed the value of HTML sitemaps. Recently, he reiterated his dissatisfaction with HTML sitemaps, saying:
“I changed my mind on HTML sitemaps over the years, they should never be needed. Sites small [and] large should always have a clear navigational structure.”
This is a slightly stronger stance than what he’s expressed in the past:
“When it comes to SEO … for small sites, your site should be crawlable anyway (and if you’re using a common CMS, it’ll almost always be fine) & for large sites, they’re not going to be useful anyway (use sitemaps, use normal cross-linking, check with a crawler of your choice).”
While Mueller makes a good point, in my opinion, we can’t just completely drop HTML sitemaps. Links and website usability still matter.
I also like what former Googler and now popular SEO expert Pedro Dias said in response to Mueller’s remarks, which reflects the reality of the web:
“Unfortunately it’s not easy to change the architecture on complex and large websites. Often, something like this, [sic!] can take years to change, as it depends on multiple teams. So, HTML sitemaps are often a welcome and temporary-ish solution. I used to dislike them… Nowadays, I see their use-cases.”
There is a compromise between both HTML and XML sitemaps, so as always in SEO, “it depends.”
Read below why, when and how to use sitemaps to benefit your website and its users.
What are sitemaps and why did they get invented?
In the early days of the web, many websites grew organically.
Websites weren’t appropriately planned by information architects according to UX best practices.
They started from scratch, and webmasters added content whenever needed in often far-from-usable ways.
Those were the days before blogs, social media or content management systems. When you built a site, you did it yourself using HTML.
In most cases, you were manually coding the HTML in a text editor, using a messy WYSIWYG editor like Dreamweaver or one of the first often custom-built and pretty cumbersome CMS tools. Either way, website structures quickly became confusing.
So out of necessity, people started adding sitemaps to their websites to make some sense of the increasing disorder. Those sitemaps were created in plain HTML and looked like any other webpage, part of the fact that they displayed a map instead of written content.
Later search engines introduced so-called XML sitemaps meant to replace manual “search engine submission.”
More than 25 years later, HTML sitemaps are less preferred. But can we really stop using HTML sitemaps, just like we dropped guest books or webrings?
HTML vs. XML sitemaps – the visible vs. the invisible
To make it short and simple, the main difference between HTML and XML sitemaps is that an HTML sitemap is usually visible to website visitors while an XML sitemap is not. (The latter primarily made for search engines.)
The XML sitemaps standard was introduced at the end of 2007 by search engines like Google, Yahoo and Bing (then MSN).
By default, they are formatted in a way that does not display well or is readable for humans. Instead, such sitemaps are used mainly to ensure proper and faster indexing by search engines.
On the other hand, just because a resource is on an XML sitemap does not mean Google and Bing will index it, or it will be findable even if it’s in the index.
Another SEO colleague Rob Watts added a tongue-in-cheek, first-hand account to the debate by saying:
“I had sites do well for years that were absent of html or xml sitemaps. I removed my tin foil hat and began using xml sitemaps and enjoyed seeing how long it took for urls to appear in serps. I don’t think I saw any noticeable improvement performance wise for their use, but it made some analyses/reports slightly easier.”
So with XML sitemaps only theoretically ensuring crawling and indexing, do we have to focus on them and neglect visible or visual sitemaps?
Why should we create sitemaps solely for Google?
Over the years, Google spokespeople told us to create websites for users, not search engines. Why make an exception when it comes to sitemaps and only create them for Google and other search engines?
It makes me wonder whether other SEO tactics that display one thing for search bots and something different for website visitors may also be a good idea.
Anyway, Google said so, so we must listen, right? Not really.
There must be a reason why Google dislikes HTML sitemaps and prefers XML.
XML sitemaps are usually automated and include all the content published as soon as it’s online.
On WordPress, numerous tools create sitemaps for you. Yoast SEO also covers you, for example, even the basic free version.
In 2020, WordPress finally added XML sitemap support to its core, so you won’t even need external plugins.
The XML code format is also machine-readable, while HTML is messy. HTML has to work despite errors, while XML breaks when you forget a character.
So at the end of the day, the XML sitemap creation works mostly behind the scenes without error-prone human hands interfering.
Also, most people will never see your XML sitemap, they wouldn’t even know where to find it as it’s not linked anywhere visibly on the site.
Making XML sitemaps work for people
Luckily, there is always some middle ground. You don’t have to make XML sitemaps invisible.
You can format XML sitemaps with stylesheets just like you do with any HTML pages.
This technique has been working as long as XML has been around. There are tutorials out there that help you with that.
Indeed for a while, it seemed that XML would replace HTML as the web standard, or it was a compromise called XHTML.
For compatibility reasons, that didn’t happen. Even though the XHTML format was cleaner and less error-prone. When an error happened, the website did break.
With HTML5 introduced instead, browsers embraced sloppily coded websites to make publishing easier but at the expense of automatic crawling.
Now you have ideally created an XML sitemap that can be read by people without coding skills. Remember that it’s still a bland list of all the content you have.
You may still consider creating a readable HTML sitemap for your website visitors.
When to use visual or HTML sitemaps
It’s not either/or. You can:
Decide to separate both visible and invisible sitemaps instead of making your XML sitemap visible to people.
Create both visible (HTML) and invisible (XML) sitemaps on your site for different reasons.
Have more than one HTML or XML sitemap.
It depends on what your site looks like and what your objectives are.
Use these questions as a guide to making a decision:
Do you want to create a quick overview page of your site? Use an HTML sitemap!
Do you want to get indexed as soon and as deeply as possible? Create an XML sitemap.
That said, there are edge cases, and many websites do not conform to typical standards.
Here are example use cases for visual or HTML sitemaps.
Legacy and/or hard-coded sites
Believe it or not, there are still legacy sites from decades ago on the web, and even hard-coded sites made manually in HTML, CSS and/or JavaScript.
In fact, you’ve probably used one quite often.
Amazon.com is almost the same site it was many years ago. It has been continuously refreshed over the years and improved to this day.
An all-out redesign would have been too risky and disruptive for the bottom line.
Some basic legacy sites may have difficulty creating an XML sitemap on the fly. So it’s not a problem when they have a hard-coded HTML sitemap instead.
Do you have to remove it because Google says so? Of course not.
Large sites
When Mueller said that you needed proper site navigation, he was generally right. But it does not always apply to larger sites.
You can’t possibly fit a highly complex site with thousands of pages into a menu without it quickly spiraling out of control.
For better findability, it’s useful for both humans and search bots to have sitemaps that reflect the overall site structure.
You don’t have to list every page, as that would make it too large.
Alternative to mega menus
Many sites attempt to fit as much as possible into their site navigation and end up having so-called mega menus.
You hover your mouse over the navigation, and the menu suddenly covers the whole site.
Users get quickly overwhelmed visually with dozens of options to select from.
Also, sometimes, if a user is not fast enough to point at the right menu item, it will disappear when they lose the level one menu.
Mega menus are tempting, especially with large companies where every stakeholder wants to be on top.
But they are often a user experience nightmare, as Smashing Magazine founder Vitaly Friedman clearly demonstrates.
Sites without internal search
When doing web outreach, you might notice that many sites make it quite hard to contact them.
Such is often the case with sites having no internal search feature. You will often struggle to locate their contact page.
A search feature would be a logical solution, yet surprisingly some sites do not have one, even those run on WordPress, which has built-in functionality.
Some themes or website owners simply remove the search input.
When I can’t find a standard page in your menu and there is no search feature available,a visual sitemap would be nice.
Orphan pages
When you manage to create a menu that is usable without becoming so huge that it does not fit some screens anymore, you will often end up with orphan pages.
Orphan pages are webpages that are not linked internally from the menu structure or inside the content.
This happens when content gets removed and updated, and some links disappear.
Oftentimes, orphan pages should be removed altogether.
Others still make sense but might not be as significant to place into the main menu or the footer links.
Such orphan pages would be fine when linked in a visible, readable sitemap so that people looking for them can find and access them.
Faster indexing
Let’s be honest! Just because a page or other resource (think image or video) is listed on an XML sitemap does not mean Google will index it.
Your site may not have enough authority to get all its content into the Google index.
This was quite a common issue for new sites in the past. Especially with newly created sites and Google having more limited computing power, some content did not make it into the index on purpose.
It’s less of an issue now, but you still may want to link content internally to boost your chances of getting indexing.
Link to the more important content on top, prominently, and/or more often.
Better link equity
When we talk about authority, we refer to the link equity that some SEOs still call “link juice.”
An HTML sitemap that shows some links on top and others at the bottom regulates link equity flow to some extent.
Pages linked on top are expected to have and receive more authority than pages at the bottom.
With XML sitemaps, there is no such hierarchy, as far as I know. All resources listed on it are just that, items in a list.
The item on top is merely the latest one, not necessarily the most important one or the one needing the highest authority.
There is no inherent hierarchy on XML sitemaps unless you code one into it by declaring that some resources are more important and, thus, should be crawled more often than others.
A visible HTML sitemap allows you to link those resources on top that matter most and thus send more users and authority their way, just like in your site navigation. So it’s not “latest” on top but most important on top, ideally.
Usability
As we are talking SEO here, the most important aspect of sitemaps, improving usability, comes at the end.
This is Search Engine Land and not UX Collective, after all. They advocate creating an (internal) UX sitemap for every website project, at least while establishing the information architecture.
HTML sitemaps improve the overall user experience, especially:
Accessibility.
Discoverability.
Findability.
Usability.
These are still the strongest use cases for HTML sitemaps, just like in the ‘90s, way before Google started using links for indexing and ranking and long before XML sitemaps were introduced.
Older users (I admit I’m among them) may even prefer clear and page-style sitemaps instead of shaky menus that vanish when you don’t touch them or search features showing irrelevant items.
Some elderly people cannot use mouse-over and mouse-click actions efficiently.
People with disabilities, particularly the visually impaired, need as few mouse actions as possible. Thus, a one-page overview might be best for them.
Useful sitemap examples in the wild
The New York Times
I often use the New York Times as an example because almost everybody knows the publication, and it’s such an authority.
This time I want to show the NYT as an example simply because they managed to do something right and stand out.
The NYT uses visible HTML sitemaps in a way that is really helpful for readers, search users and search engine bots alike.
The NYT has several sitemaps highlighted by fellow SEO Chris Long of Go Fish Digital on LinkedIn. It is a prime example of using sitemaps to benefit a website.
“60 Second SEO: The New York Times has one of the best HTML sitemaps I’ve seen. Here’s an example of how an article from 1912 is only 5 steps away from [their] home page:”
I don’t want to go too much into detail here, as Long has done an excellent job already at it.
Apple.com
Another great example of sitemap usage for a large site is Apple.com. Their plain and simple text sitemap is useful for people and search engines.
However, note how much the sitemap differs from the actual top navigation the Apple site has above the content.
The “Apple sitemap” starts with “About Apple” (not linked) and the following links:
The site navigation starts with the shop and specific products in contrast:
Apparently, the SEO team decided to make the shop and products more visible to website visitors and search bots. At the same time, the true structure or importance of the pages seems to be completely different.
In any case, the huge Apple sitemap gives users a good overview of the site contents without overloading the page despite more than a hundred links.
Moreover, they have another sitemap in the footer reflecting the top menu structure instead of the standalone “sitemap” on the dedicated page.
It is smaller both by the font size and by the number of links listed but still very useful.
So no matter where you end up, you always have a site navigation or sitemap to help you.
An “everything is easy to find” user experience is central to the Apple website design.
The post When and how to use HTML sitemaps for SEO and UX appeared first on Search Engine Land.
