The Massive Challenge of Search Engine Complexity

Courtesy Stone Temple Consulting

The complexity of generating Search results online is HUGE.  The What’s So Hard About Search? infographic from SEO expert Eric Enge at Stone Temple Consulting takes a look at the massive numbers involved with indexing and searching the Web.

Building a search engine is a very complex task. I often find myself trying to justify to people why it is that search engines can’t understand their site. They seem fixated on believing that a search engine should understand it if a human can understand it. The short answer is that with an infinite amount of time the search engine could, but the scale of the Internet makes it oh so VERY hard.

The infographic below tries to give you some sense of the scale of the problem. Please note that a few numbers are hard to truly pin down, but I pulled them from the best sources I could. For example, no one really knows how many pages there are on the web, though Majestic SEO is aware of 3.7 trillion (the number I used) or the average web page size.

Regardless, the message is the same either way. The web is a really complex place!

Designed by InfoNewt, the design gathers data from a number of different sources to put together the picture of complexity.  Combining the number of web pages, the average number of links on each web page, the amount of data online and the number of searches every minute, you begin to understand the scale of the challenge search engines face.

This design takes a different approach by citing each data source along with it’s visualization instead of gathering them all at the bottom.  I think it works well with this many different data sources, and is easier for the readers to understand where each part of the data comes from.  All of the key elements are included at the bottom: copyright, brand logo and the URL to the infographic landing page so readers can always find the original (even when blogs repost it without linking or using the embed code).

I just have to say “Cheers!” to the developer teams that tackle this problem every day.  The scope of this challenge will only get bigger in the future!

Thanks to Eric and the team at Stone Temple for being great to work with!