Four Oh Four!

The web is a messy place, and people encounter 404 “Page not found” errors from FT websites all the time. This is a page that hasn’t had a lot of love from the technology team over the last few years, but considering that we’ve disappointed the reader by not giving them what they want, we thought we should make more of an effort with the content of the error page.

This is what the old error page looks like:

Old 404 page

The copy on the page is already pretty good. It’s written in plain English, offers some helpful options including the ability to contact a real person for assistance, and apologises to the reader. But the styling is out of date, and while it’s utilitarian, it has no FT personality.

404s

The Telegraph’s in house cartoonist features on their 404 page with a topical illustration. Bloomberg famously has a GIF of a businessman smacking a computer and then disintegrating. The Verge pokes fun at clickbait style headlines, while the BBC rolls out the famous test card.

We worked with Chris Smith in the product team and devised some ideas which we published in a multi-variant test on ft.com

Basics

A good basic 404 message should be clear, focusing on the problem, not the reason for it, or technical codes or jargon. So rather than starting with an apology or the error code ‘404’, we lead with the clear message ‘Page not found’. If we’re going to be quirky, lets save it for after the reader already understands what’s going on.

We also removed references to mistyping the URL – because no-one manually types a URL with a path. Finally, we provide three ‘onward journey’ options: topics, search and home, which satisfy the three main navigational thought processes.

Copy

Now we can start to augment our basic message with some more distinctive colour.

Funny

We ran an internal survey to find the most popular ideas that amused FT staff. The winner is a list of reasons why the page wasn’t found from the perspective of multiple economic schools of thought:

Economic theories

Educational

The FT Lexicon site is an underused resource that defines many useful financial terms and cross references them with FT articles. The 404 page seemed like an ideal opportunity to give readers a quick definition of a financial term that they may not know.

Lexicon

Competitive

As a twist on the Lexicon idea, we made a variant that asks the reader to choose which term is being described by a definition. This also allows us to get an idea of whether this additional 404 page content is interesting – will readers bother to click the buttons to answer the question?

Lexicon challenge

Helpful

Finally, it makes sense to try and get the user back on track by working out what page they were looking for. Correcting typos is not a frequent problem, but instead, looking at URLs that are frequently requested and not found, we find some common patterns:

  • Appending a shortcode to www.ft.com instead of on.ft.com
  • Appending a slash (/) to a URL that doesn’t (yet!) support it
  • Redirect bugs, eg when a URL ends up with multiple /intl/ segments in it
  • Malformed URLs that still contain a valid content UUID

By baking these common patterns into some JavaScript that examines the requested URL, we can easily provide a suggested link to the page the reader probably wanted:

Helpful

Results

We replaced all 404s on the ft.com domain, including our new beta site, with this multi-variant experiment. Across all variants, we found we’re serving around 180,000 404 errors per day. However, there’s a massive bias towards bots, since humans learn from mistakes, whereas dumb robots tend to just keep requesting the same incorrect URL repeatedly. To identify the difference, the 404 page contains JavaScript that sends a beacon back to us. Users that run that JavaScript we’ll consider to be humans, though this might also catch a few particularly advanced bots.

In fact, on average it turns out we serve a mere 4,000 404s per day to JavaScript-enabled user agents, just 2% of the total. That’s pretty good – in comparison to our overall traffic, we’re not letting people down very much.

To see how each of our variants fared, we put a “Do you like this page?” vote bar at the bottom of each variant. Normalising for unique users, here are the results over the first two days (click variants for a live demo – if we’re still running it):

Variant Proportion of YESes Proportion of NOs
Auto-correct 12% 19%
Lexicon definitions 17% 22%
Lexicon game 30% 41%
Economic theories 41% 17%

The actual number of YES and NO votes is relatively unimportant since there’ll be a bias towards NO because on the face of it a 404 error is not what the user wanted, so they’re liable to say they don’t like the page.

But it’s clear that the users really like our economic theories idea – getting more YES votes than any of the others, and fewer NO votes.

Feedback

Some closing thoughts from Chris Smith:

I think we’re in a really good place at the end of the two-week lifespan of this Labs project. To go in that time from nothing to running a multi-variate experiment in live with the goal of improving, for our users, an experience which would normally only be a disappointing dead end is a worthwhile achievement. I’m looking forward to assessing the results and taking the most successful idea forward into production. I was impressed at how flexible Labs was able to be in the structure of the project, and the speed with which we went from early ideas and prototypes to a wide array of possibilities, and finally filtered them down to the final four. Can’t wait to do it again!