FT readers often ask where they can find an article they read in the paper as an online reference, for further perusal or to share it. InkLink was born from an intent to create a new hybrid and enhanced user experience for our readership; relying both on the paper and the online version of the FT.
Picture this: you are at home, reading your favourite financial newspaper with your morning coffee, spreading pages over your kitchen table. It is nigh impossible for you to take those pages along for your commute. What if you could snap the main image with your smartphone and have the online equivalent to browse on the go? This is one of the aims we set for InkLink.
Augmented Reality (AR) in newspapers
Back in 2011, long before Pokémon Go, there was a resurgence in AR applications, especially with advertisers. Most newspapers, even the FT, urged their readers to scan their adverts and see them come to life. If this innovative way of consuming content was fun for a short while, the novelty wore off rapidly. Seen as more of a gimmick the solution proved impractical both for readers -- who had to hold their phone or tablet in place after having downloaded a new application -- and editors, who would have to indicate scannable content by adding markers (which, aesthetically, can be as bad as QR codes) and sourcing content. It transpired this was a lot of preparation for a rather small pay-off.
With InkLink, we focused on the step that comes before AR, recognising a pre-specified location in the paper using Image Recognition (AR only works in the presence of patterns, or images, to recognise). Our premise is the following: can we help our readers digitally bookmark article for future reference by a simple scan of their morning paper?
What InkLink can do
Our research into InkLink has shown that the main difficulty, beyond technical constraints, is to surface article data across channels. Unfortunately, the text printed in the FT, isn’t currently matched in any way to online content. However, with manually curated content, our tool can recognise pictures from the paper and deliver up-to-date, additional content; complete with comments and related articles. It could also reveal interactive graphics or videos, not as an augmented layer, but straight out of ft.com, as a companion to the printed version.
InkLink can recognise a variety of images as long as they’re the focal point of the capture. It could even recognise the daily crossword grid (blank or partially filled in).
To achieve this result, we explored number of third-party services that focused on AR.
After looking into Image Retrieval (the technique employed by Google reverse image search), it became apparent that we didn’t need to identify elements within an image, just recognise it. That’s why we simplified our approach and are now focusing on Image Recognition. To get it working, we compared 3 different services: Blippar, Aurasma, and CraftAR by Catchoom.
Why we chose CraftAR
InkLink works fully in the browser (and is mobile-compatible, with a small caveat for iOS), without having to download an external application.
How does InkLink work?
InkLink is a web app that has 2 distinct aspects:
- An interface for editors to create new markers remotely, using a NodeJS wrapper and the Management API from CraftAR
- A client-side scanner using CraftAR JS library
Find the source on GitHub
Using a GUI, content curators can simply upload a .zip file containing images and their associated link. The NodeJS server then takes care of unzipping the file and creating “items” (an entity made of a link and 1 or more images) with their images remotely.
Once a “collection” (an entity comprising of multiple items) is created, readers can scan the latest images on the client side.
What to do next?
There are multiple applications possible for InkLink. By letting people play with it, we hope to uncover more. For example:
- Integrate it with our upcoming project of interactive crosswords, and link to the digital grid.
- Combine it with OCR to reveal crosswords solutions on a partially filled puzzle. By letting people play with it, we hope to uncover additional applications.
- Jump straight to a related interactive graphic, or video.
- Show the online comments.
- Show related/subsequent articles.
- Saving the article for later.
We are also aiming to overcome our current limitations:
- Reducing the manual process of sourcing content. Extracting images from PDFs is easily doable, but the issue of linking it back to online content persists.
- The ft.com site does not support direct links into interactive graphics or videos which are part of articles.
Want to give it a try?Let's get some caveats out of the way:
- This is an experiment, so, caveat emptor...
- We used the default/free version of CraftAR which has a limit of 1000 scans/month, so first come first served.
- You will need a screen to display these images, and a phone with camera, or a webcam, to scan the images.
- We may have re-configured InkLink to work with different images and forgotten to update this post.
- The articles may be behind the paywall, so you may receive a polite but firm request to sign up.
- Display the following images.
- On a 2nd screen/phone, go to ftlabs-inklink.herokuapp.com.
- Select 'Start scanning'.
- Align the camera to bring one of the following images into focus in the window.
- Wait for a bit...
Some images taken from the Financial Times newspaper, Friday 17 February 2017
White House 'a fine-tuned machine' (behind the paywall)
FT Data (a stream page, not behind the paywall)
Turkey border gridlock (behind the paywall)
Don't forget, there is a limit on how many scans our use of CraftAR will support, so it may be you are too late.