We are delighted to announce that UCL Computer Science student Luke Harries has won the Microsoft Prize at the 2018 University of Cambridge’s hackathon, HackCambridge.
Luke, who is currently studying for an MSc in Computer Science, was part of an international and cross-disciplinary team, formed of Filip, Information Engineering, University of Cambridge; Marcin, Mathematics and Computer Science, Brown University; and Mateusz, Mathematics, Imperial College London, and Software Engineering, BlackRock.
“HackCambridge was the perfect opportunity to return to Cambridge, where I completed my undergraduate degree in Medicine and Cognitive Neuroscience,” said Luke. “Our initial ideas ranged from trying to implement a decentralised ICO (DAICO) to controlling a VR world with a Myo armband.”
In the lead up to the event, Microsoft released a video of their Seeing AI app, which demonstrated how Saqid Shaikh, a visually impaired software developer at Microsoft, used the app he created to convert a visual world into an audible experience. The app can perform a range of functions, from describing children playing Frisbee in the park or reading a menu out loud.
The team quickly decided to build on SeeingAI's fantastic work: “It was incredible - we could only imagine the impact it had on him and many others with visual impairments!”
People with visual impairments frequently rely on a Screen Reader to navigate the internet, which reads out the written content of each page. However, when the Screen Reader reaches an image it has to rely on the “alt tag” to provide a caption. For the majority of the internet these “alt tags” are either missing, superficial or inaccurate. Luke’s team decided to bring automatic "alt-tag" generation to everyone on the internet, regardless of device.
To do this, the team used a Chrome extension to scan through every image on the page lacking an "alt tag". For each image where the "alt tag" was missing, we would send the image URL to an Azure cloud function. The cloud function would check if we already had the "alt tag" saved and if so return it. If not, we would use the Microsoft's Cognitive Services Vision API to perform object recognition and generate the description to be added as the "alt tag".
“The speed, accuracy and ease of integration with which the image descriptions were being generated would not have been possible just a few years ago, “said Luke. “We are very proud to have won the prize, and we hope our Chrome extension will empower those with visual impairments to browse the web, without having to rely on others to create the "alt tags".”
Please join us in congratulating Luke and his teammates on their success.
Find out more about HackCambridge here.
You can read Luke’s full blog on the event.
The team is currently recruiting for a small pilot study to determine the impact that their Chrome extension can have on those with visual impairments. For more information please visit their website: visualcognition.co.
