Scientists, including one of Indian-origin, have created the first fully automated computer programme that teaches itself everything there is to know about any visual concept.
Called Learning Everything about Anything, or LEVAN, the programme searches millions of books and images on the Web to learn all possible variations of a concept, then displays the results to users as a comprehensive, browsable list of images, helping them explore and understand topics quickly in great detail.
"It is all about discovering associations between textual and visual data," said Ali Farhadi, a University of Washington assistant professor of computer science and engineering.
"The programme learns to tightly couple rich sets of phrases with pixels in images. This means that it can recognise instances of specific concepts when it sees them," Farhadi said.
The programme learns which terms are relevant by looking at the content of the images found on the Web and identifying characteristic patterns across them using object recognition algorithms.
It's different from online image libraries because it draws upon a rich set of phrases to understand and tag photos by their content and pixel arrangements, not simply by words displayed in captions.
Users can browse the existing library of roughly 175 concepts. Existing concepts range from "airline" to "window," and include "beautiful," "breakfast," "shiny," "cancer," "innovation," "skateboarding," "robot," and the researchers' first-ever input, "horse."
If the concept you're looking for doesn't exist, you can submit any search term and the programme will automatically begin generating an exhaustive list of subcategory images that relate to that concept.
The technique works by searching the text from millions of books written in English and available on Google Books, scouring for every occurrence of the concept in the entire digital library.
Then, an algorithm filters out words that aren't visual. For example, with the concept "horse," the algorithm would keep phrases such as "jumping horse," "eating horse" and "barrel horse," but would exclude non-visual phrases such as "my horse" and "last horse."
Once it has learned which phrases are relevant, the programme does an image search on the Web, looking for uniformity in appearance among the photos retrieved.
When the programme is trained to find relevant images of, say, "jumping horse," it then recognises all images associated with this phrase.
"The new programme needs no human supervision, and thus can automatically learn the visual knowledge for any concept," said Santosh Divvala, a research scientist at the Allen Institute for Artificial Intelligence and an affiliate scientist at UW in computer science and engineering.
