Facebook's new marketplace gambit

The social network is entering a highly competitive zone of e-commerce where there are big players already. Tools like GrokNet should help it to carve out market share

Facebook’s Marketplace is a peer-to-peer shopping platform where its two billion-plus users upload stuff they wish to sell. It’s available in 50-odd countries including select cities in India. Facebook doesn’t get involved in deal-making and monetises Marketplace by advertising. Anything new or old that’s legal can be sold (some smart users have worked out ways to sell illegal things too). On Wednesday, the network launched its new offering, Shops, which allows small businesses to open online shops on FB and its subsidiary, Instagram.

In order to boost the Marketplace, FB has developed new AI with some stunning features involving cutting-edge computer vision. Its new tool, GrokNet, is foundational to its rallying cry of consumerism — “Make anything shoppable”.

“Grok” is a word coined by Robert Heinlein in his 1961 science fiction classic, Stranger in a Strange land. This features a human, Valentine Smith, brought up by alien Martians. Smith turns human civilisation inside-out by teaching people Martian ways to “grok” or understand something in an instinctive, immersive way. (Spoiler alert: Grokking involves cannibalism in the book.)

The official blogpost for the GrokNet system reads, “Our long-term vision is to build an all-in-one, AI lifestyle assistant that can accurately search and rank billions of products, while personalising to individual tastes. That system would make online shopping just as social as shopping with friends in real life. Going one step further, it would advance visual search to make your real-world environment shoppable. If you see something you like (clothing, furniture, electronics, etc.), you could snap a photo of it and the system would find that exact item, as well as several similar ones to purchase right then and there.”

When an item is uploaded by a seller, the system looks at the pictures and suggests attributes to list such as colour, dimensions, etc. When a buyer searches for something, the system matches it to available products, even if the search keywords don’t fit the seller’s description. Facebook is also using GrokNet to launch automatic product tagging on Facebook Pages to help buyers and sellers.

The system also offers an extraordinary new feature which turns 2D videos into 3D-like interactive views. The view can be rotated or turned 360 degrees and it is claimed to work even with poor lighting or partially obscured objects. This is among the first e-commerce platforms to offer this feature. The interactive 3-D viewer offers even more detail as to scale, etc. Make-up can be virtually applied to a picture of the face or a coat can be “worn”.

This uses a technique common in gaming — a visual-inertial Simultaneous Localisation and Mapping (SLAM) framework. In SLAM, an algorithm maps features detected in the video to a 3D view of an object. This rotating view is only available for Marketplace on iOS at the moment.

Another critical feature here is segmentation, where the computer identifies which pixels in a picture belong to which objects. This can be tough with things like clothes or masks where hair and other clothing gets in the way. The system has to precisely label every pixel.

Facebook says it developed a technique called Instance Mask Projection, which it claims to be the first system to predict obstructed or layered objects in photos, like shirts under coats. It claims above 80 per cent accuracy. GrokNet labels multiple objects in a single image by segmenting the image into parts and placing an outline around each identified item. Instance Mask Projection predicts the shape of all objects, and then individual labelled pixels are analysed for colour, and other details.

GrokNet was trained on millions of images uploaded by Facebook users, which gave it a rich data-set. Users sell all sorts of things ranging from cars to ashtrays and clothes. The AI uses 80-odd loss functions. A loss function is the measure of an algorithm’s accuracy in labelling objects in this instance. If the algo identifies an image incorrectly, the loss function will be a large number. As accuracy improves, loss function numbers reduce.

Where GrokNet scores over other machine vision systems is in the breadth of objects it recognises. Most machine vision systems are focused on identifying one type of object — say, shoes, or cars. This has seven different image sets and it can pick out objects from each.

This allows for a more natural shopping experience where, for example, a picture that features a carpet in a living room with furniture, etc., can be uploaded. The buyer can click on any object for more detail.

Facebook is entering a highly competitive zone of e-commerce where there are several big players already. Tools like GrokNet should help it to carve out market share.