Facebook developed AI for analysis of text on images

Facebooks more than 2.2 billion users share an incredible amount of photos every day. The social media giant needs to organize it every day, add it to search results and scan for potentially harmful content. A large part of these images also contain text, which also needs to be analysed.

In order to cope with that task, Facebook has built artificial intelligence called Rosetta. Today, the company revealed the existence of the assistant in a blog. Every day, the assistant analyses all the text that can be found on Facebook photos.

1 billion photos

Rosetta collects the text that appears on more than 1 billion photos shared on Facebook and Instagram on a daily basis. That text is written in a lot of different languages. The system not only analyzes standalone photos, but also individual frames in videos. To do so, it scans all the visual material in a different way than traditional text recognition software does.

Normally, only individual letters in texts are analysed and pushed through, without systems understanding the deeper context. However, Facebook’s needs went further. The company wanted a system bolts that also takes into account the context of the image. The developers of Facebook did this by providing Rosetta with predictive power.

Comprehensive analyses

Historical data is also used for the analysis of visual material, so that a visual profile is created not only of the letters, but also of the words they form and the context in which they stand. According to Facebook, this approach makes it possible for Rosetta to understand material that has never been seen before.

The system is used for a range of applications. Rosetta makes images searchable via Facebook and Instagram and also determines how and when photos or videos should be shown in timelines. Rosetta also determines whether it is potentially harmful content, so that it can be removed.

Facebook still writes in the blog that Rosetta needs more applications in the near future.