Cut-and-paste enters era of augmented reality

by Peter Grad , Tech Xplore


It was in 1901 that an author of children's books imagined an electronic image that could float over people and provide information about them. L. Frank Baum, who a year earlier penned "The Wonderful Wizard of Oz," referred to that image as a character marker. In fact, it is the first known reference to what 120 years later would commonly be referred to as augmented reality.

From SnapChat's whimsical face filters to daily evening weather forecast maps on TV, digital overlays upon real world scenes have become ubiquitous.

The pace of advances in augmented reality will only increase in coming months and years.

Some of those advances will be relatively simple. One such example was unveiled last month. Cyril Diagne, a designer and programmer currently in residence at the Google Arts and Culture Lab in Paris, showed that as mundane an operation as cut-and-paste can be turbocharged in the era of augmented reality.

He posted demonstration videos of his smartphone camera process that singles out an object, isolates it from the background and, when pointed to a desktop computer, inserts the image into a document. This is all done in under 10 seconds.

Diagne explained that he utilizes BASNet to trace the outline of an image and smoothly wipe out the background. He uses other image-tracking technology to determine precisely where the phone's camera is pointing.

4/10 - Cut & paste your surroundings to Photoshop
Code: https://t.co/cVddH3u3ik
Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF— Cyril Diagne (@cyrildiagne) May 3, 2020

"This is a prototype that uses the phone camera to capture, extract and paste objects from your surroundings directly to Photoshop," Diagne explained. "Thanks to OpenCV SIFT, an image-tracking technology, the app is able to detect where the phone is pointing at the screen, making it a seamless experience. It's part of a series of experiments I'm doing every weekend to explore how machine learning and AI can help create more digital interactions that are more natural."

His demonstration video shows him pointing his camera at three items in succession—a plant, a book and a blouse—and in each instance pointing the camera at a desktop computer, where each image flies into the document.

Images require 2.5 seconds to capture and 4 seconds to paste.

Other video examples show him capturing still images hanging on a wall transferred to a computer where they initiate video clips, and capturing the logo of hard rock band Iron Maiden on the T-shirt of his wife that, when transferred to the computer, instantly plays "Drifter" from their "Killers" album.

Diagne says there is room for improvement, including a reduction in times needed for transfer. But for now, he offers AR Cut and Paste only as a prototype and has uploaded all details and code on GitHub. A local server must be established to link the smartphone to Photoshop on the desktop computer.

Augmented reality fuels some of the most intriguing apps available today. Aside from its obvious application to innumerable games, it offers many practical uses for personal, educational and commercial needs.

Froggipedia projects a lifelike 3-D frog on any surface you point your camera at, offering a highly detailed—and pain-free—step-by-step dissection, conducted with your fingertip motions. HairStyle Pro captures your image and presents you with hundreds of hairstyles and differing colors precisely positioned over your face (including beards and sideburns for men). YouCam Makeup-Magic does the same for women seeking the perfect shade of cosmetics.

For the craftsmen or craftswoman in the home, AR Measure Kit lets you point your camera at any two targets and returns a precise measurement. If you prefer to purchase ready-made furniture, IKEA allows you to project images of its inventory of furniture and other home accessories right into your living room or kitchen.

And perhaps best of all, with an AR Visualizer, you can check out that Porsche 911 Carrera 4S that you always wanted, listing at a mere $120,600 today, and place its image on your own driveway in front of your home as you select colors, interiors and rim style and then watch the car drive off and down the road.

L. Frank Baum would be impressed.

Explore furtherEngineers develop computerized camera without optics that instead uses an ordinary window as the lens

More information: www.theverge.com/2020/5/4/2124 … world-photoshop-demo

© 2020 Science X Network