Our paper titled “Interactive Body-Driven Graphics for Augmented Video Performance” appeared to CHI 2019. Congratulations to Nazmus Saquib, and my amazing collaborators at Adobe Research- Li-Yi Wei and Wilmot Li!
“In the coming age, Computer Graphics will become an integral part of our language” – Ken Perlin
Augmented and mixed-reality technologies enable us to enhance and extend our perception of reality by incorporating virtual graphics into real-world scenes. One simple but powerful way to augment a scene is to blend dynamic graphics with live action footage of real people performing. In the past, this technique was used as a special effect for music videos, scientific documentaries, and instructional materials incorporated in the post-processing stage. Manipulating graphics in real-time is now becoming more popular in weather forecasts, live television shows, and, more recently, social media apps with video overlays.
As live-streaming becomes an increasingly powerful cultural phenomenon, an Adobe Research team is exploring how to enhance these real-time presentations with interactive graphics to create a powerful new storytelling environment. Traditionally, crafting such an interactive and expressive performance typically required technical programming or highly-specialized tools tailored for experts.
Our approach is different, and could open up this kind of presentation to a much wider range of people.
Our system leverages the rich gestural (from direct manipulation to abstract communications) and postural language of humans to interact with graphical elements.
By simplifying the mapping between gestures, postures, and their corresponding output effects, our UI enables users to create customized, rich interactions with the graphical elements.
Our user study with amateurs and professionals demonstrates the potential usage and unique aspects of this augmented medium for storytelling and presentation across a range of application domains. The resulting live presentations include an academic research paper, a cooking tutorial, interior planning, meditation tips, and educational content.
We anticipate that interactive graphics will help shape our real-time virtual communications capabilities by empowering people to leverage whole-body language, speech, and context.
Publication
Interactive Body-drive Graphics for Augmenting Video Performances
Nazmus Saquib, Rubaiat Habib, Li-Yi Wei, Wilmot Li
CHI 2019 (PDF)
Supplementary videos
Presentations and live performances by our user study participants. I’m amazed to see the creative outcomes and potential widespread application of such technologies.
Rubaiat Habib Kazi 