Note: Our work on sketching interactive AR experiences by programming by demonstration appeared to UIST 2021. Work with my excellent collaborators Germán Leiva, Jens Emil Sloth Grønbæk, Clemens Nylandsted Klokmose, Cuong Nguyen, Paul Asente. See The following UIST talk video for the presentation.
Programming by Demonstration (PbD) is a well-known technique that allows non-programmers to describe interactivity by performing examples of the expected behavior, but it has not been extensively explored for AR. We present Rapido, a novel early-stage prototyping tool to create fully interactive mobile AR prototypes from non-interactive video prototypes using PbD. In Rapido, designers use a mobile AR device to record a video prototype to capture context, sketch assets, and demonstrate interactions. They can demonstrate touch inputs, animation paths, and rules to, e.g., have a sketch follow the focus area of the device or the user’s world-space touches. Simultaneously, a live website visualizes an editable overview of all the demonstrated examples and infers a state machine of the user flow. Our key contribution is a method that enables designers to turn a video prototype into an executable state machine through PbD. The designer switches between these representations to interactively refine the final interactive prototype. We illustrate the power of Rapido’s approach by prototyping the main interactions of three popular AR mobile applications.
Rubaiat Habib Kazi 