Veronika Krauß
ABSTRACT
Current research in augmented, virtual, and mixed reality (XR) reveals a lack of tool support for designing and, in particular, prototyping XR applications. While recent tools research is often motivated by studying the requirements of non-technical designers and end-user developers, the perspective of industry practitioners is less well understood. In an interview study with 17 practitioners from different industry sectors working on professional XR projects, we establish the design practices in industry, from early project stages to the final product. To better understand XR design challenges, we characterize the different methods and tools used for prototyping and describe the role and use of key prototypes in the different projects. We extract common elements of XR prototyping, elaborating on the tools and materials used for prototyping and establishing different views on the notion of fidelity. Finally, we highlight key issues for future XR tools research.
- Deutsche Akkreditierungsstelle. 2010. Leitfaden Usability. http://www.dakks.de – retrieved in December 2019. retracted in January 2020.Google Scholar
- Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Parmit K. Chilana. 2020. Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376722Google ScholarDigital Library
- Uday A. Athavankar. 1997. Mental Imagery as a Design Tool. Cybernetics and Systems 28, 1 (1997), 25–42. https://doi.org/10.1080/019697297126236Google ScholarCross Ref
- D. Baumer, W. Bischofberger, H. Lichter, and H. Zullighoven. 1996. User interface prototyping-concepts, tools, and experience. In Proceedings of IEEE 18th International Conference on Software Engineering. Institute of Electrical and Electronics Engineers, Berlin, Germany, 532–541. https://doi.org/10.1109/ICSE.1996.493447Google ScholarCross Ref
- Michel Beaudouin-Lafon and Wendy Mackay. 2009. Prototyping Tools and Techniques. In Human-Computer Interaction: Development Process (1 ed.). CRC Press, Boca Raton, 137–160. https://doi.org/10.1201/9781420088892Google ScholarCross Ref
- Marion Buchenau and Jane Fulton Suri. 2000. Experience Prototyping. In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (New York City, New York, USA) (DIS ’00). Association for Computing Machinery, New York, NY, USA, 424–433. https://doi.org/10.1145/347642.347802Google ScholarDigital Library
- Bill Buxton. 2007. Sketching User Experiences. Morgan Kaufmann, Burlington. 139–141 pages. https://doi.org/10.1016/B978-012374037-3/50059-6Google ScholarCross Ref
- Herbert H Clark and Susan E Brennan. 1991. Grounding in communication.In Perspectives on socially shared cognition., Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley (Eds.). American Psychological Association, Washington, DC, US, 127–149. https://doi.org/10.1037/10096-006Google ScholarCross Ref
- Juliet M. Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria.Qual Sociol 13(1990), 3–21. https://doi.org/10.1007/BF00988593Google ScholarCross Ref
- Steven Dow, T. Scott Saponas, Yang Li, and James A. Landay. 2006. External Representations in Ubiquitous Computing Design and the Implications for Design Tools. In DIS ’06: Proceedings of the 6th conference on Designing Interactive systems. Association for Computing Machinery, Pennsylvania, 241–250. https://doi.org/10.1145/1142405.1142443Google ScholarDigital Library
- Sebastian Draxler, Adrian Jung, Alexander Boden, and Gunnar Stevens. 2011. Workplace Warriors: Identifying Team Practices of Appropriation in Software Ecosystems. In Proceedings of the 4th International Workshop on Cooperative and Human Aspects of Software Engineering (Waikiki, Honolulu, HI, USA) (CHASE ’11). Association for Computing Machinery, New York, NY, USA, 57–60. https://doi.org/10.1145/1984642.1984656Google ScholarDigital Library
- Christiane Floyd. 1984. A Systematic Look at Prototyping. In Approaches to Prototyping, Reinhard Budde, Karin Kuhlenkamp, Lars Mathiassen, and Heinz Züllighoven (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1–18.Google Scholar
- Maribeth Gandy and Blair MacIntyre. 2014. Designer’s Augmented Reality Toolkit, Ten Years Later: Implications for New Media Authoring Tools. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 627–636. https://doi.org/10.1145/2642918.2647369Google ScholarDigital Library
- Paul Green and Lisa Wei-Haas. 1985. The Rapid Development of User Interfaces: Experience with the Wizard of OZ Method. Proceedings of the Human Factors Society Annual Meeting 29, 5(1985), 470–474. https://doi.org/10.1177/154193128502900515Google ScholarCross Ref
- Trent Hergenrader. 2018. Collaborative worldbuilding for writers and gamers. Bloomsbury Publishing, London, Great Britain. 257 pages.Google Scholar
- Stephanie Houde and Charles Hill. 1997. Chapter 16 - What do Prototypes Prototype?In Handbook of Human-Computer Interaction (Second Edition) (second edition ed.), Marting G. Helander, Thomas K. Landauer, and Prasad V. Prabhu (Eds.). North-Holland, Amsterdam, 367–381. https://doi.org/10.1016/B978-044481862-1.50082-0Google ScholarCross Ref
- ISO/TC 159/SC 4 Ergonomics of human-system interaction. 2019. ISO 9241-210:2019 Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems (2 ed.). Standard. International Organization for Standardization, Geneva, Switzerland.Google Scholar
- Jennifer Preece, Helen Sharp, and Yvonne Rogers. 2015. Interaction Design: Beyond Human-Computer Interaction (4 ed.). Wiley. https://www.wiley.com/en-us/Interaction+Design%3A+Beyond+Human+Computer+Interaction%2C+4th+Edition-p-9781119088790Google Scholar
- Hilary Johnson and Lucy Carruthers. 2006. Supporting creative and reflective processes. International Journal of Human Computer Studies 64 (10 2006), 998–1030. Issue 10. https://doi.org/10.1016/j.ijhcs.2006.06.001Google ScholarDigital Library
- Veronika Krauß, Alexander Boden, Leif Oppermann, and René Reiners. 2021. Current Practices, Challenges, and Design Implications for Collaborative AR/VR Application Development. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 454, 15 pages. https://doi.org/10.1145/3411764.3445335Google ScholarDigital Library
- Veronika Krauß, Florian Jasche, Sheree May Saßmannshausen, Thomas Ludwig, and Alexander Boden. 2021. Research and Practice Recommendations for Mixed Reality Design – Different Perspectives from the Community. In 27th ACM Symposium on Virtual Reality Software and Technology (VRST ’21) (Osaka, Japan). Association for Computing Machinery, New York, NY, USA, 13. https://doi.org/10.1145/3489849.3489876Google ScholarDigital Library
- Carlye A. Lauff, Daniel Knight, Daria Kotys-Schwartz, and Mark E. Rentschler. 2020. The role of prototypes in communication between stakeholders. Design Studies 66(2020), 1–34. https://doi.org/10.1016/j.destud.2019.11.007Google ScholarCross Ref
- G.A. Lee, C. Nelles, M. Billinghurst, and G.J. Kim. 2004. Immersive authoring of tangible augmented reality applications. In Third IEEE and ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, Arlington, VA, USA, 172–181. https://doi.org/10.1109/ISMAR.2004.34Google ScholarDigital Library
- Germán Leiva and Michel Beaudouin-Lafon. 2018. Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-Usability. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 675–682. https://doi.org/10.1145/3242587.3242613Google ScholarDigital Library
- Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2020. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376160Google ScholarDigital Library
- Youn-Kyung Lim and Erik Stolterman. 2008. The Anatomy of Prototypes: Prototypes as Filters, Prototypes as Manifestations of Design Ideas. Transactions on Computer-Human Interaction 15, 2 (2008), 1–27. https://doi.org/10.1145/1375761.1375762Google ScholarDigital Library
- Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter. 2004. DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences. In UIST ’04: Proceedings of the 17th annual ACM symposium on User interface software and technology. Association for Computing Machinery, Santa Fe, NM, USA, 197–206. https://doi.org/10.1145/1029632.1029669Google ScholarDigital Library
- Martin Maguire. 2020. An Exploration of Low-Fidelity Prototyping Methods for Augmented and Virtual Reality. In Design, User Experience, and Usability. Design for Contemporary Interactive Environments, Aaron Marcus and Elizabeth Rosenzweig (Eds.). Springer International Publishing, Cham, 470–481.Google Scholar
- Steve Mann. 2002. Mediated reality with implementations for everyday life. Presence Connect 1(2002), 2002.Google Scholar
- Michael McCurdy, Christopher Connors, Guy Pyrzak, Bob Kanefsky, and Alonso Vera. 2006. Breaking the Fidelity Barrier: An Examination of Our Current Characterization of Prototypes and an Example of a Mixed-Fidelity Success. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI ’06). Association for Computing Machinery, New York, NY, USA, 1233–1242. https://doi.org/10.1145/1124772.1124959Google ScholarDigital Library
- Paul Milgram and Fumio Kishino. 1994. Taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems E77-D, 12(1994), 1321–1329.Google Scholar
- Brad Myers, Scott E Hudson, and Randy Pausch. 2000. Past, present, and future of user interface software tools. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 1(2000), 3–28.Google ScholarDigital Library
- Michael Nebeling, Katy Lewis, Yu-Cheng Chang, Lihan Zhu, Michelle Chung, Piaoyang Wang, and Janet Nebeling. 2020. XRDirector: A Role-Based Collaborative Immersive Authoring System. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376637Google ScholarDigital Library
- Michael Nebeling and Katy Madier. 2019. 360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300826Google ScholarDigital Library
- Michael Nebeling, Janet Nebeling, Ao Yu, and Rob Rumble. 2018. ProtoAR: Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173927Google ScholarDigital Library
- Michael Nebeling and Maximilian Speicher. 2019. The Trouble with Augmented Reality/Virtual Reality Authoring Tools. In Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers, Munich, Germany, 333–337. https://doi.org/10.1109/ISMAR-Adjunct.2018.00098Google ScholarCross Ref
- Donald A. Norman. 1993. Things That Make Us Smart -Defending Human Attributes in the Age of the Machine (1 ed.). Vol. 1. Addision-Wesley Publishing Company, USA.Google Scholar
- Marc Rettig. 1994. Prototyping for Tiny Fingers. Commun. ACM 37, 4 (April 1994), 21–27. https://doi.org/10.1145/175276.175288Google ScholarDigital Library
- Jim Rudd, Ken Stern, and Scott Isensee. 1996. Low vs. High-Fidelity Prototyping Debate. Interactions 3, 1 (Jan. 1996), 76–85. https://doi.org/10.1145/223500.223514Google ScholarDigital Library
- Sanni Siltanen. 2012. Theory and applications of marker-based augmented reality: Licentiate thesis. Ph.D. Dissertation. Aalto University, Finland. Project code: 78191.Google Scholar
- Maximilian Speicher, Brian D. Hall, and Michael Nebeling. 2019. What is Mixed Reality?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300767Google ScholarDigital Library
- Maximilian Speicher, Katy Lewis, and Michael Nebeling. 2021. Designers, the Stage Is Yours! Medium-Fidelity Prototyping of Augmented & Virtual Reality Interfaces with 360theater. Proceedings of the ACM on Human-Computer Interaction 5, EICS(2021), 1–25.Google ScholarDigital Library
- Maximilian Speicher and Michael Nebeling. 2018. GestureWiz: A Human-Powered Gesture Design Environment for User Interface Prototypes. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3173574.3173681Google ScholarDigital Library
- Gunnar Stevens and Sebastian Draxler. 2010. Appropriation of the eclipse ecosystem: Local integration of global network production. In Proceedings of COOP 2010: Proceedings of the 9th International Conference on Designing Cooperative Systems. Springer, Aix-en-Provence, France, 287–308.Google ScholarCross Ref
- Erik Stolterman. 2008. The Nature of Design Practice and Implications for Interaction Design Research. International Journal of Design 2, 1 (2008), 55–65. http://www.ijdesign.org/index.php/IJDesign/article/view/240/148Google Scholar
- Erik Stolterman, Jamie McAtee, David Royer, and Selvan Thandapani. 2009. Designerly Tools. In Undisciplined! Design Research Society Conference 2008. Sheffield Hallam University, Sheffield, UK, 16–19. http://shura.shu.ac.uk/491/Google Scholar
- Alexandra Thompson and Leigh Ellen Potter. 2018. Taking the ’A’ Out of ’AR’: Play Based Low Fidelity Contextual Prototyping of Mobile Augmented Reality. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (Melbourne, VIC, Australia) (CHI PLAY ’18 Extended Abstracts). Association for Computing Machinery, New York, NY, USA, 647–653. https://doi.org/10.1145/3270316.3271518Google ScholarDigital Library
- Balasaravanan Thoravi Kumaravel, Fraser Anderson, George Fitzmaurice, Bjoern Hartmann, and Tovi Grossman. 2019. Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 161–174. https://doi.org/10.1145/3332165.3347872Google ScholarDigital Library
- Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Bjoern Hartmann. 2020. TransceiVR: Bridging Asymmetrical Communication Between VR Users and External Collaborators. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 182–195. https://doi.org/10.1145/3379337.3415827Google ScholarDigital Library
- Haijun Xia, Sebastian Herscher, Ken Perlin, and Daniel Wigdor. 2018. Spacetime: Enabling Fluid Individual and Collaborative Editing in Virtual Reality. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 853–866. https://doi.org/10.1145/3242587.3242597Google ScholarDigital Library
- Yasuhiro Yamamoto and Kumiyo Nakakoji. 2005. Interaction design of tools for fostering creativity in the early stages of information design. International Journal of Human Computer Studies 63 (2005), 513–535. Issue 4-5 SPEC. ISS.. https://doi.org/10.1016/j.ijhcs.2005.04.023Google ScholarDigital Library
- Lei Zhang and Steve Oney. 2020. FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 342–353. https://doi.org/10.1145/3379337.3415824Google ScholarDigital Library
- Yu Zhu, Kang Zhu, Qiang Fu, Xilin Chen, Huixing Gong, and Jingyi Yu. 2016. SAVE: Shared Augmented Virtual Environment for Real-Time Mixed Reality Applications. In Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry - Volume 1(Zhuhai, China) (VRCAI ’16). Association for Computing Machinery, New York, NY, USA, 13–21. https://doi.org/10.1145/3013971.3013979Google ScholarDigital Library
Index Terms
- Elements of XR Prototyping: Characterizing the Role and Use of Prototypes in Augmented and Virtual Reality Design
Recommendations
Training Virtual Reality and Augmented Reality Prototyping without Coding
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing SystemsRecently, virtual reality (VR) and augmented reality (AR) applications have become popular. Hence, designers must be trained to be better equipped to design VR applications and games. However, current VR prototyping approaches present many obstacles to ...
What is XR? Towards a Framework for Augmented and Virtual Reality
AbstractAugmented Reality (AR), Virtual Reality (VR), Mixed Reality, and Extended Reality (often – misleadingly – abbreviated as XR) are commonly used terms to describe how technologies generate or modify reality. However, academics and ...
Highlights- XR should not be used as an abbreviation for “Extended Reality”; X should be the placeholder for “all” new reality formats.
PeriMR: a prototyping tool for head-mounted peripheral light displays in mixed reality
MobileHCI '17: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and ServicesNowadays, Mixed and Virtual Reality devices suffer from a field of view that is too small compared to human visual perception. Although a larger field of view is useful (e.g., conveying peripheral information or improving situation awareness), technical ...
Comments