We are delighted to support Jeremy Nelson, Director of the U-M XR Initiative, in kicking off his MiXR podcast. In this first episode, he sat down with me, Michael Nebeling. We discuss a lot of the developments around XR, both at U-M and outside. Some of the key points included how I found my own way into XR, and into teaching it, and I also give examples from recent research and share my perspective on the future. I really enjoyed the conversation and I hope you do, too.
Professor Nebeling and the Michigan Information Interaction Lab are thrilled to announce that Audrey Labrie has decided to join the UMSI PhD program. Audrey will be joining us from Polytechnique Montreal where she is currently finishing up her undergraduate in Software Engineering.
Audrey has a variety of interests in mixed reality interfaces, interaction techniques, accessibility, and how to learn and be productive with these evolving new sets of technologies. Recently, Audrey has been visiting with Professor Albrecht Schmidt‘s research group at LMU in Germany, where she helped create and study an mobile augmented reality German language learning app with users. The project was supervised by Lewis Chuang, Fiona Draxler and Albrecht Schmidt, and led to a publication at CHI 2020.
Audrey is currently working with Professor Jinghui Cheng at the User-Centered Design Lab at Polytechnique Montreal on two user-centered AR design projects.
We are looking forward to welcoming Audrey to U-M, the UMSI community, and our lab, and to many exciting new research projects together with her!
Professor Nebeling and the Michigan Information Interaction Lab are thrilled to announce that current MSI student, Shwetha Rajaram, has decided to join the UMSI PhD program.
Update: 2/3 positions have been filled. But we have one more opening for summer 2020! For this last position, we’re really looking for someone technical.
Summary: The Information Interaction Lab is offering up to 3 summer internship positions to students who would like to contribute to our ongoing XR design projects, some of which will be targeted for publication at the CHI 2021 conference. We also respond to a call from UMSI-CDO and highly encourage UMSI students who are experiencing distress and uncertainty about their planned summer internships due to COVID-19. We are committed to providing the following:
Opportunities for applied research methodology and skill-building; and/or work that can support the research team in some way (students will assist with the development of research prototypes in the lab; Professor Nebeling will provide a framework for the internship and guide students through each milestone)
Supervision that includes mentorship by someone that is knowledgable of the intern’s work and can provide adequate guidance, support, and learning (Professor Nebeling works with all students directly, with weekly project and mentoring meetings)
Access to external constituents, partners, and/or users (unfortunately, this will be limited given the COVID-19 pandemic but we will be conducting research that is aligned with previous research gifts from Mozilla and Disney Studios)
Learning that will benefit the intern in developing their skillset and professional experience, as well as support for meeting their learning objectives (we will have weekly virtual lab meetings to talk about project goals, ideas, and milestones)
Fair compensation: The average on-campus internship pay rate for master’s students is $15-$20/hr (we will commit to a pay rate of $20/hr and will have work of 15-20 hours per week, on average, but actual workload may vary and the expectations will be commensurate with each individual student’s personal & work situation)
More details about the projects below:
Working with UMSI professor Michael Nebeling in the Information Interaction Lab, the projects we are currently working on, at the highest level, are investigating how virtual and augmented reality interfaces can be created more easily and faster, and how XR (where X is a placeholder for VR and AR) can become more effective modes of interaction and complement existing interfaces. Our current focus as part of our involvement with the U-M wide XR Initiative is on use cases that support (remote) immersive learning and instruction through the use of XR technologies. Depending on the student’s experience and interests, they could help with the development of new tools to support this investigation, the design and evaluation of new kinds of AR/VR interfaces with a particular focus on novel interaction techniques, or both.
Some of the new research questions Professor Nebeling is asking include: How do we abstract from the complexities of tools like Unity and Unreal, yet allow designers over a short period of time in some of our tools to transition to these more advanced tools? How can we prototype complex, interactive AR/VR experiences without writing code, mostly using programming-by-demonstration? How can we enable artists to fully express their designs on paper, or in immersive authoring tools using either AR, or VR, or some combination of both? How can we appropriate complex physical objects and use them to define complex interactive behaviors in AR or VR while occluding and potentially deforming them in the process? What would a WordPress for AR/VR look like in terms of enabling concepts and the most useful getting started templates? How can we adapt a tool like PowerPoint, which works in 2D, to create “immersive slideshows” for AR/VR, without the need to design in 3D?
Desired experience & skills
The focus of the project will be determined based on student interest and experience, and can range from more technical development to more user research oriented angles. Experience shows that students with a technical background such as Computer Science do very well when working with Professor Nebeling and other lab members. Ideally, you have…
successfully completed Professor Nebeling’s AR/VR courses, SI 559 and SI 659
previous programming experience–either acquired through coursework or, ideally, through projects–with web technologies such as HTML5/CSS3, JavaScript/jQuery, and Node.js
familiarity with THREE.js/A-Frame and JavaScript, Unity and C#, and/or Unreal and C++ with Blueprint
experience with and access to* AR/VR platforms and devices (ARKit/ARCore, Oculus Rift/Quest, HTC Vive, Microsoft HoloLens, etc.)
* We will consider ways of allocating and distributing AR/VR equipment so that students can work on projects remotely.
Roland Graf, associate professor at the University of Michigan Stamps School of Art & Design, and Michael Nebeling, assistant professor at the University of Michigan School of Information, with Bryan Kreps during a “play test” for iGym. (Photo by Roger Hart, Michigan Photography)
Professor Michael Nebeling was part of the U-M team that created a first version of iGYM, a system designed for school and community-based sport or recreation facilities seeking to provide novel and accessible ways for people with different levels of mobility to play and exercise together. In our current implementation, iGYM produces an interactive gaming floor for inclusive play of a “life-size version of air hockey” using a simple form of projected augmented reality.
Funded by U-M’s ESSI program, the project achieved many of the goals that are envisioned by U-M’s exercise and sport science initiative. It involved the target audience and demonstrated large impact on the users that helped us test and improve it, in this case, the kids, their friends, and their parents. The project also received a best paper award at CHI Play 2019, and wide media coverage in December 2019 (e.g., in U.S. news, ABC news, VRScout, as well as in local news papers).
Most media articles provide Professor Roland Graf‘s perspective, our friend and collaborator and the lead researcher on the project. For Professor Nebeling, it was a very rewarding project to be a part of and something he won’t forget. While it is of course nice to have helped the team win a best paper in the HCI research discipline, the professional and personal impact on Professor Nebeling was much larger than that.
First, he was able to recruit a first selection of UMSI master’s students taking his brand new AR/VR courses, SI 559 and SI 659, Pallavi Benawri and Amy Whitesall, as research assistants. Both really played a significant role in the success of this project!
Second, with most research projects, a researcher is happy when a user study produces great results and demonstrates effectiveness of the system design. With iGYM being a system that is quite complex and difficult to study given the various design parameters, it was a little different. While overall received positively, our user study revealed some mixed feelings about the system. Despite the use of algorithms and manual fine-tuning to help adapt the difficulty and balancing model of the system, it remains a challenge to make it both fun and fair, especially since with the range of parameters we can control, it was quite easy to “overbalance” the system. The results helped us understand how we could make the system better in the future, and this is what we consider a success.
But perhaps the greatest success of this work was that we could begin to see how iGYM might be adopted in the future. To this end, Professor Nebeling really enjoyed helping organize and host two play days with the larger team, including our consultant Betsy Howell, the kids, and of course their parents. It was an amazing atmosphere and really interesting to see our research prototype in action, having our participants “break test” it through creative use, and misuse, of the features and glitches of our implementation. 🙂
Many thanks to the team, especially to Roland Graf, Pallavi Benawri, Amy Whitesall, and Besty Howell.
UPDATE: Two of our papers at CHI 2020 received Best Paper Awards. Wohoo!
We are happy to announce that we have three papers conditionally accepted at CHI 2020 featuring our work on MRAT, our toolkit for recording user interaction sessions with AR/VR applications created with Unity and producing analytics that can be inspected in a dashboard or visualized in mixed reality in-situ; XRDirector, a new collaborative immersive authoring system that adapts roles from filmmaking to coordinate multiple co-located designers, with some of them working in VR and others in AR; and an interview study on key barriers to entry for AR/VR content creators with different backgrounds and levels of expertise, to guide the design of future tools specifically for end-user programming of AR/VR experiences.
M. Nebeling, M. Speicher, X. Wang, S. Rajaram, B.D. Hall, Z. Xie, A. Raistrick, M. Aebersold, E.G. Happ, J. Wang, Y. Sun, L. Zhang, L. Ramsier, R. Kulkarni: MRAT: The Mixed Reality Analytics Toolkit. In Proc. CHI 2020. Best Paper Award
M. Nebeling, K. Madier, Y. Chang, L. Zhu, M. Chung, P. Wang, J. Nebeling: XRDirector: A Role-Based Collaborative Immersive Authoring System. In Proc. CHI 2020.
N. Ashtari, A. Bunt, J. McGrenere, M. Nebeling, P.K Chilana: Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. In Proc. CHI 2020. Best Paper Award
UPDATE: The available positions have been filled for Winter 2020. We will consider new applicants for Fall 2020, and will reach out to students in August 2020.
AR/VR web developer: Help develop a set of web-based AR/VR tools that were previously published and that are now becoming part of a larger AR/VR online creation platform. The platform will be used to support a new AR/VR MOOC specialization on Coursera currently being designed and produced by Professor Nebeling with the Center for Academic Innovation. Significant experience with JavaScript is required. Experience with A-Frame and Unity is a plus.
AR/VR Unity developer: Help create a set of novel AR/VR experiences that make use of a variety of input and output technologies. Significant experience with Unity is required. Experience with 3D animation and modeling tools is a plus.
AR/VR designer: Help create interactive 3D content for a variety of new AR/VR experiences that will be created next year as part of Professor Nebeling’s research and his efforts to support the U-M wide XR Initiative. Significant experience with 3D modeling and animation tools required. Experience with UX practices and AR/VR design methods is a plus.
Positions are paid research assistant positions — while there is an expected workload of 6-10 hours per week, there will not always be an equal amount of work every week, and students will contribute to multiple projects based on their skills, rather than being assigned to one semester-long project. There will be a short interview process, and possibly a short trial period, to make sure that candidates are a fit and have the required skills. Positions may also be taken up for credit in the form of independent study.
While we are an ambitious research group, we are also a fun team of students and faculty interested in experimenting with up-and-coming AR/VR technologies. Feel free to reach out to current or previous students working in the lab to learn about their experience.
IMPORTANT: Please apply by filling in this form. Don’t just send emails to express your interest; I will not be able to respond to them. We will review the applications entered via the form and get back to you after initial review in early January 2020. Thank you for your interest!
While the specifics and strategic goals are still in development, one of the most exciting first milestones for us in the lab will be a new set of online courses that will provide significant training in design and development of AR/VR technologies, together with frameworks for ethical, social, privacy, and security aware design of AR/VR apps, as well as guidelines and recommendations for creating new AR/VR spaces for research and development as well as teaching and learning, which will be particularly helpful for resource constrained environments such as many libraries and schools. These new learning materials will significantly expand on Professor Nebeling’s AR/VR/MR Teach-Out on Coursera, which has provided an introduction to AR/VR technologies to more than 2,400 active learners world-wide.
We have also started new collaborations with several key vendors in the AR/VR space.
We are so excited by the enthusiastic support and the new movements at U-M! If you are interested in supporting our efforts, please reach out to Professor Nebeling at nebeling@umich.edu.
We’re happy to announce that Professor Michael Nebeling has received the Disney Research|Studios Faculty Award 2019.
Michael traveled to the ACM SIGGRAPH 2019 conference in Los Angeles, CA, in July and received the award at DisPLAY, the Disney Mixer social event held during the conference.
Disney Research|Studios Director and ETH Zurich Professor, Markus Gross, and Associate Director, Bob Sumner, announced the award at the event and handed it over to Michael in a very nice ceremony.
The award, “In recognition of your scientific excellence”, is a recognition of Michael’s important research at the intersection of HCI and AR/VR over the past few years, with the goal of empowering non-programmers and user experience designers to create AR/VR experiences rapidly and cheaply.
As announced earlier, several members of the Information Interaction Lab attended CHI 2019 in Glasgow, Scotland, UK. This week, the video recordings of our presentations were released:
