INFOMMMI (Multimodal Interaction)

3rd quarter, 2024/2025, timeslot D, 7.5 ECTS

Lectures 1-4 will be given by Peter Werkhoven and address human visual, auditory and tactile perception and the use of its potential in designing novel interfaces for interacting with virtual worlds. Wolfgang Hürst will address Augmented Reality and related applications in lectures 5-7.

topics

The lectures will address the following topics:

some basics of visual, auditory and tactile perception and the effects of combining them (multimodal presentation and related interface design guidelines)
visual communication interfaces in virtual worlds (effects of non-verbal facial communication)
navigation interfaces in virtual worlds (head tracked visualization and 'cyber sickness')
manipulation in virtual worlds (traditional mouse-cursor interfaces versus virtual hand control)
virtual worlds through mobile displays (scrolling interfaces versus virtual windows)
emerging interface technology (synaesthetic media and brain machine interfaces)
augmented reality (technologies, human factors, interaction)

For related lecture dates, please refer to the course schedule. Slides from the lectures will be posted below afterwards.

Slides from the lectures

Introduction (W. Hürst)

Lecture 1: Introduction & vision, motion, stereo (P. Werkhoven)

Lecture 1 - Introduction, formalities (PDF)
Lecture 1 - Vision, motion, stereo (PDF)

Lecture 2: Hearing, touch, multimodal (P. Werkhoven)

Lecture 2 - Hearing, touch & multimodal (PDF)

Lecture 3: VR technology, human factors (P. Werkhoven)

Lecture 3 - VR technology, applications & control (PDF)

Lecture 4: Emotions, BCI & cyborgs (P. Werkhoven)

Lecture 4 - Emotions, BCI (PDF)

Lecture 5: Augmented reality - Introduction & tracking (W. Hürst)

Lecture 6: Augmented realitly - Displays (W. Hürst)

AR - 2 Displays (PDF)
(Video 1: Layar AR browser) watch for yourself if you want (optional)
Video 2: ARDefencer trailer
Video 3: Natural Feature Tracing at 30Hz on a Mobile Phone
Video 4: KLM AR app for luggage size check (website, video)
Video 5: 3D Virtual tracking and depth perception problem in mobile AR
Video 6: Cardboard AR Headset - Aryzon DIY Kit
Video 7: Light Field Technology: the Future of VR and 3D Displays. Watch the following parts:
- 0:53 min - 4:50 min, VR/AR displays introduction and current state
- 4:51 min - 5:38 min, Technology challenges
- 5:39 min - 8:38 min, Vergence-accommodation conflict
- 8:39 min - 13:29 min, Light field stereoscope
Feel free to watch the remaining part (starting at 13:30 min), too, but it is not expected.
(Video 8: Nvidia Near-Eye Light Field Display) watch for yourself if you want (optional)
Video 9: How does a light field camera work?
Video 10: Tilt Five Augmented Reality Glasses
Video 11: CastAR Augmented Reality - Through-glasses gameplay
Video 12: IllumiRoom by Microsoft Research

Lecture 7: Augmented reality - Interaction (W. Hürst)

AR - 3 Interaction (PDF)
Video 1: ARDefender (interaction; starts at 0:37 min)
Some examples for advanced VR input devices:
Video 6: Human-Computer Interaction at the Wrist
Video 7: Hyper Reality
Video 8: Augmented (hyper)Reality: Domestic Robocop
Playlist, illustrating several VR interaction techniques, including:
- Video 9: Pointing Metaphor - RayCasting
- Video 10: Pointing Metaphor - Flashlight
- Video 11: Pointing Metaphor - Aperture Selection
- Video 12: Pointing Metaphor - Image Plane
- Video 13: Grasping Metaphor - 3D Bubble Cursor
- Video 14: Pointing Metaphor - Depth Rays
- Video 15: Grasping Metaphor - GoGo Interaction
- Video 16: Indirect Metaphor - World in Miniature
- Video 18: Indirect Metaphor - Voodoo Doll
- Video 19: Hybrid Metaphor - Homer

Q&A (lectures 5-7, W. Hürst)

Q&A session (PDF)

papers to read

Multimodal perception: (*)

Paper 1: Ernst, M.O. & Bulthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162-169.
PDF (password protected)
Paper 2: Petkova, V.I., & Ehrsson, H.H. (2008). If I Were You: Perceptual Illusion of Body Swapping. PLoS ONE 3(12): e3832. doi:10.1371/journal.pone.0003832
PDF (password protected)

Navigation in (mobile) virtual environments:

Paper 3: Wolbers, T., Hegarty, M. (2010). What determines our navigational abilities? Trends in Cognitive Sciences, 14(3), 138-146
PDF (password protected)

Emotions in virtual environments:

Paper 4: Toet et al. (2022). Towards a multiscale QoE assessment of mediated social communication. Qual User Exp 7, 4.
Paper 5: Veelen, N. van et al. (2021). Tailored Immersion: Implementing Personalized Components Into Virtual Reality for Veterans With Post-Traumatic Stress Disorder. Frontiers in Virtual Reality, 2 (art 740795).

Emerging technologies:

Paper 6: Lee, V.K., Nau A.C., Laymon, C., Chan K.C., Rosario B.L., Fisher C. (2014). Successful tactile based visual sensory substitution use functions independently of visual pathway integrity. Frontiers in Human Neuroscience, 8, 291.

Augmented reality:

Paper 7: R. Skarbez et al. (2021). Revisiting Milgram and Kushino's Reality-Virtuality Continuum, Frontiers in Virtual Reality, Vol. 2, 2021. PDF (password protected).
Paper 8: J. Bailenson et al. (2024). Seeing the World through Digital Pisms: Psychological Implications of Passthrough video Usage in Mixed Reality, Journal of Technology, Mind, and Behavior PDF (password protected). See also the accompanying video.
Paper 9: J. Lee et al. (2022). User Preference for Navigation Instructions in Mixed Reality, IEEE VR 2022 conference. PDF (password protected). See also the accompanying video.

exams from previous years (part 2, lectures by W. Hürst)

The exam will be a closed book digital exam. For part 1 (lectures 1-4 by Peter Werkhoven), please refer to the example questions that Peter showed in his first lecture. For part 2 (lectures 5-7 by Wolfgang Hürst), you find the exam questions of previous years below.

Important: Be aware that the ones from 2020 and 2021 were open book exams (due to COVID, they were done from home). Also, different aspects have been covered in the lectures of previous years. Finally, before 2021, parts 1 and 2 were weighted 60-40. Since 2021, it is 50-50.