MMX-VR: Multimedia Metadata Exploration in Virtual Reality

Virtual reality (VR) presents a new approach for users to interact with computer systems and applications. However, due to its recent emergence, the designers of complex systems are still learning how to design and optimise the user experience for VR. At the same time, multimedia analytics, a field aimed at analysing and understanding multimedia collections, has been rapidly developing. The project proposed will explore a novel approach to multimedia analytics using a VR interface. Specifically, it is our aim to build a system that provides interactive visualisations in VR, yet is capable of scalable data management of large multimedia collections. We also intend to show that with such a system we can provide a superior solution to the diverse and dynamic needs of multimedia experts, especially in interactive analysis of large collections. We propose to extend an existing data model for multi- dimensional exploration by integrating traditional similarity search and interactive learning approaches into both the multimedia server software and the virtual-reality based collection explorer. The main research outcomes of the project will be a set of research articles describing a multimedia analytics platform that can be applied to various large multimedia applications. We will perform user studies of selected multimedia analytics applications and extensive performance measurements of the underlying query processing algorithms.

See also:

• Funded PhD project on Virtual Reality Exploration of Multimedia Metadata Space
• BSc and MSc project ideas

The MMX-VR project is supported by Icelandic Research Fund Grant 239772-051.

Investigators

Principal Investigator: Björn Þór Jónsson, Reykjavik University, Iceland.
Co-Investigator: Aaron Duane, Churney, Denmark.
Co-Investigator: Martin Aumüller, IT University of Copenhagen, Denmark.
Co-Investigator: Gylfi Þór Guðmundsson, Reykjavik University, Iceland.
Post-Doctoral Researcher: Omar Shahbaz Khan, Reykjavik University, Iceland.

Background Readings

Data Model:

• Björn Þór Jónsson, Grímur Tómasson, Hlynur Sigurþórsson, Áslaug Eiríksdóttir, Laurent Amsaleg, Marta Kristín Lárusdóttir. A Multi-Dimensional Data Model for Personal Photo Browsing. Proceedings of the International Conference on MultiMedia Modeling (MMM). Sydney, Australia, January 2015. [PDF]
• Snorri Gíslason, Björn Þór Jónsson, Laurent Amsaleg. Integration of Exploration and Search: A Case Study of the M³ Model. Proceedings of the International Conference on MultiMedia Modeling (MMM), Thessaloniki, Greece, January 2019. [PDF]

Software:

• Aaron Duane, Björn Þór Jónsson. ViRMA: Virtual reality multimedia analytics at the Video Browser Showdown 2022. Proceedings of the International Conference on MultiMedia Modeling (MMM). Phu Quoc, Vietnam, June 2022. [PDF]
• Aaron Duane, Björn Þór Jónsson. ViRMA: Virtual Reality Multimedia Analytics at LSC 2021. Proceedings of the ACM Workshop on Lifelog Search Challenge, LSC@ICMR 2021. Taipei, Taiwan, August 2021. [PDF]
• Grímur Tómasson, Hlynur Sigurþórsson, Björn Þór Jónsson, Laurent Amsaleg. PhotoCube: Effective and Efficient Multi-Dimensional Browsing of Personal Photo Collections. Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR), Trento, Italy, April 2011. [PDF]

General Background:

• Björn Þór Jónsson, Marcel Worring, Jan Zahálka, Stevan Rudinac, Laurent Amsaleg. Ten Research Questions for Scalable Multimedia Analytics. Proceedings of the International Conference on MultiMedia Modeling (MMM), Special Session on Perspectives on Multimedia Analytics, Miami, FL, USA, January 2016. [PDF]
• Aaron Duane, Björn Þór Jónsson, Hyowon Lee, Cathal Gurrin. LAD: An Application Design Model to Support the Analysis of Large-Scale Personal Data Collections Generated by Lifelogging. Accepted to Personal and Ubiquitous Computing, April 2023. [ Available upon request ]