Szabo, Peter, Simiscuka, Anderson Augusto ORCID: 0000-0002-0851-2452, Masneri, Stefano ORCID: 0000-0003-1965-5704, Zorrilla, Mikel ORCID: 0000-0003-2589-2490 and Muntean, Gabriel-Miro ORCID: 0000-0002-9332-4770 (2022) A CNN-based Framework for Enhancing 360° VR Experiences with Multisensorial Effects. IEEE Transactions on Multimedia . ISSN 1941-0077
Abstract
Improving user experience during the delivery of
immersive content is crucial for its success for both the content
creators and audience. Creators can express themselves better
with multisensory stimulation, while the audience can experience a higher level of involvement. The rapid development of
mulsemedia devices provides better access for stimuli such as
olfaction and haptics. Nevertheless, due to the required manual
annotation process of adding mulsemedia effects, the amount
of content available with sensorial effects is still limited. This
work introduces an innovative mulsemedia-enhancement solution
capable of automatically generating olfactory and haptic content
based on 360° video content, with the use of neural networks. Two
parallel neural networks are responsible for automatically adding
scents to 360° videos: a scene detection network (responsible
for static, global content) and an action detection network
(responsible for dynamic, local content). A 360° video dataset with
scent labels is also created and used for evaluating the robustness
of the proposed solution. The solution achieves a 69.19% olfactory
accuracy and 72.26% haptics accuracy during evaluation using
two different datasets.
Metadata
Item Type: | Article (Published) |
---|---|
Refereed: | Yes |
Uncontrolled Keywords: | Multisensory media, neural networks, image recognition, olfaction, haptics, machine learning |
Subjects: | Computer Science > Image processing |
DCU Faculties and Centres: | DCU Faculties and Schools > Faculty of Engineering and Computing > School of Electronic Engineering Research Institutes and Centres > INSIGHT Centre for Data Analytics |
Publisher: | Institute of Electrical and Electronics Engineers |
Official URL: | http://dx.doi.org/10.1109/TMM.2022.3157556 |
Copyright Information: | © 2022 The Authors. |
Funders: | EU Horizon 2020 Research and Innovation programme under Grant Agreement no. 870610 for the TRACTION project, Science Foundation Ireland (SFI) Research Centres Programme Grant Numbers SFI/12/RC/2289_P2 (Insight) and SFI/16/SP/3804 (ENABLE) |
ID Code: | 26879 |
Deposited On: | 01 Apr 2022 11:13 by Anderson Augusto Simiscuka . Last Modified 24 Mar 2023 14:25 |
Documents
Full text available as:
Preview |
PDF
- Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0 17MB |
Metrics
Altmetric Badge
Dimensions Badge
Downloads
Downloads
Downloads per month over past year
Archive Staff Only: edit this record