Date
Mar 15, 2019
Client Name
Simon Fraser University Design Research Facility
Services
User Experience Design
3D Assets Creation
Interaction and Visual Design Research
About
Simon Fraser University's Design Facility embraces students' efforts in design and discovery of novel interactions. Synesthesia VR is a prototype exploring the possibility of gaze interaction with virtual environment through colour and sound. Users are guided with colourful furniture installations followed by sounds as an audio feedback fuelling the user's interest in the space even more.
Our Clients
The esteemed clients of Simon Fraser University are researchers, undergraduate and graduate students, faculty members and Professors.
Challenge
Design and develop a Virtual Reality experience embodying user into the vision of a synesthete and their perception of Synesthesia. Wassily Kandinsky was chosen as a reference, since his research findings and documentation on synesthesia propelled the VR experience design decisions.
Research of different syntesthetes and their perception of colour to sound. The project needed clear guidelines of shapes, colours, and forms to be used in the experience. Additionally, deep research on sound libraries was required to bridge each colour & shape to sounds and tones of various instruments. For example: yellow colours would playback a range of trumpet notes.
Communicating the feeling of Synesthesia to the user inside Virtual Reality as a visual feedback was crucial to ensure the user comprehends seeing and hearing colours.
Results
Guideline on colour to sound relationship created, closely followed by the team while developing the experience.
Wassiliy Kandinsky was chosen as a role model of a Synesthete. His research and books on Synesthesia tremendously helped the team in identifying colour palette of the experience as well as the visual style of the virtual world.
Color filters were introduced as a visual playback on successful identification of shape and colour. The user would gaze at an object to see a virtual coloured overlay on their eyes approving the seen object.
> 1000 people
felt synesthesia in VR
gaze
core interaction within VR
"yellow sounds like trumpet"
most common user's reply to what yellow colour sounds like?
Process
Researching and gathering requirements to create a sense of synesthesia inside Virtual Reality. Design and prototyping of User Interactions, visual feedback, and virtual environment to embody the user into the mind of Wassily Kandinsky.
R. Murray Shafer Sound Library at SFU
Visual reference
Research literature that guide railed decision making of the team in choosing colours and sounds to recreate synesthesia in VR.
Sound Library of Classical instruments
Simon Fraser University holds one of the biggest sound libraries gladly because of R. Murray Schafer.
Our team relied heavily on the access to sound libraries to create a set of tunes for the experience.
Research and Experience Design
In this project, I lead a team of developers, 3D designers, and sound engineers to establish guidelines and rules to applying sound to colours connection.
User's interaction inside Virtual Reality was a significant key to a successful embodiment into a synesthete. The team experimented hard in search for an ideal human to computer input in VR such that the user discovers the ability to see like a synesthete.
Shape Highlight
In this preview, the gaze interaction feedback is being tested by the team. Researching the most efficient and elegant way to confirm to the user that they saw/heard the sound was a crucial point of the project.
Distance to spot a sound
What if the user is standing a little further from the objects and interacts with the environment from a far?
Can the user spot the visual feedback of small items on the bookshelf?
Balloons as a visual feedback
Adding balloons as a visual feedback to a successful interaction might be too much... Does the user embody into a synesthete? How many balloons will they see at the same time if they try and hear 10-15 objects at once?
Prototyping Interactions
Final Design
Movement on scroll
Orbit controls
Map controls
Shape Highlight
Gaze to hear a colour with a coloured filter overlay yielded as the most successful interaction from the user testers, later making its way into production.
Video showcasing the project in production. In this video, real life user interactions and emotions from the experience are captured. Some of the users were later interviewed on their understanding of synesthesia and feelings they had while being a synesthete for couple of minutes in Virtual Reality.
Experience Walkthrough
Bauhaus school in Weimar
Art Style
Wassily Kandinsky used to be a professor in Bauhaus School in Weimar, on the elements of form and colour, and directed the mural workshop.
To pay tribute to Kandinsky's influence to Bauhaus, the Art Style of Bauhaus was chosen inside virtual space.