Lead Experience Designer

Lead Experience Designer

Lead Experience Designer

Lead Experience Designer

Synesthesia VR

Synesthesia VR

Synesthesia VR

Synesthesia VR

Structural Color Gallery, Brooklyn, NY
Structural Color Gallery, Brooklyn, NY
Structural Color Gallery, Brooklyn, NY


Mar 15, 2019

Client Name

Simon Fraser University Design Research Facility


User Experience Design

3D Assets Creation

Interaction and Visual Design Research


Simon Fraser University's Design Facility embraces students' efforts in design and discovery of novel interactions. Synesthesia VR is a prototype exploring the possibility of gaze interaction with virtual environment through colour and sound. Users are guided with colourful furniture installations followed by sounds as an audio feedback fuelling the user's interest in the space even more.

Our Clients

The esteemed clients of Simon Fraser University are researchers, undergraduate and graduate students, faculty members and Professors.


Design and develop a Virtual Reality experience embodying user into the vision of a synesthete and their perception of Synesthesia. Wassily Kandinsky was chosen as a reference, since his research findings and documentation on synesthesia propelled the VR experience design decisions.

Research of different syntesthetes and their perception of colour to sound. The project needed clear guidelines of shapes, colours, and forms to be used in the experience. Additionally, deep research on sound libraries was required to bridge each colour & shape to sounds and tones of various instruments. For example: yellow colours would playback a range of trumpet notes.

Communicating the feeling of Synesthesia to the user inside Virtual Reality as a visual feedback was crucial to ensure the user comprehends seeing and hearing colours.


Guideline on colour to sound relationship created, closely followed by the team while developing the experience.

Wassiliy Kandinsky was chosen as a role model of a Synesthete. His research and books on Synesthesia tremendously helped the team in identifying colour palette of the experience as well as the visual style of the virtual world.

Color filters were introduced as a visual playback on successful identification of shape and colour. The user would gaze at an object to see a virtual coloured overlay on their eyes approving the seen object.

> 1000 people

felt synesthesia in VR


core interaction within VR

"yellow sounds like trumpet"

most common user's reply to what yellow colour sounds like?


Researching and gathering requirements to create a sense of synesthesia inside Virtual Reality. Design and prototyping of User Interactions, visual feedback, and virtual environment to embody the user into the mind of Wassily Kandinsky.

books of Kandinsky on Synesthesia

books of Kandinsky on Synesthesia

R. Murray Shafer Sound Library at SFU

Visual reference

Research literature that guide railed decision making of the team in choosing colours and sounds to recreate synesthesia in VR.

Sound Library of Classical instruments

Simon Fraser University holds one of the biggest sound libraries gladly because of R. Murray Schafer.

Our team relied heavily on the access to sound libraries to create a set of tunes for the experience.

Research and Experience Design

In this project, I lead a team of developers, 3D designers, and sound engineers to establish guidelines and rules to applying sound to colours connection.

User's interaction inside Virtual Reality was a significant key to a successful embodiment into a synesthete. The team experimented hard in search for an ideal human to computer input in VR such that the user discovers the ability to see like a synesthete.

Shape Highlight

In this preview, the gaze interaction feedback is being tested by the team. Researching the most efficient and elegant way to confirm to the user that they saw/heard the sound was a crucial point of the project.

Distance to spot a sound

What if the user is standing a little further from the objects and interacts with the environment from a far?

Can the user spot the visual feedback of small items on the bookshelf?

Balloons as a visual feedback

Adding balloons as a visual feedback to a successful interaction might be too much... Does the user embody into a synesthete? How many balloons will they see at the same time if they try and hear 10-15 objects at once?

Prototyping Interactions

Final Design

Movement on scroll

Orbit controls

Map controls

Shape Highlight

Gaze to hear a colour with a coloured filter overlay yielded as the most successful interaction from the user testers, later making its way into production.

Video showcasing the project in production. In this video, real life user interactions and emotions from the experience are captured. Some of the users were later interviewed on their understanding of synesthesia and feelings they had while being a synesthete for couple of minutes in Virtual Reality.

Experience Walkthrough

Want to mix realities together? Reach out to me:)


©2024 All Rights Reserved. Designed with 🤍

Want to mix realities together? Reach out to me:)


©2024 All Rights Reserved. Designed with 🤍

Bauhaus school in Weimar

Art Style

Wassily Kandinsky used to be a professor in Bauhaus School in Weimar, on the elements of form and colour, and directed the mural workshop.

To pay tribute to Kandinsky's influence to Bauhaus, the Art Style of Bauhaus was chosen inside virtual space.