Interactive Installation, Visual Programming, Speech Interactive

Interactive Shadow Installation

An audio interactive shadow that responds and sings songs for you





Sound & Visual
Interactive Media Art
Digital Installation

March - April 2022

Personal project

Touch Designer/
Microsoft Kinect/
Voice Tracking/
MaxMsp/Projector/ Miro

My responsibility

Ideation, prototype and present a real-time interaction performance
This individual project is developed based on my personal interest to have a responding shadow.

Design process


About Shadow
The project uses both visual and verbal interaction shows the possibility of amity between the shadow and human through dialogues, expressions and vocabularies.
Imagine talking to my shadow is one superpower that I have obtained, how can I communicate with and understand my shadow and what kind of relationship can we maintain?
Can shadow becomes an extended interface of the body?
Create a shadow that can respond to my request, and sing songs for me.
Design Preview

Responsive shadow
“My body shadow is a friend of mine. Naturally, people might assume that their shadows are insubstantial compared to the body and consciousness. But I disagree. As I spent more time observing shadows and, one day, I found out that I obtained the superpower to communicate with my shadow using the shadow language. My shadow is a good singer, and has the singing skill one could never achieve. The shadow helped and supported singing performances whenever I asked.”

The shadow has become an extended interactive interface or space of the main body.  The shadow on the wall reveals a possible connection between the body and it as it moves according to the body motion. Yet, the relationship between human and shadow is blurred as perception, self-recognition and ownership is questioned through the performance.

Have you talked to your shadow?
Are shadows presence or absence?

Ideation and Design

Interaction Mapping
The project is consist of two interactions: physical and speech based.  

In the physical interaction, audience can see a real-time interactive shadow that follows body movements.
In the speech interaction, audience can verbally communicate with the shadow and ask the shadow to sing different songs for them.

Prototyping - Visual
Using Touch Designer, I am able to create the visual representation of the shadow.

Rather than having a light source going past the edge of the body, a projector is used as a window showing the computerized figure using camera capture to mimic the body shadow. The figure of shadow camouflages itself in designed shape and movements. The projector becomes the magnification of the illusory, which further questions the boundary between real and projective.
Shadow Mode I (left):
Colorful shadow represent the active mode:
the shadow is responding to the speech

Shadow Mode II (right):
Colorless shadow represent the inactive mode:
there is no speech/request,
the shadow remains inactive

A time lag used to enhance the relationship between me and the show,
showing the shadow as a follower of the body movement.

Prototyping - Audio
I used machine learning speech recognition technique to track my speaking, and send signals to Maxmsp to activate different responses based on different requests.
I used p5.speech and to achieve speech recognition and use words as a trigger to activate music in Maxmsp.
Screen hots from Maxmsp and Visual Studio
a live recorded experiment of using words to change the song the shadow sing

Audio and Visual Interactive
The visual and verbal interaction shows the possibility of amity between the shadow and human. It is constructed through dialogues, expressions and vocabularies.
When words are said, it triggers the music and also color changes of the shadow from grey to colorful.

Final Installing