← Back to projects

Les choses qui s'en vont

Year : 2019
Category : Exhibitions & Immersion
Les choses qui s'en vont
Interactive Installation Body Tracking Young Audience Unity RealSense & Nuitrack Sound Design Museography

Les Choses qui s'en Vont (Musée de Poche)

Client: Musée de poche | Artwork: Beatrice Alemagna | Studio: Crafteke | Role: artwork preparation, Unity development, body tracking and sound design


A large format projected installation for the final room of the exhibition. Two illustrations from the book come alive under the children's gestures, captured with no controller and no touchscreen: the body is the only interface.

2 interactive scenes | Contactless skeleton tracking | Touring exhibition

Context & Challenges

  • Alive even with no audience: the installation had to run continuously in the room, image and soundtrack, behaving as a self sufficient piece of media. Interaction is not a prerequisite but a bonus: it wakes up when a child steps into frame.
  • Extending the gesture of the book: in the album, the child turns a tracing sheet to animate the image. The installation had to carry that principle over to the scale of the body, without adding anything foreign to the work.
  • An audience that does not read instructions: young children facing a projection with no button and no screen. Everything had to be understood by watching others do it.
  • Respecting a strong graphic world: Beatrice Alemagna's drawings are made with pencil, pen and brush. Any visible digital intervention would have betrayed the material.
  • Bodies too small for the sensor: a depth camera partly infers distance from the apparent size of a body, so a child reads as a more distant adult. Add to that bodies that move fast and are rarely alone in frame.

Engineering & Method

  • Artwork preparation: cleaning the album scans and creating the few animation frames needed, staying as close as possible to the original line.
  • Contactless capture: an Intel RealSense camera and the Nuitrack API for skeletons, with nothing worn or touched by the child. The rig is three elements: a projector, a projection surface, and the camera sitting at the foot of the screen.
  • Scene logic in Unity: two tableaux alternating automatically, each with its own interaction rules and sound response.
  • Continuous soundscape: birdsong at rest, wingbeats on takeoff, rainfall then wind as the rain moves away.

Technical Breakdown

1. Making skeleton tracking hold on children's bodies

Challenge: a depth camera partly infers distance from the apparent size of a body. Children being smaller than adults, the engine kept placing them too far away and the skeleton dropped out. Add constant movement, several children in frame at once, and sometimes a parent carrying one.
Solution: an Intel RealSense camera with the Nuitrack API, whose detection parameters were reworked until the skeleton held on children's bodies. This is the foundation for everything else: without reliable head and hand joints, neither interaction works.

2. The bird: rewarding stillness

Challenge: convey without a single word that you do not catch a bird by running after it.
Solution: the bird flies freely between perches. When a child reaches out, hand higher than the other, and holds the pose for about two seconds, the bird comes to land on it. Conversely, the group's agitation feeds a nervousness counter: past a threshold the bird takes off, and sharp movements interrupt its song. The installation rewards slowness, which is rare in a piece for children.
Bird scene: an outstretched finger and the bird approaching
The bird scene: the outstretched finger calls, agitation drives it away. Illustration by Beatrice Alemagna.

3. The rain: building a shelter with your body

Challenge: give the child a physical hold on something falling from the sky, without handing them a button to stop it.
Solution: the same nervousness counter brings the rain in, curtain after curtain, and the mass of water follows the group's average movement. Once the downpour is established, an invisible volume locks onto each child's head: raising both hands above your head cuts the drops passing through it. You do not command the rain, you make yourself a roof with your arms.
Rain scene: curtains of drops and a figure under an umbrella
The rain scene, in successive curtains. Illustration by Beatrice Alemagna.

4. A hand drawn instruction panel

Challenge: explain two gestures to an audience that cannot read yet, without putting text on the projection or breaking the album's world with standard pictograms.
Solution: a panel drawn entirely by hand, imitating Beatrice Alemagna's line as closely as possible, then laser engraved. Four vignettes, two per scene, show the gesture and its effect. No words are needed, and the signage belongs to the same world as the projected images.
The laser engraved wooden panel, propped on the floor in front of the projection
The engraved panel, simply propped on the floor in front of the projection.
Laser engraved panel showing the gestures for both scenes
The original drawing: point or wave for the bird, shake or raise your hands for the rain. Made for the project, in the style of the album.

5. The installation running

Note: the installation was not documented in operation, as filming children in the gallery raises image rights questions. The sequence below is recorded from the real time engine, with its soundtrack.
Both scenes in sequence. Best with sound on: the soundtrack is part of the work and runs continuously in the room, even with nobody there.
Illustrations: Beatrice Alemagna, from the album Les choses qui s'en vont.
Exhibition: conceived and produced by the Musée de poche.
Installation development: Sébastien Lebon, at Crafteke studio.
Note: visuals rendered from the real time engine, the installation not having been documented in operation.