Interview by Silvia Iacovcich
Multi-sensory artist David Yann Robert doesn’t fit the mould of the traditional artist. His kaleidoscopic career has taken many directions that ended up coexisting on the same level, fused into an eclectic process of discovery. Following the trajectory of film animation for almost a decade (where he learned to convert natural phenomena into procedural animation systems), the artist transitioned to neural networks and artificial intelligence through a scholarship in human-robot interaction design at MIT Media Lab.
Delving into Child-Robot Interaction (CRI) – a subcategory of Human-Robot Interaction (HRI) – he applied animation principles learned from movie-making to creating real robot characters while exploring the boundaries within virtual and physical by approaching the relationship between early childhood pedagogy and robotic transmedia.
Tofulandia (2009), his first contribution to mapping human-robot scenarios, works on modelling virtual identities that endure their existence in different realities. Designed to appeal to children as an immersive robot/video game, the experience focuses on building a temporary play zone where robots and humans interact in a new virtual-reality continuum.
Through a wireless joystick, we mesh with Miso, a physical robot character displayed on a 42-inch video screen which introduces us to his virtual family species, and play catch through blended reality animation models specifically designed to adapt to both universes. Experimenting with children’s imaginative abilities to co-animate new ludic systems, David investigates the impact of media content on childhood development.
The artist uses empirical research to bridge the gaps between science and esoterism, interpreting the natural kingdom as a mystical source of data we can decipher to reach a higher understanding of the macrocosm we inhabit. In Panspermia (1999), Yann Robert challenges people’s awareness, stretching the audience to realise that life is omnipresent: sentience and consciousness are not restricted to human beings – they just operate on a different timescale in nature.
Rewriting the inner biological activities of trees, he set up electrodes on the leaves, picking up the deltas and translating them into sound and graphics. The ten-year project culminated in approximately 200 synthesised sounds, using plant signals and modified animation, subsequently rearranged and composed by ambient composer Tetsu Inoue (Fax Records, DiN), a closer connection with the audience. An improvised version of it was performed live in a greenhouse in New York in 2002.
A similar approach to live performance is presented in Singing Tree (2006). The artist amplifies the biosignals of a beech tree, converting them directly into audible sound. He broadcasts over two speakers suspended in the plant – confronting the biological world using music as a powerful tool.
Yann Robert attributes a living soul to all-natural phenomena, placing animism at the core of his ethos. Life encircles us even when we don’t see it: using AI, science and sound, he gives a voice to the invisible sentient world for us to perform.
Animated Matter is a project that encapsulates this notion entirely. Envisioned in collaboration with composer/vocalist Hannah Elizabeth Cox, the album was conceived during wild-minded conversations that involved extraterrestrial entities, non-human forms of consciousness, music and plants.
The debut album dives into the Scandinavian, Scottish, and the Celtic mythical figure of the Selkie, a therianthrope being, half-human/seal, and its tragic fall into romance, expressed in sound textures and hypnotic vocals that lead us to an allegorical, dreamlike dimension.
The eight tracks are ambient lullabies that trigger the subconscious, a mesmeric/psychedelic introspective journey into the inconsistency of matter and our multi-layered essence. Encouraging ‘horizontal’ listening, the album invites the listener to get lost in the textures of sounds, abandoning our rationality and surrendering to the forces of the world.
Right: Animated Matters
For your audience that is not familiar with your work, could you tell us a bit about your background and inspirations?
Thanks for the question. I’m American/French/Vietnamese but I grew up in Brazil and have been living in the US for most of my adult life. When I was 19, I started making ambient music with my close friends and performed (as C-Lab) up and down the west coast but mostly in the SF Bay Area.
A couple of years later, I started doing commercial work in the animation/film industry and stuck with it for about a decade. I did character animation and technical direction for early computer-animated features. For the most part, I focused on studying and mimicking natural phenomena using procedural animation systems. It was a dream job, and I’ve always been grateful to my mentors, Richard Hollander, Jarrett Smith and Greg Hermanovic, who supported my development in that field.
Around 2010, I got a generous scholarship to do graduate studies in human-robot-interaction design at the MIT Media Lab. This thinks tank supports diagonal thinkers that don’t easily fit into one box or discipline.
As an artist with conceptual and technical skills, I felt fortunate to be admitted into a top robotics research lab based on a shared idea that Cynthia Breazeal (prominent social roboticist) and I were excited about the future of animation being off the screen.
She supported me in applying the algorithmic methods I learned in the computer graphics industry to animate physical robots and give them the illusion of life.
To earn a graduate degree and fit in at MIT, I had to reinvent myself, learn how to design and fabricate robots, deepen my engineering and research skills, and work harder than I had ever worked on the most stressful movie schedule.
I made some great friends that taught me a lot, and I had an epic adventure in Academia. Despite the challenges, my experience at the Media Lab changed me forever and helped me integrate the two sides of myself. On one hand, I was a research scientist inventing mixed reality robot games for children.
At the same time, I poured myself into the Art, Culture & Technology department to take advantage of the unique opportunity to train in performance art with Joan Jonas and refine my conceptual process through mentorship by artists Gedminas Urbonas and Antonio Muntadas. For the past twenty years, I’ve had an active art research practice that’s helped me explore questions that art or science on their own can not fully answer. I’m a blend of things, but primarily I’m curious.
The virtual and the physical intersect in your works (Human-Robot scenarios series). What is your standpoint towards the collapse of the boundary between them?
The idea of porous boundaries between realities makes sense to me. A visual that comes to mind is the cult animation by Roger Mainwood (Halas and Batchellor Studio) for Kraftwerk’s Autobahn, completed in 1979. Have you seen it (check out around 8m30sec) for the being slipping between wormholes?
In the next couple of years, many people are going to experience mixed reality as versions of it emerge out of labs and product design studios. Spatial computing is already really useful to overlay information on top of fact as we know it.
Still, my personal research interest is in how augmented and mixed reality can be paired with autonomous moving physical objects (e.g. robots). In those kinds of systems, the boundary between the virtual and physical can essentially be erased by modelling a kind of super physics (of multiple realities) that ensures kinetic and visual continuity across all spaces.
The trick is, to begin with, the conceptual grounding that both realities can blend into one and then develop the methods to make that happen.
For example, suppose an object moves in a virtual world, and you want that motion to transition to a moving physical object seamlessly. In that case, you’ll need to calculate the virtual one’s speed and trajectory to continue the movement in the material world with the correct acceleration/velocity so that it’s perceived as one fluid motion.
If you can fine-tune all the parameters necessary to ensure the transition across the boundary is seamless, the boundary between physical and virtual can effectively disappear. The exciting thing is that it’s wide open for experimentation by anyone.
It’s often the case that new techniques, at first, are controlled by the few that know how to make the magic happen. Still, once more artists get a hold of the tools (or build them themselves), things usually develop quickly in more exciting ways due to the very nature of the questions raised through the artistic process.
I think about choreographing blended reality experiences as a performance art & scenography challenge. It requires you (as the director) to think about agents’ (including human) bodies’ extension into other realities and use the new medium in unique ways to its capabilities for expression and representation.
What is your thinking process behind making the technological experience physical?
If by technological experience becoming physical, you mean creating a robot or embedding an AI into a body: I start by thinking about the AI’s purpose and what kind of character, back story, and body type and affordances it might need to accomplish its goals in the physical world.
My robot character design preferences are that an individual robot body should only contain one personality that slowly learns and adapts through interactions in the world. Although it’s technically feasible for various agents to share a body, I design robots to help people. Hence, I put my focus on human needs first and design human-robot interactions that support the development of positive parasocial relationships.
While I was in the Personal Robots Group’s lab at MIT, I recorded some voice memos that I later turned into a spoken word techno album. When I hear it now, I smile and cringe. Still, it accurately documents my obsessions at the time and ideas on the importance of style, character and emotions when designing robots and human-robot interactions.
Speaking more generally about making technological experiences physical: our bodies have evolved to survive and attend to physical needs by quickly picking up cues and signals. That’s why there’s emotion encoded in different types of motion. Similarly, there’s meaning in the context of interaction, and it’s very important to set up people’s expectations correctly.
A good starting point for designing physical experiences is to understand the relationship of the body in space and the evolutionary responses of human physiology. I think about how a technical system might map to the human sensory spectrum and play with those mappings until it feels right to me or makes me wonder what’s going on. It’s a process of discovery.
Trees and plants have a strong presence in your work. Can you share your ideas on the relationship between sound and trees and plants and the importance of their physical presence in your works?
Trees and plants are important to me because their type of intelligence is fascinating. Trees age well (without moving) and thrive in a networked biological environment, processing and remembering complex chemical messages being passed through the air and under the ground mediated by other organisms. My work with trees and plants started 20 years ago, and I’m not the first or last to eavesdrop on plant communication, but it’s the kind of research that keeps me endlessly curious.
Each time I create a new entry into this body of work which I approach as interspecies collaborations (first translating tree activity to audio, later to video, then listening to trees communicate with each other), I learn something exciting.
I am in awe about how little we know. Our relative ignorance is likely due to the much slower time scale that plants and trees operate than human time, our lack of patience and popular limits on what we usually consider sentient.
Our understanding has only recently begun to grow as the science of plant signalling and behaviour has gained mainstream attention and respect after being misunderstood by scientists with anthropocentric biases.
In response to the scientific community’s shortsighted rejection of early ideas on the topic, I did a performance lecture piece a few years ago honouring Bengali polymath and biophysicist Sir Jagadish Chandra Bose, one of the first (besides Charles Darwin and his son Francis) to study the minute movement in plants over a hundred years ago.
To me, science doesn’t always have all the answers, and that’s why we need more artists to lead research and ask the questions that scientists are curious about but can’t get the funding to look into or don’t want to risk their professional reputations on. How else are we going to create knowledge for the benefit of all Earth kind? Species thinking has led us into the precarious situation we find ourselves in now.
It’s going to be hard to improve the whole planet’s health if we continue to put our species first all the time. My work with trees or plants is about raising awareness and hopefully shifting people’s interactions with the environment.
My explorations into plants and trees signalling progressed unexpectedly. I initially took a technological approach by replicating early experiments and building custom electronic sensors and software. Nowadays, you can buy a product or box that does it for you, which is more convenient but constrains the interpretation of the plant’s signals to the provided sensor setup and manufactured design.
Every time a new plant music project comes out. I’m really happy because it’s the spread of the idea that matters to me. My most intimate encounters with trees, however, have been through direct contact, and tree-whispering sessions. A mentor who practices this form of spirituality shared a protocol and taught me how to ask permission and connect with trees without tech.
Animated Matter is your new project in collaboration with composer/vocalist Hannah Elizabeth Cox. Could you share with our audience a bit about the project, and how the collaboration happened? Also, what’s the story and symbolism behind the Animated Matter name?
Hannah and I met six years ago, backstage at a Flying Lotus concert in Boston. I knew Steven for several years through my close friend and recording artist Mophono in SF, who hosted him and other beat scene artists at Change the Beat, the most progressive and best thing happening in the Bay Area at the time. When I met Hannah, we were both admiring another artist’s (Strangeloop’s) sketchbook filled with extra-terrestrial drawings.
We started talking about music and plants, and non-human forms of consciousness. She lived about an hour away and would come to my studio in Chinatown on the weekend to play music. We would play all night and record around 4 am as we drifted between waking and getting close to falling asleep over our instruments and the studio gear.
Hannah has a beautiful voice, and this amazing sense of musical timing-it reminds me of the feeling I get when I watch Tarkovsky movies, the feeling that everything is slowing down to a crawl. We became friends, and I brought her on as an advisor for a commission I was doing for the Art & Nature Center of the Peabody Essex Museum in Salem, MA.
Since Hannah grew up around Maine’s forests, I asked her to help me find the specific tree I ended up collaborating with for an Earth Day program of “Tree eavesdropping” and a “Tree jam” piece that included local musician friends.
Animated Matter was formed to create ambient lullabies for the subconscious. We make music to fall asleep to. So far, we’ve heard good feedback from children and adults who agreed to sleep test Side B of the Selkie album before releasing it more widely to the general public on 02/12/21.
We’re currently developing a series of horizontal listening experiences for comfortable environments augmented by barely noticeable animated lights and occasional projections. The name Animated Matter relates to a contemporary form of animism that we feel inspires the music and art we make. The name is also related to my lifelong interest in expanding and experimenting with animation techniques.
Selkie is the first album produced with Animated matter. What was the creative and intellectual process behind the inception of this album? Was it also very much intertwined with the collaboration itself or appeared posteriorly?
Selkie was Hannah’s idea for a theme, but the deeper we got into it, the more I realised how personally connected to the myth we both are. The story of a shapeshifting seal/woman features prominently in both of our families’ Celtic roots.
I’ve always been fascinated with the blurry boundary between pre-history and myth and the symbolic purpose behind retelling old stories. My performance art mentor, Joan Jonas, was the one who pointed me in the direction of exploring the layers of the ocean as metaphors for different levels of conscious and unconscious awareness.
That idea brought together Animated Matter’s purpose with the Selkie’s origin, so we incorporated it and developed it further through the sound’s layered textures and motion. This album’s creative process involved making the material together over several recording sessions spanning multiple years and then sushi-cutting it down to the final tracks and mastering it with someone we both deeply respect: Rafael Anton Irisarri. He was kind enough to take our project on.
Hannah and I both knew what colours the music evoked and picked a beautiful photograph of the ocean sky in Maine captured by artist and photographer William Pearce Cox for the album cover.
What were the challenges for its production?
There were technical challenges during our first round of mastering. I was inexperienced and on a strict budget, so I first took the album to someone I didn’t know who advertised inexpensive mastering but had mostly worked on radio-play hip hop. He essentially pumped up all the tracks and gave me many loud, clipped files and called it done.
I learned the hard way that it’s important to take the time to find someone that understands and cares about the genre you are making and even better if they compose in a similar style. When Raf entered the picture, we were so happy with his expertise and encouragement. He produced the final digital and vinyl masters heard on the album today.
Over the multiple years, it took to finish this album. Personal transformations came about from working together with a myth about shapeshifting, which seemed to affect us both internally in different ways. In the end, I’m grateful for all the challenges because they were opportunities to grow and change, and thankfully our friends and family were there to help us through.
Self-releasing the Selkie album was challenging, but it helped us quality control the entire process, make adjustments where needed, and value the people and processes behind the scenes.
Animism (that sentience permeates all forms of matter) is strongly represented in Selkie. Could you expand on what animism represents for you?
Animism represents my own belief that the material and spiritual are not separate but rather that minerals, plants, mountains, rivers, oceans, animals, the planet and everything in the universe is sentient in a unique way. I don’t believe personhood is exclusively human. Life is primarily a process, regardless of what form it takes.
The danger of having limited ways of understanding and describing life makes it harder to identify it in all its potential forms correctly. Animism proclaims that non-human personhood is a possibility that changes and improves your relationships and interactions with everything around you if you believe it is true.
What is your chief enemy of creativity?
I’m finding it hard to think of one. If there are enemies of creativity around or inside me, I soon try to find ways to convert them into allies somehow. There’s always something to learn from the opposition, even if it’s hard or not fun to do.
You couldn’t live without…
Touch.
