Human Resource

Partners

John Butler / Butler Brothers

Curator Tommaso Ranfagni / Glasgow University / Glasgow School of Art:

Performers Martina Morger, Wassili Widmer, Alexandra Santos Ocasio and Zhuojun Sun

Challenge

Our team had the pleasure to support digital artist and filmmaker John Butler of The Butler Brothers as ISO’s first Artist in Residence. Assisting him with the development of new tools and workflows for live motion capture. The first of its kind using Smartsuit technology and trackers, we looked to explore the capabilities of this new mid-cost mocap solution in-house, without the need to hire dedicated capture facilities.

John’s project then extended into the curatorial practice of Masters student Tommaso Ranfagni, who created a live installation with John and a range of talented performance artists.

Process

John first came to us with the idea for a installation that would involve performers interacting with physical objects in a gallery space, while a real-time virtual rendition of the very same performance would play out on a big projection screen behind them. To bring his idea to life, we started looking into Rokoko’s Virtual Production suite. Combining Smartsuit and VR technology, we explored the creation of complex virtual scenes with live characters, props and virtual camera.

Experimenting with different humanoid avatars, we applied Smartsuit data to pre-made, fully modelled characters as well as custom 3D scans of our colleagues, which got us big laughs around the studio. Working in both Unity and Unreal, we developed solid workflows for real-time avatar control for characters from John’s earlier animation work.

Next, we applied VR trackers to objects, turning them into virtual props. We also used this technique on performers, creating an interesting contrast between full motion capture and positional tracking. This gave performers different possibilities to interact with the virtual world, with and without Smartsuit.

Putting performers in Smartsuits and VR headsets at the same time, we briefly explored what it would be like for them to look down at themselves in VR and see their virtual bodies. We also created a number of interesting out of body experiences, where performers could look at their avatars from different perspectives as they performed. However, these experiments were not included in the final performance.

As John’s project developed into the collaboration with Thomaso and more ambitious performance was developed with multiple performers, hosts, specially designed props, and a series of three media spaces the audience would be led through. During the event, the audience was given the chance to experience the performance through three different mediums: live at the venue, mediated via a monitor or by means of live streaming on social media. The culmination was the gallery installation with John conducting a ‘live avatar mix’ where smartsuit wearing performers had their virtual selves manipulated in real time alongside a live music score also delivered by John.

Outcome

The tools and techniques we developed have been futher developed by John in his practice and he continues to produce immersive works using smartsuit capture. The Labs continue to use the virtual production techniques in our pre-visualisation work and Thomasso graduated with a distinction in his MAster Degree in Curation!

As Thomasso wrote “Taking place in both the virtual and the physical world, HR explores themes of labour performativity and efficiency under the current neoliberal political system.”