Technical Development Summary

2024-09-30 11:31 UTC gpt-4o Open in ChatGPT ↗

Please properly format this:

Scientific & Technological Development In terms of the technical development of our project we’ve done some significant research in the fields of: camera-based motion tracking as well as on neural network architecture to develop the model that will interpret the performer’s movements for our machine to perceive. Camera-Based Pose Estimation Researched various existing models for multi camera body tracking (Free Mocap, EasyMocap, Mediapipe, MoveNet) Researched various solutions for camera calibration when using multi- camera solutions. Discussed with HLRS about the potential of them developing a multi camera tracking solution to use Insights Open Source multi camera solutions currently seem to support offline and not realtime processing. A custom solution will take time to develop and is probably not a good idea in the context of this residency We are contemplating doing professional mocap recordings to extract our training data, with optitrack, mocap suit or Captury via our connections at Target 3D (UK based motion capture & virtual production studio). We are contemplating using a Single camera tracking solution for the final performance. Neural Network Architecture Researched different models of Classifying Motion Capture Data both Supervised and Unsupervised using RNN, LSTM and Autoencoder architecture Developed custom LSTM Neural Network for Mocap Classification using archive mocap choreographies that we had on file. Started Visualising Class clustering, by performing 2D and 3D t-SNE dimension reduction on the second LSTM layer, to evaluate the model’s performance and extract feature convergence from input mocap data. Insights We chose to go with a supervised learning architecture following a model that has ample bibliography. We aim to do some analytical post processing to cross correlate the output classes with the input data. Collaboration with Scientists We’ve had some very interesting initial discussions with the team in Rome during the workshop that was organised in January where we got some initial feedback on our general technical workflow. We also had the chance to share knowledge and exchange ideas with other artists working in a similar field. Later on, in February, we travelled to Stuttgart and visited HLRS, which is our main hub. We had a chance to experiment with their available on-site technology, and test some of our initial camera tracking setups on-site where they have their LED wall. We discussed the problems that came up during those testing sessions and exchanged ideas on possible solutions. There is some shared interest with one of the scientists in HLRS who is actively researching an in-house tracking system so it was a fruitful interaction to converse with him and potentially collaborate on a solution that could benefit both of our research ambitions. As our project, apart from its technical challenges, has a significant performative aspect, it is crucial for us to collaborate with a team of tech- enthusiastic movement specialists. Both the team in HLRS as well as in Sony CSL have connected us with potential collaborators in this sector namely: the State Theatre in Stuttgart and Aterballetto in Bologna. We’ve had some very interesting initial conversations with both of these institutions and we hope we can find some common ground to collaborate during our residency, although timing and availability is definitely an issue. As we feel that our ideas and our technical approaches have now matured compared to the beginning of the residency, we hope that during the next stages of S+T+ARTS Air we will get into deeper conversations with both of our hubs, receive more meaningful feedback and expand on our common interests in a more frequent and collaborative manner.

Here’s the properly formatted version:

Scientific & Technological Development

In terms of the technical development of our project, we’ve conducted significant research in the fields of camera-based motion tracking and neural network architecture to develop a model that will interpret the performer’s movements for our machine to perceive.

Camera-Based Pose Estimation

Researched various existing models for multi-camera body tracking (Free Mocap, EasyMocap, Mediapipe, MoveNet).
Researched various solutions for camera calibration when using multi-camera systems.
Discussed with HLRS about the potential of them developing a multi-camera tracking solution for us.

Insights:

Open-source multi-camera solutions currently seem to support offline rather than real-time processing.
A custom solution will take time to develop and is likely not a feasible option within the context of this residency.
We are considering professional mocap recordings to extract our training data using OptiTrack, mocap suits, or Captury via our connections at Target 3D (a UK-based motion capture & virtual production studio).
We are also contemplating using a single-camera tracking solution for the final performance.

Neural Network Architecture

Researched different models for classifying motion capture data, both supervised and unsupervised, using RNN, LSTM, and Autoencoder architectures.
Developed a custom LSTM Neural Network for mocap classification using archived mocap choreographies.
Started visualizing class clustering by performing 2D and 3D t-SNE dimensional reduction on the second LSTM layer to evaluate the model’s performance and extract feature convergence from input mocap data.

Insights:

We chose to use a supervised learning architecture following a model with substantial supporting literature.
We aim to perform analytical post-processing to cross-correlate the output classes with the input data.

Collaboration with Scientists

We’ve had some highly engaging initial discussions with the team in Rome during the January workshop, where we received initial feedback on our technical workflow. We also had the chance to share knowledge and exchange ideas with other artists working in a similar field.

In February, we visited HLRS in Stuttgart, our main hub. We had the opportunity to experiment with their on-site technology and test some of our initial camera tracking setups using their LED wall. We discussed the challenges encountered during these testing sessions and exchanged ideas on potential solutions. One of the scientists at HLRS is actively researching an in-house tracking system, and our discussions were fruitful in potentially collaborating on a solution that could benefit both of our research objectives.

Given that our project, besides its technical challenges, has a significant performative aspect, collaboration with tech-enthusiastic movement specialists is crucial. Both the HLRS team and Sony CSL have connected us with potential collaborators, including the State Theatre in Stuttgart and Aterballetto in Bologna. We’ve had promising initial conversations with these institutions, and we hope to find common ground for collaboration during our residency, though timing and availability remain concerns.

As our ideas and technical approaches have matured since the start of the residency, we hope that during the next stages of S+T+ARTS Air, we can engage in deeper conversations with both of our hubs, receive more meaningful feedback, and expand on our common interests in a more frequent and collaborative manner.

Let me know if there’s anything else you’d like adjusted!

Please properly format this:

During this first stage of our Air residency, we’ve focused on three major artistic directions. Developing the main narrative of our performance, exploring the choreographic/movement identity of the human performer and developing some initial tests to define the language (visuals) and voice (sound design) of our digital entity. Narrative Development We have elaborated the performance’s narrative after various iterative discussions amongst our team and narrowed it down to three distinct acts. We are aiming for a 30-40 minute performance that will explore the development of a relationship between a human and a machine, the distinct power dynamics of this relationship as well as the concept of authority in this symbiotic relationship that aims to reflect our daily interactions with technology. Movement Workshops We have conducted four movement workshops with five different movement professionals (dancers & choreographers) During these workshops we discuss the performance’s main theme and how this translates into our narrative acts. Then we allow the dancer to interpret one of the acts using their own movement vocabulary in 6-10 minute improv sessions. After the improv sessions, we discuss the dancer’s choices, emotions and feelings trying to dilute how they genuinely feel when performing with a machine. We also try to understand from a movement perspective, what are the key tools (specific motions, specific limbs, specific poses) they use based on specific emotional states and how we can classify those tools to make them understandable in our Classification Model. The movement workshops also provided significant insight in how the narrative can evolve as well as how the visual language of the monolith can be developed. Visuals & Sound Development We have taken a first pass at developing some base visuals that are in line with the aesthetics we have envisioned for each narrative act and can also be used as a rudimentary Monolith visualisation in the movement workshops. We’ve had a few discussions with our sound designer collaborators elaborating on the audio identity of this digital entity and how we want this to progress through the course of the narrative. They have provided a valuable first draft for Act 1 that we have used in the movement workshops. We feel that all of the afforementioned research fields contribute significantly to our main concept which revolves around building a unique kind of non- verbal communication between a machine and a human in a performance based context. Originally when reading the Air open call we were very interested in the concept of “making the invisible visible”. In this case we are looking into exploring the communication between humans and technology and examine the power dynamics and co-dependence character of this relationship as it unfolds through an innovative non-verbal dialogue.

Here’s the properly formatted version:

Artistic Development During the First Stage of Our Air Residency

During this first stage of our Air residency, we’ve focused on three major artistic directions: developing the main narrative of our performance, exploring the choreographic/movement identity of the human performer, and conducting initial tests to define the language (visuals) and voice (sound design) of our digital entity.

Narrative Development

We have elaborated on the performance’s narrative through various iterative discussions within our team and narrowed it down to three distinct acts.
The aim is to create a 30-40 minute performance that explores the development of a relationship between a human and a machine, examining the distinct power dynamics of this relationship and the concept of authority in this symbiotic connection, which reflects our daily interactions with technology.

Movement Workshops

We have conducted four movement workshops with five different movement professionals (dancers & choreographers).
During these workshops, we discussed the performance’s main theme and how it translates into our narrative acts. Then, we allowed the dancer to interpret one of the acts using their own movement vocabulary in 6-10 minute improv sessions.
After the improv sessions, we discussed the dancer’s choices, emotions, and feelings, trying to distill how they genuinely feel when performing with a machine. We also aimed to understand, from a movement perspective, the key tools (specific motions, limbs, and poses) they use based on specific emotional states and how we can classify those tools to make them understandable within our classification model.
These movement workshops also provided significant insights into how the narrative can evolve, as well as how the visual language of the monolith can be developed.

Visuals & Sound Development

We have taken an initial pass at developing some base visuals that align with the aesthetics we’ve envisioned for each narrative act. These visuals can also serve as a rudimentary monolith visualization in the movement workshops.
We’ve engaged in discussions with our sound designer collaborators, elaborating on the audio identity of this digital entity and how we want it to progress throughout the course of the narrative. They have provided a valuable first draft for Act 1, which we have used in the movement workshops.

Conclusion

We believe that all of the aforementioned research fields contribute significantly to our main concept, which revolves around building a unique kind of non-verbal communication between a machine and a human in a performance-based context. When we initially read the Air open call, we were intrigued by the concept of “making the invisible visible.” In this case, we are exploring the communication between humans and technology, examining the power dynamics and co-dependence inherent in this relationship as it unfolds through an innovative non-verbal dialogue.

Let me know if any adjustments are needed!