Kinect Motion Data Smoothing

2024-10-08 10:26 UTC gpt-4o-canmore Open in ChatGPT ↗

You

This is how Azure Kinect maps the raw Kinect Tracking joints to an actual Unity Humanoid rig.

the problem is that the data are somewhat raw. I want to perform spectral smoothing for motion using FFT.

How can I do the smoothing before the data is injected to the humanoid rig?

using System.Collections; using System.Collections.Generic; using UnityEngine; using Microsoft.Azure.Kinect.BodyTracking; using System.Text;

public class PuppetAvatar : MonoBehaviour { public TrackerHandler KinectDevice; Dictionary<JointId, Quaternion> absoluteOffsetMap; Animator PuppetAnimator; public GameObject RootPosition; public Transform CharacterRootTransform; public float OffsetY; public float OffsetZ; private static HumanBodyBones MapKinectJoint(JointId joint) { // https://docs.microsoft.com/en-us/azure/Kinect-dk/body-joints switch (joint) { case JointId.Pelvis: return HumanBodyBones.Hips; case JointId.SpineNavel: return HumanBodyBones.Spine; case JointId.SpineChest: return HumanBodyBones.Chest; case JointId.Neck: return HumanBodyBones.Neck; case JointId.Head: return HumanBodyBones.Head; case JointId.HipLeft: return HumanBodyBones.LeftUpperLeg; case JointId.KneeLeft: return HumanBodyBones.LeftLowerLeg; case JointId.AnkleLeft: return HumanBodyBones.LeftFoot; case JointId.FootLeft: return HumanBodyBones.LeftToes; case JointId.HipRight: return HumanBodyBones.RightUpperLeg; case JointId.KneeRight: return HumanBodyBones.RightLowerLeg; case JointId.AnkleRight: return HumanBodyBones.RightFoot; case JointId.FootRight: return HumanBodyBones.RightToes; case JointId.ClavicleLeft: return HumanBodyBones.LeftShoulder; case JointId.ShoulderLeft: return HumanBodyBones.LeftUpperArm; case JointId.ElbowLeft: return HumanBodyBones.LeftLowerArm; case JointId.WristLeft: return HumanBodyBones.LeftHand; case JointId.ClavicleRight: return HumanBodyBones.RightShoulder; case JointId.ShoulderRight: return HumanBodyBones.RightUpperArm; case JointId.ElbowRight: return HumanBodyBones.RightLowerArm; case JointId.WristRight: return HumanBodyBones.RightHand; default: return HumanBodyBones.LastBone; } } private void Start() { PuppetAnimator = GetComponent(); Transform _rootJointTransform = CharacterRootTransform;

    absoluteOffsetMap = new Dictionary<JointId, Quaternion>();
    for (int i = 0; i < (int)JointId.Count; i++)
    {
        HumanBodyBones hbb = MapKinectJoint((JointId)i);
        if (hbb != HumanBodyBones.LastBone)
        {
            Transform transform = PuppetAnimator.GetBoneTransform(hbb);
            Quaternion absOffset = GetSkeletonBone(PuppetAnimator, transform.name).rotation;
            // find the absolute offset for the tpose
            while (!ReferenceEquals(transform, _rootJointTransform))
            {
                transform = transform.parent;
                absOffset = GetSkeletonBone(PuppetAnimator, transform.name).rotation * absOffset;
            }
            absoluteOffsetMap[(JointId)i] = absOffset;
        }
    }
}

private static SkeletonBone GetSkeletonBone(Animator animator, string boneName)
{
    int count = 0;
    StringBuilder cloneName = new StringBuilder(boneName);
    cloneName.Append("(Clone)");
    foreach (SkeletonBone sb in animator.avatar.humanDescription.skeleton)
    {
        if (sb.name == boneName || sb.name == cloneName.ToString())
        {
            return animator.avatar.humanDescription.skeleton[count];
        }
        count++;
    }
    return new SkeletonBone();
}

// Update is called once per frame
private void LateUpdate()
{
    for (int j = 0; j < (int)JointId.Count; j++)
    {
        if (MapKinectJoint((JointId)j) != HumanBodyBones.LastBone && absoluteOffsetMap.ContainsKey((JointId)j))
        {
            // get the absolute offset
            Quaternion absOffset = absoluteOffsetMap[(JointId)j];
            Transform finalJoint = PuppetAnimator.GetBoneTransform(MapKinectJoint((JointId)j));
            finalJoint.rotation = absOffset * Quaternion.Inverse(absOffset) * KinectDevice.absoluteJointRotations[j] * absOffset;

            if (j == 0)
            {
                // character root plus translation reading from the kinect, plus the offset from the script public variables
                finalJoint.position = CharacterRootTransform.position + new Vector3(RootPosition.transform.localPosition.x, RootPosition.transform.localPosition.y + OffsetY, RootPosition.transform.localPosition.z - OffsetZ);
            }
        }
    }
}

}

ChatGPT

To apply spectral smoothing to motion data using FFT, you’ll need to intercept the raw joint rotation and position data provided by the Kinect device, perform spectral smoothing on the data, and then pass the smoothed data to the Unity Humanoid rig. Below, I’ll provide a step-by-step guide on how to add spectral smoothing using FFT to the joint rotations before injecting the data into the Unity humanoid rig. Specifically, we’ll apply the FFT to filter the noisy data, then apply an inverse FFT to reconstruct the smoothed signal.

Here’s how to achieve this:

1. Add a Buffer for the Joint Data: To perform FFT, you’ll need to accumulate a certain amount of data for each joint over time. You can store the joint data in a sliding window buffer.

2. Perform FFT and Filtering: Use a Fast Fourier Transform (FFT) library to convert the data to the frequency domain, apply a filter to remove high-frequency components (which are often noise), and then convert the data back to the time domain using an inverse FFT.

3. Integrate Smoothed Data: Use the smoothed data in your existing logic to drive the Unity humanoid rig.

Below is an updated version of your script with comments to help you understand the new functionality:

• I’m assuming the FFT logic will use a library like MathNet.Numerics for FFT operations.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Microsoft.Azure.Kinect.BodyTracking;
using System.Text;
using MathNet.Numerics.IntegralTransforms;  // Requires MathNet.Numerics for FFT operations
using System.Linq;

public class PuppetAvatar : MonoBehaviour
{
    public TrackerHandler KinectDevice;
    Dictionary<JointId, Quaternion> absoluteOffsetMap;
    Animator PuppetAnimator;
    public GameObject RootPosition;
    public Transform CharacterRootTransform;
    public float OffsetY;
    public float OffsetZ;

    private Dictionary<JointId, Queue<Quaternion>> jointRotationBuffers;
    private int bufferSize = 60; // Adjust this depending on desired smoothing (e.g., 1 second of data at 60 FPS)

    private static HumanBodyBones MapKinectJoint(JointId joint)
    {
        // Mapping Kinect joints to Unity's Humanoid rig
        switch (joint)
        {
            case JointId.Pelvis: return HumanBodyBones.Hips;
            case JointId.SpineNavel: return HumanBodyBones.Spine;
            case JointId.SpineChest: return HumanBodyBones.Chest;
            case JointId.Neck: return HumanBodyBones.Neck;
            case JointId.Head: return HumanBodyBones.Head;
            case JointId.HipLeft: return HumanBodyBones.LeftUpperLeg;
            case JointId.KneeLeft: return HumanBodyBones.LeftLowerLeg;
            case JointId.AnkleLeft: return HumanBodyBones.LeftFoot;
            case JointId.FootLeft: return HumanBodyBones.LeftToes;
            case JointId.HipRight: return HumanBodyBones.RightUpperLeg;
            case JointId.KneeRight: return HumanBodyBones.RightLowerLeg;
            case JointId.AnkleRight: return HumanBodyBones.RightFoot;
            case JointId.FootRight: return HumanBodyBones.RightToes;
            case JointId.ClavicleLeft: return HumanBodyBones.LeftShoulder;
            case JointId.ShoulderLeft: return HumanBodyBones.LeftUpperArm;
            case JointId.ElbowLeft: return HumanBodyBones.LeftLowerArm;
            case JointId.WristLeft: return HumanBodyBones.LeftHand;
            case JointId.ClavicleRight: return HumanBodyBones.RightShoulder;
            case JointId.ShoulderRight: return HumanBodyBones.RightUpperArm;
            case JointId.ElbowRight: return HumanBodyBones.RightLowerArm;
            case JointId.WristRight: return HumanBodyBones.RightHand;
            default: return HumanBodyBones.LastBone;
        }
    }

    private void Start()
    {
        PuppetAnimator = GetComponent<Animator>();
        Transform _rootJointTransform = CharacterRootTransform;

        absoluteOffsetMap = new Dictionary<JointId, Quaternion>();
        jointRotationBuffers = new Dictionary<JointId, Queue<Quaternion>>();

        for (int i = 0; i < (int)JointId.Count; i++)
        {
            HumanBodyBones hbb = MapKinectJoint((JointId)i);
            if (hbb != HumanBodyBones.LastBone)
            {
                Transform transform = PuppetAnimator.GetBoneTransform(hbb);
                Quaternion absOffset = GetSkeletonBone(PuppetAnimator, transform.name).rotation;

                while (!ReferenceEquals(transform, _rootJointTransform))
                {
                    transform = transform.parent;
                    absOffset = GetSkeletonBone(PuppetAnimator, transform.name).rotation * absOffset;
                }
                absoluteOffsetMap[(JointId)i] = absOffset;

                // Initialize the joint rotation buffer
                jointRotationBuffers[(JointId)i] = new Queue<Quaternion>(new Quaternion[bufferSize]);
            }
        }
    }

    private static SkeletonBone GetSkeletonBone(Animator animator, string boneName)
    {
        int count = 0;
        StringBuilder cloneName = new StringBuilder(boneName);
        cloneName.Append("(Clone)");
        foreach (SkeletonBone sb in animator.avatar.humanDescription.skeleton)
        {
            if (sb.name == boneName || sb.name == cloneName.ToString())
            {
                return animator.avatar.humanDescription.skeleton[count];
            }
            count++;
        }
        return new SkeletonBone();
    }

    private void LateUpdate()
    {
        for (int j = 0; j < (int)JointId.Count; j++)
        {
            JointId jointId = (JointId)j;

            if (MapKinectJoint(jointId) != HumanBodyBones.LastBone && absoluteOffsetMap.ContainsKey(jointId))
            {
                // Add the current rotation to the buffer
                Queue<Quaternion> buffer = jointRotationBuffers[jointId];
                if (buffer.Count >= bufferSize)
                {
                    buffer.Dequeue();
                }
                buffer.Enqueue(KinectDevice.absoluteJointRotations[j]);

                // Apply spectral smoothing to the buffer
                Quaternion smoothedRotation = PerformSpectralSmoothing(buffer.ToArray());

                // Get the absolute offset
                Quaternion absOffset = absoluteOffsetMap[jointId];
                Transform finalJoint = PuppetAnimator.GetBoneTransform(MapKinectJoint(jointId));

                // Apply smoothed rotation
                finalJoint.rotation = absOffset * Quaternion.Inverse(absOffset) * smoothedRotation * absOffset;

                if (j == 0)
                {
                    // Root joint - update position with offset
                    finalJoint.position = CharacterRootTransform.position +
                                          new Vector3(RootPosition.transform.localPosition.x,
                                                      RootPosition.transform.localPosition.y + OffsetY,
                                                      RootPosition.transform.localPosition.z - OffsetZ);
                }
            }
        }
    }

    private Quaternion PerformSpectralSmoothing(Quaternion[] buffer)
    {
        // Convert the quaternion to Vector4 for FFT
        Vector4[] vectorBuffer = buffer.Select(q => new Vector4(q.x, q.y, q.z, q.w)).ToArray();

        // Perform FFT on each component
        double[] real = new double[bufferSize];
        double[] imag = new double[bufferSize];

        // Apply FFT for each component and filter out high frequencies
        for (int i = 0; i < 4; i++)
        {
            for (int j = 0; j < bufferSize; j++)
            {
                real[j] = vectorBuffer[j][i];
                imag[j] = 0;
            }

            // Perform FFT
            Fourier.Forward(real, imag, FourierOptions.Matlab);

            // Apply low-pass filter by zeroing out high frequencies
            int cutoff = bufferSize / 4; // Adjust cutoff as needed
            for (int j = cutoff; j < bufferSize - cutoff; j++)
            {
                real[j] = 0;
                imag[j] = 0;
            }

            // Perform Inverse FFT
            Fourier.Inverse(real, imag, FourierOptions.Matlab);

            // Store the smoothed value back
            for (int j = 0; j < bufferSize; j++)
            {
                vectorBuffer[j][i] = (float)real[j];
            }
        }

        // Convert back to Quaternion (use the latest smoothed value)
        Vector4 smoothedVec = vectorBuffer[bufferSize - 1];
        return new Quaternion(smoothedVec.x, smoothedVec.y, smoothedVec.z, smoothedVec.w);
    }
}

Explanation

1. Buffer Creation: A buffer (jointRotationBuffers) is used to store recent joint rotations for each joint (Quaternion values). This allows for spectral smoothing over time.

2. FFT and Smoothing:

   • PerformSpectralSmoothing() performs FFT on each of the components (x, y, z, w) of the quaternion.
   • The MathNet.Numerics library is used to perform FFT.
   • A simple low-pass filter is applied to remove high-frequency components that are likely to be noise.

3. Reconstruct Quaternion: The smoothed components are reconstructed back into a quaternion.

This approach will reduce jitter in joint movement by filtering out high-frequency noise. You can adjust the bufferSize and filter cutoff to tune the smoothing effect. The larger the buffer and the more aggressive the filtering, the smoother the animation will be, but it will also introduce more latency.

You

what is a recommended frequency for motion capture data?

ChatGPT

The recommended frequency for motion capture data depends on the specific use case and the type of movement being captured. Below, I outline typical frequency recommendations for different purposes:

1. Human Motion Capture (General Purpose)

• Frequency: 60 Hz to 120 Hz
• Reason: For capturing human motion in activities such as walking, running, or everyday gestures, 60 Hz is generally sufficient. This frequency can adequately capture the details of normal human movement without significant aliasing or loss of detail. For more precise applications, such as capturing rapid arm movements or detailed finger movements, increasing the frequency to 120 Hz can be beneficial to ensure smooth, natural interpolation and prevent motion artifacts.

2. High-Intensity or Fast Motion (Sports, Combat, Dance)

• Frequency: 120 Hz to 240 Hz
• Reason: High-intensity movements—such as those seen in sports, dancing, or combat—often involve rapid acceleration and fast, small-scale movements that can lead to motion blur or inaccurate data at lower frequencies. In these scenarios, a higher frequency of 120 Hz or even up to 240 Hz can provide greater detail, allowing better tracking of fast and dynamic movements.

3. Facial Motion Capture or Detailed Finger Movements

• Frequency: 90 Hz to 120 Hz
• Reason: Facial expressions and finger movements tend to be more subtle and require greater precision, particularly when using them for high-quality animation. A frequency of 90 Hz to 120 Hz is typically recommended to ensure the smallest nuances are captured accurately.

4. VR and Real-Time Animation

• Frequency: 60 Hz to 90 Hz
• Reason: For applications like VR or real-time animations where low latency is crucial, a frequency between 60 Hz and 90 Hz strikes a balance between accurate movement tracking and responsiveness. In VR applications, smooth tracking is essential to avoid motion sickness, which may occur due to jittery or delayed movement.

5. Medical and Biomechanical Analysis

• Frequency: 100 Hz to 240 Hz
• Reason: In biomechanical or medical contexts, where the goal is to capture fine-grained details of joint movement or analyze gait patterns, a frequency of 100 Hz or higher is preferred. This higher frequency provides sufficient accuracy for both analysis and research purposes.

Key Factors Affecting Frequency Choice

• Nature of Movement: Fast movements require higher sampling frequencies to avoid losing details or creating “jerky” animations.
• Precision vs. Latency: Higher frequencies yield more precise results, but at the cost of increased processing requirements and possibly latency.
• Use Case: Real-time use cases may prefer lower latency over extremely high precision, so a trade-off is sometimes made.

Nyquist Theorem Considerations

The Nyquist Theorem states that the sampling rate should be at least twice the highest frequency present in the signal to prevent aliasing. In the case of human motion, where most significant motion frequencies are below 10-15 Hz, a sampling rate of 30 Hz would theoretically suffice. However, practical considerations such as data quality and jitter mean that a frequency between 60 Hz and 120 Hz is ideal for most human motion.

Summary

• 60 Hz: General-purpose human motion (walking, talking, gesturing).
• 120 Hz: More dynamic activities (running, dancing, sports).
• 240 Hz: Extremely fast motion, very precise scenarios (detailed biomechanics, combat sports).
• 90-120 Hz: Subtle and precise movements (facial expressions, finger articulation).

For most standard human motion capture purposes, 60 Hz is a good starting point, while 120 Hz is often used for high-quality productions or fast-action scenarios. For highly dynamic or detailed capture (such as dance or martial arts), 240 Hz is recommended to ensure smoothness and precision.

You

So based on this, what is a good high-pass value to filter out noise, but maintain combative moves like jumping or punching?

ChatGPT

To effectively filter out noise while preserving essential components of fast, dynamic movements like jumping or punching, a good high-pass filter value must strike a balance. You need a cutoff that removes high-frequency noise (like jitter) while keeping the frequency components that represent quick, purposeful movements.

For combative movements, the following are suitable considerations:

Typical Motion Frequency Ranges

• Dynamic Human Movements: Jumping and punching typically involve frequencies in the range of 0.5 Hz to 5 Hz.
• Noise: Sensor noise and jitter are often at higher frequencies, typically above 10 Hz.

Recommended High-Pass Filter Cutoff

• A high-pass filter cutoff of around 0.5 Hz is a good starting point to filter out low-frequency drift while preserving the essential characteristics of rapid movement like punching or jumping.
• This cutoff value will:
  • Remove drift and very low-frequency noise, such as sensor bias or slow, unintended motions that could stem from Kinect tracking errors or small body sways.
  • Preserve dynamic movement since most human actions, including jumps or punches, contain frequency components well above 0.5 Hz.

Considerations for Filter Settings

• High-Pass Cutoff: 0.5 Hz:
  • This will effectively remove very slow unwanted signals (e.g., body sway, drift).
  • The essential part of movements like punching, kicking, or jumping will be retained since these movements are characterized by higher frequency components.
• Avoiding Loss of Motion Detail:
  • Be cautious not to set the high-pass cutoff too high. For example, setting it at 5 Hz or above would risk attenuating key components of rapid but broader human movements, such as torso movement during a punch.
  • 0.5 Hz to 1 Hz is generally enough to eliminate irrelevant low-frequency drift but maintain the natural dynamics of the movement.

Filter Type Suggestion

• Butterworth Filter:

  • A Butterworth filter is often a good choice due to its smooth frequency response, ensuring that essential frequency components for movement like punching or jumping are retained without excessive attenuation. A first or second-order Butterworth filter should work well for this scenario.

• Order of Filter:

  • A first or second-order high-pass filter should be sufficient. Higher-order filters can introduce latency or unwanted ripple effects, which can distort the motion tracking quality, particularly in a real-time system.

Practical Approach

1. Frequency Analysis:
   • You could analyze the frequency components of typical combative movements (like punching, jumping, or kicking) in your dataset using FFT to determine the dominant frequency bands. This will help in validating the 0.5 Hz starting point.
2. Adjust Based on Feedback:
   • Start with a 0.5 Hz high-pass cutoff and evaluate the filtered data. If critical movement details appear attenuated or “jerky,” lower the cutoff slightly. If noise is still prevalent, you might want to slightly increase it.

Summary

• Suggested High-Pass Cutoff: 0.5 Hz
  • This value filters out low-frequency drift and noise while retaining rapid, dynamic movements.
• Filter Type: A Butterworth high-pass filter with first or second-order is recommended for minimal signal distortion and smooth filtering.

This setup will help to maintain the key movement components of actions like jumping or punching while filtering out the noise introduced by sensors or unintended minor movements.

You

Apply butterworth high pass on the above implementation

ChatGPT

To implement a Butterworth high-pass filter for spectral smoothing in your Unity C# script, we can use the following approach:

1. Add Butterworth High-Pass Filter Logic:

   • Since Unity does not have a built-in signal processing library, you can create a simple Butterworth high-pass filter in C#.
   • The Butterworth filter implementation below works on quaternion components to filter out the unwanted low-frequency components.

2. Apply the Filter on Quaternion Components:

   • You will filter each of the quaternion components (x, y, z, w) separately.
   • A buffer is maintained for each component, and the Butterworth filter is applied to reduce drift and preserve rapid movement.

Here is an updated version of your code with the Butterworth high-pass filter applied:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Microsoft.Azure.Kinect.BodyTracking;
using System.Text;
using System.Linq;

public class PuppetAvatar : MonoBehaviour
{
    public TrackerHandler KinectDevice;
    Dictionary<JointId, Quaternion> absoluteOffsetMap;
    Animator PuppetAnimator;
    public GameObject RootPosition;
    public Transform CharacterRootTransform;
    public float OffsetY;
    public float OffsetZ;

    private Dictionary<JointId, Queue<Vector4>> jointRotationBuffers;
    private int bufferSize = 60; // Adjust this depending on desired smoothing (e.g., 1 second of data at 60 FPS)
    private float cutoffFrequency = 0.5f; // High-pass filter cutoff frequency in Hz
    private float samplingFrequency = 60.0f; // Sampling frequency in Hz

    private static HumanBodyBones MapKinectJoint(JointId joint)
    {
        // Mapping Kinect joints to Unity's Humanoid rig
        switch (joint)
        {
            case JointId.Pelvis: return HumanBodyBones.Hips;
            case JointId.SpineNavel: return HumanBodyBones.Spine;
            case JointId.SpineChest: return HumanBodyBones.Chest;
            case JointId.Neck: return HumanBodyBones.Neck;
            case JointId.Head: return HumanBodyBones.Head;
            case JointId.HipLeft: return HumanBodyBones.LeftUpperLeg;
            case JointId.KneeLeft: return HumanBodyBones.LeftLowerLeg;
            case JointId.AnkleLeft: return HumanBodyBones.LeftFoot;
            case JointId.FootLeft: return HumanBodyBones.LeftToes;
            case JointId.HipRight: return HumanBodyBones.RightUpperLeg;
            case JointId.KneeRight: return HumanBodyBones.RightLowerLeg;
            case JointId.AnkleRight: return HumanBodyBones.RightFoot;
            case JointId.FootRight: return HumanBodyBones.RightToes;
            case JointId.ClavicleLeft: return HumanBodyBones.LeftShoulder;
            case JointId.ShoulderLeft: return HumanBodyBones.LeftUpperArm;
            case JointId.ElbowLeft: return HumanBodyBones.LeftLowerArm;
            case JointId.WristLeft: return HumanBodyBones.LeftHand;
            case JointId.ClavicleRight: return HumanBodyBones.RightShoulder;
            case JointId.ShoulderRight: return HumanBodyBones.RightUpperArm;
            case JointId.ElbowRight: return HumanBodyBones.RightLowerArm;
            case JointId.WristRight: return HumanBodyBones.RightHand;
            default: return HumanBodyBones.LastBone;
        }
    }

    private void Start()
    {
        PuppetAnimator = GetComponent<Animator>();
        Transform _rootJointTransform = CharacterRootTransform;

        absoluteOffsetMap = new Dictionary<JointId, Quaternion>();
        jointRotationBuffers = new Dictionary<JointId, Queue<Vector4>>();

        for (int i = 0; i < (int)JointId.Count; i++)
        {
            HumanBodyBones hbb = MapKinectJoint((JointId)i);
            if (hbb != HumanBodyBones.LastBone)
            {
                Transform transform = PuppetAnimator.GetBoneTransform(hbb);
                Quaternion absOffset = GetSkeletonBone(PuppetAnimator, transform.name).rotation;

                while (!ReferenceEquals(transform, _rootJointTransform))
                {
                    transform = transform.parent;
                    absOffset = GetSkeletonBone(PuppetAnimator, transform.name).rotation * absOffset;
                }
                absoluteOffsetMap[(JointId)i] = absOffset;

                // Initialize the joint rotation buffer
                jointRotationBuffers[(JointId)i] = new Queue<Vector4>(new Vector4[bufferSize]);
            }
        }
    }

    private static SkeletonBone GetSkeletonBone(Animator animator, string boneName)
    {
        int count = 0;
        StringBuilder cloneName = new StringBuilder(boneName);
        cloneName.Append("(Clone)");
        foreach (SkeletonBone sb in animator.avatar.humanDescription.skeleton)
        {
            if (sb.name == boneName || sb.name == cloneName.ToString())
            {
                return animator.avatar.humanDescription.skeleton[count];
            }
            count++;
        }
        return new SkeletonBone();
    }

    private void LateUpdate()
    {
        for (int j = 0; j < (int)JointId.Count; j++)
        {
            JointId jointId = (JointId)j;

            if (MapKinectJoint(jointId) != HumanBodyBones.LastBone && absoluteOffsetMap.ContainsKey(jointId))
            {
                // Get the current rotation and add it to the buffer
                Quaternion currentRotation = KinectDevice.absoluteJointRotations[j];
                Vector4 currentRotationVec = new Vector4(currentRotation.x, currentRotation.y, currentRotation.z, currentRotation.w);
                Queue<Vector4> buffer = jointRotationBuffers[jointId];

                if (buffer.Count >= bufferSize)
                {
                    buffer.Dequeue();
                }
                buffer.Enqueue(currentRotationVec);

                // Apply Butterworth high-pass filter to the buffer
                Vector4 filteredRotationVec = ApplyButterworthHighPassFilter(buffer.ToArray(), cutoffFrequency, samplingFrequency);

                // Convert back to Quaternion
                Quaternion filteredRotation = new Quaternion(filteredRotationVec.x, filteredRotationVec.y, filteredRotationVec.z, filteredRotationVec.w);

                // Get the absolute offset
                Quaternion absOffset = absoluteOffsetMap[jointId];
                Transform finalJoint = PuppetAnimator.GetBoneTransform(MapKinectJoint(jointId));

                // Apply the filtered rotation
                finalJoint.rotation = absOffset * Quaternion.Inverse(absOffset) * filteredRotation * absOffset;

                if (j == 0)
                {
                    // Root joint - update position with offset
                    finalJoint.position = CharacterRootTransform.position +
                                          new Vector3(RootPosition.transform.localPosition.x,
                                                      RootPosition.transform.localPosition.y + OffsetY,
                                                      RootPosition.transform.localPosition.z - OffsetZ);
                }
            }
        }
    }

    private Vector4 ApplyButterworthHighPassFilter(Vector4[] buffer, float cutoffFrequency, float samplingFrequency)
    {
        int n = buffer.Length;
        Vector4 result = buffer[n - 1];

        if (n < 2) return result;

        // Butterworth filter parameters (second-order)
        float omega = 2.0f * Mathf.PI * cutoffFrequency / samplingFrequency;
        float tanOmega = Mathf.Tan(omega / 2.0f);
        float a0 = 1.0f / (1.0f + Mathf.Sqrt(2.0f) * tanOmega + tanOmega * tanOmega);
        float a1 = -2.0f * a0;
        float a2 = a0;
        float b1 = 2.0f * (tanOmega * tanOmega - 1.0f) * a0;
        float b2 = (1.0f - Mathf.Sqrt(2.0f) * tanOmega + tanOmega * tanOmega) * a0;

        Vector4 prevOutput = buffer[n - 2]; // Last output value (y[n-1])
        Vector4 prevPrevOutput = buffer[n - 3 >= 0 ? n - 3 : n - 2]; // y[n-2], fallback if not enough data
        Vector4 prevInput = buffer[n - 2]; // Last input value (x[n-1])
        Vector4 prevPrevInput = buffer[n - 3 >= 0 ? n - 3 : n - 2]; // x[n-2]

        // Apply Butterworth filter equation
        result = a0 * result + a1 * prevInput + a2 * prevPrevInput + b1 * prevOutput + b2 * prevPrevOutput;

        return result;
    }
}

Key Additions

1. Buffer Initialization: The script now includes a buffer (jointRotationBuffers) for each joint. The buffer stores the recent quaternion data for the joint.

2. Butterworth Filter Implementation:

   • The method ApplyButterworthHighPassFilter implements a second-order Butterworth high-pass filter for smoothing. This method works on each of the components (x, y, z, w) of the quaternion.
   • The filter parameters are set up for a second-order high-pass filter. You can adjust the cutoffFrequency and samplingFrequency to suit your specific needs.

3. Quaternion Conversion: Quaternion values are represented as Vector4 to make applying the filter to each component easier.

Parameter Adjustments

• cutoffFrequency: Adjust this based on your desired balance between noise suppression and maintaining movement dynamics. A value around 0.5 Hz works well for removing drift while preserving sharp movements.
• samplingFrequency: Set to 60 Hz if your motion capture data updates at that rate.

Final Note

This implementation uses a Butterworth high-pass filter to clean the data before injecting it into the Unity humanoid rig. The filter effectively reduces unwanted low-frequency noise while retaining the key features of rapid movements, such as punches and jumps. You can further tune the cutoff frequency and sampling rate for more refined filtering based on the specific motions you want to emphasize or smooth.

You

Is the FFT or the butterworth better for motion data?

ChatGPT

The choice between FFT-based filtering and Butterworth filtering for smoothing motion capture data depends on the specific requirements of your application, the nature of the data, and the type of movement you’re capturing. Both methods have their advantages and disadvantages, and which one is “better” will vary based on these factors. Let’s compare both techniques in the context of motion data to help you determine which is more appropriate for your use case:

1. FFT-Based Filtering

Overview: The Fast Fourier Transform (FFT) is used to convert a signal from the time domain to the frequency domain, where filtering can be applied more easily. After filtering, an Inverse FFT (IFFT) is used to convert the signal back to the time domain.

Advantages:

1. Frequency-Specific Control:

   • FFT provides precise control over the frequency bands that are retained or removed. This means you can easily visualize and filter out specific frequencies (e.g., noise above or below a certain threshold).

2. Customizable Filtering:

   • With FFT, you can apply various kinds of filters such as low-pass, high-pass, band-pass, and band-stop, providing flexibility in tailoring the filter to the specific motion data.

3. Detailed Frequency Analysis:

   • FFT allows you to perform a frequency analysis of the motion data. This can be useful if you want to understand the underlying frequency components of movements, which can help tune the filtering process.

Disadvantages:

1. Latency:

   • FFT requires buffering enough data points to perform the transformation. This introduces latency because you need a sliding window of frames before the FFT can be calculated. For real-time applications, such as a game or live animation, this latency can be problematic.

2. Complexity:

   • FFT can be more computationally expensive than other filtering methods. This could affect the performance, especially if the data needs to be processed in real time.

3. Windowing Effects:

   • When applying FFT over a limited window of data, windowing artifacts (like ringing or loss of temporal resolution) can occur. This may impact the quality of the smoothed motion.

2. Butterworth Filter

Overview: A Butterworth filter is a type of Infinite Impulse Response (IIR) filter that has a flat frequency response in the passband. Butterworth filters can be used as low-pass, high-pass, band-pass, or band-stop filters, and are commonly used in signal processing.

Advantages:

1. Low Latency:

   • Butterworth filters can be implemented incrementally, which means they can operate on each new data point as it arrives. This makes Butterworth filters more suitable for real-time processing and applications where low latency is essential.

2. Smooth Transition:

   • The frequency response of a Butterworth filter is smooth, which ensures that there are no ripples or artifacts introduced into the motion data. This is especially important for achieving natural-looking human movements.

3. Ease of Implementation:

   • Butterworth filters are conceptually simpler and easier to implement compared to FFT. They require fewer parameters and can be integrated directly into the update loop, making them well-suited for Unity real-time applications.

4. Less Computationally Intensive:

   • Butterworth filters require fewer resources compared to FFT. Since they operate directly in the time domain without needing the computational overhead of transformation and inverse transformation, they are generally more efficient for continuous filtering of incoming data.

Disadvantages:

1. Less Frequency-Specific Control:

   • Unlike FFT, Butterworth filters do not provide the ability to inspect or precisely manipulate specific frequency bands in the motion data. You can set a cutoff frequency, but it doesn’t provide the same level of flexibility as a frequency-domain approach.

2. Potential Phase Shift:

   • Butterworth filters, particularly those of higher order, can introduce phase shifts in the signal. Phase shift refers to a delay between different frequency components, which can sometimes distort motion if not corrected. However, this can be mitigated by using a bidirectional filter, which applies the filter forward and backward to eliminate phase distortion.

When to Use FFT vs. Butterworth Filter

• FFT Filtering is better if:

  1. Frequency Analysis: You need a detailed analysis of the frequency components in your motion data.
  2. Complex Filtering Requirements: You need to remove specific frequency bands or perform advanced filtering that may require a combination of high-pass, low-pass, and band-pass filters.
  3. Batch Processing: The motion data can be collected and processed in batches (i.e., non-real-time scenarios). FFT’s higher latency is less of an issue if you do not need real-time output.

• Butterworth Filter is better if:

  1. Real-Time Processing: You are working in real-time scenarios like VR or live animation, where latency must be minimized.
  2. Smooth and Continuous Filtering: You need smooth filtering for incoming data without significant delays. Butterworth filters are easy to apply on-the-fly, making them ideal for keeping motion data responsive.
  3. Computational Efficiency: You need a lightweight, computationally efficient filter to keep the frame rate high. Butterworth filters, especially of lower orders, are faster and require fewer resources.

Recommended Approach for Motion Data (e.g., Punching, Jumping)

• For real-time applications like a game or a VR avatar, a Butterworth high-pass filter is usually the preferred choice. It provides good noise reduction without the high computational cost or latency of FFT, which is important for maintaining responsiveness and real-time interactivity.
• For offline motion data analysis or scenarios where you want precise frequency control, FFT-based filtering can provide a deeper understanding of the movement data and allow for more precise frequency manipulation.

Given that your use case involves real-time Kinect data to Unity humanoid rig for actions like jumping or punching, which require immediate feedback, the Butterworth high-pass filter is generally the better choice due to:

• Low latency, ensuring smooth, real-time response.
• Simplicity in integration with your Unity project.
• Sufficient noise reduction while retaining dynamic movement.

In summary:

• If you are developing a real-time system (such as games or live animation), use the Butterworth filter.
• If you want to analyze data or perform more complex frequency-specific filtering (such as filtering out multiple noise bands), use FFT.

For your current setup involving Azure Kinect and Unity, the Butterworth high-pass filter would be a highly effective solution to reduce jitter while preserving the key characteristics of fast and dynamic human movements.

You

can we somehow visualize on the Unity game window the original signal and the smoothed signal?

ChatGPT

Yes, you can absolutely visualize both the original and smoothed signals in the Unity game window. By plotting the raw and filtered signals in real-time, you can get a better understanding of how your filtering is affecting the motion data. Here are a few different ways to visualize these signals in Unity:

Step-by-Step Approach to Visualize Signals in Unity

The most effective way to visualize the raw and smoothed data is by using line graphs in the Unity Game window. This visualization can be implemented by drawing lines representing each data point over time. Here’s how you can do it:

1. Create a Graph with Unity’s UI System: Use Unity’s Canvas, LineRenderer, or a dedicated UI framework to create line plots that visualize the signals. Below, I’ll describe how to use LineRenderer or Unity UI to visualize both the original and filtered signals.

2. Store Data for Visualization:

   • Store both raw and smoothed data over a given time frame in buffers. You can use lists to store a fixed number of recent data points, for example, the last few hundred data frames.

3. Update Graph in Real-Time:

   • Update the graph with each frame to add new data points and visualize how the signals change over time.

Implementation Steps

Step 1: Create Graph Components

• Create an empty GameObject for each graph (RawSignalGraph, FilteredSignalGraph).
• Attach a LineRenderer component to each GameObject.

Step 2: Script to Handle Visualization

Create a script that will collect data points from both the original signal and the filtered signal. This script will draw them in the Unity game window.

Here’s a complete implementation in Unity C#:

using System.Collections.Generic;
using UnityEngine;

public class MotionSignalVisualizer : MonoBehaviour
{
    public GameObject rawSignalGraph;
    public GameObject filteredSignalGraph;
    public TrackerHandler KinectDevice;

    private LineRenderer rawLineRenderer;
    private LineRenderer filteredLineRenderer;

    private int maxDataPoints = 100; // Adjust to define how many data points to visualize
    private Queue<Vector3> rawSignalData;
    private Queue<Vector3> filteredSignalData;

    public Transform jointToTrack; // Example: Track a specific joint like the wrist

    private void Start()
    {
        rawLineRenderer = rawSignalGraph.GetComponent<LineRenderer>();
        filteredLineRenderer = filteredSignalGraph.GetComponent<LineRenderer>();

        rawSignalData = new Queue<Vector3>(maxDataPoints);
        filteredSignalData = new Queue<Vector3>(maxDataPoints);

        // Initialize LineRenderers
        rawLineRenderer.positionCount = 0;
        filteredLineRenderer.positionCount = 0;
    }

    private void Update()
    {
        // Get the current raw and filtered positions
        Vector3 currentRawPosition = jointToTrack.position; // Raw data from Kinect
        Vector3 currentFilteredPosition = GetFilteredJointPosition(); // Apply your filtering logic here

        // Add raw data point
        if (rawSignalData.Count >= maxDataPoints)
        {
            rawSignalData.Dequeue();
        }
        rawSignalData.Enqueue(currentRawPosition);

        // Add filtered data point
        if (filteredSignalData.Count >= maxDataPoints)
        {
            filteredSignalData.Dequeue();
        }
        filteredSignalData.Enqueue(currentFilteredPosition);

        // Update LineRenderers
        UpdateLineRenderer(rawLineRenderer, rawSignalData);
        UpdateLineRenderer(filteredLineRenderer, filteredSignalData);
    }

    private void UpdateLineRenderer(LineRenderer lineRenderer, Queue<Vector3> dataPoints)
    {
        lineRenderer.positionCount = dataPoints.Count;
        lineRenderer.SetPositions(dataPoints.ToArray());
    }

    private Vector3 GetFilteredJointPosition()
    {
        // Apply your Butterworth filter or other smoothing technique here.
        // This function should return the filtered version of the joint position.
        // For demonstration, I'm just returning the raw position (this should be replaced with actual filtered data).
        return jointToTrack.position;
    }
}

Explanation

1. LineRenderers for Visualization:

   • The LineRenderer component is used to draw the lines representing the signal over time.
   • The script takes two GameObjects (rawSignalGraph, filteredSignalGraph) which have LineRenderer components attached to them.

2. Raw and Filtered Data Buffers:

   • rawSignalData and filteredSignalData are queues used to maintain a history of recent data points.
   • The length of these queues is determined by maxDataPoints. This ensures that the graph represents the latest portion of motion data, allowing a continuous rolling visualization.

3. Adding and Updating Data:

   • In each frame (Update() method), the latest positions of the tracked joint are retrieved and added to the appropriate queue (either raw or filtered).
   • If the queue reaches maxDataPoints, the oldest data point is removed to make space for the new one. This maintains the rolling window effect.

4. Updating the Graph:

   • The UpdateLineRenderer function takes the LineRenderer and updates its points based on the current data in the queue.

Step 3: Setting Up Unity Scene

• Create UI:

  • Add two empty GameObjects named RawSignalGraph and FilteredSignalGraph in the Unity scene.
  • Attach LineRenderer components to both objects.
  • Assign distinct colors to each LineRenderer to differentiate the raw and filtered signals.
  • Set their width and other visual parameters to make them easily visible in the game window.

• Assign GameObjects to Script:

  • Attach the MotionSignalVisualizer script to an empty GameObject in the scene.
  • Assign the RawSignalGraph and FilteredSignalGraph GameObjects to the script through the Inspector.
  • Assign the joint you wish to track (e.g., the WristLeft or WristRight transform) to the jointToTrack field.

Tips for Better Visualization

1. Scaling and Positioning:

   • You may want to scale or position the graph so that it fits properly within the Unity Game Window. Signals could be in a 3D space, and translating that into a flat graph may need additional scaling logic.

2. Graph Appearance:

   • You can add different colors or line thicknesses to make the distinction between raw and filtered signals more apparent. You could use a dashed line for one and a solid line for the other.

3. Handling 3D Vector Data:

   • Since both raw and filtered data for motion capture are represented by Vector3, you may wish to plot each component separately (X, Y, Z) using multiple LineRenderers. Alternatively, you can visualize only the magnitude or specific axis to keep it simple.

Summary

By using LineRenderers and maintaining a rolling buffer of recent data points, you can effectively visualize both the raw and smoothed signals from your motion capture setup in Unity. This approach allows you to see the real-time effects of the applied filtering, making it easier to adjust the filter parameters and verify the quality of the filtered motion data.

You

edit this fft to accept a frequency

private Quaternion PerformSpectralSmoothing(Vector4[] buffer) { // Convert the quaternion to Vector4 for FFT Vector4[] vectorBuffer = buffer;//buffer.Select(q => new Vector4(q.x, q.y, q.z, q.w)).ToArray();

    // Perform FFT on each component
    double[] real = new double[bufferSize];
    double[] imag = new double[bufferSize];

    // Apply FFT for each component and filter out high frequencies
    for (int i = 0; i < 4; i++)
    {
        for (int j = 0; j < bufferSize; j++)
        {
            real[j] = vectorBuffer[j][i];
            imag[j] = 0;
        }

        // Perform FFT
        Fourier.Forward(real, imag, FourierOptions.Matlab);

        // Apply low-pass filter by zeroing out high frequencies
        int cutoff = bufferSize / 4; // Adjust cutoff as needed
        for (int j = cutoff; j < bufferSize - cutoff; j++)
        {
            real[j] = 0;
            imag[j] = 0;
        }

        // Perform Inverse FFT
        Fourier.Inverse(real, imag, FourierOptions.Matlab);

        // Store the smoothed value back
        for (int j = 0; j < bufferSize; j++)
        {
            vectorBuffer[j][i] = (float)real[j];
        }
    }

    // Convert back to Quaternion (use the latest smoothed value)
    Vector4 smoothedVec = vectorBuffer[bufferSize - 1];
    return new Quaternion(smoothedVec.x, smoothedVec.y, smoothedVec.z, smoothedVec.w);
}

ChatGPT

To modify the PerformSpectralSmoothing function to accept a frequency cutoff parameter, you need to calculate the appropriate index for applying the filter based on the frequency cutoff and the sampling frequency. This will allow you to zero out unwanted high frequencies effectively while preserving the low-frequency components up to the desired cutoff.

Here’s how you can achieve this:

1. Add a frequency cutoff parameter to the function signature.
2. Convert frequency cutoff into an index that represents the boundary in the frequency domain.
3. Zero out components beyond the specified frequency to create a low-pass filter.

Below is the updated code:

private Quaternion PerformSpectralSmoothing(Vector4[] buffer, float frequencyCutoff, float samplingFrequency)
{
    // Convert the quaternion to Vector4 for FFT
    Vector4[] vectorBuffer = buffer; // buffer.Select(q => new Vector4(q.x, q.y, q.z, q.w)).ToArray();

    // Perform FFT on each component
    double[] real = new double[bufferSize];
    double[] imag = new double[bufferSize];

    // Apply FFT for each component and filter out frequencies above the cutoff
    for (int i = 0; i < 4; i++)
    {
        for (int j = 0; j < bufferSize; j++)
        {
            real[j] = vectorBuffer[j][i];
            imag[j] = 0;
        }

        // Perform FFT
        Fourier.Forward(real, imag, FourierOptions.Matlab);

        // Calculate cutoff index based on frequencyCutoff and samplingFrequency
        int cutoffIndex = Mathf.RoundToInt((frequencyCutoff / samplingFrequency) * bufferSize);

        // Apply low-pass filter by zeroing out high frequencies beyond cutoffIndex
        for (int j = cutoffIndex; j < bufferSize - cutoffIndex; j++)
        {
            real[j] = 0;
            imag[j] = 0;
        }

        // Perform Inverse FFT
        Fourier.Inverse(real, imag, FourierOptions.Matlab);

        // Store the smoothed value back
        for (int j = 0; j < bufferSize; j++)
        {
            vectorBuffer[j][i] = (float)real[j];
        }
    }

    // Convert back to Quaternion (use the latest smoothed value)
    Vector4 smoothedVec = vectorBuffer[bufferSize - 1];
    return new Quaternion(smoothedVec.x, smoothedVec.y, smoothedVec.z, smoothedVec.w);
}

Changes Explained:

1. Frequency Cutoff Parameter:

   • The function now accepts an additional parameter:

     float frequencyCutoff, float samplingFrequency

   • frequencyCutoff represents the cutoff frequency, above which you want to remove noise.
   • samplingFrequency represents how often the motion data is sampled, which helps calculate the corresponding frequency range in the FFT result.

2. Calculate the Cutoff Index:

   • The cutoff frequency needs to be converted to a corresponding index in the FFT result array.
   • The formula for determining the cutoff index is:

     int cutoffIndex = Mathf.RoundToInt((frequencyCutoff / samplingFrequency) * bufferSize);

   • This formula converts the desired frequency into an index that maps to the FFT bins, based on the buffer size and sampling rate.

3. Apply Low-Pass Filtering:

   • The for loop that zeroes out frequency components beyond the cutoff frequency has been updated:

     for (int j = cutoffIndex; j < bufferSize - cutoffIndex; j++)
     {
         real[j] = 0;
         imag[j] = 0;
     }

   • Here, you zero out all values beyond cutoffIndex to remove the higher frequencies. This leaves the lower-frequency components intact, allowing you to smooth the motion.

Summary

• Cutoff Frequency Parameter: The updated method now allows you to pass in a desired cutoff frequency, giving you flexibility to filter different levels of high-frequency noise.
• Frequency Calculation: The cutoff index is computed based on the cutoff frequency and the sampling frequency. This ensures that only the frequencies above the desired cutoff are zeroed out.
• Smoothing Control: By adjusting the frequencyCutoff parameter, you can control how much smoothing you want, which is particularly useful for different types of movement data (e.g., fine motor movement vs. large dynamic movements).

This updated method allows you to retain control over the filtering process, making it adaptable to different requirements for various kinds of motion capture data.

You

If my data is coming in at 60fps, what is a good buffer size, frequency cutoff and sampling frequency?

ChatGPT

When determining buffer size, frequency cutoff, and sampling frequency for motion capture data arriving at 60 frames per second (fps), the goal is to filter out noise while preserving natural movement characteristics. Let’s break down the parameters and provide some reasonable values:

1. Sampling Frequency

• Since your data is coming in at 60fps, the sampling frequency (Fs) is 60 Hz.

  • This means that the data is being updated 60 times per second, and your FFT analysis will reflect frequency components up to the Nyquist frequency, which is half of the sampling rate.
  • Nyquist Frequency: $ \frac{Fs}{2} = 30 $ Hz. This is the highest frequency that can be represented without aliasing.

  Recommended Value: 60 Hz.

2. Buffer Size

• The buffer size determines the number of frames that are used for FFT processing, which in turn determines the time window that the FFT will operate over.

• Buffer size affects both the frequency resolution and the latency of the filtering process:

  • A larger buffer provides better frequency resolution, which means you can more precisely filter specific frequency bands.
  • However, a larger buffer also introduces latency, since it requires waiting until the buffer is filled before processing can occur.

• For real-time motion capture, there is typically a trade-off between latency and smoothness:

  • Recommended Buffer Size: For 60 fps, a buffer size of 60 to 120 frames is typically good. This corresponds to 1-2 seconds of data:
    • 60 frames: Provides a 1-second window of data, balancing latency and smoothness, suitable for real-time responsiveness.
    • 120 frames: Provides a 2-second window of data, giving slightly better frequency resolution, which can be helpful if you need to remove more subtle noise.

  For most real-time applications like animation or VR avatars:

  • Start with a buffer size of 60 frames (1 second). If the smoothing isn’t sufficient, consider increasing it to 120 frames for better frequency discrimination.

  Recommended Value: 60 to 120 frames.

3. Frequency Cutoff

• The frequency cutoff determines the threshold above which higher frequencies will be removed from the motion data.

• You want to filter out high-frequency noise while retaining the frequencies associated with natural human movements. Typical human movement frequency ranges are as follows:

  • General Movement: Most meaningful human body movements (e.g., arms, legs, torso) typically occur at frequencies between 0.5 Hz to 5 Hz.
  • Noise: Sensor noise usually shows up at higher frequencies, typically above 10 Hz.

• To preserve fast, natural movements like punching or jumping while eliminating noise, you should use a low-pass filter with a cutoff between 3 to 5 Hz:

  • 3 Hz to 5 Hz will filter out high-frequency jitter while keeping the bulk of the motion data intact.
  • This ensures that you keep dynamic, deliberate movements but filter out subtle, rapid sensor errors or jittery signals.

  Recommended Value: 3 Hz to 5 Hz.

Summary of Recommended Parameters

1. Sampling Frequency (Fs): 60 Hz.

   • This is fixed based on your incoming data rate.

2. Buffer Size: 60 to 120 frames.

   • Start with 60 frames (1 second) for a good balance of latency and frequency resolution.
   • Increase to 120 frames if more frequency resolution is needed for smoother output.

3. Frequency Cutoff: 3 Hz to 5 Hz.

   • A value of 3 Hz will ensure smoother filtering, while 5 Hz will provide less aggressive filtering, preserving more dynamics.

Practical Example for Your Scenario

• Buffer Size: Start with 60 frames.
• Sampling Frequency (Fs): Set to 60 Hz.
• Frequency Cutoff (frequencyCutoff): Set to 4 Hz (middle of the recommended range).

Here’s how you would call the PerformSpectralSmoothing function:

Quaternion smoothedRotation = PerformSpectralSmoothing(buffer, 4.0f, 60.0f);

• buffer: The array of Vector4 representing quaternion components.
• 4.0f: The cutoff frequency in Hz.
• 60.0f: The sampling frequency in Hz.

Additional Considerations

• Tuning: Start with these recommended values, and tune based on visual inspection. If the filtered data seems laggy or doesn’t respond well to quick actions, try increasing the cutoff frequency slightly or reducing the buffer size.
• Latency vs. Smoothing: Larger buffer sizes will yield smoother results but may increase the latency, which could affect responsiveness in real-time applications. The same applies to a lower frequency cutoff, which will further smooth the data but could lead to a delay in rapid movement detection.

By choosing an appropriate buffer size, frequency cutoff, and sampling frequency, you can effectively smooth the motion capture data to maintain the natural quality of dynamic movements like punching or jumping while reducing noise.

You

but this filtering produces much worse deformations. is it correct that we are doing fft on the quaternions?

ChatGPT

You’re right to be concerned—applying FFT directly to quaternions is generally not recommended, as it can lead to incorrect or visually unpleasant deformations. Quaternions are used to represent rotations in a way that avoids the gimbal lock and singularity issues associated with Euler angles, and they possess non-linear properties that make direct spectral analysis challenging.

Problems with Applying FFT Directly to Quaternions

1. Non-Linearity:

   • Quaternions represent rotations as a 4D unit vector (x, y, z, w), but they are non-linear in nature. When you apply an FFT directly to the quaternion components, you lose the geometric relationships and constraints that define valid rotations.

2. Normalization:

   • Quaternions must always have a unit magnitude to represent a valid rotation (i.e., they should be normalized). Performing FFT on individual components may generate results that violate this condition, causing erroneous rotations.

3. Interpolation Issues:

   • When applying smoothing directly to quaternion components, you end up with interpolations that can become invalid or lead to unexpected results (e.g., jerky or “unnatural” rotation paths).

Recommended Approach to Smooth Motion Data

Instead of performing FFT directly on the quaternion components, it’s better to either:

1. Convert Quaternions to Euler Angles, smooth those angles, and then convert them back to quaternions.
2. Use SLERP (Spherical Linear Interpolation) to interpolate between a series of quaternion rotations.
3. Use low-pass filters directly on position or orientation data but in a way that respects quaternion properties.

Below are some approaches you can use:

1. Convert to Euler Angles and Apply FFT

1. Convert Quaternions to Euler Angles:
   • Convert each quaternion to Euler angles (pitch, yaw, roll).
   • Apply FFT and filtering to each Euler angle separately.
   • Convert filtered Euler angles back to quaternions.

This approach works because Euler angles are simpler to handle in a linear context, allowing you to use standard FFT filtering. However, Euler angles are susceptible to gimbal lock, and the conversion adds additional computational complexity.

2. Use SLERP for Smoothing

• SLERP (Spherical Linear Interpolation) is specifically designed for smoothly interpolating between two quaternions. Instead of applying an FFT, you can use SLERP to create a smoother transition between quaternions, which can effectively reduce jitter in the motion.

For example:

Quaternion smoothedRotation = Quaternion.Slerp(previousRotation, currentRotation, smoothingFactor);

• smoothingFactor should be a value between 0 and 1 that controls how much smoothing is applied.
• This method is particularly useful for reducing noise between frames while preserving valid quaternion rotations.

3. Apply a Low-Pass Filter to Quaternion Rotations

• If you want to use a low-pass filter, you can filter each component in a way that respects quaternion properties:

1. Separate Axis-Angle Filtering:

   • Convert the quaternion to an axis-angle representation.
   • Apply filtering to the angle component while maintaining the axis direction.

2. Direct Quaternion Low-Pass Filtering:

   • Instead of applying FFT, apply a simpler low-pass filtering equation that respects the quaternion nature:

   Quaternion filteredRotation = Quaternion.Lerp(previousRotation, currentRotation, smoothingFactor);

   • smoothingFactor should be a small value (0.1 to 0.3), controlling how much weight is given to the current vs. the previous frame.
   • This approach directly interpolates between quaternions, helping reduce jitter while avoiding the non-linear issues associated with FFT.

Example: Applying a Simple Low-Pass Filter to Quaternions

Below is a simplified implementation that applies a low-pass filter directly to quaternions using linear interpolation (Lerp):

private Quaternion LowPassFilterQuaternion(Quaternion previousRotation, Quaternion currentRotation, float smoothingFactor)
{
    // Apply simple linear interpolation (LERP) to smooth the rotation
    return Quaternion.Lerp(previousRotation, currentRotation, smoothingFactor);
}

// Example usage in the LateUpdate loop
private Quaternion previousRotation = Quaternion.identity;

private void LateUpdate()
{
    for (int j = 0; j < (int)JointId.Count; j++)
    {
        JointId jointId = (JointId)j;

        if (MapKinectJoint(jointId) != HumanBodyBones.LastBone && absoluteOffsetMap.ContainsKey(jointId))
        {
            Quaternion currentRotation = KinectDevice.absoluteJointRotations[j];

            // Apply low-pass filter to smooth the rotation
            float smoothingFactor = 0.1f; // Adjust for desired smoothness (0: very smooth, 1: no smoothing)
            Quaternion smoothedRotation = LowPassFilterQuaternion(previousRotation, currentRotation, smoothingFactor);
            previousRotation = smoothedRotation;

            // Get the absolute offset
            Quaternion absOffset = absoluteOffsetMap[jointId];
            Transform finalJoint = PuppetAnimator.GetBoneTransform(MapKinectJoint(jointId));

            // Apply the smoothed rotation
            finalJoint.rotation = absOffset * Quaternion.Inverse(absOffset) * smoothedRotation * absOffset;

            if (j == 0)
            {
                // Root joint - update position with offset
                finalJoint.position = CharacterRootTransform.position +
                                      new Vector3(RootPosition.transform.localPosition.x,
                                                  RootPosition.transform.localPosition.y + OffsetY,
                                                  RootPosition.transform.localPosition.z - OffsetZ);
            }
        }
    }
}

Summary

• Avoid applying FFT directly to quaternion components because quaternions are non-linear, and direct frequency-based filtering can introduce visual artifacts and incorrect deformations.
• Alternatives to FFT for Smoothing:
  1. Convert to Euler Angles and apply FFT if needed, but be cautious of gimbal lock.
  2. Use SLERP for interpolation between frames to achieve smooth and valid transitions.
  3. Apply a low-pass filter using Quaternion.Lerp for real-time smoothing that respects quaternion properties.

For your application involving real-time motion capture, using a simple low-pass filter with Quaternion.Lerp or SLERP would be the most effective way to smooth motion data while avoiding the issues that arise from working directly with quaternion components in the frequency domain. This will provide you with smoother, more natural rotations without the complex artifacts or computational overhead introduced by FFT.

You

Can I just use a weighted moving average window with Quaternion slerp?

ChatGPT

Yes, you can use a weighted moving average approach with Quaternion SLERP to achieve a smooth rotation while respecting the non-linear properties of quaternions. This approach is well-suited to smoothing rotations in real-time, as it leverages SLERP’s ability to correctly interpolate between quaternion rotations and retains valid unit quaternions throughout the smoothing process.

How to Implement Weighted Moving Average with SLERP

A weighted moving average is a technique where each value in the moving window is assigned a weight, and the smoothed value is computed based on those weights. For quaternions, SLERP can be used to smoothly interpolate between the previous rotations and the new incoming quaternion values within the window.

Here’s how you can implement this concept:

1. Maintain a Window of Quaternions:
   • Store a buffer (e.g., a queue or list) of recent quaternions, with the size of the buffer determining the “window size.”
2. Apply SLERP Iteratively:
   • Use SLERP iteratively to compute a weighted average of quaternions in the window. Assign different weights to each quaternion, with more recent quaternions often receiving higher weights.

Step-by-Step Implementation

Below is an implementation that uses a weighted moving average with SLERP to smooth a sequence of quaternions:

using System.Collections.Generic;
using UnityEngine;

public class QuaternionSmoothing : MonoBehaviour
{
    private Queue<Quaternion> rotationBuffer;  // Buffer to store recent rotations
    private int bufferSize = 5;                 // Number of frames in the moving average window
    public float[] weights;                     // Weights for the moving average
    public TrackerHandler KinectDevice;
    public Transform jointToTrack;

    private void Start()
    {
        rotationBuffer = new Queue<Quaternion>(bufferSize);

        // Example weights for a weighted moving average (must sum to 1)
        // Higher weights for more recent rotations
        weights = new float[] { 0.1f, 0.15f, 0.2f, 0.25f, 0.3f }; // Should match bufferSize
    }

    private void LateUpdate()
    {
        // Get the current rotation from the Kinect
        Quaternion currentRotation = KinectDevice.absoluteJointRotations[(int)JointId.WristLeft]; // Example joint

        // Add current rotation to the buffer
        if (rotationBuffer.Count >= bufferSize)
        {
            rotationBuffer.Dequeue(); // Remove the oldest rotation
        }
        rotationBuffer.Enqueue(currentRotation);

        // Calculate smoothed rotation using weighted SLERP
        Quaternion smoothedRotation = WeightedMovingAverageWithSLERP(rotationBuffer, weights);

        // Apply the smoothed rotation to the joint (replace this with your specific needs)
        jointToTrack.rotation = smoothedRotation;
    }

    private Quaternion WeightedMovingAverageWithSLERP(Queue<Quaternion> buffer, float[] weights)
    {
        if (buffer.Count == 0)
        {
            return Quaternion.identity;
        }

        // Start from the first quaternion in the buffer
        Quaternion weightedAverage = Quaternion.identity;
        Quaternion[] quaternions = buffer.ToArray();

        // Apply SLERP iteratively to compute the weighted moving average
        weightedAverage = quaternions[0];
        float totalWeight = weights[0];

        for (int i = 1; i < buffer.Count; i++)
        {
            weightedAverage = Quaternion.Slerp(weightedAverage, quaternions[i], weights[i] / totalWeight);
            totalWeight += weights[i];
        }

        return weightedAverage;
    }
}

Explanation of the Code

1. Buffer of Quaternions:

   • The rotationBuffer stores a fixed number of recent rotations (bufferSize). As new rotations are added, old ones are removed to maintain a sliding window of data.

2. Weights for Weighted Moving Average:

   • The weights array contains weights for each quaternion in the buffer. The weights are designed to sum to 1.0. This is a weighted moving average where more recent rotations may have higher weights to give them more influence on the final result.
   • You can adjust the weights array based on how much smoothing you need. For example, if you want the current frame to have more influence, assign a higher weight to the most recent quaternion.

3. Weighted SLERP Calculation:

   • The WeightedMovingAverageWithSLERP method iteratively applies SLERP to calculate the final weighted average.
   • The first quaternion is taken as the starting value (weightedAverage).
   • Each quaternion in the buffer is then SLERPed with the existing weightedAverage value, using its corresponding weight. The weights are normalized by the cumulative total weight.

Considerations and Tuning

1. Buffer Size:

   • The buffer size (bufferSize) controls how many quaternions are included in the moving average. A larger buffer will result in smoother rotations but also introduce more latency.
   • A value of 3 to 5 is a good starting point. You can experiment with larger sizes for smoother results but bear in mind that larger buffers may introduce lag.

2. Weights:

   • The weights array should have the same length as the buffer and should sum to 1.
   • For more smoothing, you can make the weights more even (e.g., 0.2, 0.2, 0.2, 0.2, 0.2).
   • If you want more recent data to have greater influence, assign higher weights to the newer rotations (e.g., 0.1, 0.15, 0.2, 0.25, 0.3).

3. SLERP and Rotations:

   • SLERP (Spherical Linear Interpolation) ensures that all interpolations maintain a unit quaternion, which keeps all rotations valid.
   • Using SLERP for a weighted average maintains the smoothness and geometric properties of the quaternion, resulting in more natural-looking rotations without the risk of artifacts.

4. Normalization of Weights:

   • Ensure the sum of weights is 1. This helps to maintain consistency and avoid unintended rotation biases.

Why This Approach is Effective

• No Deformations: By using SLERP to average quaternions, you avoid the problems of gimbal lock and non-linear interpolation artifacts that can occur when processing quaternion components directly.
• More Natural Smoothing: Weighted SLERP provides a smoother transition between rotations, suitable for real-time applications like motion capture and character animation.
• Adaptability: You can easily modify the weights and buffer size to fine-tune the amount of smoothing and responsiveness required for your specific application, such as fast combat moves or general body motion.

Summary

Using a weighted moving average with Quaternion SLERP is an effective way to smooth quaternion-based rotation data, as it respects the non-linear and unit properties of quaternions. This approach will provide you with smooth, natural rotations that maintain valid geometric relationships, ensuring your avatar’s movements are smooth and realistic without unwanted artifacts. By tuning the buffer size and weights, you can control how responsive or smooth the resulting rotations are, making it highly adaptable to your needs.

You

what is a good function for weighting average windows of movement data?

ChatGPT

Choosing a good function for weighting the average windows of movement data depends on the specific type of motion you want to achieve—whether it’s a very smooth output or a quick response to new inputs. The weights determine how much influence each frame in the window has on the final output, allowing you to emphasize either recent movements (for responsiveness) or past movements (for smoothing). Below, I’ll cover a few popular and effective weighting functions:

1. Linear Weighting

In linear weighting, each sample is given a linearly increasing or decreasing weight. This approach ensures that more recent data points have more influence compared to older data points. It is straightforward and works well for achieving a balance between responsiveness and smoothing.

• Formula: $ w_i = \frac{i + 1}{N} $ where:

  • $ w_i $ is the weight for the $ i $-th data point.
  • $ i $ starts from 0 for the oldest data point up to $ N-1 $ for the newest.
  • $ N $ is the total number of data points in the buffer.

• Example Weights (for buffer size of 5): [0.2, 0.4, 0.6, 0.8, 1.0]

• Use Case: When you want newer data points to have more influence while still considering past data. It’s good for general-purpose smoothing where recent changes need to be reflected moderately quickly.

2. Exponential Weighting (EMA)

Exponential weighting provides a smoother output and gives exponentially decreasing weight to older data points. This means the most recent values have significantly more influence.

• Formula: $ w_i = (1 - \alpha)^{N-i} $ where:

  • $ \alpha $ is the smoothing factor, typically between 0.1 and 0.3.
  • $ N $ is the total number of data points.
  • $ i $ is the index of each data point, with $ i = N-1 $ being the most recent.

• Example Weights (for buffer size of 5, α = 0.2): [0.4096, 0.512, 0.64, 0.8, 1.0]

• Use Case: Exponential weighting is effective when you need strong smoothing while allowing the system to still respond to new data. It’s a common approach for velocity or rotational smoothing in motion capture applications.

3. Gaussian Weighting

Gaussian weighting applies a bell-shaped distribution to the data points, assigning higher weights to the middle of the window and decreasing weights towards the edges.

• Formula: $ w_i = e^{-\frac{(i - \mu)^2}{2\sigma^2}} $ where:

  • $ \mu $ is the mean of the window (usually the midpoint).
  • $ \sigma $ controls the spread (how quickly weights decrease as you move away from the center).

• Example Weights (for buffer size of 5, μ = 2, σ = 1.0): [0.1353, 0.3679, 0.6065, 0.3679, 0.1353]

• Use Case: Gaussian weighting is useful when you want to emphasize data in the middle of the window, making it suitable for predictive movement or oscillatory behavior, where the entire window represents a meaningful event that you want to give more attention to.

4. Triangular Weighting

This is similar to Gaussian weighting but less computationally expensive. It applies linearly increasing weights up to the midpoint, then decreasing weights symmetrically.

• Formula: $ w_i = \frac{1 - |i - \mu|}{\mu} $ where:

  • $ \mu $ is the midpoint of the window.

• Example Weights (for buffer size of 5): [0.2, 0.4, 0.6, 0.4, 0.2]

• Use Case: Triangular weighting is simpler to implement than Gaussian and is good for applications that require smooth averaging but with less complexity.

5. Uniform Weighting (Simple Moving Average)

This method assigns equal weight to all data points in the window, making it the simplest form of moving average.

• Formula: $ w_i = \frac{1}{N} $ where:

  • $ N $ is the number of points in the window.

• Example Weights (for buffer size of 5): [0.2, 0.2, 0.2, 0.2, 0.2]

• Use Case: This method is ideal if you want to give equal importance to all recent data without any emphasis on newer or older points. It’s suitable for very general, broad smoothing, but it lacks responsiveness.

Example Implementation with Quaternion SLERP

For motion smoothing with Quaternion SLERP, a linear or exponential weighting approach often provides the best balance of smoothness and responsiveness. Below, I show an example of using linear weights with SLERP:

private Quaternion WeightedMovingAverageWithSLERP(Queue<Quaternion> buffer)
{
    if (buffer.Count == 0)
    {
        return Quaternion.identity;
    }

    // Linear weights for a buffer size of 5, where recent values have more influence
    float[] weights = new float[buffer.Count];
    float totalWeight = 0f;
    for (int i = 0; i < buffer.Count; i++)
    {
        weights[i] = (i + 1) / (float)buffer.Count; // Linear weight calculation
        totalWeight += weights[i];
    }

    Quaternion weightedAverage = buffer.ToArray()[0];
    Quaternion[] quaternions = buffer.ToArray();

    // Apply SLERP with weights
    for (int i = 1; i < buffer.Count; i++)
    {
        float normalizedWeight = weights[i] / totalWeight; // Normalize weights so they sum to 1
        weightedAverage = Quaternion.Slerp(weightedAverage, quaternions[i], normalizedWeight);
    }

    return weightedAverage;
}

Summary of Weighting Function Recommendations

• Linear Weighting: Useful for general-purpose smoothing while giving more recent frames more influence. Good for dynamic but controlled movement.
• Exponential Weighting (EMA): Best for stronger smoothing while still maintaining responsiveness. Ideal for sudden movements like combat or sports.
• Gaussian Weighting: Suitable when predictive or oscillatory behavior needs to be highlighted, but computationally more intensive.
• Triangular Weighting: A less expensive approximation of Gaussian weighting for smooth emphasis on the central window.
• Uniform Weighting (Simple Moving Average): Good for basic smoothing, but lacks emphasis on recent data, leading to slower response times.

Practical Suggestion

For real-time motion capture applications that require smoothing but with quick adaptation to changes, use exponential or linear weighting with SLERP:

• Start with a linear weighting approach if you need moderate smoothing.
• Use exponential weighting if there is noticeable jitter that needs stronger attenuation.

In both cases, applying SLERP to calculate the weighted average helps maintain valid quaternions and ensures smooth, natural-looking rotations without introducing artifacts or instability in the character movement.

You

apply exponential smoothing

ChatGPT

To apply exponential smoothing to the quaternion rotation data in your Unity implementation, you can use SLERP to blend the incoming quaternion with the previous smoothed value. This effectively gives more influence to the more recent data, allowing for a smooth output while maintaining responsiveness to new motion data. Exponential smoothing is controlled by a smoothing factor (α), which dictates the influence of recent vs. past data.

Here’s how to implement exponential smoothing with quaternions using SLERP:

• Smoothing Factor (α):
  • A value between 0 and 1.
  • A higher value (e.g., 0.8) makes the system more responsive (quick to react to new data but less smooth).
  • A lower value (e.g., 0.1) makes the system smoother (more stable but slower to react to new changes).

The exponential smoothing formula for rotation using quaternions can be represented as: $ Q_{\text{smoothed}} = \text{SLERP}(Q_{\text{previousSmoothed}}, Q_{\text{current}}, \alpha) $ Where:

• Q_previousSmoothed is the quaternion from the previous frame after smoothing.
• Q_current is the current quaternion (new incoming data).
• α is the smoothing factor.

Implementation in Unity

Below is an implementation in Unity C# that applies exponential smoothing using Quaternion.Slerp:

using UnityEngine;

public class ExponentialQuaternionSmoothing : MonoBehaviour
{
    public TrackerHandler KinectDevice;
    public Transform jointToTrack; // The joint transform in Unity that you want to animate
    public float smoothingFactor = 0.2f; // Smoothing factor (0 < α ≤ 1). Higher values = more responsiveness, lower values = smoother

    private Quaternion previousSmoothedRotation = Quaternion.identity; // Store the previous smoothed rotation

    private void Start()
    {
        // Initialize the previous smoothed rotation to the initial rotation of the tracked joint
        if (jointToTrack != null)
        {
            previousSmoothedRotation = jointToTrack.rotation;
        }
    }

    private void LateUpdate()
    {
        // Assuming you are smoothing the rotation of a specific joint, for example WristLeft
        Quaternion currentRotation = KinectDevice.absoluteJointRotations[(int)JointId.WristLeft]; // Replace with your specific joint index

        // Apply exponential smoothing using SLERP
        Quaternion smoothedRotation = Quaternion.Slerp(previousSmoothedRotation, currentRotation, smoothingFactor);

        // Update the joint's rotation in Unity
        jointToTrack.rotation = smoothedRotation;

        // Store the smoothed rotation for use in the next frame
        previousSmoothedRotation = smoothedRotation;
    }
}

Explanation

1. Variables:

   • smoothingFactor: The exponential smoothing factor (α) that controls the influence of the new quaternion on the final result.
     • Values closer to 0 (e.g., 0.1) provide stronger smoothing.
     • Values closer to 1 (e.g., 0.9) make the rotation more responsive to recent data.
   • previousSmoothedRotation: Stores the previous smoothed rotation value to be used in the current frame’s calculation.

2. Smoothing with SLERP:

   • The Quaternion.Slerp function is used to interpolate between the previousSmoothedRotation (from the last frame) and currentRotation (new incoming data) based on the smoothingFactor.
   • SLERP (Spherical Linear Interpolation) ensures that the resulting quaternion is a valid rotation, avoiding problems like gimbal lock or losing the unit quaternion constraint.

3. Store the Smoothed Value:

   • After computing the smoothedRotation, it is stored in previousSmoothedRotation so it can be used in the next frame to continue the exponential smoothing.

Tuning the Smoothing Factor

• The smoothing factor (α) determines the balance between smoothness and responsiveness:
  • Lower values (e.g., 0.1 to 0.3): Provide smoother, more stable output at the cost of lag in reacting to sudden changes.
  • Higher values (e.g., 0.7 to 0.9): Provide more responsiveness, which means the smoothed value will more quickly reflect new incoming data, but it may be less stable or slightly jittery.

For motion capture applications where you need to reduce noise but still reflect dynamic movements, a value of around 0.2 to 0.5 is often a good starting point. You can tune this based on the specific nature of the motion you are capturing (e.g., rapid combat moves might require a higher value to avoid lag).

Example Use Cases

• Slow and Smooth Motion (e.g., Yoga poses): Use a lower smoothing factor, around 0.1.
• Dynamic Movements (e.g., boxing, running): Use a higher smoothing factor, around 0.5 to 0.7.
• Interactive Applications (e.g., VR avatar mimicry): Use a value that balances between smoothness and responsiveness, such as 0.3 to 0.4.

Advantages of Exponential Smoothing with SLERP

1. Maintains Valid Rotations:

   • Since SLERP ensures that the interpolated quaternion remains unit length, the resulting rotations are valid and do not suffer from deformations or numerical instability.

2. Adaptability:

   • The smoothing factor can be easily adjusted to achieve the desired balance between responsiveness and smoothness, which is crucial in different motion capture scenarios.

3. Low Computational Complexity:

   • Unlike FFT or other frequency-domain approaches, exponential smoothing is computationally efficient and suitable for real-time applications where low latency is essential.

Summary

Using exponential smoothing with Quaternion.Slerp is an excellent method for reducing jitter in motion capture data while preserving the smoothness of natural human movements. By adjusting the smoothing factor (α), you can control how responsive the motion is to new input while avoiding the issues that arise from directly applying spectral analysis to quaternion components.

This method ensures that your animated character moves smoothly in a way that looks natural, without introducing the artifacts and inaccuracies that can occur with other filtering methods when working with rotation data.

You

let’s take a step back.

implement a weighted moving average, where the weights are governed by a parametric exponential function

ChatGPT

Got it. Let’s create a weighted moving average implementation where the weights are defined by a parametric exponential function. This approach allows us to apply an exponentially decaying influence to each rotation in the buffer. More recent data points will have higher weights, which naturally emphasizes new input and maintains a smoother output.

The parametric exponential function for weights can be defined as:

$ w_i = e^{-\lambda \cdot i} $

Where:

• $ w_i $ is the weight for the $i$-th data point (where $i = 0$ represents the most recent data point).
• $ \lambda $ is a decay parameter that determines how quickly the weights decrease:
  • Higher values of $\lambda$ give more emphasis on recent data and reduce the impact of older data quickly.
  • Lower values provide a slower decay, resulting in a smoother but less responsive output.

We will use Quaternion SLERP for averaging to maintain valid quaternion rotations.

Implementation in Unity C#

Here’s a complete Unity implementation of a weighted moving average with weights governed by a parametric exponential function:

using System.Collections.Generic;
using UnityEngine;

public class WeightedMovingAverageWithExponentialDecay : MonoBehaviour
{
    public TrackerHandler KinectDevice;
    public Transform jointToTrack; // The joint transform in Unity that you want to animate
    public int bufferSize = 5; // Number of frames in the moving average window
    public float lambda = 0.5f; // Decay parameter for exponential weighting (higher values = more emphasis on recent data)

    private Queue<Quaternion> rotationBuffer; // Buffer to store recent quaternions

    private void Start()
    {
        // Initialize the buffer
        rotationBuffer = new Queue<Quaternion>(bufferSize);
    }

    private void LateUpdate()
    {
        // Get the current rotation from the Kinect
        Quaternion currentRotation = KinectDevice.absoluteJointRotations[(int)JointId.WristLeft]; // Replace with your specific joint index

        // Add current rotation to the buffer
        if (rotationBuffer.Count >= bufferSize)
        {
            rotationBuffer.Dequeue(); // Remove the oldest rotation to keep the buffer size fixed
        }
        rotationBuffer.Enqueue(currentRotation);

        // Calculate the smoothed rotation using weighted moving average with exponential decay
        Quaternion smoothedRotation = CalculateWeightedMovingAverageWithExponentialDecay(rotationBuffer, lambda);

        // Apply the smoothed rotation to the joint
        jointToTrack.rotation = smoothedRotation;
    }

    private Quaternion CalculateWeightedMovingAverageWithExponentialDecay(Queue<Quaternion> buffer, float lambda)
    {
        if (buffer.Count == 0)
        {
            return Quaternion.identity; // If the buffer is empty, return identity quaternion
        }

        Quaternion[] quaternions = buffer.ToArray();
        int count = quaternions.Length;

        // Calculate weights using the exponential decay function
        float[] weights = new float[count];
        float totalWeight = 0.0f;

        for (int i = 0; i < count; i++)
        {
            weights[i] = Mathf.Exp(-lambda * i); // Calculate weight for each element in the buffer
            totalWeight += weights[i];
        }

        // Normalize weights so that their sum is equal to 1
        for (int i = 0; i < count; i++)
        {
            weights[i] /= totalWeight;
        }

        // Apply SLERP with weights to calculate the weighted average
        Quaternion weightedAverage = quaternions[0]; // Start with the most recent quaternion

        for (int i = 1; i < count; i++)
        {
            weightedAverage = Quaternion.Slerp(weightedAverage, quaternions[i], weights[i]);
        }

        return weightedAverage;
    }
}

Explanation of Code

1. Buffer Initialization:

   • A queue (rotationBuffer) is used to store a fixed number of recent quaternions, with a maximum size of bufferSize.
   • As new rotations are added, older ones are removed, maintaining a sliding window of recent rotations.

2. Exponential Weights Calculation:

   • The weights are calculated using an exponential decay function: $ w_i = e^{-\lambda \cdot i} $.
   • lambda controls how quickly the weights decay:
     • Higher values of lambda mean more recent data points dominate, leading to a more responsive result.
     • Lower values provide slower decay and result in smoother movement.

3. Normalization of Weights:

   • The calculated weights are normalized so that their sum equals 1.
   • This ensures that the final weighted average is a true representation of all the quaternions in the buffer, without overemphasizing any single value.

4. Weighted Moving Average with SLERP:

   • The SLERP function is used to iteratively calculate the weighted average.
   • The SLERP operation is applied iteratively, starting with the most recent quaternion.
   • SLERP is well-suited to this task because it maintains the unit properties of quaternions, preventing invalid rotations.

Tuning the Decay Parameter (λ)

• lambda controls the emphasis on recent data versus older data:
  • A larger value (e.g., 1.0 or higher) will prioritize more recent quaternions, making the result more responsive to changes.
  • A smaller value (e.g., 0.1 to 0.3) will provide a smoother result by giving more influence to the older rotations in the buffer.

Example Values for Parameters

• bufferSize:
  • Start with a buffer size of 5.
  • Increase it if you need smoother results (e.g., 10) or decrease for more responsiveness.
• lambda:
  • Start with 0.5 as a middle-ground value.
  • Increase if you want the character to react more quickly to new movements (e.g., 1.0).
  • Decrease if you need smoother movements (e.g., 0.2).

Advantages of This Approach

1. Smoothness with Responsiveness:

   • By using an exponential decay function, you can provide more weight to recent movements while retaining some influence from older movements.
   • This ensures that the final output is smooth but responsive enough to accurately reflect the user’s intent.

2. Maintains Valid Quaternions:

   • Using SLERP maintains the validity of the quaternions, avoiding issues like non-normalization and ensuring the rotations are smooth and natural.

3. Adaptability:

   • The decay parameter (lambda) is easy to adjust, allowing for different levels of smoothness or responsiveness based on the specific type of movement you are aiming for (e.g., gentle motion vs. rapid reactions).

Summary

The weighted moving average with parametric exponential decay is an effective technique to smooth quaternion rotation data while still emphasizing more recent information. This implementation uses SLERP to ensure that all rotations are valid and provides a flexible approach to tune the movement smoothing using the decay parameter (lambda).

• A lower lambda value gives you smoother movements suitable for applications like dance or slow gestures.
• A higher lambda value makes the motion more responsive, which is useful for dynamic activities like sports or combat.

By adjusting the buffer size and lambda parameter, you can tailor the behavior to suit different types of movements and interaction contexts, ensuring a balance between smoothness and real-time responsiveness.