Pose Classification NN

2025-03-14 12:39 UTC o3-mini-high Open in ChatGPT ↗

Below is an outline of how you can build a simple classifier neural network for classifying body poses based on limb direction vectors:

1. Data Representation

Input Features:
Represent each pose as a numerical vector. For example, if you have the direction (or orientation) for each limb, you might have a vector where each limb contributes several features (e.g., the x, y, z components of a unit vector).
Example: For 10 limbs with 3 components each, your input vector would be of size 30.
Labels:
Your pose categories should be encoded appropriately. If you have, say, 5 different poses, you can use one-hot encoding for the target labels when using categorical crossentropy loss.

2. Neural Network Architecture

A straightforward choice is to use a multi-layer perceptron (MLP) with:

Input Layer: Size equal to your feature vector length.
Hidden Layers: One or two dense layers with a non-linear activation (ReLU is a common choice). You might also add dropout layers for regularization.
Output Layer: A dense layer with as many neurons as pose categories using a softmax activation for multi-class classification.

3. Example Using TensorFlow/Keras

Below is a sample Python code snippet demonstrating how to build, compile, and train such a model:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import Adam

# Example configuration
num_features = 30   # Adjust based on your vector size (e.g., 10 limbs * 3 components)
num_classes = 5     # Number of pose categories

# Build the model
model = Sequential([
    Dense(64, input_dim=num_features, activation='relu'),
    Dropout(0.5),  # Regularization to prevent overfitting
    Dense(32, activation='relu'),
    Dense(num_classes, activation='softmax')  # Softmax for multi-class classification
])

# Compile the model
model.compile(
    loss='categorical_crossentropy',  # Use categorical crossentropy for multi-class classification
    optimizer=Adam(),
    metrics=['accuracy']
)

# Assume X_train, y_train, X_val, y_val are your training and validation datasets
# For example purposes, the training might look like this:
# model.fit(X_train, y_train, epochs=50, batch_size=32, validation_data=(X_val, y_val))

4. Training and Evaluation

Data Preprocessing:
Make sure your input data is normalized. For instance, if your limb direction vectors are already unit vectors, they may be in a good form; otherwise, consider scaling them to a consistent range.
Splitting Data:
Divide your dataset into training, validation, and test sets to assess the model’s generalization.
Hyperparameter Tuning:
Experiment with different numbers of neurons, layers, dropout rates, learning rates, and batch sizes to see what works best for your specific dataset.
Monitoring:
Use callbacks like early stopping based on validation loss to prevent overfitting.

5. Additional Considerations

Feature Engineering:
Depending on the complexity of your poses, you might consider including additional features such as relative angles between limbs or joint positions.
Data Augmentation:
If your dataset is small, consider augmenting your data (e.g., by adding small rotations or noise) to help the model generalize better.
Evaluation Metrics:
Besides accuracy, you might want to look at confusion matrices to understand which poses are misclassified and adjust your model or data accordingly.

This should give you a solid starting point for creating your pose classification neural network. Adjust the model architecture and parameters based on the specifics of your data and the performance you observe during training.

Below is a complete Python example using TensorFlow/Keras that builds, trains, and evaluates a neural network classifier for body poses. This code assumes that you have already prepared your dataset as NumPy arrays (or similar) for training and validation. For illustration, dummy data creation is commented out—you can remove the dummy data block if you have your own data.

import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping
import matplotlib.pyplot as plt

# ---------------------------
# 1. Data Preparation
# ---------------------------
# Assume your dataset is already loaded into these variables:
# X_train, y_train, X_val, y_val

# Uncomment the following block if you need dummy data for testing.
# Replace this with your actual dataset.
# ------------------------------------------------------------------------------
# num_samples_train = 1000
# num_samples_val = 200
# num_features = 30  # e.g., 10 limbs * 3 components each
# num_classes = 5    # number of pose categories
#
# # Dummy feature data: random numbers (in real cases, these are your pose vectors)
# X_train = np.random.random((num_samples_train, num_features))
# X_val = np.random.random((num_samples_val, num_features))
#
# # Dummy label data: random integers converted to one-hot encoding
# y_train = tf.keras.utils.to_categorical(np.random.randint(0, num_classes, num_samples_train), num_classes=num_classes)
# y_val = tf.keras.utils.to_categorical(np.random.randint(0, num_classes, num_samples_val), num_classes=num_classes)
# ------------------------------------------------------------------------------

# ---------------------------
# 2. Build the Model
# ---------------------------
model = Sequential([
    Dense(64, input_dim=30, activation='relu'),  # input_dim should match the number of features
    Dropout(0.5),  # Helps to prevent overfitting
    Dense(32, activation='relu'),
    Dense(5, activation='softmax')  # Output layer: 5 classes with softmax activation
])

# ---------------------------
# 3. Compile the Model
# ---------------------------
model.compile(
    loss='categorical_crossentropy',  # Suitable for multi-class classification
    optimizer=Adam(),
    metrics=['accuracy']
)

# ---------------------------
# 4. Train the Model
# ---------------------------
# Using EarlyStopping to monitor validation loss and avoid overfitting
early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

history = model.fit(
    X_train,
    y_train,
    epochs=50,
    batch_size=32,
    validation_data=(X_val, y_val),
    callbacks=[early_stop]
)

# ---------------------------
# 5. Evaluate the Model
# ---------------------------
val_loss, val_accuracy = model.evaluate(X_val, y_val)
print("Validation Loss: {:.4f}".format(val_loss))
print("Validation Accuracy: {:.4f}".format(val_accuracy))

# ---------------------------
# 6. Plot Training History
# ---------------------------
plt.figure(figsize=(12, 4))

# Plot loss
plt.subplot(1, 2, 1)
plt.plot(history.history['loss'], label='Train Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Loss over Epochs')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()

# Plot accuracy
plt.subplot(1, 2, 2)
plt.plot(history.history['accuracy'], label='Train Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.title('Accuracy over Epochs')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()

plt.tight_layout()
plt.show()

Explanation

Data Preparation:
Replace the dummy data block with your actual data arrays:
- X_train and X_val: Should be arrays of shape (num_samples, 30) if you have 30 features.
- y_train and y_val: Should be one-hot encoded arrays corresponding to the pose categories.
Model Architecture:
The model is a simple MLP:
- First dense layer with 64 neurons and ReLU activation.
- A dropout layer to mitigate overfitting.
- A second dense layer with 32 neurons.
- An output layer with 5 neurons and softmax activation for classification.
Compilation & Training:
The model uses the Adam optimizer and categorical crossentropy loss function. Early stopping is used to prevent overfitting if the validation loss stops improving.
Evaluation & Visualization:
After training, the model is evaluated on the validation set and training history is plotted to review loss and accuracy trends.

This complete code snippet should serve as a solid starting point for your pose classification neural network. Adjust the network architecture and training parameters as needed to suit your specific dataset and application.

Below are the steps to convert your TensorFlow/Keras model into a format suitable for Unity Sentis and a sample C# script to perform runtime inference.

1. Converting Your Model to ONNX

Unity Sentis accepts ONNX models, so you need to convert your trained Keras model into the ONNX format. You can do this with the tf2onnx library. For example:

import tensorflow as tf
import tf2onnx

# Load your pre-trained Keras model (assumes a .h5 file)
model = tf.keras.models.load_model("pose_classifier.h5")

# Define an input signature: here we assume each sample is a 30-dimensional vector.
spec = (tf.TensorSpec((None, 30), tf.float32, name="input"),)

# Convert the model and save to an ONNX file.
output_path = "pose_classifier.onnx"
model_proto, _ = tf2onnx.convert.from_keras(model, input_signature=spec, opset=13, output_path=output_path)
print("ONNX model saved to", output_path)

This code converts your model to ONNX format by providing an input signature that matches your pose vector (e.g., 30 features). You can adjust the dimensions as needed.

2. Importing the ONNX Model into Unity

Add the ONNX file:
Place the pose_classifier.onnx file into your Unity project’s Assets folder.
Install Unity Sentis:
Make sure that the Unity Sentis package is installed in your project (via the Package Manager). Unity Sentis is Unity’s runtime neural network inference engine optimized for platforms like mobile.
Create a Model Asset:
In Unity, create a Sentis model asset (or use the ONNX file directly if Sentis supports that workflow). Typically, you can assign the ONNX file to a public field in your C# script.

3. Runtime Inference in C# Using Unity Sentis

Below is a sample C# script that demonstrates how to load the Sentis model, prepare input data, run inference, and process the output:

using UnityEngine;
using Unity.Sentis;  // Ensure the Sentis namespace is available

public class PoseInference : MonoBehaviour
{
    // Assign your ONNX model asset via the Unity Inspector.
    public SentisModelAsset modelAsset;
    private InferenceEngine engine;

    void Start()
    {
        // Initialize the inference engine with the Sentis model.
        engine = new InferenceEngine(modelAsset);
    }

    /// <summary>
    /// Runs inference on the provided input vector.
    /// </summary>
    /// <param name="inputData">A float array representing a single pose (e.g., length 30).</param>
    /// <returns>Output predictions as a float array.</returns>
    public float[] RunInference(float[] inputData)
    {
        // Create a tensor from the input data.
        // Here, we assume a single sample with 30 features (adjust dimensions if needed).
        Tensor inputTensor = new Tensor(1, inputData.Length, inputData);

        // Execute the model.
        Tensor outputTensor = engine.Execute(inputTensor);

        // Retrieve the results as a float array.
        float[] outputData = outputTensor.ToReadOnlyArray();

        // Dispose tensors to free resources.
        inputTensor.Dispose();
        outputTensor.Dispose();

        return outputData;
    }

    void OnDestroy()
    {
        // Properly dispose the inference engine when done.
        engine.Dispose();
    }
}

Key Points in the C# Code:

Model Asset:
The SentisModelAsset field should be assigned in the Unity Inspector with your ONNX model file (or the Sentis-processed asset).
Inference Engine Initialization:
The InferenceEngine is initialized in the Start() method, which loads the model for runtime inference.
Running Inference:
The RunInference method creates a tensor from your input vector (assumed to be a 1×30 tensor), executes the model, and retrieves the prediction output as a float array.
Resource Management:
Tensors and the inference engine are disposed of properly to manage memory.

Final Notes

Model Conversion:
Adjust the input signature and dimensions in the Python conversion script if your actual data format differs.
Sentis API:
The code above assumes a basic API structure for Unity Sentis. Depending on your Unity version and the Sentis package version, you might need to adjust namespace references or method names. Always refer to the latest Unity Sentis documentation for any updates.
Debugging:
Test the model in the Unity Editor first to ensure that the input data, tensor dimensions, and output parsing align with your model’s architecture.

This workflow should let you convert your Keras model to ONNX and integrate it into Unity for real-time inference using Unity Sentis.