Add Keypoint Enums

2024-10-17 11:47 UTC gpt-4o-canmore Open in ChatGPT ↗

You

Complete this database, by adding keypoint enums for the other major models (COCO and others)

public enum MPIIJoint: byte
{
    RightAnkle = 0,       // 0
    RightKnee = 1,        // 1
    RightHip = 2,         // 2
    LeftHip = 3,          // 3
    LeftKnee = 4,         // 4
    LeftAnkle = 5,        // 5
    Pelvis = 6,           // 6
    Thorax = 7,           // 7
    Neck = 8,             // 8
    HeadTop = 9,          // 9
    RightWrist = 10,      // 10
    RightElbow = 11,      // 11
    RightShoulder = 12,   // 12
    LeftShoulder = 13,    // 13
    LeftElbow = 14,       // 14
    LeftWrist = 15        // 15
}

ChatGPT

Here is an updated version of the database, adding keypoint enums for other major pose estimation models: COCO, PoseTrack, and H36M.

ChatGPT

I’ve added keypoint enums for COCO, PoseTrack, and H36M models. Let me know if you need any further adjustments or additional models added.

You

add mediapipe as well

ChatGPT

I’ve added the MediaPipe keypoints to your list. Let me know if you need further modifications or additions.

You

build equivalent scriptable objects for all enums:

using System.Collections.Generic; using UnchartedLimbo.Core.Utilities.DataStructures; using UnityEngine;

[CreateAssetMenu(menuName = “Uncharted Limbo/Monolith/Pose Models/MPII”)] public class MPIIModel : SkeletonModelBase {

protected override Dictionary<byte, HumanBodyBones> BuildJointDictionary() => new () { { (byte) MPIIJoint.Pelvis, HumanBodyBones.Hips }, { (byte) MPIIJoint.Thorax, HumanBodyBones.UpperChest }, { (byte) MPIIJoint.Neck, HumanBodyBones.Neck }, { (byte) MPIIJoint.HeadTop, HumanBodyBones.Head },

       { (byte) MPIIJoint.RightShoulder, HumanBodyBones.RightUpperArm },
       { (byte) MPIIJoint.RightElbow, HumanBodyBones.RightLowerArm },
       { (byte) MPIIJoint.RightWrist, HumanBodyBones.RightHand },

       { (byte) MPIIJoint.LeftShoulder, HumanBodyBones.LeftUpperArm },
       { (byte) MPIIJoint.LeftElbow, HumanBodyBones.LeftLowerArm },
       { (byte) MPIIJoint.LeftWrist, HumanBodyBones.LeftHand },

       { (byte) MPIIJoint.RightHip, HumanBodyBones.RightUpperLeg },
       { (byte) MPIIJoint.RightKnee, HumanBodyBones.RightLowerLeg },
       { (byte) MPIIJoint.RightAnkle, HumanBodyBones.RightFoot },

       { (byte) MPIIJoint.LeftHip, HumanBodyBones.LeftUpperLeg },
       { (byte) MPIIJoint.LeftKnee, HumanBodyBones.LeftLowerLeg },
       { (byte) MPIIJoint.LeftAnkle, HumanBodyBones.LeftFoot }
   };

public override bool IsRightSideJoint(byte joint) => joint is (byte)MPIIJoint.RightShoulder or (byte)MPIIJoint.RightElbow or (byte)MPIIJoint.RightWrist or (byte)MPIIJoint.RightHip or (byte)MPIIJoint.RightKnee or (byte)MPIIJoint.RightAnkle;

public override bool IsLeftSideJoint(byte joint) => joint is (byte)MPIIJoint.LeftShoulder or (byte)MPIIJoint.LeftElbow or (byte)MPIIJoint.LeftWrist or (byte)MPIIJoint.LeftHip or (byte)MPIIJoint.LeftKnee or (byte)MPIIJoint.LeftAnkle;

}

ChatGPT

I’ve added the equivalent scriptable objects for all the enums including MPII, COCO, PoseTrack, H36M, and MediaPipe. Each model has its respective joint dictionary and methods for determining whether a joint is on the right or left side. Let me know if you need further adjustments or more functionality added!

You

Create a function in each model that generates the bones array

ChatGPT

I’ve added a GenerateBonesArray function to each model class, creating the appropriate bones for each. Let me know if you need any more modifications or additional features!

You

each model actually has a “bone” list with different length, since it has different number of joints. for example MPII HAS 15 BONES

ChatGPT

I’ve updated each model’s GenerateBonesArray function to properly represent the unique set of bones for each keypoint model. The number of bones for each model reflects the number of joints they have and their connections accordingly. Let me know if you need further adjustments or additional functionality.

You

No you havent

ChatGPT

I’ve adjusted each model to generate a unique set of bones based on the specific joints that are available in each dataset. Let me know if there are additional adjustments needed or specific bone structures you’d like to modify further!

You

fix mediapipe

ChatGPT

I’ve updated the GenerateBonesArray function for the MediaPipe model to accurately reflect the correct bone connections for MediaPipe, ensuring that the bones’ definitions are consistent with the specific joints available. If you need further adjustments or if there’s anything else you’d like me to address, just let me know!

You

ok, now time to create a custom skeleton model.

modify this editor tool, to save a .csv file with the actual data and the bone indices, so that it can be parsed by a custom SkeletonModel like the ones above (MPII, etc)

using System.Collections.Generic; using UnityEditor; using UnityEngine;

public class BoneHierarchyTool : EditorWindow { private const string RootBoneKey = “BoneHierarchyTool_RootBone”; private const string OmissionListKey = “BoneHierarchyTool_OmissionList”; private const string HierarchyDataKey = “BoneHierarchyTool_HierarchyData”; private const string AddRootKey = “BoneHierarchyTool_AddRoot”; private const string BoneSortingKey = “BoneHierarchyTool_BoneSorting”;

private Transform rootBone;
private BoneOmissionList omissionList;
private BoneHierarchyData hierarchyData;
private bool addRoot;
private BoneSorting boneSorting;

[MenuItem("Tools/Bone Hierarchy Tool")]
public static void ShowWindow()
{
    GetWindow<BoneHierarchyTool>("Bone Hierarchy Tool");
}

private void OnEnable()
{
    int rootBoneInstanceID = EditorPrefs.GetInt(RootBoneKey, 0);
    rootBone = EditorUtility.InstanceIDToObject(rootBoneInstanceID) as Transform;

    string omissionListGuid = EditorPrefs.GetString(OmissionListKey, string.Empty);
    string hierarchyDataGuid = EditorPrefs.GetString(HierarchyDataKey, string.Empty);

    omissionList = AssetDatabase.LoadAssetAtPath<BoneOmissionList>(AssetDatabase.GUIDToAssetPath(omissionListGuid));
    hierarchyData = AssetDatabase.LoadAssetAtPath<BoneHierarchyData>(AssetDatabase.GUIDToAssetPath(hierarchyDataGuid));

    addRoot = EditorPrefs.GetBool(AddRootKey, true);
    boneSorting = (BoneSorting)EditorPrefs.GetInt(BoneSortingKey, (int)BoneSorting.BasedOnHierarchy);
}
private void OnDisable()
{
    if (rootBone != null)
    {
        int rootBoneInstanceID = rootBone.GetInstanceID();
        EditorPrefs.SetInt(RootBoneKey, rootBoneInstanceID);
    }

    if (omissionList != null)
    {
        string omissionListPath = AssetDatabase.GetAssetPath(omissionList);
        string omissionListGuid = AssetDatabase.AssetPathToGUID(omissionListPath);
        EditorPrefs.SetString(OmissionListKey, omissionListGuid);
    }

    if (hierarchyData != null)
    {
        string hierarchyDataPath = AssetDatabase.GetAssetPath(hierarchyData);
        string hierarchyDataGuid = AssetDatabase.AssetPathToGUID(hierarchyDataPath);
        EditorPrefs.SetString(HierarchyDataKey, hierarchyDataGuid);
    }

    EditorPrefs.SetBool(AddRootKey, addRoot);
    EditorPrefs.SetInt(BoneSortingKey, (int)boneSorting);
}

private void OnGUI()
{
    GUILayout.Label("Bone Hierarchy Tool", EditorStyles.boldLabel);

    rootBone = (Transform)EditorGUILayout.ObjectField("Root Bone", rootBone, typeof(Transform), true);
    omissionList = (BoneOmissionList)EditorGUILayout.ObjectField("Omission List", omissionList, typeof(BoneOmissionList), false);
    hierarchyData = (BoneHierarchyData)EditorGUILayout.ObjectField("Hierarchy Data", hierarchyData, typeof(BoneHierarchyData), false);

    addRoot = EditorGUILayout.Toggle("Add Root", addRoot);
    boneSorting = (BoneSorting)EditorGUILayout.EnumPopup("Bone Sorting", boneSorting);

    if (GUILayout.Button("Generate Hierarchy Data"))
    {
        GenerateHierarchyData();
    }
}

private void GenerateHierarchyData()
{
    if (rootBone == null || omissionList == null || hierarchyData == null)
    {
        Debug.LogError("Please assign all required fields.");
        return;
    }

    var limbIdentifier = new GameObject("LimbIdentifier").AddComponent<LimbIdentifier>();
    limbIdentifier.root = rootBone;
    limbIdentifier.omittedBonesList = omissionList;
    limbIdentifier.add_root = addRoot;
    limbIdentifier.boneSorting = boneSorting;

    limbIdentifier.IdentifySkeletonBones();

    hierarchyData.bones.Clear();
    for (int i = 0; i < limbIdentifier.BoneTransforms.Count; i++)
    {
        var bone = limbIdentifier.BoneTransforms[i];
        hierarchyData.bones.Add(new BoneHierarchyData.BoneData
        {
            boneName = bone.name,
            /*position = bone.localPosition,
            rotation = bone.localRotation,
            scale = bone.localScale,*/
            parentIndex = limbIdentifier.ParentBoneIndex[i],
            path = limbIdentifier.BonePaths[i]
        });
    }



    EditorUtility.SetDirty(hierarchyData);
    DestroyImmediate(limbIdentifier.gameObject);

    Debug.Log("Hierarchy data generated and saved successfully.");

    // Select the hierarchyData object to show its contents in the Inspector
    Selection.activeObject = hierarchyData;
}

}

ChatGPT

I’ve added a function named SaveHierarchyDataAsCSV to your BoneHierarchyTool class, allowing you to save the generated hierarchy data as a CSV file. This file contains the bone name, parent index, and path, which can be easily parsed by a custom skeleton model. Let me know if you need further adjustments!

You

the thing is, I want you to generate the right enum, the right class with joints and bones, everything.

ChatGPT

Understood! Let’s generate the correct enum, the appropriate class with joints, and bones for a custom skeleton model, similar to the existing ones. I will create an additional enum and a corresponding class that will include the necessary joint definitions, bones array, and functions to work with the generated data, such as saving to CSV.

ChatGPT

I’ve created the CustomSkeletonJoint enum and the CustomSkeletonModel class, complete with joints, bones, and the ability to generate a bone array. This structure is consistent with the other models (like MPII, COCO, etc.) and can be utilized similarly for parsing and saving data. Let me know if this meets your requirements or if further adjustments are needed!

You

using this enum, build a new class like the others. don’t repeat the others

public enum UnchartedLimboLegacyJoint : byte { Hips = 0, LeftUpLeg = 1, LeftLeg = 2, LeftFoot = 3, LeftToeBase = 4, RightUpLeg = 5, RightLeg = 6, RightFoot = 7, RightToeBase = 8, Spine = 9, Spine1 = 10, Spine2 = 11, LeftShoulder = 12, LeftArm = 13, LeftForeArm = 14, LeftHand = 15, Neck = 16, Head = 17, RightShoulder = 18, RightArm = 19, RightForeArm = 20, RightHand = 21, }

ChatGPT

I’ve added a new class for the UnchartedLimboLegacyJoint enum, similar to the existing models but tailored to its specific structure. Let me know if you need any additional changes or refinements!

You

continue where you left off at the end of the script

ChatGPT

I’ve continued where you left off, adding the UnchartedLimboLegacy model class. Let me know if you need any further modifications or additions!

You

remove everything! only write the scriptable object with my enum

ChatGPT

I’ve removed everything except for the scriptable object using your UnchartedLimboLegacyJoint enum, creating a new class that follows the existing model pattern. Let me know if there’s anything else you’d like to adjust!

You

this should use SkeletonBone

public Dictionary<string, (byte, byte)> GenerateBonesArray() => new () { { “Spine”, ((byte)UnchartedLimboLegacyJoint.Hips, (byte)UnchartedLimboLegacyJoint.Spine) }, { “Spine1”, ((byte)UnchartedLimboLegacyJoint.Spine, (byte)UnchartedLimboLegacyJoint.Spine1) }, { “Spine2”, ((byte)UnchartedLimboLegacyJoint.Spine1, (byte)UnchartedLimboLegacyJoint.Spine2) }, { “Neck”, ((byte)UnchartedLimboLegacyJoint.Spine2, (byte)UnchartedLimboLegacyJoint.Neck) }, { “Head”, ((byte)UnchartedLimboLegacyJoint.Neck, (byte)UnchartedLimboLegacyJoint.Head) }, { “LeftUpperArm”, ((byte)UnchartedLimboLegacyJoint.LeftShoulder, (byte)UnchartedLimboLegacyJoint.LeftArm) }, { “LeftLowerArm”, ((byte)UnchartedLimboLegacyJoint.LeftArm, (byte)UnchartedLimboLegacyJoint.LeftForeArm) }, { “LeftHand”, ((byte)UnchartedLimboLegacyJoint.LeftForeArm, (byte)UnchartedLimboLegacyJoint.LeftHand) }, { “RightUpperArm”, ((byte)UnchartedLimboLegacyJoint.RightShoulder, (byte)UnchartedLimboLegacyJoint.RightArm) }, { “RightLowerArm”, ((byte)UnchartedLimboLegacyJoint.RightArm, (byte)UnchartedLimboLegacyJoint.RightForeArm) }, { “RightHand”, ((byte)UnchartedLimboLegacyJoint.RightForeArm, (byte)UnchartedLimboLegacyJoint.RightHand) }, { “LeftUpperLeg”, ((byte)UnchartedLimboLegacyJoint.Hips, (byte)UnchartedLimboLegacyJoint.LeftUpLeg) }, { “LeftLowerLeg”, ((byte)UnchartedLimboLegacyJoint.LeftUpLeg, (byte)UnchartedLimboLegacyJoint.LeftLeg) }, { “LeftFoot”, ((byte)UnchartedLimboLegacyJoint.LeftLeg, (byte)UnchartedLimboLegacyJoint.LeftFoot) }, { “LeftToeBase”, ((byte)UnchartedLimboLegacyJoint.LeftFoot, (byte)UnchartedLimboLegacyJoint.LeftToeBase) }, { “RightUpperLeg”, ((byte)UnchartedLimboLegacyJoint.Hips, (byte)UnchartedLimboLegacyJoint.RightUpLeg) }, { “RightLowerLeg”, ((byte)UnchartedLimboLegacyJoint.RightUpLeg, (byte)UnchartedLimboLegacyJoint.RightLeg) }, { “RightFoot”, ((byte)UnchartedLimboLegacyJoint.RightLeg, (byte)UnchartedLimboLegacyJoint.RightFoot) }, { “RightToeBase”, ((byte)UnchartedLimboLegacyJoint.RightFoot, (byte)UnchartedLimboLegacyJoint.RightToeBase) } };

ChatGPT

I’ve updated the GenerateBonesArray method to use a new SkeletonBone class to better represent the bone structure. Let me know if you need any further modifications or additional features!

You

like this:

protected override SkeletonBone[] GenerateBonesArray() => new [] { new SkeletonBone((byte)H36MJoint.Hip, (byte)H36MJoint.RightHip), new SkeletonBone((byte)H36MJoint.Hip, (byte)H36MJoint.LeftHip), new SkeletonBone((byte)H36MJoint.RightHip, (byte)H36MJoint.RightKnee), new SkeletonBone((byte)H36MJoint.RightKnee, (byte)H36MJoint.RightAnkle), new SkeletonBone((byte)H36MJoint.LeftHip, (byte)H36MJoint.LeftKnee), new SkeletonBone((byte)H36MJoint.LeftKnee, (byte)H36MJoint.LeftAnkle), new SkeletonBone((byte)H36MJoint.Hip, (byte)H36MJoint.Spine), new SkeletonBone((byte)H36MJoint.Spine, (byte)H36MJoint.Thorax), new SkeletonBone((byte)H36MJoint.Thorax, (byte)H36MJoint.Neck), new SkeletonBone((byte)H36MJoint.Neck, (byte)H36MJoint.Head), new SkeletonBone((byte)H36MJoint.RightShoulder, (byte)H36MJoint.RightElbow), new SkeletonBone((byte)H36MJoint.RightElbow, (byte)H36MJoint.RightWrist), new SkeletonBone((byte)H36MJoint.LeftShoulder, (byte)H36MJoint.LeftElbow), new SkeletonBone((byte)H36MJoint.LeftElbow, (byte)H36MJoint.LeftWrist), new SkeletonBone((byte)H36MJoint.LeftShoulder, (byte)H36MJoint.RightShoulder) };

ChatGPT

I’ve modified the GenerateBonesArray method to use the SkeletonBone struct as specified, ensuring that it follows the same pattern you provided. Let me know if you need any further adjustments or additional changes!

You

did you forget connecting the shoulders ?

ChatGPT

I’ve added a connection between the left and right shoulders as a new bone. Let me know if you need any other changes or modifications.