Type Hint Error Fix

2024-09-11 20:03 UTC gpt-4o Open in ChatGPT ↗

You

in this python code I get this error:

def _generate_gaussian( self, keypoints: np.ndarray, keypoints_visible: Optional[np.ndarray] = None ) -> tuple[ndarray, ndarray, ndarray, ndarray]:

tuple[ndarray, ndarray, ndarray, ndarray]: TypeError: ‘type’ object is not subscriptable

ChatGPT

The error you’re seeing (TypeError: 'type' object is not subscriptable) occurs because you’re using tuple[ndarray, ndarray, ndarray, ndarray] which is only valid in Python 3.9 and later versions. However, the ndarray type you’re using is probably from numpy, which should be referenced with the full path np.ndarray.

To fix this error, you have a few options depending on the version of Python you’re using.

Solution 1: For Python 3.9+ (use tuple with type hints)

If you’re using Python 3.9 or later, ensure that np.ndarray is correctly imported from numpy:

from typing import Optional
import numpy as np
from numpy import ndarray

def _generate_gaussian(
    self,
    keypoints: np.ndarray,
    keypoints_visible: Optional[np.ndarray] = None
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    # Your function implementation here

Solution 2: For Python versions older than 3.9

In versions older than 3.9, the tuple type hint syntax like tuple[X, Y] isn’t supported. You should instead use the Tuple from the typing module:

from typing import Optional, Tuple
import numpy as np

def _generate_gaussian(
    self,
    keypoints: np.ndarray,
    keypoints_visible: Optional[np.ndarray] = None
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    # Your function implementation here

By using Tuple from the typing module, you can avoid the 'type' object is not subscriptable error, and the function will work correctly for Python versions below 3.9.

Let me know if you need further assistance!

You

change the code so that it works for both python versions

ChatGPT

To make your code compatible with both Python 3.9+ and older versions, you can check the Python version at runtime and apply the appropriate import depending on which version is being used.

Here’s an example that dynamically handles the type annotations for both Python 3.9+ and earlier versions:

import sys
from typing import Optional
import numpy as np

# Conditional import based on Python version
if sys.version_info >= (3, 9):
    tuple_type = tuple  # Python 3.9+ allows tuple[x, y] directly
else:
    from typing import Tuple
    tuple_type = Tuple  # For older Python versions

def _generate_gaussian(
    self,
    keypoints: np.ndarray,
    keypoints_visible: Optional[np.ndarray] = None
) -> tuple_type[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    # Your function implementation here
    pass

Explanation:

1. Version check: The sys.version_info tuple is used to check whether the Python version is 3.9 or higher.
2. Conditional type assignment: If the version is 3.9+, we use the standard tuple type hint. If the version is older, we import Tuple from typing and use that for type annotations.
3. Usage: The tuple_type is used to annotate the return type so it will work in both environments.

This approach ensures compatibility across Python versions while keeping the code clean. Let me know if you need further clarification or modifications!

You

why can’t i pass named arguments to these?

parser.add_argument(‘det_config’, help=‘Config file for detection’) parser.add_argument(‘det_checkpoint’, help=‘Checkpoint file for detection’) parser.add_argument( ‘pose3d_estimator_config’, type=str, default=None, help=‘Config file for the 3D pose estimator’) parser.add_argument( ‘pose3d_estimator_checkpoint’, type=str, default=None, help=‘Checkpoint file for the 3D pose estimator’)

ChatGPT

The issue you’re encountering is that positional arguments (those without a -- prefix) cannot have default values or be optional. In your code, the arguments det_config, det_checkpoint, pose3d_estimator_config, and pose3d_estimator_checkpoint are being defined as positional arguments, meaning that the user must provide values for them when running the script.

Key Points:

• Positional arguments: These are mandatory and must be passed in the order they are defined when the script is executed. You can’t have a default value for them.
• Named (optional) arguments: These are optional and can be provided in any order, typically prefixed with -- or -. They can have default values.

Fix: Use named arguments for the ones that should be optional or have defaults

If you want to allow named arguments with default values, you need to convert the arguments pose3d_estimator_config and pose3d_estimator_checkpoint to named arguments by prefixing them with --.

Here’s the corrected version of your code:

import argparse

# Initialize parser
parser = argparse.ArgumentParser(description="Example script")

# Positional arguments
parser.add_argument('det_config', help='Config file for detection')
parser.add_argument('det_checkpoint', help='Checkpoint file for detection')

# Named arguments (with default values and optional)
parser.add_argument(
    '--pose3d_estimator_config',  # Named argument with '--'
    type=str,
    default=None,  # Default value
    help='Config file for the 3D pose estimator'
)
parser.add_argument(
    '--pose3d_estimator_checkpoint',  # Named argument with '--'
    type=str,
    default=None,  # Default value
    help='Checkpoint file for the 3D pose estimator'
)

# Parse the arguments
args = parser.parse_args()

# Example use of the arguments
print(args.det_config)
print(args.det_checkpoint)
print(args.pose3d_estimator_config)
print(args.pose3d_estimator_checkpoint)

Explanation of changes:

1. Positional arguments (det_config and det_checkpoint) are kept as they are because they don’t have default values and are mandatory.
2. Named arguments (pose3d_estimator_config and pose3d_estimator_checkpoint) are now optional, prefixed with --, and can be passed as named arguments in the command line. They have default values of None.

How to use it:

python script.py det_config.yaml det_checkpoint.pth --pose3d_estimator_config pose3d_config.yaml --pose3d_estimator_checkpoint pose3d_checkpoint.pth

If the named arguments (--pose3d_estimator_config and --pose3d_estimator_checkpoint) are omitted, they will default to None.

Let me know if this works for you!

You

Instead of processing only one video in —input, I want to process an entire batch of videos in a folder. I also want to provide info on the progress of the batch.

only provide relevant code, and not the whole script

Copyright (c) OpenMMLab. All rights reserved.

import logging import mimetypes import os import time from argparse import ArgumentParser from typing import List

import cv2 import json_tricks as json import mmcv import mmengine import numpy as np from mmengine.logging import print_log

from mmpose.apis import inference_topdown, init_model from mmpose.registry import VISUALIZERS from mmpose.structures import (PoseDataSample, merge_data_samples, split_instances) from mmpose.utils import adapt_mmdet_pipeline from mmpose.visualization import Pose3dLocalVisualizer

try: from mmdet.apis import inference_detector, init_detector has_mmdet = True except (ImportError, ModuleNotFoundError): has_mmdet = False

from rtmpose3d import * # noqa: F401, F403

def parse_args(): parser = ArgumentParser() parser.add_argument(‘—det_config’, help=‘Config file for detection’) parser.add_argument(‘—det_checkpoint’, help=‘Checkpoint file for detection’) parser.add_argument( ‘—pose3d_estimator_config’, type=str, default=None, help=‘Config file for the 3D pose estimator’) parser.add_argument( ‘—pose3d_estimator_checkpoint’, type=str, default=None, help=‘Checkpoint file for the 3D pose estimator’) parser.add_argument(‘—input’, type=str, default=”, help=‘Video path’) parser.add_argument( ‘—show’, action=‘store_true’, default=False, help=‘Whether to show visualizations’) parser.add_argument( ‘—disable-rebase-keypoint’, action=‘store_true’, default=False, help=‘Whether to disable rebasing the predicted 3D pose so its ’ ‘lowest keypoint has a height of 0 (landing on the ground). Rebase ’ ‘is useful for visualization when the model do not predict the ’ ‘global position of the 3D pose.’) parser.add_argument( ‘—disable-norm-pose-2d’, action=‘store_true’, default=False, help=‘Whether to scale the bbox (along with the 2D pose) to the ’ ‘average bbox scale of the dataset, and move the bbox (along with the ’ ‘2D pose) to the average bbox center of the dataset. This is useful ’ ‘when bbox is small, especially in multi-person scenarios.’) parser.add_argument( ‘—num-instances’, type=int, default=1, help=‘The number of 3D poses to be visualized in every frame. If ’ ‘less than 0, it will be set to the number of pose results in the ’ ‘first frame.’) parser.add_argument( ‘—output-root’, type=str, default=”, help=‘Root of the output video file. ’ ‘Default not saving the visualization video.’) parser.add_argument( ‘—save-predictions’, action=‘store_true’, default=False, help=‘Whether to save predicted results’) parser.add_argument( ‘—device’, default=‘cuda:0’, help=‘Device used for inference’) parser.add_argument( ‘—det-cat-id’, type=int, default=0, help=‘Category id for bounding box detection model’) parser.add_argument( ‘—bbox-thr’, type=float, default=0.5, help=‘Bounding box score threshold’) parser.add_argument(‘—kpt-thr’, type=float, default=0.3) parser.add_argument( ‘—use-oks-tracking’, action=‘store_true’, help=‘Using OKS tracking’) parser.add_argument( ‘—tracking-thr’, type=float, default=0.3, help=‘Tracking threshold’) parser.add_argument( ‘—show-interval’, type=int, default=0, help=‘Sleep seconds per frame’) parser.add_argument( ‘—thickness’, type=int, default=1, help=‘Link thickness for visualization’) parser.add_argument( ‘—radius’, type=int, default=3, help=‘Keypoint radius for visualization’) parser.add_argument( ‘—online’, action=‘store_true’, default=False, help=‘Inference mode. If set to True, can not use future frame’ ‘information when using multi frames for inference in the 2D pose’ ‘detection stage. Default: False.’)

args = parser.parse_args()
return args

def process_one_image(args, detector, frame: np.ndarray, frame_idx: int, pose_estimator, pose_est_results_last: List[PoseDataSample], pose_est_results_list: List[List[PoseDataSample]], next_id: int, visualize_frame: np.ndarray, visualizer: Pose3dLocalVisualizer): """Visualize detected and predicted keypoints of one image.

Pipeline of this function:

                          frame
                            |
                            V
                    +-----------------+
                    |     detector    |
                    +-----------------+
                            |  det_result
                            V
                    +-----------------+
                    |  pose_estimator |
                    +-----------------+
                            |  pose_est_results
                            V
                   +-----------------+
                   | post-processing |
                   +-----------------+
                            |  pred_3d_data_samples
                            V
                     +------------+
                     | visualizer |
                     +------------+

Args:
    args (Argument): Custom command-line arguments.
    detector (mmdet.BaseDetector): The mmdet detector.
    frame (np.ndarray): The image frame read from input image or video.
    frame_idx (int): The index of current frame.
    pose_estimator (TopdownPoseEstimator): The pose estimator for 2d pose.
    pose_est_results_last (list(PoseDataSample)): The results of pose
        estimation from the last frame for tracking instances.
    pose_est_results_list (list(list(PoseDataSample))): The list of all
        pose estimation results converted by
        ``convert_keypoint_definition`` from previous frames. In
        pose-lifting stage it is used to obtain the 2d estimation sequence.
    next_id (int): The next track id to be used.
    pose_lifter (PoseLifter): The pose-lifter for estimating 3d pose.
    visualize_frame (np.ndarray): The image for drawing the results on.
    visualizer (Visualizer): The visualizer for visualizing the 2d and 3d
        pose estimation results.

Returns:
    pose_est_results (list(PoseDataSample)): The pose estimation result of
        the current frame.
    pose_est_results_list (list(list(PoseDataSample))): The list of all
        converted pose estimation results until the current frame.
    pred_3d_instances (InstanceData): The result of pose-lifting.
        Specifically, the predicted keypoints and scores are saved at
        ``pred_3d_instances.keypoints`` and
        ``pred_3d_instances.keypoint_scores``.
    next_id (int): The next track id to be used.
"""
# pose_dataset = pose_estimator.cfg.test_dataloader.dataset
pose_det_dataset_name = pose_estimator.dataset_meta['dataset_name']

# First stage: conduct 2D pose detection in a Topdown manner
# use detector to obtain person bounding boxes
det_result = inference_detector(detector, frame)
pred_instance = det_result.pred_instances.cpu().numpy()

# filter out the person instances with category and bbox threshold
# e.g. 0 for person in COCO
bboxes = pred_instance.bboxes
bboxes = bboxes[np.logical_and(pred_instance.labels == args.det_cat_id,
                               pred_instance.scores > args.bbox_thr)]

# estimate pose results for current image
pose_est_results = inference_topdown(pose_estimator, frame, bboxes)

# post-processing
for idx, pose_est_result in enumerate(pose_est_results):
    pose_est_result.track_id = pose_est_results[idx].get('track_id', 1e4)

    pred_instances = pose_est_result.pred_instances
    keypoints = pred_instances.keypoints
    keypoint_scores = pred_instances.keypoint_scores
    if keypoint_scores.ndim == 3:
        keypoint_scores = np.squeeze(keypoint_scores, axis=1)
        pose_est_results[
            idx].pred_instances.keypoint_scores = keypoint_scores
    if keypoints.ndim == 4:
        keypoints = np.squeeze(keypoints, axis=1)

    keypoints = -keypoints[..., [0, 2, 1]]

    # rebase height (z-axis)
    if not args.disable_rebase_keypoint:
        keypoints[..., 2] -= np.min(
            keypoints[..., 2], axis=-1, keepdims=True)

    pose_est_results[idx].pred_instances.keypoints = keypoints

pose_est_results = sorted(
    pose_est_results, key=lambda x: x.get('track_id', 1e4))

pred_3d_data_samples = merge_data_samples(pose_est_results)
pred_3d_instances = pred_3d_data_samples.get('pred_instances', None)

if args.num_instances < 0:
    args.num_instances = len(pose_est_results)

# Visualization
if visualizer is not None:
    visualizer.add_datasample(
        'result',
        visualize_frame,
        data_sample=pred_3d_data_samples,
        det_data_sample=pred_3d_data_samples,
        draw_gt=False,
        draw_2d=True,
        dataset_2d=pose_det_dataset_name,
        dataset_3d=pose_det_dataset_name,
        show=args.show,
        draw_bbox=True,
        kpt_thr=args.kpt_thr,
        convert_keypoint=False,
        axis_limit=400,
        axis_azimuth=70,
        axis_elev=15,
        num_instances=args.num_instances,
        wait_time=args.show_interval)

return pose_est_results, pose_est_results_list, pred_3d_instances, next_id

def main(): assert has_mmdet, ‘Please install mmdet to run the demo.’

args = parse_args()

assert args.show or (args.output_root != '')
assert args.input != ''
assert args.det_config is not None
assert args.det_checkpoint is not None

detector = init_detector(
    args.det_config, args.det_checkpoint, device=args.device.lower())
detector.cfg = adapt_mmdet_pipeline(detector.cfg)

pose_estimator = init_model(
    args.pose3d_estimator_config,
    args.pose3d_estimator_checkpoint,
    device=args.device.lower())

det_kpt_color = pose_estimator.dataset_meta.get('keypoint_colors', None)
det_dataset_skeleton = pose_estimator.dataset_meta.get(
    'skeleton_links', None)
det_dataset_link_color = pose_estimator.dataset_meta.get(
    'skeleton_link_colors', None)

pose_estimator.cfg.model.test_cfg.mode = 'vis'
pose_estimator.cfg.visualizer.radius = args.radius
pose_estimator.cfg.visualizer.line_width = args.thickness
pose_estimator.cfg.visualizer.det_kpt_color = det_kpt_color
pose_estimator.cfg.visualizer.det_dataset_skeleton = det_dataset_skeleton
pose_estimator.cfg.visualizer.det_dataset_link_color = det_dataset_link_color  # noqa: E501
pose_estimator.cfg.visualizer.skeleton = det_dataset_skeleton
pose_estimator.cfg.visualizer.link_color = det_dataset_link_color
pose_estimator.cfg.visualizer.kpt_color = det_kpt_color
visualizer = VISUALIZERS.build(pose_estimator.cfg.visualizer)

if args.input == 'webcam':
    input_type = 'webcam'
else:
    input_type = mimetypes.guess_type(args.input)[0].split('/')[0]

if args.output_root == '':
    save_output = False
else:
    mmengine.mkdir_or_exist(args.output_root)
    output_file = os.path.join(args.output_root,
                               os.path.basename(args.input))
    if args.input == 'webcam':
        output_file += '.mp4'
    save_output = True

if args.save_predictions:
    assert args.output_root != ''
    args.pred_save_path = f'{args.output_root}/results_' \
        f'{os.path.splitext(os.path.basename(args.input))[0]}.json'

if save_output:
    fourcc = cv2.VideoWriter_fourcc(*'mp4v')

pose_est_results_list = []
pred_instances_list = []
if input_type == 'image':
    frame = mmcv.imread(args.input, channel_order='rgb')
    _, _, pred_3d_instances, _ = process_one_image(
        args=args,
        detector=detector,
        frame=args.input,
        frame_idx=0,
        pose_estimator=pose_estimator,
        pose_est_results_last=[],
        pose_est_results_list=pose_est_results_list,
        next_id=0,
        visualize_frame=frame,
        visualizer=visualizer)

    if args.save_predictions:
        # save prediction results
        pred_instances_list = split_instances(pred_3d_instances)

    if save_output:
        frame_vis = visualizer.get_image()
        mmcv.imwrite(mmcv.rgb2bgr(frame_vis), output_file)

elif input_type in ['webcam', 'video']:
    next_id = 0
    pose_est_results = []

    if args.input == 'webcam':
        video = cv2.VideoCapture(0)
    else:
        video = cv2.VideoCapture(args.input)

    (major_ver, minor_ver, subminor_ver) = (cv2.__version__).split('.')
    if int(major_ver) < 3:
        fps = video.get(cv2.cv.CV_CAP_PROP_FPS)
    else:
        fps = video.get(cv2.CAP_PROP_FPS)

    video_writer = None
    frame_idx = 0

    while video.isOpened():
        success, frame = video.read()
        frame_idx += 1

        if not success:
            break

        pose_est_results_last = pose_est_results

        # First stage: 2D pose detection
        # make person results for current image
        (pose_est_results, pose_est_results_list, pred_3d_instances,
         next_id) = process_one_image(
             args=args,
             detector=detector,
             frame=frame,
             frame_idx=frame_idx,
             pose_estimator=pose_estimator,
             pose_est_results_last=pose_est_results_last,
             pose_est_results_list=pose_est_results_list,
             next_id=next_id,
             visualize_frame=mmcv.bgr2rgb(frame),
             visualizer=visualizer)

        if args.save_predictions:
            # save prediction results
            pred_instances_list.append(
                dict(
                    frame_id=frame_idx,
                    instances=split_instances(pred_3d_instances)))

        if save_output:
            frame_vis = visualizer.get_image()
            if video_writer is None:
                # the size of the image with visualization may vary
                # depending on the presence of heatmaps
                video_writer = cv2.VideoWriter(output_file, fourcc, fps,
                                               (frame_vis.shape[1],
                                                frame_vis.shape[0]))
            video_writer.write(mmcv.rgb2bgr(frame_vis))

        if args.show:
            # press ESC to exit
            if cv2.waitKey(5) & 0xFF == 27:
                break
            time.sleep(args.show_interval)

    video.release()

    if video_writer:
        video_writer.release()
else:
    args.save_predictions = False
    raise ValueError(
        f'file {os.path.basename(args.input)} has invalid format.')

if args.save_predictions:
    with open(args.pred_save_path, 'w') as f:
        json.dump(
            dict(
                meta_info=pose_estimator.dataset_meta,
                instance_info=pred_instances_list),
            f,
            indent='\t')
    print(f'predictions have been saved at {args.pred_save_path}')

if save_output:
    input_type = input_type.replace('webcam', 'video')
    print_log(
        f'the output {input_type} has been saved at {output_file}',
        logger='current',
        level=logging.INFO)

if name == ‘main’: main()

ChatGPT

To process an entire folder of videos and track the progress of the batch, you can modify the relevant parts of your code as follows:

1. Accept a directory as input instead of a single video file.
2. Iterate over the videos in the directory.
3. Track and display the progress of processing each video.

Here’s how you can adjust the code:

1. Modify the input argument to handle a folder:

Update the --input argument to accept either a folder or a single file.

parser.add_argument('--input', type=str, default='', help='Video file or folder path')

2. Add logic to process multiple videos:

If --input is a folder, loop through all videos in that folder. You can use os.listdir to get all video files, and ensure that the input type is recognized as video.

if os.path.isdir(args.input):
    # Input is a folder, process each video file in the folder
    video_files = [
        os.path.join(args.input, f) for f in os.listdir(args.input)
        if mimetypes.guess_type(f)[0].startswith('video')
    ]
else:
    # Input is a single file
    video_files = [args.input]

3. Track and display progress:

Use the tqdm library (or manually print progress) to display the progress of each video file being processed.

First, install tqdm if it’s not installed:

pip install tqdm

Then use tqdm to wrap around the iteration over the video files:

from tqdm import tqdm

# Iterate over all video files in the folder
for video_idx, video_file in enumerate(tqdm(video_files, desc='Processing videos')):
    print(f'\nProcessing video {video_idx + 1}/{len(video_files)}: {video_file}')

    # Initialize the video capture for the current video file
    video = cv2.VideoCapture(video_file)

    if not video.isOpened():
        print(f'Failed to open video {video_file}')
        continue

    # Process the video (you can reuse the existing video processing code here)
    frame_idx = 0
    pose_est_results = []
    while video.isOpened():
        success, frame = video.read()
        if not success:
            break

        # Processing the frame (this part is already implemented in your code)
        pose_est_results_last = pose_est_results
        frame_idx += 1

        (pose_est_results, pose_est_results_list, pred_3d_instances, next_id) = process_one_image(
            args=args,
            detector=detector,
            frame=frame,
            frame_idx=frame_idx,
            pose_estimator=pose_estimator,
            pose_est_results_last=pose_est_results_last,
            pose_est_results_list=pose_est_results_list,
            next_id=next_id,
            visualize_frame=mmcv.bgr2rgb(frame),
            visualizer=visualizer
        )

        # Handle saving, visualization, etc. (reuse the logic from your code)

    # Clean up after processing the video
    video.release()

4. Full integration of changes:

Here’s the relevant snippet showing how the batch processing and progress tracking work together:

import os
import mimetypes
from tqdm import tqdm

# Check if input is a folder or a single video file
if os.path.isdir(args.input):
    # Input is a folder, get all video files
    video_files = [
        os.path.join(args.input, f) for f in os.listdir(args.input)
        if mimetypes.guess_type(f)[0] and mimetypes.guess_type(f)[0].startswith('video')
    ]
else:
    # Input is a single file
    video_files = [args.input]

# Iterate over all video files
for video_idx, video_file in enumerate(tqdm(video_files, desc='Processing videos')):
    print(f'\nProcessing video {video_idx + 1}/{len(video_files)}: {video_file}')

    # Open the video file
    video = cv2.VideoCapture(video_file)
    if not video.isOpened():
        print(f'Failed to open video {video_file}')
        continue

    # Initialize the video writer, pose estimation results, etc.
    next_id = 0
    pose_est_results = []
    frame_idx = 0
    while video.isOpened():
        success, frame = video.read()
        if not success:
            break

        pose_est_results_last = pose_est_results
        frame_idx += 1

        # Process each frame (reusing your existing frame processing logic)
        (pose_est_results, pose_est_results_list, pred_3d_instances, next_id) = process_one_image(
            args=args,
            detector=detector,
            frame=frame,
            frame_idx=frame_idx,
            pose_estimator=pose_estimator,
            pose_est_results_last=pose_est_results_last,
            pose_est_results_list=pose_est_results_list,
            next_id=next_id,
            visualize_frame=mmcv.bgr2rgb(frame),
            visualizer=visualizer
        )

        # Handle saving or displaying the output
        # (reuse your existing logic for writing video frames or showing visualizations)

    # Release video resources after processing
    video.release()

What this does:

• Handles a folder of videos: If --input is a folder, it processes each video file inside.
• Tracks progress: It shows progress with tqdm to indicate how many videos have been processed and how many are left.

You

change the main()

def main(): assert has_mmdet, ‘Please install mmdet to run the demo.’

args = parse_args()

assert args.show or (args.output_root != '')
assert args.input != ''
assert args.det_config is not None
assert args.det_checkpoint is not None

detector = init_detector(
    args.det_config, args.det_checkpoint, device=args.device.lower())
detector.cfg = adapt_mmdet_pipeline(detector.cfg)

pose_estimator = init_model(
    args.pose3d_estimator_config,
    args.pose3d_estimator_checkpoint,
    device=args.device.lower())

det_kpt_color = pose_estimator.dataset_meta.get('keypoint_colors', None)
det_dataset_skeleton = pose_estimator.dataset_meta.get(
    'skeleton_links', None)
det_dataset_link_color = pose_estimator.dataset_meta.get(
    'skeleton_link_colors', None)

pose_estimator.cfg.model.test_cfg.mode = 'vis'
pose_estimator.cfg.visualizer.radius = args.radius
pose_estimator.cfg.visualizer.line_width = args.thickness
pose_estimator.cfg.visualizer.det_kpt_color = det_kpt_color
pose_estimator.cfg.visualizer.det_dataset_skeleton = det_dataset_skeleton
pose_estimator.cfg.visualizer.det_dataset_link_color = det_dataset_link_color  # noqa: E501
pose_estimator.cfg.visualizer.skeleton = det_dataset_skeleton
pose_estimator.cfg.visualizer.link_color = det_dataset_link_color
pose_estimator.cfg.visualizer.kpt_color = det_kpt_color
visualizer = VISUALIZERS.build(pose_estimator.cfg.visualizer)

if args.input == 'webcam':
    input_type = 'webcam'
else:
    input_type = mimetypes.guess_type(args.input)[0].split('/')[0]

if args.output_root == '':
    save_output = False
else:
    mmengine.mkdir_or_exist(args.output_root)
    output_file = os.path.join(args.output_root,
                               os.path.basename(args.input))
    if args.input == 'webcam':
        output_file += '.mp4'
    save_output = True

if args.save_predictions:
    assert args.output_root != ''
    args.pred_save_path = f'{args.output_root}/results_' \
        f'{os.path.splitext(os.path.basename(args.input))[0]}.json'

if save_output:
    fourcc = cv2.VideoWriter_fourcc(*'mp4v')

pose_est_results_list = []
pred_instances_list = []
if input_type == 'image':
    frame = mmcv.imread(args.input, channel_order='rgb')
    _, _, pred_3d_instances, _ = process_one_image(
        args=args,
        detector=detector,
        frame=args.input,
        frame_idx=0,
        pose_estimator=pose_estimator,
        pose_est_results_last=[],
        pose_est_results_list=pose_est_results_list,
        next_id=0,
        visualize_frame=frame,
        visualizer=visualizer)

    if args.save_predictions:
        # save prediction results
        pred_instances_list = split_instances(pred_3d_instances)

    if save_output:
        frame_vis = visualizer.get_image()
        mmcv.imwrite(mmcv.rgb2bgr(frame_vis), output_file)

elif input_type in ['webcam', 'video']:
    next_id = 0
    pose_est_results = []

    if args.input == 'webcam':
        video = cv2.VideoCapture(0)
    else:
        video = cv2.VideoCapture(args.input)

    (major_ver, minor_ver, subminor_ver) = (cv2.__version__).split('.')
    if int(major_ver) < 3:
        fps = video.get(cv2.cv.CV_CAP_PROP_FPS)
    else:
        fps = video.get(cv2.CAP_PROP_FPS)

    video_writer = None
    frame_idx = 0

    while video.isOpened():
        success, frame = video.read()
        frame_idx += 1

        if not success:
            break

        pose_est_results_last = pose_est_results

        # First stage: 2D pose detection
        # make person results for current image
        (pose_est_results, pose_est_results_list, pred_3d_instances,
         next_id) = process_one_image(
             args=args,
             detector=detector,
             frame=frame,
             frame_idx=frame_idx,
             pose_estimator=pose_estimator,
             pose_est_results_last=pose_est_results_last,
             pose_est_results_list=pose_est_results_list,
             next_id=next_id,
             visualize_frame=mmcv.bgr2rgb(frame),
             visualizer=visualizer)

        if args.save_predictions:
            # save prediction results
            pred_instances_list.append(
                dict(
                    frame_id=frame_idx,
                    instances=split_instances(pred_3d_instances)))

        if save_output:
            frame_vis = visualizer.get_image()
            if video_writer is None:
                # the size of the image with visualization may vary
                # depending on the presence of heatmaps
                video_writer = cv2.VideoWriter(output_file, fourcc, fps,
                                               (frame_vis.shape[1],
                                                frame_vis.shape[0]))
            video_writer.write(mmcv.rgb2bgr(frame_vis))

        if args.show:
            # press ESC to exit
            if cv2.waitKey(5) & 0xFF == 27:
                break
            time.sleep(args.show_interval)

    video.release()

    if video_writer:
        video_writer.release()
else:
    args.save_predictions = False
    raise ValueError(
        f'file {os.path.basename(args.input)} has invalid format.')

if args.save_predictions:
    with open(args.pred_save_path, 'w') as f:
        json.dump(
            dict(
                meta_info=pose_estimator.dataset_meta,
                instance_info=pred_instances_list),
            f,
            indent='\t')
    print(f'predictions have been saved at {args.pred_save_path}')

if save_output:
    input_type = input_type.replace('webcam', 'video')
    print_log(
        f'the output {input_type} has been saved at {output_file}',
        logger='current',
        level=logging.INFO)

ChatGPT

Here’s a revised version of your main() function to support processing an entire folder of videos and provide progress tracking for the batch:

Updated main() function:

def main():
    assert has_mmdet, 'Please install mmdet to run the demo.'

    args = parse_args()

    assert args.show or (args.output_root != '')
    assert args.input != ''
    assert args.det_config is not None
    assert args.det_checkpoint is not None

    detector = init_detector(
        args.det_config, args.det_checkpoint, device=args.device.lower())
    detector.cfg = adapt_mmdet_pipeline(detector.cfg)

    pose_estimator = init_model(
        args.pose3d_estimator_config,
        args.pose3d_estimator_checkpoint,
        device=args.device.lower())

    det_kpt_color = pose_estimator.dataset_meta.get('keypoint_colors', None)
    det_dataset_skeleton = pose_estimator.dataset_meta.get(
        'skeleton_links', None)
    det_dataset_link_color = pose_estimator.dataset_meta.get(
        'skeleton_link_colors', None)

    pose_estimator.cfg.model.test_cfg.mode = 'vis'
    pose_estimator.cfg.visualizer.radius = args.radius
    pose_estimator.cfg.visualizer.line_width = args.thickness
    pose_estimator.cfg.visualizer.det_kpt_color = det_kpt_color
    pose_estimator.cfg.visualizer.det_dataset_skeleton = det_dataset_skeleton
    pose_estimator.cfg.visualizer.det_dataset_link_color = det_dataset_link_color
    pose_estimator.cfg.visualizer.skeleton = det_dataset_skeleton
    pose_estimator.cfg.visualizer.link_color = det_dataset_link_color
    pose_estimator.cfg.visualizer.kpt_color = det_kpt_color
    visualizer = VISUALIZERS.build(pose_estimator.cfg.visualizer)

    # Check if input is a folder or a single file
    if os.path.isdir(args.input):
        video_files = [
            os.path.join(args.input, f) for f in os.listdir(args.input)
            if mimetypes.guess_type(f)[0] and mimetypes.guess_type(f)[0].startswith('video')
        ]
    else:
        video_files = [args.input]

    for video_idx, video_file in enumerate(tqdm(video_files, desc='Processing videos')):
        print(f'\nProcessing video {video_idx + 1}/{len(video_files)}: {video_file}')

        input_type = mimetypes.guess_type(video_file)[0].split('/')[0]

        if args.output_root == '':
            save_output = False
        else:
            mmengine.mkdir_or_exist(args.output_root)
            output_file = os.path.join(args.output_root,
                                       os.path.basename(video_file))
            save_output = True

        if args.save_predictions:
            args.pred_save_path = f'{args.output_root}/results_' \
                f'{os.path.splitext(os.path.basename(video_file))[0]}.json'

        if save_output:
            fourcc = cv2.VideoWriter_fourcc(*'mp4v')

        pose_est_results_list = []
        pred_instances_list = []

        if input_type == 'image':
            frame = mmcv.imread(video_file, channel_order='rgb')
            _, _, pred_3d_instances, _ = process_one_image(
                args=args,
                detector=detector,
                frame=frame,
                frame_idx=0,
                pose_estimator=pose_estimator,
                pose_est_results_last=[],
                pose_est_results_list=pose_est_results_list,
                next_id=0,
                visualize_frame=frame,
                visualizer=visualizer)

            if args.save_predictions:
                pred_instances_list = split_instances(pred_3d_instances)

            if save_output:
                frame_vis = visualizer.get_image()
                mmcv.imwrite(mmcv.rgb2bgr(frame_vis), output_file)

        elif input_type in ['webcam', 'video']:
            next_id = 0
            pose_est_results = []

            video = cv2.VideoCapture(video_file)

            (major_ver, minor_ver, subminor_ver) = (cv2.__version__).split('.')
            if int(major_ver) < 3:
                fps = video.get(cv2.cv.CV_CAP_PROP_FPS)
            else:
                fps = video.get(cv2.CAP_PROP_FPS)

            video_writer = None
            frame_idx = 0

            while video.isOpened():
                success, frame = video.read()
                frame_idx += 1

                if not success:
                    break

                pose_est_results_last = pose_est_results

                (pose_est_results, pose_est_results_list, pred_3d_instances,
                 next_id) = process_one_image(
                    args=args,
                    detector=detector,
                    frame=frame,
                    frame_idx=frame_idx,
                    pose_estimator=pose_estimator,
                    pose_est_results_last=pose_est_results_last,
                    pose_est_results_list=pose_est_results_list,
                    next_id=next_id,
                    visualize_frame=mmcv.bgr2rgb(frame),
                    visualizer=visualizer)

                if args.save_predictions:
                    pred_instances_list.append(
                        dict(
                            frame_id=frame_idx,
                            instances=split_instances(pred_3d_instances)))

                if save_output:
                    frame_vis = visualizer.get_image()
                    if video_writer is None:
                        video_writer = cv2.VideoWriter(output_file, fourcc, fps,
                                                       (frame_vis.shape[1],
                                                        frame_vis.shape[0]))
                    video_writer.write(mmcv.rgb2bgr(frame_vis))

                if args.show:
                    if cv2.waitKey(5) & 0xFF == 27:
                        break
                    time.sleep(args.show_interval)

            video.release()

            if video_writer:
                video_writer.release()

        else:
            args.save_predictions = False
            raise ValueError(f'file {os.path.basename(video_file)} has invalid format.')

        if args.save_predictions:
            with open(args.pred_save_path, 'w') as f:
                json.dump(
                    dict(
                        meta_info=pose_estimator.dataset_meta,
                        instance_info=pred_instances_list),
                    f,
                    indent='\t')
            print(f'predictions have been saved at {args.pred_save_path}')

        if save_output:
            print_log(
                f'the output {input_type} has been saved at {output_file}',
                logger='current',
                level=logging.INFO)

Key Changes:

1. Batch processing of videos in a folder:

   • If args.input is a folder, all video files in the folder are processed.
   • The code now loops over each video and processes them individually.

2. Progress tracking with tqdm:

   • The progress of video processing is shown using the tqdm progress bar.

3. Each video has its own output file:

   • The output video and prediction results are saved with unique filenames based on each input video file.

This update allows you to process an entire batch of videos, track the progress of each, and save the results for each video.

You

I get a json of prediction data and I want to display in Unity using debug.drawLine and Gizmos.

the json contains { “meta_info”: { “dataset_name”: “h3wb”, “num_keypoints”: 133, “keypoint_id2name”: { “0”: “nose”, “1”: “left_eye”, “2”: “right_eye”, “3”: “left_ear”, “4”: “right_ear”, “5”: “left_shoulder”, “6”: “right_shoulder”, “7”: “left_elbow”, “8”: “right_elbow”, “9”: “left_wrist”, “10”: “right_wrist”, “11”: “left_hip”, “12”: “right_hip”, “13”: “left_knee”, “14”: “right_knee”, “15”: “left_ankle”, “16”: “right_ankle”, “17”: “left_big_toe”, “18”: “left_small_toe”, “19”: “left_heel”, “20”: “right_big_toe”, “21”: “right_small_toe”, “22”: “right_heel”, “23”: “face-0”, “24”: “face-1”, “25”: “face-2”, “26”: “face-3”, “27”: “face-4”, “28”: “face-5”, “29”: “face-6”, “30”: “face-7”, “31”: “face-8”, “32”: “face-9”, “33”: “face-10”, “34”: “face-11”, “35”: “face-12”, “36”: “face-13”, “37”: “face-14”, “38”: “face-15”, “39”: “face-16”, “40”: “face-17”, “41”: “face-18”, “42”: “face-19”, “43”: “face-20”, “44”: “face-21”, “45”: “face-22”, “46”: “face-23”, “47”: “face-24”, “48”: “face-25”, “49”: “face-26”, “50”: “face-27”, “51”: “face-28”, “52”: “face-29”, “53”: “face-30”, “54”: “face-31”, “55”: “face-32”, “56”: “face-33”, “57”: “face-34”, “58”: “face-35”, “59”: “face-36”, “60”: “face-37”, “61”: “face-38”, “62”: “face-39”, “63”: “face-40”, “64”: “face-41”, “65”: “face-42”, “66”: “face-43”, “67”: “face-44”, “68”: “face-45”, “69”: “face-46”, “70”: “face-47”, “71”: “face-48”, “72”: “face-49”, “73”: “face-50”, “74”: “face-51”, “75”: “face-52”, “76”: “face-53”, “77”: “face-54”, “78”: “face-55”, “79”: “face-56”, “80”: “face-57”, “81”: “face-58”, “82”: “face-59”, “83”: “face-60”, “84”: “face-61”, “85”: “face-62”, “86”: “face-63”, “87”: “face-64”, “88”: “face-65”, “89”: “face-66”, “90”: “face-67”, “91”: “left_hand_root”, “92”: “left_thumb1”, “93”: “left_thumb2”, “94”: “left_thumb3”, “95”: “left_thumb4”, “96”: “left_forefinger1”, “97”: “left_forefinger2”, “98”: “left_forefinger3”, “99”: “left_forefinger4”, “100”: “left_middle_finger1”, “101”: “left_middle_finger2”, “102”: “left_middle_finger3”, “103”: “left_middle_finger4”, “104”: “left_ring_finger1”, “105”: “left_ring_finger2”, “106”: “left_ring_finger3”, “107”: “left_ring_finger4”, “108”: “left_pinky_finger1”, “109”: “left_pinky_finger2”, “110”: “left_pinky_finger3”, “111”: “left_pinky_finger4”, “112”: “right_hand_root”, “113”: “right_thumb1”, “114”: “right_thumb2”, “115”: “right_thumb3”, “116”: “right_thumb4”, “117”: “right_forefinger1”, “118”: “right_forefinger2”, “119”: “right_forefinger3”, “120”: “right_forefinger4”, “121”: “right_middle_finger1”, “122”: “right_middle_finger2”, “123”: “right_middle_finger3”, “124”: “right_middle_finger4”, “125”: “right_ring_finger1”, “126”: “right_ring_finger2”, “127”: “right_ring_finger3”, “128”: “right_ring_finger4”, “129”: “right_pinky_finger1”, “130”: “right_pinky_finger2”, “131”: “right_pinky_finger3”, “132”: “right_pinky_finger4” }, “keypoint_name2id”: { “nose”: 0, “left_eye”: 1, “right_eye”: 2, “left_ear”: 3, “right_ear”: 4, “left_shoulder”: 5, “right_shoulder”: 6, “left_elbow”: 7, “right_elbow”: 8, “left_wrist”: 9, “right_wrist”: 10, “left_hip”: 11, “right_hip”: 12, “left_knee”: 13, “right_knee”: 14, “left_ankle”: 15, “right_ankle”: 16, “left_big_toe”: 17, “left_small_toe”: 18, “left_heel”: 19, “right_big_toe”: 20, “right_small_toe”: 21, “right_heel”: 22, “face-0”: 23, “face-1”: 24, “face-2”: 25, “face-3”: 26, “face-4”: 27, “face-5”: 28, “face-6”: 29, “face-7”: 30, “face-8”: 31, “face-9”: 32, “face-10”: 33, “face-11”: 34, “face-12”: 35, “face-13”: 36, “face-14”: 37, “face-15”: 38, “face-16”: 39, “face-17”: 40, “face-18”: 41, “face-19”: 42, “face-20”: 43, “face-21”: 44, “face-22”: 45, “face-23”: 46, “face-24”: 47, “face-25”: 48, “face-26”: 49, “face-27”: 50, “face-28”: 51, “face-29”: 52, “face-30”: 53, “face-31”: 54, “face-32”: 55, “face-33”: 56, “face-34”: 57, “face-35”: 58, “face-36”: 59, “face-37”: 60, “face-38”: 61, “face-39”: 62, “face-40”: 63, “face-41”: 64, “face-42”: 65, “face-43”: 66, “face-44”: 67, “face-45”: 68, “face-46”: 69, “face-47”: 70, “face-48”: 71, “face-49”: 72, “face-50”: 73, “face-51”: 74, “face-52”: 75, “face-53”: 76, “face-54”: 77, “face-55”: 78, “face-56”: 79, “face-57”: 80, “face-58”: 81, “face-59”: 82, “face-60”: 83, “face-61”: 84, “face-62”: 85, “face-63”: 86, “face-64”: 87, “face-65”: 88, “face-66”: 89, “face-67”: 90, “left_hand_root”: 91, “left_thumb1”: 92, “left_thumb2”: 93, “left_thumb3”: 94, “left_thumb4”: 95, “left_forefinger1”: 96, “left_forefinger2”: 97, “left_forefinger3”: 98, “left_forefinger4”: 99, “left_middle_finger1”: 100, “left_middle_finger2”: 101, “left_middle_finger3”: 102, “left_middle_finger4”: 103, “left_ring_finger1”: 104, “left_ring_finger2”: 105, “left_ring_finger3”: 106, “left_ring_finger4”: 107, “left_pinky_finger1”: 108, “left_pinky_finger2”: 109, “left_pinky_finger3”: 110, “left_pinky_finger4”: 111, “right_hand_root”: 112, “right_thumb1”: 113, “right_thumb2”: 114, “right_thumb3”: 115, “right_thumb4”: 116, “right_forefinger1”: 117, “right_forefinger2”: 118, “right_forefinger3”: 119, “right_forefinger4”: 120, “right_middle_finger1”: 121, “right_middle_finger2”: 122, “right_middle_finger3”: 123, “right_middle_finger4”: 124, “right_ring_finger1”: 125, “right_ring_finger2”: 126, “right_ring_finger3”: 127, “right_ring_finger4”: 128, “right_pinky_finger1”: 129, “right_pinky_finger2”: 130, “right_pinky_finger3”: 131, “right_pinky_finger4”: 132 }, “upper_body_ids”: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90 ], “lower_body_ids”: [ 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22 ], “flip_indices”: [ 0, 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11, 14, 13, 16, 15, 20, 21, 22, 17, 18, 19, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 50, 51, 52, 53, 58, 57, 56, 55, 54, 68, 67, 66, 65, 70, 69, 62, 61, 60, 59, 64, 63, 77, 76, 75, 74, 73, 72, 71, 82, 81, 80, 79, 78, 87, 86, 85, 84, 83, 90, 89, 88, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111 ], “flip_pairs”: [ [ 2, 1 ], [ 1, 2 ], [ 4, 3 ], [ 3, 4 ], [ 6, 5 ], [ 5, 6 ], [ 8, 7 ], [ 7, 8 ], [ 10, 9 ], [ 9, 10 ], [ 12, 11 ], [ 11, 12 ], [ 14, 13 ], [ 13, 14 ], [ 16, 15 ], [ 15, 16 ], [ 20, 17 ], [ 21, 18 ], [ 22, 19 ], [ 17, 20 ], [ 18, 21 ], [ 19, 22 ], [ 39, 23 ], [ 38, 24 ], [ 37, 25 ], [ 36, 26 ], [ 35, 27 ], [ 34, 28 ], [ 33, 29 ], [ 32, 30 ], [ 30, 32 ], [ 29, 33 ], [ 28, 34 ], [ 27, 35 ], [ 26, 36 ], [ 25, 37 ], [ 24, 38 ], [ 23, 39 ], [ 49, 40 ], [ 48, 41 ], [ 47, 42 ], [ 46, 43 ], [ 45, 44 ], [ 44, 45 ], [ 43, 46 ], [ 42, 47 ], [ 41, 48 ], [ 40, 49 ], [ 58, 54 ], [ 57, 55 ], [ 55, 57 ], [ 54, 58 ], [ 68, 59 ], [ 67, 60 ], [ 66, 61 ], [ 65, 62 ], [ 70, 63 ], [ 69, 64 ], [ 62, 65 ], [ 61, 66 ], [ 60, 67 ], [ 59, 68 ], [ 64, 69 ], [ 63, 70 ], [ 77, 71 ], [ 76, 72 ], [ 75, 73 ], [ 73, 75 ], [ 72, 76 ], [ 71, 77 ], [ 82, 78 ], [ 81, 79 ], [ 79, 81 ], [ 78, 82 ], [ 87, 83 ], [ 86, 84 ], [ 84, 86 ], [ 83, 87 ], [ 90, 88 ], [ 88, 90 ], [ 112, 91 ], [ 113, 92 ], [ 114, 93 ], [ 115, 94 ], [ 116, 95 ], [ 117, 96 ], [ 118, 97 ], [ 119, 98 ], [ 120, 99 ], [ 121, 100 ], [ 122, 101 ], [ 123, 102 ], [ 124, 103 ], [ 125, 104 ], [ 126, 105 ], [ 127, 106 ], [ 128, 107 ], [ 129, 108 ], [ 130, 109 ], [ 131, 110 ], [ 132, 111 ], [ 91, 112 ], [ 92, 113 ], [ 93, 114 ], [ 94, 115 ], [ 95, 116 ], [ 96, 117 ], [ 97, 118 ], [ 98, 119 ], [ 99, 120 ], [ 100, 121 ], [ 101, 122 ], [ 102, 123 ], [ 103, 124 ], [ 104, 125 ], [ 105, 126 ], [ 106, 127 ], [ 107, 128 ], [ 108, 129 ], [ 109, 130 ], [ 110, 131 ], [ 111, 132 ] ], “keypoint_colors”: { “ndarray”: [ [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 247, 34, 5 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 247, 34, 5 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ] ], “dtype”: “uint8”, “shape”: [ 133, 3 ], “Corder”: true }, “num_skeleton_links”: 65, “skeleton_links”: [ [ 15, 13 ], [ 13, 11 ], [ 16, 14 ], [ 14, 12 ], [ 11, 12 ], [ 5, 11 ], [ 6, 12 ], [ 5, 6 ], [ 5, 7 ], [ 6, 8 ], [ 7, 9 ], [ 8, 10 ], [ 1, 2 ], [ 0, 1 ], [ 0, 2 ], [ 1, 3 ], [ 2, 4 ], [ 3, 5 ], [ 4, 6 ], [ 15, 17 ], [ 15, 18 ], [ 15, 19 ], [ 16, 20 ], [ 16, 21 ], [ 16, 22 ], [ 91, 92 ], [ 92, 93 ], [ 93, 94 ], [ 94, 95 ], [ 91, 96 ], [ 96, 97 ], [ 97, 98 ], [ 98, 99 ], [ 91, 100 ], [ 100, 101 ], [ 101, 102 ], [ 102, 103 ], [ 91, 104 ], [ 104, 105 ], [ 105, 106 ], [ 106, 107 ], [ 91, 108 ], [ 108, 109 ], [ 109, 110 ], [ 110, 111 ], [ 112, 113 ], [ 113, 114 ], [ 114, 115 ], [ 115, 116 ], [ 112, 117 ], [ 117, 118 ], [ 118, 119 ], [ 119, 120 ], [ 112, 121 ], [ 121, 122 ], [ 122, 123 ], [ 123, 124 ], [ 112, 125 ], [ 125, 126 ], [ 126, 127 ], [ 127, 128 ], [ 112, 129 ], [ 129, 130 ], [ 130, 131 ], [ 131, 132 ] ], “skeleton_link_colors”: { “ndarray”: [ [ 0, 255, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 51, 153, 255 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 128, 0 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 255, 153, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 102, 178, 255 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 255, 51, 51 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ], [ 0, 255, 0 ] ], “dtype”: “uint8”, “shape”: [ 65, 3 ], “Corder”: true }, “dataset_keypoint_weights”: { “ndarray”: [ 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 ], “dtype”: “float32”, “shape”: [ 133 ] }, “sigmas”: { “ndarray”: [ 0.026000000536441803, 0.02500000037252903, 0.02500000037252903, 0.03500000014901161, 0.03500000014901161, 0.07900000363588333, 0.07900000363588333, 0.07199999690055847, 0.07199999690055847, 0.06199999898672104, 0.06199999898672104, 0.10700000077486038, 0.10700000077486038, 0.08699999749660492, 0.08699999749660492, 0.08900000154972076, 0.08900000154972076, 0.06800000369548798, 0.06599999964237213, 0.06599999964237213, 0.09200000017881393, 0.09399999678134918, 0.09399999678134918, 0.041999999433755875, 0.0430000014603138, 0.04399999976158142, 0.0430000014603138, 0.03999999910593033, 0.03500000014901161, 0.03099999949336052, 0.02500000037252903, 0.019999999552965164, 0.023000000044703484, 0.028999999165534973, 0.03200000151991844, 0.03700000047683716, 0.03799999877810478, 0.0430000014603138, 0.04100000113248825, 0.04500000178813934, 0.013000000268220901, 0.012000000104308128, 0.010999999940395355, 0.010999999940395355, 0.012000000104308128, 0.012000000104308128, 0.010999999940395355, 0.010999999940395355, 0.013000000268220901, 0.014999999664723873, 0.008999999612569809, 0.007000000216066837, 0.007000000216066837, 0.007000000216066837, 0.012000000104308128, 0.008999999612569809, 0.00800000037997961, 0.01600000075995922, 0.009999999776482582, 0.017000000923871994, 0.010999999940395355, 0.008999999612569809, 0.010999999940395355, 0.008999999612569809, 0.007000000216066837, 0.013000000268220901, 0.00800000037997961, 0.010999999940395355, 0.012000000104308128, 0.009999999776482582, 0.03400000184774399, 0.00800000037997961, 0.00800000037997961, 0.008999999612569809, 0.00800000037997961, 0.00800000037997961, 0.007000000216066837, 0.009999999776482582, 0.00800000037997961, 0.008999999612569809, 0.008999999612569809, 0.008999999612569809, 0.007000000216066837, 0.007000000216066837, 0.00800000037997961, 0.010999999940395355, 0.00800000037997961, 0.00800000037997961, 0.00800000037997961, 0.009999999776482582, 0.00800000037997961, 0.028999999165534973, 0.02199999988079071, 0.03500000014901161, 0.03700000047683716, 0.04699999839067459, 0.026000000536441803, 0.02500000037252903, 0.024000000208616257, 0.03500000014901161, 0.017999999225139618, 0.024000000208616257, 0.02199999988079071, 0.026000000536441803, 0.017000000923871994, 0.020999999716877937, 0.020999999716877937, 0.03200000151991844, 0.019999999552965164, 0.01899999938905239, 0.02199999988079071, 0.03099999949336052, 0.028999999165534973, 0.02199999988079071, 0.03500000014901161, 0.03700000047683716, 0.04699999839067459, 0.026000000536441803, 0.02500000037252903, 0.024000000208616257, 0.03500000014901161, 0.017999999225139618, 0.024000000208616257, 0.02199999988079071, 0.026000000536441803, 0.017000000923871994, 0.020999999716877937, 0.020999999716877937, 0.03200000151991844, 0.019999999552965164, 0.01899999938905239, 0.02199999988079071, 0.03099999949336052 ], “dtype”: “float32”, “shape”: [ 133 ] } }, “instance_info”: [ { “frame_id”: 1, “instances”: [ { “keypoints”: [ [ -177.5, -146.0, 259.0 ], [ -180.0, -152.5, 187.5 ], [ -183.0, -146.0, 255.5 ], [ -149.0, -144.5, 285.5 ], [ -178.0, -149.0, 272.5 ], [ -91.5, -138.0, 281.0 ], [ -172.5, -144.5, 265.5 ], [ -60.5, -134.5, 199.5 ], [ -179.5, -145.0, 177.5 ], [ -103.5, -137.0, 133.5 ], [ -202.5, -154.5, 108.0 ], [ -84.0, -143.5, 129.0 ], [ -141.5, -144.5, 130.5 ], [ -145.5, -153.0, 180.5 ], [ -148.5, -136.0, 44.5 ], [ -72.0, -136.0, 117.5 ], [ -141.0, -138.0, 95.5 ], [ -175.0, -131.5, 97.0 ], [ -49.5, -119.0, 100.5 ], [ -58.0, -145.5, 109.0 ], [ -148.5, -140.5, 102.0 ], [ -160.0, -145.5, 115.5 ], [ -135.5, -144.0, 108.0 ], [ -168.0, -147.5, 259.0 ], [ -164.5, -148.0, 269.0 ], [ -159.0, -147.5, 258.5 ], [ -150.5, -147.5, 255.0 ], [ -142.0, -147.0, 254.0 ], [ -141.0, -147.0, 0.0 ], [ -141.5, -146.5, 0.0 ], [ -142.0, -147.0, 383.5 ], [ -142.5, -146.5, 383.5 ], [ -158.0, -146.0, 383.5 ], [ -0.0, -147.0, 383.5 ], [ -0.0, -148.0, 383.5 ], [ -0.0, -148.0, 383.5 ], [ -170.0, -149.5, 383.5 ], [ -172.0, -150.0, 383.5 ], [ -287.5, -149.0, 383.5 ], [ -287.5, -161.0, 383.5 ], [ -173.5, -148.0, 255.0 ], [ -173.0, -148.5, 255.5 ], [ -172.0, -148.5, 255.0 ], [ -168.5, -148.5, 255.0 ], [ -170.0, -151.0, 257.5 ], [ -169.5, -153.0, 256.5 ], [ -169.5, -153.0, 256.5 ], [ -172.0, -150.5, 255.5 ], [ -180.5, -149.5, 255.5 ], [ -180.5, -149.0, 383.5 ], [ -167.5, -156.5, 257.0 ], [ -168.0, -156.0, 255.5 ], [ -168.0, -154.0, 253.0 ], [ -168.5, -153.5, 254.5 ], [ -172.5, -152.5, 253.5 ], [ -173.0, -153.5, 254.5 ], [ -168.5, -153.5, 253.0 ], [ -165.5, -152.5, 253.5 ], [ -166.5, -151.5, 255.5 ], [ -179.0, -148.0, 253.0 ], [ -173.0, -152.5, 253.5 ], [ -170.0, -147.5, 256.0 ], [ -167.0, -146.5, 258.5 ], [ -167.0, -146.5, 258.0 ], [ -172.0, -149.0, 256.0 ], [ -167.0, -150.5, 256.5 ], [ -163.5, -148.5, 256.0 ], [ -167.5, -149.0, 383.5 ], [ -287.5, -148.0, 383.5 ], [ -167.5, -153.0, 383.5 ], [ -167.0, -150.5, 383.5 ], [ -159.0, -147.0, 255.0 ], [ -171.5, -146.5, 255.0 ], [ -172.0, -146.5, 255.5 ], [ -168.0, -147.0, 245.5 ], [ -167.0, -146.5, 244.5 ], [ -166.0, -147.0, 220.5 ], [ -158.5, -147.5, 383.5 ], [ -158.5, -147.0, 383.5 ], [ -0.0, -147.0, 383.5 ], [ -173.5, -147.5, 383.5 ], [ -168.0, -147.0, 244.0 ], [ -158.5, -146.5, 254.5 ], [ -159.0, -147.0, 198.5 ], [ -164.5, -146.5, 253.5 ], [ -170.0, -146.5, 255.0 ], [ -167.0, -147.0, 256.5 ], [ -166.5, -147.5, 383.5 ], [ -167.0, -147.5, 245.5 ], [ -171.5, -147.0, 244.5 ], [ -163.0, -146.5, 245.5 ], [ -56.5, -137.0, 114.0 ], [ -209.5, -153.5, 116.0 ], [ -215.0, -153.5, 102.5 ], [ -227.5, -153.5, 98.0 ], [ -232.5, -152.5, 98.5 ], [ -225.0, -154.0, 105.0 ], [ -235.5, -137.0, 103.0 ], [ -245.0, -154.0, 98.5 ], [ -253.0, -164.0, 97.5 ], [ -225.5, -155.5, 105.0 ], [ -237.5, -154.5, 102.0 ], [ -250.0, -154.0, 96.5 ], [ -258.0, -134.5, 95.0 ], [ -226.5, -155.5, 105.0 ], [ -236.0, -155.0, 100.5 ], [ -248.5, -163.0, 94.5 ], [ -255.0, -134.0, 93.0 ], [ -224.5, -154.5, 102.5 ], [ -231.0, -154.5, 99.5 ], [ -241.0, -154.0, 96.5 ], [ -247.5, -163.0, 89.0 ], [ -205.0, -165.0, 103.5 ], [ -214.0, -162.5, 103.5 ], [ -221.0, -163.5, 102.5 ], [ -229.0, -163.5, 101.5 ], [ -235.5, -164.0, 99.0 ], [ -230.5, -163.5, 107.5 ], [ -241.5, -163.5, 105.5 ], [ -249.0, -164.0, 102.0 ], [ -255.5, -164.0, 99.0 ], [ -230.0, -162.0, 105.0 ], [ -241.5, -162.5, 101.0 ], [ -250.5, -163.0, 96.5 ], [ -258.0, -163.0, 92.5 ], [ -227.5, -162.0, 101.0 ], [ -238.0, -162.5, 97.0 ], [ -246.0, -162.5, 92.5 ], [ -253.0, -162.5, 89.0 ], [ -223.5, -166.0, 95.5 ], [ -232.0, -162.5, 92.0 ], [ -238.0, -162.0, 89.5 ], [ -243.5, -162.5, 87.0 ] ], “keypoint_scores”: [ 0.40531837940216064, 0.18405357003211975, 0.48211470246315, 0.35187196731567383, 0.5293264389038086, 0.5780988931655884, 0.5900246500968933, 0.5260855555534363, 0.6153478026390076, 0.23395457863807678, 0.7824482917785645, 0.4690173268318176, 0.4369870722293854, 0.19321686029434204, 0.3326764702796936, 0.16863475739955902, 0.2548399865627289, 0.13458845019340515, 0.24586138129234314, 0.1940145045518875, 0.3006875514984131, 0.2972128391265869, 0.2561021149158478, 0.25699782371520996, 0.2917252779006958, 0.2868095934391022, 0.20242246985435486, 0.12991748750209808, 0.07530181109905243, 0.08372682332992554, 0.07973313331604004, 0.08082561939954758, 0.08322370052337646, 0.08724988251924515, 0.09648165106773376, 0.0917154848575592, 0.08026520162820816, 0.10444790124893188, 0.07916475832462311, 0.10067783296108246, 0.34404414892196655, 0.3354242444038391, 0.3061371445655823, 0.31848448514938354, 0.2686695456504822, 0.24636971950531006, 0.1797993779182434, 0.16639792919158936, 0.13428223133087158, 0.08970092982053757, 0.2647181451320648, 0.2930288314819336, 0.2564682960510254, 0.22240296006202698, 0.2082076072692871, 0.1408882886171341, 0.2067408710718155, 0.1718032956123352, 0.14241179823875427, 0.24420103430747986, 0.2488202154636383, 0.2917303740978241, 0.32381951808929443, 0.2617563009262085, 0.21968156099319458, 0.18884295225143433, 0.12793710827827454, 0.10158038884401321, 0.07726319134235382, 0.08719799667596817, 0.12367592751979828, 0.12063349038362503, 0.09244374930858612, 0.09473835676908493, 0.1265525072813034, 0.09923699498176575, 0.10613573342561722, 0.09252621978521347, 0.08461447060108185, 0.07353158295154572, 0.09780298173427582, 0.09970812499523163, 0.1306171417236328, 0.08838710188865662, 0.16716501116752625, 0.10561677068471909, 0.13161274790763855, 0.08248808234930038, 0.10283485800027847, 0.10413061082363129, 0.16424304246902466, 0.23419132828712463, 0.19241678714752197, 0.26241201162338257, 0.3421594202518463, 0.40774664282798767, 0.3441167175769806, 0.4232301115989685, 0.48388534784317017, 0.5152691602706909, 0.3605597913265228, 0.3908516764640808, 0.5086203217506409, 0.5281122326850891, 0.3994280695915222, 0.40851712226867676, 0.464418888092041, 0.4809495210647583, 0.42895451188087463, 0.38435274362564087, 0.3607679307460785, 0.3870113492012024, 0.7765405178070068, 0.7765754461288452, 0.7446026802062988, 0.7401175498962402, 0.7086427211761475, 0.8852808475494385, 0.901991605758667, 0.8843626976013184, 0.8852169513702393, 0.8562625646591187, 0.916162371635437, 0.9145952463150024, 0.9311339855194092, 0.8531612753868103, 0.9007415771484375, 0.9027521014213562, 0.9489120841026306, 0.8101211786270142, 0.8358651399612427, 0.8502238392829895, 0.8781735897064209 ], “bbox”: [ [ 1872.845703125, 1453.1700439453125, 2177.105712890625, 1739.99072265625 ] ], “bbox_score”: 1.0 } ] },

ChatGPT

To display the skeleton represented in the JSON data using Debug.DrawLine and Gizmos in Unity, you can extract the keypoint positions and link them together based on the skeleton links defined in the JSON file. Here’s an approach to achieve this:

Steps:

1. Parse the JSON Data: Read the JSON data to extract keypoints and skeleton links.
2. Convert Keypoints to Unity Positions: Unity uses a different coordinate system (Y-up) compared to most 3D applications (Z-up). Adjust the coordinates if necessary.
3. Draw the Skeleton: Use Debug.DrawLine or Gizmos.DrawLine to draw lines between connected keypoints.

Example Code:

This code assumes you have a method to load the JSON data, and then you can use it in a MonoBehaviour to draw the skeleton:

using System.Collections.Generic;
using UnityEngine;

public class SkeletonVisualizer : MonoBehaviour
{
    [System.Serializable]
    public class Keypoint
    {
        public float x;
        public float y;
        public float z;
    }

    [System.Serializable]
    public class SkeletonLink
    {
        public int start;
        public int end;
    }

    [System.Serializable]
    public class Instance
    {
        public List<Keypoint> keypoints;
    }

    [System.Serializable]
    public class FrameData
    {
        public List<Instance> instances;
    }

    [System.Serializable]
    public class JsonData
    {
        public List<SkeletonLink> skeleton_links;
        public List<FrameData> instance_info;
    }

    public TextAsset jsonDataFile;  // JSON data file
    private JsonData jsonData;

    // You can set these dynamically in the inspector for color and line thickness
    public Color skeletonColor = Color.red;
    public float lineWidth = 0.02f;

    void Start()
    {
        LoadJsonData();
    }

    void LoadJsonData()
    {
        // Parse JSON data from the TextAsset
        jsonData = JsonUtility.FromJson<JsonData>(jsonDataFile.text);
    }

    void Update()
    {
        if (jsonData != null)
        {
            foreach (var frame in jsonData.instance_info)
            {
                foreach (var instance in frame.instances)
                {
                    DrawSkeleton(instance.keypoints);
                }
            }
        }
    }

    void DrawSkeleton(List<Keypoint> keypoints)
    {
        foreach (var link in jsonData.skeleton_links)
        {
            if (link.start < keypoints.Count && link.end < keypoints.Count)
            {
                Keypoint kp1 = keypoints[link.start];
                Keypoint kp2 = keypoints[link.end];

                // Convert keypoints to Unity's Vector3 (adjust axes as needed)
                Vector3 pos1 = new Vector3(kp1.x, kp1.y, kp1.z);
                Vector3 pos2 = new Vector3(kp2.x, kp2.y, kp2.z);

                // Draw lines between keypoints
                Debug.DrawLine(pos1, pos2, skeletonColor, Time.deltaTime);
            }
        }
    }

    void OnDrawGizmos()
    {
        if (jsonData != null)
        {
            foreach (var frame in jsonData.instance_info)
            {
                foreach (var instance in frame.instances)
                {
                    DrawGizmoSkeleton(instance.keypoints);
                }
            }
        }
    }

    void DrawGizmoSkeleton(List<Keypoint> keypoints)
    {
        Gizmos.color = skeletonColor;
        foreach (var link in jsonData.skeleton_links)
        {
            if (link.start < keypoints.Count && link.end < keypoints.Count)
            {
                Keypoint kp1 = keypoints[link.start];
                Keypoint kp2 = keypoints[link.end];

                // Convert keypoints to Unity's Vector3 (adjust axes as needed)
                Vector3 pos1 = new Vector3(kp1.x, kp1.y, kp1.z);
                Vector3 pos2 = new Vector3(kp2.x, kp2.y, kp2.z);

                // Draw lines between keypoints using Gizmos
                Gizmos.DrawLine(pos1, pos2);
            }
        }
    }
}

Explanation:

• Keypoint and SkeletonLink classes: These classes represent a keypoint’s 3D coordinates and the connections between keypoints (skeleton links).
• LoadJsonData(): This method loads the JSON data from a TextAsset.
• DrawSkeleton(): This method uses Debug.DrawLine to draw lines between keypoints based on the skeleton links defined in the JSON.
• OnDrawGizmos(): Uses Gizmos.DrawLine for drawing lines in the editor (if you want to visualize the skeleton in the Unity editor).

Steps to Use:

1. Attach this script to a GameObject in Unity.
2. Drag your JSON file (saved as a TextAsset) into the jsonDataFile field of the script.
3. Run the game, and the keypoints will be drawn as lines using Debug.DrawLine.

This will visualize the 3D pose skeleton from your JSON data in Unity. You may need to adjust coordinate systems depending on how your keypoints are defined in the JSON.

You

Make the entire thing DOTS friendly and blittable, by using structs of simple types

ChatGPT

To make the system DOTS-friendly and use structs with simple types (blittable types), we need to:

1. Use Unity’s ECS (Entity Component System): We’ll create components for keypoints and skeleton links.
2. Ensure Blittable Structs: We’ll use simple blittable types like float3 and ensure that the structs contain only blittable data (no references or complex types like List<>).
3. Convert Keypoint and SkeletonLink to be components that Unity’s ECS system can process.
4. Draw Skeleton Using Jobs: We’ll create a job system to draw lines between the keypoints, leveraging Unity’s Burst compiler and job system for performance.

Here’s how you can refactor the code for Unity DOTS:

Step 1

You

NO ENTITIES

ChatGPT

If you want to make the system DOTS-friendly without using entities and still ensure that everything is blittable and efficient using structs with simple types, we can:

• Replace all complex types like List<T> with arrays or native collections like NativeArray<T>.
• Ensure that all data structures are blittable.
• Use Unity’s Job System and Burst Compiler to parallelize and optimize the drawing.

Here’s how you can refactor the system:

1. Define Blittable Structs

We will replace List<Keypoint> and List<SkeletonLink> with blittable arrays. We’ll use float3 for positions instead of Vector3 to keep things simple and performant.

2. Refactor the Keypoint and SkeletonLink into Blittable Structs

using UnityEngine;
using Unity.Burst;
using Unity.Collections;
using Unity.Jobs;
using Unity.Mathematics;

public struct Keypoint
{
    public float3 position; // Blittable type, holds x, y, z coordinates
}

public struct SkeletonLink
{
    public int start; // Index of start keypoint
    public int end;   // Index of end keypoint
}

public struct FrameData
{
    public NativeArray<Keypoint> keypoints; // Blittable array
}

public struct SkeletonData
{
    public NativeArray<SkeletonLink> skeletonLinks; // Blittable array
}

3. Job System for Drawing Lines

We will use Unity’s Job System to run the drawing logic in parallel. This improves performance by taking advantage of multiple CPU cores.

using Unity.Burst;
using Unity.Collections;
using Unity.Jobs;
using Unity.Mathematics;
using UnityEngine;

public class SkeletonVisualizer : MonoBehaviour
{
    public Color skeletonColor = Color.red;

    // Define skeleton and frame data
    private SkeletonData skeletonData;
    private FrameData frameData;

    // NativeArrays for keypoints and skeleton links
    private NativeArray<Keypoint> keypoints;
    private NativeArray<SkeletonLink> skeletonLinks;

    // Create a job for drawing lines between keypoints
    [BurstCompile]
    struct DrawSkeletonJob : IJobParallelFor
    {
        [ReadOnly] public NativeArray<Keypoint> keypoints;
        [ReadOnly] public NativeArray<SkeletonLink> skeletonLinks;
        public Color skeletonColor;

        public void Execute(int index)
        {
            SkeletonLink link = skeletonLinks[index];

            float3 pos1 = keypoints[link.start].position;
            float3 pos2 = keypoints[link.end].position;

            // Draw line between the keypoints using Debug.DrawLine
            Debug.DrawLine(pos1, pos2, skeletonColor);
        }
    }

    private void Start()
    {
        // Example initialization: initialize arrays with sample data
        InitializeData();
    }

    void InitializeData()
    {
        // Define sample keypoints
        keypoints = new NativeArray<Keypoint>(5, Allocator.Persistent);
        keypoints[0] = new Keypoint { position = new float3(0, 0, 0) };
        keypoints[1] = new Keypoint { position = new float3(1, 0, 0) };
        keypoints[2] = new Keypoint { position = new float3(1, 1, 0) };
        keypoints[3] = new Keypoint { position = new float3(0, 1, 0) };
        keypoints[4] = new Keypoint { position = new float3(0.5f, 0.5f, 1) };

        // Define sample skeleton links (which keypoints to connect)
        skeletonLinks = new NativeArray<SkeletonLink>(4, Allocator.Persistent);
        skeletonLinks[0] = new SkeletonLink { start = 0, end = 1 };
        skeletonLinks[1] = new SkeletonLink { start = 1, end = 2 };
        skeletonLinks[2] = new SkeletonLink { start = 2, end = 3 };
        skeletonLinks[3] = new SkeletonLink { start = 3, end = 0 };

        // Assign the skeleton and frame data
        skeletonData.skeletonLinks = skeletonLinks;
        frameData.keypoints = keypoints;
    }

    private void Update()
    {
        // Schedule the job to draw the skeleton
        var job = new DrawSkeletonJob
        {
            keypoints = frameData.keypoints,
            skeletonLinks = skeletonData.skeletonLinks,
            skeletonColor = skeletonColor
        };

        // Schedule the job to run on multiple threads
        JobHandle handle = job.Schedule(skeletonData.skeletonLinks.Length, 64);
        handle.Complete(); // Ensure the job is completed before continuing
    }

    private void OnDestroy()
    {
        // Dispose of NativeArrays to prevent memory leaks
        if (keypoints.IsCreated)
        {
            keypoints.Dispose();
        }

        if (skeletonLinks.IsCreated)
        {
            skeletonLinks.Dispose();
        }
    }
}

Key Changes:

1. Blittable Structs:

   • The Keypoint and SkeletonLink structs are blittable because they only contain primitive data types like float3 and int.

2. NativeArray Usage:

   • We use NativeArray<Keypoint> and NativeArray<SkeletonLink> for holding keypoint positions and skeleton connections. These arrays are allocated in Persistent memory, making them suitable for long-term use across multiple frames.

3. Job System:

   • The DrawSkeletonJob is a parallel job that loops over the skeleton links and draws lines between the keypoints using Debug.DrawLine.

4. Burst Compiler:

   • The DrawSkeletonJob is marked with [BurstCompile], allowing the Burst compiler to optimize the job for performance. This can significantly improve performance, especially for large datasets.

5. Memory Management:

   • NativeArray instances are properly disposed of in the OnDestroy method to avoid memory leaks.

Usage in Unity:

• Attach the SkeletonVisualizer script to a GameObject in your scene.
• During Start, the system will initialize sample data. You can replace this with the actual data parsed from your JSON file.
• The job system will efficiently draw the skeleton each frame using Debug.DrawLine.

This approach makes the code DOTS-friendly without using entities, ensuring that all the data structures are blittable and leveraging Unity’s Job System and Burst Compiler for efficient execution.