Body pose tracking

A simple adapter in Python is developed in MARSH-Sim public repository based on RealSense Pose Detector which was fairly easy to run and works, tested with Intel RealSense D455F.

The application sends the HUMAN_BODY_POSE message. Currently the 3D system of coordinates is from camera's point of view:

  • X: forward
  • Y: right
  • Z: down

The coloring scale goes between 0 or missing data - deep blue and 2 meters or more - deep red. The camera can measure greater distances, but this makes the view of cockpit more legible.

screenshot of tracking adapter

Installation

Note that the RealSense cameras require a USB-C cable with USB 3 connections.

The project is managed with uv. This command should take care of dependencies and start the program:

uv run pose.py

Windows

Intel RealSense SDK, get the latest stable release from the Releases section on the right.

Tracked points

Currently we're using Google's MediaPipe Pose Landmarker. The landmark model predicts the location of 33 full-body landmarks (see figure below), each with (x, y, z, visibility). The current model does not use depth data directly, we're projecting the positions identified on 2D image into 3D space using RealSense afterwards. This means that the position of points is centered on the joint in 2D, but on the closest surface to camera in depth.

Messages are sent only for points that have visibility higher than the minimum threshold. The definitions HUMAN_BODY_POINT start with the same indices as this model's output:

diagram of point positions on human body

Alternate models

For skeletal pose estimation Intel recommends (1)(2):