How it works:
Marker-based motion capture is a form of motion capture technology that involves the use of tracking markers strategically placed on the participant’s body to create a local coordinate system defining the underlying local anatomy of the individual. This coordinate system is developed using anatomical landmarks identified by a specialist operator, who palpitates the regions of the body segments. This process can take up to 45 minutes and requires participants to dress in specific, tight-fitting clothing to prevent marker obstruction from skewing results. Once determined, the tracking coordinates can be used to ascertain the pose and orientation of the individual’s body while carrying out a variety of specific activities. There are two types of markers utilized with this technology. The first of which are active markers, which emit infrared light for detection by infrared camera sensors. As they emit light themselves, active markers require participants to carry power sources on their physical body as they perform movements or connect to an external battery source using wires, all of which may act as a potential constrictor of free movement, thus skewing data. To avoid this data bias, passive markers act as a potential alternative, which consists of retroreflective markers placed on the skin that reflect infrared light that is emitted from the infrared cameras themselves. The camera’s sensors then detect the infrared light to identify the local anatomical coordinate system.
Why not marker-based?
There are several limitations to the use of marker-based technology. Its long duration (approximately 2 hours for one participant), invasive process (uncomfortable clothing in a specifically controlled laboratory environment) involving physical bodily palpitations by a trained operator, as well as the requirement to carry a power source or connect to a power source through wiring limits the accuracy of any potential data collection. Additionally, the heavy commitment factor involved in the study may deter potential subjects from participating altogether. Thus, researchers have proposed novel technological approaches that address these limitations and bridge these gaps to create a more efficient approach to motion technology.
While the benefits of marker-based are still appreciated within the field of biokinematics, transitioning to markerless motion capture technology (MLMC) allows researchers to advance their studies through larger study samples, greater proficiency, and a more comfortable, naturalistic, and adaptable data collection environment.
Unlike marker-based motion capture, MLMC is not dependent on the usage of physical markers to identify the local anatomical coordinate system of the body. Rather than placing passive or active markers on the participant's body, MLMC uses videos to capture participants moving organically in their environment. These videos are processed with advanced machine learning that identifies rigid body segments to determine their pose and orientation in reference to a global reference frame. Once these rigid segments have been identified, video processing allows for the tracking of these segments, which then creates a 3-dimensional (3D) visualization of participant movements. There are two main types of MLMC: depth-based and video-based. Other methods include using Wi-Fi or LIDAR. [1]
Depth-based:
Depth-base motion capture uses depth cameras that measure the distance from each pixel to the camera’s position; it achieves this depth analysis from the infrared emitter, which creates patterns of infrared light off the objects being perceived by the infrared sensor.[2] Depth perception paired with RGB sensing allows the cameras to understand positionality and the 3-dimensional structure of objects within space. By using RGB sensing in conjunction with depth-based sensors, the limited range of depth-based sensors in detecting shapes farther than 5 meters away is overcome.
Video-based:
Video-based motion capture uses one or multiple camera views to record video data or participant movement, which is analyzed post-processing. Prior to analysis, video processing requires computer vision techniques and deep learning convolutional neural networks. Software such as Theia3D uses key points on the body that signify anatomical rigid body segments based on previously calculated joint localization biomarkers to create a visual hull of the body’s skeleton as the participant performs the designated tasks. Computer visioning requires pose estimation and orientation to identify common body features and create key points for joint localization. To specify specific joint biomarkers in the human body with a higher degree of efficiency, rigid body segments are predicted for tracking 3D pose estimation can only fully occur with the use of the six degrees of freedom (6DOF). The 6DOF refers to the ability of the rigid body to translate along three different axes and rotate around three different axes. Using these biomarkers and 6DOF, deep learning techniques can be applied to predict the position of anatomical keypoints and construct the 3D model of the skeleton.[3]
Though markerless motion capture (MLMC) is an efficient and innovative system, it is still a recent technology without equivalent specificity and credibility within the biomechanics field. As a result, there still exist several limitations to its full integration within the field.
Requires specialized coding knowledge, causing researchers in biomechanics difficulty in adopting its technology.
Sony limitations: cannot concurrently synchronize cameras to force plates during data collection, thus requiring a third-party device to synchronize the data
Optitrack limitations: downloads videos directly onto system hard drive rather than on camera memory cards, forces system to utilize unnecessary space for storage rather than allocating computer resources towards data handling and processing. Not a mobile system due to the higher amounts of hardware required during data collection. Moreover, Optitrack requires the manual creation of intrinsic calibrations code for each camera (compared to Sony RX0 II, which contains default intrinsic calibrations due to the small deviations in lens distortion between cameras)
Theia3D limitations: Despite the various biomarkers in Theia’s software being advantageous to increasing the specificity of kinematic tracking, users cannot access which key points are being used on the data recordings. The transparency of the system’s use of key points may pose a limitation if studies require targeting of specific biomarkers or omission of others. Moreover, if one of the markers poses an issue in joint localization, users would not be able to identify which of the key points are at fault
As software and algorithms develop, raw data needs to be re-processed to maintain proper versioning of the data
Troubleshooting the system may not be straightforward in answering questions such as: choosing optimal camera views; distance of the cameras; or which colour gamut or white balance to use. These answers depend entirely on the training dataset used and the machine learning or computer vision solution developed
Theia3D is a markerless motion capture that uses over 120 different anatomical key points of the body to create an accurate pose estimation with less variability between sessions, or day-to-day collections, compared to marker-based motion capture [4]. Theia3D has been shown to yield stronger results compared to other markerless motion capture software; this is due to Theia's use of internally created biomarkers as opposed to the use of keypoints accumulated from open-access datasets [4]. The validity of joint localization keypoints in open-access repositories has consistently produced inaccurate results in other markerless motion capture studies [4]. The frequent accuracy of Theia’s markerless pose estimation highlights the advantage of its unique joint localization biomarkers.
The HMRL lab utilizes Theia3D recommended hardware to increase the compatibility and efficiency of the software with the camera and equipment. The Sony RX0 II cameras come with a network switch and are used with camera control boxes. One of the camera control boxes is denoted the “master” and given a unique IP address, whilst the rest of the boxes are prescribed as clients. The switch handles the protocol of communicating the IP addresses between cameras. Thus, when connected to the switch, all synced cameras adopt the time code of the master, allowing for multi-camera frame-by-frame synchronization of multi-angled video data collection.
Optitracks allow for force plate synchronization with cameras simultaneous to video data recording and support a higher frame rate while at the same resolution as the Sony cameras.
The frame rate selected for each data collection depends on the activity the participant is to perform, and the capabilities of the system used to collect data. Slow activities do not require large frame rates, with 60 Hz being sufficient. For faster activities, a larger frame rate of 120 Hz or above is required. This would include activities such as running, jumping, or cross-cutting.
Similar to frame rates selection, cut-off frequencies are dependent on the type of activity the participant is performing. Slow activities like yoga require a frequency of 4 Hz, while moderately-paced activities, such as walking, require a frequency of 8 Hz. Faster activities, like running or jumping, require a frequency of 20 Hz.
Theia3D does support 6DOF, but only in the legs and arms. The shoulder can support 3DOF or 6DOF and the ankle can support 3DOF or 6DOF. However, the knee can only support 2DOF or 3DOF.
*DOF = degrees of freedom
There are several possibilities as to why a camera view is not showing up in the GUI. Check to ensure that all these requirements are met if a camera view isn’t showing up:
Only one camera is set as the “master” control box, with the rest set as client
All camera batteries are sufficiently charged
All cables are connected to the cameras
No cameras are set to or stuck in ‘Formatting’
If the issue persists, contact Sony IT services to troubleshoot the issue further. There may be a hardware issue present.
Motive stores data by directly downloading videos into the selected system’s hard drive. As a result, if background programs are running on your system, it will cause the software to buffer and result in dropped frames or camera views dropping in and out. To resolve this, close all background programs and restart motive to clear the software buffer.
To optimize the data collection process, refer to Theia’s recommendations in utilizing software and implement each suggestion in data collection practices.
Pixel height: Theia recommends maintaining subjects at a pixel height of 500 pixels.
Capture volume: While Theia doesn’t directly recommend a specific capture volume, it is suggested to utilize whichever volume allows participants to consistently maintain a pixel height of 500 pixels. The HMRL lab uses a capture volume of approximately 15 m (length) by 8 m (width) but 4.5 m (height).
Number of cameras: Theia recommends using a minimum of 6 cameras around the capture volume for data collection. However, the HMRL conducts data collections using 8 cameras around the volume to maximize the different viewpoints of each participant.
References:
[1] Gutierrez Farewik, L., Olsson, F., Destro, A., & Davit, S. (2022). Markerless motion capture using iPad pro with LiDAR camera adjusted with artificial neural networks. Gait & Posture, 97, S72–S73. https://doi.org/10.1016/j.gaitpost.2022.07.055
[2] Lannan, N., Zhou, L., & Fan, G. (2022). A Multiview Depth-based Motion Capture Benchmark Dataset for Human Motion Denoising and Enhancement Research. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 426–435. https://doi.org/10.1109/CVPRW56347.2022.00058
[3] Kanko, Robert & Laende, Elise & Selbie, Scott & Deluzio, Kevin. (2020). Inter-session repeatability of Theia3D markerless motion capture gait kinematics. 10.1101/2020.06.23.155358.
[4] Wade, L., Needham, L., McGuigan, P., & Bilzon, J. (2022). Applications and limitations of current markerless motion capture methods for clinical gait biomechanics. PeerJ, 10, e12995. https://doi.org/10.7717/peerj.12995