Action recognition is the task of detecting and classifying human actions in video, determining not just what objects are present in a frame but what the people in it are doing. Human pose estimation is the input that makes this reliable: by reducing a person to a compact skeleton of keypoints, pose estimation strips away background clutter and clothing variation, leaving a signal that action models can classify accurately. This guide covers both techniques together, with the goal being behavior detection for safety and security: turning pose and action into an alert a human operator can act on.
Key Takeaways
Action recognition detects and classifies human actions within video sequences, analyzing spatial and temporal data together to interpret what is happening rather than just what is visible in a single frame. A single image can show a person mid-air; only the sequence of frames around it reveals whether that person is jumping, falling, or being thrown.
Human pose gives this task a compact, robust signal to work from. Rather than classifying actions from raw pixels (which vary enormously with lighting, clothing, camera angle, and background), a pose-based system reduces each person to a small set of body keypoints (shoulders, elbows, hips, knees) and classifies movements based on how those points change position over time. This abstraction is what makes action recognition tractable at the level of accuracy required for safety applications.
The task is important for a wide range of applications: surveillance and public safety (detecting aggression, falls, trespassing), healthcare (monitoring patient movement, detecting falls in elderly care), sports analytics (technique analysis, performance tracking), and human-computer interaction (gesture control). This guide focuses specifically on the safety and security use case, where the body, its movement, and its relationship to other people and the environment are the signal that determines whether an alert should fire.
Two architectural approaches dominate action recognition systems, and the choice between them shapes every downstream design decision.
Estimate pose from each frame first (extracting keypoints and skeleton structure), then classify the action from the sequence of keypoints rather than from raw pixels. This approach is computationally efficient at the classification stage (a skeleton is a tiny fraction of the data volume of a video frame), robust to background and clothing variation, and inherently privacy-preserving since the action classifier never needs to see raw pixels after the pose extraction step.
Classify directly from pixels or optical flow, without an intermediate pose representation. Convolutional neural networks recognize spatial features directly from RGB frames, and two-stream networks combine RGB frames and optical flow to achieve high accuracy by capturing both appearance and motion information in parallel branches. 3D CNNs capture spatial and temporal features simultaneously by treating a video clip as a volumetric input rather than a sequence of independent images. Recurrent neural networks (LSTMs, GRUs) are effective for long-term activity analysis, modeling how a sequence of features evolves across many frames where the action itself unfolds over an extended period.
Pose-first pipelines are the better choice when the action is primarily defined by body movement and posture (most safety-relevant behaviors fall into this category), compute is constrained, or privacy is a deployment requirement. Appearance-first pipelines are necessary when the action depends on context that pose alone cannot capture, such as interaction with an object (picking up a bag, opening a door) where the object itself carries essential information the skeleton does not encode.
Pose-based methods work off skeleton sequences, tracking keypoints over time rather than raw pixels. That's what makes them lightweight and privacy-preserving: the model never sees a face or a background, just joint positions moving frame to frame. They also hold up well when someone changes clothes or lighting shifts, since appearance never factored into the prediction in the first place. The gap shows up when context matters. A skeleton alone can't tell you if someone's hand is near a weapon or an object on the ground, so anything involving object interaction slips past this approach.
2D CNN plus RNN architectures take RGB frames and feed them through a sequential model, usually an LSTM or GRU sitting on top of a convolutional backbone. This setup captures appearance detail the pose-based approach misses entirely, plus a basic sense of how things unfold over time. Where it falls short is fine-grained temporal modeling. An RNN processing per-frame features one at a time doesn't have the same handle on rapid, subtle motion changes that a model built to reason across space and time jointly would have.
Two-stream architectures split the problem in two: one branch processes RGB frames, the other processes optical flow, and the two get fused before a final decision. This is the setup that typically delivers the highest accuracy on this list, since motion gets modeled explicitly rather than inferred indirectly through frame sequences. That accuracy comes at a real cost. Running two parallel branches, computing optical flow on top of that, means meaningfully higher compute overhead than either pose-based or 2D CNN plus RNN approaches.
3D CNNs take a different route: instead of frames processed individually, the model ingests a volumetric clip and runs convolutions across height, width, and time simultaneously. This gives it strong spatiotemporal features without needing a separate optical flow branch or a recurrent layer bolted on afterward. It's also the most expensive option in the group. Full 3D convolutions carry a heavier compute cost than any of the other three methods, which matters directly if you're trying to run this on an edge device with a fixed power and latency budget.
Pose estimation outputs a structured representation of a person's body: a set of keypoints (joint locations such as shoulders, elbows, wrists, hips, knees, ankles), each with a confidence score indicating the model's certainty in that specific location, connected into a skeleton that represents the body's structure. This keypoint-and-skeleton representation is what feeds into the action classification stage of the pipeline described above.
Early pose estimation relied on geometric and feature-based methods: hand-crafted features, pictorial structure models, and classical computer vision techniques that predated deep learning entirely. These approaches were limited in accuracy and struggled with occlusion and viewpoint variation, which is why deep learning-based pose estimation has almost entirely displaced them in production systems.
Deep-learning methods like OpenPose use Part Affinity Fields: a representation that encodes not just where each keypoint is located, but how keypoints connect to each other, allowing the model to correctly associate detected joints with the right person even in a multi-person scene with overlapping bodies. This was a foundational innovation that made reliable multi-person pose estimation from a single image practical for the first time.
The COCO keypoint format (17 points: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles) is the most widely used standard, supported by most major frameworks and pretrained models. MediaPipe's BlazePose extends this to 33 keypoints, adding finer detail around the hands, feet, and face that some applications require. The choice of keypoint set should match the granularity the target action actually needs: detecting a fall or a fight rarely requires hand-level detail, while gesture recognition typically does.
Monocular cameras are commonly used for affordability, since a single standard camera provides sufficient input for most 2D pose estimation models, avoiding the cost and calibration complexity of stereo or depth camera rigs. The trade-off is that monocular pose estimation produces 2D keypoints by default; recovering 3D pose from a single camera requires either a model specifically designed for monocular 3D estimation or multiple camera views.
The major deep-learning pose-estimation model families differ primarily in backbone architecture and how they represent the keypoint prediction problem.
Heatmap-based models predict a confidence map (heatmap) for each keypoint across the entire image, then extract the keypoint location as the peak of that heatmap. This approach, used by HRNet and most high-accuracy models, preserves spatial resolution information and tends to produce more accurate localization. Direct coordinate regression models predict keypoint x,y coordinates directly as numerical outputs, which is computationally simpler and faster but historically less accurate, though recent architectures (including SimCC-based approaches like RTMPose) have substantially closed this gap by treating coordinate prediction as a classification problem over discretized positions rather than direct regression.
HRNet maintains high-resolution representations throughout the network, rather than the downsample-then-upsample pattern used by ResNet-based architectures. This preserves fine spatial detail critical for precise keypoint localization throughout the entire forward pass, rather than recovering it only at the end, and HRNet remains a mature, well-documented choice for production top-down pipelines even though newer architectures have since matched or exceeded its peak accuracy.
ViTPose applies a largely unmodified Vision Transformer backbone to keypoint estimation, using self-attention to model long-range dependencies between joints across the entire image rather than relying purely on local convolutional receptive fields. This lets the model reason about relationships between distant body parts (the position of a wrist relative to the opposite hip, for instance) more directly than convolutional architectures, and ViTPose++ extensions trained across multiple datasets simultaneously have achieved state-of-the-art results on several major benchmarks at once. Recent advances in efficient attention mechanisms (window attention, pooling-window attention) are reducing the computational overhead of transformer-based pose models, making them increasingly viable for practical deployment rather than research-only use.
HRNet runs on a high-resolution CNN architecture and outputs heatmap regression for each joint. Its real strength is precise localization, which is why it shows up so often in production top-down pipelines where accuracy per keypoint matters more than raw speed.
OpenPose took a different path with its multi-stage CNN, using Part Affinity Fields to associate joints across multiple people in a single pass. It pioneered the bottom-up, multi-person approach and still shows up as a research baseline or in legacy systems that were built around it years ago, even though newer architectures have since overtaken it on accuracy.
ViTPose and its successor ViTPose++ moved the field toward Vision Transformers, still relying on heatmap regression but capturing long-range dependencies across the image that convolutional networks tend to miss. This is currently the state of the art for accuracy, and it's the model family teams reach for when maximum precision matters more than inference speed.
RTMPose pairs a CSPNeXt backbone with SimCC, a coordinate classification approach rather than dense heatmap prediction. That combination gives it a strong balance between speed and accuracy, which is exactly why it tends to land in production systems that need both without compromising heavily on either.
MoveNet uses a lightweight CNN with heatmap regression, optimized specifically for speed. It's built for single-person tracking on mobile and edge hardware, where compute budget is tight and full-scene multi-person accuracy isn't the priority.
YOLO11 Pose takes yet another approach: a single-stage detector doing direct regression, handling detection and pose estimation in one forward pass instead of two separate stages. That architecture is what makes it well suited for multi-person, real-time scenarios where every millisecond of pipeline overhead counts.
For applications where each camera or frame contains one subject (a fitness application, a single-occupant vehicle cabin, a controlled workstation), single-person models trade multi-person handling capability for substantial speed gains.
MoveNet offers two variants: Lightning, optimized for ultra-fast inference, and Thunder, optimized for higher accuracy at somewhat higher compute cost. MoveNet processes images in under 7 milliseconds on a Google Pixel 4 (TensorFlow team benchmarks, 2021), making real-time pose estimation feasible even on modest mobile devices without dedicated AI acceleration hardware. Both variants detect 17 COCO-format keypoints and are widely deployed precisely because of this computational efficiency advantage over larger models.
MediaPipe Pose (built on the BlazePose architecture) identifies 33 landmark locations, extending beyond the standard COCO set to include additional detail useful for fitness, physical therapy, and gesture applications. MediaPipe Pose uses a detector-tracker pipeline: a person detector identifies the region of interest in early frames, then a lighter tracking model follows the subject across subsequent frames without re-running full detection on every frame, which is a significant efficiency gain for video applications where the subject does not move dramatically between consecutive frames.
Recommended single-person scenarios: MoveNet Lightning for the tightest compute and latency budgets (wearables, low-power embedded devices); MoveNet Thunder or MediaPipe Pose where moderate accuracy improvement is worth a modest compute increase; either model is a reasonable default whenever the deployment can guarantee a single subject per camera view, which simplifies the pipeline considerably relative to multi-person handling.
Most real-world safety and security deployments involve more than one person in frame, which requires a fundamentally different architectural approach than single-person models.
First run a person detector (commonly an object-detection model like YOLO) to localize each individual in the frame, then run single-person pose estimation independently on each detected bounding box crop. This approach generally achieves higher per-person accuracy because each pose estimation pass operates on a cropped, normalized region, but compute cost scales with the number of people in the frame, which becomes a real constraint in genuinely crowded scenes.
Detect all keypoints across the entire image first (without first detecting individual people), then group those keypoints into distinct skeletons using an association method (Part Affinity Fields, as pioneered by OpenPose, is the classic example). This approach has more consistent compute cost regardless of how many people are in frame, which makes it better suited to dense crowds where a top-down approach's per-person cost would become prohibitive, at some cost to per-person accuracy relative to top-down methods.
DETRPose is designed for real-time multi-person pose estimation using a transformer-based detection-and-pose architecture, representing the first practical real-time transformer approach specifically built for the multi-person case rather than adapting a single-person transformer model. YOLO11 Pose, which performs object detection and pose estimation in a single forward pass, reports 89.4% mAP at an IoU threshold of 0.5 on the COCO Keypoints benchmark while maintaining real-time inference speeds above 200 FPS on standard GPU hardware (Roboflow benchmark, 2025), making it a strong default choice for multi-person scenarios that need both speed and accuracy.
Bottom-up architectures or single-pass detection-and-pose models like YOLO11 Pose generally outperform classic top-down pipelines as person count increases, because their compute cost does not scale linearly with the number of individuals present, which is the dominant constraint in transit platforms, stations, and other high-density public spaces.
HRNet's high-resolution strategy maintains parallel multi-resolution feature streams throughout the network, repeatedly exchanging information between high-resolution and lower-resolution branches rather than collapsing to a single low-resolution bottleneck and then upsampling. This architectural choice is what gives HRNet (and its successors) superior accuracy for fine keypoint localization relative to architectures that lose spatial precision through aggressive early downsampling.
Model-based approaches take a different strategy entirely: rather than predicting keypoints directly, model-based methods use predefined skeletal models (or, more elaborately, parametric body mesh models like SMPL) and fit the model's parameters to match the observed image. SMPL and similar mesh-based approaches produce a full 3D body surface representation, not just a sparse set of joint locations, which is valuable for applications needing detailed body shape information (biomechanics, virtual try-on, detailed motion capture) beyond what a skeleton alone provides.
When to prefer high-resolution models: when keypoint localization precision directly affects downstream decision quality, for example distinguishing a near-fall recovery from an actual fall, where a few pixels of joint position error can change the classification outcome. For coarser safety classifications (is this person standing, sitting, or lying down), the localization precision advantage of high-resolution and specialized architectures matters less, and a faster, lighter model is usually the better operational choice.
Keypoint representation records a discrete set of joint locations, typically as (x, y) coordinates with an associated confidence score, providing a sparse but computationally efficient representation of the body. Heatmap representation instead encodes, for each keypoint, a full probability distribution across the image, which is denser but allows the model to express uncertainty and multi-modal ambiguity (a partially occluded joint might have two plausible locations) more naturally than a single coordinate.
A skeletal representation connects keypoints with line segments approximating bone structure, sufficient for most action and behavior classification tasks since the relative motion of joints is what carries the behavioral signal. A mesh representation (as produced by SMPL-style model-based approaches) models the full body surface, capturing shape and volume in addition to joint position, which is unnecessary overhead for most safety applications but valuable where body shape itself is informative.
A small set of core joints (shoulders, hips, knees) is typically sufficient for whole-body actions like walking, running, falling, or fighting. Tasks requiring finer discrimination (gesture recognition, sign language, detailed gait analysis) need the larger keypoint sets that extend to hands and feet, such as the 33-point MediaPipe format. Selecting more keypoints than a task requires adds compute cost without improving the actual classification the system needs to make.
Recovering accurate 3D pose, particularly from a single monocular camera, faces persistent challenges from depth ambiguity (a 2D projection is consistent with multiple 3D poses) and occlusion (a joint hidden behind the body or another object cannot be directly observed, and its position must be inferred). These challenges are why most production safety applications use 2D pose estimation, which avoids the depth ambiguity problem entirely, reserving 3D pose estimation for applications where depth information is essential to the task and the additional complexity and error are justified.
Public benchmark datasets are the standard reference for evaluating both pose estimation and action recognition models.
MPII provides 16 2D keypoints with occlusion labels and remains a classic benchmark for both single-person and multi-person pose estimation work.
MSCOCO offers 17 keypoints along with bounding boxes, and it functions as the dominant standard benchmark across the entire field. Most published pose estimation results get measured against it, so it's the dataset to check first when comparing model claims.
Human3.6M scales up considerably, with 3.6 million human poses captured indoors. It's the go-to large-scale benchmark for 3D pose estimation, though the indoor setting means it doesn't reflect the lighting or clutter variation found in real-world deployments.
PoseTrack takes a different angle entirely. Instead of scoring single frames in isolation, it provides large-scale video-level annotations built specifically for tracking people across frames. That makes it the standard reference when a model needs to maintain identity over time, not just estimate a pose in one still image.
MoVi combines mesh annotations with 2D and 3D keypoints in the same dataset. Pairing skeletal ground truth with mesh-based ground truth gives it a use case neither pure keypoint datasets nor pure mesh datasets cover on their own.
3DPW captures in-the-wild activities, and its real value is that the 3D pose ground truth was recorded outside a lab. Most 3D pose datasets get built in controlled motion-capture studios, so 3DPW fills a gap that matters a lot once a model has to handle real environments instead of a rig with perfect lighting and no occlusion.
NTU RGB+D is a large-scale benchmark built around 3D skeleton data, and it's the dataset most teams reach for when training or evaluating skeleton-based action classification models.
Kinetics-400 takes an appearance-based approach instead, covering hundreds of action classes using raw RGB video rather than skeleton data. It's become the standard benchmark for teams working on appearance-based action recognition, the counterpart to what NTU RGB+D does for skeleton-based methods.
Annotating keypoints across video frames is significantly more labor-intensive than single-image annotation; semi-automated annotation (using a pretrained model to propose keypoints, with human correction) substantially reduces this cost. Temporal consistency across frames should be checked explicitly, since frame-by-frame independent annotation can introduce jitter that a sequential model will then have to learn to ignore rather than learning genuine motion.
Object Keypoint Similarity (OKS) is the standard metric for evaluating keypoint localization accuracy, analogous to IoU for bounding boxes but accounting for keypoint-specific scale and visibility. Keypoint mAP, computed by averaging precision across OKS thresholds, is the standard single-number summary metric reported in published benchmarks (as seen in the YOLO11 Pose and HRNet figures cited earlier in this guide).
Temporal augmentations for video training include varying playback speed, temporal cropping (selecting different windows of a longer clip), and frame dropout, all of which expose the model to a wider range of temporal patterns than the raw training set alone would provide and improve robustness to variation in how quickly different individuals perform the same action.
Some safety-relevant poses (a person mid-fall, a specific aggressive posture) are rare in naturally collected footage. Synthetic data generation, using motion capture data retargeted to varied body types or simulation environments, can expand training coverage for these rare but operationally important cases without waiting for sufficient real-world examples to accumulate.
Training a single model with a combined loss for both pose estimation accuracy and downstream action classification accuracy can improve both tasks relative to training them as fully separate, sequential stages, because the pose representation that the model learns becomes directly shaped by what is useful for the action classification objective, not just by raw keypoint localization accuracy in isolation.
Skeleton-based methods analyze human joint coordinates for action recognition rather than processing raw video directly, and this approach has become the dominant paradigm for safety-relevant action recognition specifically because of its computational efficiency and robustness benefits. Deep learning significantly improves action recognition performance across the board relative to classical, hand-engineered feature approaches, which is consistent with the broader pattern across computer vision over the past decade.
This is where pose estimation and action recognition become an operational safety system rather than a research benchmark exercise. The full chain runs: pose, then action, then behavior, then alert.
Pose extracts the skeleton for every person in frame, every frame. Action classifies short sequences of that skeleton data into discrete categories: walking, running, falling, raising an arm, crouching. Behavior aggregates action classifications over a longer time window and across multiple people to recognize higher-level patterns: a fight (two people in close proximity with rapid, opposing limb movements), a fall (a sudden transition from standing to horizontal pose followed by no recovery), loitering (low movement in a restricted zone over an extended period), or aggression more broadly. Alert packages the detected behavior into a structured notification, with the supporting pose and action evidence attached, routed to a human operator for review.
Aggression and violence recognition is the safety-relevant action category that gets the most attention in transit and public-space deployments. This is fundamentally a temporal pattern recognition problem built on pose and action sequences rather than a single-frame classification task: a raised arm alone is ambiguous (a wave, a punch, a reach), but a raised arm combined with rapid forward acceleration, proximity to another person, and a defensive posture in that other person is a much stronger signal. Pose-based action recognition is well suited to capturing exactly this kind of compound, temporally-extended pattern.
Anomaly detection for unusual behavior complements this targeted aggression recognition with a more open-ended approach. Anomaly detection identifies suspicious behaviors in real time using action recognition outputs as input features, flagging behavior sequences that deviate from learned normal patterns even when the specific behavior was never explicitly trained as a named category. This combination, action recognition for known behavior types plus anomaly detection for the unknown, is the practical architecture most production behavior-detection systems converge on, and it is used in video surveillance for suspicious action recognition broadly across public safety, transit, and retail loss-prevention applications. See edge AI CCTV threat detection and video anomaly detection for rare safety events for the adjacent systems this connects to.
Skeleton or pose data is far less identifying than raw video. A skeleton with 17 or 33 points carries almost none of the biometric and contextual information present in a raw video frame: no face detail, no clothing, no background that could reveal location or context beyond what the pose itself implies. For transit and public-space deployments, where regulatory and public trust concerns around continuous video surveillance are significant, an architecture that performs pose extraction on-device and discards the raw frame immediately after, retaining and transmitting only the skeleton sequence, meaningfully reduces the privacy footprint of the entire system while still enabling accurate behavior detection.
InTechHouse case study: Pose-to-alert behavior detection in transit CCTV with PESA
InTechHouse implemented a pose-to-action-to-alert pipeline for unsafe passenger behavior detection in a transit operator's onboard CCTV system, developed in partnership with PESA. The system needed to detect aggressive behavior and falls within a moving vehicle cabin in real time, using cameras already installed for general surveillance purposes.
The full chain ran onboard on embedded hardware: pose extraction processed each camera frame to keypoints, the action classification stage operated on short sliding windows of pose sequences, and the behavior layer aggregated action outputs across all detected individuals in the cabin to flag aggression or fall events. The entire chain completed within the frame budget required to keep pace with live video, with no cloud dependency at any stage.
Using pose data rather than transmitting or storing raw video for the behavior-classification stage materially reduced the system's privacy footprint: once keypoints were extracted, the raw frame was not retained beyond the buffer needed for the next pose extraction pass, and only skeleton sequences and resulting alert metadata persisted. This was a meaningful point in the system's data protection documentation for a deployment processing footage of identifiable passengers in a public transit setting.
For real-time behavior detection, the pose-and-action pipeline should sustain at least 15–30 FPS to avoid missing fast-developing events like a fall or the onset of a physical altercation. Lower frame rates (5–10 FPS) may be acceptable for slower-developing behaviors like loitering detection, where the relevant time scale is seconds to minutes rather than a single video frame.
Pose estimation and action classification models trained at full precision should be quantized (typically to INT8) before deployment to embedded hardware, reducing model size and increasing inference throughput with minimal accuracy loss when calibration is done properly. See model quantization for edge deployment for the detailed mechanics.
Pruning removes low-contribution weights and connections from a trained model, further reducing compute cost, which compounds with quantization to bring larger, more accurate models within the latency budget of constrained edge hardware.
NVIDIA Jetson platforms (Orin Nano through AGX Orin) are the common choice for multi-camera, multi-person pose and action pipelines requiring substantial compute. Mobile CPUs and lightweight NPUs are sufficient for single-person, single-camera deployments using models like MoveNet Lightning, given its sub-10-millisecond inference time on commodity mobile hardware. See edge AI hardware for in-vehicle systems for the full hardware selection framework.
Benchmark figures reported by model authors are typically measured on specific reference hardware (a particular GPU, a particular phone model) that may not match the deployment target. Computational resources available on the actual edge device, and computational overhead from preprocessing, postprocessing, and any multi-camera aggregation logic, must be measured directly on that device before committing to a model and pipeline architecture for production.
Reducing input resolution is one of the most direct levers for trading accuracy against speed; running an ablation that measures accuracy and latency at several candidate resolutions (for example, 256x256 down to 128x128) on the target hardware gives a concrete, deployment-specific basis for the resolution decision rather than relying on published benchmark defaults.
Production deployments should specifically measure performance degradation when keypoints are partially occluded, since this is a near-universal condition in crowded real-world scenes and rarely well represented in clean benchmark test sets. A model that reports strong aggregate accuracy on a standard benchmark can still degrade substantially under occlusion conditions specific to a deployment's actual camera angles and crowd density.
Accuracy figures without corresponding latency numbers, measured on the same hardware, are not directly useful for a real-time deployment decision. Every benchmark claim made for a production system should report both, tied explicitly to a specific dataset and a specific hardware configuration, since the spatial resolution and compute budget interact directly with both metrics.
Claims should be reproducible: tied to a named dataset, a named hardware configuration, and a documented evaluation protocol, rather than presented as a bare percentage figure with no context for how it was measured.
Occlusion and inter-person association remain the central unsolved problems in crowded multi-person pose estimation. When two or more people overlap in the frame, the model must not only estimate each person's pose despite partial visibility, but correctly associate each detected keypoint with the right individual, a problem that becomes substantially harder as crowd density increases and is directly relevant to the crowded transit reality this guide's use case operates in: a busy platform or cabin is exactly the dense, occlusion-heavy scenario where these challenges are most acute.
Generalization across viewpoints and datasets is an ongoing limitation: models trained primarily on datasets captured from typical eye-level or slightly elevated camera angles can underperform when deployed with the overhead or extreme-angle camera placements common in transit and security installations. Improving generalization to these deployment-relevant viewpoints, rather than only optimizing for the camera angles represented in standard benchmarks, is a practical priority distinct from chasing benchmark-leaderboard accuracy.
Prioritizing real-time performance on edge devices remains a persistent tension against the field's broader trend toward larger, more accurate (and more compute-hungry) architectures like full-scale transformer models. The practical state of the art for edge-deployed safety systems increasingly favors models specifically designed for the speed-accuracy trade-off (RTMPose, YOLO11 Pose, MoveNet) over the highest-accuracy research models, which are often impractical to deploy within real-time edge constraints regardless of their benchmark leaderboard position.
Self-supervised learning for unlabeled video is an active research direction relevant to this domain because labeled pose and action data, especially for rare safety-relevant behaviors, is expensive and slow to collect at scale. Self-supervised approaches that can learn useful pose and motion representations from unlabeled video, then require only a small amount of labeled data for the final behavior classification task, could substantially reduce the data collection burden for new deployment environments.
Match the model to the deployment's specific constraints: single-person or multi-person scene, target hardware compute budget, required keypoint granularity, and accuracy requirements for the specific behaviors being detected.
Collect representative footage from the actual deployment environment (camera angle, lighting, typical crowd density) rather than relying solely on generic public benchmark datasets, and ensure rare but operationally important behaviors are adequately represented, using synthetic augmentation where real examples are too scarce.
Build the full pose-to-action-to-behavior-to-alert chain, validated end-to-end on the target hardware, with explicit measurement of total pipeline latency, not just individual model inference time in isolation.
Once deployed, track inference latency, detection confidence distributions, and (where feedback is available) false positive and false negative rates over time, since model performance can drift as the deployment environment changes (new camera angles added, seasonal lighting shifts, changes in typical crowd patterns).
Landmark papers: Cao et al., "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields" (the OpenPose paper); Sun et al., "Deep High-Resolution Representation Learning for Human Pose Estimation" (HRNet); Xu et al., "ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation."
Benchmark datasets: MSCOCO Keypoints, MPII Human Pose, Human3.6M, PoseTrack, NTU RGB+D, Kinetics-400 (all detailed in the Datasets section above).
Implementation toolkits: MediaPipe (Google) for fast, well-documented single and multi-person pose estimation with mobile-first deployment support; Ultralytics YOLO11 Pose for combined detection-and-pose in a single real-time pipeline; MMPose (OpenMMLab) for a broad library of research and production-grade pose model implementations including RTMPose.
This is a practical resource list rather than an exhaustive literature review; teams building a production system should expect to evaluate two or three candidate models directly on their own deployment data and hardware rather than selecting purely from published benchmark leaderboards.
Human pose estimation is the computer vision task of detecting the locations of key body joints (such as shoulders, elbows, hips, and knees) from an image or video frame, producing a structured skeleton representation of a person's posture. It is the foundational input for most modern action recognition systems, because classifying movement from a compact set of joint positions is more robust and computationally efficient than classifying directly from raw pixels.
Action recognition works by analyzing a sequence of frames (or, in pose-based pipelines, a sequence of extracted skeletons) to classify what movement or activity is occurring, using both spatial information (body configuration at a given moment) and temporal information (how that configuration changes over time). Models range from pose-based classifiers that operate on keypoint sequences to appearance-based approaches like two-stream networks and 3D CNNs that learn directly from RGB video and optical flow.
Yes. Aggression and violence detection from CCTV is typically built on pose-based action recognition: extracting skeletons for each person in frame, classifying short sequences of movement (proximity, rapid limb motion, defensive posture in a second person) as aggressive or non-aggressive, and flagging confirmed sequences for operator review. This is a temporal pattern recognition task rather than a single-frame classification, since aggressive behavior is defined by how a sequence of movements unfolds, not by any single static pose.
The standard pipeline extracts spatial information per frame (either raw appearance features or pose keypoints), then applies a temporal model (an RNN, a 3D CNN, or a transformer) to classify how that spatial information changes across a sequence of frames. Pose-based pipelines do this on a compact skeleton representation; appearance-based pipelines do it on raw pixels or optical flow. The right choice depends on compute constraints, privacy requirements, and whether the target actions depend primarily on body movement or also require object-interaction context that pose alone does not capture.
Yes, substantially. A skeleton representation, typically 17 to 33 keypoint coordinates per person per frame, carries almost none of the identifying detail present in raw video: no facial features, no clothing detail, no readable background context. For public-space and transit deployments where privacy regulation and public trust are significant concerns, an architecture that extracts pose on-device and discards the raw frame immediately, retaining only the skeleton sequence and resulting behavior alerts, meaningfully reduces the system's privacy footprint while still supporting accurate behavior detection.
An expert in Artificial Intelligence, professor and researcher, who has authored numerous scientific publications and led international projects focused on AI, machine learning, and data-driven systems.
His work connects academic research with industrial applications, applying advanced AI models to practical challenges across sectors such as defense, telecommunications, smart industry, and cybersecurity. He has extensive experience in designing and implementing intelligent systems in complex, high-demand environments.
In addition to his technical work, Prof. Andrysiak shares insights on AI trends and applications as a speaker, mentor, and author, contributing to discussions on the role of AI in modern technology and digital transformation.
This initial conversation is focused on understanding your product, technical challenges, and constraints.
No sales pitch - just a practical discussion with experienced engineers.
Share a few details about your product and context. We’ll review the information and suggest the most appropriate next step.
