Small object detection is the task of identifying and localizing objects that occupy only a few pixels in an image: a person at 40 meters, a weapon partially concealed in a crowd, a defect a few millimeters across on a production line. Standard detectors handle these objects poorly because they carry almost no discriminative visual information after the network downsamples the input. This guide covers small object detection in video, in real time, in the deployments where missing one matters: transit safety, industrial inspection, security. The core technique covered is SAHI (Slicing Aided Hyper Inference), the slicing-based approach that recovers detail standard inference discards.
Key Takeaways
Small object detection is the subfield of object detection focused on identifying and localizing objects that are small relative to the image frame, typically defined (per the COCO benchmark convention) as objects occupying fewer than 32x32 pixels in the original image. Small objects often occupy only a few pixels of an image, which puts them at a fundamental information disadvantage relative to larger objects: there is simply less visual signal for the network to learn from.
The scope of this problem spans most safety and security computer vision deployments. Small object detection is crucial for applications including autonomous driving (a pedestrian or cyclist at distance), surveillance (a weapon or unattended bag in a crowded frame), remote sensing and aerial imagery (vehicles or structures in satellite or drone footage), and industrial inspection (a defect a few millimeters across on a production line). In every one of these real world applications, the object that matters most operationally is frequently the smallest one in the frame.
A structural reason this problem persists: most public training datasets focus on medium-to-large objects, which limits small object detection performance for models trained on them. COCO, ImageNet, and similar general-purpose datasets are dominated by object instances that are easy to label and visually prominent. A model trained on this distribution learns to detect what it has seen most: medium and large objects. Small-object performance on these models is a secondary capability, not a primary design target, which is why specialized techniques are needed to close the gap.
A small object's pixel footprint is its entire information budget. At 20x20 pixels, an object provides far less texture, edge, and shape information than a 200x200 pixel object of the same class. Small objects lack discriminative features relative to the background, making the classification task itself harder, independent of the localization challenge.
Convolutional networks repeatedly downsample the input through pooling and strided convolution to build a manageable feature map size. Small objects lose detail as images pass through neural networks: an object that starts at 20x20 pixels in the input may be represented by a single feature map cell, or disappear entirely, by the time the network reaches its deepest layers. Tiny objects often become blurred or lost during this downsampling process before the detection head ever sees them.
In crowded scenes, small objects are disproportionately affected by occlusion because they have so little visible area to begin with. Occlusion reduces the visible area significantly for any object, but for a small object, even partial occlusion can remove the majority of its already-limited pixel footprint, pushing it below the threshold the detector needs for a confident classification.
A detection model trained primarily on a narrow range of object sizes generalizes poorly to objects far outside that range. Cluttered backgrounds compound this: a small object surrounded by texture-rich background clutter has a worse signal-to-noise ratio than the same object against a clean background.
Common evaluation metrics like IoU (Intersection over Union) perform poorly as a measure of detection quality for small objects. A one or two pixel localization error on a 20x20 pixel object produces a much larger relative IoU penalty than the same pixel error on a 200x200 pixel object, even though the practical impact (did the system detect the object at all) may be identical. This metric distortion means standard mAP scores understate how well or poorly a model actually performs on small targets, and it is the reason size-stratified evaluation is essential, covered later in this guide.
The net effect of these compounding factors is elevated false negatives: detecting small targets reliably requires the model and the inference pipeline to work against several independent sources of information loss simultaneously, not just one.
A feature map is the output of a convolutional layer: a spatial grid where each cell encodes learned features (edges, textures, shapes, eventually semantic concepts) for the corresponding region of the input image. Shallow layers produce high-resolution feature maps with fine spatial detail but limited semantic abstraction. Deep layers produce low-resolution feature maps with rich semantic information but coarse spatial precision. Small object detection needs both simultaneously: spatial precision to localize a small target, and semantic richness to classify it correctly.
Feature Pyramid Networks (FPN) solve this by combining information from multiple layers of the network. The architecture builds a top-down pathway with lateral connections: deep, semantically rich feature maps are upsampled and merged with shallow, spatially precise feature maps at each scale. FPN enhances multi-scale feature learning, preserving meaningful information during aggregation rather than relying on a single feature map scale for all object sizes. Shallow layers contribute fine spatial details that benefit small-object localization; deep layers contribute the contextual information needed for reliable classification.
Common feature-fusion strategies beyond standard FPN include PANet (which adds a bottom-up path augmentation to shorten the information path from low-level to top-level features), BiFPN (which uses weighted bidirectional connections to fuse features more efficiently), and attention-based fusion modules that learn which scale of feature to weight more heavily for a given spatial region. The practical recommendation across architectures: preserve high-resolution details by not collapsing the feature pyramid too aggressively, particularly in the early layers feeding the detection head responsible for small-object predictions.
Anchor-based detectors (Faster R-CNN, the early YOLO versions, RetinaNet) predict bounding boxes as offsets from a predefined set of anchor boxes at each spatial location. Faster R-CNN is a two-stage detector: a region proposal network generates candidate object regions, then a second stage classifies and refines each region. This two-stage approach generally achieves higher accuracy, including on small objects, at the cost of inference speed.
Anchor-free detectors (FCOS, CenterNet, and later YOLO versions) predict object centers or keypoints directly without predefined anchors, simplifying the architecture and often improving performance on objects with unusual aspect ratios. Anchor-free approaches have become the dominant pattern in recent real-time object detection models because they reduce the hyperparameter tuning burden associated with anchor box design.
YOLO models excel in real-time object detection because they process the entire image in a single forward pass, making them the practical default for video applications where frame rate matters. The trade-off relevant to small object detection: YOLO models use fixed input sizes, which directly affects small-object detection performance. YOLOv8, for example, is trained on images with a maximum side length of 640 pixels (Ultralytics documentation, 2024). When a high-resolution source image (a 4K security camera frame, for instance) is resized down to 640x640 for inference, small objects that were already marginal in the original resolution become sub-pixel or near-invisible after the resize. Images below 640x640 in their native resolution are themselves considered low resolution for the purposes of small-object work, since there is no recoverable detail to extract regardless of inference strategy.
Transformer and CNN hybrid trends. Vision Transformer-based detectors (DETR and its variants) use self-attention to model long-range spatial relationships across the entire image, which can help with small-object context (recognizing that a small blob near a doorway is more likely a person than the same blob in open sky). Hybrid CNN-transformer architectures combine convolutional backbones for efficient local feature extraction with transformer-based heads for global context, an active area of architecture research for improving small-object recall without the full computational cost of pure transformer models.
Multi-scale detection heads. Modern detectors attach separate prediction heads to different levels of the feature pyramid, with each head specialized for a different object size range. The head attached to the highest-resolution feature map handles small objects; the head attached to the lowest-resolution, most semantically rich feature map handles large objects. This architecture decision is the most direct way to give a model dedicated capacity for small-object prediction rather than treating all scales identically.
Receptive-field enlargement. Dilated convolutions increase the receptive field of a convolutional layer without reducing spatial resolution, by inserting gaps between kernel elements. This lets a layer see more spatial context around a small object without the resolution loss that comes from standard strided downsampling, directly addressing the tension between context and detail that small-object detection requires.
Attention mechanisms. Spatial and channel attention modules learn to weight feature map regions and channels by relevance, effectively suppressing background clutter and amplifying the signal from small, salient regions. This is particularly useful when cluttered backgrounds dominate the frame and a small object's signal would otherwise be diluted in the feature representation.
Image pyramids. Running inference on the image at multiple resolutions (an image pyramid) lets the network detect objects at multiple scales using the same model, at the cost of multiple forward passes. This is conceptually related to the slicing approach covered in the SAHI section, but operates on resolution rather than spatial tiling.
Contextual awareness. Models that incorporate scene context (what other objects and regions are nearby) improve small-object detection because the surrounding context disambiguates objects that would otherwise be too small to classify confidently from appearance alone. A blurry small shape next to a road is more likely a vehicle; the same shape on a platform is more likely a bag.
Anchor optimization or removal. YOLOv5 uses an auto-anchor algorithm with K-means clustering to learn anchor box sizes directly from the training dataset's object size distribution, rather than using generic anchor sizes that may not match the target deployment's object scale profile. For deployments dominated by small objects, this anchor optimization step measurably improves recall relative to generic anchor configurations.
Bounding box regression refinement. Shape-IoU improves bounding-box regression accuracy by incorporating the shape and scale properties of the bounding box into the loss function, rather than relying purely on overlap-based losses (standard IoU, GIoU, DIoU) that disproportionately penalize small-box localization error as discussed in the metrics challenge above. Refined loss functions like Shape-IoU produce more stable training gradients for small object instances, improving convergence and final detection accuracy.
Lightweight backbones for edge deployment. For real-time inference on constrained hardware, lightweight backbones (MobileNet, EfficientNet-Lite, GhostNet) trade some representational capacity for significantly reduced compute cost. Combined with the multi-scale and attention techniques above, lightweight backbones can retain much of the small-object detection accuracy of larger models at a fraction of the inference cost, which matters directly for the real-time edge deployments covered later in this guide.
The latest versions of major detection frameworks (YOLOv8 through the YOLO11 generation, RT-DETR) incorporate several of these techniques natively, reducing the custom engineering required to achieve competitive small-object performance out of the box. Fine-tuning on a representative dataset remains necessary regardless of base architecture choice.
SAHI (Slicing Aided Hyper Inference) is a generic, model-agnostic technique that improves small-object detection accuracy by slicing the input image into overlapping tiles, running inference on each tile independently at full resolution, and merging the per-tile detections back into global image coordinates with a non-maximum suppression step to remove duplicates from the overlapping regions.
The core insight: a small object that would be a few pixels after resizing a full 4K image down to a model's 640x640 input becomes a much larger, more detectable object when the model instead processes a 640x640 crop from the original full-resolution image. SAHI recovers the detail lost at full-frame resolution by ensuring the model never has to downsample away the small object's pixel information in the first place.
SAHI increases object detection AP by 6.8%, 5.1%, and 5.3% for FCOS, VFNet, and TOOD detectors respectively on the VisDrone and xView aerial datasets, using inference-only slicing with no model retraining (Akyon et al., 2022). When combined with slicing-aided fine-tuning, where the training data is also sliced to better match the inference-time distribution, the cumulative improvement reaches 12.7%, 13.4%, and 14.5% AP for the same three detectors. In industrial defect inspection applications, SAHI has demonstrated roughly 2x improvement in small-defect detection rates over standard full-frame inference (semiconductor defect detection research, 2023).
Standard full-frame inference is the baseline here: no accuracy gain to speak of, but it runs at 1x cost and needs no retraining.
Image pyramid, or multi-resolution inference, buys a moderate bump in small-object accuracy. That comes at 2 to 4x the inference cost depending on how many scales you test, and it doesn't require retraining either.
SAHI used purely at inference delivers a real jump: 6.8 to 14.5% AP gain on small objects. The catch is cost, which climbs 4 to 16x depending on slice count, though no retraining is needed.
Pairing SAHI with slicing-aided fine-tuning pushes accuracy even further, the highest gain on this list, at the same 4 to 16x inference cost. This one does require retraining, so budget for that step.
Adaptive slicing, known as ASAHI, is the interesting middle ground: high accuracy gains with lower compute overhead, running 20 to 25% faster than fixed SAHI. Retraining is optional here, which makes it a flexible choice depending on how much engineering time you want to invest upfront.
The real-time trade-off is direct and unavoidable: SAHI multiplies the number of inference passes per frame by the number of slices. A 4-slice configuration roughly quadruples inference cost relative to single-pass inference on the same hardware; a 12-slice configuration roughly increases it twelvefold. On an edge compute budget with a fixed frame-rate requirement, slice size, overlap percentage, and slice count are the levers that trade directly against achievable frame rate.
Larger slices reduce slice count (lower compute cost) but recover less detail for very small objects. Higher overlap percentage reduces the risk of an object being split across a slice boundary (which can cause a missed detection or a poorly localized partial detection) but increases the redundant computation and the post-processing burden of merging duplicate detections. A fixed slice size, as used in the original SAHI implementation, produces inconsistent redundant computation across images of varying resolution. Adaptive slicing approaches (ASAHI) that scale slice count to image resolution rather than using a fixed slice dimension reduce computational overhead by 20–25% while maintaining or improving detection accuracy across both small and medium object categories (recent adaptive slicing research, 2026).
For post-processing, several merging methods handle the redundant detections produced across overlapping slices: standard NMS (Non-Maximum Suppression), Soft-NMS (which decays confidence rather than hard-suppressing overlapping boxes), NMM (Non-Maximum Merging, which merges rather than discards overlapping boxes), and Greedy NMM. The choice affects both accuracy and the additional latency this merging step adds to the pipeline, which matters on a real-time edge budget where every millisecond is accounted for.
A single video frame gives a detector one chance to find a small object. Video gives the system many chances across consecutive frames, and that temporal information is underused in most small-object detection pipelines.
The core idea: motion and temporal consistency can confirm a faint small-object detection that a single frame would miss or reject as a low-confidence false positive. A weapon partially visible for one frame, then occluded, then visible again three frames later is a stronger signal than any single frame in isolation. A model evaluating frames independently discards this information; a model that incorporates temporal context can recover it.
Running a lightweight object tracker (Kalman filter-based, or a learned tracker like ByteTrack) alongside per-frame detection allows the system to maintain a track for an object even through frames where the per-frame detector's confidence drops below threshold. If an object was confidently detected in frames N and N+2, the tracker can interpolate or confirm its presence in frame N+1 even if the detector alone would have missed it. This is particularly valuable for moving objects that pass through partial occlusion: a person briefly hidden behind a pillar maintains a track rather than generating a false negative followed by a new, disconnected detection.
Aggregating confidence scores across a sliding window of frames (rather than acting on any single frame's output) reduces the false positive rate from transient noise while improving recall for objects with marginal per-frame confidence. A small object detected with 0.4 confidence in three consecutive frames is a more reliable signal than a single 0.6-confidence detection with no temporal corroboration.
The trade-off is latency and complexity: temporal approaches inherently introduce some delay (waiting for confirming frames) and require maintaining state across the video stream, which adds engineering complexity relative to stateless per-frame inference. For systems where real time performance with minimal added latency is critical (a collision avoidance system, for example), the temporal window must be kept short, and the architecture must be designed so the added latency does not violate the safety timing budget.
Higher image resolution in training data directly improves small-object detection by preserving the pixel information the model needs to learn discriminative features. Beyond resolution, targeted augmentation strategies for small objects include copy-paste augmentation (pasting small object crops from other images into new backgrounds, multiplying the effective number of small-object training examples), mosaic augmentation (combining four training images into one, which effectively creates smaller relative object sizes and forces the model to learn small-object features), and scale jittering (randomly resizing training images to expose the model to a wider range of relative object sizes than the original dataset provides).
Standard IoU-based losses penalize small-object localization error disproportionately, as covered earlier. Shape-IoU and similar refined loss formulations incorporate box shape and scale awareness, producing training signals that are more proportionate across the full range of object sizes present in a dataset, which improves convergence specifically for the small-object tail of the size distribution.
Training on specific, representative datasets improves small-object detection over general-purpose datasets. A model intended for transit CCTV small-object detection (weapons, unattended bags) benefits far more from a training set built from representative transit camera footage than from a general object detection dataset, even a large one, because the object scale distribution, camera angle, and background clutter characteristics of the target deployment are what the model actually needs to learn.
Reducing false positives from duplicate or near-duplicate bounding boxes (a common artifact of multi-scale and slicing-based inference approaches) requires careful NMS threshold tuning, calibrated against the IoU distribution of true duplicate detections in the specific deployment's object size and density profile.
Standard evaluation practice for object detection understates small-object performance unless metrics are explicitly stratified by object size.
Intersection over Union (IoU) measures bounding-box overlap between a predicted box and the ground truth box, the foundational metric for determining whether a detection counts as a true positive. As established earlier, IoU thresholds often misclassify small-object predictions as false negatives even when the detection correctly identified and roughly localized the object, because the relative penalty for a fixed pixel error scales inversely with object size.
mAP evaluates performance across categories by averaging precision-recall performance, typically at IoU thresholds from 0.5 to 0.95. Reported as a single aggregate number, mAP can mask substantial differences in small-object versus large-object performance, since large objects dominate most benchmark datasets numerically and therefore dominate the aggregate score.
Report AP-small as a distinct metric, following the COCO convention of partitioning ground truth objects into small (under 32x32 pixels), medium, and large size buckets and computing AP independently within each bucket. This is the single most important evaluation practice change for any team building small-object detection systems: a model with strong overall mAP can have materially weaker AP-small, and that gap is invisible without explicit size stratification.
Given the metric distortion described above, evaluating tiny-object performance at a lower IoU threshold (0.3 instead of the standard 0.5) more accurately reflects operational usefulness for use cases where detecting the object's presence matters more than pixel-precise localization, such as flagging a suspicious small object for human review.
Precision-recall curves visualize the trade-off between false positives and false negatives across the full range of confidence score thresholds, letting a team select an operating threshold appropriate to the deployment's tolerance for each error type. For safety applications, recall (catching every true positive) is typically prioritized over precision (minimizing false alarms), within the constraint that excessive false positives degrade operator trust.
Ground truth labels for small objects are themselves error-prone: human annotators miss small objects more often than large ones during dataset labeling, and incorporating human verification passes specifically targeting small-object regions improves both the training data quality and the reliability of evaluation metrics computed against that ground truth.
Collect training data representative of the actual deployment environment and object scale distribution. Apply human verification specifically to small-object annotations, since these are most prone to labeling errors and omissions.
Use a feature pyramid architecture with dedicated small-object detection heads. Apply targeted augmentation (copy-paste, mosaic, scale jittering) to expand effective small-object training data volume. Consider slicing-aided fine-tuning if SAHI will be used at inference time, since training on slices better matches the inference-time data distribution.
Benchmark the trained model with and without SAHI slicing on a held-out test set, measuring AP-small specifically. This evaluation determines whether the accuracy gain from SAHI justifies its compute cost for the specific deployment.
Quantize and prune the model for the target edge hardware before finalizing the deployment configuration. On a constrained device, the SAHI tiling strategy and the model size must fit the latency budget together: a larger slice count combined with an unoptimized model will exceed most real-time edge budgets. See model quantization for edge deployment and edge AI hardware for in-vehicle systems for the hardware selection and optimization process.
The computational resources required for SAHI-based inference must be sized against the actual deployment hardware, not a development workstation. A model and slicing configuration that achieves 30 FPS on a development GPU may achieve a fraction of that on an embedded NPU, requiring slice count or model size adjustments to meet the real-time requirement.
InTechHouse case study: Small object detection in transit CCTV with PESA
InTechHouse implemented a small object detection pipeline for a transit operator's onboard CCTV system, developed in partnership with PESA, targeting safety-critical objects on the order of a few pixels in a crowded cabin environment: small bags, partially concealed objects, and individuals at the edge of camera coverage.
Full-frame inference on the deployment hardware missed a significant fraction of small-object instances at the resolution required for real-time processing. The team implemented a tuned slicing strategy: a moderate slice count selected specifically to fit within the onboard compute budget while still recovering sufficient detail for the smallest target objects, with overlap calibrated to avoid splitting objects across slice boundaries in the camera's typical object size range.
Staying within the real-time budget required careful balance: the slicing configuration was benchmarked directly on the target embedded hardware, not a development workstation, and the final configuration traded some theoretical maximum accuracy for a slice count that reliably sustained the required frame rate. The resulting pipeline materially improved small-object recall over the full-frame baseline while operating within the latency constraints of the onboard safety system.
The YOLO family (YOLOv8 through YOLO11) for the best balance of real-time speed and small-object accuracy when combined with the techniques above; RT-DETR for transformer-based detection with competitive small-object context modeling; FCOS, VFNet, and TOOD as anchor-free alternatives that have been directly benchmarked with SAHI in published research.
The open-source SAHI framework (github.com/obss/sahi) integrates directly with Detectron2, MMDetection, and YOLOv5/v8, providing the slicing, inference, and merging pipeline without requiring custom implementation. This is the practical starting point for any team evaluating slicing-aided inference.
FiftyOne provides dataset visualization, quality analysis, and model evaluation tooling well suited to identifying small-object-specific failure modes in both training data and model predictions. Roboflow supports dataset annotation, augmentation pipeline configuration, and model training workflows with built-in support for small-object-focused augmentation strategies.
COCO remains the standard general-purpose object detection benchmark, with a well-established small/medium/large size stratification convention. VisDrone, an aerial imagery dataset with a high density of genuinely small objects (vehicles and pedestrians from drone altitude), and xView, a satellite imagery dataset for remote sensing applications, are the standard benchmarks used in published SAHI research and remain the most relevant public benchmarks for teams evaluating small-object detection approaches against published results.
Active research directions for small object detection include adaptive feature fusion architectures that adjust fusion weighting based on the specific object size distribution present in a given image rather than using fixed fusion weights, and lightweight fusion modules designed specifically for edge deployment that capture most of the accuracy benefit of full FPN architectures at a fraction of the compute cost. Semi-supervised learning approaches that can improve small-object detection performance using partially labeled data are also an active area, relevant because comprehensively labeling every small object instance in a large training set is expensive and error-prone, as discussed in the human verification section above.
The state of the art in small object detection continues to shift toward adaptive, resolution-aware inference strategies (like ASAHI) that reduce the brute-force compute cost of fixed slicing approaches while retaining most of the accuracy benefit, an important direction for any team operating under real-time edge compute budgets.
For production deployment, three priorities consistently determine success: preserve multi-scale features throughout the model architecture rather than collapsing to a single feature resolution; use SAHI or an adaptive slicing variant when the deployment's object scale distribution and compute budget justify the added inference cost; and evaluate with small-object-specific metrics (AP-small, lower IoU thresholds for tiny-object review) rather than relying on aggregate mAP, which can hide exactly the failure mode that matters most for safety-critical small-object applications.
To discuss a small object detection pipeline for your specific deployment, including hardware sizing and slicing configuration for your real-time budget, talk to the InTechHouse team.
Small object detection is the computer vision task of identifying and localizing objects that occupy a small portion of an image, typically under 32x32 pixels by the COCO benchmark convention. Standard object detectors handle these objects poorly because downsampling layers in the network discard most of a small object's limited pixel information before the detection head ever processes it, making specialized techniques (feature pyramids, slicing-based inference, targeted training data) necessary for reliable performance.
SAHI (Slicing Aided Hyper Inference) is a model-agnostic inference technique that slices an input image into overlapping tiles, runs object detection on each tile at full resolution, and merges the resulting detections back into the original image's coordinate system. By avoiding the resolution loss that comes from resizing a large image down to a model's standard input size, SAHI recovers detail for small objects that would otherwise be lost, with published results showing AP improvements of 6.8% to 14.5% depending on the detector and whether slicing-aided fine-tuning is also applied.
Small objects carry inherently less visual information than large objects, and that limited information is further degraded as it passes through a neural network's downsampling layers, often disappearing or becoming indistinguishable from background noise by the time the detection head processes it. Compounding factors include occlusion (which removes a disproportionate share of a small object's already-limited visible area), cluttered backgrounds (which reduce signal-to-noise ratio), and training datasets that are dominated by medium and large object instances, giving models limited exposure to learn robust small-object features.
The two complementary approaches are slicing-based inference (SAHI) to recover spatial detail lost at standard inference resolution, and temporal information from consecutive video frames to confirm low-confidence detections that a single frame would miss. Tracking-assisted detection maintains object identity across frames even through brief occlusion or detection confidence dips, and temporal smoothing aggregates confidence across a sliding window of frames to improve both recall and false-positive suppression relative to frame-independent inference.
Yes, but it requires deliberately balancing slicing configuration against the available compute budget. SAHI's accuracy improvement comes at the cost of multiplying inference passes per frame, which on constrained edge hardware can quickly exceed a real-time frame rate budget if slice count and model size are not jointly optimized. Practical deployments use adaptive slicing strategies, lightweight model backbones, and hardware-aware quantization to fit the slicing-based pipeline within the target frame rate on the deployment's specific edge AI compute unit, validated through direct benchmarking on that hardware rather than a development workstation.
An expert in Artificial Intelligence, professor and researcher, who has authored numerous scientific publications and led international projects focused on AI, machine learning, and data-driven systems.
His work connects academic research with industrial applications, applying advanced AI models to practical challenges across sectors such as defense, telecommunications, smart industry, and cybersecurity. He has extensive experience in designing and implementing intelligent systems in complex, high-demand environments.
In addition to his technical work, Prof. Andrysiak shares insights on AI trends and applications as a speaker, mentor, and author, contributing to discussions on the role of AI in modern technology and digital transformation.
This initial conversation is focused on understanding your product, technical challenges, and constraints.
No sales pitch - just a practical discussion with experienced engineers.
Share a few details about your product and context. We’ll review the information and suggest the most appropriate next step.
