From DETR to YOLOv8: Transformer Evolution in Object Detection

Introduction

Object detection is one of the most fundamental tasks in computer vision. From the R-CNN series to YOLO, two-stage and single-stage detectors have continuously evolved over the past decade. However, the emergence of DETR in 2020 completely changed the game — it was the first to introduce Transformer architecture into object detection, replacing traditional anchor-based methods with an end-to-end approach.

This article starts from DETR's core ideas, progressively introduces its evolution, and shares the challenges and solutions I encountered in engineering practice.

DETR's core contribution is transforming object detection from "heuristic design" to "end-to-end learning" — a paradigm-level revolution.

DETR Architecture Analysis

DETR (Detection Transformer) models object detection as a set prediction problem. It uses the Transformer encoder-decoder structure with the Hungarian matching algorithm to directly predict a fixed number of bounding boxes, completely eliminating anchors and NMS post-processing.

The overall architecture consists of three core components:

• CNN Backbone — Uses ResNet to extract multi-scale feature maps, encoding input images into high-dimensional semantic representations
• Transformer Encoder — Establishes global dependencies within feature maps through self-attention, capturing long-range context
• Transformer Decoder — Uses learnable object queries with cross-attention to encoder output, directly producing prediction results

However, original DETR has two major issues: slow training convergence (requiring 500 epochs) and poor small object detection. Deformable DETR introduced deformable attention, focusing attention on key points, improving convergence speed by 10x while significantly boosting small object detection performance.

Attention Mechanism Optimization

The core computational bottleneck of Transformers lies in the attention mechanism. Standard self-attention has O(n²) complexity, which is prohibitively expensive for high-resolution feature maps. In engineering practice, we adopted the following optimization strategies:

• Deformable Attention — Focuses on only a few key sampling points, reducing complexity from O(n²) to O(n while maintaining detection accuracy
• Multi-Scale Attention — Performs attention computation across different resolution feature maps, balancing detection of large and small objects
• Linear Attention — Uses kernel approximation to replace softmax attention with linear attention, further accelerating in specific scenarios
• Flash Attention — Uses IO-aware tiled computation to reduce GPU HBM access frequency, improving practical runtime efficiency

Key insight: deformable attention reduces global attention complexity from O(n²) to O(n), making real-time inference possible. In actual deployment, we observed only 0.3 mAP drop in detection accuracy, but a 4x improvement in inference speed.

Model Optimization: TensorRT & ONNX

Porting DETR models from PyTorch to production requires multiple optimization stages. Here is our complete optimization pipeline:

• ONNX Export — Uses torch.onnx.export to export models to ONNX format, handling dynamic axes and custom operators
• Graph Optimization — Uses ONNX Simplifier for constant folding, redundant node elimination, and operator fusion
• TensorRT Quantization — Applies INT8 quantization with calibration datasets to ensure acceptable accuracy loss
• CUDA Graph Optimization — Uses CUDA Graph to capture inference computation graphs, eliminating kernel launch overhead

The optimized model achieves 12ms end-to-end inference latency on NVIDIA T4 GPU (640x640 input), a 3.75x speedup over the original PyTorch model.

PyTorch Inference Code

Below is the complete inference pipeline code, including model loading, preprocessing, inference, and post-processing:

Benchmark Results

We conducted comprehensive benchmarks on the COCO 2017 validation set, comparing performance across different models and deployment approaches:

• Original DETR (ResNet-50) — 42.0 mAP, 45ms latency (V100)
• Deformable DETR (ResNet-50) — 46.2 mAP, 28ms latency (V100)
• Deformable DETR + TensorRT FP16 — 46.0 mAP, 15ms latency (T4)
• Deformable DETR + TensorRT INT8 — 45.5 mAP, 12ms latency (T4)
• YOLOv8-L (baseline) — 52.9 mAP, 8ms latency (T4)

While DETR series still slightly trails YOLO in pure latency metrics, it clearly excels in tasks requiring precise spatial reasoning and complex scene understanding (e.g., occlusion handling, dense objects).

Selection advice: For precision-first, object-dense industrial scenarios, Deformable DETR + TensorRT is recommended; for latency-sensitive real-time video streams, YOLO remains the more pragmatic choice.

Engineering Deployment

Deploying DETR series models to production faces several key challenges:

• Dynamic Batch — Different video streams have varying frame rates and resolutions, requiring dynamic batch size support via TensorRT's dynamic shape mechanism
• Model Hot-swap — Designed an A/B testing framework supporting online detection model switching without service interruption
• Multi-stream Concurrency — Uses CUDA Stream for parallel inference across multiple video streams, fully utilizing GPU compute resources
• Monitoring & Alerting — Integrated Prometheus + Grafana for inference latency, GPU utilization, and detection accuracy drift monitoring

Ultimately, through TensorRT INT8 quantization + CUDA Graph optimization, we reduced inference latency from 45ms to 12ms, meeting real-time video stream processing requirements. The system processes over 5 million frames daily with GPU utilization stable above 75%.

Summary & Outlook

Transformer applications in object detection are just beginning. From DETR to Deformable DETR, and the latest DINO and Co-DETR, this field is rapidly evolving. Future directions worth watching:

• Vision-Language Fusion — Combining CLIP and other vision-language models for open-vocabulary object detection
• Self-Supervised Pre-training — Reducing dependence on labeled data, lowering model training costs
• Edge Deployment — Through knowledge distillation and model compression, pushing DETR to edge devices

For engineering teams, the key is finding the balance between model accuracy and inference efficiency. Through careful architecture design and engineering optimization, Transformer-based detectors can absolutely achieve real-time performance in production environments.