End-to-end multicamera event-image stitching and object detection on the edge

Wide Field-of-View (FoV) and high-resolution imaging are crucial capabilities for visual surveillance. However, they impose significant computational demands, posing significant challenges for embedded platforms targeting real-time performance. Event-based cameras help in curtailing computational effort by capturing only brightness changes at the pixel level, thereby reducing data volume while enhancing temporal resolution and dynamic range. Moreover, their inherent sensitivity to object edges improves the visibility of camouflage patterns. This work presents a real-time, wide-FoV event-based vision system based on a dual-camera setup with onboard object detection and tracking. The proposed system implements a low-latency end-to-end pipeline encompassing data capture, event-stream processing, image stitching, object detection, and tracking. Custom CUDA kernels are developed for efficient event processing and stitching, while a YOLOv8-based detector is evaluated in combination with multiple tracking algorithms. With the dual-camera configuration generating up to 20 million events per second per camera, end-to-end object detection and tracking is achieved in under 30 milliseconds (30-40 FPS) on an NVIDIA Jetson AGX Orin. The system demonstrates linear scalability, establishes a baseline for real-time multi-camera event-based vision-at-the-edge platforms, and provides the first embedded implementation exploiting event-based stitching and detection.