Multicamera dataset enables low cost screen detection and pointing estimation
DABACO: A Multi-Camera Dataset and Benchmark for Screen Localization and Pointing Estimation
Computer Vision and Pattern RecognitionMultimedia
Summary
Detecting screens and where someone is pointing at them using cameras is important for making affordable interactive devices. The paper introduces the DABACO Dataset, which contains video clips from multiple inexpensive cameras set up on small computers. This dataset helps researchers and developers test and improve methods for finding screens and estimating pointing gestures, using real video rather than just pictures or synthetic data. The authors also provide tools to evaluate how well these methods work, along with some example approaches as a starting point.
What this means in practice
- •For embedded device engineers: Develop and test camera-based screen detection and pointing algorithms using realistic multicamera video data and evaluation tools for low-cost platforms.
- •For interactive hardware developers: Build affordable interactive devices that rely on camera-based pointing and screen detection with reliable benchmarks from real video data on embedded hardware.
Authors
Óscar Gómez-Cárdenes, José Gil Marichal-Hernández, Juan Manuel Martín-Doñas
Abstract
Screen localization and pointing estimation are key to low-cost interactive devices. Yet developing and evaluating these algorithms requires realistic data: synthetic captures cannot fully reproduce the optical distortion, rolling shutter, motion blur, and display processing of a physical acquisition, and most existing datasets provide static images rather than the video needed to assess continuous pointing. In this paper, we introduce the DABACO Dataset. Developed within the DABACO (Dispositivo Apuntador de BAjo COste, or Low-Cost Pointing Device) project, this dataset supports the development and evaluation of screen detection and camera-based pointing algorithms for embedded systems. It comprises video sequences captured with multiple low-cost camera sensors, including monochrome global-shutter and color rolling-shutter modules, on embedded platforms such as the Raspberry Pi 4B and ESP32-S3. We present a systematic annotation pipeline combining temporary visual watermarking, optical-flow tracking, manual verification, and marker removal. Both the original marked captures and the marker-free images, together with explicit corner annotations and modification masks, are released to support auditing and the study of potential reconstruction bias. In addition, we release an open-source evaluation toolkit with two reference baselines: a classical screen-detection pipeline based on edge and contour geometry, and a closed-vocabulary, off-the-shelf YOLOv8 segmentation model evaluated without dataset-specific training. Both are evaluated on a general-purpose computer using Intersection over Union, corner error, and pointing error, establishing initial reference results for future embedded implementations. Overall, DABACO addresses a gap in existing resources and is intended to support the development and evaluation of new low-cost screen-localization and pointing systems.