Yang Liu✉, Gregory W. Wornell, William T. Freeman, and Frédo Durand✉. Science Advances, 10 (2), eadj3608, 2024. [project] [code] [video] [highlight] [doi] [bibtex]
Embedded sensors in smart devices pose privacy risks, often unintentionally leaking user information. We investigate how combining an ambient light sensor with a device display can capture an image of touch interaction without a camera. By displaying a known video sequence, we use the light sensor to capture reflected light intensity variations partially blocked by the touching hand, formulating an inverse problem similar to single-pixel imaging. Due to sensors’ heavy quantization and low sensitivity, we propose an inversion algorithm involving an $\ell_p$-norm dequantizer and a deep denoiser as natural image priors, to reconstruct images from the screen’s perspective. We demonstrate touch interactions and eavesdropping hand gestures on an off-the-shelf Android tablet. Despite limitations in resolution and speed, we aim to raise awareness of potential security/privacy threats induced by the combination of passive and active components in smart devices, and promote the development of ways to mitigate them.
Initial Poster video at ICCP 2021.
https://www.youtube.com/embed/PoEFci3csLc
2022.04
https://www.youtube.com/embed/8-cyA-LJB4A
@article{Liu24PDI,
author = {Liu, Yang and Wornell, Gregory W. and Freeman, William T. and Durand, Fr{\\'e}do},
title = {Imaging Privacy Threats from an Ambient Light Sensor},
journal = {Science Advances},
year = {2024},
month = {1},
volume = {10},
number = {2},
pages = {eadj3608},
url = {<https://doi.org/10.1126/sciadv.adj3608>},
publisher = {AAAS},
doi = {10.1126/sciadv.adj3608},
type = {Journal Article}
}