GhostImage Camera Attack

Why would this happen?

This is essentially what people know as the lens flare effect, aka, ghost effect. Due to the imperfection of lenses (e.g., reflectance), light beams coming from the light source are not only refrected, but also reflected among lenses before they reach the CMOS image sensor. Theorectically, ghosts are always there but they are normally too weak to be visible.

All right... How successful is the attack?

To demonstrate the attack, we trained a neural network for recognizing US traffic signs. We vary the distance between the projector and the camera. The table below shows the attack success rates. Digital results are based on simulation, while perception results are experimental.

Did you do anything to overcome these issues?

Yes, we take into account the channel effects (including color distortion and auto-exposure control), and also take advantage of the non-robustness of the neural networks. Specifically, the channel model is able to mimic the channel effects in an end-to-end fashion, such that when given an input image, the model can output the image that is as if the image is projected by the projector and then captured by the camera.

Then how do the adversarial examples and channel model perform?

Pretty well. See the bar plots below, and also some examples of the optimal adversarial ghost patterns.

I'm sold! How do I cite this work?

Thanks! This work is accepted RAID 2020. The video of the coference presentation can be found here. You can cite the paper as follows. Also, the project is open-sourced.