Ziwei Liu
Jun Luo
ABSTRACT
Optical camera communication (OCC) enabled by LED and embedded cameras has attracted extensive attention, thanks to its rich spectrum availability and ready deployability. However, the close interactions between OCC and the indoor spaces have created two major challenges. On one hand, the stripe pattern incurred by OCC may greatly damage the accuracy of image-based object recognition. On the other hand, the patterns inherent to indoor spaces can significantly degrade the decoding performance of reflected OCC. To this end, we propose CORE-Lens as a pipeline to make the mutual interference transparent to existing OR and OCC algorithms. Essentially, CORE-Lens treats the two challenges as two sides of a signal mixture issue: the signals transmitted by OCC get mixed with background images so well that their features become entangled. Consequently, CORE-Lens exploits the idea of disentangled representation learning to separate the mixed signals in the feature space: while the GAN-reconstructed clean background images are used to perform object recognition, OCC decoding is conducted on the residual of the original image after subtracting the reconstructed background. Our extensive experiments on evaluating the real-life performance of CORE-Lens evidently demonstrate its superiority over conventional approaches.
- Navid Bani Hassan, Zabih Ghassemlooy, Stanislav Zvanovec, Mauro Biagi, Anna Maria Vegni, Min Zhang, and Pengfei Luo. 2019. Non-Line-of-Sight MIMO Space-Time Division Multiplexing Visible Light Optical Camera Communications. OSA/IEEE Journal of Lightwave Technology 37, 10 (2019), 2409--2417.Google Scholar
Cross Ref
- Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (2013), 1798--1828.Google ScholarDigital Library
- David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. 2017. Variational Inference: A Review for Statisticians. Journal of the American statistical Association 112, 518 (2017), 859--877.Google ScholarCross Ref
- Léon Bottou. 2012. Stochastic Gradient Descent Tricks. In Neural Networks: Tricks of the Trade. Springer, 421--436.Google Scholar
- Gary Bradski and Adrian Kaehler. 2000. OpenCV. Dr. Dobb's Journal of Software Tools 3 (2000), 2.Google Scholar
- Chao Cai, Rong Zheng, and Jun Luo. 2022. Ubiquitous Acoustic Sensing on Commodity IoT Devices: A Survey. IEEE Communications Surveys & Tutorials 24, 1 (2022), 432--454.Google ScholarCross Ref
- Shao-Qi Chen, Xue-Fen Chi, and Te-Yu Li. 2021. Non-line-of-sight Optical Camera Communication Aided by a Pilot. Opt. Lett. 46, 14 (2021), 3348--3351.Google ScholarCross Ref
- Weixuan Chen and Daniel McDuff. 2018. DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. In Proc. of the 15th IEEE ECCV. 349--365.Google ScholarDigital Library
- Zhe Chen, Chao Cai, Tianyue Zheng, Jun Luo, Jie Xiong, and Xin Wang. 2021. RF-Based Human Activity Recognition Using Signal Adapted Convolutional Neural Network. IEEE Trans. on Mobile Computing (2021), 1--13.Google ScholarCross Ref
- Zhe Chen, Tianyue Zheng, Chao Cai, and Jun Luo. 2021. MoVi-Fi: Motion-robust Vital Signs Waveform Recovery via Deep Interpreted RF Sensing. In Proc. of the 27th ACM MobiCom. 392--405.Google ScholarDigital Library
- Chi-Wai Chow, Yang Liu, Chien-Hung Yeh, Yun-Han Chang, Yun-Shen Lin, Ke-Ling Hsu, Xin-Lan Liao, and Kun-Hsien Lin. 2021. Display Light Panel and Rolling Shutter Image Sensor Based Optical Camera Communication (OCC) Using Frame-Averaging Background Removal and Neural Network. OSA/IEEE Journal ofLightwave Technology 39, 13 (2021), 4360--4366.Google ScholarCross Ref
- Yuanhao Cui, Fan Liu, Xiaojun Jing, and Junsheng Mu. 2021. Integrating Sensing and Communications for Ubiquitous IoT: Applications, Trends, and Challenges. IEEE Network 35, 5 (2021), 158--167.Google ScholarDigital Library
- Christos Danakis, Mostafa Afgani, Gordon Povey, Ian Underwood, and Harald Haas. 2012. Using a CMOS Camera Sensor for Visible Light Communication. In Proc. of IEEE GLOBECOM Workshops. 1244--1248.Google ScholarCross Ref
- CORE-Lens Dataset. 2022. https://github.com/XavierLiu888/Dataset-V1.Google Scholar
- Shuya Ding, Zhe Chen, Tianyue Zheng, and Jun Luo. 2020. RF-Net: A Unified Meta-Learning Framework for RF-Enabled One-Shot Human Activity Recognition. In Proc. of the 18th ACM SenSys. 517--530.Google ScholarDigital Library
- GigaDevice. 2022. GD32F330G8U6 - GD32 ARM Cortex-M4 Microcontroller. https://www.gigadevice.com/microcontroller/gd32f330g8u6/. Online; accessed 4 March 2022.Google Scholar
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Proc. of NIPS. 2672--2680.Google Scholar
- Jie Hao, Yanbing Yang, and Jun Luo. 2016. CeilingCast: Energy Efficient and Location-Bound Broadcast through LED-Camera Communication. In Proc. of the 35th IEEE INFOCOM. 1--9.Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. of the 29th IEEE/CVF CVPR. 770--778.Google ScholarCross Ref
- Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In Proc. of the 5th ICLR. 1--22.Google Scholar
- Ke-Ling Hsu, Yu-Chun Wu, Yu-Cheng Chuang, Chi-Wai Chow, Yang Liu, Xin-Lan Liao, Kun-Hsien Lin, and Yi-Yuan Chen. 2020. CMOS Camera Based Visible Light Communication (VLC) using Grayscale Value Distribution and Machine Learning Algorithm. Opt. Express 28, 2 (2020), 2427--2432.Google ScholarCross Ref
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks. In Proc. of the 30th IEEE/CVF CVPR. 4700--4708.Google ScholarCross Ref
- Huawei Device Co., Ltd. 2022. Huawei Mate 30 Pro. https://consumer.huawei.com/sg/phones/mate30-pro/. Online; accessed 4 March 2022.Google Scholar
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In Proc. of the IEEE CVPR. 1125--1134.Google Scholar
- Shuang Jiang, Zhiyao Ma, Xiao Zeng, Chenren Xu, Mi Zhang, Chen Zhang, and Yunxin Liu. 2020. SCYLLA: QoE-aware Continuous Mobile Vision with FPGA-based Dynamic Deep Neural Network Reconfiguration. In Proc. of the 39th IEEE INFOCOM. 1369--1378.Google ScholarDigital Library
- Cristo Jurado-Verdu, Victor Guerra, Vicente Matus, Jose Rabadan, and Rafael Perez-Jimenez. 2021. Convolutional autoencoder for exposure effects equalization and noise mitigation in optical camera communication. Opt. Express 29,15 (Jul 2021), 22973--22991.Google ScholarCross Ref
- Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. In Proc. of ICLR. 1--14.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet Classification with Deep Convolutional Neural Networks. Commun. ACM 60, 6 (2017), 84--90.Google ScholarDigital Library
- Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. 2018. Variational Inference of Disentangled Latent Concepts from Unlabeled Observations. In Proc. of the 6th ICLR. 1--16.Google Scholar
- Siddharth Krishna Kumar. 2017. On Weight Initialization in Deep Neural Networks. arXiv preprint arXiv:1704.08863 (2017).Google Scholar
- Hui-Yu Lee, Hao-MinLin, Yu-Lin Wei, Hsin-I Wu, Hsin-Mu Tsai, and Kate Ching-Ju Lin. 2015. Rollinglight: Enabling Line-of-Sight Light-to-Camera Communications. In Proc. of the 13th ACM MobiSys. 167--180.Google ScholarDigital Library
- Yun-Shen Lin, Yang Liu, Chi-Wai Chow, Yun-Han Chang, Dong-Chang Lin, Shao-Hua Song, Ke-Ling Hsu, and Chien-Hung Yeh. 2021. Z-Score Averaging Neural Network and Background Content Removal for High Performance Rolling Shutter based Optical Camera Communication (OCC). In Optical Fiber Communication Conference (OFC) 2021. F1A.4.Google ScholarCross Ref
- Hongbo Liu, Bo Liu, Cong Shi, and Yingying Chen. 2017. Secret Key Distribution Leveraging Color Shift Over Visible Light Channel. In Proc. of the IEEE CNS. 1--9.Google ScholarCross Ref
- Liqiong Liu and Lian-Kuan Chen. 2021. Li-poster: Real-time Non-line-of-sight Optical Camera Communication for Hand-held Smartphone Applications. In Proc. of OSA OFC. M1B.9.Google ScholarCross Ref
- Liqiong Liu, Yang Hong, and Lian-Kuan Chen. 2018. A Frame Averaging based Signal Tracing (FAST) Algorithm for Optical Camera Communications. Asia Communications and Photonics Conference (ACP) 2018, Su3D.3.Google ScholarCross Ref
- Xiaochen Liu, Pradipta Ghosh, Oytun Ulutan, BS Manjunath, Kevin Chan, and Ramesh Govindan. 2019. Caesar: Cross-Camera Complex Activity Recognition. In Proc. of the 17th ACM SenSys. 232--244.Google ScholarDigital Library
- Ziwei Liu, Lin Yang, Yanbing Yang, Rengmao Wu, Lei Zhang, Liangyin Chen, Die Wu, and Jun She. 2021. Improved Optical Camera Communication Systems using a Freeform Lens. Opt. Express 29, 21 (2021), 34066--34076.Google ScholarCross Ref
- Minghuang Ma, Haoqi Fan, and Kris M Kitani. 2016. Going Deeper into First Person Activity Recognition. In Proc. of the 29th IEEE CVPR. 1894--1903.Google ScholarCross Ref
- Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Soheil Feizi. 2022. A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes. In Proc. of the IEEE CVPR. 19087--19097.Google ScholarCross Ref
- Huynh Nguyen, Archan Misra, and Youngki Lee. 2016. LightSense: Exploiting Smart Bulbs for Practical Multimodal Localization. In Proc. of the 14th IEEE PerCom. 1--4.Google ScholarCross Ref
- Jingyi Ning, Lei Xie, Yi Li, Yingying Chen, Yanling Bu, Baoliu Ye, and Sanglu Lu. 2022. MoiréPose: Ultra High Precision Camera-to-Screen Pose Estimation based on Moire Pattern. In Proc. of the 28th ACM MobiCom. 1--14.Google ScholarDigital Library
- NVIDIA. 2022. GET SUPER POWERS GEFORCE RTX 2070 SUPER. https://www.nvidia.com/en-me/geforce/graphics-cards/rtx-2070-super/. Online; accessed 4 March 2022.Google Scholar
- Misa Ogura and Ravi Jain. 2022. FlashTorch. https://github.com/MisaOgura/flashtorch. Online; accessed 4 March 2022.Google Scholar
- A Pai, A Veeraraghavan, and A. Sabharwal. 2021. HRVCam: Robust Camera-based Measurement of Heart Rate Variability. J. Biomed Opt 26 (2021), 1--23.Google ScholarCross Ref
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. arXiv preprint arXiv:1912.01703 (2019).Google Scholar
- PyTorch. 2022. PyTorch Mobile. https://pytorch.org/mobile/home/. Online; accessed 4 March 2022.Google Scholar
- Nasir Saeed, Shuaishuai Guo, Ki-Hong Park, Tareq Y. Al-Naffouri, and Mohamed-Slim Alouini. 2019. Optical Camera Communications: Survey, Use Cases, Challenges, and Future Trends. Physical Communication 37 (2019), 100900.Google ScholarDigital Library
- Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv preprint arXiv:1312.6034 (2013).Google Scholar
- Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In Proc. of IEEE CVPR. 1--9.Google ScholarCross Ref
- Zhao Tian, Charles J. Carver, Qijia Shao, Monika Roznere, Alberto Quattrini Li, and Xia Zhou. 2020. PolarTag: Invisible Data with Light Polarization. In Proc. of the 21st HotMobile. 74--79.Google ScholarDigital Library
- UNW. 2022. UMW SI2310A N-Channel Power MOSFET. https://www.semiee.com/file/Source10/UMW-SI2310A.pdf. Online; accessed 4 March 2022.Google Scholar
- Tim Van Erven and Peter Harremos. 2014. Rényi Divergence and Kullback-Leibler Divergence. IEEE Transactions on Information Theory 60, 7 (2014), 3797--3820.Google ScholarCross Ref
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. Proc. of NIPS 30 (2017), 1--11.Google Scholar
- Purui Wang, Lilei Feng, Guojun Chen, Chenren Xu, Yue Wu, Kenuo Xu, Guobin Shen, Kuntai Du, Gang Huang, and Xuanzhe Liu. 2020. Renovating Road Signs for Infrastructure-to-Vehicle Networking: A Visible Light Backscatter Communication and Networking Approach. In Proc. of the 26th ACM MobiCom. 1--13.Google ScholarDigital Library
- Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y. Zhao. 2021. Backdoor Attacks Against Deep Learning Systems in the Physical World. In Proc. of the IEEE CVPR. 6202--6211.Google Scholar
- Yue Wu, Purui Wang, Kenuo Xu, Lilei Feng, and Chenren Xu. 2020. Turboboosting Visible Light Backscatter Communication. In Proc. of the ACM SIGCOMM. 186--197.Google ScholarDigital Library
- Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. 2021. Noise or Signal: The Role of Image Backgrounds in Object Recognition. In Proc. of the 9th ICLR. 1--28.Google Scholar
- Chenren Xu, Shuang Jiang, Guojie Luo, Guangyu Sun, Ning An, Gang Huang, and Xuanzhe Liu. 2022. The Case for FPGA-Based Edge Computing. IEEE Transactions on Mobile Computing 21, 7 (2022), 2610--2619.Google ScholarCross Ref
- Xieyang Xu, Yang Shen, Junrui Yang, Chenren Xu, Guobin Shen, Guojun Chen, and Yunzhe Ni. 2017. PassiveVLC: Enabling Practical Visible Light Backscatter Communication for Battery-Free IoT Applications. In Proc. of the 23rd ACM MobiCom. 180--192.Google ScholarDigital Library
- Yanbing Yang, Jie Hao, and Jun Luo. 2017. CeilingTalk: Lightweight Indoor Broadcast Through LED-Camera Communication. IEEE Trans. on Mobile Computing 16, 12 (2017), 3308--3319.Google ScholarCross Ref
- Yanbing Yang, Jie Hao, Jun Luo, and Sinno Jialin Pan. 2017. CeilingSee: Device-free occupancy inference through lighting infrastructure based LED sensing. In Proc. of the 15th IEEE PerCom. 247--256.Google ScholarCross Ref
- Yanbing Yang and Jun Luo. 2018. Boosting the Throughput of LED-Camera VLC via Composite Light Emission. In Proc. of the 37th IEEE INFOCOM. 315--323.Google ScholarDigital Library
- Yanbing Yang, Jun Luo, Chen Chen, Zequn Chen, Wen-De Zhong, and Liangyin Chen. 2021. Pushing the Data Rate of Practical VLC via Combinatorial Light Emission. IEEE Trans. on Mobile Computing 20, 5 (2021), 1979--1992.Google ScholarCross Ref
- Yanbing Yang, Jiangtian Nie, and Jun Luo. 2017. ReflexCode: Coding with Superposed Reflection Light for LED-Camera Communication. In Proc. of the 23rd ACM MobiCom. 193--205.Google ScholarDigital Library
- Zitong Yu, Wei Peng, Xiaobai Li, Xiaopeng Hong, and Guoying Zhao. 2019. Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-End Deep Learning Solution with Video Enhancement. In Proc. of the IEEE/CVF ICCV. 151--160.Google ScholarCross Ref
- Tianyue Zheng, Zhe Chen, Shujie Zhang, Chao Cai, and Jun Luo. 2021. MoRe-Fi: Motion-robust and Fine-grained Respiration Monitoring via Deep-Learning UWB Radar. In Proc. of the 19th ACM SenSys. 111--124.Google ScholarDigital Library
- Shilin Zhu, Chi Zhang, and Xinyu Zhang. 2017. Automating Visual Privacy Protection Using a Smart LED. In Proc. of the 23rd ACM MobiCom. 329--342.Google ScholarDigital Library
Index Terms
- CORE-lens: simultaneous communication and object recognition with disentangled-GAN cameras
Recommendations
Joint source channel decoding and iterative symbol combining with turbo trellis-coded modulation
A new joint source channel decoding (JSCD) scheme for turbo trellis-coded modulation is proposed which exploits symbol-a-priori probabilities to improve decoding performance. To obtain symbol-a-priori probabilities, we propose a new method which uses a ...
A survey of design and implementation for optical camera communication
Visible Light Communication (VLC) has emerged as a promising technology for wireless communication owing to advantages such as bandwidth, license, coexistence and security. Additionally, it was officially standardized by IEEE 802.15.7 in 2011. However, ...
Epipolar geometry for prism-based single-lens stereovision
In order to simplify the design and implementation of a stereo vision system, prism has been used to capture stereo images with a single camera. This kind of system not only provides advantages over traditional two-camera stereo, but also reduces the ...
Comments