Light and Sound for Improved Artificial Vision

This project proposes to combine vision and ultrasound sensors together with Artificial Intelligence (AI) to perform robust and improved multimodal detection and identification of objects. The proposed approach is a multimodal perception combining light and sound with the aim of achieving enhanced resolution and accuracy, improved adaptability and robustness, contextual understanding, and improved field of view. The vision sensor is based on a newly proposed custom innovative pixel architecture allowing not only to detect pixel illumination but to integrate in-pixel AI functionalities such as convolutional neural network with limited circuit overhead, hence reducing computational and storage requirements. The pixel output constitutes a pre-processed signal that is fed to an on-chip AI processor integrated side-by-side with the pixel array with the objective of performing detection and identification. The ultrasound sensor is based on a newly proposed air-coupled meta-surface based Piezoelectric Micromachined Ultrasound Transducer (META-PMUT) based on acoustic meta-materials concepts, enabling beam collimation and amplification, improving detection limits and directionality as well as enhancing resolution and depth measurement in various applications. The output reflected from objects will be used as signature signals to perform object detection and identification using the AI processing unit. The fusion of these sensors augmented with AI processing constitutes a very powerful platform capable of performing robust object detection and identification in any lighting condition, making them ideal for a variety of applications including but not limited to obstacle avoidance, identification and detection, robotics, security and surveillance, virtual reality, drones, healthcare, agriculture, industrial automation, and autonomous systems.

Hamad Bin Khalifa University (HBKU)