2D-3D reasoning and augmented reality

CVT’s work in change detection supports deployed improvised explosive devices (IEDs). These algorithms look at multiple passes of video data to detect change signatures of buried roadside IEDs. The recent integration of machine learning-based road-detection methods has significantly improved our change detection performance. SRI is developing novel anomaly detection and anomaly-guided change detection methods for next-generation systems. Specifically, CVT is developing a transformer-based joint spatiotemporal model encompassing multiple space and time resolutions. Transformer networks enable retention of properties of various data modalities, namely geography, weather, seasonal variations, knowledge of typical events and activities of interest, thus providing modularity that enables multimodality, scalability, and explainability.

CVT has developed an end-to-end pipeline that fuses multi-modal data in deep embedding space for specific tasks—such as target detection and recognition—by directly optimizing target metrics and learning the optimal contribution/control of each mode to the results. This pipeline has been applied to different modalities, including electro-optic/infrared images, hyperspectral imaging, and LIDAR/RADAR data. CVT has also incorporated scene-contextual information to further improve performance of the target task.

CVT created a new approach to continually acquire, fine-tune and transfer knowledge to optimize tasks such as target classification. Our approach advances state-of-the-art transfer learning and continual learning methods to create an in-situ algorithm-training environment to streamline the training of classifiers to new, unknown sensor data in real time.