ARLIS, through its INSURE academic consortium, is supporting a research project to develop context-aware multimodal information retrieval systems - a next-generation capability that could transform how intelligence and security professionals access and analyze complex data.
The project, led by Dr. Alan McMillan at the University of Wisconsin and supported by Dr. Michael Brundage at ARLIS, focuses on a cutting-edge AI approach called Retrieval-Augmented Generation (RAG). RAG helps AI systems pull in new, up-to-date information during tasks, something standalone models can’t do. While RAG has mostly been used with text, this project aims to combine it with large multimodal models (LMMs) that can handle all types of data at once.
This project, entitled Context-Aware Multimodal Information Retrieval System, could be a game-changer for intelligence work. Today, many analysts are forced to jump between systems or manually interpret complex data, slowing down the decision-making process. By integrating RAG and LMMs, the team hopes to build tools that deliver faster, more relevant, and more complete insights - right when they’re needed.
The project will focus on three areas where this technology can have an immediate impact: keeping naval equipment mission-ready, analyzing historical government documents for declassification, and helping interpret medical images in fast-moving environments.
“This is about giving decision-makers better tools to understand the full picture faster,” said Dr. Brundage. “We’re aiming to create AI systems that can search across multiple formats and actually respond to what a human user needs in the moment.”
The 12-month effort will explore how to build these tools using open-source AI models, measure their performance, and test them in real-world security and defense scenarios. It also emphasizes the importance of human oversight, making sure AI tools work alongside people, not instead of them.
“In today’s national security landscape, we don’t just need more data, we need smarter ways to connect and understand it across formats and contexts,” said Dr. McMillan. “By developing practical multi-modal systems to do RAG with LMMs, we’re aiming to build systems that help analysts cut through complexity and ultimately do their jobs more effectively.”