ARLIS, through its Intelligence and Security University Research Enterprise (INSURE) academic consortium, is funding a study with Arizona State University (ASU) researchers to identify factors that increase the adoption of AI and autonomy for intelligence tasks.
The project, AI Technology Evaluation Study via Testbed (AI TEST), hopes to validate an evaluation tool that measures “trustability” in AI-enabled systems. As the volume and complexity of information in intelligence and security domains continue to grow, emerging AI technologies must align with decision-makers’ needs and adapt to dynamic operational environments. One way for AI systems to align to these needs and dynamic contexts is by assessing whether human operators have appropriate trust in the technologies and whether that relationship can be sustained and fostered over time.
Led by Dr. Erin Chiou, Associate Professor of Human Systems Engineering at ASU, and Dr. Jamie Gorman, Professor of Human Systems Engineering at ASU, and supported by Dr. Julie Marble, Director of the Intelligent Human-Machine Systems Division at ARLIS, the research team is focusing on a core concept they call responsivity – a measure of how well an AI system can anticipate human needs, adapt to changing conditions, and support collaborative, goal-aligned interaction. Unlike more commonly used metrics that assess only reliability or usability, responsivity criteria offer an all-inclusive estimate of how AI systems perform as dynamic systems with people.
“I am very pleased ARLIS can support this project. Operationalizing concepts that increase people’s willingness to rely on AI and autonomous systems enables engineers to develop better systems that people adopt quickly with less training,” said Dr. Julie Marble. “This project helps us define what makes an AI system feel reliable and helpful in real-world, high-pressure situations, which in turn enables more rapid adoption of AI and technologies for national security. In addition, operationalizing these concepts will allow faster development and deployment of tools, which require less training, and increase the efficiency and throughput of intelligence analysis.”
The ASU team is adapting their previously developed, interactive testing environment called MASTOPIA, a testbed that leverages OpenAI’s GPT models, RAG-based architecture, a simulated text-based dataset from the VAST Challenge, and a Streamlit interface wrapper. The testbed simulates a high-stakes intelligence reporting task that allows researchers to evaluate human-AI system performance based on how quickly and effectively the combined team responds to the scenario under examination.
The research hypothesizes that AI systems that better match responsivity characteristics will align with higher trust perceptions, which will lead to overall system performance benefits for national security use cases. Study participants’ interactions with the AI system, including actions and communication patterns, will be recorded and analyzed using real-time team cognition metrics. The goal is to quantify interdependency shifts over time and evaluate how responsivity influences trust perceptions and effectiveness in human-AI collaboration.
“We’re looking at what really makes AI feel like a good partner - not just whether it gets the job done, but whether it listens, adapts, and supports human goals,” said Dr. Erin Chiou. “With AI TEST, we’re building a simple way to measure that kind of trust and teamwork.”