Research Reports

Cognitive Assistant for Training Cost Estimators

PUBLIC RELEASE
October 2024

COMPLETED
July 2024

AUTHORS:  Dr. Daniel Selva1, Dr. Theodora Chaspari1, Dr. Alejandro Salado2
TEXAS A&M UNIVERSITY 1, UNIVERSITY OF ARIZONA 2

In an incubate Phase I of this project (September 2021 to June 2022), an AIRC research team worked with the sponsor and other stakeholders to define the use case for cognitive assistants. The project was approved for a Phase II with the goal of further developing the agent and validating it with real users. A previous report described the activities conducted during the first half of the Phase II effort (September 2022 to September 2024). In that first year of the Phase II effort, the team developed a second version of the agent that leverages Large Language Models (LLMs) to make the system more flexible, extensible, and easier to maintain. In addition, they developed a three-module online course on space systems with slides, example questions, and quizzes for testing the effectiveness of the tool.

This report documents the second half of the Phase II effort (October 2023 to July 2024). During these eight months, the researchers refined the software tool, added new instructional materials, and validated the effectiveness of the tool in the lab with 51 student subjects and 22 real users at the Office of Cost Assessment and Program Evaluation (CAPE) and the Air Force Cost Analysis Agency (AFCAA).

Results from the lab experiment showed that students who used the cognitive assistant scored six points higher in the test than students without access to the assistant, suggesting that this technology is promising to improve the efficiency and effectiveness of workforce training. This improvement is likely due at least in part to the assistant simply increasing the time the student is interacting with the material. The software tool and accompanying documentation including a transition plan document have been delivered to the sponsor for transition into production.