Research Reports

Bot Automation Using Large Language Models (LLMs) and Plugins

PUBLIC RELEASE
January 2025

COMPLETED
July 2024

AUTHORS: Dr. Naren Ramakrishnan, Dr. Patrick Butler, Mr. Brian Mayer, Mr. Andrew Neeser
VIRGINIA TECH

The aim of this research study was to create tools that automate information extraction pipelines to support business processes in contract and procurement management. The research team was specifically asked to explore opportunities to use Large Language Models (LLMs) to accomplish this task. After reviewing the problem space and the potential solutions, the team designed and created a tool to generate reports on the status of entries from the Contractor Performance Assessment Reporting System (CPARS), broken down by contracting division.

This tool automates the extraction of the Contracting Officer’s Representative (COR) status information. The team also explored methods for using LLM pipelines to automate other potential contractual management tasks and presented some demonstrations of possible uses. The research indicated that LLMs have significant potential to enhance contract and procurement management processes, e.g., automating field extraction from existing contracts, assisting contract generation and customization, rapid contract analysis, and streamlining routine document processing tasks.

Based on demonstrations, the sponsor agreed on their potential. Yet, while the potential benefits are substantial, there are concerns with data privacy and security, accuracy and reliability, legal and compliance issues, and integration with existing systems. To mitigate these concerns and maximize benefits, the research team suggests focusing on local, open-source LLM solutions like LLaMA or Phi. These models can be deployed on-premises, ensuring data privacy and security while providing powerful LLM capabilities including customization and specialization.