Events

Model-Based TEMP Strategy & Integrated Decision Support Key Workshop

August 2-3, 2023

USC Information Sciences Institute | Hybrid

The United States (US) Department of Defense (DoD) and its supporting industry, research, and academia partners have proposed using Digital Engineering (DE) methods and tools to update traditional systems engineering (SE) and test and evaluation (T&E) practices to improve acquisition outcomes and accelerate traditional processes.

One critical element of the T&E process is the development of the Test and Evaluation Master Plan (TEMP), in which the program office, systems engineers, and testers come together to document an adequate test and evaluation strategy that spans major program milestones. DE provides opportunities to enhance the TEMP by directly incorporating data and models into the process of documenting the test strategy and program.

A new core element of the TEMP is the Integrated Decision Support Key (IDSK) and Evaluation Matrix family of related tables (e.g., requirements, evaluation measures and metrics, key program decisions, test events, etc.). The IDSK derives acquisition decisions from the developmental and operational test-based evaluations; the Evaluation Matrix links high level system performance metrics to their supporting test data.

This workshop will explore Digital Engineering, TEMPs, IDSKs to identify best practices, new methods and tools, and build consensus on where digital transformation will have the largest impact on T&E processes and documentation.

Workshop Objectives

Gather the T&E community from across the DoD, industry, and academia to discuss current state of capabilities, research gaps, and synergies, and develop a common vision for digitally transforming T&E practices.

Presentations will include demonstrations of tools and practices. Topics will include:

IDSK examples for T&E Across the Acquisition Life-cycle

Approaches to Model-Based IDSK

Best Practices in Digital Transformation

Linkages between legacy system performance, modeling, performance assessment, and decision-support data/information

IDSK application across the Acquisition Lifecyle and integration of Digital Twins

Speakers

Randy Saunders-sq-1
Dr. Randy Saunders
Johns Hopkins University Applied Physics Laboratory
Biography

Dr. Saunders is a Software and Systems Analysis who has held positions of increasing responsibility with 40+ years of experience. Roles and responsibility include: Principle analyst for Director, Operational Test and Evaluation program to digitally transform operational test strategies through the development of a Model-Based Test and Evaluation Master Plan concept. Project technical lead for the Air Force A9 Bayesian Enterprise Analysis Model (BEAM), a new statistical modeling paradigm to support enterprise scale analysis. Analysis lead on the OSD technical team for the Integrated Defense Analysis Capability (IDAC) development effort. Cyber security analyst for developmental and operation test organizations, focusing analysis on policy and process improvements. Research analyst for information technology efficiencies improvement across the intelligence community.

Dr. Laura Freeman
Dr. Laura Freeman
Virginia Tech National Security Institute
Biography

Dr. Freeman is a Research Professor of Statistics and dual hatted as the Deputy Director of the Virginia Tech National Security Institute and Assistant Dean for Research for the College of Science. Her research leverages experimental methods for conducting research that brings together cyber-physical systems, data science, artificial intelligence (AI), and machine learning to address critical challenges in national security.

Suzanne Beers-sq-1
Dr. Suzanne Beers
MITRE
Biography

Dr. Beers is MITRE’s Defense Systems Engineering Department Manager, leading personnel and portfolio management of six OSD-level test and evaluation (T&E), systems engineering and strategic analysis projects. In addition to her Department Manager responsibilities, Dr. Beers is the Technical Lead for OSD/DTE&A’s Integrated Decision Support Key initiative. This work – which she will be briefing during the workshop – includes evolving the IDSK concept from separate developmental and operational evaluation frameworks and applying it to both acquisition programs of record and full-spectrum (S&T to sustainment – especially to prototyping and experimentation) decision-support informed by operational and technical capability evaluation.
Dr. Beers joined MITRE in Colorado Springs after retiring from the USAF in October 2008. Her final assignment was as the Commander, Air Force Operational Test and Evaluation Center’s Detachment 4 at Peterson AFB leading the OT&E of new space, missile, and missile defense systems.
Dr. Beers holds a plethora of degrees (five) from a wide variety of institutions of higher learning…giving her plenty of cheering flexibility during college football season…including a B.S. in Mechanical Engineering from The Ohio State University and a Ph.D. in Electrical Engineering from Georgia Institute of Technology. Dr. Beers caught the triathlon bug, spending the last decade swimming, biking, running or talking triathlon with fellow addicts. She has since traded-in the triathlon hobby for astrophotography.

Caitlin Szymendera-sq-1
Ms. Caitlin Szymendera
NAVAIR
Biography

Ms. Caitlin Szymendera is the current Director for RDT&E Data and Technology for NAVAIR’s Engineering & Cyber Warfare Group. She also serves as the Test & Evaluation Tools Lead for the Department of the Navy’s (DON) Test & Evaluation Executive, working to modernize and standardize the digital landscape across the DON Test & Evaluation community. Ms. Szymendera spent the first half of her career as a Mission Systems Flight Test Engineer for the Presidential and CH-53K Helicopter Programs before transitioning to enterprise process improvement and digital tools program management.

Hans Mair-sq-1
Mr. Hans Mair
Johns Hopkins University Applied Physics Laboratory
Biography

Mr. Mair is a national security analyst and systems engineer analyzing complex adaptive system performance in support of consequential national security decisions.

Jason Bigger-sq-1
Mr. Jason Bigger
U.S. Army
Biography

Mr. Bigger is a Test Subject Matter Expert within the Army’s Future Long Range Assault Aircraft Program Management Office (PMO) under PEO Aviation. He has been a test engineer in PEO Aviation for over 20 years and has a BSE from Univ. of Alabama Huntsville (UAH).

George Michael Winter-sq-1
Mr. George Michael Winter
U.S. Army
Biography

Mr. Winter is a model-based systems engineer within the Army’s Future Long Range Assault Aircraft Program Management Office (PMO) under PEO Aviation. He has a MS in Systems Engineering from Kennesaw State University, a MS in Mechanical Engineering from Purdue University, and a BS in Mechanical Engineering from the Georgia Institute of Technology.

Space Force_Headshot
Maj Caleb “CJ” Werner and Mr. Ken Sexe
U.S. Space Force
Biography

The Digital Transformation and Command Tests Authority out of Space Systems Command have partnered to revolutionize the way Test & Evaluation is executed across space. Being in Space System Integration Office, both teams collaborate to bring new techniques and change the way Acquisitions is done in the Space Force. Although a young team, this group has created the Model Based T&E Reference Model (MBTERM) which has become a flagship of Enterprise Digital Transformation and Engineering not only across the Space Force but has garnered support from stakeholders all across the DoD and IC. Being in a new service and new organization, the MBTERM team is able to design with multi-domain warfare in mind and is dedicated to support not only the attack and defend mission in space, but allow the Space Enterprise T&E function to support mission partners.

Trisha Radocaj-sq-1
Ms. Trisha Radocaj
Johns Hopkins University Applied Physics Laboratory
Biography

Ms. Radocaj is a Systems Engineer in Force Projection Section, Model-Based & Digital Mission Engineering. She has been supporting Test & Evaluation Planning with model-base strategies that relate mission planning to system implementation and operational, developmental and contractor test. Her background includes analysis of launch vehicle and satellite propulsion systems supporting Air Force projects, as well as unmanned control systems and communication systems supporting the Navy.

James Ferry-sq-1
Dr. James Ferry
Metron, Inc.
Biography

Dr. Ferry is a Senior Research Scientist at Metron, Inc. He has an SB in Mathematics from MIT and a PhD in Applied Mathematics from Brown University. He has been the Principal Investigator for a variety of projects for DARPA, ONR, and the intelligence community. His areas of expertise include Bayesian reasoning, network science, data fusion, and fluid dynamics.

Kelli Esser-sq-1
Dr. Kelli Esser
Virginia Tech National Security Institute
Biography

Dr. Esser is the Associate Director for the Intelligent Systems Division of the Virginia Tech National Security Institute. She is a recognized leader and change agent with over eighteen years of experience working with national and homeland security agencies to improve the planning, management, and coordination of research development and innovation investments across public and private partners. Dr. Esser’s research has contributed to major policy and implementation changes across government in the areas of test and evaluation, capabilities-based planning, and operational assessments in support of technology development, transition, and deployment.

Alexander_Headshot
Mr. Matthew Alexander
MITRE
Biography

Mr. Alexander is a Senior Systems Engineer with MITRE supporting the Space Training and Readiness Command and the Space Systems Command. He has used a range of methods, techniques, and tools to digitally execute multiple systems engineering and mission engineering methods and processes for the Space Force. He has also performed Digital Engineering to positively impact the U.S. Army, the U.S. Air Force, the U.S. Navy, the U.S. Space Force, and a Joint Program Office of a highly visible program. He has earned recognition from government, MITRE, and contractor top leadership. Non-DoD systems engineering included work for the FAA, DHS, IRS, CMS, and VA. Matt earned a Bachelors of Science and then a Masters of Science in Systems Engineering from the John Hopkins University. He took a Worcester Polytechnic Institute MBSE course taught by MBSE industry thought leader Sanford Friedenthal and is taking an MIT Professional Education course on the New Space Economy in October 2023.

Mulligan_Headshot
Mr. David Mulligan
MITRE
Biography

Mr. Mulligan is the Project Lead for the Space Training and Readiness Command (STARCOM) within the Space Enterprise Architecting Department of MITRE’s Air and Space Force Center where he’s responsible for directing the work for MITRE’s support to the Test & Evaluation Enterprise, as well as the STARCOM & Space Systems Center National Space Test & Training Complex (NSTTC) Integrated Program Office. Since joining MITRE in 2010, Dave has been involved in a range of activities, spanning a variety of roles leading and supporting requirements development, cyber, test and evaluation and program assessments of space capabilities. His experiences span direct support to a program office to major command to OSD Research & Engineering Developmental Test, Evaluation & Assessments. Dave earned a B.S. in Computer Science from Embry-Riddle Aeronautical University, a MBA from Webster University, a M.S. in Systems Engineering from the Air Force Institute of Technology. Dave is also a 27-year US Air Force veteran with linguist, software development, network management, tactical communications, radar evaluation, cyber and space protection experiences.

Craig Arndt-sq-1
Dr. Craig Arndt
George Tech Research Institute
Biography

Dr. Arndt has extensive experience as a senior executive and technology leader in research, education, engineering and defense, homeland security and intelligence technologies, with extensive experience as an innovative leader in industry, academia, and government. He is often called upon as an industry expert to chair major program reviews; lead new programs, bid and proposal efforts, in the areas of C4ISR and sensor systems, Intellectual Properties and Data Rights, Human Factors, and Systems Engineering.
Dr. Arndt currently serves as a principal research engineer on the research faculty of the George Tech Research Institute (GTRI) in the System Engineering Research division of the Electronic Systems Lab. Dr. Arndt is a licensed Professional Engineer (PE), and has over 40 years of professional engineering experience though the defense and government engineering community.
Dr. Arndt holds engineering degrees in electrical engineering, systems engineering and human factors engineering and a Masters of Arts in strategic studies from the US Naval War college. He served as Professor and Chair of the engineering department at the Defense Acquisition University, and as technical director of the Homeland security FFRDC at the MITRE Corporation.
In industry, Dr. Arndt has served as a manager at all levels of engineering management including director of systems engineering, CTO and Vic President of Engineering, and has led business units, companies, and engineering organizations of all sizes located both in the United States and internationally.
As an active member of the academic and professional community, he has served as a technical expert for the Army and Defense Science Boards, the CNO’s Strategic Studies Group (SSG), the National Science Foundation, the International Standards Organization (ISO), the IEEE, and other public and private technical organizations, and for several major universities.
Dr. Arndt served as an engineering officer in the US Navy and retired from the Navy in 2013. He is married to Dorothy Becker and has two children and three grandchildren.

Brentson J. Bell Jr.-sq-1
Mr. Brentson J. Bell Jr.
U.S. Army Cyber Resiliency and Training
Biography

Mr. Bell currently serves as a Deputy Product Manager for Cyber Resiliency and Training (PdM CRT) for the Program Executive Office for Simulation Training and Instrumentation (PEO STRI). Mr. Bell is responsible for all aspects of acquisition for a Major Defense Acquisition Program, the Persistent Cyber Training Environment (PCTE), enabling over 9,000 Cyber Mission Force operators to meet and exceed the Department of Defense Cyber Mission Readiness strategy.

Mr. Bell was previously the Deputy Product Director for Aviation Logistics for the Program Executive Officer of Enterprise Information Systems (PEO EIS). Mr. Bell established the product management office overseeing a 170-person operation for a Major Acquisition Information System program. He directed the planning and execution of a $750 million program which supported the development, testing, and fielding of the total Army Aviation Logistics portfolio within the Army enterprise capability modernizing logistics oversight of over 6,000 Army aircraft.

Mr. Bell also supported the Deputy Assistant Secretary of Defense Developmental Test & Evaluation (DASD(DT&E)) Director as a Staff Specialist. He was responsible for providing oversight to several Major Acquisition Programs, reviewing their Integration, Test and Evaluation Strategies, Test and Evaluation Master Plans, System Engineering Plans, and evaluation reports providing program assessment recommendations to OSD leadership.

William Corey Bogler-sq-1
Mr. William Corey Bogler
U.S. Army Cyber Resiliency and Training
Biography

Mr. Bogler serves as the Senior Test & Integration Officer on the PCTE program. After graduating from East Stroudsburg University with a BS in Computer Science, Mr. Bogler started his career developing software for the Paladin self-propelled howitzer before transitioning to PEO STRI, where he led efforts to implement Modular Open Systems Approaches for the MILES live training product line.

In his time as Product Manager – Cyber Resiliency and Training, he has led both the Operations and Test teams and is a member of the CRT Leadership team.

John “JD” Donaldson-sq-1
Mr. John “JD” Donaldson
U.S. Army Cyber Resiliency and Training
Biography

After Mr. Donaldson earned his BS in Computer Science and a commission in the US Army, he served a rewarding 24-year career as a Signal Officer and paratrooper, during which he learned that leadership is service, and gained a lifelong passion for travel and adventure. Since hanging up his jump boots, John has worked in cybersecurity, while enjoying everything that life with Orlando as your homebase has to offer. As part of the innovative Persistent Cyber Training Environment test team, he has broken new ground in cybersecurity testing of Defense Department acquisition programs, helping to earn the team special designation as “Pathfinders” by the Assistant Secretary of the Army (Acquisition, Logistics and Technology).

Gilbert Duverglas-sq-1
Mr. Gilbert Duverglas
U.S. Army Cyber Resiliency and Training
Biography

Mr. Duverglas is the Operational Test Lead for the Persistent Cyber Training Environment (PCTE). With nearly four years of dedicated experience with PCTE, Gilbert Duverglas has played a pivotal role in ensuring the program’s success by aiding in the development of the DevSecOps processes.

O'Neill_Headshot
Mr. Davin O’Neill
MITRE
Biography

Mr. O’Neill, with the MITRE Corporation, is a highly experienced technical leader with 20+ years of diverse experience in integrated architecture development, MBSE, software engineering, and digital engineering. With a BS in Physics and MS in Materials Engineering, he has made significant contributions to various domains, including ballistic missile defense and C4ISR systems. Currently supporting Army Test and Evaluation Command’s digital transformation, Davin develops model-based enterprise architectures, incorporating intelligent dashboards and simulation tools.

Jonathan Brown-sq-1
Mr. Jonathan Brown
Agile Data Decisions, Inc
Biography

Mr. Brown has led business development efforts focusing on the Department of Defense for over 15 years for software and robotics companies. He has led teams that have won multiple Phase I, II, and III SBIR and STTR contracts with the Air Force, Army, Navy, Marine Corps, and DARPA. In addition, he has secured multiple contracts from SOCOM and from DoD prime contractors. Mr. Brown previously worked for AnthroTronix, Inc, Silver Spring, MD; RE2 Robotics, Pittsburgh, PA; and Think-A-Move, Ltd, Beachwood, OH. Mr. Brown received his M.B.A. from the University of Chicago Booth School of Business and his B.A. from Connecticut College.

Location

USC Information Sciences Institute
4676 Admiralty Way, Marina Del Rey, CA 90292

Directions

  FROM

Los Angeles International Airport
1 World Way, Los Angeles, CA 90045

Icon-Line

Get on CA-1 N/S Sepulveda Blvd
5 min (1.0 mi) CA-1 N turns slightly right and becomes CA-1 N/Lincoln Blvd

Follow CA-1 N/Lincoln Blvd to Fiji Way
9 min (3.7 mi)

Drive to Admiralty Way in Marina del Rey
2 min (0.4 mi)

  TO

USC Information Sciences Institute
4676 Admiralty Way #1001, Marina Del Rey, CA 90292

Hotels

13534 Bali Way, Marina Del Rey, CA 90292
(310) 301-1000

4200 Admiralty Way, Marina Del Rey, CA 90292
310-301-2000

13480 Maxella Ave, Marina Del Rey, CA 90292
310-822-8555

4630 Via Marina, Marina Del Rey, CA 90292
310-439-2908

6161 West Centinela Avenue, Culver City, CA 90230
310-6491776

Point of Contact

G.Kerr_sq

Technical

Senior Research Associate, Systems and Digital Engineering, Virginia Tech National Security Institute
Tara Kelly

Registration

Research Project Manager, Systems Engineering Research Center (SERC), AIRC