Empirical Assessment of Cybersecurity Competencies Through Human-Generative Artificial Intelligence (GenAI) Teaming

  • Dariusz Witko

    Student thesis: Doctoral ThesisDoctor of Philosophy

    Abstract

    The critical shortage of skilled cybersecurity professionals, with over 750,000 unfilled positions in the United States (U.S.), combined with rising practitioner burnout and the complexity of modern cyber threats, poses significant risks to national security. As Generative Artificial Intelligence (GenAI) emerges as a potential tool to support cybersecurity operations, its role in assisting human analysts with high-demand tasks offers both opportunities and challenges. While GenAI can improve efficiency, it also introduces adversarial risks if manipulated by malicious actors. This study investigated how human-GenAI collaboration can address these challenges, focusing on the fundamental cybersecurity knowledge, skills, and task completion required for effective teamwork, particularly within military contexts where adaptable technologically fluent and savvy "T-SAVVY" soldiers operate in technology-intensive environments. Using a mixed-methods design, this study engaged 31 novice participants in realistic cybersecurity activities conducted in a commercial Cyber Range. These participants performed tasks independently and with GenAI support (e.g. ChatGPT), to assess the extent to which GenAI enhances cybersecurity competency. This research unfolded in three phases: Phase I involved validating the fundamental cybersecurity competencies and experimental procedures through a Delphi method with 20 Subject Matter Experts (SMEs). In the subsequent phases, novice participants completed tasks in a commercial Cyber Range, designed to measure both competency with and without GenAI support. Furthermore, this research explored the ability to detect adversarial GenAI prompts. Statistical analyses, including Analysis of Variance (ANOVA) and Analysis of Covariance (ANCOVAs), helped evaluate demographic impacts on performance and determine whether GenAI significantly improves cybersecurity outcomes. Results from ANOVA and ANCOVA analyses confirmed that GenAI support significantly improved mean competency (F=23.523; p-value The findings are expected to contribute to the Body of Knowledge (BoK) by defining key competencies for GenAI-augmented teams and proposing guidelines for human-GenAI collaboration in the context of cybersecurity. This research aimed to enhance workforce readiness, offering strategies to build a resilient cybersecurity workforce equipped to navigate the complexities of a digitally dependent era.
    Date of AwardJan 1 2025
    Original languageEnglish
    SupervisorYair Levy (Supervisor), Greg Simco (Advisor), Laurie P. Dringus (Advisor) & Melissa Carlton (Advisor)

    Cite this

    '