Pilot testing of experimental procedures to measure user's judgment errors in simulated social engineering attacks

Distracted users appear to have difficulties correctly distinguishing between legitimate and malicious emails or search engine results. Additionally, mobile phone users appear to have a more challenging time identifying malicious content due to the smaller screen size and the limited security features in mobile phone applications. Thus, the goal of this research study was to conduct a pilot test and validate a set of field experiments based on Subject Matter Experts (SMEs) feedback to assess users’ judgment when exposed to two types of simulated social engineering attacks: phishing and Potentially Malicious Search Engine Results (PMSER), based on the interaction of the environment (distracting vs. non-distracting) and type of device used (mobile vs. computer). This paper provides the results from the pilot test we conducted using recruited volunteers consisting of 10 participants out of 20 volunteers invited. Due to COVID-19 restrictions, all interactions in this pilot testing were conducted remotely. These restrictions somewhat limited our ability to control the testing environment to ensure a completely non-distractive environment during these parts of the study; however, a significant attempt was made to ensure such a non-distractive environment was genuinely adhered to during that part of the study. Our initial pilot testing results indicate that the findings were counterintuitive for the Phishing Intelligence Quotient (IQ) tests. In contrast, results of the PMSER were intuitive with improved detection on a computer compared to mobile. We conclude with a discussion on the study limitations and further research.