Abstract P460: Attack of the (Survey) Bots: How We Determined That Our Anonymous Survey Was Full of False Data

Abstract

Introduction: Given the nursing shortage in the US and increased focus on reducing burnout, more research on nurses' and nursing assistants’ strategies to mitigate the adverse effects of night shifts and rotating schedules is needed. However, recruitment challenges limit study sample sizes and, consequently, the generalizability of results. Social media platforms provide opportunities to reach a broader sample, yet data quality assessment requires further investigation. Purpose: This study examined the feasibility of using X (formerly Twitter) to recruit nurses and nursing assistants throughout the US to complete a survey on work schedules, fatigue countermeasures use, and sleep and general health. Methods: The anonymous survey was advertised on X with the following tweet: “Nurses and nursing assistants: How does your work schedule and sleep affect your health? What do you do to lessen your fatigue when working? Let us know by completing the Scheduling, Sleep, and Health Survey and a chance for an Amazon gift card: [survey link]. Please RT & share! #Nurse #NurseLife.” Inclusion criteria were: 1) age ≥ 18; 2) being a registered nurse, licensed practical nurse, or nursing assistant; and 3) worked ≥ 36 hours/week during the last 4 weeks. The survey included questions on sociodemographics, shift type and schedule, fatigue countermeasure use, mealtimes/location when working, and sleep health and several PROMIS measures (General Health, Fatigue, and Cognitive Function). After survey completion, respondents were directed to a new survey to provide their contact information to be entered to win a $50 Amazon gift card, with 20 randomly selected recipients. The survey incorporated data verification questions to detect potentially bot-generated or suspicious responses. Results: In under 24 hours, 512 surveys were recorded (495 contained complete data) before deactivating the survey link. Responses were categorized as: 1) likely genuine (n = 2); 2) likely bot-generated (n = 472); and 3) uncertain (i.e., possible but not likely such as worked 6 or 7 shifts in the last 7 days; n = 19). Identifying likely bot-generated responses involved evaluating four components from the data verification questions: 1) inconsistencies in mealtimes/locations compared to reported shift type in the past week (n = 243); 2) failure to meet the criterion for hours worked (i.e., shift length and number of shifts totaling < 36 hours in the last 7 days; n = 143); 3) discrepancies in reported shifts between the past week and the previous 4 weeks (n = 61); and 4) responses with large blocks of identical data (n = 27). Conclusions: Since over 95% of the responses were likely from bots, it may not be feasible to use X for valid, anonymous survey data collection. Targeted recruitment methods within a healthcare system or specialty groups (e.g., operating room or critical care nurses) may provide more valid, but less generalizable, data.