Online Experimental Design Research Articles

‘Are ad endorsements really annoying?’ The impact of advertising in short-form tourism videos on tourists’ information processing

ABSTRACT Short-form tourism videos show high effectiveness in online destination marketing. However, some disruptive factors (i.e. embedded endorsement advertisements) diminish their marketing effectiveness by burdening the viewer’s cognitive processing and breaking one’s online interactive experience with the original tourism-related content in the video. Using a between-subjects online experimental design and 354 valid data from short-form video users, this study explored the effect of interruption by advertisements on the cognitive experience evoked by information stimuli from the perspective of information processing. This study also assessed the difference in short-form tourism video information processing between disturbed and undisturbed situations. In general, results revealed that the formation of destination-related mental imagery in short-form tourism videos is influenced by information processing fluency, and co-predicted both the attitudes and behavioural intentions of travel consumers. Meanwhile, the presence of advertising information in short-form videos notably obstructed users’ information processing fluency as well as the formation of mental imagery, and it also attenuated their destination visit intentions. This study provides a theoretical basis for the moderate interpretation of endorsement advertisements in short-form tourism videos and sheds light on insights related to tourism marketing strategies for short-form tourism videos for relevant practising practitioners.

Advertising Artificial Intelligence (AI) Agents: The Effects of Social Presence, Sincerity, and Social Benefit Appeals

The present study investigates the distinct advantages that conversational artificial intelligence (AI) agents provide consumers, including social benefits and communication sincerity. Additionally, this research examines how the exhibition of these attributes in advertisements impacts consumer perceptions. In doing so, this study represents an initial look at how advertisers can strategically promote conversational AI agents to consumers. The present study employed a 2 (AI agent sincerity condition vs. control condition) × 2 (AI agent benefits appeal: social benefits condition vs. utilitarian benefits condition) between-subjects online experimental design with a sample of 256 U.S. adults. Study results suggest that promoting social benefits of AI agents and emphasizing a sincere personality influence consumer social presence perceptions in human–AI interaction. Consumers’ perceived social presence leads to more favorable attitudes toward the advertisement and the use of the AI agent. In contrast, without the mediating role of social presence, accentuating the utilitarian benefits of AI agents in the advertisement is more effective than highlighting social advantages in influencing consumers’ positive attitudes. The current study contributes to the literature on consumer AI interactions, social presence, parasocial interactions, and brand personalities.

Pigeonhole Design: Balancing Sequential Experiments from an Online Matching Perspective

Practitioners and academics have long appreciated the benefits of covariate balancing when they conduct randomized experiments. For web-facing firms running online A/B tests, however, it still remains challenging in balancing covariate information when experimental subjects arrive sequentially. In this paper, we study an online experimental design problem, which we refer to as the online blocking problem. In this problem, experimental subjects with heterogeneous covariate information arrive sequentially and must be immediately assigned into either the control or the treated group. The objective is to minimize the total discrepancy, which is defined as the minimum weight perfect matching between the two groups. To solve this problem, we propose a randomized design of experiment, which we refer to as the pigeonhole design. The pigeonhole design first partitions the covariate space into smaller spaces, which we refer to as pigeonholes, and then, when the experimental subjects arrive at each pigeonhole, balances the number of control and treated subjects for each pigeonhole. We analyze the theoretical performance of the pigeonhole design and show its effectiveness by comparing against two well-known benchmark designs: the matched-pair design and the completely randomized design. We identify scenarios when the pigeonhole design demonstrates more benefits over the benchmark design. To conclude, we conduct extensive simulations using Yahoo! data to show a 10.2% reduction in variance if we use the pigeonhole design to estimate the average treatment effect. This paper was accepted by George Shanthikumar, data science. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.02184 .

Easy to read, easy to judge: Assessing readability as a heuristic for credibility in news

Despite calls for journalists and media agencies to address a disconnect between news audiences and news prose, content continues to increase in its difficulty to read and comprehend for the masses. While readability is often associated with audience comprehension and engagement, studies have neglected to assess whether readability is a factor in audience assessments of the credibility of content. Using an online experimental design, this study examines whether readability acts as a heuristic that helps news consumers make credibility judgments of news. Results show that readability tends not to be a predictor of credibility perception, regardless of partisanship strength or media use preferences. Theoretical and practical implications are discussed.

Credibility and altered communication styles of AI graders in the classroom

AbstractBackgroundEducation is often the primary arena for exploring and integrating new technologies. AI and human‐machine communication (HMC) are prevalent in the classroom, yet we are still learning how student perceptions of these tools will impact education.ObjectivesWe sought to understand student perceptions of credibility related to written feedback attributed to a human or an AI grader (Study One). We also investigated how corrective messages containing verbal immediacy and social support influenced student perceptions of an AI grader's credibility based on feedback in an evaluated essay (Study Two).MethodsWe used an online experimental design to assess the perceived credibility of a grader. In Study One, we randomly assigned students (N = 155) to a condition that contained a paragraph they were told was evaluated by a human or an AI grader. In Study Two (N = 222), we investigated ways of increasing perceptions of an AI grader's credibility by writing messages with higher/lower levels of immediacy and social support.ResultsIn Study One, the students rated both the human and AI grader as credible (yet rated the AI grader lower on goodwill). The data suggest that students in Study Two attributed more goodwill (i.e., caring) to the AI grader when the feedback included more verbal immediacy.ConclusionsOur results highlight the importance of student perceptions and communication styles when integrating technology into education. The two studies imply that students viewed the human and AI graders as competent, caring, and trustworthy, specifically when feedback included more immediacy cues.

The effects of different billboard advertisement language on TV sports audiences’ recognition, recall and search intention of the sponsor signage

PurposeThe impact of various factors on how TV sports audiences perceive sport event sponsors’ billboards around sports facilities has been the subject of extensive research. Despite numerous factors that impact the effectiveness of sponsor signage at sporting events, there has been a lack of research regarding the language used for such signage around sports facilities’ billboards. Therefore, this study aims to investigate the effects of billboard advertisement language on TV sports audiences’ recognition, recall and search intention to sponsor signage.Design/methodology/approachThis study employed an online experimental design. Participants (n = 925) were recruited from two linguistically different regions: Chinese and English. Participants were randomly assigned to one of two conditions: watching tennis video matches with billboard advertisements presented in either the Roman alphabet exclusively or in a combination of the Roman alphabet and Chinese characters.FindingsThis study revealed that although language cannot significantly impact audiences’ unaided recall of a brand, it does have a discernible effect on brand recognition and search intention among audiences. Additionally, people are more likely to search for brands in their native language. Participants from various regions tend to have different recognition rates and search intentions for sport sponsors.Originality/valueThis is the first manuscript examining the use of different languages in relation to audiences’ recognition and recall of sports sponsorship. It provides empirical evidence of the importance of carefully considering the language used in sponsor signage around stadium billboards to optimize the efficacy of sponsorships at sports events.

Mental health literacy for social phobia in Ghana: Investigation of gender stereotypes and previous experience for recognition rates and prejudice.

Mental health literacy (MHL) research has been of substantial interest internationally. Nevertheless, the interplay between beliefs, attitudes, previous experience with mental disorders, and knowledge of Ghanaians on specific mental disorders remains to be understood. The present study explored the interconnectedness between gender stereotypes, prejudice, previous experience with social phobia, and MHL among the general population in Ghana. Six hundred and one Ghanaians were recruited for an online experimental study design using a survey approach for data gathering. Respondents were randomly assigned to one of the two conditions (i.e., male and female vignettes) depicting symptoms of social phobia for a hypothetical person. Participants further completed self-reported measures including gender stereotypes (based on Ambivalent Sexism Inventory) and prejudice. Results revealed a 15.5% recognition rate for social phobia. Recognition rates of social phobia did not differ by the experimental condition or by the gender of participants. However, personal experience of social phobia was positively related to an increased likelihood of correctly labeling social phobia among men in the female vignette condition, whereas correct recognition of social phobia was negatively related to prejudice among women in the male vignette condition. In the male vignette condition, men with more hostile sexism attitudes toward men exhibited more prejudice toward their hypothetical male counterpart. In contrast, women with hostile sexism attitudes toward men exhibited less prejudice, but greater benevolent sexism attitudes toward men was associated with more prejudice toward the hypothetical male in the vignette. Findings from the current study emphasize the role of the cultural milieu in shaping effective mental health interventions. The results also have implications for promoting MHL to reduce prejudice in Ghana and other developing countries in the region.

Learning to embrace one’s stress: the selective effects of short videos on youth’s stress mindsets

ABSTRACTBackground and ObjectivesStress is not inherently negative. As youth will inevitably experience stress when facing the various challenges of adolescence, they can benefit from developing a stress-can-be-enhancing mindset rather than learning to fear their stress responses and avoid taking on challenges. We aimed to verify whether a rapid intervention improved stress mindsets and diminished perceived stress and anxiety sensitivity in adolescents.Design and MethodsAn online experimental design randomly exposed 233 Canadian youths aged 14–17 (83% female) to four videos of the Stress N’ Go intervention (how to embrace stress) or to control condition videos (brain facts). Validated questionnaires assessing stress mindsets, perceived stress, and anxiety sensitivity were administered pre- and post-intervention, followed by open-ended questions.ResultsThe intervention content successfully instilled a stress-can-be-enhancing mindset compared to the control condition. Although Bayes factor analyses showed no main differences in perceived stress or anxiety sensitivity between conditions, a thematic analysis revealed that the intervention helped participants to live better with their stress.ConclusionsOverall, these results suggest that our intervention can rapidly modify stress mindsets in youth. Future studies are needed to determine whether modifying stress mindsets is sufficient to alter anxiety sensitivity in certain adolescents and contexts.

Influences of mental accounting on consumption decisions: asymmetric effect of a scarcity mindset.

A scarcity mindset is considered to impact consumer behaviors. Our research aimed to examine the moderating effect of the scarcity mindset on the relationship between mental accounting and hedonic (vs. utilitarian) consumption. We conducted an online experimental design (mental accounting: windfall gains vs. hard-earning gains; consumption: hedonic products vs. utilitarian products) and verified our hypotheses in two distinct samples: a student sample and an adult sample. Our results showed that consumers who received windfall gains tended to use it for hedonic consumption rather than utilitarian consumption. Intriguingly, such an effect was insignificant under a high level of a scarcity mindset but significant under a low level of the scarcity mindset. Moreover, consumers who received hard-earning gains tended to spend the money on utilitarian (vs. hedonic) consumption. However, we did not detect the impact of the scarcity mindset on such effects. Our research suggested an asymmetric effect of the scarcity mindset on hedonic (vs. utilitarian) consumption under two different mental accounts. It highlights the important role of the scarcity mindset in consumer behaviors, which leaves avenues for future research to understand marketing promotion strategies for distinct products.

Digital Educational Strategies to Teach Skin Self-examination to Individuals at Risk for Skin Cancer

ABSTRACT Skin cancer is the most common cancer in the United States, and early detection of melanoma may lead to diagnosis of thinner and more treatable cancers, resulting in improved survival rates. This study examined the effects of message interactivity (high vs. low) and imagery (cartoon, real human character, or customized imagery preference) on accuracy of identifying abnormal skin lesions (ASL) and skin self-examination (SSE) intention. This study employed a 3 (cartoon character vs. real person vs. customization) x 2 (high interactivity vs. low interactivity) between-subjects online experimental design. Participants at risk for skin cancer were randomly assigned to one of the six conditions and completed a survey after reviewing the educational materials. Univariate analyses were conducted to detect group differences on the accuracy of identifying ASL and intention to conduct SSE in the next 3 months. Among 321 participants who completed the study, the mean age was 36.61 years, 56.7% were females, 76.1% had a college or higher degree, and over 60% self-identified as non-Hispanic White. Individuals in the high interactivity and customization group (compared to the low interactivity and cartoon group) were more likely to accurately identify ASL. Individuals in the high interactivity and customization or low interactivity and real person imagery groups (compared to the low interactivity and cartoon group) reported higher intention to conduct SSE in the next 3 months. These results suggest that customization and interactivity may be beneficial for educational programs or intervention design to improve both melanoma identification and SSE intention.

Reducing Choice-Blindness? An Experimental Study Comparing Experienced Meditators to Non-Meditators.

The mindfulness trait is an intrinsic characteristic of one's disposition that facilitates awareness of the present moment. Meditation has proven to enhance situational awareness. In this study, we compared the performance of participants that were split into two groups depending on their experience in mindfulness meditation (a control group naive to mindfulness meditation and a group of experienced mindfulness meditators). Choice-blindness happens when people fail to notice mismatches between their intentions and the consequences of decisions. Our task consisted of decisions where participants chose one preferred female facial image from a pair of images for a total of 15 decisions. By reversing the decisions, unbeknownst to the participants, three discrepancies were introduced in an online experimental design. Our results indicate that the likelihood of detecting one or more manipulations was higher in the mindful group compared to the control group. The higher FMI scores of the mindful group did not contribute to this observation; only the practice of mindfulness meditation itself did. Thus, this could be explained by better introspective access and control of reasoning processes acquired during practice and not by the latent characteristics that are attributed to the mindfulness trait.

Pigeonhole Design: Balancing Sequential Experiments from an Online Matching Perspective

Practitioners and academics have long appreciated the benefits that experimentation brings to firms. For online web-facing firms, however, it still remains challenging in balancing covariate information when experimental units arrive sequentially in online field experiments. In this paper, we study a novel online experimental design problem, which we refer to as the "Online Blocking Problem." In this problem, experimental units with heterogeneous covariate information arrive sequentially and must be immediately assigned into either the control or the treatment group, with an objective of minimizing the total discrepancy, which is defined as the minimum weight perfect matching between the two groups. To solve this problem, we propose a novel experimental design approach, which we refer to as the "Pigeonhole Design." The pigeonhole design first partitions the covariate space into smaller spaces, which we refer to as pigeonholes, and then, when the experimental units arrive at each pigeonhole, balances the number of control and treatment units for each pigeonhole. We analyze the theoretical performance of the pigeonhole design and show its effectiveness by comparing against two well-known benchmark designs: the match-pair design and the completely randomized design. We conduct extensive simulations to study the numerical performance of the different designs and conclude with practical suggestions.

Effectiveness of message framing in changing COVID-19 vaccination intentions: Moderating role of travel desire.

This research examines the effectiveness of message framing, message appeal and information content in changing respondents' COVID-19 vaccination intentions through influencing their vaccine risk perceptions. Furthermore, the moderating effect of travel desire on the relationship between vaccine risk perceptions and changing vaccination intentions is examined. In doing so, two rounds of data that were collected from the same respondents. The first survey recorded respondents' vaccination intentions, travel desire and socio demographics. A follow up survey tested cause-and-effect relationships on the proposed relationships using a 2 (message frame: gain, loss) x 2 (message appeal: rational, emotional) ×2 (information content: subjective, objective) between-subjects online experimental design. Findings suggest that loss-framed messages are more effective than gain-framed and emotional-rational messages in reducing risk perceptions and, thus, changing vaccination intentions. Travel desire is found to moderate the effect of vaccine risk perception on vaccination intentions by weakening the negative effect of vaccine risk perception on vaccination intention. Findings show the importance of travel desire along with message framing and message appeal on changing individuals’ COVID-19 vaccination intentions in public health communications.

Why display motion on packaging? The effect of implied motion on consumer behavior

Visual packaging elements play a crucial role in influencing consumer behavior in stores. Front of pack (FOP) formats frequently present images of the product in motion (i.e., implied motion), especially in food categories. Despite the popularity of implied motion in real-world business, little research has been done to understand its effect on consumer behavior. To fill this gap, the present study adopts an evolutionary lens to investigate the impact of implied motion as a packaging design technique on consumers’ attention, product evaluation, purchase intention and choice. We carried out two experiments using realistic milk and orange juice packaging. Specifically, Experiment 1 was conducted in a lab using an eye tracking method to provide an objective measure of attention. The findings show that implied motion significantly increases visual attention and consequently generates more frequent choices. Using an online experimental design and declarative measures, Experiment 2 indicates that implied motion also enhances freshness, which translates into greater tastiness, product attractiveness and, ultimately, purchase intention. The study provides marketers with an inexpensive yet efficient way to enhance in-store marketing performance by incorporating implied motion into the FOP design.

How terrorism cues affect attitude polarization over undocumented immigrants via negative emotions and information avoidance: A terror management theory perspective

ABSTRACT Using an online experiment design on Americans, this study investigates how terrorist attack death cues affect individual-level attitude polarization regarding undocumented immigrants via distinct negative emotions (i.e., anxiety, anger, and sadness) and information avoidance. First, we find that exposure to terrorist attack death cues increases anxiety, anger, and sadness. Of these negative emotions, anxiety increases information avoidance, and such information avoidance reduces attitude polarization. We revisit the traditional affect-free claim of terror management theory (TMT) and connect the theory with discrete negative emotions and information avoidance to better understand the underlying mechanisms of attitude polarization in threatening situations. Our findings suggest that the role information avoidance plays in managing death anxiety in response to threats will require more systematic examination, as information avoidance can counterbalance tendencies toward selective exposure when experiencing threats that evoke death anxiety.

Hospitality employee and customer role in value co-creation: Personal, organizational and collaborative outcomes

• Hospitality employees evaluate co-creation higher than customers. • Co-creation positively affects wellbeing, satisfaction, competitive advantage perceptions. • Customers value co-innovation, co-marketing, co-production, and co-recovery differently. • Co-created value is mediator among four co-creation types and outcomes for customers. • Employees do not differentiate among value co-creation types. With the growing frequency and importance of peer-to-peer service exchanges, the role of front-line employees for traditional hospitality service providers is changing. This study is the first to examine differences in value co-creation appraisal between front-line employees and customers. Respondents ( N = 720) were equally assigned to either an employee or customer role and performed that role for four different online co-creation scenarios based on destination resort context: co-innovation, co-creation of marketing, co-production, and co-recovery. The results of between-within online experimental design were analysed using repeated measures ANOVA and PLS-SEM. Co-created value was appraised significantly higher by employees and positively affected well-being, satisfaction, and competitive advantage for both employees and customers. Co-created value served as a mediator between the four types of co-creation and outcomes only for customers. The study contributes to understanding roles as knowledge-based resources in value co-creation and provides practical implications for successful value co-creation in hospitality.

The Infectious Disease Threat (IDT) Appraisal Model: How Perceptions of IDT Predictability and Controllability Predict Individuals’ Responses to Risks

ABSTRACT Grounded in the multidisciplinary field of strategic risk and health communication, this study proposed and tested a new infectious disease threat (IDT) appraisal model, focused on mapping individuals’ coping strategy preferences as predicted by their perceived predictability and controllability of the disease. A 2 (predictability: high vs. low) × 2 (controllability: high vs. low) within-subjects online experimental design (N = 1,032 U.S. adults) was employed, in which four IDT scenarios (sexually transmitted infection [STI]; waterborne ID; foodborne ID; vector-borne ID) were shown to participants in a counterbalanced fashion, to examine the effects of IDT appraisals on how individuals cope with outbreaks. Results support the hypothesized model, in which assessments of predictability, controllability, and responsibility of an IDT situation drive individuals’ affect valence, information seeking, and conative reactions in passive and active ways. Findings further provide insights into what information seeking strategies and IDT coping behaviors individuals prefer based on their differential IDT appraisals, thus suggesting how public health authorities and risk communication professionals can optimally communicate about infectious diseases to help individuals understand these situations and respond appropriately.

Citizens’ perceptions of closing the gender pay gap: an experimental study

ABSTRACT The issue of pay equity has been a topic of interest in our society. One area that has been understudied is the perceptions of the citizenry toward pay disparities between women and men. The purpose of this study is to examine the perceptions of the citizenry in efforts to close the pay gap. We rely on an online experimental design, where we ask survey respondents their views on efforts by a local government to promote pay equity. We find that, regardless of how female and male employees are actually performing, citizens have favorable responses towards efforts to promote pay equity.

An online experiment on the influence of online user comments on attitudes toward a minority group

In recent years, there is interest in examining the effects of user comments on online content consumers’ attitudes and perceptions. Building on theoretical foundations from social psychology and using an online experimental design, we investigated whether exposure to online comments attached to news content affects attitudes toward refugees. We recruited students from a public university in the Mediterranean region and, after administering a printed pretest on their attitudes toward refugees, we assigned them into three groups: one that read a series of positive comments about refugees (positive experimental group), one that read a series of negative comments about refugees (negative experimental group) and one that did not read comments (control group). The comments and a neutral reference news video were communicated to the participants via email during a five-day period after the pretest. After the implementation of the experimental stimuli, we administered a modified version of the pretest as a posttest. We found that exposure to positive comments reduced prejudice and that the intensity of prior prejudice was positively associated with the magnitude of prejudice reduction. Exposure to negative comments did not affect prejudice. A repeat posttest, administered one week after the posttest, showed stability of the observed attitude change.

Social norms and free-riding in influenza vaccine decisions in the UK: an online experiment

BackgroundNudge-based social norm messages conveying high influenza vaccination coverage levels signal a strong social norm, encouraging vaccination, but also a low risk of infection, discouraging vaccination and promoting free-riding. The complex interplay between these two signals can result in ambiguous vaccination decision-making at varying coverage levels. We aimed to measure different vaccination coverage levels’ (VCLs) effect on influenza vaccination intention through an online experiment. MethodsUK residents aged 18 years or older were eligible to participate in this online experiment and recruited via Prolific. They were stratified by gender and randomly assigned to a control group with no message (n=202) or one of seven treatment groups (n=1 163) with different messages of VCLs (ie, proportion of vaccinated people [10%, 25%, 50%, 65%, 75%, 85%, or 95%]) in the respondents’ environment. Effect on respondents’ vaccination intention was measured with self-reported intention and three elicited behaviour measures: opening an online map locating nearby private flu jab providers; time looking at this map; and downloading a calendar reminder to vaccinate. Linear regressions, probit, logistic, and double hurdle models were used, controlling for population behaviour perceptions, risk attitudes, and behavioural and socioeconomic characteristics collected through individual questionnaires. FindingsBetween May 3 and August 20, 2018, 1365 participants were eligible to participate. Those treated with coverage levels at 65% or higher reported significantly greater stated vaccination intention (greater intention 13·2% [95% CI 6·32–20·08], p<0·0001 if VCL=65%; 15·82% [8·65–23], p<0·0001 if VCL=75%; 18·12% [10·45–25·79], p <0·0001 if VCL=85%; 20·22% [11·98–28·45], p<0·0001 if VCL=95%) than the control group. These participants were also more likely to look at the map of vaccination locations (increased likelihood 14·6 percentage points [3·65–25·55], p=0·0090; 20·81 percentage points [9·73–31·88], p<0·0001; 18·4 percentage points [6·57–30·23], p=0·0023; 14·12 percentage points [1·27–26·96], p=0·031), and spent longer looking at the map (longer viewing 0·52 s [0·13–0·91], p=0·0094; 0·74 s [0·34–1·14], p<0·0001; 0·65 s [0·23–1·08], p=0·0025; 0·5 s [0·04–0·96], p=0·032) than the control group. Coverage lower than 65% did not lead to any significant differences between treatment and control groups for any of these measures. There was no significant effect of any treatment VCL on calendar download intention. InterpretationAverage vaccination intention was higher at lower coverage levels (potential social norm compliance) but lower at higher coverage levels (possible free-riding). Because this study examines vaccination intention rather than uptake and uses an online experimental design that might dampen generalisability, further research should be done. Still, policy makers should consider this curvilinear effect when designing vaccination interventions as using lower social norm messages might nudge vaccination better than extremely high messages. FundingImperial College Business School, National Institute for Health Research Health Protection Research Unit, UK Medical Research Council, Department for International Development, Medical Research Council Centre for Global Infectious Disease Analysis.