Instructional Manipulation Research Articles

Are Attention Check Questions a Threat to Scale Validity?

Attention checks have become increasingly popular in survey research as a means to filter out careless respondents. Despite their widespread use, little research has empirically tested the impact of attention checks on scale validity. In fact, because attention checks can induce a more deliberative mindset in survey respondents, they may change the way respondents answer survey questions, posing a threat to scale validity. In two studies, we tested this hypothesis (N = 816). We examined whether common attention checks—instructed‐response items (Study 1) and an instructional manipulation check (Study 2)—impact responses to a well‐validated management scale. Results showed no evidence that they affect scale validity, both in reported scale means and tests of measurement invariance. These findings allow researchers to justify the use of attention checks without compromising scale validity and encourage future research to examine other survey characteristic‐respondent dynamics to advance our use of survey methods.

Capturing the scene: efficacy test of the re-enactment investigative instruction

PurposeAs eyewitnesses provide the most valuable information for criminal investigations, it is important to further develop and test techniques for collecting eyewitness testimony so that they meet the major objective of a police interview: obtaining details pertaining to criminal actions. The purpose of this paper is to test a new instruction – the re-enactment investigative instruction – formulated to collect the most fine-grained details of a criminal event as accurately as possible. It leads the interviewee to decompose all directly recollected actions into the most minimal actions so that the event can be accurately re-enacted.Design/methodology/approachIn all, 40 participants individually viewed a video depicting a robbery, were randomly assigned to a re-enactment or structured interview (SI) group and then interviewed face-to-face. Each interview was comprised of two free recall phases and a questioning phase. Manipulation of the re-enactment instruction took place in the second free recall phase of the re-enactment interviews (RIs).FindingsThe RI elicited more correct information compared to the SI (d=1.14), and slightly but not significantly less incorrect information (d=0.09). Participants in the RI condition reported significantly more details pertaining to general and specific actions.Practical implicationsThe re-enactment instruction shows the potential to increase witness recall in a way that promotes recall of both additional correct information and investigative-relevant information.Originality/valueThe instruction provides witnesses a retrieval strategy that facilitates overcoming both the gap between memory availability and accessibility and the gap between memory availability and output regulation, eliciting more details with no significant increase of errors.

Comparing Grids With Vertical and Horizontal Item-by-Item Formats for PCs and Smartphones

Much research has been done comparing grids and item-by-item formats. However, the results are mixed, and more research is needed especially when a significant proportion of respondents answer using smartphones. In this study, we implemented an experiment with seven groups ( n = 1,476), varying the device used (PC or smartphone), the presentation of the questions (grids, item-by-item vertical, item-by-item horizontal), and, in the case of smartphones only, the visibility of the “next” button (always visible or only visible at the end of the page, after scrolling down). The survey was conducted by the Netquest online fieldwork company in Spain in 2016. We examined several outcomes for three sets of questions, which are related to respondent behavior (completion time, lost focus, answer changes, and screen orientation) and data quality (item missing data, nonsubstantive responses, instructional manipulation check failure, and nondifferentiation). The most striking difference found is for the placement of the next button in the smartphone item-by-item conditions: When the button is always visible, item missing data are substantially higher.

The effect of arousal on regulation of negative emotions using cognitive reappraisal: An ERP study

Because the effectiveness of the emotion regulation strategy cognitive reappraisal may vary with emotion intensity, we investigated how stimulus arousal affects reappraisal success. Participants up- and down-regulated emotional responses using cognitive reappraisal to low and high arousing unpleasant pictures while the electroencephalogram (EEG) was recorded. Up-regulation resulted in more negative self-reported valence, while down-regulation resulted in less negative self-reported valence regardless of stimulus arousal, suggesting that subjective reappraisal success does not vary with emotional intensity. Participants felt that down-regulation of emotional responses to low arousing unpleasant pictures was easiest, which is in line with previous findings that participants showed a greater preference for reappraisal in low than high arousing situations. The late positive potential (LPP) amplitude was enhanced by down-regulation of high arousing unpleasant pictures. Even though this effect was unexpected and is opposite to the typical effect of down-regulation on the LPP, it is in line with several previous studies. Potential explanations for LPP regulation effects in the unexpected direction, such as strategy selection and task design, are evaluated. Suggestions and recommendations for future research are discussed, including using trial-by-trial manipulation of regulation instructions and studying the effect of stimulus arousal on up- and down-regulation of positive emotions.

When Does More Money Work? Examining the Role of Perceived Fairness in Pay on the Performance Quality of Crowdworkers

This research adds to the rich discussion on whether increases in payment to crowdworkers lead to increases in performance quality by introducing the concept of perceived fairness in pay (PFP). PFP refers to the belief that one is fairly compensated for their work. We examine whether PFP mediates the impact of payment amount on the performance quality of crowdworkers. We conducted a field experiment with 152 crowdworkers performing a button-clicking (BC) task and an instructional manipulation check (IMC) task. PFP mediated the impact of payment amount on performance quality in the BC task but not in the IMC task. PFP also mediated the impact of payment amount on satisfaction and task time. Results suggest that PFP can help us better understand the relationship between payment and performance quality in crowdsourcing.

Survey Satisficing Inflates Stereotypical Responses in Online Experiment: The Case of Immigration Study.

Though survey satisficing, grudging cognitive efforts required to provide optimal answers in the survey response process, poses a serious threat to the validity of online experiments, a detailed explanation of the mechanism has yet to be established. Focusing on attitudes toward immigrants, we examined the mechanism by which survey satisficing distorts treatment effect estimates in online experiments. We hypothesized that satisficers would display more stereotypical responses than non-satisficers would when presented with stereotype-disconfirming information about an immigrant. Results of two experiments largely supported our hypotheses. Satisficers, whom we identified through an instructional manipulation check (IMC), processed information about immigrants' personality traits congruently with the stereotype activated by information provided about nationality. The significantly shorter vignette reading time of satisficers corroborates their time-efficient impression formation based on stereotyping. However, the shallow information processing of satisficers can be rectified by alerting them to their inattentiveness through use of a repeated IMC.

Effects of gender, rape-supportive attitudes, and explicit instruction on perceptions of women's momentary sexual interest.

Contemporary models of male-initiated sexual aggression toward female acquaintances implicate misperception of women's sexual interest. This study investigated the effects of gender, rape-supportive attitudes and an instructional manipulation on college students' sexual-interest judgments. Two hundred seventy-six women and 220 men judged the cues of momentary sexual interest expressed by photographed women; half received instruction on the differential validity of nonverbal cues of sexual interest for estimation of women's momentary sexual interest. Participants also completed an assessment of rape-supportive attitudes. Overall, college students' perceptions of women's momentary sexual interest are compromised both nomothetically and idiographically. Both male and female college students relied not only on women's nonverbal affect but also on the provocativeness of women's clothing and attractiveness when judging women's sexual interest. Men and women showed similar average ratings, but women relied more than men on women's affect, whereas men relied more than women on women's attractiveness. Both male and female students who endorsed more rape-supportive attitudes, relative to their peers, relied less on women's affect and more on women's clothing style and attractiveness. Explicit instruction regarding the greater validity of women's affective than nonaffective cues enhanced focus on nonverbal affective cues and decreased focus on clothing style and attractiveness. Although higher rape-supportive attitudes predicted more deficits in processing cues of sexual interest, explicit instruction proved to be effective for both higher-risk and lower-risk participants. These findings highlight the generalizability of the well-established effects of explicit instruction on category learning to sexual perception and may point to procedures that eventually could be incorporated into augmented prevention programs for sexual aggression on college campuses.

Distributional Effects and Individual Differences in L2 Morphology Learning

AbstractSecond language (L2) learning outcomes may depend on the structure of the input and learners’ cognitive abilities. This study tested whether less predictable input might facilitate learning and generalization of L2 morphology while evaluating contributions of statistical learning ability, nonverbal intelligence, phonological short‐term memory, and verbal working memory. Over three sessions, 54 adults were exposed to a Russian case‐marking paradigm with a balanced or skewed item distribution in the input. Whereas statistical learning ability and nonverbal intelligence predicted learning of trained items, only nonverbal intelligence also predicted generalization of case‐marking inflections to new vocabulary. Neither measure of temporary storage capacity predicted learning. Balanced, less predictable input was associated with higher accuracy in generalization but only in the initial test session. These results suggest that individual differences in pattern extraction play a more sustained role in L2 acquisition than instructional manipulations that vary the predictability of lexical items in the input.Open PracticesThis article has been awarded Open Materials and Open Data badges. All materials and data are publicly accessible via the Open Science Framework at https://osf.io/n8spz. Learn more about the Open Practices badges from the Center for Open Science: https://osf.io/tvyxz/wiki.

ERPs Differentially Reflect Automatic and Deliberate Processing of the Functional Manipulability of Objects.

It is known that the functional properties of an object can interact with perceptual, cognitive, and motor processes. Previously we have found that a between-subjects manipulation of judgment instructions resulted in different manipulability-related memory biases in an incidental memory test. To better understand this effect we recorded electroencephalography (EEG) while participants made judgments about images of objects that were either high or low in functional manipulability (e.g., hammer vs. ladder). Using a between-subjects design, participants judged whether they had seen the object recently (Personal Experience), or could manipulate the object using their hand (Functionality). We focused on the P300 and slow-wave event-related potentials (ERPs) as reflections of attentional allocation. In both groups, we observed higher P300 and slow wave amplitudes for high-manipulability objects at electrodes Pz and C3. As P300 is thought to reflect bottom-up attentional processes, this may suggest that the processing of high-manipulability objects recruited more attentional resources. Additionally, the P300 effect was greater in the Functionality group. A more complex pattern was observed at electrode C3 during slow wave: processing the high-manipulability objects in the Functionality instruction evoked a more positive slow wave than in the other three conditions, likely related to motor simulation processes. These data provide neural evidence that effects of manipulability on stimulus processing are further mediated by automatic vs. deliberate motor-related processing.

Implicitly learned suppression of irrelevant spatial locations.

How do we ignore a salient, irrelevant stimulus whose location is predictable? A variety of studies using instructional manipulations have shown that participants possess the capacity to exert location-based suppression. However, for the visual search challenges we face in daily life, we are not often provided explicit instructions and are unlikely to consciously deliberate on what our best strategy might be. Instead, we might rely on our past experience-in the form of implicit learning-to exert strategic control. In this paper, we tested whether implicit learning could drive spatial suppression. In Experiment 1, participants searched displays in which one location contained a target, while another contained a salient distractor. An arrow cue pointed to the target location with 70% validity. Also, unbeknownst to the participants, the same arrow cue predicted the distractor location with 70% validity. Results showed facilitated RTs to the predicted target location, confirming target enhancement. Critically, distractor interference was reduced at the predicted distractor location, revealing that participants used spatial suppression. Further, we found that participants had no explicit knowledge of the cue-distractor contingencies, confirming that the learning was implicit. In Experiment 2, to seek further evidence for suppression, we modified the task to include occasional masked probes following the arrow cue; we found worse probe identification accuracy at the predicted distractor location than control locations, providing converging evidence that observers spatially suppressed the predicted distractor locations. These results reveal an ecologically desirable mechanism of suppression, which functions without the need for conscious knowledge or externally guided instructions.

Answering Without Reading: IMCs and Strong Satisficing in Online Surveys

Some respondents of online surveys click responses at random. Screeners or instructional manipulation checks (IMC) have become customary for identifying this strong form of satisficing. This research first analyzes the factors that condition IMC failures using an online panel survey carried out in Spain (2011–2015). Our data show that the probability of passing a screener depends mainly on the screener’s difficulty, the individuals’ intrinsic motivations for answering the survey, and past failures. We then address the substantive consequences of omitting those who fail to pass IMCs. We find that this strategy introduces an additional source of bias in descriptive analyses. The article ends with a discussion of the implications that these findings have for the use of IMCs.

Sources of Placebo-Induced Relief From Nausea: The Role of Instruction and Conditioning.

It is well documented that expectancies alter the nauseous response. However, the lack of integration in research examining sources of expectancy has limited our understanding of how expectancies are formed and, consequently, our ability to intervene. The present study explored the role of both instructions and conditioning in placebo-induced relief from nausea. The study used a 2 × 2 between-subjects design with instruction and conditioning as factors with 56 healthy volunteers. The instruction manipulation involved randomizing participants to receive information that a sham treatment (a peppermint essence vapor) would reduce nausea or no such instructions. The conditioning manipulation involved further randomizing participants to have the first administration of this sham treatment paired with a surreptitious reduction in galvanic vestibular stimulation (GVS) intensity or no prior pairing. Nausea was induced through GVS. On test, all groups received the same level of GVS with the sham treatment present. On test, participants who received instruction had significantly lower nauseous response scores than those who did not (F(1,46) = 6.71, p = .013), and those who received conditioning also reported less nausea than those who did not (F(1,46) = 5.20, p = .027), with the interaction between the two not reaching statistical significance (F(1,46) = 2.33, p = .13). These findings indicate that placebo responding in nausea can be induced both through positive instructions and as little as one pairing of a treatment with a reduction in nausea, as well as their combination. This suggests that using placebo effects to complement antiemetic therapy may offer an important method of further reducing nausea in the clinic.

Testing the Level of Social Desirability During Job Interview on White-Collar Profession.

Social desirability as a tendency to present oneself in a better light rather than in a truthful manner is common feature presented during job interviews. Previous studies mainly focused on blue-collar professions and therefore authors researched contrary set of white-collar professions in three sub-studies with four different participant groups (legal professions; police officers; controls and university students influenced by scenarios; overall N = 636). It was hypothesized that candidates for legal profession would show similar tendency toward social desirability, when compared with controls. Furthermore, police officers were hypothesized to show similar levels of social desirability as legal professions. Lastly, participants in the instruction manipulation condition were hypothesized to show increased levels of social desirability in tender situation as compared to the honest situation. All groups were tested with balanced Inventory of Desirable Responding (BIDR, Paulhus, 1984). Statistical analyses revealed statistically significant differences for both subscales of BIDR when comparing legal professions and control group. Similarly, increased levels of social desirability were detected in police officer candidates as well as in university students in the tender situation compared with students in the honest situation. The overall results indicated that it is typical for white-collar candidates to adapt to the testing situation and it cannot be expected to see different behavior from legal profession candidates as was originally expected.

In a context of time: the impact of delay and exposure time on the emergence of memory context effects.

Research on context-mediated facilitation of recognition memory distinguishes between the effects of reinstating the exact same context previously associated with a target and a context that is familiar but not directly associated with the target. As both effects are difficult to produce reliably in recognition experiments, attention has turned to measures that may explain inconsistencies, such as the extent to which instructions encourage association between targets and contexts. The aim of the current study was to examine the distinctive and interactive effects of three factors that may lead to variability in context effects (CEs), namely type of instructions given at learning, delay between learning and test, and exposure time for targets and contexts at learning. Using a comprehensive paradigm developed by Vakil and colleagues, with photographs of faces serving as target and context stimuli, both exposure time and delay were shown to be associated with the occurrence of CEs and appeared to interact with one another in determining the nature of these effects. Unlike several previous studies, false alarms did not increase when foils were presented with familiar contexts. Also unexpectedly, the instruction manipulation did not appear to strengthen target-context binding. It may instead have increased attention to contexts at the expense of targets, as suggested by the finding that direct memory for context improved under associative instruction conditions. Overall, the study demonstrates the importance of understanding and controlling various factors that may potentially influence the emergence of both reinstatement and familiarity-based CEs, among them exposure time and learning-to-test delay.

Flipping the Managerial Accounting Principles Course: Effects on Student Performance, Evaluation, and Attendance

Abstract Purpose The purpose of the study is to examine the effects of “flipping the classroom” on student performance, evaluation, and attendance in managerial accounting principles. Design/methodology/approach The study uses a crossed within-participants research design (each student experiencing both traditional instruction and simplified flipped instruction) allowing for control of individual differences between students; repeated-measures regression analysis for overall effects; quantile regression for performance-segregated effects. Findings Flipping the classroom resulted in significant performance improvement, particularly for lower performing students. Course evaluations indicate a few instructor-related ratings were lower for the flipped approach. Attendance was lower under the flipped approach for initial class meetings where the instructional manipulation occurred. Research limitations/implications The study design included a weak form of flipping. A stronger form of flipping with greater incentives for class preparation as well as lecture videos could have stronger results. Practical implications Flipping the classroom could be effective for application-oriented accounting courses, particularly for lower performing students. Originality/value This is one of very few studies on flipping providing evidence of effectiveness using a crossed within-participants research design.

Resistance to instructed reversal of the learned predictiveness effect.

The learned predictiveness effect is a widely observed bias towards previously predictive cues in novel situations. Although the effect is generally attributed to an automatic attentional shift, it has recently been explained as the product of controlled inferences about the predictive value of cues. This view is supported by the susceptibility of learned predictiveness to instruction manipulation. However, recent research has shown conflicting results. Three experiments investigated the parameters of the instructed reversal effect in a human causal learning task, to determine the relative contribution of automatic and controlled attention processes. Experiment 1 showed that reversal instructions abolished, but did not reverse, the learned predictiveness effect, although length of initial training had no effect on the extent to which predictive cues subsequently captured attention. Experiment 2 explored whether particular causal scenarios lend themselves more readily to instructed reversal, but still failed to establish a significant reversal effect. Experiment 3 demonstrated a significant reversal effect when nonpredictive cues were explicitly and individually identified as the causes of outcomes. However, this effect was considerably weaker than the learned predictiveness effect when predictive cues were identified in the same way. Taken together, the results are inconsistent with a purely controlled account of learned predictiveness and provide support for dual-process theories of learning and attention.

It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks

Instructional manipulation checks (IMCs) have become popular tools for identifying inattentive participants in online studies. IMCs function by attempting to trick inattentive participants into responding incorrectly. However, from a conversational perspective, question characteristics are part of the researcher’s contribution to the conversation, and IMCs may teach participants that there is “more than meets the eye,” prompting systematic thinking on subsequent tricky-seeming questions in an attempt to avoid being tricked. In two online studies, participants responded to a simple task either before or after completing an IMC. As expected, answering an IMC prior to the task improved performance on items that benefit from increased systematic thinking—namely, the Cognitive Reflection Test (Study 1), and a probabilistic reasoning task (Study 2). We conclude that IMCs change attention rather than merely measure attention and discuss implications for their use in online studies.

The Impact of Instructions on Generalization of Conditioned Fear in Humans

Generalization of conditioned fear has been implicated in the maintenance and proliferation of fear in anxiety disorders. The role of cognitive processes in generalization of conditioning is an important yet understudied issue. Vervliet et al. (2010) tested generalization of fear to a visual stimulus of a particular color and shape paired with electric shock. Test stimuli shared either the color or shape of the CS+. Prior to conditioning, participants were instructed that either color or shape would be predictive of shock. Generalization was stronger to the stimulus containing the instructed feature, suggesting that instructions impacted generalization of fear. However, the result may also reflect the impact of instructions on attention and learning during the conditioning phase. In the present study, the instructional manipulation was given after the conditioning phase to control for any impact of instructions on learning. A similar result to that reported by Vervliet et al. was observed. On self-reported expectancy of shock, generalization was greater to the test stimulus that included the instructed stimulus feature. The same pattern was observed on skin conductance, although it did not reach statistical significance. The findings indicate that explicitly instructed information affected generalization of conditioned fear independently of any impact on learning, pointing to the role of cognitive processes in human fear generalization. They also support the utility of cognitive therapy approaches, which are employed after fear has already developed, in addressing clinical overgeneralization.

Automaticity and cognitive control in the learned predictiveness effect.

In novel contexts, learning is biased toward cues previously experienced as predictive compared with cues previously experienced as nonpredictive. This is known as learned predictiveness. A recent finding has shown that instructions issued about the causal status of cues influences the expression of learned predictiveness, suggesting that controlled, volitional processes play a role in this effect. Three experiments are reported further investigating the effects of instructional manipulations on learned predictiveness. Experiment 1 confirms the influence of inferential processes, extending previous work to suggest that instructions affect associative memory as well as causal reasoning. Experiments 2 and 3 used a procedure designed to tease apart inferential and automatic contributions to the bias by presenting instructed causes that were previously predictive and previously nonpredictive. The results demonstrate that the prior predictiveness of cues influences subsequent learning over and above the effect of explicit instruction. However, it appears that the relationship between explicit instruction and predictive history is interactive rather than additive. Potential explanations for this interactivity are discussed.

Manipulation of cognitive load variables and impact on auscultation test performance.

Health profession educators have identified auscultation skill as a learning need for health professional students. This article explores the application of cognitive load theory (CLT) to designing cardiac and respiratory auscultation skill instruction for senior-level undergraduate nursing students. Three experiments assessed student auscultation performance following instructional manipulations of the three primary components of cognitive load: intrinsic, extraneous, and germane load. Study 1 evaluated the impact of intrinsic cognitive load by varying the number of diagnoses learned in one instruction session; Study 2 evaluated the impact of extraneous cognitive load by providing students with single or multiple examples of diagnoses during instruction; and Study 3 evaluated the impact of germane cognitive load by employing mixed or blocked sequences of diagnostic examples to students. Each of the three studies presents results that support CLT as explaining the influence of different types of cognitive processing on auscultation skill acquisition. We conclude with a discussion regarding CLT's usefulness as a framework for education and education research in the health professions.