Web-based Assessments Research Articles

Comparison of office‐based versus home web‐based clinical assessments for Parkinson's disease

The feasibility and validity of Web-based assessments in Parkinson's disease is unknown. The objectives of this study were to develop and to compare home Web-based assessments with office-based assessments. We tested feasibility and validity using a longitudinal, randomized crossover design. Patients were assessed at baseline and after 6 and 12 weeks using both assessments including the Unified Parkinson's Disease Rating Scale, the Unified Dyskinesia Rating scale, timed tests, and quality-of life and Non-Motor Symptoms questionnaires. Forty-two patients were included (22 men, 20 women; mean age, 64.7 ± 9.0 years). Only 2 patients (5%) dropped out. The mean intraclass correlation coefficient between Web- and office-based assessments ranged from 0.67 (first visit) to 0.75 (last visit) and 0.81 and 0.82 for doctor- and patient-administered scales, respectively. No differences in responsiveness (P = 0.63), and data precision (P = 0.11) were found, but Web-based assessments had fewer missing values (P = 0.01). Web-based assessments offer a feasible format for assessing PD-related impairment from home.

Synchronous Collection of Multisource Feedback Evaluations Does Not Increase Inter‐rater Reliability

Most multisource feedback (MSF) evaluations are performed asynchronously, with raters reflecting on the subject's behavior. Numerous studies have demonstrated poor inter-rater reliability of MSF. This may be due to cognitive biases that are inherent in such a process. We sought to determine if within- and between-rater group reliability is increased when evaluations are gathered synchronously and relate to a specific patient interaction. This was a survey at a university emergency department (ED) of 30 emergency medicine (EM) residents. ED nurses and faculty anonymously participated in asynchronous MSF assessment of resident performance from February to April 2010 using a Web-based survey, the Emergency Medicine Humanism Scale (EM-HS). In May 2010, a second round of MSF collection was conducted in the ED. At the conclusion of patient encounters, the EM-HS was synchronously obtained from ED nurses and faculty. Evaluators were instructed to assess the resident based on the patient encounter, placing aside any preconceptions of resident performance, attitude, or behavior. Evaluators rated resident performance using a 1-9 scale ("needs improvement" to "outstanding"). The mean rating for each of the questions and the total score provided by each evaluator class was calculated for each EM resident. Differences between the asynchronous and synchronous ratings were compared with t-tests. Pearson correlations were used to measure agreement in scores within and between nurse and faculty rater groups. Correlations > 0.70 were deemed acceptable and are reported with 95% confidence intervals (CIs). Twenty-one of 30 residents had assessments collected by both asynchronous and synchronous methods. A total of 699 Web-based (asynchronous) assessments were completed by nurses and 149 by faculty. Synchronous nurse and faculty assessments were obtained in 105 resident-patient encounters. There was no difference in faculty ratings between the MSF collection methods. Nurses assigned slightly (but significantly) higher ratings during synchronous collection. Correlation of the total MSF score between asynchronous and synchronous feedback collection methods within the faculty rater group was poor (0.18, 95% CI = -0.22 to 0.60). Correlation of the total MSF score between asynchronous and synchronous feedback collection methods within nurse rater groups was moderate (0.63, 95% CI = 0.27 to 0.83). Correlations between faculty-nurse rater groups for the total MSF collected asynchronously and synchronously were moderate (0.39, 95% CI = -0.05 to 0.7; and 0.44, 95% CI 0.01 to 0.73, respectively). Synchronous collection of MSF did not provide clinically different EM-HS scores within rater groups and did not result in improved correlations. Our small, single-center study supports asynchronous collection of MSF.

Social constraints, genetic vulnerability, and mental health following collective stress

A repeat-length polymorphism of the serotonin promoter gene (5-HTTLPR) has been associated with depression and posttraumatic stress disorder (PTSD) in trauma-exposed individuals reporting unsupportive social environments. We examine the contributions of the triallelic 5-HTTLPR genotype and social constraints to posttraumatic stress (PTS) symptoms in a national sample following the September 11, 2001 (9/11) terrorist attacks. Saliva was collected by mail from 711 respondents (European American subsample n = 463) of a large national probability sample of 2,729 adults. Respondents completed web-based assessments of pre-9/11 mental and physical health, acute stress 9 to 23 days post-9/11, PTS symptoms, and social constraints on disclosure regarding fears of future terrorist attacks 2-3 years post-9/11. Social constraints were positively associated with PTS symptoms 2-3 years post-9/11. The triallelic 5-HTTLPR genotype was not directly associated with PTS symptoms, but it interacted with social constraints to predict PTS symptoms 2-3 years post-9/11: Social constraints were more strongly associated with PTS symptoms for individuals with any s/lg allele than for homozygous la/la individuals. Constraints on disclosing fears about future terrorism moderate the 5-HTTLPR genotype-PTS symptom association even when indirectly exposed to collective stress.

Targeted Transition Assessment Leading to Job Placement for Young Adults with Disabilities in Rural Areas

In rural areas, transition planning for young adults with disabilities presents considerable challenges, including linkage to employment upon completion of special education services. Two assessment programs, described in this article, work in concert to assist transition teachers and youth with disabilities in rural areas and include: (a) web-based job preference and job matching assessments, and (b) a social network assessment. An internet-based assessment ( www.yesjobsearch.com ) presents 2-4 min of video for up to 120 entry-level jobs and allows a job seeker to select preferred ones. A job matching assessment determines the extent to which the job seeker's current skill level matches preferred jobs. Once preferred and well-matched jobs are identified, the social network assessment identifies family, friends, acquaintances, and resources associated with the job seeker's most-preferred/best-matched job. The article describes use of these assessments to facilitate job placement for youth with disabilities in rural areas.

Shyness mindset: Applying mindset theory to the domain of inhibited social behavior

This study applies mindset theory to the domain of inhibited social behavior. Incoming college freshmen ( N = 93) completed web-based assessments upon beginning college and 7 months later. Time 1 shyness mindset predicted changes in performance anxiety from Time 1 to Time 2; those who viewed inhibited social behavior as fixed showed increases in performance anxiety and those that viewed shyness as malleable showed decreases. The effect was partially mediated through college belongingness. Shyness mindset did not moderate changes in interaction anxiety. More research is warranted to determine if shyness mindset is a useful target for intervention during the transition to a new social environment.

Validation of an Online Assessment of Orthopedic Surgery Residents' Cognitive Skills and Preparedness for Carpal Tunnel Release Surgery

Surgical competence requires both cognitive and technical skills. Relatively little is found in the literature regarding the value of Web-based assessments to measure surgery residents' mastery of the competencies. To determine the validity and reliability of 2 online instruments for predicting the cognitive preparedness of residents for performing carpal tunnel release surgery. Twenty-eight orthopedic residents and 2 medical school students responded to an online measure of their perception of preparedness and to an online cognitive skills assessment prior to an objective structured assessment of technical skills, in which they performed carpal tunnel release surgery on cadaveric specimens and received a pass/fail assessment. The 2 online assessments were analyzed for their internal reliability, external correlation with the pass/fail decision, and construct validity. The internal consistency of the perception of preparedness measure was high (α = .92) while the cognitive assessment was less strong (α = .65). Both instruments demonstrated moderately strong correlations with the pass/fail decision, with Spearman correlation of .606 (P = .000) and .617 (P = .000), respectively. Using logistic regression to analyze the predictive strength of each instrument, the perception of preparedness measure demonstrated a 76% probability (η(2) = .354) and the cognitive skills assessment a 73% probability (η(2) = .381) of correctly predicting the pass/fail decision. Analysis of variance modeling resulted in significant differences between levels at P < .005, supporting good construct validity. The online perception of preparedness measure and the cognitive skills assessment both are valid and reliable predictors of readiness to successfully pass a cadaveric motor skills test of carpal tunnel release surgery.

Validation of Web-based Psychological Assessments

Validation of Web-based Psychological Assessments

A tailored internet-plus-email intervention for increasing physical activity among ethnically-diverse women

Objective To evaluate the feasibility and efficacy of an individually tailored, Internet-plus-email physical activity intervention designed for adult women. Method Healthy and ethnically-diverse adult females ( N = 156) (mean age = 42.8 years, 65% Caucasian) from California were randomly assigned to an intervention (access to a tailored website and weekly emails) or wait-list control group. Participants completed web-based assessments of physical activity, stage of behavior change, and psychosocial variables at baseline, one month, two months, and three months. Data were collected during 2006–2007. Multilevel random coefficient modeling examined group differences in rates of change. Results As compared to the control condition, the intervention group increased walking (+ 69 versus + 32 min per week) and total moderate-to-vigorous physical activity (+ 23 versus − 25 min per week) after three months. The intervention did not impact stage of behavior change or any of the other psychosocial variables. Conclusion A tailored, Internet-based intervention for adult women had a positive effect on walking and moderate-to-vigorous physical activity in an ethnically-diverse sample. However, given the lack of comparable research contact in the control group, these findings should be taken cautiously.

Reliability, compliance, and security in web-based course assessments

Pre- and postcourse assessment has become a very important tool for education research in physics and other areas. The web offers an attractive alternative to in-class paper administration, but concerns about web-based administration include reliability due to changes in medium, student compliance rates, and test security, both question leakage and utilization of web resources. An investigation was carried out in introductory astronomy courses comparing pre- and postcourse administration of assessments using the web and on paper. Overall no difference was seen in performance due to the medium. Compliance rates fluctuated greatly, and factors that seemed to produce higher rates are identified. Notably, email reminders increased compliance by 20%. Most of the 559 students complied with requests to not copy, print, or save questions nor use web resources; about 1% did copy some question text and around 2% frequently used other windows or applications while completing the assessment.

A Comparison of Web-Based Concept Mapping Tasks for Alternative Assessment in Distance Teacher Education

Three sections of the same distance education class completed a series of Web-based concept map assessments using one of two methods. Open-ended maps applied in section 1 led students to conduct more relational thinking overall, but variance in map items was very high introducing more subjectivity in scoring. Preselected term mapping applied in sections 2—3 led students to correctly classify many concepts and express proper relationships compared to an instructor's map. Identifying expected concept sets from the instructor's maps caused students some difficulty. The high volume of readings associated with the task appeared to further this problem. Students touted many benefits of mapping, including synthesizing and connecting course material, reading more intentionally, and thinking critically.

Social Motives and the Interaction Between Descriptive and Injunctive Norms in College Student Drinking

Social norms are a key determinant of young adult drinking, yet little research has evaluated potential interactive effects among different types of norms. The present research was designed to evaluate perceptions of friends' approval of drinking (i.e., injunctive norms) as a moderator of the relationship between perceived prevalence of friends' drinking (i.e., descriptive norms) and personal alcohol consumption. We also evaluated whether social drinking motives further influence this relationship. Participants included 1,400 first-year college students (61% women) who completed Web-based assessments of descriptive and injunctive norms, personal drinking, and social drinking motives. Results revealed that both descriptive and injunctive norms regarding close friends were uniquely and positively associated with drinking behavior. The relationship between perceived descriptive norms and personal alcohol consumption was stronger among those who also perceived their friends as being more approving of drinking but only among students who reported stronger social motives for drinking. Descriptive and injunctive norms are distinct constructs and are not interchangeable, having both unique and interactive effects, and vary as a function of social motives. Results are considered in terms of their implications for brief interventions.

Are Social Norms the Best Predictor of Outcomes Among Heavy-Drinking College Students?

This research was designed to evaluate the relative contribution of social norms, demographics, drinking motives, and alcohol expectancies in predicting alcohol consumption and related problems among heavy-drinking college students. Participants included 818 (57.6% women) first-year undergraduates who reported at least one heavy-drinking episode in the previous month. In addition to providing demographic information (gender and fraternity/sorority membership) participants completed Web-based assessments of social norms (perceived descriptive norms regarding typical student drinking, injunctive norms regarding friends' and parents' approval), motives (social, enhancement, coping, and conformity), and expectancies and evaluations of positive and negative alcohol effects. Regression results indicated that descriptive and injunctive norms were among the best predictors of college student drinking. With respect to alcohol problems, results indicated that coping motives accounted for the largest proportion of unique variance. Finally, results revealed that alcohol consumption mediated the relationships between predictors and problems for social norms, whereas coping motives, negative expectancies, and evaluation of negative effects were directly associated with alcohol problems despite having relatively weak or null unique associations with consumption. The results of this study substantiate social norms as being among the best predictors of alcohol consumption in this population and suggest that drinking to cope is a better predictor of problems. The findings are discussed in terms of practical prevention and treatment implications.

Digital Photography and Web-based Assignments in an Urban Field Course: Snapshots from Berlin

This paper reflects on the use of digital photography in urban-based human geography fieldwork. It draws on the authors' experience of introducing digital photography into the teaching and assessment of a level 3 undergraduate field course in Berlin. To begin they outline how they sought to use simple digital technologies in order to facilitate student engagement within Berlin and to widen the range of assessment avenues available to students. They concentrate in particular on how it provided the opportunity to develop Web-based assessments. In the second half of the paper they reflect on student experience of the use of digital cameras and Web-based assessment. While acknowledging some of the problems with using technology in this way, they conclude by arguing that digital photography not only enhances the quality of student engagement with urban environments but also provides a way of more effectively integrating the field-, class- and Web-based components of urban field courses.

Social Anxiety as a Moderator of the Relationship Between Perceived Norms and Drinking

College students overestimate the drinking of their peers, and the more they overestimate, the more they drink. The present research was designed to evaluate social anxiety as a moderator of the relationship between perceived norms and drinking among college men and women. Participants included 1,217 first-year residence-hall students (62.8% women) who completed Web-based assessments of social anxiety, perceived norms, and self-reported drinking. Results replicated previous research in that students overestimated the drinking of their peers (d = 0.75, p < .001). Students who had higher social anxiety drank somewhat more but did not differ from students who had lower social anxiety on perceived norms. However, the relationship between perceived norms and drinking was stronger among students who had higher social anxiety (d = 0.92, p < .001) relative to less socially anxious students (d = 0.02, p = NS). Higher levels of social anxiety were associated with a stronger relationship between perceived norms and drinking for both men (d = 0.86, p < .001) and women (d = 0.50, p < .001) but stronger for men (d = 0.26, p < .001). These results corroborate previous literature, which suggests that social factors are important determinants of drinking in this population and suggest that social anxiety is associated with susceptibility to peer influences on drinking. Additional work evaluating whether reductions in social anxiety may ameliorate the impact of perceived norms on drinking would be worthwhile.

Tuning medical education for rural-ready practice: designing and resourcing optimally

In an effort to bring doctors back to the bush the Australian government has resourced a number of rural clinical schools (RCS). At the RCS in the University of Western Australia students were allocated in small groups to rural sites for the entire fifth year of a six-year course, sitting the same final examinations as city students. Key factors guiding the successful outcome were the resourcing and implementation of the infrastructure and teaching and learning pedagogy. In designing support, the disconnection of students from their city colleagues was anticipated as an issue, as was the pedagogical indoctrination of the teachers. The curriculum implementation was adapted in this light. The role of the Web in teaching and learning, and their status as ‘student colleagues’ and independent learners were pivotal aspects. As students settled at their site, their confidence grew and their anxiety over urban disconnection dissipated. By benchmarking themselves using Web-based formative assessments and in formative ‘objective structured clinical examinations’ staged for them by the RCS, the students received ongoing feedback on their progress. This model of embedding students in rural centres for an extended period with rural practitioners as teachers was successfully implemented at multiple sites geographically vastly separate.

Student Behavior in the Virtual Environment: Using Web-based Quizzes to Promote Student Learning

This exploratory comparative study examined the efficacy of Web-based formative assessments in promoting student performance on in-class summative assessments. Online quizzes were incorporated into one section of a foundational undergraduate social work class (N= 25). Pre- and post-tests were administered to the experimental group (N= 25) and a comparison group (N= 25) which did not have access to the formative assessments. A comparison of pre- and post-test mean differences neared significance (p = .051), with students who took the Web-based quizzes performing only slightly better than those who did not. Instructor evaluations and final course grades were analyzed for triangulation purposes. Implications for the use of instructional technology to broaden pedagogical approaches are discussed.

The Use of the Internet in Psychological Research: Comparison of Online and Offline Questionnaires

The Internet can be an effective medium for the posting, exchange, and collection of information in psychology-related research and data. The relative ease and inexpensiveness of creating and maintaining Web-based applications, associated with the simplicity of use via the graphic-user interface format of form-based surveys, can establish a new research frontier for the social and behavioral sciences. To explore the possible use of Internet tools in psychological research, this study compared Web-based assessment techniques with traditional paper-based methods of different measures of Internet attitudes and behaviors in an Italian sample. The collected data were analyzed to identify both differences between the two samples and in the psychometric characteristics of the questionnaires. Even if we found significant differences between the two samples in the Internet attitudes and behaviors, no relevant differences were found in the psychometric properties of the different questionnaires. This result, similar to the ones previously obtained in Web-based assessments of personality constructs, is even more interesting given the lack of control on the characteristics of the online sample. These finding suggests that, if sampling control and validity assessment is provided, Internet-based questionnaires can be a suitable alternative to more traditional paper-based measures.

Where Did All the Data Go? Internet Security for Web‐Based Assessments

The purpose of this article is to enumerate a series of security steps that might be taken by those researchers or organizations that are contemplating web‐based tests and performance assessments. From a security viewpoint, much of what goes on with web‐based transactions is similar to other general computer activity, but the recommendations here focus on what can be done to avoid the loss, compromising, or modification of data collected by or stored through the Internet.

Getting Up to Speed on the Information Highway: Integrating Web-based Resources into Business Communication Pedagogy

Building on student and faculty enthusiasm for Web resources, we addressed four significant questions as we revised a business communications class to incorporate the Internet: • What Web resources should we include? • How do we make sure we select high quality sites and assessment tools? • How do we effectively integrate Web tools with existing pedagogies? • Will students find Web assignments valuable, easy to access, and useful for course assignments? We integrated Web resources with other course pedagogies in the undergraduate business communications class and identified Websites relevant for teaching employment communication, including e-mail etiquette, online resumes, and elec tronic job searches. Student evaluations of our Web-enhanced course indicate that most found the Web-based assessments and tools accessible and valuable in fur thering their self-understanding and in leading them to explore other sites not required for class. We developed guidelines for using the new technology to support teaching and learning strategies.

Computer-based performance assessments: a solution to the narrow measurement and reporting of problem-solving

Although performance assessments test for higher order thinking and problem solving, they rarely report students' thinking process data back to teachers, students, or the public. Web-based database-backed performance assessments provide a viable means for concurrently reporting both performance and thinking process data. In the research conducted here, we report our findings from a study that assessed student problem solving using networked computers. Both performance and process data could be reported back to teachers and students such that they could diagnose and understand how they performed and what problem-solving processes contributed to or detracted from their performance.