Multilevel Repeated Measures Analyses Research Articles

Investigating the promise of automated writing evaluation for supporting formative writing assessment at scale

ABSTRACT We investigated the promise of a novel approach to formative writing assessment at scale that involved an automated writing evaluation (AWE) system called MI Write. Specifically, we investigated elementary teachers’ perceptions and implementation of MI Write and changes in students’ writing performance in three genres from Fall to Spring associated with this implementation. Teachers in Grades 3–5 (n = 14) reported that MI Write was usable and acceptable, useful, and desirable; however, teachers tended to implement MI Write in a limited manner. Multilevel repeated measures analyses indicated that students in Grades 3–5 (n = 570) tended not to increase their performance from Fall to Spring except for third graders in all genres and fourth graders’ narrative writing. Findings illustrate the importance of educators utilising scalable formative assessments to evaluate and adjust core instruction.

Combining Effect and Process Evaluation on European Preschool Children's Snacking Behavior in a Kindergarten-Based, Family-Involved Cluster Randomized Controlled Trial: The ToyBox Study.

This study aimed at (1) studying the effect of the standardized ToyBox intervention on European preschoolers’ snacking behavior, and (2) studying whether a higher process evaluation score from teachers and parents/caregivers was associated with a more positive result for preschoolers’ snack intake. A sample of 4970 preschoolers (51.4% boys, 4.74 ± 0.44 years) from six European countries provided information on snack intake with the use of a Food Frequency Questionnaire. To investigate the effect of the intervention, multilevel repeated measures analyses were executed for the total sample and the six country-specific samples. Furthermore, questionnaires to measure process evaluation were used to compute a total process evaluation score for teachers and parents/caregivers. No significant intervention effects on preschoolers’ snack intake were found (all p > 0.003). In general, no different effects of the intervention on snack intake were found according to kindergarten teachers’ and parents’/caregivers’ process evaluation scores. The lack of effects could be due to limited intervention duration and dose. To induce larger effects on preschoolers’ snack intake, a less standardized intervention which is more tailored to the local needs might be needed.

Effectiveness of a family-, school- and community-based intervention on physical activity and its correlates in Belgian families with an increased risk for type 2 diabetes mellitus: the Feel4Diabetes-study

BackgroundThe study aimed to investigate the effectiveness of the European Feel4Diabetes intervention, promoting a healthy lifestyle, on physical activity and its correlates among families at risk for type 2 diabetes mellitus (based on the Finnish Diabetes Risk Score) in Belgium.MethodsThe Feel4Diabetes intervention involved three components: family, school and community component, with the family component consisting of 6 counseling sessions for families at risk. Main outcomes were objectively measured physical activity levels and its subjectively measured correlates. The final sample consisted of 454 parents (mean age 39.4 years; 72.0% women) and 444 children (mean age 8.0 years; 50.1% girls). Multilevel repeated measures analyses were performed to assess intervention effectiveness after 1 year.ResultsIn parents, there was no significant intervention effect. In children, there were only significant negative effects for moderate to vigorous physical activity (p = 0.05; ηp2 = 0.008) and steps (p = 0.03; ηp2 = 0.006%) on weekdays, with physical activity decreasing (more) in the intervention group.ConclusionsThe F4D-intervention lacks effectiveness on high-risk families’ physical activity and its correlates in Belgium. This could partially be explained by low attendance rates and a large drop-out. To reach vulnerable populations, future interventions should invest in more appropriate recruitment (e.g. more face-to-face contact) and more bottom-up development of the intervention (i.e. co-creation of the intervention with the target group).Trial registrationThe Feel4Diabetes-study was prospectively registered at clinicaltrials.gov as NCT02393872 on 20 March 2015.

Effects of Enriched Thematic Multi-Sensory Stimulation on BPSD in A Beach Room: A Pilot Study among Nursing-Home Residents with Dementia

Background: Studies of the effectiveness of psychosocial interventions indicate that sensory stimulation is a promising approach to reduce psychological and behavioural disturbances in people with dementia. A relatively new approach is enriched multi-sensory stimulation. Enriched multi-sensory stimulation includes the simultaneous stimulation of multiple senses by stimuli that are related to positive themes. Methods: A pilot - randomized controlled trial was conducted with repeated measurements and two conditions: (1) the experimental condition involving enriched multi-sensory stimulation in a thematic Beach Room; (2) the control condition involving visits to the nursing home’s standard Grand Café room without enriched multi-sensory stimulation. 49 nursing-home residents were randomly assigned to one of the two conditions, in which they participated three times a week for 30 minutes during 12 weeks. Outcomes were measured at three points in time. The primary outcomes - depression, agitation and apathy - were measured with the MDS-Depression Rating Scale, the Cohen-Mansfield Agitation Inventory and the Apathy Evaluation Scale respectively. The secondary outcomes — sleep-wake patterns, observed behaviour and mood — were measured with the Actiwatch activity monitor, and the INTERACT and FACE observation scales respectively. Multilevel repeated measures analyses were used for analyses. Results: In total 35 residents completed the follow-up: 18 residents in the Beach Room and 17 in the Grand Café environment. Sleep improved among residents who visited the Grand Café environment. These residents also showed fewer feelings of depression. No improvements in the selected primary and secondary outcome measures were found among the residents who visited the Beach Room. Conclusions: Enriched multi-sensory stimulation is not necessarily more effective in reducing psychological and behavioural symptoms among nursing-home residents with dementia than stimulation within a less enriched environment. More research is needed.

Effect and process evaluation of a real-world school garden program on vegetable consumption and its determinants in primary schoolchildren.

ObjectivesThis study aimed to investigate the effectiveness of a school garden program on children’s vegetable consumption and determinants and to gain insight into the process of the program.MethodsThe “Taste Garden” is a real-world nine-week school garden program developed and implemented by a local organization. A total of 350 children (149 intervention group, 201 control group) filled out questionnaires on vegetable consumption, determinants and process of the program. Additionally, teachers filled out a process evaluation questionnaire. For effect evaluation, interaction effects (time x group) were considered, using multilevel repeated measures analyses in MLwiN 3.02. Interaction effects were repeated, taking into account quality of implementation (time x implementation group). Process evaluation was descriptively assessed with SPSS 24.0.ResultsOverall, beside some practical concerns of teachers, the program was well perceived by teachers and children. However, an intervention effect of “The Taste Garden” was only found for knowledge (p = 0.02), with a very small effect size (0.55%). When taking into account implementation quality, only small effects were found for awareness (p between 0.005 and 0.007 and an effect size of 0.63%) and knowledge (p between 0.04 and 0.09 and an effect size of 0.65%).CconclusionsEvaluation of the real-world “Taste Garden” program, which was positively perceived by teachers, showed no effects on vegetable consumption and small effects on its determinants. Adaptations of the current format and longer follow-up periods are therefore recommended.

"Can't live with or without them:" Transitions and young adults' perceptions of sibling relationships.

Extant research documents how siblings' relationships develop from childhood through adolescence; yet, we know little about how sibling relationships change in young adulthood. Rooted in life course theory, this 2-wave longitudinal study investigated changes in sibling closeness and conflict, and the roles of life transitions and sibling similarity in life stage. Participants included 273 young adults from 180 families who reported on 340 sibling relationships (Time 1 M age = 24.45, SD = 5.33; Time 2 M age = 30.23, SD = 5.33). Multilevel repeated measures analysis of covariance indicated that, on average, siblings' perceptions of conflict declined over the course of young adulthood. Additionally, patterns of change in closeness and conflict were linked to life transitions surrounding coresidence, parenthood, and similarity in employment. The discussion addresses findings in regard to life course theory and similarity in life transitions. (PsycINFO Database Record

Effect and process evaluation of a kindergarten-based, family-involved cluster randomised controlled trial in six European countries on four- to six-year-old children\u2019s steps per day: the ToyBox-study

BackgroundThe ToyBox-intervention is a theory- and evidence-based intervention delivered in kindergartens to improve four- to six-year-old children’s energy balance-related behaviours and prevent obesity. The current study aimed to (1) examine the effect of the ToyBox-intervention on increasing European four- to six-year-old children’ steps per day, and (2) examine if a higher process evaluation score from teachers and parents was related to a more favourable effect on steps per day.MethodsA sample of 2438 four- to six-year-old children (51.9% boys, mean age 4.75 ± 0.43 years) from 6 European countries (Belgium, Bulgaria, Germany, Greece, Poland and Spain) wore a motion sensor (pedometer or accelerometer) for a minimum of two weekdays and one weekend day both at baseline and follow-up to objectively measure their steps per day. Kindergarten teachers implemented the physical activity component of the ToyBox-intervention for 6 weeks in total, with a focus on (1) environmental changes in the classroom, (2) the child performing the actual behaviour and (3) classroom activities. Children’s parents received newsletters, tip cards and posters. To assess intervention effects, multilevel repeated measures analyses were conducted for the total sample and the six intervention countries separately. In addition, process evaluation questionnaires were used to calculate a total process evaluation score (with implementation and satisfaction as a part of the overall score) for teachers and parents which was then linked with the physical activity outcomes.ResultsNo significant intervention effects on four- to six-year-old children’ steps per weekday, steps per weekend day and steps per average day were found, both in the total sample and in the country-specific samples (all p > 0.05). In general, the intervention effects on steps per day were least favourable in four- to six-year-old children with a low teachers process evaluation score and most favourable in four- to six-year-old children with a high teachers process evaluation score. No differences in intervention effects were found for a low, medium or high parents’ process evaluation score.ConclusionThe physical activity component of the ToyBox-intervention had no overall effect on four- to six-year-old children’ steps per day. However, the process evaluation scores showed that kindergarten teachers that implemented the physical activity component of the ToyBox-intervention as planned and were satisfied with the physical activity component led to favourable effects on children’s steps per day. Strategies to motivate, actively involve and engage the kindergarten teachers and parents/caregivers are needed to induce larger effects.

Multibehavioural Interventions with a Focus on Specific Energy Balance-Related Behaviours Can Affect Diet Quality in Preschoolers from Six European Countries: The ToyBox-Study.

The present study aimed to examine whether a multibehavioural intervention with a focus on specific energy balance-related behaviours can affect total diet quality and its four subcomponents in European preschoolers and to investigate if these intervention effects differed by socioeconomic status (SES). Parents/caregivers of 3.5 to 5.5 year-olds (n = 4968) recruited through kindergartens in six European countries within the ToyBox-study completed questionnaires on socio-demographics and a food frequency questionnaire on their preschoolers’ diet. To assess intervention effects and differences by SES, multilevel repeated measures analyses were conducted. In contrast to no significant difference in total diet quality, in both the intervention and control group, the dietary quality and dietary equilibrium increased, with a larger increase in the intervention group (mean difference quality: +3.4%; mean difference equilibrium: +0.9%) compared to the control group (quality: +1.5%; equilibrium: +0.2%). SES was not a significant moderator for intervention effects on total diet quality, nor for the four subcomponents. This study indicates that multibehavioural interventions with a focus on specific energy balance-related behaviours in preschoolers not only affect those targeted behaviours, but can also have more generalized effects. The ToyBox-intervention effects were similar for both lower and high SES preschoolers.

Effect and process evaluation of a kindergarten-based, family-involved intervention with a randomized cluster design on sedentary behaviour in 4- to 6- year old European preschool children: The ToyBox-study.

BackgroundThe aim of the present study evaluated the effect and process of the ToyBox-intervention on proxy-reported sedentary behaviours in 4- to 6-year-old preschoolers from six European countries.MethodsIn total, 2434 preschoolers’ parents/primary caregivers (mean age: 4.7±0.4 years, 52.2% boys) filled out a questionnaire, assessing preschoolers’ sedentary behaviours (TV/DVD/video viewing, computer/video games use and quiet play) on weekdays and weekend days. Multilevel repeated measures analyses were conducted to measure the intervention effects. Additionally, process evaluation data were included to better understand the intervention effects.ResultsPositive intervention effects were found for computer/video games use. In the total sample, the intervention group showed a smaller increase in computer/video games use on weekdays (ß = -3.40, p = 0.06; intervention: +5.48 min/day, control: +8.89 min/day) and on weekend days (ß = -5.97, p = 0.05; intervention: +9.46 min/day, control: +15.43 min/day) from baseline to follow-up, compared to the control group. Country-specific analyses showed similar effects in Belgium and Bulgaria, while no significant intervention effects were found in the other countries. Process evaluation data showed relatively low teachers’ and low parents’ process evaluation scores for the sedentary behaviour component of the intervention (mean: 15.6/24, range: 2.5–23.5 and mean: 8.7/17, range: 0–17, respectively). Higher parents’ process evaluation scores were related to a larger intervention effect, but higher teachers’ process evaluation scores were not.ConclusionsThe ToyBox-intervention had a small, positive effect on European preschoolers’ computer/video games use on both weekdays and weekend days, but not on TV/DVD/video viewing or quiet play. The lack of larger effects can possibly be due to the fact that parents were only passively involved in the intervention and to the fact that the intervention was too demanding for the teachers. Future interventions targeting preschoolers' behaviours should involve parents more actively in both the development and the implementation of the intervention and, when involving schools, less demanding activities for teachers should be developed.Trial registrationclinicaltrials.gov NCT02116296

A Teacher-Focused Intervention to Decrease PE Students’ Amotivation by Increasing Need Satisfaction and Decreasing Need Frustration

Intervention-induced gains in need satisfaction decrease PE students' amotivation. The present study adopted a dual-process model to test whether an intervention could also decrease need frustration and hence provide a second supplemental source to further decrease students' PE amotivation. Using an experimental, longitudinal research design, 19 experienced PE teachers (9 experimental, 10 control) and their 1,017 students participated in an intervention program to help teachers become both more autonomy supportive and less controlling. Multilevel repeated measures analyses showed that students of teachers in the experimental group reported greater T2, T3, and T4 perceived autonomy support, need satisfaction, and engagement and lesser T2, T3, and T4 perceived teacher control, need frustration, and amotivation than did students of teachers in the control group. Multilevel structural equation modeling analyses confirmed the hypothesized dual-process model in which both intervention-induced increases in need satisfaction and intervention-induced decreases need frustration decreased students' end-of-semester amotivation. We discuss the theoretical and practical implications of this new finding on the dual antecedents of diminished amotivation.

Effect and Process Evaluation of a Cluster Randomized Control Trial on Water Intake and Beverage Consumption in Preschoolers from Six European Countries: The ToyBox-Study.

BackgroundWithin the ToyBox-study, a kindergarten-based, family-involved intervention was developed to prevent overweight and obesity in European preschoolers, targeting four key behaviours related to early childhood obesity, including water consumption. The present study aimed to examine the effect of the ToyBox-intervention (cluster randomized controlled trial) on water intake and beverage consumption in European preschoolers and to investigate if the intervention effects differed by implementation score of kindergartens and parents/caregivers.MethodA sample of 4964 preschoolers (4.7±0.4 years; 51.5% boys) from six European countries (Belgium, Bulgaria, Germany, Greece, Poland, Spain) was included in the data analyses. A standardized protocol was used and parents/caregivers filled in socio-demographic data and a food-frequency questionnaire. To assess intervention effects, multilevel repeated measures analyses were conducted for the total sample and for the six country-specific samples. Based on the process evaluation questionnaire of teachers and parents/caregivers, an implementation score was constructed. To assess differences in water intake and beverage consumption by implementation score in the total sample, multilevel repeated measures analyses were performed.ResultsLimited intervention effects on water intake from beverages and overall beverage consumption were found. However, important results were found on prepacked fruit juice consumption, with a larger decrease in the intervention group compared to the control group. However, also a decline in plain milk consumption was found. Implementation scores were rather low in both kindergartens and parents/caregivers. Nevertheless, more favorable effects on beverage choices were found in preschoolers whose parents/caregivers and kindergarten teachers had higher implementation scores compared to those with lower implementation scores.ConclusionThe ToyBox-intervention can provide the basis for the development of more tailor-made interventions. However, new strategies to improve implementation of interventions should be created.

The effect of a kindergarten-based, family-involved intervention on objectively measured physical activity in Belgian preschool boys and girls of high and low SES: the ToyBox-study

BackgroundThe ToyBox-study developed an evidence- and theory-based intervention to improve preschoolers’ energy balance-related behaviours – including physical activity (PA) – by targeting the kindergarten environment and involving their parents/caregivers. The present study aimed to examine the effect of the ToyBox-intervention on increasing Belgian preschoolers’ objectively measured PA levels.MethodsA sample of 472 preschoolers (4.43 ± 0.55 years; 55.1% boys) from 27 kindergartens (15 intervention, 12 control kindergartens) in Flanders, Belgium were included in the data analyses. Preschoolers wore an ActiGraph accelerometer for six consecutive days and were included in the data analyses if they had a minimum of two weekdays and one weekend day, both at baseline and follow-up (one year later). Preschoolers’ PA outcomes were estimated for an average day, weekday, weekend day, during school hours, and during after school hours. To assess intervention effects, multilevel repeated measures analyses were conducted for the total sample, and for sub-groups (according to sex, kindergarten levels of socio-economic status (SES) and risk groups (low levels of PA at baseline)) of preschoolers.ResultsSmall intervention effects were found in the total sample. Most intervention effects were found in boys and in preschoolers from high SES kindergartens. Boys from the intervention group had an increase in vigorous PA (ß = 1.47, p = 0.03) and moderate-to-vigorous PA (ß = 1.27, p = 0.03) from baseline to follow-up, whereas PA levels in boys from the control group stagnated or decreased. In preschoolers from high SES kindergartens, the largest effects were found for PA outcomes during school hours and during after school hours.ConclusionThe results from the Belgian sample demonstrate that effects of the PA-component of the ToyBox-intervention on objectively measured PA were found in preschool boys and in preschoolers from high SES kindergartens, which means that the ToyBox-intervention was mainly effective in those sub-groups. Future interventions should search for alternative strategies to increase preschoolers’ PA levels in preschool girls and preschoolers from low SES kindergartens, as these are the most important at-risk groups regarding PA.

An Evaluation of Six Brief Interventions That Target Drug-Related Problems in Correctional Populations

Finding brief effective treatments for criminal justice populations is a major public need. The CJ-DATS Targeted Intervention for Corrections (TIC), which consists of six brief interventions (communication, anger, motivation, criminal thinking, social networks, and HIV/sexual health), was tested in separate federally-funded randomized control studies. In total, 1,573 criminal justice-involved individuals from 20 correction facilities participated (78 % males; 54 % White). Multilevel repeated measures analyses found significant gains in knowledge, attitudes, and psychosocial functioning (criteria basic to knowledge, attitude, and practices (KAP) and TCU treatment process models). While improvements were less consistent in criminal thinking, overall evidence supported efficacy for the TIC interventions.

Male methamphetamine-user inmates in prison treatment: During-treatment outcomes

Psychosocial functioning and criminal thinking of methamphetamine-using inmates were examined before and after their completion of primary treatment in three in-prison drug treatment programs (one “outpatient” and two different modified therapeutic communities [TCs]). The sample consisted of 2,026 adult male inmates in 30 programs in Indiana. Data included background, psychosocial functioning, criminal thinking, and therapeutic engagement indicators. Multilevel repeated measures analysis was used to evaluate changes during treatment, and multilevel covariate analysis adjusted for sample differences in tests of between-treatment differences. Significant improvements were found for all three treatments, but the two modified TCs showed significantly better progress than did outpatient treatment housed among the general prison population. Significant predictors of treatment progress included baseline psychosocial functioning and background, wherein higher psychosocial functioning and lower criminal thinking orientation predicted stronger therapeutic engagement. However, treatment engagement level was found to mediate during-treatment improvement and initial criminal thinking.

During-Treatment Outcomes Among Female Methamphetamine-Using Offenders in Prison-Based Treatments

An increasingly important treatment group is the expanding population of methamphetamine-using female offenders. This study focused on women methamphetamine-using offenders (n = 359) who were treated either in a modified therapeutic community (TC) program (“Clean Lifestyle is Freedom Forever” [CLIFF]-TC: n = 234) designed for non-violent offenders with significant impairment from methamphetamine use, or the standard outpatient (OTP) treatment (OTP: n = 125). All participants were assessed on motivation, psychological and social functioning, and treatment engagement before and during treatment. A multilevel repeated measures analysis examined changes between intake and end of Phase 2 treatment. Both CLIFF-TC and the traditional OTP treatments were shown to improve psychosocial functioning, with significant changes on measures of self esteem, depression, anxiety, decision making, hostility, risk taking, and criminal thinking errors. Effect size comparisons indicated treatment gains were larger in the CLIFF-TC than in the OTP group. Both groups rated treatment engagement measures of participation, satisfaction, and counselor rapport to be very high. These results have positive implications for managing and improving treatment of methamphetamine-using women offenders because psychological improvements during treatment have been linked to better post release outcomes.

Multilevel Modeling of Longitudinal Growth and Change: Substantive Effects or Regression Toward the Mean Artifacts?

Regression toward the mean artifacts (RTMAs) are ubiquitous phenomena, seducing researchers, policy makers, and practitioners down the path of offering substantive interpretations of statistical artifacts. The use of highly sophisticated statistical tools can mislead otherwise knowledgeable researchers into thinking that this well-known problem is no longer relevant. Here we evaluated multilevel models of growth and change in relation to RMTAs, using simulated data to represent students nested within schools for which there were initial school differences due to selection based on T1 (pretest) achievement scores, but no school differences in achievement growth over four years. In the unconditional multilevel growth modeling approach (i.e., a multilevel repeated measures analysis), there were substantial RTMAs; apparent growth over time was more positive for schools with initially lower T1 school-average achievement and the sizes of these RTMAs varied inversely with the simulated reliability and stability of the measures. In the conditional multilevel covariance approach (i.e., multilevel path models with adjustment for pretest covariates) there were no RTMAs, but some advantages of the growth modeling approach were lost. In a hybrid approach (growth models of residualized change scores) in which T1 achievement was considered as a covariate and growth model components were constructed from T2, T3, and T4 achievement, RTMAs associated with initial achievement differences were eliminated. Because assignment to schools in this example was completely determined by a known T1 (pretest) achievement score, both the conditional and hybrid approaches eliminated RTMAs, but none of the approaches would be generally effective if selection was based on unmeasured variables. The results demonstrate that multilevel growth models provide no protection from RTMAs that can easily be misinterpreted as substantively meaningful results rather than statistical artifacts.