Evaluation Of Learning Curve Research Articles

Learning curve evaluation using cumulative summation analysis—a clinical example of pediatric robot-assisted laparoscopic pyeloplasty

BackgroundThe cumulative summation (CUSUM) method for learning curve analysis remains under-utilized in the surgical literature in general, and is described in only a small number of publications within the field of pediatric surgery. This study introduces the CUSUM analysis technique and applies it to evaluate the learning curve for pediatric robot-assisted laparoscopic pyeloplasty (RP). MethodsClinical data were prospectively recorded for consecutive pediatric RP cases performed by a single-surgeon. CUSUM charts and tests were generated for set-up time, docking time, console time, operating time, total operating room time, and postoperative complications. Conversions and avoidable operating room delay were separately evaluated with respect to case experience. Comparisons between case experience and time-based outcomes were assessed using the Student's t-test and ANOVA for bi-phasic and multi-phasic learning curves respectively. Comparison between case experience and complication frequency was assessed using the Kruskal–Wallis test. ResultsA total of 90 RP cases were evaluated. The learning curve transitioned beyond the learning phase at cases 10, 15, 42, 57, and 58 for set-up time, docking time, console time, operating time, and total operating room time respectively. All comparisons of mean operating times between the learning phase and subsequent phases were statistically significant (P=<0.001–0.01). No significant difference was observed between case experience and frequency of post-operative complications (P=0.125), although the CUSUM chart demonstrated a directional change in slope for the last 12 cases in which there were high proportions of re-do cases and patients <6months of age. ConclusionsThe CUSUM method has a valuable role for learning curve evaluation and outcome quality monitoring. In applying this statistical technique to the largest reported single surgeon series of pediatric RP, we demonstrate numerous distinctly shaped learning curves and well-defined learning phase transition points.

Evaluation of the learning curve for a board-certified veterinary surgeon performing laparoendoscopic single-site ovariectomy in dogs.

To define the learning curve and evaluate the outcome for a board-certified veterinary surgeon performing laparoendoscopic single-site (LESS) ovariectomy in dogs. Retrospective case review and learning curve evaluation with a skill acquisition model. Animals-27 client-owned dogs. Procedures-Between April 2011 and December 2012, 27 dogs underwent elective LESS ovariectomy performed by a single experienced board-certified laparoscopic surgeon by means of the same technique. Medical records for these patients were reviewed to determine whether a learning curve could be detected. A commercially available multitrocar port was inserted through a 15- to 20-mm incision at the umbilicus, and LESS ovariectomy was performed with articulating graspers, a bipolar vessel-sealing device, and a 30° telescope. Surgical performance of the surgeon was quantified with an exponential skill acquisition model, and how skill was gained with repetition of the same novel surgical procedure was examined. Median patient body weight was 20 kg (44 lb; range, 3.5 to 41 kg [7.7 to 90.2 lb]). Median surgical time was 35 minutes (range, 20 to 80 minutes). Median patient age was 314 days (range, 176 to 2,913 days). The skill acquisition model revealed that a comparable surgeon could reach 90% of optimal surgery performance after approximately 8 procedures (8.6, 95% confidence interval, 0.5 to 16.6 procedures). According to the model, with each surgery, surgical time would be expected to decrease by 27% (95% confidence interval, 2% to 52%). Complications were limited to minor hemorrhage due to a splenic laceration and a postoperative incisional infection. Follow-up information was available for all 27 cases. All owners were satisfied and indicated that they would pursue LESS ovariectomy again. The learning curve for LESS ovariectomy was short and definable. Short-term outcome was excellent. Results of this study suggested that an experienced laparoscopic surgeon may anticipate achieving proficiency with this technique after performing approximately 8 procedures.

Systematic Review of Learning Curves for Minimally Invasive Abdominal Surgery

To determine how minimally invasive surgical learning curves are assessed and define an ideal framework for this assessment. Learning curves have implications for training and adoption of new procedures and devices. In 2000, a review of the learning curve literature was done by Ramsay et al and it called for improved reporting and statistical evaluation of learning curves. Since then, a body of literature is emerging on learning curves but the presentation and analysis vary. A systematic search was performed of MEDLINE, EMBASE, ISI Web of Science, ERIC, and the Cochrane Library from 1985 to August 2012. The inclusion criteria are minimally invasive abdominal surgery formally analyzing the learning curve and English language. 592 (11.1%) of the identified studies met the selection criteria. Time is the most commonly used proxy for the learning curve (508, 86%). Intraoperative outcomes were used in 316 (53%) of the articles, postoperative outcomes in 306 (52%), technical skills in 102 (17%), and patient-oriented outcomes in 38 (6%) articles. Over time, there was evidence of an increase in the relative amount of laparoscopic and robotic studies (P < 0.001) without statistical evidence of a change in the complexity of analysis (P = 0.121). Assessment of learning curves is needed to inform surgical training and evaluate new clinical procedures. An ideal analysis would account for the degree of complexity of individual cases and the inherent differences between surgeons. There is no single proxy that best represents the success of surgery, and hence multiple outcomes should be collected.

Irreversible Electroporation Ablation (IRE) of Unresectable Soft Tissue Tumors: Learning Curve Evaluation in the First 150 Patients Treated

BackgroundIrreversible electroporation (IRE) is a novel technology that uses peri-target discrete probes to deliver high-voltage localized electric current to induce cell death without thermal-induced coagulative necrosis. “Learnability” and consistently effective results by novice practitioners is essential for determining acceptance of novel techniques. This multi-center prospectively-collected database study evaluates the learning curve of IRE.MethodsAnalysis of 150 consecutive patients over 7 institutions from 9/2010-7/2012 was performed with patients treated divided into 3 groups A (1st 50 patients treated), B (2nd 50) and C (3rd 50 patients treated) chronologically and analyzed for outcomes.ResultsA total of 167 IRE procedures were performed, with a majority being liver(39.5%) and pancreatic(35.5%) lesions. The three groups were similar with respect to co-morbidities and demographics. Group C had larger lesions (3.9vs3cm,p=0.001), more numerous lesions (3.2vs2.2,p=0.07), more vascular invasion(p=0.001), underwent more associated procedures(p=0.001) and had longer operative times(p<0.001). Despite this, they had similar complication and high-grade complication rates(p=0.24). Attributable morbidity rate was 13.3%(total 29.3%) and high-grade complications were seen in 4.19%(total 12.6%). Pancreatic lesions(p=0.001) and laparotomy(p=0.001) were associated with complications.ConclusionThe review represents that single largest review of IRE soft tissue ablation demonstrating initial patient selection and safety. Over time, complex treatments of larger lesions and lesions with greater vascular involvement were performed without a significant increase in adverse effects or impact on local relapse free survival. This evolution demonstrates the safety profile of IRE and speed of graduation to more complex lesions, which was greater than 5 cases by institution. IRE is a safe and effective alternative to conventional ablation with a demonstrable learning curve of at least 5 cases to become proficient.

Comparative Outcomes and Assessment of Trifecta in 500 Robotic and Laparoscopic Partial Nephrectomy Cases: A Single Surgeon Experience

We report a comparative analysis of a large series of laparoscopic and robotic partial nephrectomies performed by a high volume single surgeon at a tertiary care institution. We retrospectively reviewed the medical charts of 500 patients treated with minimally invasive partial nephrectomy by a single surgeon between March 2002 and February 2012. Demographic and perioperative data were collected and statistically analyzed. R.E.N.A.L. (radius, exophytic/endophytic properties, nearness of tumor to the collecting system or sinus in mm, anterior/posterior and location relative to polar lines) nephrometry score was used to score tumors. Those scored as moderate and high complexity were designated as complex. Trifecta was defined as a combination of warm ischemia time less than 25 minutes, negative surgical margins and no perioperative complications. Two groups were identified, including 261 patients with robotic and 231 with laparoscopic partial nephrectomy. Demographics were similar in the groups. The robotic group was significantly more morbid (Charlson comorbidity index 3.75 vs 1.26), included more complex tumors (R.E.N.A.L. score 5.98 vs 7.2), and had lower operative (169.9 vs 191.7 minutes) and warm ischemia (17.9 vs 25.2 minutes) time, intraoperative (2.6% vs 5.6%, each p <0.001) and postoperative (24.53% vs 32.03%, p = 0.004) complications, and positive margin rate (2.9% vs 5.6%, p <0.001). Thus, a higher overall trifecta rate was observed for robotic partial nephrectomy (58.7% vs 31.6%, p <0.001). The laparoscopic group had longer followup (3.43 vs 1.51 years, p <0.001) and no significant difference in postoperative changes in renal function. Main study limitations were the retrospective nature, arbitrary definition of trifecta and shorter followup in the RPN group. Our large comparative analysis shows that robotic partial nephrectomy offers a wider range of indications, better operative outcomes and lower perioperative morbidity than laparoscopic partial nephrectomy. Overall, the quest for trifecta seems to be better accomplished by robotic partial nephrectomy, which is likely to become the new standard for minimally invasive partial nephrectomy.

Cost Analysis of Pediatric Robot-Assisted and Laparoscopic Pyeloplasty

An increasing percentage of pediatric pyeloplasties are being performed with assistance of the da Vinci® Surgical System. A review of the recent literature shows decreased operative times and length of hospital stays when robotic procedures are performed, although there are few published data comparing the cost of pediatric robotic and pure laparoscopic pyeloplasty. We reviewed a representative sample of pyeloplasties performed at our institution and performed a cost analysis. We retrospectively identified 23 robot-assisted and 23 laparoscopic pyeloplasties performed at our institution between August 2008 and April 2012. Total cost was calculated from direct and indirect costs provided by our billing department. Robotic procedures were shorter than pure laparoscopic procedures (200 vs 265 minutes, p <0.001) but there was no significant difference in the total cost of the 2 procedures ($15,337 vs $16,067, p <0.46). When compared to laparoscopic cases, subgroup analysis demonstrated decreased operative times (140 vs 265 minutes, p <0.00001) and total cost ($11,949 vs $16,067, p <0.0001) in robotic cases where stents were placed in an antegrade fashion. With widespread use the cost of robotic instrumentation may decrease, and experience may further shorten operative times. However, it currently remains to be seen whether robotic technology will become a cost-effective replacement for pure laparoscopy in the management of pediatric ureteropelvic junction obstruction.

Robotic-assisted Versus Traditional Laparoscopic Partial Nephrectomy: Comparison of Outcomes and Evaluation of Learning Curve

To examine the transition to robot-assisted laparoscopic partial nephrectomy (RALPN) from pure laparoscopic partial nephrectomy (LPN) and investigate the learning curve (LC). RALPN has emerged as a minimally invasive alternative to nephron-sparing surgery. A total of 150 consecutive patients were identified who underwent LPN or RALPN in the initial experience of a single surgeon since 2006. The perioperative data were evaluated using appropriate comparative tests. The LC was investigated by examining the operative times, warm ischemia times (WITs), and estimated blood loss (EBL) in groups of 25 consecutive patients. To account for laparoscopic LC, the outcomes of patients who underwent surgery in 2009 or later were also compared. Of the 150 patients, 102 and 48 underwent LPN and RALPN, respectively. The patient and tumor characteristics were similar. The mean operative time (193 vs 152 minutes, P < .001), WIT (18.0 vs 14.0, P < .001), and EBL (245 vs 122 mL, P = .001) favored RALPN. Improvements in the operative time (P = .01), WIT (P = .006), and EBL (P = .01) were noted as experience increased in the LPN cohort and was most pronounced after the first 25 LPN patients. Since 2009, 55 and 44 patients underwent LPN and RALPN, respectively. Although the absolute differences were less, the operative time (182 vs 150, P < .001), WIT (15.3 vs 13.3, P < .001), and EBL (206 vs 118, P = .005) favored RALPN. RALPN appears to have shorter operative and ischemia times and less blood loss compared with LPN. After a LC of approximately 25 cases, the transition from LPN to RALPN can be undertaken without an additional LC and can be associated with immediate benefits.

Complications After Robotic Partial Nephrectomy at Centers of Excellence: Multi-Institutional Analysis of 450 Cases

We evaluated the incidence of perioperative complications after robotic partial nephrectomy. We retrospectively reviewed the records of patients treated with robotic assisted partial nephrectomy across the 4 participating institutions. Demographic, blood loss, warm ischemia time, and intraoperative and postoperative complication data were collected. All complications were graded according to the Clavien classification system. A total of 450 consecutive robotic assisted partial nephrectomies were done between June 2006 and May 2009. Overall 71 patients (15.8%) had a complication, including intraoperative and postoperative complications in 8 (1.8%) and 65 (14.4%), respectively. Hemorrhage developed in 2 patients (0.2%) intraoperatively and in 22 (4.9%) postoperatively. Seven patients (1.6%) had urine leakage. As classified by the Clavien system, complications were grade I-II in 76.1% of cases and grade III-IV in 23.9%. Robotic assisted partial nephrectomy was converted to open or conventional laparoscopic surgery in 3 patients (0.7%) and to radical nephrectomy in 7 (1.6%). There were no deaths. Current data indicate that robotic assisted partial nephrectomy is safe. Most postoperative complications are Clavien grade I or II, or can be managed conservatively.

Comparison of the Learning Curve and Outcomes of Robotic Assisted Pediatric Pyeloplasty

We compared the learning curve and outcomes in children undergoing robotic assisted laparoscopic pyeloplasty during the initiation of a robotic surgery program compared to the benchmark of open pyeloplasty. The records of our first consecutive 33 children undergoing robotic assisted laparoscopic pyeloplasty from 2006 to 2009 were retrospectively reviewed and compared to those of age and gender matched children who underwent open repair done by senior faculty surgeons before the initiation of our robotic surgery program. We compared operative time, complications, postoperative pain, length of stay and surgical success for 2 surgeons who adopted the robotic approach at an academic teaching institution. We found no significant differences in length of stay, pain score or surgical success at a median followup of 16 months. The number of complications was similar and they tended to be early and technical in the robotic assisted laparoscopic pyeloplasty group. Overall average operative time was 90 minutes longer (38%) for robotic assisted laparoscopic pyeloplasty (p <0.004). When evaluated chronologically, there was evidence of a learning curve. After 15 to 20 robotic cases overall operative times for robotic assisted laparoscopic cases was consistently within 1 SD of our average open pyeloplasty time with no significant difference in overall operative time (p = 0.23). Of the decrease in overall operative time 70% was due to decreased pyeloplasty time rather than peripheral time. There was similar safety and efficacy with robotic assisted laparoscopic pyeloplasty, although complications tended to be technical and early in our initial experience. Operative time decreased with experience and after 15 to 20 cases it was similar to that of open pyeloplasty with similar outcomes and surgical success.

Assessment and learning curve evaluation of endobronchial ultrasound skills following simulation and clinical training

Endobronchial ultrasound is a revolutionary diagnostic pulmonary procedure. The use of a computer endobronchial ultrasound simulator could improve trainee procedural skills before attempting to perform procedures on patients. This study aims to compare endobronchial ultrasound performance following training with simulation versus conventional training using patients. A prospective study of pulmonary medicine and thoracic surgery trainees. Two cohorts of trainees were evaluated using simulated cases with performance metrics measured by the simulator. Group 1 received endobronchial ultrasound training by performing 15 cases on an endobronchial ultrasound simulator (n=4). Group 2 received endobronchial ultrasound training by doing 15-25 cases on patients (n=9). Total procedure time was significantly shorter in group 1 than group 2 (15.15 (±1.34) vs 20.00 (±3.25) min, P<0.05). The percentage of lymph nodes successfully identified was significantly better in group 1 than group 2 (89.8 (±5.4) vs 68.1 (±5.2), P < 0.05). There was no difference between group 1 and group 2 in the percentage of successful biopsies (100.0 (±0.0) vs 90.4 (±11.5), P=0.13). The learning curves for simulation trained fellows did not show an obvious plateau after 19 simulated cases. Using an endobronchial ultrasound simulator leads to more rapid acquisition of skill in endobronchial ultrasound compared with conventional training methods, as assessed by an endobronchial ultrasound simulator. Endobronchial ultrasound simulators show promise for training with the advantage of minimizing the burden of procedural learning on patients.

Complications in 2200 Consecutive Laparoscopic Radical Prostatectomies: Standardised Evaluation and Analysis of Learning Curves

BackgroundLaparoscopic radical prostatectomy (LRP) represents an established treatment modality for localised prostate cancer. ObjectiveTo report standardised complication rates for LRP, evaluate the development of complication rates over time, and show changes within the learning curves of laparoscopic surgeons. Design, setting, and participantsWe conducted a standardised analysis of 2200 consecutive patients who underwent LRP between 1999 and 2008 at a single institution. InterventionLRP was performed using a transperitoneal (n=871) or extraperitoneal (n=1329) retrograde Heilbronn technique. Five surgeons operated on 96% of the patients. MeasurementsComplications were classified according to the modified Clavien system. Total complication rates and changes over time were analysed. Three generations of surgeons were defined for evaluation of learning curves. Results and limitationsMinor complications occurred in 21.7% of patients (Clavien 1: 6.8%; Clavien 2: 14.9%); anaemia requiring transfusion (10.4%) dominated. Early reinterventions were necessary in 6.7% of patients (Clavien 3a: 3.6%; Clavien 3b: 1.5%; Clavien 4a: 1.5%; Clavien 4b: 0.1%). Late Clavien 3b complications occurred in 4.7% of patients—most of them anastomotic strictures. Mortality was 0.1% (Clavien 5). There was a significant decrease in overall complication rates over time, resulting predominantly from decreasing Clavien 1–2 events. Learning curves of third-generation surgeons plateaued earlier compared to the first generation (250 vs 700 cases). The limitation of this study is that data concerning comorbidity were not included. ConclusionsLRP is a safe procedure characterised by an acceptable profile of complications. Specifically, few major complications are reported. According to the complication rates, the learning curve of third-generation surgeons is significantly shorter compared to first- and second-generation surgeons.

Robot-Assisted Partial Nephrectomy: Evaluation of Learning Curve for an Experienced Renal Surgeon

The learning curve for robot-assisted partial nephrectomy (RAPN) has not been extensively studied. We therefore evaluated the learning curve of RAPN for a fellowship-trained laparoscopic surgeon with extensive prior experience with laparoscopic partial nephrectomy (LPN). We also examined the potential effect of tumor size on the learning curve. We prospectively evaluated 38 consecutive patients undergoing RAPN by a single surgeon (S.B.B.). Sixteen patients had tumors <2 cm, and 22 patients had tumors >2 cm. Warm ischemia times and overall operative times were recorded as indices of learning progression. Average operative time for tumors <2 cm was 131.9 minutes (115.3-148.5 minutes) and for tumors >2 cm was 145.8 minutes (131.1-160.5 minutes). The difference between the operative times for tumors <2 and >2 cm was not statistically significant (p = 0.23). Average warm ischemia time for tumors <2 cm was 21 minutes (16.9-25.1 minutes) and for tumors >2 cm was 24.7 minutes (21.3-28.1 minutes). This difference was also not statistically significant (p = 0.20). Defined by the overall operative time, the learning curve for RAPN was 16 cases, and by ischemic time, the learning curve was 26 cases. Tumor size did not have an effect on the learning curve. The learning curve for RAPN is short for surgeons already experienced with LPN. The learning curve for portions performed under warm ischemia is slightly longer, implying that the critical portions of the procedure require more experience to become facile. Tumor size does not appear to have a significant impact on the learning curve for surgeons experienced with LPN.

Virtual Dissection CT Colonography: Evaluation of Learning Curves and Reading Times with and without Computer-aided Detection

To prospectively evaluate the learning curves and reading times of inexperienced readers who used the virtual dissection reading method for retrospective computed tomographic (CT) colonography data sets, with and without concurrent computer-aided detection (CAD). An Institutional Review Board approved this study; informed consent was waived. Four radiologists without experience in CT colonography evaluated 100 optical colonoscopy-proved data sets of 100 patients (49 men, 51 women; mean age, 59 years +/- 13 [standard deviation]; range, 21-85 years) by using the virtual dissection reading method. Two readers used concurrent CAD. Data sets were read during five consecutive 1-day sessions (20 data sets per session). Polyp detection and false-positive rates, receiver operating characteristics (ROCs), and reading times were calculated for individual, CAD group, and non-CAD group readings. Diagnostic values were compared by calculating the 95% confidence intervals (CIs) around the relative risk. Areas under ROC curves (AUCs) (Hanley and McNeil for paired analysis and z statistics for unpaired analysis) and reading times (Wilcoxon signed rank test) were compared across the sessions, within each session and for the whole study. The range of detection rates was 79 of 111 (.71 [95% CI: .61, .79]) to 91 of 111 (.82 [95% CI: .73, .88]). The range of false-positive rates was 17 of 111 (.15 [95% CI: .09, .23]) to 22 of 111 (.20 [95% CI: .12, .28]). All readers' AUCs rose from session 1 to session 4; this rise was significant (P < .05) for the non-CAD group. Only during session 1 was the CAD group AUC (.83) higher than the non-CAD group AUC (.54) (P < .05). Comparison of CAD and non-CAD reading times showed no significant difference for the whole study or during each session (P > .05). The virtual dissection reading technique allows short learning curves, which may be improved by the concurrent use of CAD, without significant effect on average reading time.

Robot-assisted laparoscopic cholecystectomy: Evaluation of learning curve, advantages and limitations

Robot-assisted laparoscopic cholecystectomy: Evaluation of learning curve, advantages and limitations

Routine ileoscopy at colonoscopy: a prospective evaluation of learning curve and skill-keeping line

Ileoscopy is the criterion standard for mucosal lesions of terminal ileum but is performed in 5% of colonoscopy activity, and the need for training is undefined. To assess the learning curve and skill-keeping line of ileoscopy. Prospective randomized study. Single GI endoscopy unit. Adult outpatients referred for colonoscopy. Patient randomization to ileocolonoscopy by trainees and seniors. Ileoscopy skill was divided into the following: ileocecal valve (ICV) intubation (success rate, time) and ileal exploration (length, time). Consecutive blocks of 10 procedures were used to calculate the learning curve and the skill-keeping line. Variables considered to influence ICV intubation were the following: endoscopist experience, ICV morphology, patient age, gender, body mass index, diverticular disease, and preceding abdominal/pelvic surgery. Learning curve showed that competency in ICV intubation (80% success rate) and ileal exploration (16 cm) was achieved after 50 procedures. The skill-keeping line showed that ileoscopy was easy (97% success rate), fast (1 minute), and well accepted up to 45 cm of the ileum. Thin-lipped and volcanic ICV are the most difficult and easiest to intubate, respectively. Crohn's disease and adenomas of the proximal edge of ICV were diagnosed in 2.2%. Training in ileoscopy is recommended to achieve competency, and endoscopists should consider practicing ICV intubation to maintain and increase their skill. Ileoscopy can be difficult when the ICV is thin lipped or single bulged, and easy when it has a volcanic morphology.

Prospective evaluation of learning curve for laparoscopic radical prostatectomy: Identification of factors improving operative times

Objectives To evaluate the time required for each component of laparoscopic radical prostatectomy (LRP) performed by a single surgeon to identify the factors that expedite the learning curve. LRP is a technically demanding procedure with a lengthy learning curve. Methods The LRP procedure was divided into 12 steps, and the time for each step was prospectively recorded during the first 50 consecutive patients undergoing LRP by a single surgeon. The operations were divided into five groups of 10, and the average times for each step were compared and correlated with surgeon observations and changes in surgical technique. Results Statistically significant progressive improvement was seen in the total time of the LRP procedure (269.4 minutes in period 1 versus 205.4 minutes in period 4, P <0.05). Regarding the specific steps, improvement occurred in the time needed for dissection of the vas deferens and seminal vesicles (51.8 minutes for period 1 versus 25.3 minutes for period 4, P <0.01 and 31.2 minutes for period 5, P <0.03), apical incision (16.7 minutes for period 1 versus 6.3 minutes for period 4, P <0.03 and 5.7 minutes for period 5, P <0.02), and division of the rectourethralis (13.5 minutes for period 1 versus 3.4 minutes for period 5, P <0.05). The time needed for vesicourethral anastomosis remained unchanged throughout the study (average 48.5 minutes). Conclusions With experience, the operative times for defined components of LRP improve. Patient selection for a low body mass index and smaller prostate may expedite the procedure. Application of a fixed retractor system, early identification of the vas deferens beneath the peritoneum in a more lateral position, and slow meticulous dissection and ligation of the dorsal vein complex are factors that may expedite the learning curve for LRP.

Nonparametric Evaluation of Learning Curves by Hierarchical Contingeney Analysis

AbstractHierarchical contingency analysis (HCA) is derived from the Perli‐Hommel‐Lehmacher (1986) closed test procedure for nonparametrical evaluation of learning curves of a 2 x 2‐factorial experiment. By HCA, univariate main effects are detected without Bonferroni alpha adjustment, as is shown by a numerical example from gold fish shock avoidance conditioning. Alternative approaches to nonparametrical evaluation of MANOVA designs with and without repeated measurements are discussed.