Measuring the amorphous: Substantive and methodological outcomes from concept maps

Abstract

AbstractBackgroundAccreditation organizations broadened program assessment criteria in ways that present challenges for the evaluation of learning outcomes. This is especially the case in courses where there are not narrowly defined questions or definitive solutions, such as engineering ethics. While protocols for learning assessment exist, there is limited research exploring sociotechnical learning outcomes in a manner that combines theoretical, empirical, and procedural aspects of assessment.Purpose/HypothesisThis paper shares 3 years of research into the effectiveness of concept maps as an assessment tool for engineering students in courses that prioritize professional skills development and sociotechnical thinking. We show that concept maps can offer evidence of knowledge formation and learning outcomes associated with courses that introduce complex problems with multiple possible interpretations or viable solutions.Design/MethodA concept mapping activity was completed by 614 undergraduate engineering students at the start and end of three different courses to evaluate sociotechnical thinking. Student‐level longitudinal changes were evaluated using paired t tests and simple linear regression. The concept mapping activity was improved iteratively in response to pilot tests and focus groups.ResultsUndergraduate engineers demonstrated greater complexity in the representation of sociotechnical relationships as observed in the structure and content of the concept maps. The methodological results offer lessons about the administration and analysis of concept maps often excluded from conversations on course and curriculum level assessment.ConclusionsConcept maps offer insights into student learning outcomes, and they can be deployed and analyzed with minimal resources. However, assessments must be carefully designed to account for administrative and analytical challenges.