Measuring Materials Research: The Material Advancement Progression, or MAP Scale

Abstract

Scientists support their theories with numerical data, measured quantities, and objective metrics. Can academic research be measured in a similar manner? Here, we propose a new scale to measure the progress of materials science: the Material Advancement Progression (MAP) Scale. Scientists support their theories with numerical data, measured quantities, and objective metrics. Can academic research be measured in a similar manner? Here, we propose a new scale to measure the progress of materials science: the Material Advancement Progression (MAP) Scale. Science is both a collaborative and self-aggregating endeavor, building upon the past success and works of others (and oneself). Many technologies are motivated by the continuous and progressive research of a developing materials system, concept, or idea, as human depth of knowledge increases and technological tools are improved. There is no endgame for scientific progress—it is in constant motion. At the same time, science is intimately linked with objective measurement. The scientific method is rooted in systematic observation and experiment to test a hypothesis. One could say that without measure, there is no objectivity in observation. Combining these two defining characteristics—constant progression and measurement—it would be highly beneficial if there existed a means to measure scientific research. A standard metric to compare some facet of study “A” with study “B,” so to speak. For better or for worse, scientists’ desire to somehow assess research has resulted in a focus on citations. The thought process is that if a certain paper is cited more often, it has more value than a paper cited less often. Superficially it makes sense—authors tend to cite works that support their findings, so highly cited papers must be the foundation of science! As a result, when it comes to judging research success, universities and collegiate peers seem to cling to the notion of journal impact factors. Entire meta-studies can be written on the use (and abuse) of impact factors, which is beyond the scope of the discussion here. The key point is that the impact factor of a journal—or the number of citations for a paper, or the h-index of an author—do not convey any information about the contents of the publication or work. Impact factors, h-indices, and related bibliometrics reflect scientific output and citations, but they fail to assess the research itself. It’s akin to listing the ratings of a television show (e.g., total number of viewers) when I ask what the show is about. Beyond bibliometrics, how can one measure scientific research? Materials science is an interesting case, as materials frequently serve as a link from fundamental discovery to enabled technological applications. Material systems that end up in widespread use undergo a journey, from initial discovery, through the course of research and development, to ultimate societal impact. Indeed, materials researchers typically work on challenges with such a journey to this final societal use in mind. As drivers of technology, a key impetus for materials development is to exploit their properties for some structural or functional goal. While many papers tout the ultimate applications in the introduction or conclusion (e.g., “The behavior demonstrated herein will ultimately lead to cutting-edge biomedical implants…” or similar), how close they are to that application is typically ambiguous. A reader may ponder, if the work is a proof-of-concept study, or a brand-new observed phenomena. Are the findings more sophisticated experiments extended from prior work? When will I see this material in practical use? Such questions very much span the scope of Matter, from nano to macro and from fundamental science to application. Matter aims to be a platform to demonstrate such progress, a moment in history as materials are developed. Each manuscript can be thought of as a “snapshot” of this journey (Figure 1). Indeed, one way to look at academic journals is as contemporary albums—a collection of snapshots of cutting-edge science. Just as your parents’ wedding photographs differ from your children’s (in terms of fashion trends and hair styles), research articles of today differ from those in the early 20th century in terms of findings, characterization approaches, and even literary style. If we consider each manuscript as a moment-in-time in a particular material’s journey, it would be prudent to measure how far along the path it may be by using a metric to measure research to indicate and classify the progress from discovery to societal impact (and not just bibliometrics). Matter has developed a novel numerical scale to quantify the research progress—a Material Advancement Progression (MAP) scale—to assess (as well as categorize) the development state of each published material. The MAP scale was partially motivated by the concept of technology readiness levels (TRLs) used for product development. TRLs are a systematic measurement system that supports assessments of the maturity of a particular technology and the consistent comparison of maturity between different types of technology. The framework has been successfully used by the NASA for space technology planning as well as the U.S. Department of Defense for technology development and weapons system acquisitions. The primary advantage of TRLs was as an efficient means to track development timelines (and deadlines), as well as associated risks and costs. Whereas TRLs are typically used to track development milestones (e.g., prototyping, testing, etc.), we intend our scale to be more general and academic. The MAP scale consists of six levels, with the goal to span “discovery” to real-world “development,” as depicted in Figure 2. It is the intent that research progression can be associated with MAP benchmarks. By design, the scale is balanced between more fundamental aspects (levels 1, 2, and 3) and more applied, functional aims (levels 4, 5, and 6), reflected in the colors of the associated icons. There is a clear differentiation between the progression of a newly observed scientific phenomena, principle, or material system to focused technology development addressing specific functions for one or more potential identified applications. Of note, the MAP system does not represent or presume any difficulty or challenges for any stage of material development, or a comparison of importance or “ranking” between technologies. Manuscripts both assigned a 2 need not be the same level of “impact,” nor does a 3 imply less of an achievement than a 2, etc. The assigned MAP is merely an assessment of current state, with primary significance intrinsic to the specific project scope. With more details, the levels are as described below. MAP 1 represents a breakthrough in materials science, encompassing previously unknown material responses or phenomenon (without prior demonstration or current/complete understanding), or undocumented materials (e.g., new structure or composition without precedent) or new characterization approaches implementing novel physics and/or exploiting novel material phenomena. If the material existed previously, or the behavior was observed previously in a similar material system, or the experimental technique is derivative, the manuscript cannot be classified as MAP 1. MAP 2 studies involve primary description of novel materials systems and associated properties, structure, synthesis, or characterization approach(es). The underlying principles and material properties/structures quantified, observed, or explained for the first time a particular system, but the system itself is known. Example MAP 2 studies include first-reported measurements of conductivity, strength, or hydrophobicity of a new material (or composite system); first-reported structural configuration of known material; first composite assembly of two-well known components, etc. Novel characterization techniques can be included if they are exploiting new phenomena or response or are innovatively applied. MAP 2 studies may include fundamental computational predictions of proposed theoretical systems. If it cannot be physically synthesized, it cannot be classified as MAP 1. Incremental improvements of measurements and/or dependency studies of behaviors/values cannot be classified as MAP 2 if prior published properties can be referenced as benchmark values. MAP 3 encompasses expanded understanding of a material system, properties, or behavior due to variation in conditions (effects of temperature, pressure, light, material combinations, etc.), combinations, scale, etc., including theoretical descriptions of previously discovered or observed behaviors and/or phenomena. Dependency studies should still be novel. Example MAP 3 include basic principles and material properties and structures and behaviors quantified and extended from prior reports and/or knowledge. Such works may delineate limits of performance and/or property range. MAP 3 studies need not include or imply intended application. MAP 4 shifts from fundamental understanding to functionality, with the development or design of proof-of-concept system for a specific application. At this level, basic principles and material properties/structures are quantified and extended with designed intent. These papers can involve bringing together the necessary “pieces” for proof of function or application via analysis or experiment, such as composite materials or material performance measured with functionality in mind. Examples could include measuring hydrogen storage of an MOF system for energy storage or demonstrating membrane diffusivity for ion removal, etc. The system need not be application-ready—simply proof of desired response/behavior. MAP 5 moves beyond concept and simple behavior to an enhanced materials system response, including characterization, optimization, and/or development of existing system with improved functionality for application. Material combination, system architecture, exploited response, use-case, etc. must be novel; however, the underlying physics, chemistry, and/or material components have been investigated prior and understood in principle. System need not be application ready, but physical operational conditions are considered. MAP 6 involves the response of material system exploited, characterized, demonstrated for specific application, with final system-ready material use and/or implementation. MAP 6 studies can focus on the improvement/development of technology already in use. As each level is achieved and surpassed, the technology advances, and a new set of challenges is usually discovered. Only when all of these challenges are overcome can a material's use become widespread. How will the MAP scale be applied in practice? Upon submission of a manuscript, the author may propose a MAP value (with a short justification). Prior key works leading up to the submission can also be listed (not necessarily the authors’ previous publications), providing a nominal research path leading to the current findings. In this way, the scope of the current study, as well as how it relates to a larger project or goal, can be explicitly defined. Note that a specific assessment is somewhat dependent on the material system and thus can be subject to a researcher’s interpretation. Assessment will be justified by consensus via peer review. In the event of a dispute or impasse, the editor retains the right to arbitrate. Also, many papers or studies may breach more than one MAP category (e.g., a new material system developed, combined with a novel proof-of-concept application). In such cases, two principles will be applied for assessment: (1) the lowest applicable MAP scale value will be prioritized, and (2) the primary focus of the manuscript will be prioritized; i.e., is the material development more of a conceptual advance or the novel application? Ultimately, the MAP assessment evaluates the maturity of a material's development. Due to the nature of academic research (with a focus on fundamental science), Matter anticipates submissions favoring lower MAP numbers (very rarely a 5 or 6). However, this should not discourage the submission of more developed systems, wherein system development plays a key role in ultimate application. Depending on the system, there are still academic studies, particularly regarding optimization and risk assessment, novel environmental applications of existing technologies (e.g., new use), and/or integration of new technologies or materials with established technologies. The only requirement is conceptual novelty, which can occur at any MAP value. There are clear benefits to adoption of the MAP scale. For one, research progress is explicitly defined for a publication, thereby producing a record of project development or progress (benchmarks) that can be used to justify funding and/or grants to relevant agents both in the past and future. One could imagine initial findings on a project being at a MAP 2 or MAP 3 level, while late-stage works progress to MAP 5 or MAP 6 levels. Thus, the project explicitly shows the development of a particular system rather than stagnate at a conceptual or technological roadblock. Authors can no longer overestimate or hype the impact of their findings by putting the cart before the horse, so-to-speak, increasing the integrity of the work. Moreover, the work can be easily categorized by interested parties. For example, the public and industry personnel, researchers, and/or scientists may have more interest in studies with high MAP values and thus closer to commercialization. With more widespread adoption of the scale, status and current state of emerging technologies are easily understood. By tracking a (growing) dataset of materials science publications with MAP scale assignments, material platforms can be tracked from discovery to application and from basic principles to consumer products. This is very much in the spirit of the Materials Genome Initiative—a multi-agency initiative designed to create a new era of policy, resources, and infrastructure in an effort to discover, manufacture, and deploy advanced materials twice as fast at a fraction of the cost. The Editorial team of Matter is looking forward to the adoption of the MAP scale system to its manuscripts and hope the concept will proliferate across the field of materials science. Such assessment has likely been considered across multiple material system—we have just quantified it and labeled it explicitly. We hope the MAP classification will help make materials matter.