Common Set Of Metrics Research Articles

Towards a Common Metrics and Evaluation Framework for Assessment of Older Adults and Caregivers Interacting with Artificial Intelligence

Artificial intelligence (AI) has applications in assisting older adults to age in place and provide support to them and their caregivers as their cognition declines with age. However, effective assessment methods of this technology are needed in order to benchmark their performance and a common set of metrics and evaluation methods would enable such assessments to be compared to one another. To this end, we propose a common framework for human-AI interaction involving care recipients and their care networks. From the results of a literature review exercise, a framework with sample metrics, related measures, qualified evaluation tools, and contextual factors that impact assessment are reviewed. This paper provides a sample of common metrics in one of the framework’s measurement spaces (human-AI interaction) and discusses some of the impacts of contextual factors and how use of the common metrics and evaluation framework can be used for meta-analysis and to guide future research. Additional future articles are planned to cover the other measurement spaces in the framework (system performance, task performance, and well-being), including their particular common metrics and evaluation methods. This effort aims to provide guidance for researchers in this domain as well as highlight measurement gaps that can be filled by future research.

The computational and energy cost of simulation and storage for climate science: lessons from CMIP6

Abstract. The Coupled Model Intercomparison Project (CMIP) is one of the biggest international efforts aimed at better understanding the past, present, and future of climate changes in a multi-model context. A total of 21 model intercomparison projects (MIPs) were endorsed in its sixth phase (CMIP6), which included 190 different experiments that were used to simulate 40 000 years and produced around 40 PB of data in total. This paper presents the main findings obtained from the CPMIP (the Computational Performance Model Intercomparison Project), a collection of a common set of metrics, specifically designed for assessing climate model performance. These metrics were exclusively collected from the production runs of experiments used in CMIP6 and primarily from institutions within the IS-ENES3 consortium. The document presents the full set of CPMIP metrics per institution and experiment, including a detailed analysis and discussion of each of the measurements. During the analysis, we found a positive correlation between the core hours needed, the complexity of the models, and the resolution used. Likewise, we show that between 5 %–15 % of the execution cost is spent in the coupling between independent components, and it only gets worse by increasing the number of resources. From the data, it is clear that queue times have a great impact on the actual speed achieved and have a huge variability across different institutions, ranging from none to up to 78 % execution overhead. Furthermore, our evaluation shows that the estimated carbon footprint of running such big simulations within the IS-ENES3 consortium is 1692 t of CO2 equivalent. As a result of the collection, we contribute to the creation of a comprehensive database for future community reference, establishing a benchmark for evaluation and facilitating the multi-model, multi-platform comparisons crucial for understanding climate modelling performance. Given the diverse range of applications, configurations, and hardware utilised, further work is required for the standardisation and formulation of general rules. The paper concludes with recommendations for future exercises aimed at addressing the encountered challenges which will facilitate more collections of a similar nature.

Research Agenda for Implementation of Principles of Care for Children and Youth With Special Health Care Needs.

Children and youth with special health care needs (CYSHCN) have a range of medical, educational, and support service needs to achieve optimal health and wellness. Principles of care for CYSHCN have been well described, but the literature is lacking particularly on implementation and integration of care across different settings and systems. The objective of this manuscript is to define a research agenda for principles of care for CYSHCN. Literature review examined principles of care for CYSHCN. Existing research gaps and priorities for principles of care were drawn from the literature review, a recently developed national research agenda for CYSHCN, and stakeholder consensus. Specific implementation areas of inquiry include family partner roles within and across systems; life course approach for CYSHCN; roles and training of interdisciplinary team members; and implementation, spread, and sustainability studies. Proposed methods include implementation science-based and comparative effectiveness research. A common set of metrics including health care utilization, clinical outcomes, and family and provider needs should be considered to evaluate implementation of principles of care. Implementation science and comparative effectiveness methods are needed to further understanding about how to adopt and spread principles of care for CYSHCN. The evolving demographics of CYSHCN add relevance and urgency for research findings.

Disturbance ecology and the problem of n = 1: A proposed framework for unifying disturbance ecology studies to address theory across multiple ecological systems

Abstract Disturbances are a key part of ecosystem dynamics at multiple scales. They can maintain ephemeral habitat, alter local and landscape biodiversity, drive carbon balance changes and trigger whole ecosystem regime shifts. Yet, there are few theories and only limited frameworks underlying disturbance ecology by which scientists and practitioners can anticipate the impacts of novel disturbances or changes in disturbance regimes. Much of the challenge in developing and testing theories lies in the diversity of ecosystems and disturbance processes and the general inability to (a) properly replicate studies and (b) compare results across disparate systems, from bacterial communities to boreal forests. General syntheses of disturbance processes have identified key aspects of disturbance impact and response—resource stock change, resource flux change, spatial dynamics and trait diversity—that should apply across systems. Yet, application and testing remain challenging due to a lack of a common set of metrics and baselines for comparisons across systems. Here, I propose a discipline‐wide effort to develop and publish standardized metrics (in addition to study‐specific data) such that the field as a whole can build and test general theory that works across ecological systems. Ten metrics spanning event type, location/design data and biotic composition comprise this suite of ‘minimum descriptors’ and are directly derived from current disturbance ecology theory. Several examples of cross‐system applicability, from bacterial studies to palaeoecological disturbances, are noted. These are example metrics, intended to open the conversation about the most appropriate and theory‐based lines of commonality across the field, and they may be further refined through cross‐disciplinary conversation. The result of standardizing metrics will be a coordinated dataset that allows for inter‐system testing of theory, meaning disturbance ecology studies can move past disparate, seemingly unique systems with little replication to integrated research. To phrase it in another way, disturbance ecology studies should be planned, in part, to contribute to future quantitative meta‐analyses. If done, we (as a community) will be much better prepared to address the challenges of emerging disturbances, novel ecosystems, and no‐analogue changes in disturbance regimes which are anticipated with climate change and increasing human pressures.

Incorporating shared measurement in social change initiatives: benefits, challenges, and lessons learned

Abstract Collective impact is increasingly being adopted as a systems-based approach for addressing complex social problems. Yet one of the critical components, shared measurement, remains challenging to implement (Hanleybrown, Kania and Kramer, 2012). Based around a common set of metrics, shared measurement is foundational to offering multi-organizational initiatives ongoing feedback that can allow them to share the lessons learned, make adjustments, and document the outcomes of their joint efforts. This paper examines the lessons practitioners have learned as they applied shared measurement strategies across diverse initiatives. We scrutinized food systems initiatives, in particular, hypothesizing that the complex, adaptive nature of food systems makes it more challenging to apply a commonly agreed upon set of measures in comparison to other sectors. Through seventeen semi-structured interviews, we found that interviewees shared similar perceptions about the multiple benefits of shared measurement whether they worked in food systems or other fields. Based on the major challenges they encountered, three primary lessons emerged about what it takes to establish a successful shared measurement system: trust across collaborating organizations, a streamlined process, and sufficient funding.

Plant Pest Impact Metric System (PPIMS): Framework and guidelines for a common set of metrics to classify and prioritise plant pests

Agricultural stakeholders need a common set of metrics to evaluate plant pest impacts to facilitate transparency and harmonisation of pest management and prioritisation across spatial scales and jurisdictions. We propose a classification system that articulates, defines and classifies the magnitude of impacts (historical, current or potential) of pest species (alien and native) in plant production systems. Metrics were identified and criteria defined through consideration of economic parameters, risk assessment standards and guidance tools, discussions with pest risk assessment practitioners and recent advances in environmental impact classification schemes. Twenty metrics were identified and assigned to one of four key metric types: spatiotemporal, market-driven, primary response and mid-to long-term response. Host crop value, Market access, Feasibility of management and Reversibility were identified as disruptor metrics, likely to influence overall classification by at least twice that of other metrics. Application of the system found it was able to classify well-known pests by importance, capturing changes in impact status as the management programme progressed for one pest, and how it was influenced by the geographic scale of assessment for another. Our work demonstrates the value of integrating plant protection science with invasion biology to derive a comprehensive measure of pest impact in agroecosystems that can be utilised by all plant biosecurity stakeholders.

Streamlined eco‐engineering approach helps define environmental flows for tropical Andean headwaters

Abstract Applying the environmental flows concept to human‐altered lotic ecosystems continues to face many practical challenges and barriers. Here, we modify a previously proposed framework, the Eco‐Engineering Decision Scaling, for application to part of the water supply system of Quito, Ecuador. Specifically, we used feedback from engineers and water managers to develop a common set of metrics for defining flow‐ecology relationships and assess managed‐flow impacts on stream ecology. At 12 sites over 3 years, we collected flow and benthic invertebrate data (taxonomic richness, taxa important for fish, functional feeding groups, and water quality‐sensitive taxa) during wet and dry seasons. We then used these data to identify flow thresholds (relative to unmanaged flows) where flow withdrawal caused visible ecological impacts. For this system, reduction of flow to 20% of the annual median was detrimental to benthic communities, while reductions to 40% of the annual median flow caused a variety of responses in the system. A trade‐off analysis of weighted metrics showed that a 50% benthic fauna richness could be sustained if dry season flows were maintained between 28% and 40% of the unmanaged median annual flow. This study provides a roadmap for bridging between eco‐engineering theoretical frameworks and the adoption of the environmental flows concept as actionable management thresholds.

Making the Most of the Chicago Benchmarking Collaborative

In this case, lessons from the Chicago Benchmarking Collaborative illustrate key principles of collaborative action and the importance of using data to achieve SMART goals. In 2015, the Chicago Benchmarking Collaborative (CBC) was a network of seven agencies in Chicago, Illinois, serving 12,000 low-income residents. Each of the agencies had early childhood, school-age children, and adult education programs. At the prompting of the Chicago Community Trust, they came together to (1) benchmark their education programs outputs and outcomes; (2) learn and share best practices through developing a common set of metrics and measurements and implementing these measurements into a case management software system; and (3) share the costs of the case management software system to be used for program evaluation and continuous quality improvement. Three aspects of CBC are particularly noteworthy. First, there are no joint program activities or clients among these agencies. Their exchange is limited to sharing data and other information. This makes CBC distinct from collaborations formed to begin a program or to advocate for a policy. Second, the group requires each agency to enter data on a timely basis and to set SMART goals based on the data reports. The agencies are held mutually accountable for their work to achieve their own SMART goals during the year and report on progress. Third, CBC used monetary incentives to ensure that data entry and SMART goal action remained a priority for each agency.

Platform-Independent Benchmarks for Task and Motion Planning

We present the first platform-independent evaluation method for task and motion planning (TAMP). Previously point, various problems have been used to test individual planners for specific aspects of TAMP. However, no common set of metrics, formats, and problems have been accepted by the community. We propose a set of benchmark problems covering the challenging aspects of TAMP and a planner-independent specification format for these problems. Our objective is to better evaluate and compare TAMP planners, foster communication, and progress within the field, and lay a foundation to better understand this class of planning problems.

Building the Evidence Base for Tele-Emergency Care: Efforts to Identify a Standardized Set of Outcome Measures.

To enhance the quality of emergency department (ED) care, some rural hospitals have adopted the use of telemedicine (tele-ED). Without a common set of metrics, it is difficult to quantify the impact of this technology. To address this limitation, the Health Resources and Services Administration funded the identification and testing of a core set of measures that could be used to build a business case for the value of tele-ED care. A comprehensive environmental scan was conducted to identify existing measures relevant to assessing ED care and the use of telemedicine. Identified measures were assessed against a set of criteria and pilot tested in rural hospitals. The environmental scan identified numerous ED-specific measures and a limited set of telehealth-specific measures, but no clearly defined measures specific to tele-ED. Applying evaluation criteria to the measures revealed that few have a well-established evidence base, and fewer have undergone the rigorous testing needed to establish statistical reliability and validity. Nevertheless, a parsimonious set of measures was identified that met many of the evaluation criteria. Pilot testing indicated that collecting data using these measures was feasible. For tele-ED benefits to be widely acknowledged, more research is required to demonstrate that care delivered using tele-ED care is as high quality, if not more so, than in-person care. This requires researchers to consistently use a set of clearly defined measures. The use of clearly defined and standardized measures will aid interpretation and permit replication in multiple studies, furthering acceptance of study findings.

Scene background initialization: A taxonomy

The availability of an initial background model that describes the scene without foreground objects is the prerequisite for a wide range of applications, ranging from video surveillance to computational photography. Limited attention to the problem is given in the literature on background modeling, that mainly regards model representation and updating. Therefore, we propose a taxonomy study for background initialization, providing the basis for a fair and easy comparison of existing and future methods, on a common dataset of groundtruthed sequences, with a common set of metrics, and based on reproducible results. Experimental results highlight the most promising approaches as well as main open issues for background initialization.

Is specialization an evolutionary dead end? Testing for differences in speciation, extinction and trait transition rates across diverse phylogenies of specialists and generalists.

Specialization has often been claimed to be an evolutionary dead end, with specialist lineages having a reduced capacity to persist or diversify. In a phylogenetic comparative framework, an evolutionary dead end may be detectable from the phylogenetic distribution of specialists, if specialists rarely give rise to large, diverse clades. Previous phylogenetic studies of the influence of specialization on macroevolutionary processes have demonstrated a range of patterns, including examples where specialists have both higher and lower diversification rates than generalists, as well as examples where the rates of evolutionary transitions from generalists to specialists are higher, lower or equal to transitions from specialists to generalists. Here, we wish to ask whether these varied answers are due to the differences in macroevolutionary processes in different clades, or partly due to differences in methodology. We analysed ten phylogenies containing multiple independent origins of specialization and quantified the phylogenetic distribution of specialists by applying a common set of metrics to all datasets. We compared the tip branch lengths of specialists to generalists, the size of specialist clades arising from each evolutionary origin of a specialized trait and whether specialists tend to be clustered or scattered on phylogenies. For each of these measures, we compared the observed values to expectations under null models of trait evolution and expected outcomes under alternative macroevolutionary scenarios. We found that specialization is sometimes an evolutionary dead end: in two of the ten case studies (pollinator-specific plants and host-specific flies), specialization is associated with a reduced rate of diversification or trait persistence. However, in the majority of studies, we could not distinguish the observed phylogenetic distribution of specialists from null models in which specialization has no effect on diversification or trait persistence.

Guidelines for evaluating performance of oyster habitat restoration

Restoration of degraded ecosystems is an important societal goal, yet inadequate monitoring and the absence of clear performance metrics are common criticisms of many habitat restoration projects. Funding limitations can prevent adequate monitoring, but we suggest that the lack of accepted metrics to address the diversity of restoration objectives also presents a serious challenge to the monitoring of restoration projects. A working group with experience in designing and monitoring oyster reef projects was used to develop standardized monitoring metrics, units, and performance criteria that would allow for comparison among restoration sites and projects of various construction types. A set of four universal metrics (reef areal dimensions, reef height, oyster density, and oyster size–frequency distribution) and a set of three universal environmental variables (water temperature, salinity, and dissolved oxygen) are recommended to be monitored for all oyster habitat restoration projects regardless of their goal(s). In addition, restoration goal‐based metrics specific to four commonly cited ecosystem service‐based restoration goals are recommended, along with an optional set of seven supplemental ancillary metrics that could provide information useful to the interpretation of prerestoration and postrestoration monitoring data. Widespread adoption of a common set of metrics with standardized techniques and units to assess well‐defined goals not only allows practitioners to gauge the performance of their own projects but also allows for comparison among projects, which is both essential to the advancement of the field of oyster restoration and can provide new knowledge about the structure and ecological function of oyster reef ecosystems.

Conceptualising a New Knowledge Management Logic Model for Global Health: A Case-Study Approach

Over the past few decades, knowledge management (KM) has become well-established in many fields, particularly in business. Several KM models have been at the forefront of promoting KM in businesses and organisations. However, the applicability of these traditional KM models to the global health field is limited by their focus on KM processes and activities with few linkages to intended outcomes. This paper presents the new Knowledge Management for Global Health (KM4GH) Logic Model, a practical tool that helps global health professionals plan ways in which resources and specific KM activities can work together to achieve desired health program outcomes. We test the validity of this model through three case studies of global and field-level health initiatives: an SMS-based mobile phone network among community health workers (CHWs) and their supervisors in Malawi, a global electronic Toolkits platform that provides health professionals access to health information resources, and a netbook-based eHealth pilot among CHWs and their clients in Bangladesh. The case studies demonstrate the flexibility of the KM4GH Logic Model in designing various KM activities while defining a common set of metrics to measure their outcomes, providing global health organisations with a tool to select the most appropriate KM activities to meet specific knowledge needs of an audience. The three levels of outcomes depicted in the model, which are grounded in behavioural theory, show the progression in the behaviour change process, or in this case, the knowledge use process, from raising awareness of and using the new knowledge to contributing to better health systems and behaviours of the public, and ultimately to improving the health status of communities and individuals. The KM4GH Logic Model makes a unique contribution to the global health field by helping health professionals plan KM activities with the end goal in mind.

A situation that we had never imagined

There is no consensus regarding a common set of metrics for robot task complexity in associated human-robot interactions. This paper is an attempt to address this issue by proposing a new metric so...

A situation that we had never imagined: post-Fukushima virtual collaborations for determining robot task metrics

There is no consensus regarding a common set of metrics for robot task complexity in associated human-robot interactions. This paper is an attempt to address this issue by proposing a new metric so that the educational potential when using robots can be further developed. Tasks in which students in Japan and UK interact in a 3D virtual space to collaboratively program robots to navigate mazes have resulted in quantitative data of immersion, circuit task complexity and robot task complexity. The data has subsequently been collated to create a proposed new metric for tasks involving robots, which we have termed task fidelity. The paper proposes that task fidelity is a quantitative measure of a set robot task in relation to a learner's solution. By quantifying task fidelity educators utilising robots in schools and in higher education will be able to provide tasks commensurate with the expected successful outcomes achieved by the learners.

Outcome measures for multiple sclerosis

Background: This review determined the most commonly used, and reliable, measures for assessing clinical outcomes for multiple sclerosis (MS).Objectives: It was anticipated that this would facilitate the development of a common set of metrics, and aid reaching a consensus regarding the outcome measures that are typically used in the field of MS clinical research.Major findings: A thorough literature review of clinical outcome measures for MS produced 166 measures that have been used in this context. This list was then refined by discussion with a panel of consultant neurologists, which reduced the list to 23 commonly employed tools. This shortlist was then further refined through surveying 41 centres for MS treatment, which reduced the shortlist to 16 measures. The properties of these scales, in terms of their symptom/function domains, their specificity for MS, their administration characteristics, and their reliability and validity for MS, are all discussed.Conclusions: Conclusions regarding the development of potential sets of assessment measures for MS, which encompass broad symptom/function domains, and which are sensitive to the practical requirements of administration within clinical contexts, are explored.

Comparative evaluation of existing and new methods for correcting ocular artifacts in electroencephalographic recordings

EEG signals are often contaminated by ocular artifacts (OAs), in particular when they are recorded for a subject that is, in principle, awake, such as in a study of drowsiness. It is generally desirable to detect and/or correct these OAs before interpreting the EEG signals. We have identified 11 existing methods for dealing with OAs. Their study allowed us to create 16 new methods. We performed a comparative performance evaluation of the resulting 27 distinct methods using a common set of data and a common set of metrics. The data was recorded during a driving task of about two hours in a driving simulator. This led to a ranking of all methods, with five emerging clear winners, comprising two existing methods and three new ones.

Lost in Translation: Degree Definition and Quality in a Globalized World

Within the United States, defining the meaning of a degree and comparing the quality of institutions on a common set of metrics is no simple matter. And indeed, we resist such comparisons. We are p...

A Roundtable Discussion Alarm Safety: A Collaborative Effort

<i>A Roundtable Discussion</i> Alarm Safety: A Collaborative Effort