Evaluation Methodology Research Articles

DMPC-based control solution for mobile robots platoon based on ZigBee communication

The integration of mobile robots into various applications, operating collaboratively in groups known as platoons, presents challenges in communication, control, and synchronization. This paper introduces a control solution for mobile robot platoons based on a DMPC algorithm and the ZigBee protocol for V2V communication. This communication standard was selected for its low power consumption, cost-effectiveness, and reliable data transmission over short distances. Furthermore, a testbed and methodology for the experimental evaluation of the proposed control solution were developed. The results demonstrate the efficacy and feasibility of the proposed control and communication solutions.

Improvement of the system for creating intellectual property objects

The aim of this article is to identify the institutional and economic-psychological factors that affect the effectiveness of protecting intellectual property rights, with the goal of improving the efficiency of innovation in priority areas of scientific and technological development. In this study, we used methods of analysis, systematization, and classification to extract relevant information related to the efficiency of innovative activity from regulatory and academic sources. We also analyzed patent data for five years to track the dynamics of application filings and patent reviews. Additionally, we applied methods of functional analysis to labor functions in training future patent and economic engineers. Finally, we employed the methodology of expert evaluation to assess the rating of indicators for innovative performance. Based on a comprehensive analysis of normative and scientific approaches to evaluating the effectiveness of innovative activities, the main directions and ways to improve it have been identified, and prospects for future research have been outlined. The paper also discusses the activities of faculty and researchers, as well as the potential for further research in this area

A methodology for preliminary benefit evaluation of Distributed Generation to drive private investments

Private investments in distributed generation (DG) are typically driven by the interests of private investor, often overlooking possible benefits for the distribution operator. This work presents a methodology that enables power distribution utilities to assess the potential for attracting investments in their networks, considering a regulatory framework where utilities could actively encourage private DG investors. The methodology involves calculating and integrating the benefits associated with investment deferral, losses reduction and reliability improvement into the energy price negotiated between the utility and the DG owner. The proposed approach was applied to an adapted test grid, considering real regulatory aspects related to technical losses and reliability. The results provide support for utilities to perform preliminary financial analyses to determine the appropriate incentives for private DG investors, identifying cases with competitive energy prices for generators. Key contributions of this methodology include a clear and reasonable metric for quantifying the capacity to attract investments based on standard utility data and the importance of incorporating regulatory aspects related to technical losses and reliability, that are often neglected in similar studies. Furthermore, the methodology not only promotes new opportunities for DG investors but also addresses challenges associated with network congestion.

Sustainable practices in construction: Exposing the Potential of Waste as a Resource.

Abstract This paper examines the critical issues facing the construction and demolition industry in the countries of the European Union, with a particular focus on Germany. It presents strategies for waste management and sustainable development goals in construction sector. The construction industry is responsible for a significant proportion of waste production, which underscores the urgency of addressing the impending waste management crisis. The paper highlights the necessity for the implementation of circular-oriented construction practices that prioritize recycling and the reduction of waste, aiming to improve the efficient use of resources in building design and construction. Using a real non-residential building demolition as a case study, the paper outlines specific steps to determine building materials and document demolition procedures. This study presents a variety of selective demolition strategies, with consideration of end-of-life scenarios for materials, including reuse, recycling, and downcycling. The analysis highlights the complexity of the demolition process, emphasizing the necessity for comprehensive evaluation methodologies. By closing the material cycle and prioritizing reuse and recycling, it is possible to achieve sustainable and long-term value retention of building materials. Additionally, methods such as life cycle assessment and material flow analysis for waste streams are introduced. The environmental impact of the demolition of the “Elementum” office building in Munich was analyzed and the CO2 emissions calculated, in relation to transport distances. Results were presented on the reduction in CO2 emissions due to the elimination of transport to landfill and the associated impact on the Life Cycle Assessment. By considering factors such as resource conservation, energy consumption, greenhouse gas emissions, waste generation and pollution, stakeholders can make informed decisions to mitigate negative environmental impacts.

Challenges and Alternatives to Evaluation Methods and Regulation Approaches for Medical Apps as Mobile Medical Devices: International and Multidisciplinary Focus Group Discussion.

The rapid proliferation of medical apps has transformed the health care landscape by giving patients and health care providers unprecedented access to personalized health information and services. However, concerns regarding the effectiveness and safety of medical apps have raised questions regarding the efficacy of randomized controlled trials (RCTs) in the evaluation of such apps and as a requirement for their regulation as mobile medical devices. This study aims to address this issue by investigating alternative methods, apart from RCTs, for evaluating and regulating medical apps. Using a qualitative approach, a focus group study with 46 international and multidisciplinary public health experts was conducted at the 17th World Congress on Public Health in May 2023 in Rome, Italy. The group was split into 3 subgroups to gather in-depth insights into alternative approaches for evaluating and regulating medical apps. We conducted a policy analysis on the current regulation of medical apps as mobile medical devices for the 4 most represented countries in the workshop: Italy, Germany, Canada, and Australia. We developed a logic model that combines the evaluation and regulation domains on the basis of these findings. The focus group discussions explored the strengths and limitations of the current evaluation and regulation methods and identified potential alternatives that could enhance the quality and safety of medical apps. Although RCTs were only explicitly mentioned in the German regulatory system as one of many options, an analysis of chosen evaluation methods for German apps on prescription pointed toward a "scientific reflex" where RCTs are always the chosen evaluation method. However, this method has substantial limitations when used to evaluate digital interventions such as medical apps. Comparable results were observed during the focus group discussions, where participants expressed similar experiences with their own evaluation approaches. In addition, the participants highlighted numerous alternatives to RCTs. These alternatives can be used at different points during the life cycle of a digital intervention to assess its efficacy and potential harm to users. It is crucial to recognize that unlike analog tools, digital interventions constantly evolve, posing challenges to inflexible evaluation methods such as RCTs. Potential risks include high dropout rates, decreased adherence, and nonsignificant results. However, existing regulations do not explicitly advocate for other evaluation methodologies. Our research highlighted the necessity of overcoming the gap between regulatory demands to demonstrate safety and efficacy of medical apps and evolving scientific practices, ensuring that digital health innovation is evaluated and regulated in a way that considers the unique characteristics of mobile medical devices.

Application of Compositional Nutrient Diagnosis to Maize Cultivated in Tropical Soils in Brazilian Semi-Arid

ABSTRACT Nutritional diagnosis, a technique used to maximize the economic yield of agricultural holdings and to protect the environment, is the focus of this study. Nutrient availability is one of the critical limiting factors in maize grain yield. The study, which aimed to establish and compare the nutritional standards obtained by the Compositional Nutrient Diagnosis (CND) method in maize grown in tropical soils in a semi-arid region, is based on a thorough examination of a database containing information from 198 experimental plots collected between 2011 and 2019 in the semi-arid region of Ceará, Brazil. The evaluation methodology was the multivariate CND calculation method. The mathematical models used to analyze the relationship between the nutrient’s CND-r2 index and leaf concentrations revealed coefficients of determination (R2) for nitrogen (N) and magnesium (Mg) of 0.36 and 0.66, respectively. The other values exceeded 70% (R2 ≥ 0.70), while phosphurus (P), calcium (Ca), cupper (Cu), iron (Fe), and manganese (Mn) showed values above 90% (R2 ≥ 0.90). These results not only highlight the high accuracy of these models in studying the interactions between nutrient coefficients and their leaf concentrations but also reassure the reader about the reliability of the research. The study’s thoroughness instills confidence in the research’s validity.

Quantitative assessment of concrete microstructure in the diagnosis of surface damage to industrial floor

Diagnostic assessment of industrial floor using the methodology of quantitative evaluation of concrete microstructure is presented. Symptoms of damage included surface dusting and delamination of the top layer. A petrographic analysis of the concrete composition in core specimens was carried out. Excessive air content, accumulation of air voids and the occurrence of oriented cracks, local variability of the phase composition of cement hydration products, including the occurrence of carbonated areas were observed and associated with the premature surface damage

Research on Packaging Design Elements for Expanding Market Share of Fruit Juice Beverages in China

Purpose: By examining Chinese consumers’ preferences and needs in fruit juice beverage packaging design, this study provides scientifically grounded strategies and solutions for researching and developing packaging designs in the juice beverage industry. Research design, data, and methodology: This study systematically analyzes the packaging design of fruit juice beverages and introduces a demand evaluation methodology for fruit juice beverage packaging by integrating the Kano model and the Analytic Hierarchy Process (AHP). Results: Based on the calculation results, ‘Nontoxic’ and ‘Clear ingredient information’ were important in Must-Be Quality Element. ‘Portable’ and ‘Lightweight performance’ were the top 2 in One-Dimensional Quality Element. In the Attractive Quality Element, ‘Unique pattern’ was identified as the most important factor. Conclusions: The analysis results of this study suggest the importance of fruit juice beverage packaging design elements preferred by Chinese consumers and are expected to be used as reference material by Korean companies in the future when preparing plans to enter and expand into the Chinese beverage market.

Nursing effectiveness reconsidered: Some fundamental reflections on the nature of nursing.

Despite being considered the proverbial backbone of our healthcare systems, nursing still seems to struggle to scientifically demonstrate its contribution to care experiences and patient outcomes. This leads to erosive tendencies that threaten the development of the profession and its progress as an academic discipline. With this paper, we want to contribute to the theoretical discourse concerning the nature of nursing and the research into its effectiveness. We begin by outlining a set of prevailing paradoxes and their consequences relating to nursing and nursing research: the issue of demonstrating its unique contribution despite a clear societal mandate; a discrepancy between subjectively experienced effectiveness and objectively ascertainable effectiveness; and a mismatch between theoretical premises of nursing and task-oriented cultures in practice environments. Using an example of a seemingly simple nursing intervention, we intend to demonstrate the qualities and complexities of nursing. We further illustrate this by drawing on several of our research projects using theory-based evaluation methodologies. From these illustrative examples, we distil two insights relating to nursing interventions that we consider fundamental: the nurse, as a person, is central to its unique effectiveness; and there is always an interplay between context, intervention and its intended effect. We summarise our considerations and argue the case for conceiving research designs in alignment with theoretical premises of nursing.

A framework for human evaluation of large language models in healthcare derived from literature review

With generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices. To overcome such significant obstacles to healthcare LLM developments and deployments, we propose QUEST, a comprehensive and practical framework for human evaluation of LLMs covering three phases of workflow: Planning, Implementation and Adjudication, and Scoring and Review. QUEST is designed with five proposed evaluation principles: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.

Causal message-passing for experiments with unknown and general network interference

Randomized experiments are a powerful methodology for data-driven evaluation of decisions or interventions. Yet, their validity may be undermined by network interference. This occurs when the treatment of one unit impacts not only its outcome but also that of connected units, biasing traditional treatment effect estimations. Our study introduces a framework to accommodate complex and unknown network interference, moving beyond specialized models in the existing literature. Our framework, termed causal message-passing, is grounded in high-dimensional approximate message-passing methodology. It is tailored for multiperiod experiments and is particularly effective in settings with many units and prevalent network interference. The framework models causal effects as a dynamic process where a treated unit's impact propagates through the network via neighboring units until equilibrium is reached. This approach allows us to approximate the dynamics of potential outcomes over time, enabling the extraction of valuable information before treatment effects reach equilibrium. Utilizing causal message-passing, we introduce a practical algorithm to estimate the total treatment effect, defined as the impact observed when all units are treated compared to the scenario where no unit receives treatment. We demonstrate the effectiveness of this approach across five numerical scenarios, each characterized by a distinct interference structure.

Using RS and GIS for risk management of natural disasters consequences: The case of cultural heritage in Jinan city, China

The preservation of cultural heritage is confronted with significant challenges due to its extensive history and the increasing impact of climate change, particularly natural disasters. Instead of solely investing resources in post-disaster restoration efforts, implementing a proactive risk-management strategy for natural disasters is a more effective approach. This study introduces an analytical and evaluative methodology grounded in remote sensing (RS) and geographic information systems (GIS) to bridge the existing gap in understanding natural disaster risks to cultural heritage sites in Jinan, China. By leveraging a combination of RS data and established methodologies such as the Soil Conservation Service Curve Number, Maximum entropy, and the Revised Universal Soil Loss Equation, we conducted an in-depth analysis of the risks posed by various disasters including floods, landslides, earthquakes, and erosion. Furthermore, GIS and the Analytic Hierarchy Process were utilized to facilitate the risk assessment. Detailed disaster risk maps based on these assessments were produced. Our findings revealed that approximately 28.95 % of Jinan's cultural heritage sites face moderate to severe risks from natural disasters. Cultural heritage sites in Changqing District, Gangcheng District, and Laiwu District are particularly vulnerable to such calamities. These outcomes serve as crucial references for enhancing the safeguarding and management of cultural heritage, while informing disaster prevention and mitigation strategies in Jinan.

Treatment of restless legs syndrome and periodic limb movement disorder: an American Academy of Sleep Medicine clinical practice guideline.

This guideline establishes clinical practice recommendations for Treatment of Restless Legs Syndrome (RLS) and Periodic Limb Movement Disorder (PLMD) in adults and pediatric patients. The American Academy of Sleep Medicine (AASM) commissioned a task force of experts in sleep medicine to develop recommendations and assign strengths based on a systematic review of the literature and an assessment of the evidence using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology. The task force provided a summary of the relevant literature and the certainty of evidence, the balance of benefits and harms, patient values and preferences, and resource use considerations that support the recommendations. The AASM Board of Directors approved the final recommendations. The following good practice statement is based on expert consensus, and its implementation is necessary for the appropriate and effective management of patients with RLS. The following recommendations are intended as a guide for clinicians in choosing a specific treatment for RLS and PLMD in adults and children. Each recommendation statement is assigned a strength ("Strong" or "Conditional"). A "Strong" recommendation (i.e., "We recommend…") is one that clinicians should follow under most circumstances. The recommendations listed below are ranked in the order of strength of recommendations and grouped by class of treatments within each PICO question. Some recommendations include remarks that provide additional context to guide clinicians with implementation of this recommendation.

A Survey on the Design of Virtual Reality Interaction Interfaces

Virtual reality (VR) technology has made remarkable progress in recent years and will be widely used in the future. As a bridge for information exchanges between users and VR systems, the interaction interface is pivotal for providing users with a good experience and has emerged as a key research focus. In this review, we conducted a comprehensive search of the Web of Science and CNKI databases from 2011 to 2023 to identify articles dedicated to VR interaction interface design. Through a meticulous analysis of 438 articles, this paper offers a substantial contribution to the emerging field of VR interactive interface research, providing an in-depth review of the principal research advancements. This review revealed that the majority of studies are centered on practical case analyses within specific application scenarios, employing empirical evaluation methods to assess objective or subjective metrics. We then concentrated on elucidating the foundational principles of interface design and their evaluation methodologies, providing a reference for future research endeavors. Additionally, the limitations, challenges, and future directions in VR interaction interface design research were discussed, highlighting the need for further research in design evaluation to continuously refine the development of standards and guidelines for VR interactive interface design. According to the findings of this review, there is a necessity to enhance research on information design for multi-channel interactive interfaces. Furthermore, it is essential to focus on the diverse characteristics of users to propose more inclusive design solutions. Adopting interdisciplinary approaches could lead to breakthroughs in the creation of personalized and adaptive VR interaction interfaces.

Оценка готовности российских регионов к использованию облачных услуг

The objective of the study is to develop methodological approaches for the comparative assessment of Rus-sian regions’ readiness to adopt cloud services and the intensity of their usage, in order to substantiate strategic directions for the development of the infocommunication sector at the regional level. The approach is based on the construction of integrated indices of economic and infrastructural conditions favorable to cloud service adoption, as well as an index of the intensity of their usage. The selection of statistical indicators included in these indices is justified. A methodology for the comparative evaluation of Russian regions’ readiness to utilize cloud services is proposed. Readiness assessments of Russian regions for the development of cloud services were constructed based on current data from Rosstat for 2022 and 2023. The regions were grouped analo-gously to the BCG matrix and Gartner’s quadrant. Framework recommendations were developed, taking into account the identified specifics of regional conditions for the development of cloud service markets and the index of cloud service usage intensity. These recommendations can be used to substantiate strategic directions for the development of the infocommunication sector in Russian regions.

Systematic review of Janus kinases inhibitors for rheumatoid arthritis: methodology, reporting, and quality of evidence evaluation.

To evaluate the methodological, reporting and evidence quality of systematic reviews or meta-analyses of Janus kinases (JAK) inhibitors for the treatment of rheumatoid arthritis (RA). Our study systematically retrieved reviews from various databases, spanning from inception to June 2024. Two evaluators independently assessed the methodological, reporting, and evidence quality of each review using the AMSTAR-2 and PRIAMA2020 tools. The evidence quality was evaluated according to GRADE criteria. Six aspects were evaluated: publication year, study type, homogeneity, risk of publication bias, AMSTAR-2 methodology, and PRIAMA2020 reporting quality. Excel 2016 facilitated conversion of scores into radar plots. Following stringent selection criteria, a total of 18 relevant studies were identified. The AMSTAR-2 scores ranged from 4 to 13 points, with five studies rated as low quality and the remaining 13 as critically low quality. All studies encompassed populations, interventions, controls, and outcome measures, demonstrating commendable integrity. However, there is room for improvement in study protocol development and registration, comprehensive search strategies, inclusion and exclusion criteria, conflict of interest disclosure, and discussion of heterogeneity. PRIAMA2020 assessments ranged from 14.5 to 21 points, with two studies scoring below 15 points due to increased bias risk from data transformation and sensitivity analysis. Notably, all reviews (100%) adhered to PRIAMA2020 guidelines for certain items but none met all criteria. GRADE evaluation included 446 outcome measures, with 158 of moderate, 156 of low, and 132 of very low quality, indicating JAK inhibitors is effective in improving RA. According to radar chart, the average rank score was 13.13. One study achieved a balanced score across all dimensions, while 11 exceeded the average, five showed significant differences in PRIAMA2020 scores, and four in AMSTAR two scores. Despite summarizing the efficacy and safety of JAK inhibitors in treating RA, the included studies exhibited poor methodological and reporting quality, along with low-quality evidence overall. Therefore, caution is warranted among decision-makers regarding the use of JAK inhibitors in RA treatment. Urgent requirements include high-quality, multicenter studies investigating JAK inhibitors for RA. https://www.crd.york.ac.uk/PROSPERO, identifier 413415.

Оценка рисков реактивации разломов при бурении горизонтальной скважины по результатам одномерного геомеханического моделирования

The article presents main factors affecting the fault stability. Concepts of critically stressed faults and critical pore pressure are discussed. The methodologies for evaluation of stressed-deformed state of rocks in the vicinity of fault zones during reservoir depletion and injection, well drilling are analyzed. 1D geomechanical models of fishbone sidetracks and main borehole of the well N are constructed. The fault stability evaluation is made using Mohr – Coulomb failurecriterion from data of insitustresses and near-wellborestressescalculated from Kirsch wellbore solution. Optimal mud weight ranges are determined for safe drilling of fishbone boreholes accounting fault reactivation risks. Basic recommendations for optimal drilling through the interval of unstable shales are made.

Applying queueing theory to evaluate wait-time-savings of triage algorithms

In the past decade, artificial intelligence (AI) algorithms have made promising impacts in many areas of healthcare. One application is AI-enabled prioritization software known as computer-aided triage and notification (CADt). This type of software as a medical device is intended to prioritize reviews of radiological images with time-sensitive findings, thus shortening the waiting time for patients with these findings. While many CADt devices have been deployed into clinical workflows and have been shown to improve patient treatment and clinical outcomes, quantitative methods to evaluate the wait-time-savings from their deployment are not yet available. In this paper, we apply queueing theory methods to evaluate the wait-time-savings of a CADt by calculating the average waiting time per patient image without and with a CADt device being deployed. We study two workflow models with one or multiple radiologists (servers) for a range of AI diagnostic performances, radiologist’s reading rates, and patient image (customer) arrival rates. To evaluate the time-saving performance of a CADt, we use the difference in the mean waiting time between the diseased patient images in the with-CADt scenario and that in the without-CADt scenario as our performance metric. As part of this effort, we have developed and also share a software tool to simulate the radiology workflow around medical image interpretation, to verify theoretical results, and to provide confidence intervals for the performance metric we defined. We show quantitatively that a CADt triage device is more effective in a busy, short-staffed reading setting, which is consistent with our clinical intuition and simulation results. Although this work is motivated by the need for evaluating CADt devices, the evaluation methodology presented in this paper can be applied to assess the time-saving performance of other types of algorithms that prioritize a subset of customers based on binary outputs.

Guía de práctica clínica para el tamizaje de cáncer de mama en el Seguro Social del Perú (EsSalud)

Introduction. This article provides an overview of the clinical practice guideline (CPG) for breast cancer screening in the Social Security of Peru (EsSalud). Objective. To offer evidence-based clinical recommendations for breast cancer screening within EsSalud. Methods. The CPG for breast cancer screening was developed by establishing a guideline development group (local GDG) consisting of medical specialists and methodologists. The local GDG formulated 5 clinical questions to be answered by this CPG. Systematic searches of systematic reviews and, when it was considered pertinent, primary studies were conducted in Pubmed and Cochrane CENTRAL during 2022. The evidence to answer each of the posed clinical questions was selected. The quality of the evidence was evaluated using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) methodology. In periodic work meetings, the local GDG used the GRADE methodology to review the evidence and formulate the recommendations, points of good clinical practice, and the flowchart of screening Finally, the CPG was approved with Resolution No. 68-IETSI-ESSALUD-2022. Results. The CPG examined 5 clinical questions related to physical examination and diagnostic imaging for breast cancer screening. This led to the development of 8 recommendations (2 strong and 6 conditional), 7 best practice points, and 1 flowchart. Conclusion. This article outlines the methodology and evidence-based findings of the CPG for breast cancer screening at EsSalud.

Financial Performance Evaluation of Small Finance Banks: A CAMEL Analysis

Small finance banks (SFBs) are a special category of banks incepted by the Reserve Bank of India in 2015 to cater to the financial needs of the underserved population of the country with the aim of financial inclusion. They serve the priority sector and the unorganised sector of the country. This article analyses and evaluates the performance of these banks on the basis of various financial parameters such as capital adequacy, asset management, management quality, earning quality and liquidity derived from the CAMEL framework. Following a robust evaluation methodology, namely data envelopment analysis (DEA), a sample of all the SFBs operating in India to date is taken. As per the composite rating of CAMEL, results show that the majority of SFBs exhibit satisfactory performance. Utkarsh SFB Limited is ranked at the first position and is followed by Fincare SFB Limited on the second position. Jana SFB Limited stands at the last position amongst all the banks, preceded by ESAF SFB Limited. The results of DEA show average inefficiency among SFBs. The calculated efficiency scores of SFBs highlight that Capital SFB, Fincare SFB, Jana SFB, Shivalik SFB and Utkarsh SFB are the most efficient banks, with a technical efficiency score of 1. ESAF SFB has the lowest efficiency score. Scale inefficiency is the major reason for inefficiency. The study provides valuable insights to the authorities, managers and investors about the performance of these banks and gives them a future vision of the sustenance of these banks.