Process Evaluation Model Research Articles

Forecasting Unemployment in Russia Using Machine Learning Methods

In this paper, we forecast the dynamics of unemployment in Russia using several machine learning methods: random forest, gradient boosting, elastic net, and neural networks. The scientific contribution of this paper is threefold. First, along with feed-forward, fully connected neural networks, we use sequence-to-sequence model recurrent neural networks, which take the time-series structure of the sample dataset into account. Second, in addition to univariate long short-term memory models, we include additional macroeconomic indicators in order to estimate multivariate recurrent neural networks. Third, the model evaluation process considers revisions of statistical information in real-time datasets. In order to increase the model’s predictive performance, we use additional unstructured indicators: search queries and news indices. Relative to the structural model of unemployment dynamics, the mean absolute forecast error for one month ahead is reduced by 65%, to 0.12 percentage points of the unemployment rate in the recurrent neural networks and long short-term memory models, and by 56%, to 0.14 percentage points in the modified gradient boosting algorithms. When accounting for revisions of statistical information, further reduction of the root-mean-square error by the models proposed is revealed, which highlights the importance of accounting for possible changes in the calculation of the values of macroeconomic indicators.

A novel based-performance degradation Wiener process model for real-time reliability evaluation of lithium-ion battery

Real-time reliability evaluation of lithium-ion battery plays a vital role in guaranteeing the safety of energy storage system and its related products. However, it is difficult to predict and evaluate the remaining useful life and reliability of cell with accurate mathematical models, which is related to the complexity and variability of performance degradation during service. In this paper, a novel based-performance degradation Wiener process model is established based on battery degradation data and Bayesian updating algorithm. Firstly, three types of reliability evaluation models are constructed based on Wiener process degradation model, and the laboratory tests of 8 Lithium-ion cells (cylindrical 18650) are carried out to verify the validity and accuracy of the models, the results show that the binary random parameters evaluation model (BRPEM) is able to more accurately describe the degradation process of lithium-ion performance. Then, the Bayesian updating algorithm based on the BRPEM is employed to fuse the degradation data between 100 and 300 cycles of 5# cell in the test and real-time monitoring degradation data, the real-time reliability evaluation of the battery is realized. Finally, the results of life probability density function and reliability function show that the proposed model can achieve scientific and accurate reliability evaluation of Lithium-ion battery.

Conceptual Framework for Evaluation of Diploma Program in Automotive Technology in Vocational Colleges in Malaysia

This study was conducted to build a conceptual framework for evaluating diploma programs in automotive technology in vocational colleges in Malaysia. The researcher used a conceptual framework based on the study's theoretical framework, which focused on the context, input, process, and product (CIPP) evaluation model of Stufflebeam (1971). The findings indicate that this conceptual framework is suitable for use in this study at the next stage and can be adapted by other researchers.

The evaluation of Disaster Resilient Village Program to support National Security matters

Disaster Resilient Village is a village with capability to independently adapt and deal with disaster threat, and quickly recover from the adverse effect of disaster. This research employed Context, Input, Process, and Product (CIPP) evaluation model to evaluate the Disaster Resilient Village program in Tanjung Benoa Village. As the result, the Context aspect of the Disaster Resilient Village program in Tanjung Benoa Village is appropriate in accordance with the Regulation of the Head of the National Disaster Management Agency of the Republic of Indonesia Number 1 of 2012. On the Input aspect, Tanjung Benoa village has established a Disaster Risk Reduction Forum that consists of the local community representatives. On the Process aspect, the village has developed Disaster-Safe School and Communities that routinely conduct simulations on monthly basis and collaborate with the hospitality sector to develop evacuation sites in the event of a tsunami disaster. On Product aspect, namely community capacity building and community participation, the program resulted in the increasing trust among domestic and foreign tourists and the improvement of Tanjung Benoa people’s welfare. In conclusion, Disaster Resilient Village program in Tanjung Benoa Village has been going well.

An analytic network process model for hospital facilities management performance evaluation

PurposeThis study aims to establish a rigorous model that can pragmatically evaluate the facilities management (FM) performance of hospitals.Design/methodology/approachAmong the applicable performance indicators that were identified from extant literature, a focus group study shortlisted ten key performance indicators (KPIs) in four categories (safety, physical, financial and environmental) and verified their practicality. Using the analytic network process (ANP) method to process the focus group’s responses yielded importance weightings for the KPIs and developed the intended evaluation model. This model was then validated by a case study.FindingsFrom the empirical data collected, two types of FM performance data and two scenarios of KPI scores were identified. To process these data and scores, a robust calculation method was devised and then proved useful in obtaining an overall score for holistic hospital FM performance. The case study confirmed the appropriateness and validity of the model developed.Research limitations/implicationsThrough illustrating how the ANP method could be applied to develop an FM performance evaluation model, the study contributes knowledge to the multi-criteria decision-making domain. Despite the geographical limitation of the model established (i.e. centered around a group of hospitals investigated in Hong Kong), the study can serve as a reference for developing performance evaluation models for other buildings or infrastructures globally.Practical implicationsThe model constitutes a practical tool for evaluating the FM performance of hospitals. Using this model on a regular basis will enable performance benchmarking and hence, continuous improvement of FM services.Originality/valueThe ANP model established is the first of its kind tailored for evaluation of hospital FM performance.

Clinical placements as a challenging opportunity in midwifery education: A qualitative study.

AimThe aim of this qualitative study was to explore some of the existing statuses of clinical education and examine the strengths and weaknesses of the clinical faculty programme.DesignA qualitative study.MethodsThis qualitative study was designed through the directed content analysis approach, which was completed according to the Context, Input, Process, and Product evaluation model. The purposeful sampling method of maximum variation was employed to select 33 participants, and the data were collected through in‐depth individual interviews and focus group discussion.ResultsAfter the interview transcripts were analysed, the participants' opinions of the clinical faculty programme and the relevant challenges, dimensions and factors were classified into four categories and sixteen subcategories. The categories were compiled in the CIPP evaluation model, including context, input, process and product. Based on the content analysis results, appropriate planning was recommended for teaching effective clinical skills to students.

Evaluation of agricultural extension model sites approach in Iran

Evaluation provides effective feedback for development plans and programs. In this respect, it is of utmost importance to ensure that the outputs of agricultural extension and education projects are compatible with the ones expected. Therefore, the main purpose of this study was to evaluate agricultural extension model sites approach from actors’ perspectives and to analyze their gaps via the context, input, process, and product (CIPP) evaluation model. The study was quantitative, applied, survey-based, and causal-comparative in terms of nature, purpose, methodology, and type of research, respectively. The samples included 150 main and follower farmers from a total number of 40 model sites and 37 subject-matter experts selected using the random and purposive sampling methods, respectively. The data collection instrument was a researcher-made questionnaire, whose reliability was confirmed through computing Cronbach’s alpha coefficient (0.75 ≤ α ≤ 0.90) and its validity was established by a panel of experts. The data were further analyzed using the SPSS 24 and comparative statistical tests. The comparison results demonstrated that the mean values of the experts’ views towards all items at different evaluation stages (i.e., context, input, process, product, output, and re-engineering) were higher compared with those of the farmers. Moreover, the farmers’ perspectives at the context and the input evaluation stages were not the same as those of the experts, and their satisfaction with the project had boosted as they had approached the output evaluation stage. The results of the gap analysis similarly indicated that the largest negative gap between the views of the experts and the farmers was associated with the input evaluation stage and the smallest gap was related to the site re-engineering. Accordingly, much more attention should be paid to building and maintaining the trust of farmers during the early stages of planning for and implementation of agricultural extension model sites.

Evaluation of the implementation of the sekolah penggerak curriculum using the context, input, process and product evaluation model in high schools

Learning evaluation is to find out how far the progress, development, and success of students after carrying out the learning process within the specified time period is, including the evaluation of the innovative learning curriculum, namely the <em>sekolah penggerak</em> curriculum. The purpose of this study is to describe the evaluation of the school curriculum using the Context, Input, Process and Product (CIPP) evaluation model. Qualitative research designed in the form of case study research. The research subjects were the principal, curriculum representative, and school supervisor. Determination of research subjects was carried out purposively. Data collection techniques using observation sheets and data analysis techniques used are descriptive statistics. Research findings using the Context, Input, Process and Product (CIPP) is seen from the Context, Input, Process and Product aspects that the implementation of the <em>sekolah penggerak</em> curriculum is in accordance with the learning objectives. The implementation of this curriculum innovation is expected to have an impact on the quality of learning outcomes, the quality of the main competencies, the quality of teachers, the acceleration of school digitization and the profile of Pancasila students.

Pedati Model Online Learning in University

Since the COVID-19 pandemic hit the world, various changes have occurred in the world of education. Teachers are required to be able to develop online learning so that learning activities can run smoothly. The purpose of this study was to analyze the effectiveness of online learning in universities. The research subjects were lecturers totalling 22 people. The research object includes the effectiveness of online learning programs in terms of context, input, process, and product. The approach used in this study is evaluative. The research model design used is the CIPP evaluation model (Context, Input, Process and Product Evaluation Model). The data were analyzed descriptively by calculating the percentage. Next, the data were analyzed by the T-Score formula. The level of effectiveness of online learning was analyzed through the quadrant of the "Glickman" model. The results of data analysis showed that online learning in PGSD, in terms of context, descriptively classified as very good, with an effectiveness percentage of 18.18%. Online learning at PGSD Undiksha in terms of input, descriptively is very good, with an effectiveness percentage of 0%. In terms of process, online learning at PGSD Undiksha is descriptively classified as very good, with an effectiveness percentage of -9.09%. In terms of products, online learning at PGSD Undiksha descriptively is classified as very good, with an effectiveness percentage of 18.18%. It can be concluded that the online learning that has been held is excellent and effective.

Multidimensional Item Response Theory in the Style of Collaborative Filtering.

This paper presents a machine learning approach to multidimensional item response theory (MIRT), a class of latent factor models that can be used to model and predict student performance from observed assessment data. Inspired by collaborative filtering, we define a general class of models that includes many MIRT models. We discuss the use of penalized joint maximum likelihood to estimate individual models and cross-validation to select the best performing model. This model evaluation process can be optimized using batching techniques, such that even sparse large-scale data can be analyzed efficiently. We illustrate our approach with simulated and real data, including an example from a massive open online course. The high-dimensional model fit to this large and sparse dataset does not lend itself well to traditional methods of factor interpretation. By analogy to recommender-system applications, we propose an alternative "validation" of the factor model, using auxiliary information about the popularity of items consulted during an open-book examination in the course.

Spatial analysis and modelling for primary healthcare site selection in Midnapore town, West Bengal.

Unprecedented and chaotic growth of cities results in reducing open spaces and water bodies, worsening infrastructure facilities and changes in ecological morphology. This unregulated growth of the urban population led to uneven distribution of urban amenities, facilities and healthcare services. Considering this, the study aimed to draw attention to the existing spatial pattern of healthcare facility centres as well as to find out the possible sites for the provision of healthcare facility centres in the municipal ward (micro-scale) of Midnapore town. This prototype study was conducted using Analytical Hierarchy Process (AHP) and Ordinary Least Square (OLS) evaluation model based on various criteria through Arc GIS environment. The findings indicate that the spatial distribution patterns of existing public healthcare centres were significantly dispersed. Weights based on a set of criteria were calculated by AHP and OLS algorithm and generated suitability evaluation maps classified from 1 (poor suitable) to 4 (most suitable). According to the employed criteria in this study unveil those existing hospitals and primary healthcare centres have not been located in the appropriate locations. The model is found to be valid for the given study area and there is no significant difference between AHP and OLS results. Further, it can be used for preparing the suitability map for the other areas with similar geo-environmental conditions for the proviso of healthcare services as well as will be most effective in preventing disease progression and reducing healthcare inequality on a large scale.Supplementary InformationThe online version contains supplementary material available at 10.1007/s10708-021-10528-w.

A process model for quality in use evaluation of clinical decision support systems

ContextClinical decision support systems (CDSSs) are used to help healthcare professionals in making decisions, offering them a tool for improved medical care practices based on monitoring and management procedures. Although CDSSs exhibit many advantages, challenges remain in terms of their adoption in the clinician community. One such issue is related to user satisfaction and the system reliability. Ensuring the quality of CDSSs is a way to improve their acceptance and adoption. ObjectiveThis study aims to propose a process model for evaluation of the quality in use characteristics of a CDSS to identify deficiencies that reduce its use by healthcare professionals. MethodsWe reviewed the existing literature on CDSS assessment and developed a process model based on the international standards ISO/IEC 25010 System and software quality models, and ISO/IEC 25022 Measurement of quality in use. To select measures for evaluating these characteristics, we adopted the Goal-Question-Metric (GQM) method. We evaluated the quality in use characteristics because they can represent system usability. Measurement of these characteristics helps us understand user needs, improve the user experience, and mitigate the low acceptance of CDSS, particularly by the primary users. ResultsWe developed a process model for measuring the quality in use (QiU) characteristics of CDSSs, explaining its applicability through an illustrative example focused on the characteristics of satisfaction and efficiency. ConclusionWe consider that the proposed process model will benefit the CDSS adoption and contribute to the improvement of the quality of such systems by measuring its QiU.

ACCELERATING STUDENTS’ CAPABILITIES IN ACCOMPLISHING FINAL PAPERS: AN EVALUATION

This paper aimed at evaluating such aspects as self-efficacy, research methodology, types of supervisor, and academic writing ability in accelerating students accomplishing final papers. 214 students of bachelor and master programs at Universitas Riau - Indonesia were selected through purposive sampling technique as the participants. This research used Context, Input, Process, and Product Evaluation Model to evaluate those variables. The research findings revealed that (a) there was no significant difference in self-efficacy on the aspects of gender ((sig. 0.219 > p 0.05) and learning support facilities ((sig. 0.073 > p 0.05); (b) in research methodology on the aspect of gender ((sig. 0.508 > p 0.05); (c) in academic writing ability on the aspects of gender (sig. 0.939 > p 0.05); and (d) in types of supervisors on the aspects of gender (sig.0.618 > p 0.05), sources of educational cost (sig. 0.474 > p 0.05), and learning support facilities (sig. 0.773 > p 0.05). However, (a) there was a significant difference in self-efficacy on the aspects of profession (sig. 0.000 < p 0.05), and sources of educational cost (sig. 0.014 < p 0.05); (b) in research methodology on the aspects of profession (sig. 0.000 < p 0.05), sources of educational cost (sig. 0.004 < p 0.05), and learning support facilities (sig. 0.001 < p 0.05); (c) in academic writing ability on the aspects of profession (sig. 0.013 < p 0.05 ), sources of educational support (sig. 0.014 < p 0.05), and learning support facilities (sig. 0.000 < p 0.05); and (d) in types of supervisors on the aspect of profession (sig. 0.022 < p 0.05). Partially, self-efficacy contributed 59.7%; research methodology (10.4%) and types of supervisor (0.1%) ---- which is totally 70.2% towards the students’ academic writing ability in accelerating students accomplishing final papers.

Evaluasi Program Praktek Lapangan Industri Jurusan Teknik Elektro Universitas Negeri Padang

This study aims to determine the condition of the context, input, process, and product components in the PLI program at the Department of Electrical Engineering, Padang State University. The type of research used in evaluation research with a quantitative approach. This study uses the CIPP context, input, process, and product evaluation model developed by Daniel Stufflebeam with a population of 79 students majoring in electrical engineering who have carried out PLI. The data collection of this research is in the form of a questionnaire. Data analysis techniques used are data verification, data classification, and data processing. The results of this study indicate that the implementation of the PLI program in the Department of Electrical Engineering has been categorized as good, but there are several indicators that must be improved in the future. In the context evaluation, the PLI program objective indicator obtained a respondent's achievement level of 90.54% categorized as very good, the PLI program environmental indicator with a respondent achievement level of 86.75% categorized as good. The input evaluation consists of five indicators, namely supporting facilities with 88% of respondents categorized as good, indicators of sources of funds with 60.8% of respondents being categorized as adequate, program relevance indicators with 85.8% of respondents being categorized as good, indicators of direction, and guidance from majors and industries, the achievement level of respondents was 81.9% categorized as good, and in the Student HR indicator, the achievement level of respondents was 81.4% categorized as good. Process evaluation, indicators of preparation for the implementation of the PLI program with the level of achievement of respondents 84.6% categorized as good, indicators of implementation of the PLI program at the level of achievement of respondents 87.6% categorized as good, indicators of monitoring the implementation of PLI with the level of achievement of respondents 53.5% categorized as not good, and indicators barriers to the implementation of the PLI respondents' achievement level of 83.5% is categorized as good. Product evaluation obtained a respondent's level of achievement of 88.01% which was categorized as good.

Performance of the Taiwan Earth System Model in Simulating Climate Variability Compared With Observations and CMIP6 Model Simulations

AbstractThis study evaluates the performance of the Taiwan Earth System Model version 1 (TaiESM1) in simulating the observed climate variability in the historical simulation of the Coupled Model Intercomparison Phase 6 (CMIP6). TaiESM1 is developed on the basis of the Community Earth System Model version 1.2.2, with the inclusion of several new physical schemes and improvements in the atmosphere model. The new additions include an improved triggering function in the cumulus convection scheme, a revised distribution‐based formula in the cloud fraction scheme, a new aerosol scheme, and a unique scheme for three‐dimensional surface absorption of shortwave radiation that accounts for the influence of complex terrains. In contrast to the majority of model evaluation processes, which focus mainly on the climatological mean, this evaluation focuses on climate variability parameters, including the diurnal rainfall cycle, precipitation extremes, synoptic eddy activity, intraseasonal fluctuation, monsoon evolution, and interannual and multidecadal atmospheric and oceanic teleconnection patterns. A series of intercomparisons between the simulations of TaiESM1 and CMIP6 models and observations indicate that TaiESM1, a participating model in CMIP6, can realistically simulate the observed climate variability at various time scales and are among the leading CMIP6 models in terms of many key climate features.

Modeling of the process approach to the life cycle of measuring instruments

The problem of increasing the reliability and competitiveness of products in all sectors of the national economy can be solved only on the basis of obtaining complete and reliable measurement information. This is facilitated by modern measuring instruments (MI), which are complex hardware and software systems. The relevance of the study is due to the fact that modern MI need an effective assessment of their quality at all stages of the life cycle (LC). This requires the development of appropriate assessment methods both at the stage of production and operation of MI. The expediency of using the process approach to MI LC stages and its advantages over the functional approach were proved. The process approach allows more effective assessment of MI quality indicators at different LC stages and is compatible with the construction of modern quality management systems. Mathematical modeling was carried out, a set process model of the MI LC was developed and its representation as a process V-model was carried out. This allows studying the interaction of processes of all MI LC stages and performing process quality management at all MI LC stages. Mathematical modeling was carried out, a set process model of evaluation of quality indicators of MI LC stages was developed and its representation as a process V-model was carried out. This allows evaluating quality indicators of the MI and its components throughout the MI LC. Recommendations for the use of international standards were formulated, in particular for project management planning, measurement processes, system requirements throughout the MI LC, risk analysis and management at the LC stages. This should help increase efficiency in achieving the planned results at all stages of the MI LC

The impact of emergency remote teaching on a blended engineering course: perspectives and implications for the future.

Higher education institutions globally were forced to transition to remote teaching and learning when the Covid-19 pandemic impacted the world in 2020. The rushed, unplanned nature of the transition led to the approach being labeled Emergency Remote Teaching (ERT). This paper evaluates the impact of ERT on a blended course in engineering using a descriptive case study approach applying the Context, Input, Process and Product (CIPP) evaluation model. The context analysis highlighted the need for consistent training in the use of technology, technical support for stakeholders, greater access to the Internet and timely, targeted communication. Students appreciated the convenience of online classes and accessibility to recorded lectures and labs allowing them to review at their own pace. There was a perception that the new learning environment placed some students at a disadvantage. These findings suggested a need to ensure deliberate planning for online learning from the start and attention to building a community of learners. Findings from the study can contribute to a university’s exploration of the academic enterprise. These findings can also help identify mitigating factors for effective online learning.Supplementary InformationThe online version contains supplementary material available at 10.1007/s43545-021-00172-z.

Evaluation of the Chemistry curriculum at the lower secondary level: feedback from educators

Abstract Mauritius is presently witnessing a major educational reform, which has necessitated a review of the lower secondary curriculum in all subjects, including Science. Educators were called to evaluate the Science curriculum (which includes Biology, Chemistry and Physics components) by providing feedback on the curriculum content and content organisation. One hundred and thirty-three educators took part in this exercise. This paper focuses on a detailed analysis of the feedback obtained from educators regarding the lower secondary Chemistry curriculum (LSCC) to identify the challenges encountered in its implementation. The Process component of the Stufflebeam’s (Stufflebeam DL. The CIPP model for evaluation. In: Evaluation models. Dordrecht: Springer; 2000:279–317 pp) Context, Input, Process and Product (CIPP) Evaluation model has been used as a framework for the study. Based on qualitative thematic analysis, four main challenges were identified in the implementation of the LSCC namely, ensuring curriculum continuity, avoiding curriculum overload, the need for more laboratory experiences and knowledge transfer.

Model evaluation for policy insights: Reflections on the forum process

Model evaluation is best considered as a process for communicating with the policymaking or policy-advising community. Six decades of energy modelling have witnessed increasing complexity in these systems, a situation that raises a number of important challenges in using them effectively in policymaking organizations. When used as a learning rather than forecasting tool, these systems can be evaluated individually one by one or through joint efforts to compare them in multi-model exercises. After summarizing the evolution of energy modelling and efforts to evaluate them since the first oil embargo, this essay provides a guide to future evaluation collaborations by highlighting a few challenges that would improve the value of these studies for the policymaking community. These challenges range broadly and cover topics such as enhancing the engagement of the model user, ventilating the models’ complexity with intuitive insights, using simple models to demonstrate key parameters or responses, applying judicious occasional meta-analysis when there is value added, reporting model responses and calibrating them for decisionmakers, considering retrospective evaluation for a past period (when possible), selecting standardized or modeler-choice baseline conditions, selectively developing policy or diagnostic alternative cases, and institutionalising the model evaluation process for a specific topic or region.

Study on Comprehensive Evaluation Method for Construction Suitability of Precast Concrete Beam-column Connections

The key difference between prefabricated structure and cast-in-place structure is the connection of components. The comprehensive reliability of the connection directly determines the overall performance of the structure, and also affects its application scope in the building structure. It is particularly important to select the appropriate beam-column connections in the construction. In recent years, with the continuous development of precast concrete structure, a large number of precast beam-column connections have been invented. However, in view of different construction environment, construction cost and building function requirements, the selection of precast beam column connections is often difficult to choose. Therefore, this paper is based on the Analytical Hierarchy Process (AHP) evaluation model, combined with the evaluation index of the precast concrete beam-column connections, to provide a scientific method for selecting suitable precast beam-column connections for engineering construction.