Pentaho Data Integration Research Articles

Business Intelligence Implementation for Groceries Commodity Price Data Analytics

As a startup company specializing in agricultural technology, currently PT XYZ does not have any effective and integrated tools to analyze groceries commodity for purchase and sale data-driven decision making. This research will integrating multiple data sources for accurate analysis, tailoring the business intelligence (BI) system to PT XYZ’s specific needs, and developing a robust solution that supports comprehensive decision-making. BI implementation solution will be developed using Pentaho Data Integration & Apache Superset. Applied research used as its method, that contains processes to gather user requirements to gather user needs from each information, design the solution, and dashboard development. The study focused on analyzing data from both internal sources, such as purchase and sales transactions, and external sources, including market price data, using these BI tools to provide comprehensive insights. The developed dashboard contains several sections that allow users to see price recommendation price and get the insight for the last seven days trend, monitor the market position, see team performance, and find the summary about estimated and actual margin comparison. Pentaho successfully handled ETL and data modeling, while Apache Superset enabled straightforward dashboard setup, though chart customization is limited.

Automated Data Quality Control System in Health and Demographic Surveillance System

The automated data quality management system serves as a comprehensive solution developed to enhance the precision, dependability, and uniformity of data within data-driven organizations. Such systems play an important role in eliminating the shortcomings associated with manual data quality management, which is prevalent in health and demographic surveillance systems (HDSS). The ongoing difficulty of ensuring data quality through manual processing hinders the HDSS's capacity to optimize data quality effectively. To address this challenge, our study adopted design science methodologies to provide guidelines for the design and implementation of the automated data quality control system. The open source technologies (Pentaho data integration, R Studio, SQL, Windows task scheduler) were used to facilitate the automation and validation of the incoming and database resident data. The quality of data has vastly improved since the implementation of the proposed system. The findings suggest that the automated data quality control system exhibited superior performance compared to the manual methods, thereby minimizing errors and time-wasting efforts.

Implementation of Data Warehouse and Star Schema for Optimizing Property Business Decision Making

The property industry is a sector that faces intense competition, so innovation is needed to plan business strategies. PT Wijaya Karya Realty faces several problems, such as determining strategic locations for its property products, such as houses or shophouses, and comparing the price of company houses with the market price of homes in specific locations. To answer these needs, it is necessary to implement a data warehouse and Extract, Transform, Load (ETL). In this research, Pentaho Data Integration (PDI) is used as a tool for managing and cleaning data that can be used as analysis material to assist in data-based decision-making. Apart from that, Mondrian can be used through the workbench scheme to improve data analysis effectively. Designing a star schema that includes a dimensional table and fact table can help optimize the analysis process better. The implementation of a data warehouse allows companies to make in-depth decisions based on critical variables. The research results show that the implementation of Star Schema can provide information for companies in decision-making and business strategy planning in the property sector.

Cleenex: Support for User Involvement during an Iterative Data Cleaning Process

The existence of large amounts of data increases the probability of occurring data quality problems. A data cleaning process that corrects these problems is usually an iterative process, because it may need to be re-executed and refined to produce high-quality data. Moreover, due to the specificity of some data quality problems and the limitation of data cleaning programs to cover all problems, often a user has to be involved during the program executions by manually repairing data. However, there is no data cleaning framework that appropriately supports this involvement in such an iterative process, a form of human-in-the-loop, to clean structured data. Moreover, data preparation tools that somehow involve the user in data cleaning processes have not been evaluated with real users to assess their effort. Therefore, we propose Cleenex, a data cleaning framework with support for user involvement during an iterative data cleaning process, and conduct two data cleaning experimental evaluations: an assessment of the Cleenex components that support the user when manually repairing data with a simulated user; and a comparison, in terms of user involvement, of data preparation tools with real users. Results show that Cleenex components reduce the user effort when manually cleaning data during a data cleaning process, for example, the number of tuples visualized is reduced in 99%. Moreover, when performing data cleaning tasks with Cleenex, real users need less time/effort (e.g., half the clicks) and, based on questionnaires, prefer it to the other tools used for comparison, OpenRefine and Pentaho Data Integration.

Impact of business intelligence on incident management in the control center of a security company

The main objective of this article is to implement business intelligence (BI) technology to improve incident management in the control center of a security company; making use of the Ralph Kimball method for the development of the data mart, contemplating the phases: planning of projects, definition of requirements, design of technological architecture, performing a dimensional modeling a star with 9 dimensions and 1 table made, also used the structured query language (MySQL) database, extract, transform, and load (ETL). Development the pentaho data integration, which allowed the migration, transformation and cleaning of the data from the online transaction processing (OLTP) and online analytical processing (OLAP) database, obtaining as a result that the BI managed to increase the level of the service by 28.42% and reducing the frequency index by 25%.

Publication of Linked Open Data – A Systematic Literature Review for Identifying Problems and Technical Tools Supporting the Process

On the Internet, we find a large amount of information from government institutions that has been published in open format. However, only a part of these data is available in standard formats such as Resource Description Framework (RDF), and to a lesser extent, is published as Linked Open Data (LOD). The main objective of the research presented in this paper is to identify problems and tools used in the process of publishing LOD with the purpose of establishing a basis for the construction of a future framework that will help public institutions to facilitate such processes. To fulfill the objective, we conducted a systematic literature review in order to assess the state-of-the-art in this matter. The contribution of this work is to identify the frequent problems that arise in the LOD publishing process. It also provides a detail of the frameworks proposed in scientific papers grouping the technical tools by phases that correspond to the LOD publication life cycle. In addition, it compiles the characteristics of the ETL (Extract-Transform-Load) tools that predominate in this review, such as Pentaho Data Integration (Kettle) and OpenRefine.

Implementation of Business Intelligence In Analyzing Data Using Tableau at PT Global Bintan Permata

PT Global Bintan Permata is member group Yafindo (Batam, Indonesia) which is engaged in the distribution and import goods consumption and goods consumption throughout Indonesia . one _ business being run is distributing sugar _ throughout modern market and delivery some are via transportation and some are done in a manner independent . The target market namely the big modern market like Alfamart , Alfamidi , Indomarco , Indogrosir , Tiptop, and there is also a small modern market like Totalfruits , Grandlucky , etc. _ Of the many transactions made by the company this company _ This Still Not yet own Dashboard application for monitoring activity operations and transactions . So from it is needed something deep data visualization dashboard shape . Method used to data warehouse design is use method kimball nine-step technology. For the Extract, Transform and Load (ETL) process for data warehouse design using tools in the Pentaho Data Integration application , meanwhile For dashboard visualization is done with use the Tableau Public app . Result of application Tableau Public app is form generated data visualization dashboard reports containing information in form the data visualization it consists of from transaction and marketing data in 2021. Dashboard provides information For know How development sale both product and sales are also developing transactions with customers in 2021 later will become base in taking decisions required by management within PT Global Bintan Permata for help in the process of taking decision.

Analysis of Student Performance Applying Data Mining Techniques in a Virtual Learning Environment

Students' academic performance is a key factor for educational institutions and society, which is an important indicator of the quality of the teaching-learning process and the appropriation of knowledge. Its analysis allows an understanding of the behavior of students and teachers, generating valuable knowledge for making timely academic decisions. In this study, the following phases were carried out: (i) identification of factors that influence the academic performance of engineering university students, (ii) early prediction of academic success, and (iii) identification of use patterns in a virtual learning environment (VLE). The Knowledge Discovery in Databases (KDD) methodology was applied based on predictive and descriptive data mining techniques, using academic and socioeconomic data and interactions (resources and activities) with the VLE. The tools and programming languages ​​used were Pentaho Data Integration for data integration and processing; Jupyter Notebook, Python, and Scikit-Learn for correlation analysis and prediction modeling; and R Studio for the clustering task. The results show that VLE resources such as files, links, and activities such as participation in forums are factors that lead to good academic performance. On the other hand, it was possible to make predictions of academic success (pass or not) with an accuracy greater than 95% and to identify the main patterns of use of the VLE. The group with excellent academic performance (grades 9 to 10) is recognized for using file-type resources and high participation in class and forum activities.

Implementation of Business intelligence in Know Your Customer (KYC) for credit card customers’ loan repayment status

Nowadays, the increasing technological improvements followed by data demand made businesses and organizations follow these improvements to continue operating. The Know Your Customer (KYC) assessment is still manually done, followed by the growing amount of data, resulting in data accumulation and affecting the time organizations spend analyzing the prospective applicant data. The unavailability of tools that can help review documents and data leads to more data accumulation and timeconsuming KYC assessments. This paper aims to create a Business Intelligence (BI) system to help financial organizations analyze, process applicant data, and determine applicants’ eligibility who are willing to get credit card services. This paper may be utilized by an enterprise operating in the financial sector that follows the KYC procedure to identify applicants who require credit card services. The development of BI systems is predicted to help reduce the time spent to validate application data, particularly in the banking or financial industry. This study has designed three dimension tables and a fact table using Microsoft SQL Server 18 for the data warehouse. Pentaho Data Integration is used for the ETL process, and Tableau creates the dashboard. The dashboard contains general information and the loan repayment status of an applicant. Two pivot tables were created using Microsoft Excel to summarize the loan repayment status of an applicant.

Design of a Sales Performance System for SMEs based on Business Intelligence and Data Warehouse

The influence of information technology today is powerful. It impacts people's lives because technological changes are running so fast and affect the way of thinking and behaving in competition in the business world and organizations. Small and Medium Enterprises (SMEs) must be able to adapt to this technology to maintain their business. It means that digitizing SMEs means integrating technology into all business activities. In this study, Toko Cerme is the object of research. The Toko Cerme is a SMEs in the form of a minimarket located in Central Java, Indonesia. The Toko Cerme takes advantage of technology to help run business processes so that they can be managed optimally. In running its business, The Toko Cerme is currently using an information system to input product data and transaction activities. The purpose of this research is to propose a Design of a Sales Performance System based on Business Intelligence and a Data Warehouse to support business processes at the Toko Cerme so that it can efficiently process data and information in the future. From the research that the authors conducted, it can be concluded that the results of this study are the creation of a data warehouse and business intelligence design using the nine steps Kimbal method. At the same time, Pentaho Data Integration (PDI) is a tool. The design is used as a reference in producing information relating to sales transactions.
 
 

Penerapan Business Intelligence & Online Analytical Processing untuk Data-Data Penelitian dan Luarannya pada Perguruan Tinggi Menggunakan Pentaho

Utilization of ICT in various sectors is needed, especially those related to business impact and strategy. Information technology is the backbone of the sustainability of businesses, companies, and organizations. Companies that are able to utilize ICT well, therefore indirectly adapt to the times and strive to excel from competitors. The method used refers to the steps recommended by Carlo Vercellis. The software and tools used are open source based, such as Pentaho Data Integration for processing extract, transform, load (ETL), Pentaho Community Edition for dashboards, Pentaho Report Designer for report generation, and Mondrian OLAP for displaying multidimensional data. The results of this study conclude application of business intelligence in universities is very easy and efficient, the use of a dashboard that is presented visually and interactively is very helpful for leaders in viewing existing research data. So it is very helpful for organizations, especially university leaders in making decisions.

Development of Business Intelligence For Integration of Financial Reporting System In PT XYZ

The research aims to determine the relationship between two or more variables. Data collection with Literature and field research with questionnaires. The result of the research that is more influential in BAPPEDA Pemprov DKI Jakarta is the Business Intelligence variable, so it needs to be prioritized and given special attention to its implementation in the field. The research method used is associative research, designing Data Warehouse architecture, planning Data Warehouse sources, dimensional data modeling, filling in Data Warehouse. And the results of this study can be used as a reference for further follow-up with the DKI Jakarta Provincial Government BAPPEDA to be prioritized in the near future. The conclusion from the results of this study is that a transaction data warehouse designed using PDI (Pentaho Data Integration), is very helpful in collecting transaction data from original transaction data that already has a database and only as OLTP (Online Transactional Processing) data so that can be used as data OLAP. (Online Analysis Processing) so that it is analyzed using OLAP.

Perancang Data Warehouse Dan Visualisasi Data Mutu Penerimaan Beras

Rice is a staple food consumed by a large portion of the Indonesian population. Each region has its own rice production so that it has different qualities.. Indonesia itself has specific standards for good quality rice. In order for rice can be distributed evenly throughout the archipelago, Indonesia has a rice management organization, one of which is PT Food Station Tjipinang Jaya. Rice from various suppliers must be recorded and checked for quality. Making a Data Warehouse needs to be implemented so that it is easily collected and analyzes the data received and can be used as a reference for decision making. To build a data warehouse can use Extract, Transform and Load (ETL) available in Pentaho Data Integration. Data that has been entered into the data warehouse can be visualized using Python to make further decisions.

Utilization Extract, Transform, Load For Developing Data Warehouse In Education Using Pentaho Data Integration

The utilization of data warehouses in various fields is an absolute necessity. A data warehouse is a database that contains large amounts of data that aims to help organizations, fields, and institutions specifically for decision making. Data warehouses can produce important information in the future. Loading data from various sources and processed through an ETL (Extract, Transform, Load) process that displays data consistently is the basis for creating a data warehouse architecture. The development of a data warehouse in education will provide significant benefits for the progress of education. Integration of data and processing results stored in the data warehouse can be the basis for evaluating better planning. Development of data warehouse adopt the multidimensional modelling method which consists of four stages: select the business process, declare the grain, select dimensions, and identify facts. This stage produces a data warehouse architecture and influences and contributes to the advanced information technology in education.

RANCANGAN DATA WAREHOUSE AKADEMIK POLITEKNIK PARIWISATA BALI BERBASIS LINUX OPENSUSE

This study aims to design an academic data warehouse at Politeknik Pariwisata Bali because there are several information systems that require a system capable of handling databases of several existing information system applications and perform analysis. A data warehouse is the right solution for handling databases of several information systems that are not yet integrated. The research method begins with literature study, identification of problems, determining research objectives, designing solutions, namely designing an academic data warehouse, determining the hardware and software specifications needed, and ending with discussions and drawing conclusions. The result of this research is an academic data warehouse design that can be implemented properly. Besides, the results of this study are also accompanied by specifications for open-source hardware and software to save costs. The data warehouse design produced in this study uses a database of 3 information systems which is the main business process of a college, namely: Academic Information System, Information System for New Student Admissions, Research Information System. The three databases of the information system are internal data sources using the MySQL DBMS. The resulting academic data warehouse design uses open source software, namely: openSUSE, Pentaho Data Integration, MySQL DBMS.

RDFAdaptor: Efficient ETL Plugins for RDF Data Process

Abstract Purpose The interdisciplinary nature and rapid development of the Semantic Web led to the mass publication of RDF data in a large number of widely accepted serialization formats, thus developing out the necessity for RDF data processing with specific purposes. The paper reports on an assessment of chief RDF data endpoint challenges and introduces the RDF Adaptor, a set of plugins for RDF data processing which covers the whole life-cycle with high efficiency. Design/methodology/approach The RDFAdaptor is designed based on the prominent ETL tool—Pentaho Data Integration—which provides a user-friendly and intuitive interface and allows connect to various data sources and formats, and reuses the Java framework RDF4J as middleware that realizes access to data repositories, SPARQL endpoints and all leading RDF database solutions with SPARQL 1.1 support. It can support effortless services with various configuration templates in multi-scenario applications, and help extend data process tasks in other services or tools to complement missing functions. Findings The proposed comprehensive RDF ETL solution—RDFAdaptor—provides an easy-to-use and intuitive interface, supports data integration and federation over multi-source heterogeneous repositories or endpoints, as well as manage linked data in hybrid storage mode. Research limitations The plugin set can support several application scenarios of RDF data process, but error detection/check and interaction with other graph repositories remain to be improved. Practical implications The plugin set can provide user interface and configuration templates which enable its usability in various applications of RDF data generation, multi-format data conversion, remote RDF data migration, and RDF graph update in semantic query process. Originality/value This is the first attempt to develop components instead of systems that can include extract, consolidate, and store RDF data on the basis of an ecologically mature data warehousing environment.

Data analysis and data warehouse design based on Pentaho data integration (kettle) to support the determination of student learning achievement

Data analysis has become a vital requirement in various fields. The results of data analysis can be used as a data warehouse design that is expected to help in solving problems, evaluating learning outcomes and supporting decision making. In the field of education, student achievement is a reference for achieving quality learning. This success can be seen from the achievement of student learning completeness that is the basis for evaluating and making decisions. The data warehouse design can be used as a basis in knowing student learning progress that can be seen from the value obtained. The diversity of data makes managing values difficult and delayed, data warehouse design using Pentaho can help and simplify data integration. The results of the analysis and design of the data warehouse will be presented in a multidimensional form that can be seen through dimension tables and fact tables.

Tableau Business Intelligence Using the 9 Steps of Kimball’s Data Warehouse & Extract Transform Loading of the Pentaho Data Integration Process Approach in Higher Education

This paper discusses about how to build a data warehouse (DW) in business intelligence (BI) for a typical marketing division in a university. This study uses a descriptive method that attempts to describe the object or subject under study as it is, with the aim of systematically describing the facts and characteristics of the object under study precisely. In the elaboration of the methodology, there are four phases that include the identification and source data collection phase, the analysis phase, the design phase, and then the results phase of each detail in accordance with the nine steps of Kimball’s data warehouse and the Pentaho Data Integration (PDI). The result is a tableau as a tool of BI that does not have complete ETL tools. So, the process approach in combining PDI and DW as a data source certainly makes a tableau as a BI tool more useful in presenting data thus minimizing the time needed to obtain strategic data from 2-3 weeks to 77 minutes.

Business Intelligence Design for Data Visualization and Drug Stock Forecasting

Klinik Pratama is one form of service provided by the Ministry of Communication and Information of the Republic of Indonesia in protect employees from health disorders that could affect employee productivity. In its development, the clinic often finds problems, one of them is often a shortage or excess the drug stock on a running period. Therefore, it be required a design of an Business Intelligence that manages complex data into a data visualization forecasting of the future stock of drugs. Historical data processing of the drug is done with process of Extract, Transform and Load (ETL) using the Spoon Pentaho Data Integration tools. While the visualization of drug stock data and forecast results is done using Microsoft Power BI (Business Intelligence) tools and for forecasting is done with Artificial Neural Network method by RStudio tools. The results of forecasting the amount of stock out of drug samples using the Artificial Neural Network method obtained an MSE value of 67.72 and RMSE 8.22 which means that this forecast has a good ability with the resulting error rate is relatively small. From this research, the Klinik Pratama of the Ministry of Communication and Information can easily understand and analyze drug stock data and can support operational decision making The results of forecasting the amount of stock out of drug samples using the Artificial Neural Network method obtained an MSE value of 67.72 and RMSE 8.22 which means that this forecast has a good ability because the resulting error rate is relatively small. From this research, Klinik Pratama of the Ministry of Communication and Information can easily understand and analyze drug stock data and can support operational decision making The results of forecasting the amount of stock out of drug samples using the Artificial Neural Network method obtained an MSE value of 67.72 and RMSE 8.22 which means that this forecast has a good ability with the resulting error rate is relatively small. From this research, Klinik Pratama of the Ministry of Communication and Information can easily understand and analyze drug stock data and can support operational decision making.

Implementation of a learning healthcare system for sickle cell disease.

ABSTRCTObjectiveUsing sickle cell disease (SCD) as a model, the objective of this study was to create a comprehensive learning healthcare system to support disease management and research. A multidisciplinary team developed a SCD clinical data dictionary to standardize bedside data entry and inform a scalable environment capable of converting complex electronic healthcare records (EHRs) into knowledge accessible in real time.Materials and MethodsClinicians expert in SCD care developed a data dictionary to describe important SCD-associated health maintenance and adverse events. The SCD data dictionary was deployed in the EHR using EPIC SmartForms, an efficient bedside data entry tool. Additional data elements were extracted from the EHR database (Clarity) using Pentaho Data Integration and stored in a data analytics database (SQL). A custom application, the Sickle Cell Knowledgebase, was developed to improve data analysis and visualization. Utilization, accuracy, and completeness of data entry were assessed.ResultsThe SCD Knowledgebase facilitates generation of patient-level and aggregate data visualization, driving the translation of data into knowledge that can impact care. A single patient can be selected to monitor health maintenance, comorbidities, adverse event frequency and severity, and medication dosing/adherence.ConclusionsDisease-specific data dictionaries used at the bedside will ultimately increase the meaningful use of EHR datasets to drive consistent clinical data entry, improve data accuracy, and support analytics that will facilitate quality improvement and research.