Conventional Data Collection Methods Research Articles

CALLM: Enhancing Clinical Interview Analysis through Data Augmentation with Large Language Models.

The global prevalence of mental health disorders is increasing, leading to a significant economic burden estimated in trillions of dollars. In automated mental health diagnosis, the scarcity and imbalance of clinical data pose considerable challenges for researchers, limiting the effectiveness of machine learning algorithms. To cope with this issue, this paper aims to introduce a novel clinical transcript data augmentation framework by leveraging large language models (CALLM). The framework follows a "patient-doctor role-playing" intuition to generate realistic synthetic data. In addition, our study introduces a unique "Textbook-Assignment-Application" (T-A-A) partitioning approach to offer a systematic means of crafting synthetic clinical interview datasets. Concurrently, we have also developed a "Response-Reason" prompt engineering paradigm to generate highly authentic and diagnostically valuable transcripts. By leveraging a fine-tuned DistilBERT model on the E-DAIC PTSD dataset, we achieved a balanced accuracy of 0.77, an F1-score of 0.70, and an AUC of 0.78 during test set evaluations, which showcase robust adaptability in both Zero-Shot Learning (ZSL) and Few-Shot Learning (FSL) scenarios. We further compare the CALLM framework with other data augmentation methods and PTSD diagnostic works and demonstrates consistent improvements. Compared to conventional data collection methods, our synthetic dataset not only demonstrates superior performance but also incurs less than 1% of the associated costs.

Enhancing Energy Consumption Prediction in Smart Homes: A Convergence-Aware Federated Transfer Learning Approach

Achieving accurate energy consumption prediction can be challenging, particularly in residential buildings, which experience highly variable consumption behavior due to changes in occupants and the construction of new buildings. This variability, combined with the potential for privacy breaches through conventional data collection methods, underscores the need for novel approaches to energy consumption forecasting. The proposed study suggests a new approach to predict energy consumption, utilizing Federated Learning (FL) to train a global model while ensuring local data privacy and transferring knowledge from information-rich to information-poor buildings. The proposed method learns the transferable knowledge from the source building without any privacy leakage and utilizes it for target buildings. Since the performance of the global model could be negatively affected by some participating nodes with poor performance due to noisy or limited data, we propose a client selection strategy on the server based on the normal distribution for choosing the best possible participants for the global model. Our method enables clients to participate selectively in the aggregation procedure to avoid model divergence due to poor performance. The proposed model is evaluated and conducts in-depth analyses of energy consumption patterns. We validate the performance by comparing its MAE, MSE, and R2 values to those of existing traditional and ensemble models. Our findings indicate that the proposed FL-based model with selective client participation outperforms its counterpart methods regarding predictive accuracy and robustness. The source code is available on Github

Real‐Time Air Quality Index Detection through Regression‐Based Convolutional Neural Network Model on Captured Images

ABSTRACTOur research addresses the imperative need for an efficient air quality monitoring and forecasting system to mitigate the significant health risks of air pollution. Departing from conventional binary data collection methods, we employ image‐based techniques to overcome inherent limitations. A pioneering aspect of our work involves the development of a novel model capable of predicting six air pollutants (PM2.5, PM10, O3, CO, SO2, NO2) along with Air Quality Index for Bengaluru, Delhi, and Tamil Nadu, achieving a commendable mean absolute error of 0.1432 on the test set. The efficacy of our approach is validated through a meticulously curated dataset comprising approximately 5455 images. We emphasize the significance of normalization by presenting outputs before and after, shedding light on the impact of parameters with varying ranges and strategies employed to mitigate such discrepancies. A detailed analysis of our model's best and worst outputs provides valuable insights into its strengths and limitations. To enhance user accessibility, we introduce an innovative image‐based, real‐time, user‐friendly dashboard that allows users to conveniently assess a location's air pollution levels by uploading an image. This holistic approach offers a promising avenue for accurate air quality prediction and real‐time monitoring.

Structured timeline mapping as a data collection methodology: a new perspective for research on environmental adaptation

ABSTRACT The process of adaptation is dynamic and involves a temporal component. However, conventional methods of data collection have mostly been used in research on environmental adaptations and provide a rather static view. The structured timeline mapping method presented here is grounded in social sciences, designed for the West African environment, and constructed to capture time. The different steps for its construction and implementation are outlined along with the underlying reasoning that led to the choices made. It appears both respondents and interviewers benefit from this method since the recall process is facilitated for the first and the visual support gives a global view of the interview at all stages for the latter. Additionally, this methodology offers four major benefits. First, temporality is the central information collected, with both the timing and the duration of the elements. Second, this approach enables the observation of linkages and interrelations between adaptations or between adaptations and perceived changes in the environment. Third, the participants can self-reflect on their situation and bring elements to the analysis. Lastly, structured timeline mapping results in a more holistic view of the respondent’s adaptation journey. Therefore, this data collection methodology provides a complementary view of the adaptation journey.

The Problem of Data Extraction in Social Media: A Theoretical Framework

In today's rapidly evolving digital landscape, the pervasive growth of social media platforms has resulted in an era of unprecedented data generation. These platforms are responsible for generating vast volumes of data on a daily basis, forming intricate webs of patterns and connections that harbor invaluable insights crucial for informed decision-making. Recognizing the significance of exploring social media data, researchers have increasingly turned their attention towards leveraging this data to address a wide array of social research issues. Unlike conventional data collection methods such as questionnaires, interviews, or focus groups, social media data presents unique challenges and opportunities, demanding specialized techniques for its extraction and analysis. However, the absence of a standardized and systematic approach to collect and preprocess social media data remains a gap in the field. This gap not only compromises the quality and credibility of subsequent data analysis but also hinders the realization of the full potential inherent in social media data. This paper aims to bridge this gap by presenting a comprehensive framework designed for the systematic extraction and processing of social media data. The proposed framework offers a clear, step-by-step methodology for the extraction and processing of social media data for analysis. In an era where social media data serves as a pivotal resource for understanding human behavior, sentiment, and societal dynamics, this framework offers a foundational toolset for researchers and practitioners seeking to harness the wealth of insights concealed within the vast expanse of social media data.

TransJakarta Service Evaluation in Controlling COVID-19 Transmission Using Twitter Sentiment Analysis

This study attempted to understand passenger perception of using public transport by utilizing Twitter data about the services of the TransJakarta Busway. Tweets were the main data source to capture users’ responses toward these services. Users’ perceptions were analyzed by sentiment analysis using a naïve Bayes algorithm. Furthermore, content analysis was used to inform improvements in service maintenance. The findings showed that the pandemic had a major impact on TransJakarta services, from a decrease in users, route closures, and fleet reductions to changes in user behavior. Most Tweets were negative regarding (1) poor bus frequency, leading to long queues and passenger overcrowding at bus stops and inside buses; (2) failure to maintain social distancing measures; (3) frequent violations of the 50% bus capacity reduction during peak hours, and showing a lack of consideration in measuring demand size during peak hours; (4) staff’s weak control of implementing the health protocol exacerbated poor services. This study suggests service improvement based on peak hour demand analysis to offset the implications of a 50% capacity restriction by providing proper bus frequencies and headway arrangements considerable enough to avoid crowding, followed by optimal monitoring of health protocol by staff. Tweet data may inform poor management in controlling the transmission of COVID-19 on public transportation. Hence, using Twitter data could replace conventional data collection methods like user interviews. Beneficial information from Tweet data can be captured at relatively low costs. Therefore, it may aid the evaluation of PPKM policy implementation to create more resilient public transportation during pandemics.

Post COVID-19: New breakthroughs and the future of behavioural research data collection

Behavioural researchers have been faced with challenges associated with the choice of data collection methods that is timely and cost-effective for all situations. Several studies have examined various means of collecting data while some electronic means of data collection have been explored. However, there is a need for a study that compares the conventional and contemporary data collection methods in terms of profile, perceptions and prospects. Therefore, this study examined the new breakthroughs and the future of behavioural research data collection in post COVID-19 era. The study is underpinned by connectivism learning theory within ex-post facto design with a sample of one hundred and twenty-six (126) behavioural science researchers. Post COVID-19 Data Collection Methods Scale-Forms App (r=0.86) was used, and the data collected were analysed using frequency count and t-test. The findings showed that there were more users of breakthrough methods 47 (37.3%) than conventional 39 (30.9%) and mixed method 40 (31.7%). Conventional methods were less available than new breakthrough methods. There is a significant difference in the perception, challenges and prospects of the conventional and breakthroughs in behavioural research data collection methods, all in favour of new breakthroughs. It is, therefore, recommended that behavioural researchers, as well as other researchers, avail themselves of the opportunities offered by the new breakthrough to advance their research endeavours in post COVID-19 era.

Citizen Science for Water Quality Monitoring in the Meki River, Ethiopia: Quality Assurance and Comparison with Conventional Methods

A lack of water quality information for many water bodies around the world makes it difficult to identify global change and discover early signs of myriad threats to freshwater resources. This problem is widely seen in Ethiopia due to absence of regular monitoring. Citizen science has a great potential to fill these gaps in water quality data, but there is concern about the accuracy of data collected by citizen scientists. Moreover, there is a gap to engage citizen scientists in water quality monitoring, and there is still insufficient awareness of how citizen scientists can become part of a collaborative scheme. This study aimed to evaluate the accuracy of water quality collected by citizen scientists and characterize the water quality of the Meki River with the involvement of citizen scientists. The suitability of the river water for irrigation was evaluated using a combination of citizen science and conventional water quality data collection methods. Water temperature, turbidity, ammonia, phosphate, nitrate, nitrite, total alkalinity, total hardness, and pH were analyzed by both citizen scientists and in a conventional laboratory. The citizen scientists’ data, expressed as percent of synthetic standard solution concentrations, indicated good agreement for selected water quality parameters: 123.8 ± 24.7% for PO43−, 115.6 ± 6.3% for NO3−, 105.8 ± 7.4% for pH, and 133.3 ± 23.6% for NH4+. Thus, citizen scientists can monitor and collect water quality data accurately. From the results, the Meki River water can be used for irrigation, but pollution sources should be controlled to reduce further quality deterioration as the population increases.

Global innovation and competition in quantum technology, viewed through the lens of patents and artificial intelligence

In this work we elucidate international trends in the field of quantum technology (QT) by analysing a global patent database built from an operational definition of QT that was generated through the curated application of artificial intelligence (AI). In doing so, we demonstrate how the sophisticated use of intellectual property information, enhanced by the artful deployment of AI techniques, may produce more reliable and useful revelations for policymakers and managers about global innovation in emerging fields of technology than is possible through conventional methods of data collection and analysis. We also demonstrate the utility of this approach for reliably characterising the evolving constituent sub-fields of QT. By adopting a hybrid human-AI approach to both the definition and the analysis of QT, we have produced some novel insights about global innovation and national organisational profiles in the QT field, particularly concerning dynamic competition between the USA and China.

Evaluation of Low-Cost UV-Vis Spectroscopy for Measuring Nitrate Using Synthetic Water Samples

Highlights A novel low-cost, modular spectroscopy system is compared to a standard system using synthetic samples, principal component analysis, and partial least squares regression. The information contained in the data produced by the two systems is similar according to principal component analysis. The low-cost system was able to accurately predict nitrate concentrations in concentrated and diluted samples using partial least squares regression. The methodology could be applied to water quality analysis in agriculture and water resources management. Abstract. Water quality data collection is an essential component of water systems management. For instance, the effective management of nutrients in hydroponic systems is necessary for maximizing yields efficiently and sustainably. Additionally, nutrients in natural and engineered waterbodies must be monitored to ensure they are meeting the required chemical characteristics for their ecological and social functions. However, conventional water quality data collection methods place limitations on water systems management due to their high resource requirements. Nitrate (NO3) is a major nutrient in ecological and agricultural systems, which can be reliably measured with ultraviolet-visible (UV-Vis) spectroscopy, a highly established technique for water quality analysis. The goal of this research was to evaluate a novel, low-cost, modular UV-Vis spectroscopy setup (GatorSpec) for the measurement of NO3 concentration in chemically complex solutions. UV-Vis absorbance of synthetic samples was measured using the GatorSpec and a commonly used bench-top laboratory spectroscopy system, the NanoDrop2000C. These data were analyzed using principal component analysis (PCA) to compare the spectral data produced by each system and partial least squares (PLS) regression to compare their ability to predict NO3 concentration. Results showed that data from both measurement systems were similar, indicating that the low-cost GatorSpec provided similar measurement accuracy to that of the laboratory reference system, the NanoDrop2000C. The PLS results revealed that for the diluted samples, the models derived from both systems were very good at predicting NO3 concentration. With these outcomes, it can be concluded that the GatorSpec is effective at measuring NO3 concentration in complex solutions and is comparable in performance to that of the NanoDrop2000C. In the future, this low-cost setup could be used to manage NO3 concentrations more efficiently in various applications such as hydroponic plant production, environmental monitoring, and stormwater treatment, which, in turn, could reduce the economic and environmental costs of these systems. Keywords: Low-cost, Synthetic samples, Ultraviolet-visible absorption spectroscopy, Water quality.

Willingness to participate in geolocation-based research.

Among the new research possibilities offered by smartphones, collecting geolocation data (e.g., GPS coordinates) holds a prominent position, allowing the investigation of individuals' mobility with greater precision and less effort than conventional data collection methods. However, geolocation data are still affected by errors (e.g., inaccurate recorded locations) and limitations (e.g., inability to record the purpose of a trip). Sending a survey right in the moment an event of interest is detected using geolocation data can add the missing information, while reducing memory errors that typically affect conventional surveys (sent some days/weeks after). However, the possibilities offered by both geolocation data and in-the-moment surveys triggered by geolocation data are limited by individuals' willingness to participate. This paper assesses such willingness using a conjoint experiment carried out on a sample of 1,016 members of an opt-in online panel in Spain. The effects on such willingness to accept the conditions offered to participants and their personal characteristics are also studied. The results show that asking panelists to participate in in-the-moment surveys does not negatively affect willingness compared to only sharing geolocation data. However, the conditions offered to panelists for their participation (mainly project duration and incentives) have a strong influence on their willingness. Furthermore, panelists less concerned with privacy and safety, and more experienced in sharing social media content, installing apps and using Google Maps, are more willing to participate. Finally, answers to open questions reveal that the main reason for participating is getting the incentive, while not participating is primarily related to privacy issues.

A New U-Net Based Convolutional Neural Network for Estimating Caribou Lichen Ground Cover from Field-Level RGB Images

High-quality ground-truth data are critical for developing reliable Earth Observation (EO) based geospatial products. Conventional methods of collecting these data are either subject to an unknown amount of human error and bias or require extended time in the field to complete (i.e., point-intercept assessments). Digital photograph classification (DPC) may address these drawbacks. In this study, we first assess the performance of a DPC method developed through licensed software to estimate ground cover percentage (%) of bright lichens, a critical caribou forage in fall and winter when other food resources are scarce. We then evaluate the feasibility of replicating this workflow in an open-source environment with a modified U-net model to improve processing time and scalability. Our results indicate that DPC is appropriate for generating ground-truth data in support of large-scale EO-based lichen mapping within the boreal forests of eastern Canada. Our final open-sourced classification model, Lichen Convolutional Neural Network (LiCNN), is comparably accurate yet more efficient than the licensed workflow. Therefore, the LiCNN approach successfully addresses the mentioned shortcomings of conventional ground-truth data collection methods efficiently and without the need for specialized software.

OD-Based Partition Technique to Improve Arterial Signal Coordination Using Connected Vehicle Data

Maximizing two-way arterial bands often plays a critical role in signal coordination along urban arterials. With an increasing number of signals involved, the arterial bands tend to shrink to some extent resulting in inefficient arterial signal coordination, and not all vehicles fully take advantage of the arterial bands to travel through the entire corridor. In response to that, system partition is a technique for handling arterials with many signals. Rather than designing end-to-end signal coordination, efficient arterial signal coordination is highly reliant on traffic origin and destination (OD) patterns on the arterial, which have been difficult to obtain using conventional data collection methods. The emerging big data sources, such as connected vehicles, provide great potential to gather such invaluable OD information for improving arterial signal coordination. This research proposes an easy-to-implement OD-based partition technique to improve arterial signal coordination by utilizing vehicle trajectory data automatically collected from connected vehicles. The proposed signal timing technique was tested using an arterial with 17 signalized intersections in Orange County, California. The results demonstrated that the OD-based partition technique improved the arterial average travel speed by 2.7% and 12.1% for the eastbound and westbound directions, respectively. At the same time, the proposed technique shortened the arterial average travel time by 2.6% and 11.1% for the eastbound and westbound directions, respectively. The total travel time was shortened both for the main-street and side-street major traffic flows.

Monitoring the well-being of vulnerable transit riders using machine learning based sentiment analysis and social media: Lessons from COVID-19.

Using open-source data, we show that despite significant reductions in global public transit during the COVID-19 pandemic, ∼20% of ridership continues during social distancing measures. Current urban transport data collection methods do not account for the distinct behavioural and psychological experiences of the population. Therefore, little is known about the travel experience of vulnerable citizens that continue to rely on public transit and their concerns over risk, safety and other stressors that could negatively affect their health and well-being. We develop a machine learning approach to augment conventional transport data collection methods by curating a population segmented Twitter dataset representing the travel experiences of ∼120,000 transit riders before and during the pandemic in Metro Vancouver, Canada. Results show a heightened increase in negative sentiments, differentiated by age, gender and ethnicity associated with public transit indicating signs of psychological stress among travellers during the first and second waves of COVID-19. Our results provide empirical evidence of existing inequalities and additional risks faced by citizens using public transit during the pandemic, and can help raise awareness of the differential risks faced by travellers. Our data collection methods can help inform more targeted social-distancing measures, public health announcements, and transit monitoring services during times of transport disruptions and closures.

Hydrological Perspectives on Integrated, Coordinated, Open, Networked (ICON) Science

AbstractHydrologic sciences depend on data monitoring, analyses, and simulations of hydrologic processes to ensure safe, sufficient, and equal water distribution. These hydrologic data come from but are not limited to primary (lab, plot, and field experiments) and secondary sources (remote sensing, UAVs, hydrologic models) that typically follow FAIR Principles (Findable, Accessible, Interoperable, and Reusable: (go-fair.org)). Easy availability of FAIR data has become possible because the hydrology‐oriented organizations have pushed the community to increase coordination of the protocols for generating data and sharing model platforms. In addition, networking at all levels has emerged with an invigorated effort to activate community science efforts that complement conventional data collection methods. However, it has become difficult to decipher various complex hydrologic processes with increasing data. Machine learning, a branch of artificial intelligence, provide more accurate and faster alternatives to better understand different hydrological processes. The Integrated, Coordinated, Open, Networked (ICON) framework provides a pathway for water users to include and respect diversity, equity, and inclusivity. In addition, ICONs support the integration of peoples with historically marginalized identities into this professional discipline of water sciences. This article comprises three independent commentaries about the state of ICON principles in hydrology and discusses the opportunities and challenges of adopting them.

Benchmarking machine learning algorithms by inferring transportation modes from unlabeled GPS data

Many traffic-related applications, e.g. traffic demand modeling, rely on conventional data collection methods such as travel surveys. These methods can be demanding in terms of cost and time, which results in low network coverage and limited representativeness. On-motion sensors, e.g. smartphones, offer the opportunity to replace such methods and compensate for the aforementioned drawbacks by collecting positioning data automatically. GPS data consist of positioning records each of which has geographical coordinates and a timestamp associated. These data, however, require cleansing and processing before being put into use. Information about the used transportation mode is missing from this kind of data unless travelers were specifically asked to report it. In the literature, supervised machine learning (ML) algorithms were successful in inferring transportation modes from GPS data. However, these algorithms, unlike unsupervised ML algorithms, require training data that are not always available. This paper aims to investigate the capability of unsupervised ML algorithms to infer transportation modes from real GPS data extracted from smartphones. Therefore, we used two datasets to benchmark different unsupervised ML algorithms with different input attributes. The paper also investigates the feasibility of using a pre-trained model for unlabeled real data. Finally, we compared the best performing unsupervised setup to the supervised ML algorithms recommended in the literature. The results suggest that the recommended unsupervised setups can reach an overall inferring accuracy of 93%.

Mobile Crowd-sensing Applications: Data Redundancies, Challenges, and Solutions

Conventional data collection methods that use Wireless Sensor Networks (WSNs) suffer from disadvantages such as deployment location limitation, geographical distance, as well as high construction and deployment costs of WSNs. Recently, various efforts have been promoting mobile crowd-sensing (such as a community with people using mobile devices) as a way to collect data based on existing resources. A Mobile Crowd-Sensing System can be considered as a Cyber-Physical System (CPS), because it allows people with mobile devices to collect and supply data to CPSs’ centers. In practical mobile crowd-sensing applications, due to limited budgets for the different expenditure categories in the system, it is necessary to minimize the collection of redundant information to save more resources for the investor. We study the problem of selecting participants in Mobile Crowd-Sensing Systems without redundant information such that the number of users is minimized and the number of records (events) reported by users is maximized, also known as the Participant-Report-Incident Redundant Avoidance (PRIRA) problem. We propose a new approximation algorithm, called the Maximum-Participant-Report Algorithm (MPRA) to solve the PRIRA problem. Through rigorous theoretical analysis and experimentation, we demonstrate that our proposed method performs well within reasonable bounds of computational complexity.

Application and Evolution of Airborne LiDAR Technology for Topographic Data Acquisition Practice in the Department of Survey and Mapping Malaysia

The mapping industry is one of the areas that is always given attention to balance the rapid development of current technology. The application of Light Detection and Ranging (LiDAR) technology in the mapping industry opens up a wide dimension of discussion involving industry users as well as academics. LiDAR technology is now a common method for faster and higher quality topographic data collection than conventional topographic data collection methods. Observation data that is generally in the form of high-density point (point cloud) can also be applied in various uses, especially in the field of mapping and terrain analysis. Therefore, this paper will discuss related LiDAR technology including basic information or principles of LiDAR technology, the latest developments of LiDAR methods, and work processes involved from the point of view of the Department of Survey and Mapping Malaysia (JUPEM).

Reinforcement learning based metric filtering for evolutionary distance metric learning

Data collection plays an important role in business agility; data can prove valuable and provide insights for important features. However, conventional data collection methods can be costly and time-consuming. This paper proposes a hybrid system R-EDML that combines a sequential feature selection performed by Reinforcement Learning (RL) with the evolutionary feature prioritization of Evolutionary Distance Metric Learning (EDML) in a clustering process. The goal is to reduce the features while maintaining or increasing the accuracy leading to less time complexity and future data collection time and cost reduction. In this method, features represented by the diagonal elements of EDML matrices are prioritized using a differential evolution algorithm. Further, a selection control strategy using RL is learned by sequentially inserting and evaluating the prioritized elements. The outcome offers the best accuracy R-EDML matrix with the least number of elements. Diagonal R-EDML focusing on the diagonal elements is compared with EDML and conventional feature selection. Full Matrix R-EDML focusing on the diagonal and non-diagonal elements is tested and compared with Information-Theoretic Metric Learning. Moreover, R-EDML policy is tested for each EDML generation and across all generations. Results show a significant decrease in the number of features while maintaining or increasing accuracy.

Implementation of online questionnaires in the General Household Survey in Hong Kong

The General Household Survey (GHS) of Hong Kong is a monthly household survey on labour force characteristics and household income. Traditionally, data collection was conducted via face-to-face interviews through field visits and computer-assisted telephone interviews (CATI). In July 2017, the Census and Statistics Department (C&SD) introduced an online questionnaire (OQ) in the GHS to enhance its services to survey respondents. Considering the importance of GHS statistics, the tight monthly survey cycle and the complexities involved in adding a new data collection mode, a phase-by-phase approach was adopted with close monitoring of data quality of the survey estimates. With satisfactory trial results, the OQ was fully implemented in the GHS in January 2019. Being a convenient and privacy-assured channel, the OQ helps entice sampled households to respond to the GHS and increases the contact rate of those households who are more difficult to reach by the conventional data collection methods. During COVID-19 when face-to-face interviews had to be adjourned, the OQ provided a powerful channel for data collection. With targeted messages appealing for response by OQ sent to sampled households in a more proactive manner, the take-up rate of the OQ increased by as much as 10 percentage points, partly compensating the loss due to the suspension of field visits such that monthly labour force statistics could continue to be published during the epidemic. The paper was prepared under the kind mentorship of Ms Lisa Bersales, former Chief Statistician of the Philippines.