Intra-device Variability Research Articles

Technical characterisation of digital stethoscopes: towards scalable artificial intelligence-based auscultation

Digital stethoscopes can enable the development of integrated artificial intelligence (AI) systems that can remove the subjectivity of manual auscultation, improve diagnostic accuracy, and compensate for diminishing auscultatory skills. Developing scalable AI systems can be challenging, especially when acquisition devices differ and thus introduce sensor bias. To address this issue, a precise knowledge of these differences, i.e., frequency responses of these devices, is needed, but the manufacturers often do not provide complete device specifications. In this study, we reported an effective methodology for determining the frequency response of a digital stethoscope and used it to characterise three common digital stethoscopes: Littmann 3200, Eko Core, and Thinklabs One. Our results show significant inter-device variability in that the frequency responses of the three studied stethoscopes were distinctly different. A moderate intra-device variability was seen when comparing two separate units of Littmann 3200. The study highlights the need for normalisation across devices for developing successful AI-assisted auscultation and provides a technical characterisation approach as a first step to accomplish it.

Comparison between the CASIA SS-1000 and Pentacam in measuring corneal curvatures and corneal thickness maps

PurposeTo compare the intra-device repeatability and inter-device reproducibility between two anterior segment imaging instruments, the CASIA SS-1000 (Tomey Corp., Nagoya, Japan) and Pentacam (OCULUS, Arlington, WA) in measuring anterior segment parameters.MethodsSingle-center, prospective clinical trial. Participants ≥20 years of age were included. One eye was randomly selected, each imaged by three CASIA SS-1000 devices and three Pentacam devices by three different examiners. Each photographer operated a pair of devices, one CASIA SS-1000 and one Pentacam. The image order for each participant was determined by a random permutation table. Three images were taken from each device. A total of 18 images were taken for each eye. Ratios of the standard deviations, referenced as (CASIA/Pentacam), were calculated to compare the repeatability and reproducibility of the two imaging instruments.ResultsIn all, 66 participants with a mean age of 46.4 years (±21.7) were enrolled in the study. All repeatability ratios and intra-device variability were less than 1 (anterior corneal curvature: flat = 0.86, steep = 0.85; posterior corneal curvature: flat = 0.43, steep = 0.61; and map: thinnest = 0.22; central = 0.24, 2 mm = 0.26, 4 mm = 0.27, and 6 mm = 0.30). All reproducibility ratios, which measure the inter-device variability, were less than 1 (anterior corneal curvature: flat = 0.58, steep = 0.73; posterior corneal curvature: flat = 0.25, steep = 0.31; and pachymetry map: thinnest = 0.20; central = 0.20; 2 mm = 0.20; 4 mm = 0.19; and 6 mm = 0.22). A ratio of less than 1 indicates that the CASIA SS-1000 has more consistent measurements.ConclusionsThe CASIA SS-1000 was found to have better repeatability and reproducibility compared to the Pentacam for both corneal curvature and pachymetry maps. This greater consistency may require further study to determine whether the decreased variability can be translated into improved clinical results.

Multi-instrument assessment of fine and ultrafine titanium dioxide aerosols

ABSTRACT The measurement of fine (diameter: 100 nanometers–2.5 micrometers) and ultrafine (UF: < 100 nanometers) titanium dioxide (TiO2) particles is instrument dependent. Differences in measurements exist between toxicological and field investigations for the same exposure metric such as mass, number, or surface area because of variations in instruments used, operating parameters, or particle-size measurement ranges. Without appropriate comparison, instrument measurements create a disconnect between toxicological and field investigations for a given exposure metric. Our objective was to compare a variety of instruments including multiple metrics including mass, number, and surface area (SA) concentrations for assessing different concentrations of separately aerosolized fine and UF TiO2 particles. The instruments studied were (1) DustTrak™ DRX, (2) personal DataRAMs™ (PDR), (3) GRIMMTM, and (4) diffusion charger (DC). Two devices of each field-study instrument (DRX, PDR, GRIMM, and DC) were used to measure various metrics while adjusting for gravimetric mass concentrations of fine and UF TiO2 particles in controlled chamber tests. An analysis of variance (ANOVA) was used to apportion the variance to inter-instrument (between different instrument-types), inter-device (within instrument), and intra-device components. Performance of each instrument-device was calculated using root mean squared error compared to reference methods: close-faced cassette and gravimetric analysis for mass and scanning mobility particle sizer (SMPS) real-time monitoring for number and SA concentrations. Generally, inter-instrument variability accounted for the greatest (62.6% or more) source of variance for mass, and SA-based concentrations of fine and UF TiO2 particles. However, higher intra-device variability (53.7%) was observed for number concentrations measurements with fine particles compared to inter-instrument variability (40.8%). Inter-device variance range(0.5–5.5%) was similar for all exposure metrics. DRX performed better in measuring mass closer to gravimetric than PDRs for fine and UF TiO2. Number concentrations measured by GRIMMs and SA measurements by DCs were considerably (40.8–86.9%) different from the reference (SMPS) method for comparable size ranges of fine and UF TiO2. This information may serve to aid in interpreting assessments in risk models, epidemiologic studies, and development of occupational exposure limits, relating to health effect endpoints identified in toxicological studies considering similar instruments evaluated in this study.

Precision of bit slicing with in-memory computing based on analog phase-change memory crossbars

Abstract In-memory computing is a promising non-von Neumann approach to perform certain computational tasks efficiently within memory devices by exploiting their physical attributes. However, the computational accuracy achieved with this approach has been rather low, owing to significant inter-device variability and inhomogeneity across an array as well as intra-device variability and randomness from the analog memory devices. Bit slicing, a technique for constructing a high precision processor from several modules of lower precision, is a promising approach for overcoming this accuracy limitation. However, a systematic study to assess the precision ultimately achieved by bit slicing with analog in-memory computing has so far been lacking. In this work, we assess the computational error from bit slicing when performing in-memory matrix-vector multiplications. Using accurate models of phase-change memory crossbar arrays, we demonstrate that unlike in digital processors where bit slicing is used to extend the dynamic range of the number representation, bit slicing with in-memory computing should aim at minimizing the error from the analog matrix representation through averaging within a given dynamic range. The results are validated using a prototype phase-change memory chip and the impact on the neural network inference accuracy on CIFAR-10 and ImageNet benchmarks is evaluated.

Performance Assessment of a Commercial Continuous-Wave Near-Infrared Spectroscopy Tissue Oximeter for Suitability for Use in an International, Multi-Center Clinical Trial.

Despite the wide range of clinical and research applications, the reliability of the absolute oxygenation measurements of continuous wave near-infrared spectroscopy sensors is often questioned, partially due to issues of standardization. In this study, we have compared the performances of 13 units of a continuous wave near-infrared spectroscopy device (PortaMon, Artinis Medical Systems, NL) to test their suitability for being used in the HEMOCOVID-19 clinical trial in 10 medical centers around the world. Detailed phantom and in vivo tests were employed to measure the precision and reproducibility of measurements of local blood oxygen saturation and total hemoglobin concentration under different conditions: for different devices used, different operators, for probe repositioning over the same location, and over time (hours/days/months). We have detected systematic differences between devices when measuring phantoms (inter-device variability, <4%), which were larger than the intra-device variability (<1%). This intrinsic variability is in addition to the variability during in vivo measurements on the forearm muscle resulting from errors in probe positioning and intrinsic physiological noise (<9%), which was also larger than the inter-device differences (<3%) during the same test. Lastly, we have tested the reproducibility of the protocol of the HEMOCOVID-19 clinical trial; that is, forearm muscle oxygenation monitoring during vascular occlusion tests over days. Overall, our conclusion is that these devices can be used in multi-center trials but care must be taken to characterize, follow-up, and statistically account for inter-device variability.

A systematic scoping review to identify the design and assess the performance of devices for antenatal continuous fetal monitoring.

Antepartum fetal monitoring aims to assess fetal development and wellbeing throughout pregnancy. Current methods utilised in clinical practice are intermittent and only provide a 'snapshot' of fetal wellbeing, thus key signs of fetal demise could be missed. Continuous fetal monitoring (CFM) offers the potential to alleviate these issues by providing an objective and longitudinal overview of fetal status. Various CFM devices exist within literature; this review planned to provide a systematic overview of these devices, and specifically aimed to map the devices' design, performance and factors which affect this, whilst determining any gaps in development. A systematic search was conducted using MEDLINE, EMBASE, CINAHL, EMCARE, BNI, Cochrane Library, Web of Science and Pubmed databases. Following the deletion of duplicates, the articles' titles and abstracts were screened and suitable papers underwent a full-text assessment prior to inclusion in the review by two independent assessors. The literature searches generated 4,885 hits from which 43 studies were included in the review. Twenty-four different devices were identified utilising four suitable CFM technologies: fetal electrocardiography, fetal phonocardiography, accelerometry and fetal vectorcardiography. The devices adopted various designs and signal processing methods. There was no common means of device performance assessment between different devices, which limited comparison. The device performance of fetal electrocardiography was reduced between 28 to 36 weeks' gestation and during high levels of maternal movement, and increased during night-time rest. Other factors, including maternal body mass index, fetal position, recording location, uterine activity, amniotic fluid index, number of fetuses and smoking status, as well as factors which affected alternative technologies had equivocal effects and require further investigation. A variety of CFM devices have been developed, however no specific approach or design appears to be advantageous due to high levels of inter-device and intra-device variability.

Technical validity and usability of a novel smartphone-connected spirometry device for pediatric patients with asthma and cystic fibrosis.

BackgroundDiagnosis and follow‐up of respiratory diseases traditionally rely on pulmonary function tests (PFTs), which are currently performed in hospitals and require trained personnel. Smartphone‐connected spirometers, like the Air Next spirometer, have been developed to aid in the home monitoring of patients with pulmonary disease. The aim of this study was to investigate the technical validity and usability of the Air Next spirometer in pediatric patients.MethodsDevice variability was tested with a calibrated syringe. About 90 subjects, aged 6 to 16, were included in a prospective cohort study. Fifty‐eight subjects performed conventional spirometry and subsequent Air Next spirometry. The bias and the limits of agreement between the measurements were calculated. Furthermore, subjects used the device for 28 days at home and completed a subject‐satisfaction questionnaire at the end of the study period.ResultsInterdevice variability was 2.8% and intradevice variability was 0.9%. The average difference between the Air Next and conventional spirometry was 40 mL for forced expiratory volume in 1 second (FEV1) and 3 mL for forced vital capacity (FVC). The limits of agreement were −270 mL and +352 mL for FEV1 and −403 mL and +397 mL for FVC. About 45% of FEV1 measurements and 41% of FVC measurements at home were acceptable and reproducible according to American Thoracic Society/European Respiratory Society criteria. Parents scored difficulty, usefulness, and reliability of the device 1.9, 3.5, and 3.8 out of 5, respectively.ConclusionThe Air Next device shows validity for the measurement of FEV1 and FVC in a pediatric patient population.

Comparison of Choriocapillaris Flow Measurements between Two Optical Coherence Tomography Angiography Devices

Purpose: To evaluate choriocapillaris (CC) perfusion in healthy subjects using 2 different optical coherence tomography angiography (OCT-A) devices. Procedures: Macular OCT-A imaging (36 eyes of 36 subjects) was performed using Optovue AngioVue and Zeiss AngioPlex devices. CC decorrelation signal index was assessed, and CC data were analyzed regarding intra-device variability, inter-device correlation, age, signal strength, and fields of view. Results: The intra-device variability of CC measurements in the 3 × 3 mm² field was 5.3 and 2.6% (Angiovue and Angioplex, coefficients of variation; 6 × 6 mm²: 8.0 and 2.8%, respectively). Mean CC decorrelation signal index in 3 × 3 mm² was 104.3 ± 6.7 (Angiovue) and 81.3 ± 9.2 (Angioplex) (6 × 6 mm²: 95.6 ± 8.1, 81.1 ± 6.5) with high correlation between both devices (3 × 3 mm²: p = 0.0053; 6 × 6 mm²: p = 0.0139). CC decorrelation signal index in 3 × 3 mm² was significantly higher in subjects aged ≤58 years compared to subjects aged ≥59 years (Angiovue: 107.3 ± 3.6, 101.3 ± 7.7, p = 0.0156; Angioplex: 84.6 ± 7.6, 78.0 ± 9.5, p = 0.0371). Signal strength was 64.6 ± 8.9 (Angiovue) and 9.5 ± 0.8 (Angioplex). Conclusion: Both devices showed low intra-device variability and a high inter-device correlation. CC decorrelation signal index was negatively correlated with advancing age.

Calibration of RO-based temperature sensors for a toolset for measuring thermal behavior of FPGA devices

In this work, we present calibration effort for Ring Oscillator-based temperature sensors in Field Programmable Gate Array (FPGA) devices and a toolset suitable for the analysis of thermal behavior of FPGAs. The toolset allows for the automatic synthesis of a unified temperature measurement and heat generation infrastructure combined with the necessary control structures and communication interfaces. The tools insert temperature sensors and heaters based on Ring Oscillators (RO) through the low level manipulations on Xilinx Design Language (XDL) descriptions of FPGA resource allocations. The purpose of the kit is to support rapid setup of thermal behavior experiments by providing basic heating and sensing elements with verified properties as well as an area-optimised IP core for control of the experiments. Furthermore to evaluate inter- and intra-device variability we compare results of temperature sensor calibration using our toolset and six devices from two FPGA families: Spartan-3E and Spartan-6.

Isolation and counting of multiple cell types using an affinity separation device

A simple device for the separation of cells by phenotype is described. Cells are separated/isolated using capture antibodies on a glass chip. Unlike other “sandwich” type assays, the readout is performed without labels using transmission microscopy, simplifying cell enumeration. T and B lymphocytes from cell culture or whole blood were separated using antibodies for the CD4, CD19, and CD71 antigens. The separation slides were found to reproducibly bind cells by antigen expression, allowing for accurate enumeration of mixed cell samples. Inter- and intra-device variability was evaluated, and the issue of nonspecific binding is addressed. We envision that this type of cell separation technique could be used in remote settings, as sample preparation is minimal and the analysis time is rapid (20 min from sample acquisition to final readout).

Instrumentation as a source of variability in the application of fluorescence spectroscopic devices for detecting cervical neoplasia

We report on a study designed to assess variability among three different fluorescence spectroscopy devices, four fiber optic probes, and three sets of optical calibration standards to better understand the reproducibility of measurements and interdevice comparisons of fluorescence spectroscopic data intended for clinical diagnostic use. Multiple measurements are acquired from all sets of standards using each combination of spectrometer, fiber optic probe, and optical standard. Data are processed using standard calibration methods to remove instrument-dependant responses. Processed spectra are analyzed using an analysis of variance to assess the percent variance explained by each factor that was statistically significant. Analysis of processed data confirms statistically significant differences among the spectrometers and fiber optic probes. However, no differences are found when varying calibration standards or measurement date and time. The spectrometers and fiber optic probes are significant sources of variability, but appropriate data processing substantially reduces these effects. Studies of inter- and intradevice variability are important methodological issues for optical device trials and must be included in the quality assurance studies for the clinical trial design.

A comparison of two single-headed and two multi-headed allergen skin test devices

Given the potential differences in performance of skin test devices, the purpose of this study was to prospectively assess the performance of two single-headed and two multiheaded devices for allergy skin testing in terms of wheal size, sensitivity, specificity, intradevice variability, and pain level. Two single-headed devices (Greer Pick, Duotip-Test) and two multiheaded devices (Multi-Test II, OMNI) were tested in 15 subjects in a prospective partially blind fashion looking at wheal reactions and pain using histamine and glycerol-saline on the arms and back. Differences among devices in wheal size and pain were noted. Sensitivity, specificity, and intradevice variability were calculated. Differences between corner and interior heads in multiheaded devices were analyzed. No significant differences were observed in wheal size between Greer Pick (7.1+/-1.4 mm) and Duotip-Test (7.2+/-1.6 mm). Multiheaded devices were significantly different in wheal size compared with each other and to the single-headed devices (Multi-Test II, 5.4+/-1.7 mm; OMNI, 3.3+/-1.2 mm). Single-headed devices were more sensitive (100% each, 95% CI of 92-100%) than the multiheaded devices. Multi-Test II was significantly more sensitive (83%, 95% CI of 78-87%) than OMNI (57%, 95% CI of 51-62%). There was significant intradevice variability for the multiheaded devices with corner heads being significantly more sensitive than interior heads. Specificities for all devices were equally good (-97%). Pain was greater for multiheaded devices than single-headed devices but was generally mild. In conclusion, this study supports the idea that single-headed devices may be more sensitive and consistent than multiheaded devices. Multi-Test II is more sensitive than OMNI. In multiheaded devices, corner heads are more sensitive than interior heads.

Evaluation of a vaporizing device (Volcano®) for the pulmonary administration of tetrahydrocannabinol

What is currently needed for optimal use of medicinal cannabinoids is a feasible, nonsmoked, rapid-onset delivery system. Cannabis "vaporization" is a technique aimed at suppressing irritating respiratory toxins by heating cannabis to a temperature where active cannabinoid vapors form, but below the point of combustion where smoke and associated toxins are produced. The goal of this study was to evaluate the performance of the Volcano vaporizer in terms of reproducible delivery of the bioactive cannabinoid tetrahydrocannabinol (THC) by using pure cannabinoid preparations, so that it could be used in a clinical trial. By changing parameters such as temperature setting, type of evaporation sample and balloon volume, the vaporization of THC was systematically improved to its maximum, while preventing the formation of breakdown products of THC, such as cannabinol or delta-8-THC. Inter- and intra-device variability was tested as well as relationship between loaded- and delivered dose. It was found that an average of about 54% of loaded THC was delivered into the balloon of the vaporizer, in a reproducible manner. When the vaporizer was used for clinical administration of inhaled THC, it was found that on average 35% of inhaled THC was directly exhaled again. Our results show that with the Volcano a safe and effective cannabinoid delivery system seems to be available to patients. The final pulmonal uptake of THC is comparable to the smoking of cannabis, while avoiding the respiratory disadvantages of smoking.

Comparison of test devices for skin prick testing

Allergy skin testing guides developing avoidance plans and writing an immunotherapy prescription. The goal for the allergist is to apply allergen skin testing to the appropriate patient population by using a device that minimizes both false-negative and false-positive findings while minimizing patient discomfort. New skin testing devices continue to be developed with a trend toward production of multiheaded devices. Data on the performance of these devices in a head-to-head prospective fashion are limited. Our goal was to study 8 commonly used devices to compare their performance in a head-to-head fashion. In a prospective, double-blind fashion, the performance of 8 skin test devices was evaluated. Devices were tested with histamine and saline on both the arms and back of each subject. Devices were rotated over 4 testing sessions, at least a week apart, so each device was tested in each anatomic testing location. Performance elements examined included wheal, flare, pain, sensitivity, specificity, and intradevice variability. We found significant differences in all areas of device performance among all devices examined. Multiheaded devices also demonstrated significant intradevice variability and were more painful than single devices. Furthermore, multiheaded devices had larger reactions on the back, whereas single devices had larger reactions on the arms. Statistically significant differences exist among all devices tested. Providers should consider this data when choosing a device that suits their practice setting and ensure that technicians are sufficiently trained on the correct use of that device.