Incomplete Observations Research Articles

Sea surface height data reconstruction via inter and intra layer features based on dual attention

Understanding of geoscience relies on spatio-temporal continuous fields, such as Sea Surface Height (SSH). Data reconstruction for restoring spatio-temporal continuous and gridded maps from incomplete observations and inaccuracy interpolation products has long been a crucial challenge in the marine geoscience. Despite remarkable progress, most existing learning-based reconstruction methods neglect to fully and discriminatively utilize the inter-layer and intra-layer features. The inter-layer semantics are complementary, and the significance of intra-layer components varies with frequency. To address these issues, we propose a multi-layer Feature Combination Network based on Attention mechanism (FCANet) for SSH data reconstruction. Specifically, a novel trainable Multi-layer Feature Combination Block (MFCB) is developed to enrich the features through inter-layer dependencies. Dual attention mechanism is introduced into the MFCB to enhance saliency of intra-layer features by adaptively rescaling the spatial-wise and channel-wise features. Furthermore, experimental results on SSH variable demonstrate the superiority of our FCANet over state-of-the-art reconstruction methods in the metrics of RMSE and SSIM.

FINE: A Framework for Distributed Learning on Incomplete Observations for Heterogeneous Crowdsensing Networks

Numerous crowdsensing applications have been developed recently in mobile social networks and vehicle networks. How to implement an accurate distributed learning process to estimate parameters of an unknown model in crowdsensing is a significant issue because centralised learning methods produce unreliable data gathering, expensive central servers, and privacy concerns. Due to this, we propose FINE, a distributed learning framework for imperfect data and non-smooth estimation, along with its design, analysis, and assessment. Our design, which is focused on creating a workable framework for learning parameters in crowdsensing networks accurately and efficiently, generalises earlier learning techniques by supporting heterogeneous dimensions of data records observed by various nodes as well as minimization based on non-smooth error functions.In particular, FINE makes use of a distributed dual average technique that efficiently minimises non-smooth error functions and a novel distributed record completion algorithm that enables each node to get the global consensus through effective communication with neighbours. All of these algorithms converge, as shown by our analysis, and the convergence rates are also obtained to support their efficacy. Through experiments on synthetic and actual networks, we assess how well our framework performs

Tensor train factorization under noisy and incomplete data with automatic rank estimation

As a powerful tool in analyzing multi-dimensional data, tensor train (TT) decomposition shows superior performance compared to other tensor decomposition formats. Existing TT decomposition methods, however, either easily overfit with noise, or require substantial fine-tuning to strike a balance between recovery accuracy and model complexity. To avoid the above shortcomings, this paper treats the TT decomposition in a fully Bayesian perspective, which includes automatic TT rank determination and noise power estimation. Theoretical justification on adopting the Gaussian-product-Gamma priors for inducing sparsity on the slices of the TT cores is provided, thus allowing the model complexity to be automatically determined even when the observed tensor data is noisy and contains many missing values. Furthermore, using the variational inference framework, an effective learning algorithm on the probabilistic model parameters is derived. Simulations on synthetic data demonstrate that the proposed algorithm accurately recovers the underlying TT structure from incomplete noisy observations. Further experiments on image and video data also show its superior performance to other existing TT decomposition algorithms.

Mortality Modeling of Partially Observed Cohorts Using Administrative Death Records

New advances in data linkage provide mortality researchers with access to administrative datasets with millions of mortality records and rich demographic covariates. Although these new datasets allow for high-resolution mortality research, administrative mortality records often have technical limitations, such as limited mortality coverage windows and incomplete observation of survivors. We describe a method for fitting truncated distributions that can be used for estimating mortality differentials in administrative data. We apply this method to the CenSoc datasets, which link the United States 1940 Census records to Social Security administrative mortality records. Our approach may be useful in other contexts where administrative data on deaths are available. As a companion to the paper, we release the R package gompertztrunc, which implements the methods introduced in this paper.

Dynamic Vehicle Pose Estimation with Heuristic L-Shape Fitting and Grid-Based Particle Filter

Vehicle pose estimation with LIDAR plays a crucial role in autonomous driving systems. It serves as the fundamental basis for functions such as tracking, path planning, and decision-making. However, the majority of current vehicle pose estimation techniques struggle to produce satisfactory results when faced with incomplete observation measurements, such as L-shaped point cloud clusters without side contours or those including side-view mirrors. In addition, the requirement for real-time results further increases the difficulty of the pose estimation task. In this paper, we present a vehicle Pose Estimation method with Heuristic L-shape fitting and grid-based Particle Filter (PE-HL-PF). We design a geometric shape classifier module to divide clusters into symmetrical and asymmetrical ones according to their shape features. Furthermore, a contour-based heuristic L-shape fitting module is introduced for asymmetrical clusters, and a structure-aware grid-based particle filter is used to estimate the pose of symmetrical clusters. PE-HL-PF first utilizes a heuristic asymmetrical module that selects dominant contours fitting orientation in a heuristic manner, thereby avoiding the need for a complex traversal search. Additionally, a symmetrical module based on particle filtering is incorporated to enhance the stability of orientation estimation. This method achieves significant improvements in both the runtime efficiency and pose estimation accuracy of incomplete point clouds. Compared with state-of-the-art pose estimation methods, our PE-HL-PF demonstrates a notable performance improvement. Our method can estimate the pose of thousands of objects in less than 1 millisecond, a significant improvement over previous methods. The results of experiments performed on the KITTI dataset validate the effectiveness of our approach.

The importance of theory at the Information Systems Journal

The importance of theory at the <i>Information Systems Journal</i>

Sparse array radar forward-looking imaging based on fast atomic norm minimization

ABSTRACT The sparse array imaging radar could largely reduce the system complexity, but it also suffers from poor azimuthal resolution due to the limited aperture length and incomplete observation caused by non-uniform array configuration. In this letter, a radar forward-looking imaging method with improved imaging quality and reduced system complexity is proposed. Firstly, the matrix completion method is utilized to recover the missing signals of sparse array. Then, based on the plane-wave assumption, a fast 2-dimensional atomic norm minimization (ANM) is adopted to solve the positions and amplitudes of scatterers. As a continues compressed sensing method, the result of ANM-based method earns sharper main-lobes and much lower-level side-lobes compared to the fast Fourier transform -based method. Although the resolution of proposed method is still limited by Rayleigh resolution, the imaging quality can be greatly improved by emphasizing main-lobes and avoiding the perturbation of side-lobes. Simulations and experiments are carried out to demonstrate the performance of the proposed method.

Water use characteristics of the artificial forests black locust (Robinia pseudoacacia) and Chinese pine (Pinus tabulaeformis) on the Loess Plateau of China

AbstractThe broad leaf, deciduous black locust (Robinia pseudoacacia) and needle leaf, evergreen Chinese pine (Pinus tabulaeformis) are the major afforestation species on the Loess Plateau of China, accounting for 29.5% and 15.9% of the afforestation area, respectively. The decay of black locust and drying of the forest land soil in this region have caused serious concerns among scientific community and local management authorities. Understanding the hydrological characteristics of the afforestation plantation may provide clues for choosing proper species. However, previous studies demonstrated diverse or even contradictory understandings due to incomplete observations of the water balance. We used a model to reconstruct the water balance based on 5 years (2015–2019) of incomplete soil water and transpiration observations at the field for these two species. The R‐squares of linear regression analysis of simulated and observed soil water in black locust and Chinese pine plantations were 0.69 and 0.90, respectively; and transpiration was 0.62 and 0.69, respectively. Based on the reconstructed water balance components, we found that (1) black locust and Chinese pine had similar annual evapotranspiration in quantity. However, the Chinese pine transpired less and evaporated more water from the canopy than the black locust, with their ratios to the annual precipitation 50.7% and 69.7%, 26.3% and 17.2%, respectively; (2) the Chinese pine has higher evapotranspiration than the black locust in the non‐growing season, accounting for 19.5% and 9.2% of the annual evapotranspiration, respectively; and (3) the Chinese pine is more tolerable to soil water drought than black locust. The findings may suggest that the Chinese pine is a more favourable species for afforestation than the black locust from the point view of drought resistance in this region.

Algebraic Conditions to the Uniqueness Theorem for a Coupled System of Wave Equations

We propose a new rank condition on the coupling matrices, which, together with Kalman’s classic rank condition, can guarantee the uniqueness of a solution for a large class of coupled systems of wave equations with incomplete observations. We then establish the approximate controllability and the approximate synchronization of the underlying system in the general situation.

Factors contributing to the late diagnosis of appendicitis in children: the results of a retrospective study

BACKGROUND: The diagnosis of acute appendicitis in children is often established late, which leads to complications, lethal outcomes are recorded.&#x0D; AIM: To identify and analyze the factors contributing to the late diagnosis of acute appendicitis in children.&#x0D; MATERIALS AND METHODS: A retrospective analysis of 279 case histories of children was carried out. The duration of symptoms is divided into time 1 (before hospitalization) and time 2 (from hospitalization to surgery). Patients were divided into two groups: 1st with destructive uncomplicated appendicitis (phlegmonous, gangrenous), 2nd with complicated appendicitis (unrestricted peritonitis, infiltrate, abscess). The causes of late hospitalization, symptoms indicated in the case histories, examinations performed, postoperative complications were analyzed. The diagnosis was established on the basis of intraoperative data, histological examination.&#x0D; RESULTS: In the 1st group, time 1 and time 2 were significantly less than in the 2nd (p 0.001; 0.028). Late presentation (24 hours) in 21.9%, misdiagnosis in 12.2% of cases. In the hospital, the observation of the patient 12 hours in 21.1% of patients. The frequency of presence / absence of a description of clinical and laboratory signs of acute appendicitis is from 100 to 19.7%. The 2nd group was characterized by: duration of symptoms 24 hours, repeated vomiting (p 0.001), febrile condition (p 0.001), increase in neutrophils 90% (8792%). Ultrasound examination with evaluation of the appendix in 20.1% of cases with a diagnostic accuracy of 89.3%. Laparoscopic appendectomy in 53.2% Alvarado scores are significantly higher in 2nd group (810) than in 1st group (79, p 0.001). In 1st group, complications were significantly less than 3.4% than in 2nd group 22.7 % (p 0.001).&#x0D; CONCLUSIONS: Factors contributing to the late diagnosis of appendicitis in children are: late treatment, diagnostic errors at the prehospital stage, incomplete clinical examination and observation, insufficient use of laboratory and instrumental methods.

Wind energy forecasting with missing values within a fully conditional specification framework

Wind power forecasting is essential to power system operation and electricity markets. As abundant data became available thanks to the deployment of measurement infrastructures and the democratization of meteorological modeling, extensive data-driven approaches have been developed within both point and probabilistic forecasting frameworks. These models usually assume that the dataset at hand is complete and overlook missing value issues that often occur in practice. In contrast to that common approach, we here rigorously consider the wind power forecasting problem in the presence of missing values, by jointly accommodating imputation and forecasting tasks. Our approach can infer the joint distribution of input features and target variables at the model estimation stage based on incomplete observations only. We place emphasis on a fully conditional specification method, owing to its desirable properties, e.g., being assumption-free when it comes to these joint distributions. Then, at the operational forecasting stage, with available features at hand, one can issue forecasts by implicitly imputing all missing entries. The approach is applicable to both point and probabilistic forecasting, while yielding competitive forecast quality in both simulated and real-world case studies. The results confirm that by using a powerful universal imputation method based on a fully conditional specification, the proposed universal imputation approach is superior to the common impute-then-predict approach, especially in the context of probabilistic forecasting.

State of the art, challenges and prospects of wide-area event identification on transmission systems

The proliferation of advanced metering devices such as phasor measurement units (PMUs) along with communication systems readiness has opened new horizons for centralized protection and control of transmission systems. Wide-area event identification (WAEI) is considered an indispensable enabling block to these advanced applications. This paper is aimed at scrutinizing existing WAEI methods and discussing their prospects and shortcomings in improving the situational awareness of complex transmission systems. The disturbances of interest are those that significantly impact system operation and stability, namely short-circuit faults, line outages, and generation outages. The reluctance of system operators to entrust WAEI methods is discussed and linked to the inability of existing methods to deal with real-world challenges such as communication latencies, temporarily incomplete network observability, and the loss of the time synchronization signal. The superimposed-circuit concept is detailed and promoted as a powerful methodology with great unleashed potential for addressing these problems. The paper ends with remarks on the remaining research gaps that need to be addressed to fulfill the needs of power system operators, thus facilitating the uptake of WAEI methods in practice.

On the Separation of Estimation and Control in Risk-Sensitive Investment Problems under Incomplete Observation

On the Separation of Estimation and Control in Risk-Sensitive Investment Problems under Incomplete Observation

Uncertainty Quantification for Nonconvex Tensor Completion: Confidence Intervals, Heteroscedasticity and Optimality

We study the distribution and uncertainty of nonconvex optimization for noisy tensor completion—the problem of estimating a low-rank tensor given incomplete and corrupted observations of its entries. Focusing on a two-stage estimation algorithm proposed by Cai <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , we characterize the distribution of this nonconvex estimator down to fine scales. This distributional theory in turn allows one to construct valid and short confidence intervals for both the unseen tensor entries and the unknown tensor factors. The proposed inferential procedure enjoys several important features: (1) it is fully adaptive to noise heteroscedasticity, and (2) it is data-driven and automatically adapts to unknown noise distributions. Furthermore, our findings unveil the statistical optimality of nonconvex tensor completion: it attains un-improvable <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{2}$ </tex-math></inline-formula> accuracy—including both the rates and the pre-constants—when estimating both the unknown tensor and the underlying tensor factors.

Ecological Forecasting and Operational Information Systems Support Sustainable Ocean Management

In times of rapid change and rising human pressures on marine systems, information about the future state of the ocean can provide decision-makers with time to avoid adverse impacts and maximise opportunities. An ecological forecast predicts changes in ecosystems and its components due to environmental forcing such as climate variability and change, extreme weather conditions, pollution, or habitat change. Here, we summarise examples from several sectors and a range of locations. We describe the need, approach, forecast performance, delivery system, and end user uptake. This examination shows that near-term ecological forecasts are needed by end users, decisions are being made based on forecasts, and there is an urgent need to develop operational information systems to support sustainable ocean management. An operational information system is critical for connecting to decision makers and providing an enduring approach to forecasting and proactive decision making. These operational systems require significant investment and ongoing maintenance but are key to delivering ecological forecasts for societal benefits. Iterative forecasting practices could provide continuous improvement by incorporating evaluation and feedback to overcome the limitations of the imperfect model and incomplete observations to achieve better forecast outcomes and accuracy.

Obstacle avoidance planning of autonomous vehicles using deep reinforcement learning

Obstacle avoidance path planning in a dynamic circumstance is one of the fundamental problems of autonomous vehicles, counting optional maneuvers: emergency braking and active steering. This paper proposes emergency obstacle avoidance planning based on deep reinforcement learning (DRL), considering safety and comfort. Firstly, the vehicle emergency braking and lane change processes are analyzed in detail. A graded hazard index is defined to indicate the degree of the potential risk of the current vehicle movement. The longitudinal distance and lateral waypoint models are established, including the comfort deceleration and stability coefficient considerations. Simultaneously, a fuzzy PID controller is installed to track to satisfy the stability and feasibility of the path. Then, this paper proposes a DRL process to determine the obstacle avoidance plan. Mainly, multi-reward functions are designed for different collisions, corresponding penalties for longitudinal rear-end collisions, and lane-changing side collisions based on the safety distance, comfort reward, and safety reward. Apply a special DRL method-DQN to release the planning program. The difference is that the long and short-term memory (LSTM) layer is utilized to solve incomplete observations and improve the efficiency and stability of the algorithm in a dynamic environment. Once the policy is practiced, the vehicle can automatically perform the best obstacle avoidance maneuver in an emergency, improving driving safety. Finally, this paper builds a simulated environment in CARLA and is trained to evaluate the effectiveness of the proposed algorithm. The collision rate, safety distance difference, and total reward index indicate that the collision avoidance path is generated safely, and the lateral acceleration and longitudinal velocity satisfy the comfort requirements. Besides, the method proposed in this paper is compared with traditional DRL, which proves the beneficial performance in safety and efficiency.

Testing for main fixed effects: The symmetry assumption and monotone incomplete data

We consider the balanced two-way mixed effects design with some empty cells. A test procedure for the hypothesis of no main fixed effects is developed under violation of the assumption of variance homogeneity and symmetry. The asymptotic null distribution of the test statistics is studied under the condition that the number of levels of the random effects tends to infinity as both the number of complete and incomplete observations tend to infinity. An illustrative example is given.

Feasibility of Novel Magnetically Controlled Cable Capsule Endoscopy System In Vitro Experiments for Gastric Examination.

Background Magnetically controlled capsule endoscopy has been shown to be feasible for the examination of gastric diseases. However, there may be problems, such as low image quality, incomplete esophageal observation, and capsule retention. We developed a novel magnetically controlled cable capsule endoscopy (MCCCE) system and evaluated its feasibility through in vitro experiments. Methods Three experienced endoscopists performed MCCCE on the plastic stomach model and the excised porcine stomach model 5 times, respectively. Outcomes included handle ability, examination time, examination completion, and image quality. The examination completion was accessed by other two blinded endoscopists, and the image quality was compared with conventional gastroscopy (Olympus, GIF-290). Results The performance of MCCCE in vitro experiments is excellent, with a mean operation time of 18.5 minutes in the plastic stomach model and 17.3 minutes in the excised porcine stomach model. The visualization rate of the gastric mucosa is >90% in the plastic stomach model and 75–90% in the excised porcine stomach model. The images of MCCCE are very clear, with good color resolution and no image distortion, which seem to be comparable to conventional gastroscopy. Conclusions MCCCE system is feasible for gastric examination in vitro experiments, living animal experiments and human trials need to be further conducted.

An Interrelated Imitation Learning Method for Heterogeneous Drone Swarm Coordination

The proliferation of small drones has boosted diverse intelligent services, in which the effective swarm coordination plays a vital role in enhancing execution efficiency. However, owing to unreliable air communication and heterogeneous computation capabilities, it is difficult to achieve coordinated actions particularly in distributed scenarios with incomplete observations. In this article, we utilize the generative adversarial imitation learning (GAIL) model to coordinate the drones’ maneuvers by imitating the peer's demonstrations. However, incomplete observations will lead to inaccurate imitation policies. In order to recover true environment states, we encode historical observation-action trajectories into latent belief representations, which are trained in correlation to imitation policies. Moreover, by merging the trace of historical contexts, the prediction of future states and the action-assisted guidance information, we gain robust belief representations, which lead to more accurate imitation policies. We evaluate the algorithm performance via the drones’ formation control task. Experiment results display the superiorities on imitation accuracy, execution time and energy cost.

Traffic Data Recovery From Corrupted and Incomplete Observations via Spatial-Temporal TRPCA

Traffic information can be used for real-time traffic management and long-term transportation planning to increase traffic efficiency and safety. However, data containing both missing and deviating values, can seriously affect the accuracy of traffic information, even leading to incorrect results in traffic data analysis. In this paper, we propose a novel tensor-based data recovery method named spatial-temporal tensor robust principal component analysis (ST-TRPCA) to recover traffic data from corrupted and incomplete observations. Specifically, we not only fully account for the spatial-temporal properties of traffic data to increase the data recovery accuracy, but also utilize tensor factorization and its low-dimensional representation to improve computational efficiency. The extensive experimental results performed on real-world traffic dataset under various scenarios show that ST-TRPCA outperforms other state-of-the-art methods in both missing data recovery and anomaly detection, especially when the traffic data are severely corrupted.