Auxiliary Stream Research Articles

TAG-fusion: Two-stage attention guided multi-modal fusion network for semantic segmentation

In the current research, leveraging auxiliary modalities, such as depth information or point cloud information, to improve RGB semantic segmentation has shown significant potential. However, existing methods mainly use convolutional modules for aggregating features from auxiliary modalities, thereby lacking sufficient exploitation of long-range dependencies. Moreover, fusion strategies are typically limited to singular approaches. In this paper, we propose a transformer-based multimodal fusion framework to better utilize auxiliary modalities for enhancing semantic segmentation results. Specifically, we employ a dual-stream architecture for extracting features from RGB and auxiliary modalities, respectively. We incorporate both early fusion and deep feature fusion techniques. At each layer, we introduce mixed attention mechanisms to leverage features from other modalities, guiding and enhancing the current modality's features before propagating them to the subsequent stage of feature extraction. After the extraction of features from different modalities, we employ an enhanced cross-attention mechanism for feature interaction, followed by channel fusion to obtain the final semantic features. Subsequently, we provide separate supervision to the network on the RGB stream, auxiliary stream, and fusion stream to facilitate the learning of representations for different modalities. The experimental results demonstrate that our framework exhibits superior performance across diverse modalities. Specifically, our approach achieves state-of-the-art results on the NYU Depth V2, SUN-RGBD, DELIVER and MFNet datasets.

Utilizing Silhouette and Head Information for Improved Cloth-changing Person Re-Identification

In recent years, significant progress has been made in person re-identification (ReID). However, as the time span increases, the phenomenon of changing clothes often occur, and the performance of methods based on person re-identification may decrease in this case. In this article, we focus on solving the problem of cloth-changing person re-identification from a single RGB image, which is more challenging, and improving the accuracy and robustness of traditional methods for recognition. We propose a cloth-changing Feature Regularization learning framework by fusing Silhouette and Head information (SHFR), a two-stream framework which transfers the silhouette information and head information learned by auxiliary stream to the main stream and supplements features unrelated to clothing. Specifically, the main stream approach we adopt is the traditional cloth-changing person re-identification network. In the auxiliary stream, we use the DeepLabV3 semantic segmentation model to extract person silhouette features, and the FCHD fully convolutional head detection model to extract head information with advanced semantic information. We concatenate the head information and background information into the silhouette as input for BackBone to continue training the network. In order to utilize silhouette information and head information, we regularize the cloth-changing features on both main stream and auxiliary stream, allowing main stream model to pay more attention to features unrelated to clothing and improve the accuracy of model recognition. We conduct extensive experiments on our model on the PRCC dataset, and the results show that our method has highly competitive performance.

A Structure-Aware Convolutional Neural Network for Automatic Diagnosisof Fungal Keratitis with In Vivo Confocal Microscopy Images.

Fungal keratitis (FK) is a common and severe corneal disease, which is widely spread in tropical and subtropical areas. Early diagnosis and treatment are vital for patients, with confocal microscopy cornea imaging being one of the most effective methods for the diagnosis of FK. However, most cases are currently diagnosed by the subjective judgment of ophthalmologists, which is time-consuming and heavily depends on the experience of the ophthalmologists. In this paper, we introduce a novel structure-aware automatic diagnosis algorithm based on deep convolutional neural networks for the accurate diagnosis of FK. Specifically, a two-stream convolutional network is deployed, combining GoogLeNet and VGGNet, which are two commonly used networks in computer vision architectures. The main stream is used for feature extraction of the input image, while the auxiliary stream is used for feature discrimination and enhancement of the hyphae structure. Then, the features are combined by concatenating the channel dimension to obtain the final output, i.e., normal or abnormal. The results showed that the proposed method achieved accuracy, sensitivity, and specificity of 97.73%, 97.02%, and 98.54%, respectively. These results suggest that the proposed neural network could be a promising computer-aided FK diagnosis solution.

HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection

The High-Resolution Transformer (HRFormer) can maintain high-resolution representation and share global receptive fields. It is friendly towards salient object detection (SOD) in which the input and output have the same resolution. However, two critical problems need to be solved for two-modality SOD. One problem is two-modality fusion. The other problem is the HRFormer output's fusion. To address the first problem, a supplementary modality is injected into the primary modality by using global optimization and an attention mechanism to select and purify the modality at the input level. To solve the second problem, a dual-direction short connection fusion module is used to optimize the output features of HRFormer, thereby enhancing the detailed representation of objects at the output level. The proposed model, named HRTransNet, first introduces an auxiliary stream for feature extraction of supplementary modality. Then, features are injected into the primary modality at the beginning of each multi-resolution branch. Next, HRFormer is applied to achieve forwarding propagation. Finally, all the output features with different resolutions are aggregated by intra-feature and inter-feature interactive transformers. Application of the proposed model results in impressive improvement for driving two-modality SOD tasks, e.g., RGB-D, RGB-T, and light field SOD.https://github.com/liuzywen/HRTransNet

Heat Transfer Analysis and Magnetohydrodynamics Effect on Peristaltic Transport of Ree–Eyring Fluid in Rotating Frame

This paper discusses Ree–Eyring fluid’s peristaltic transport in a rotating frame and examines the impacts of Magnetohydrodynamics (MHD). The results deal with systematically (analytically) applying each of the governing equations of Ree–Eyring fluid, the axial and secondary velocities, flow rate due to auxiliary stream, and bolus. The effects of some distinctive variables, such as Hartman number, heat source/sink, and amplitude ratio, are taken under consideration and illustrated through graphs.

Gait Recognition Method of Underground Coal Mine Personnel Based on Densely Connected Convolution Network and Stacked Convolutional Autoencoder

Biological recognition methods often use biological characteristics such as the human face, iris, fingerprint, and palm print; however, such images often become blurred under the limitation of the complex environment of the underground, which leads to low identification rates of underground coal mine personnel. A gait recognition method via similarity learning named Two-Stream neural network (TS-Net) is proposed based on a densely connected convolution network (DenseNet) and stacked convolutional autoencoder (SCAE). The mainstream network based on DenseNet is mainly used to learn the similarity of dynamic deep features containing spatiotemporal information in the gait pattern. The auxiliary stream network based on SCAE is used to learn the similarity of static invariant features containing physiological information. Moreover, a novel feature fusion method is adopted to achieve the fusion and representation of dynamic and static features. The extracted features are robust to angle, clothing, miner hats, waterproof shoes, and carrying conditions. The method was evaluated on the challenging CASIA-B gait dataset and the collected gait dataset of underground coal mine personnel (UCMP-GAIT). Experimental results show that the method is effective and feasible for the gait recognition of underground coal mine personnel. Besides, compared with other gait recognition methods, the recognition accuracy has been significantly improved.

Saliency Inside: Learning Attentive CNNs for Content-based Image Retrieval.

In content-based image retrieval (CBIR), one of the most challenging and ambiguous tasks are to correctly understand the human query intention and measure its semantic relevance with images in the database. Due to the impressive capability of visual saliency in predicting human visual attention that is closely related to the query intention, this paper attempts to explicitly discover the essential effect of visual saliency in CBIR via qualitative and quantitative experiments. Toward this end, we first generate the fixation density maps of images from a widely used CBIR dataset by using an eye-tracking apparatus. These ground-truth saliency maps are then used to measure the influence of visual saliency to the task of CBIR by exploring several probable ways of incorporating such saliency cues into the retrieval process. We find that visual saliency is indeed beneficial to the CBIR task, and the best saliency involving scheme is possibly different for different image retrieval models. Inspired by the findings, this paper presents two-stream attentive CNNs with saliency embedded inside for CBIR. The proposed network has two streams that simultaneously handle two tasks. The main stream focuses on extracting discriminative visual features that are tightly related to semantic attributes. Meanwhile, the auxiliary stream aims to facilitate the main stream by redirecting the feature extraction to the salient image content that human may pay attention to. By fusing these two streams into the Main and Auxiliary CNNs (MAC), image similarity can be computed as the human being does by reserving conspicuous content and suppressing irrelevant regions. Extensive experiments show that the proposed model achieves impressive performance in image retrieval on four public datasets.

On Problem of Mathematical Modelling of Thermo-Physical Processes in Regenerative Water-Evaporating Coolers

For cooling the air environment of industrial premises water-evaporating air, conditioners are being increasingly applied. The simplicity of their construction, ecological safety and low power consumption distinguish them from the coolers of other types. Cooling the processed air is due to the loss of energy for the evaporation of moisture from the surface of the water-wetted plates that form air channels. As a result of this process, cooled air is often saturated with moisture, which limits the possibilities for the operation of the coolers of this type. In these cases, more complex coolers of indirect principle without such drawback should be applied. The most effective modification of indirect cooling is the installation of recuperative principle units. The paper presents a mathematical model of heat-mass transfer in such water-evaporating coolers. The scheme of realization of this model based on an iterative algorithm of solution of the system of finite–difference linear equations that takes into account longitudinal and transverse thermal conductivity of the heat transfer plates is suggested. The possibility of obtaining the optimal values of the redistribution of the main and auxiliary air flows through the substantiation of the aerodynamic resistance of the output grid is proved. This allows refusing the inclusion in the additional system cooling fan unit for discharging an auxiliary stream of air.

Robust Multiview Three-Dimensional Video Communications Based on Distributed Video Coding

With the rapid development of electronic and communications technology, multiview video services are becoming more and more feasible in practice. However, robustly delivering many views over error-prone channels is a rather challenging task. Due to the recent development of distributed video coding, this paper proposes two alternative error-resilient schemes for multiview video transmission based on the Wyner–Ziv (WZ) coding technique. A lightweight-encoder error-resilient scheme requires no communications between the cameras at the encoder, whereas the temporal and interview redundancy can be explored to generate the side information at the decoder. In this case, it is not only robust to channel losses but also has independent encoders with low encoding complexity. On the other hand, a complex-encoder error-resilient scheme refers to the scheme that employs an auxiliary redundant stream encoded according to a WZ video coding approach to protect a primary stream encoded with a standard multiview video coding codec. At the receiver, the error-concealed reconstructed frames are used as the side information by the WZ decoder. Therefore, this method maintains the original multiview bitstream unaltered by simply adding WZ bits for protection. Experimental results conducted over a simulated error-prone channel reveal that the two proposed schemes have better performance than the forward error correction by means of the Reed–Solomon and intrarefresh approaches.

Lossless data hiding algorithm for encrypted images with high capacity

In this paper, a novel reversible data hiding algorithm for encrypted images is proposed. In encryption phase, chaotic sequence is applied to encrypt the original image. Then the least significant bits (LSBs) of pixels in encrypted image are losslessly compressed to leave place for secret data. With auxiliary bit stream, the lossless compression is realized by the Hamming distance calculation between the LSB stream and auxiliary stream. At receiving terminal, the operation is flexible, that is, it meets the requirement of separation. With the decryption key, a receiver can get access to the marked decrypted image which is similar to the original one. With data-hiding key, the receiver can successfully extract secret data from the marked encrypted image. With both keys, the receiver can get secret data and the exactly original image. Compared with existing methods, experiments show the feasibility and efficiency of the proposed method, especially in aspect of embedding capacity, embedding quality and error-free recovery with increasing payload.

Auxiliary stream for optimizing memory access of video decoders

Due to the ever increasing resolution and frame rate of mainstream video sequences, memory access has become the main performance bottleneck of video decoding. To reduce the required off-chip memory, many decoders employ on-chip cache. However, they cannot distinguish whether a data block is reusable due to the lack of the information of undecoded Macro Blocks (MBs), thus often evicting reusable data from the cache and preserving non-reusable data in the cache, which will lead to a waste of off-chip memory bandwidth. In this paper, we manage to make full use of cache from a novel perspective, i.e., auxiliary bitstream. Concretely speaking, since the memory access behavior of video decoding is determined in video encoding, the encoder can pack the memory access behaviors of video decoding as auxiliary bitstream, which can inform the decoder whether a data block will be reused by future MBs. Hence, such an auxiliary stream can enable optimal management of cache. To effectively reduce the size of auxiliary bitstream, we propose an Auxiliary Prior Information Coding (APIC) method complying with the current video standards. For future video standards, we introduce a Super Block scan Order (SBO) for MB organization to further reduce the bitrate overhead of auxiliary bitstream. The above ideas are evaluated on a number of representative video sequences. The additional prior information can reduce the required off-chip memory bandwidth for motion compensation by over 35% (for a 60 kB cache), while only causing less than 2.3% bitrate increase for high definition (HD) videos.

HNO3-activated mesoporous carbon catalyst for direct dehydrogenation of propane to propylene

Abstract Mesoporous carbon activated with HNO3 was demonstrated to be robust metal-free catalyst for direct dehydrogenation of propane without any auxiliary stream, exhibiting high selectivity and stability. The final propane conversion is 22.4% with stable propylene selectivity of 86.6% after 50 hours. HNO3 activation introduces more basic oxygen groups than the pristine carbon, which is believed to be active site in the dehydrogenation process.

Assessing the value of high‐resolution isotope tracer data in the stepwise development of a lumped conceptual rainfall–runoff model

AbstractA study was undertaken in a small agricultural catchment in north east Scotland with the objective of assessing the value of high‐resolution isotope data for integration within a lumped, conceptual hydrological model to improve calibration and aid evaluation. Daily samples of precipitation and stream flow, collected over a year, were analysed for deuterium using new laser spectroscopy technology. The utility of such high‐resolution isotope data was evaluated in relation to the associated uncertainty which was contextualized in relation to uncertainties over hydrometric data and the influence of different sampling resolutions. The simulations were evaluated against model and data errors using auxiliary stream deuterium time series in addition to discharge. The lumped conceptual catchment isotope model (CIM) was developed and adequately reflects flow dynamics and deuterium peaks, but a simple assumption of ‘good mixing’ is not able to fully reproduce the daily stream deuterium dynamic. Using auxiliary data for model evaluation, however, significantly constrained acceptable behavioural parameter sets and therefore reduces the model's degree of freedom. The data indicate that isotopic variability in the stream response is not adequately captured using weekly tracer data. The input resolution of precipitation deuterium concentrations, which were much more variable, proved to be crucial. This approach has provided further assessment of the value of tracers in hydrological modelling, demonstrating their usefulness in terms of model conceptualization, development and calibration, which outweighs the additional uncertainty. Copyright © 2010 John Wiley & Sons, Ltd.

Hyphenation of centrifugal partition chromatography with electrospray ionization mass spectrometry using an active flow‐splitter device for characterization of flavonol glycosides

Online coupling of centrifugal partition chromatography to electrospray ionization mass spectrometry (CPC/ESI-MS) was investigated for the separation and characterization of flavonol glycosides. Structural identification and purification monitoring of analytes on milligram scale were demonstrated to be possible by using an active flow-splitter device which transfers automatically and successively, at discrete frequencies, small aliquots of the chromatographic effluent to an independent auxiliary stream directed to an ESI quadrupole mass spectrometer. The CPC protocol used a biphasic solvent system composed of ethyl acetate/ethanol/water (4.5:1:4.5, v/v/v) in isocratic mode. During the separation process, continuous acquisition of mass spectral data of the isolated flavonols from the effluent was performed in the negative ion mode with an auxiliary stream composed of 50 mM ammonium acetate/ethanol (2:8, v/v) delivered by a secondary pump. To demonstrate the potential of this hyphenated technique, flavonol glycosides from an apple peel extract were identified, purified and quantitatively analyzed. Calibration curves and limits of detection are also detailed.

Thermodynamic analysis of different two-stage transcritical carbon dioxide cycles

The aim of this paper is the thermodynamic evaluation and optimisation of different two-stage transcritical carbon dioxide cycles. Five different cycles are studied: basic single-stage cycle, single-throttling with two-stage compression cycle, split cycle, phase separation cycle and single-stage cycle coupled with a gas cooling circuit. Each basic cycle is analysed for the effect of internal heat transfer between different streams of refrigerants. In the case of two-stage compression, intermediate cooling between the compressor stages is present. An analysis on the Plank cycle for intermediate pressure higher than critical one is performed. Each cycle is optimised with regards to energy performance, calculating the optimal values of both the upper and the intermediate pressures. In the case of split cycle, the ratio of the mass flow rate in the main stream to the one in the auxiliary stream is also optimised.

Zerotree-based stereoscopic video CODEC

Due to the provision of a more natural representation of a scene in the form of left and right eye views, a stereoscopic imaging system provides a more effective method for image/video display. Unfortunately the vast amount of information that must be transmitted/stored to represent a stereo image pair/video sequence, has so far hindered its use in commercial applications. However, by properly exploiting the spatial, temporal and binocular redundancy, a stereo image pair or a sequence could be compressed and transmitted through a single monocular channel's bandwidth without unduly sacrificing the perceived stereoscopic image quality. We propose a timely and novel framework to transmit stereoscopic data efficiently. We propose a timely and novel framework to transmit stereoscopic data efficiently. We present a new technique for coding stereo video sequences based on discrete wavelet transform (DWT) technology. The proposed technique particularly exploits zerotree entropy (ZTE) coding that makes use of the wavelet block concept to achieve low bit rate stereo video coding. One of the two image streams, namely, the main stream, is independently coded by a zerotree video CODEC, while the second stream, namely, the auxiliary stream, is predicted based on disparity compensation. A zerotree video CODEC subsequently codes the residual stream. We compare the performance of the proposed CODEC with a discrete cosine transform (DCT)-based, modified MPEG-2 stereo video CODEC. We show that the proposed CODEC outperforms the benchmark CODEC in coding both main and auxiliary streams.

Motion/Disparity Compensated Multiview Sequence Coding

A multiview sequence coding technique is proposed in this paper. A multiview sequence encoder must be able to reduce redundancies in both the time and view domains because the amount of data increases as the number of views (or cameras) increases. We define a new coding structure, a group of a group of pictures (GGOP), which is compatible with MPEG-2 and is flexible with a baseline distance. The GGOP could have several possible types, e.g., one-I type, two-I type, etc., according to the number of reference frame sequences. This permits one of these types to be selected according to the baseline distance among the cameras. In addition, the GGOP is designed for MPEG-2 compatibility. The proposed multiview sequence encoder consists of preprocessing stages, disparity estimation/compensation, motion estimation/compensation, residual coding, rate control, and entropy coding. It generates two types of bit streams, the main bit stream and the auxiliary bit stream. The main bit stream contains information concerning reference sequences including I-pictures, in order to maintain MPEG-2 compatibility. The auxiliary bit stream contains information concerning the remaining multiview sequences except for the sequences that include I-pictures. The proposed encoder shows improvements in compression ratio as well as in peak signal-to-noise ratio (PSNR) compared with conventional methods. The dependency of the GGOP on the baseline distance between cameras is also confirmed.

The Tornado Flow Configuration—An Effective Method for Screening of a Solar Reactor Window

The working fluid in solar receivers, utilized for effecting chemical reactions, is usually flown through a sealed enclosure provided with a quartz window. When one of the reactants or products of reaction is a powder, care must be taken to prevent contact of the incandescent powder particles with the window, in order to obviate its destruction by overheating. Attempts made in the past to screen the window against particle deposition by a “curtain” of an auxiliary gas stream showed that very substantial flow rates of auxiliary gas (30—80% of the main stream flow rate) were necessary for perfect window screening. The heat absorbed by the auxiliary gas stream represented a major loss of energy. In an effort to reduce the auxiliary stream flow rate to a minimum, a certain flow pattern akin to the natural tornado phenomenon has recently been developed in our laboratory. It enabled effective reactor window screening by an auxiliary gas flow rate less than 5% of the main gas flow rate. The tornado effect is discussed and demonstrated by a smoke flow visualization technique.

Flow-injection determination of lead by hydride generation and conductometric detection.

Plumbane produced from the lead analyte in a flow-injection manifold by reaction with sodium borohydride is passed through a porous poly(tetrafluoroethylene) membrane in a gas-diffusion cell. The hydride reacts with bromine in the acceptor stream resulting in ionization which is detected by conductivity measurement. Direct mixing of the carrier with a reagent stream yields a limit of detection of approximately 1 mg/L. An improved detection limit of about 200 microg/L can be achieved by the incorporation of an auxiliary stream containing persulphate as oxidizing agent. The application of the method to the determination of lead in road dust and soil samples is demonstrated.

On the evolution of Rossby waves, generated by wind stress in a closed basin, incorporating total mass conservation

Abstract The evolution of Rossby waves generated in a closed basin by applied stationary wind stress is considered, taking into account the total mass conservation constraint. The behavior of the wave field for large time t and the formation of a Sverdrup regime in the open ocean are analyzed by neglecting the frictionally induced wave decay. The consistent quasigeostrophic formulation of the problem is developed and the conditions are found for the applicability of relations derived. Using the vertical mode expansion reduces the 3D problem to a set of 2D problems. A method of solving the forced 2D problem with the mass constraint is suggested. The method consists of solving several particular problems, the most important of which are the forced problem for the auxiliary stream function Ψτ with zero boundary values and the Volterra integral equation of the second kind for time-dependent boundary values of the stream function Ψ. To simplify the analysis, the 1D model of the forced problem considered is offered. The analytical solution of the problem with zero boundary values of the stream function Ψ is found. It is known that, in general, such a solution does not stabilize with time but for large values of m (the ratio of the basin dimension to the Rossby radius of deformation) and t it tends to the Sverdrup solution in the open ocean. The time-averaged stream function Ψ is introduced, which tends to the Sverdrup solution in the open ocean for all m. A solution of the consistent problem satisfying the total mass conservation constraint is obtained and analyzed. It is pointed out that the stream function Ψ does not tend to the Sverdrup solution outside the western boundary layer for large values of m and t, which differs drastically from the corresponding behavior of Ψ in the problem with zero boundary values. This demonstrates the failure of the solution of the inconsistent problem with zero boundary values to describe the wave motion for large t. It is shown that the establishment of the Sverdrup solution for large t can be seen only if the time-averaged stream function Ψ is considered. It is proved for all m that outside the western boundary layer Ψ tends to the Sverdrup solution (minus the integral of Ψ in x over the whole basin) for large t. The evolution of the total energy is also discussed.