Media Retrieval Research Articles

<i>c</i>-SNE: Deep Cross-modal Retrieval based on Subjective Information using Stochastic Neighbor Embedding

Cross-modal information retrieval based on subjective information aims to enable flexible media retrieval services, such as allowing users to specify, for example, an image to search audio clips. The resulting audio clips should have an impression similar to the specified image. Existing methods focus on building cross-media cross-modal relationships using objective information (such as the standard caption). However, such a relation can be built only between the pieces of media that are originally related, which limits the flexibility of cross-modal media retrieval. This research leverages subjective information in the media clips for similarity calculation to achieve more flexibility. We propose a novel cross-modal stochastic neighbor embedding technique called c-SNE. c-SNE can extract features of subjective information from pieces of media and map them in the common embedding space. It is a learning technique to bridge the heterogeneous gap between the modal distributions using label-weighted SNE. It allows users to find the media that share the same subjective information with a query medium. Our experimental results on the benchmark datasets demonstrate that the proposed method effectively performs in cross-modal distribution alignment and retrieval. Furthermore, our user study with ten users with 600 data points confirmed that c-SNE outperforms three related methods in the actual usage situation from the users' perspective.

Deep learning model based multimedia retrieval and its optimization in augmented reality applications.

With the uproar of touchless technology, the Virtual Continuum has seen some spark in the upcoming products. Today numerous gadgets support the use of Mixed Reality / Augmented Reality (AR)/ Virtual Reality. The Head Mounted Displays (HMDs) like that of Hololens, Google Lens, Jio Glass manifested reality into virtuality. Other than the HMDs many organizations tend to develop mobile AR applications to support umpteen number of industries like medicine, education, construction. Currently, the major issue lies in the performance parameters of these applications, while deploying for mobile application’s graphics performance, latency, and CPU functioning. Many industries pose real-time computation requirements in AR but do not implement an efficient algorithm in their frameworks. Offloading the computation of deep learning models involved in the application to the cloud servers will highly affect the processing parameters. For our use case, we will be using Multi-Task Cascaded Convolutional Neural Network (MTCNN) which is a modern tool for face detection, using a 3-stage neural network detector. Therefore, the optimization of communication between local application and cloud computing frameworks needs to be optimized. The proposed framework defines how the parameters involving the complete deployment of a mobile AR application can be optimized in terms of retrieval of multimedia, its processing, and augmentation of graphics, eventually enhancing the performance. To implement the proposed algorithm a mobile application is created in Unity3D. The mobile application virtually augments a 3D model of a skeleton on a target face. After the mentioned experimentation, it is found that average Media Retrieval Time (1.1471 μ s) and Client Time (1.1207 μ s) in the local application are extremely low than the average API process time (288.934ms). The highest time latency is achieved at the frame rate higher than 80fps.

Effectiveness of Water Hyacinth (Eichhornia crassipes) and Water Spinach (Ipomoea aquatica) to Reduce Nitrate and Phosphate Concentrations in Cimulu River Water, Tasikmalaya City, Indonesia

Water hyacinth and water spinach can be used as phytoremediation agents in the Cimulu River because these two plants can grow and adapt in the Cimulu River. This study compared the effectiveness of the two plants in reducing nitrate and phosphate in Cimulu river water. The research was conducted from September to October 2021. The research was conducted ex-situ and in-situ. Media retrieval from the Cimulu River was carried out in situ in the middle of the Cimulu River, Tawang District, Tasikmalaya City, Indonesia. aquatic plants, analysis of pH, temperature, and sampling were carried out ex-situ in the backyard of the house at the LIK Complex, Tasikmalaya City, Indonesia. Analysis of water samples such as DO, BOD, nitrate, and phosphate was carried out ex-situ at the Water Resources Management Laboratory of FPIK Padjadjaran University. The method used is an experimental method, with three treatments consisting of control treatment, treatment A (water hyacinth), and treatment B (water spinach). Each aquarium in treatments A and B contained aquatic plants weighing 200 g each. The results of the research conducted for 28 days showed that water hyacinth weighing 200 g was more effective in reducing nitrate and phosphate than water spinach with the same weight. Furthermore, water hyacinth can reduce nitrate by 61.72% and phosphate by 78.27% with the absorption rate of nitrate in one day reaching 0.006 g/m2 and phosphate 0.038 g/m2. The growth rates of water hyacinth in weeks I, II, III, and IV were 5.03±0.971%, 6.31±0.971%, 6.80±0.321%, and 7.18±0.277%. Meanwhile, other water parameters that can be reduced are BOD from 37.3 mg/L to 8.10 mg/L.

Latex bag contrast system for the diagnosis of oesophageal and rectal strictures in two cats

AbstractA common problem we often encounter during traditional contrast radiography is that contrast media are not adequately retained in the target region, especially during oesophagography and proctography. Patients with oesophageal disorders often have dysphagia or regurgitation, which increase the risk of aspiration pneumonia. Stercoroma in patients with rectal disease often prevents clinicians from obtaining good‐quality proctography for diagnosis. In the present case report, we introduce a novel contrast diagnostic system that has been used to diagnose oesophageal and rectal strictures. Latex bags (commercial latex condoms) are thin with good elasticity, producing a delicate interface for contrast filling, thereby providing better image quality. The watertight design of the latex bag system facilitates the retrieval of contrast media after radiographic examination. Furthermore, it significantly decreases the risk of aspiration pneumonia during the diagnostic process.

Application of new media retrieval system based on Neutrosophic social marketing

The new media is a crucial ply role in social marketing. The new media growth rapidly in social marketing so, select consumers and companies for competition. So, the new media is a complex task for companies and firms. The new media retrieval contains much content such as image retrieval, audio, and video retrieval. The new media retrieval contains many criteria, and each criterion contains sub-criteria. So, use an MCDM method for dealing with this problem. The new media retrieval system is analyzed under a neutrosophic environment. The Decision-making trial and evaluation laboratory (DEMATEL) integrated with single-valued neutrosophic. The DEMATEL method is used to show the importance and impact criteria of new media retrieval for helping companies in social marketing competition. An example is provided for showing calculations.

Application of new media retrieval system based on Neutrosophic in marketing

The new media is a crucial ply role in the market. The new media growth rapidly in the market so, select consumers and companies for competition. So, the new media is a complex task for companies and firms. The new media retrieval contains much content such as image retrieval, audio and video retrieval. The new media retrieval contains many criteria, and each criterion contains sub-criteria. So, use an MCDM method for dealing with this problem. The new media retrieval system is analyzing under a neutrosophic environment. The Decision-making trial and evaluation laboratory (DEMATEL) integrated with single-valued neutrosophic. DEMATEL method used to show importance and impact criteria of new media retrieval for helping companies in market competition. An example is provided for showing calculations.

Social media image classification and retrieval method based on deep hash algorithm

Social media image classification and retrieval method based on deep hash algorithm

Cross-Media Intelligent Perception and Retrieval Analysis Application Technology Based on Deep Learning Education

This paper proposes a research on cross-media intelligent perception and retrieval analysis technology based on deep learning education in view of the complex learning knowledge environment of artificial intelligence of traditional cross-media retrieval technology, which is unable to obtain retrieval information timely and accurately. Based on the cross-media theory, this paper analyzes the pre-processing of cross-media data, transforms from a single media form such as text, voice, image, and video to a cross-media integration covering network space and physical space, and designs a cross-media intelligent perception platform system. With multi-core method of typical correlation analysis algorithm, this paper develops a new joint learning framework based on the joint feature selection and subspace learning cross-modal retrieval. What’s more, the algorithm is tested experimentally. The results show that the retrieval analysis technology is significantly better than the traditional media retrieval technology, which can effectively identify text semantics and visual image classification, and can better maintains the relevance of data content and the consistency of semantic information, which has important reference value for cross-media applications.

2019 Tony Kent Strix Memorial Award Winner

In 2019, the UK electronic information Group (UKeiG), in partnership with the International Society for Knowledge Organisation UK (ISKO UK), the Royal Society of Chemistry Chemical Information and Computer Applications Group (RSC CICAG) and the British Computer Society Information Retrieval Specialist Group (BCS IRSG), was delighted to announce that the winner of the prestigious Tony Kent Strix Memorial Award for 2019, in recognition of his major and sustained contributions to the field of information retrieval, was Professor Ingemar J. Cox, Department of Computer Science at University College London.
 Professor Cox is Head of UCL’s Media Futures Research Group and a Professor at the Department of Computer Science at the University of Copenhagen. His current research interests include information retrieval and data analytics of online social media, twitter and query logs.

Maneuvers of Multi Perspective Media Retrieval

Recently, Learning Machines have achieved a measure of success in the representation of multiple views. Since the effectiveness of data mining methods is highly dependent on the ability to produce data representation, learning multi-visual representation has become a very promising topic with widespread use. It is an emerging data mining guide that looks at multidisciplinary learning to improve overall performance. Multi-view reading is also known as data integration or data integration from multiple feature sets. In general, learning the representation of multiple views is able to learn the informative and cohesive representation that leads to the improvement in the performance of predictors. Therefore, learning multi-view representation has been widely used in many real-world applications including media retrieval, native language processing, video analysis, and a recommendation program. We propose two main stages of learning multidisciplinary representation: (i) alignment of multidisciplinary representation, which aims to capture relationships between different perspectives on content alignment; (ii) a combination of different visual representations, which seeks to combine different aspects learned from many different perspectives into a single integrated representation. Both of these strategies seek to use the relevant information contained in most views to represent the data as a whole. In this project we use the concept of canonical integration analysis to get more details. Encouraged by the success of in-depth reading, in-depth reading representation of multiple theories has attracted a lot of attention in media access due to its ability to read explicit visual representation.

Use of loss limit approach to zero in scattering-based parameter retrieval of elastic micro-structured media

The parameter retrieval method based on scattering data is used to derive dynamic constitutive parameters of solids with periodic structure. There are inherent ambiguities in the real part of the retrieved wavenumber and the sign of impedance, though the latter has a one-to-one correspondence with the direction of energy flux. Moreover, for lossless structures there can be multiple solution branches that satisfy passivity and continuity of the wavenumber as a function of frequency, leading to potentially double positive or double negative (in terms of density and modulus) overall descriptions of the micro-structured medium. The continuity of wavenumber for lossy unit cells is used to unambiguously determine their constitutive parameters and by taking the limit when loss approaches zero, one can determine the stable solution branches of lossless micro-structures. The two loss models of nonzero damping and complex modulus are compared in terms of their energy loss characteristics and retrieved parameters. These models are employed in both time and frequency domain calculations. The lossy solutions demonstrate that from double-negative and double-positive solutions, only the latter is the stable solution branch in a pass-band of a lossless 1D unit cell. A 2D photonic crystal example is used to demonstrate that the wavenumber appears to jump, discontinuously, at the transition point between two consecutive stop-bands, hence rendering the existing methods based on continuity of wavenumber ineffective. In contrast, the proposed approach based on taking limit of lossy solutions can successfully be used to determine stable overall properties of such media. Finally, certain features of half- and full-cycle stop-bands are discussed.

SketchHelper: Real-Time Stroke Guidance for Freehand Sketch Retrieval

Text-based retrieval systems have been popular, but content-based retrieval systems have gained widespread acceptance in recent years to directly retrieve diverse media based on their visual content, such as color, texture, and shape. Among many content-based retrieval systems, sketch-based media retrieval systems have attracted attention recently with the proliferation of tablet PCs and smart mobile devices. Sketch-based retrieval requires the user to draw a freehand sketch query, but freehand drawing can be challenging for those with limited drawing skills. This degrades retrieval performance, since successful retrieval depends on the quality of the sketch query image drawn by the user. To address this issue, we propose a real-time stroke guidance for freehand sketch retrieval that continuously displays next-stroke shadow sketches on the canvas based on the user's step-by-step partial strokes. We train a stroke guidance network that learns the mapping between the step-wise stroke relations to predict the user's next stroke. The proposed stroke guidance for freehand sketch retrieval system runs on a five step next-stroke prediction model that identifies candidate next-stroke sketches from a database of millions of sketches. The system retrieves variable number of sketch object classes at different drawing stages. During the initial sketching stage, diverse drawing possibilities are covered by retrieving multiple sketch classes; as the sketching progresses, the intended sketch class is narrowed down to one. Deep binary hashing is employed for efficient similarity matching of relevant next-stroke sketches. We extend the Google QuickDraw dataset to create a five step sketch stroke database. Qualitative and quantitative experiments are conducted to verify the effectiveness of the proposed system, which can be utilized for drawing guidance, tracing, and sketch retrieval. Tracing refers to the act of copying the shadowed line of a guiding image by drawing over its lines.

Supervised graph regularization based cross media retrieval with intra and inter-class correlation

With the rapid development of internet technology, mining and retrieving the information from internet accurately is an urgent problem, among which, cross media retrieval becomes a hot spot of current research. This paper proposes a cross media retrieval approach, which learns two couples of projections based on different retrieval tasks. We first learn a common subspace to project heterogeneous media data to the isomorphic subspace, to measure the similarity of the heterogeneous media data in the isomorphic subspace. Second, we build isomorphic and heterogeneous adjacent graphs to preserve the correlations of the cross media data. Then we combine the two processes together to learn a common subspace. We also consider intra-class and inter-class similarity of images or texts in the unified framework. Third, the L2 norm is used to perform feature selection for different media data. Experimental results on three datasets demonstrate the effectiveness of the proposed approach.

Attribute-enhanced metric learning for face retrieval

Metric learning is a significant factor for media retrieval. In this paper, we propose an attribute label enhanced metric learning model to assist face image retrieval. Different from general cross-media retrieval, in the proposed model, the information of attribute labels are embedded in a hypergraph metric learning framework for face image retrieval tasks. The attribute labels serve to build a hypergraph, in which each image is abstracted as a vertex and is contained in several hyperedges. The learned hypergraph combines the attribute label to reform the topology of image similarity relationship. With the mined correlation among multiple facial attributes, the reformed metrics incorporates the semantic information in the general image similarity measure. We apply the metric learning strategy to both similarity face retrieval and interactive face retrieval. The proposed metric learning model effectively narrows down the semantic gap between human and machine face perception. The learned distance metric not only increases the precision of similarity retrieval but also speeds up the convergence distinctively in interactive face retrieval.

COSA: Cloud Object Storage Archive for deep archival of digital data

The Cornell Lab of Ornithology gathers, utilizes and archives a wide variety of digital assets ranging from details of a bird observation to photos, video and sound recordings. Some of these datasets are fairly small, while others are hundreds of terabytes. In this presentation we will describe how the Lab archives these datasets to ensure the data are both loss-less and recoverable in the case of a widespread disaster, how the archival strategy has evolved over the years and explore in detail the current hybrid cloud storage management system. The Lab runs eBird and several other citizen science programs focused on birds where individuals from around the globe enter their sightings into a centralized database. The eBird project alone stores over 500,000,000 observations and the underlying database is over a terabyte in size. Birds of North America, Neotropical Birds and All About Birds are online species accounts comprising a wide range of authoritative live history articles maintained in a relatively small database. Macaulay Library is the world’s largest image, sound and video archive with over 6,000,000 cuts totaling nearly 100 TB of data. The Bioacoustics Research Program utilizes automated recording units (SWIFTs) in the forests of the US, jungles of Africa and in all seven oceans to record the environment. These units record 24 hours a day and gather a tremendous about of raw data, over 200 TB to date with an expected rate of an additional 100TB per year. Lastly, BirdCams run by the lab add a steady stream of media detailing the reproductive cycles of a number of species. The lab is committed to making these archives of the natural world available for research and conservation today. More importantly, ensuring these data exist and are accessible in 100 years is a critical component of the Lab data strategy. The data management system for these digital assets has been completely overhauled to handle the rapidly increasing volume and to utilize on-premises systems and cloud services in a hybrid cloud storage system to ensure data are archived in a manner that is redundant, loss-less and insulated from disasters yet still accessible for research. With multimedia being the largest and most rapidly growing block of data, cost rapidly becomes a constraining factor of archiving these data in redundant, geographically isolated facilities. Datasets with a smaller footprint, eBIrd and species accounts allow for a wider variety of solutions as cost is less of a factor. Using different methods to take advantage of differing technologies and balancing cost vs recovery speed, the Lab has implemented several strategies based on data stability (eBird data are constantly changing), retrieval frequency required for research and overall size of the dataset. We utilize Amazon S3 and Glacier as our media archive, we tag each media in Glacier with a set of basic DarwinCore metatdata fields that key back to a master metadata database and numerous project specific databases. Because these metadata databases are much smaller in size, yet critical in searching and retrieval of a required media file, they are archived differently with up to the minute replication to prevent any data loss due to an unexpected disaster. The media files are tagged with a standard set of basic metadata and in the case where the metadata databases were unavailable, retrieval of specific media and basic metadata can still occur. This system has allowed the lab to place into long term archive hundreds of terabytes of data, store them in redundant, geographically isolated locations and provide for complete disaster recovery of the data and metadata.

A Review on Correlation Maximized Similarity Measurement in Cross Media Retrieval Method

A Review on Correlation Maximized Similarity Measurement in Cross Media Retrieval Method

Joint graph regularization based semantic analysis for cross-media retrieval: a systematic review

The large number of heterogeneous data are rapidly increasing in the internet and most data consist of audio, video, text and images. The searching of the required data from the large database is difficult and time taking process. The single media retrieval is used to get the needed data from the large dataset and it has the drawback, it can only retrieve the single media only. If the query is given as the text and acquired result are present in text. The users demand the cross-media retrieval for their queries and it is very consistent in providing the result. This helps the users to get more information regarding to their queries. Finding the similarities between the heterogeneous data is very complex. Many research is done on the cross-media retrieval with different methods and provide the different result. The aim is to analysis the different cross-media retrieval with the joint graph regularization (JGR) to understand the various technique. The most of researches are using the parameter of MAP, precision and recall for their research.

Deep Learning for Mobile Multimedia

Deep Learning (DL) has become a crucial technology for multimedia computing. It offers a powerful instrument to automatically produce high-level abstractions of complex multimedia data, which can be exploited in a number of applications, including object detection and recognition, speech-to- text, media retrieval, multimodal data analysis, and so on. The availability of affordable large-scale parallel processing architectures, and the sharing of effective open-source codes implementing the basic learning algorithms, caused a rapid diffusion of DL methodologies, bringing a number of new technologies and applications that outperform, in most cases, traditional machine learning technologies. In recent years, the possibility of implementing DL technologies on mobile devices has attracted significant attention. Thanks to this technology, portable devices may become smart objects capable of learning and acting. The path toward these exciting future scenarios, however, entangles a number of important research challenges. DL architectures and algorithms are hardly adapted to the storage and computation resources of a mobile device. Therefore, there is a need for new generations of mobile processors and chipsets, small footprint learning and inference algorithms, new models of collaborative and distributed processing, and a number of other fundamental building blocks. This survey reports the state of the art in this exciting research area, looking back to the evolution of neural networks, and arriving to the most recent results in terms of methodologies, technologies, and applications for mobile environments.

Real-time social media retrieval with spatial, temporal and social constraints

Search in social network is continuously being expanded to enhance user experience. Besides basic textual retrieval, users can also search based on features such as spatial proximity, temporal freshness and/or social closeness. To efficiently process each advanced query type, customized indexing mechanisms have been developed. However, such mechanisms only perform well for the query types that they were designed for; moreover, they are not readily adaptable to support other query types. In this paper, we propose an interval-at-a-time (IAAT) framework as a first attempt to provide a one-size-fits-all solution to social media retrieval with spatial, temporal and social constraints. In addition, the algorithm relies on inverted index only, which makes it compatible with conventional search engines. The inverted lists are sorted by document id and the insertion is very fast because only append operation is involved. Experiments conducted on two large-scale Twitter datasets show that though IAAT is a unified strategy, it performs better than most of the state-of-the-art customized solutions in a variety of query types.

Guest Editorial: Big Media Data: Understanding, Search, and Mining

The paper in this special section focus on big data in the media industry. Big media data is a new research area, and has been attracting a lot of research interests in both industry and academia. This editorial, as the fourth part of this special issue, introduces four papers on fine-grained image classification, object retrieval, and applications of social media.