Foundations and Trends® in Computer Graphics and Vision

This review study looks at two important areas of recent work: improvements in computer vision algorithms and the incorporation of computer graphics into computer vision systems. We explore the development of computer vision in Part 1, highlighting its many uses in various fields as UAV image processing and automobile navigation. Even while real-time processing is essential in many situations, classic local algorithms which are highly valued for their speed often sacrifice image quality in favour of global algorithms. But recent work reveals subtle modifications to matching computations and data gathering techniques that support local algorithms, leading to performance that approaches that of global algorithms with respect to matching rate. In Part 2, an innovative method for combining computer vision and graphics is shown for creating a new telepresence and collaboration platform. Three key features of this system are its smooth integration of the physical and virtual worlds for input and output, its ability to enable remote collaboration between users, and its ability to facilitate interaction between different 3D graphics programs. This system is designed for future high-bandwidth networks and sends real-time fusions of dynamic computer graphics and vision data. This paper highlights the visual dimension and the mutually beneficial relationship between computer graphics and vision for telepresence and collaborative applications. Preliminary studies show promising possibilities for this technology to create immersive spaces suitable for various cooperative tasks across high-bandwidth networks. Our goal in doing this thorough assessment is to clarify the course of these

This work takes a look at computer vision and graphics as interdisciplinary, multidisciplinary and transdisciplinary fields. Computer vision and graphics belong to the wide area of imaging but acquired their scientific autonomy with their own needs, goals, and methods. These two have brought many methodological contributions in interaction with physics, applied mathematics, etc. New concepts and paradigms were established and partly solved. They belong to the core of the two fields. They also generated a lot of important applications with societal impacts and, in turn, these targets led to new theoretical issues. This way, they reinforced their place as autonomous disciplines and investigated new research spaces. Computer vision and computer graphics have influenced the development of new sensing/actuating technologies as well. Therefore, their identification as such has been beneficial over the last decades, and trying to unify them very likely would be counterproductive in the short term for the many applications they address. This work discusses a key area where major examples can be found - medical applications.

The paper demonstrates how discrete exterior calculus (DEC) tools may be useful in computer vision and graphics. A variational approach provides a link with mechanics. Our development of DEC includes discrete differential forms, discrete vector fields and the operators acting on these. This development of a discrete calculus, when combined with the methods of discrete mechanics and other recent work is likely to have promising applications in a field like computer vision which offers such a rich variety of challenging variational problems to be solved computationally. As a specific example we consider the problem of template matching and show how numerical methods derived from a discrete exterior calculus are starting to play an important role in solving the equations of averaged template matching. We also show some example applications using variational problems from computer graphics and mechanics to demonstrate that formulating the problem discretely and using discrete methods for solution can lead to efficient algorithms.

A human being has a remarkable ability to make 3D connection on seeing 2D images. For example, although transparent objects are invisible to a large extent, we can mentally infer the 3D solid almost instantly and effortlessly. This ability, however, sometimes leads to interesting problems. M. C. Escher was a renowned artist who specialized in embedding impossible figures into architectural drawings. Impossible figure is a special kind of drawing consisting of multiple geometrically possible units connected by linear structures. When the figure is viewed as a whole, structural inconsistencies arise and confuse our visual perception. The theme of this thesis consists of an appearance-based computational framework for modeling the invisibles and the impossibles, that is transparent objects and impossible figures. We applied our appearance-based model in computer graphics, allowing for the first time the rendering of transparent objects in a new background scene, as well as the rendering of impossible figures at high frame rates, both without any tedious 3D modeling. The key in our approach lies in bringing the user into the modeling loop, i.e., human-computer interaction (HCI). Human-computer interaction, which can be understood as the supplying of a small amount of hints to help automatic computer algorithm to solve difficult problems, has recently gained a lot of attention in research in computer vision and interactive techniques. The main issue lies in improving computer's performance by user interaction using a simple interface. This thesis presents a HCI approach to model and render transparent objects (the invisibles) and impossible figures (the impossibles) which are traditionally very difficult in computer vision and graphics. In this thesis, we exploit our remarkable human visual system to provide prior knowledge, and propose an HCI approach to transfer such prior information in the form of a few simple hints via an easy and intuitive user interface. The computer algorithm then automatically performs the rest of the processing. We first transform the problem of transparent layer extraction into a soft-segmentation problem, where simple user's hints are available in the form of rough strokes drawn on the image. Then, with the aid of a human user who can easily recognize transparent objects given a single photo, we derive a practical and interactive approach to solve the problem of matting and compositing of transparent and refractive objects. Traditionally, these tasks can only be achieved using multiple images or 3D models. Finally, we present two approaches to model and render impossible figures, which is a long-standing problem in computer vision and computer graphics.

Image processing is concerned with the analysis and manipulation of images by computer. Providing a thorough treatment of image processing, with an emphasis on those aspects most used in computer graphics and vision, this fully revised second edition concentrates on describing and analyzing the underlying concepts of this subject. As befits a modern introduction to this topic, a good balance is struck between discussing the underlying mathematics and the main topics of signal processing, data discretization, the theory of color and different color systems, operations in images, dithering and half-toning, warping and morphing, and image processing. Significantly expanded and revised, this easy-to-follow text/reference reflects recent trends in science and technology that exploit image processing in computer graphics and vision applications. Stochastic image models and statistical methods for image processing are covered, as is probability theory for image processing, and a focus on applications in image analysis and computer vision. Features: Includes 5 new chapters and major changes throughout Adopts a conceptual approach with emphasis on the mathematical concepts and their applications Introduces an abstraction paradigm that relates mathematical models with image processing techniques and implementation methods - used throughout to help understanding of the mathematical theory and its practical use Motivates through an elementary presentation, opting for an intuitive description where needed Contains adopted innovative formulations whenever necessary for clarity of exposition Provides numerous examples and illustrations, as an aid to understanding Focuses on the aspects of image processing that have importance in computer graphics and vision applications Offers a comprehensive introductory chapter for instructors This comprehensive text imparts a good conceptual understanding of the topic, as a basis for further study, and is suitable both as a textbook and a professional reference. The current extended edition is a must-have resource and guide for all studying or interested in this field.

Efficient rendering of photo‐realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo‐realistic images from hand‐crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo‐realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state‐of‐the‐art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photorealistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. Specifically, our emphasis is on the type of control, i.e., how the control is provided, which parts of the pipeline are learned, explicit vs. implicit control, generalization, and stochastic vs. deterministic synthesis. The second half of this state‐of‐the‐art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free‐viewpoint video, and the creation of photo‐realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.

Deep neural networks have been part of many breakthroughs in computer graphics and vision research. In the context of visual content synthesis, deep learning models have achieved impressive performance in the image domain. However, adapting the successes of image synthesis models to the video domain has been difficult, arguably due to the lack of sufficiently strong inductive biases that encourage the models to capture the temporal-dynamic nature of video data. Inductive bias refers to the prior knowledge incorporated into the learning models to explicitly drive the learning process toward the solutions that capture meaningful structures from data, which is critical to help the model generalize beyond the training data. Successful deep neural network architectures, such as convolutional neural networks (CNN), while effective in representing image data thanks to the spatial inductive bias, often lack the inductive biases relating to the dynamic nature of videos. Mai argues that designing such inductive biases can benefit from the domain knowledge of video processing literature. Their primary motivation in this thesis is to demonstrate that the knowledge acquired from traditional computer vision and graphics literature can serve as effective inductive biases for designing deep learning models for video synthesis. This dissertation provides the initial steps toward verifying that insight via two case studies. In the first case study, Mai explored adapting the standard CNN architecture to perform video frame interpolation. Early CNN-based methods for frame generation followed the direct prediction approach, thus ineffective in learning to capture motion information. Inspired by traditional video frame interpolation techniques that established frame interpolation as a joint process of motion estimation and pixel re-sampling, Mai presented the CNN-based frame interpolation framework that incorporated such insight into the synthesis model via the novel AdaConv layer. That serves as a functional inductive bias and enables the first deep learning model for high-quality video frame interpolation. In the second case study, Mai explored adapting the recent Implicit Neural Representation (INR) to a novel motion-adjustable video representation. Viewing modern INR frameworks as a form of non-linear transform from a frequency domain to the image domain, and inspired by the success of phase-based motion modelling in the classical computer vision literature, they presented a simple modification to the standard image-based INR model that allows for not only video reconstruction but also a variety of motion editing tasks.

Optimal transport is a long‐standing theory that has been studied in depth from both theoretical and numerical point of views. Starting from the 50s this theory has also found a lot of applications in operational research. Over the last 30 years it has spread to computer vision and computer graphics and is now becoming hard to ignore. Still, its mathematical complexity can make it difficult to comprehend, and as such, computer vision and computer graphics researchers may find it hard to follow recent developments in their field related to optimal transport. This survey first briefly introduces the theory of optimal transport in layman's terms as well as most common numerical techniques to solve it. More importantly, it presents applications of these numerical techniques to solve various computer graphics and vision related problems. This involves applications ranging from image processing, geometry processing, rendering, fluid simulation, to computational optics, and many more. It is aimed at computer graphics researchers desiring to follow optimal transport research in their field as well as optimal transport researchers willing to find applications for their numerical algorithms.

Computer vision and image processing are rapidly evolving fields with broad applications across numerous domains, including healthcare, autonomous driving, surveillance, and entertainment. These fields have transformed from simple data recording techniques into sophisticated systems that incorporate digital image processing, pattern recognition, machine learning, and computer graphics. This evolution has prompted interdisciplinary interest, pushed the technology’s boundaries and expanded its practical uses. This paper offers a comprehensive survey of recent advancements in computer vision, focusing on image processing and its applications across various fields. It delves into the theoretical foundations and technologies that make computer vision a valuable tool for interpreting images and videos, extracting relevant information, recognizing patterns, and understanding events. The ability of computer vision to analyze large datasets across multiple application domains makes it instrumental in tasks such as object identification, facial recognition, scene understanding, and even real-time action prediction. This versatility has established computer vision as a key driver of data-driven insights in both scientific and commercial sectors. The study categorizes computer vision into four main areas: image processing, object recognition, machine learning, and computer graphics. Each of these categories is essential to the functionality of modern computer vision systems. Image processing involves techniques for enhancing image quality and extracting important features. Object recognition and machine learning enable the identification of specific elements within images and allow systems to learn from large datasets, enhancing accuracy over time. Computer graphics, on the other hand, aid in visualizing and interpreting processed data. By offering insights into the latest techniques and evaluating their performance, this survey highlights the current state of computer vision while shedding light on future trends. Computer vision’s expanding utility across various fields underscores its critical role in driving interdisciplinary innovation and addressing complex challenges.

In one abstract view, computer graphics deals with building computer models or representations and then displaying them by some method or theory to produce a high-quality image. Computer vision is a dual to computer graphics. It starts with an image or animation and deduces the model representation for the computer. Computer graphics goes down from model to image, and computer vision goes up from image to model. This is, of course, a simplification, but it serves as a good basis to understand recent developments intersecting the two fields. Computer graphics and computer vision are truly complementary disciplines quickly approaching convergence. The broad study of computer-based imagery extends beyond these two fields to include the areas of human-computer interaction, visualization and image processing. Ongoing research and development will continue to forge this bond, and we'll begin to see real-world products emerge from these efforts. Then we will see the fruits of this convergence.

List of Figures. List of Tables. Preface. Contributing Authors. Introduction A. Leonardis, et al. 1. From images to virtual and augmented reality A. Zisserman, et al. 2. Surface reconstruction from multiple views using apparent contours and surface testure G. Cross, A. Zisserman. 3. Consistent projective reconstruction from multiple views M. Urban, et al. 4. Accurate natural surface reconstruction from polynocular stereo R. Sara. 5. Building models from sensor data: An application shared by the computer vision and the computer graphics community G. Roth. 6. Acquiring range images of objects with non-uniform reflectance using high dynamic scale radiance maps D. Skocaj, A. Leonardis. 7. Dynamic view interpolation without affine reconstruction R.A. Manning, C.R. Dyer. 8. Facial motion capturing using an explanation-based approach H. Tao, Th.S. Huang. 9. Image-based 3D modeling: Modeling from reality L. Van Gool, et al. 10. Computer vision and graphics techniques for modeling dressed humans N. Jojica, T.S. Huang. 11. Urban site models: Accurate, detailed, rapid and inexpensive F.W. Leberl, et al. 12. Medical visualisation, biomechanics, figure animation and robot teleoperation: Themes and links G.J. Clapworthy, et al. 13. Can virtual look real? A review of virtual studio techniques A. Wojdala. 14. Real-time 3D-teleimmersion K. Daniilidis, et al. 15. Augmented reality: A problem in need of many computer vision-based solutions G. Klinker. 16. Registration methods for harmoniousintegration of real world and computer generated objects G. Simon, et al. 17. 3D object tracking using analysis/synthesis techniques A. Gagalowicz, P. Gerard. 18. Augmented reality by integrating multiple sensory modalities for underwater scene understanding V. Murino, A. Fusiello. Index.

The Internet has become an unprecedented source of visual information about our world, with millions of people uploading photos and videos to media-sharing sites at staggering rates. Virtually all of the world's famous landmarks and cities (and many not-so-famous ones) have been photographed hundreds of thousands or millions of times, and billions of these photos can be found on photo-sharing websites. This richness and variety make such Internet photo collections extremely attractive as a source of data for applications ranging from mapping and visualization to social science. However, a prerequisite to many such applications is recovering structure — often in the form of 3D geometry — from these massive, unorganized collections of imagery. This ever-growing collection of visual data opens up fundamental new questions in computer vision and computer graphics, where traditional techniques designed for small, controlled sets of images cannot be readily applied. This article surveys recent work on applying geometric computer vision to large, unstructured photo collections, as well as applications enabled by these new techniques in scene visualization, location recognition, image editing, and other areas of computer vision and graphics.

Computer graphics is a rapidly expanding field of science and technology, and education in this area is increasingly important. Development in computer graphics and the requirements of different applications have led to an explosion of subdisciplines.These share many fundamental principles, but each has a specialized orientation, with its own techniques and approaches to computer graphics. The basic disciplines, such as geometric modeling of single objects and 2D and 3D scenes, need efficient algorithms that involve computational geometry and knowledge of parallel computation. Other disciplines are oriented toward realistic visualization, and the enhancement of human/machine interaction leads to developing fields such as user interfaces, interactive modeling, multimedia and hypermedia and visual languages. A strong link is also required with related disciplines such as computer vision and image processing.Many of these subjects integrate knowledge derived from different sciences such as mathematics, physics, computer science and engineering, into a variety of applications. In Italy, computer graphics courses are taught in a number of curricula of traditional disciplines such as computer science, mathematics, physics, architecture and engineering. The choice was not to establish specific departments on computer graphics, but to give each discipline an overall sense of computer graphics as a subject into which they can fit their own specialities.This has led a number of universities to propose courses in computer graphics at both fundamental and advanced levels, especially in scientific departments. The variety of topics in computer graphics is so wide that they cannot be easily compressed in one course or a series of courses, but must be split according to the application requirements. A typical feature of computer graphics courses is that they are not usually intended for a single department's students, but they are of broad interest.

Computer Vision, Imaging and Computer Graphics - Theory and Applications : International Joint Conference, VISIGRAPP 2013, Barcelona, Spain, February 21-24, 2013, Revised Selected Papers

Computer graphics “Made in Germany”: Darmstadt, the leading “Computer Graphics and Visual Computing Hub” in Europe: The way from 1975 to 2014