Publications

2010
Physical Reproduction of Materials with Specified Subsurface Scattering
Hasan M, Fuchs M, Matusik W, Pfister H, Rusinkiewicz S. Physical Reproduction of Materials with Specified Subsurface Scattering. ACM Transactions on Graphics 2010;3Abstract
Physical Reproduction of Materials with Specified Subsurface Scattering Authors Milos Hasan; Martin Fuchs; Wojciech Matusik; Hanspeter Pfister; Szymon Rusinkiewicz Abstract We investigate a complete pipeline for measuring, modeling, and fabricating objects with specified subsurface scattering behaviors. The process starts with measuring the scattering properties of a given set of base materials, determining their radial reflection and transmission profiles. We describe a mathematical model that pre- dicts the profiles of different stackings of base materials, at arbi- trary thicknesses. In an inverse process, we can then specify a de- sired reflection profile and compute a layered composite material that best approximates it. Our algorithm efficiently searches the space of possible combinations of base materials, pruning unsat- isfactory states imposed by physical constraints. We validate our process by producing both homogeneous and heterogeneous com- posites fabricated using a multi-material 3D printer. We demon- strate reproductions that have scattering properties approximating complex materials.
Images
Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows
Sunkavalli K, Zickler T, Pfister H. Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows. European Conference on Computer Vision (ECCV) 2010;:251-264 .Abstract
Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows Authors Kalyan Sunkavalli; Todd Zickler; Hanspeter Pfister Abstract Photometric stereo relies on inverting the image formation process, and doing this accurately requires reasoning about the visibility of light sources with respect to each image point. While simple heuristics for shadow detection suffice in some cases, they are susceptible to error. This paper presents an alternative approach for handling visibility in photometric stereo, one that is suitable for uncalibrated settings where the light directions are not known. A surface imaged under a finite set of light sources can be divided into regions having uniform visibility, and when the surface is Lambertian, these regions generally map to distinct three-dimensional illumination subspaces. We show that by identifying these subspaces, we can locate the regions and their visibilities, and in the process identify shadows. The result is an automatic method for uncalibrated Lambertian photometric stereo in the presence of shadows, both cast and attached.
Paper Slides Images
2009
Capture and Modeling of Non-Linear Heterogeneous Soft Tissue
Bickel B, Baecher M, Otaduy M, Matusik W, Pfister H, Gross M. Capture and Modeling of Non-Linear Heterogeneous Soft Tissue. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH) 2009;28(3)Abstract
This paper introduces a data-driven representation and modeling technique for simulating non-linear heterogeneous soft tissue. It simplifies the construction of convincing deformable models by avoiding complex selection and tuning of physical material parameters, yet retaining the richness of non-linear heterogeneous behavior. We acquire a set of example deformations of a real object, and represent each of them as a spatially varying stress-strain relationship in a finite-element model. We then model the material by non-linear interpolation of these stress-strain relationships in strain-space. Our method relies on a simple-to-build capture system and an efficient run-time simulation algorithm based on incremental loading, making it suitable for interactive computer graphics applications. We present the results of our approach for several nonlinear materials and biological soft tissue, with accurate agreement of our model to the measured data.
Paper Video Image Image Image Image Image
Image Restoration using Online Photo Collections
Dale K, Johnson MK, Sunkavalli K, Matusik W, Pfister H. Image Restoration using Online Photo Collections. IEEE International Conference on Computer Vision 2009;Abstract
We present an image restoration method that leverages a large database of images gathered from the web. Given an input image, we execute an efficient visual search to find the closest images in the database; these images define the input’s visual context. We use the visual context as an image-specific prior and show its value in a variety of image restoration operations, including white balance correction, exposure correction, and contrast enhancement. We evaluate our approach using a database of 1 million images downloaded from Flickr and demonstrate the effect of database size on performance. Our results show that priors based on the visual context consistently out-perform generic or even domain-specific priors for these operations.
Paper Image
Multiphase Geometric Couplings for the Segmentation of Neural Processes
Reina VA, Miller E, Pfister H. Multiphase Geometric Couplings for the Segmentation of Neural Processes. IEEE Conference on Computer Vision and Pattern Recognition 2009;:2020-2027.Abstract
The ability to constrain the geometry of deformable models for image segmentation can be useful when information about the expected shape or positioning of the objects in a scene is known a priori. An example of this occurs when segmenting neural cross sections in electron microscopy. Such images often contain multiple nested boundaries separating regions of homogeneous intensities. For these applications, multiphase level sets provide a partitioning framework that allows for the segmentation of multiple deformable objects by combining several level set functions. Although there has been much effort in the study of statistical shape priors that can be used to constrain the geometry of each partition, none of these methods allow for the direct modeling of geometric arrangements of partitions. In this paper, we show how to define elastic couplings between multiple level set functions to model ribbon-like partitions. We build such couplings using dynamic force fields that can depend on the image content and relative location and shape of the level set functions. To the best of our knowledge, this is the first work that shows a direct way of geometrically constraining multiphase level sets for image segmentation. We demonstrate the robustness of our method by comparing it with previous level set segmentation methods.
Paper Video Image
Scalable and Interactive Segmentation and Visualization of Neural Processes in EM Datasets
Jeong W-K, Beyer J, Hadwiger M, Vazquez A, Pfister H, Whitaker RT. Scalable and Interactive Segmentation and Visualization of Neural Processes in EM Datasets. IEEE Transactions on Visualization and Computer Graphics 2009;15:1505-1514.Abstract
Recent advances in scanning technology provide high resolution EM (Electron Microscopy) datasets that allow neuroscientists to reconstruct complex neural connections in a nervous system. However, due to the enormous size and complexity of the resulting data, segmentation and visualization of neural processes in EM data is usually a difficult and very time-consuming task. In this paper, we present NeuroTrace, a novel EM volume segmentation and visualization system that consists of two parts: a semi-automatic multiphase level set segmentation with 3D tracking for reconstruction of neural processes, and a specialized volume rendering approach for visualization of EM volumes. It employs view-dependent on-demand filtering and evaluation of a local histogram edge metric, as well as on-the-fly interpolation and ray-casting of implicit surfaces for segmented neural structures. Both methods are implemented on the GPU for interactive performance. NeuroTrace is designed to be scalable to large datasets and data-parallel hardware architectures. A comparison of NeuroTrace with a commonly used manual EM segmentation tool shows that our interactive workflow is faster and easier to use for the reconstruction of complex neural processes.
Paper Video Image
2008
Particle-based Sampling and Meshing of Surfaces in Multimaterial Volumes
Meyer M, Whitaker R, Kirby RM, Ledergerber C, Pfister H. Particle-based Sampling and Meshing of Surfaces in Multimaterial Volumes. IEEE Transactions on Visualization and Computer Graphics 2008;14(6):1372-1379.Abstract
Methods that faithfully and robustly capture the geometry of complex material interfaces in labeled volume data are important for generating realistic and accurate visualizations and simulations of real-world objects. The generation of such multimaterial models from measured data poses two unique challenges: first, the surfaces must be well-sampled with regular, efficient tessellations that are consistent across material boundaries; and second, the resulting meshes must respect the nonmanifold geometry of the multimaterial interfaces. This paper proposes a strategy for sampling and meshing multimaterial volumes using dynamic particle systems, including a novel, differentiable representation of the material junctions that allows the par ticle system to explicitly sample corners, edges, and surfaces of material intersections. The distributions of particles are controlled by fundamental sampling constraints, allowing Delaunay-based meshing algorithms to reliably extract water tight meshes of consistently high-quality.
Paper Image
Real-Time Face Pose Estimation from Single Range Images
Breitenstein MD, Kuettel D, Weise T, van Gool L, Pfister H. Real-Time Face Pose Estimation from Single Range Images. In: IEEE Conference on Computer Vision and Pattern Recognition. 2008 p. 1-8.Abstract
We present a real-time algorithm to estimate the 3D pose of a previously unseen face from a single range image. Based on a novel shape signature to identify noses in range images, we generate candidates for their positions, and then generate and evaluate many pose hypotheses in parallel using modern graphics processing units (GPUs). We developed a novel error function that compares the input range image to precomputed pose images of an average face model. The algorithm is robust to large pose variations of plusmn90deg yaw, plusmn45deg pitch and plusmn30deg roll rotation, facial expression, partial occlusion, and works for multiple faces in the field of view. It correctly estimates 97.8% of the poses within yaw and pitch error of 15deg at 55.8 fps. To evaluate the algorithm, we built a database of range images with large pose variations and developed a method for automatic ground truth annotation.
Paper Video Image
Volume MLS Ray Casting
Ledergerber C, Guennebaud G, Meyer M, Bacher M, Pfister H. Volume MLS Ray Casting. IEEE Transactions on Visualization and Computer Graphics 2008;14(6):1539-1546.Abstract
The method of Moving Least Squares (MLS) is a popular framework for reconstructing continuous functions from scattered data due to its rich mathematical proper ties and well-understood theoretical foundations. This paper applies MLS to volume rendering, providing a unified mathematical framework for ray casting of scalar data stored over regular as well as irregular grids. We use the MLS reconstruction to render smooth isosurfaces and to compute accurate derivatives for high-quality shading effects. We also present a novel, adaptive preintegration scheme to improve the efficiency of the ray casting algorithm by reducing the overall number of function evaluations, and an efficient implementation of our framework exploiting modern graphics hardware. The resulting system enables high-quality volume integration and shaded isosurface rendering for regular and irregular volume data.
Paper Image
What do color changes reveal about an outdoor scene?
Sunkavalli K, Romeiro F, Matusik W, Zickler T, Pfister H. What do color changes reveal about an outdoor scene?. In: IEEE Conference on Computer Vision and Pattern Recognition. 2008Abstract
In an extended image sequence of an outdoor scene, one observes changes in color induced by variations in the spectral composition of daylight. This paper proposes a model for these temporal color changes and explores its use for the analysis of outdoor scenes from time-lapse video data. We show that the time-varying changes in direct sunlight and ambient skylight can be recovered with this model, and that an image sequence can be decomposed into two corresponding components. The decomposition provides access to both radiometric and geometric information about a scene, and we demonstrate how this can be exploited for a variety of visual tasks, including color-constancy, background subtraction, shadow detection, scene reconstruction, and camera geo-location.
Paper Video Slides Image
Multidimensional Adaptive Sampling and Reconstruction for Ray Tracing
Hachisuka T, Jarosz W, Weistroffer RP, Dale K, Humphreys G, Zwicker M, Jensen HW. Multidimensional Adaptive Sampling and Reconstruction for Ray Tracing. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 2008;27Abstract
We present a new adaptive sampling strategy for ray tracing. Our technique is specifically designed to handle multidimensional sample domains, and it is well suited for efficiently generating images with effects such as soft shadows, motion blur, and depth of field. These effects are problematic for existing image based adaptive sampling techniques as they operate on pixels, which are possibly noisy results of a Monte Carlo ray tracing process. Our sampling technique operates on samples in the multidimensional space given by the rendering equation and as a consequence the value of each sample is noise-free. Our algorithm consists of two passes. In the first pass we adaptively generate samples in the multidimensional space, focusing on regions where the local contrast between samples is high. In the second pass we reconstruct the image by integrating the multidimensional function along all but the image dimensions. We perform a high quality anisotropic reconstruction by determining the extent of each sample in the multidimensional space using a structure tensor. We demonstrate our method on scenes with a 3 to 5 dimensional space, including soft shadows, motion blur, and depth of field. The results show that our method uses fewer samples than Mitchell’s adaptive sampling technique while producing images with less noise.
Paper Image
2007
Automatic Pose Estimation for Range Images on the GPU
Germann M, Breitenstein M, Park I-K, and Pfister H. Automatic Pose Estimation for Range Images on the GPU. In: 3-D Digital Imaging and Modeling (3DIM). Sixth International Conference on. Montreal, QC: 2007 p. 81-90.Abstract
Object pose (location and orientation) estimation is a common task in many computer vision applications. Although many methods exist, most algorithms need manual initialization and lack robustness to illumination variation, appearance change, and partial occlusions. We propose a fast method for automatic pose estimation without manual initialization based on shape matching of a 3D model to a range image of the scene. We developed a new error function to compare the input range image to pre-computed range maps of the 3D model. We use the tremendous data- parallel processing performance of modern graphics hardware to evaluate and minimize the error function on many range images in parallel. Our algorithm is simple and accurately estimates the pose of partially occluded objects in cluttered scenes in about one second.
Paper
Display Pre-Filtering for Multi-View Video Compression
Zwicker M, Yea S, Vetro A, Forlines C, Matusik W, Pfister H. Display Pre-Filtering for Multi-View Video Compression. In: Proceedings of the 15th international conference on Multimedia. ACM; 2007 p. 1046-1053.Abstract
Multi-view 3D displays are preferable to other stereoscopic display technologies because they provide autostereoscopic viewing from any viewpoint without special glasses. However, they require a large number of pixels to achieve high image quality. Therefore, data compression is a major issue for this approach. In this paper, we present a framework for efficient compression of multi-view video streams for multi-view 3D displays. Our goal is to optimize image quality without increasing the required data bandwidth. We achieve this by taking into account a precise notion of the multi-dimensional display bandwidth. The display bandwidth implies that scene elements that appear at a given distance from the display become increasingly blurry as the distance grows. Our main contribution is to enhance conventional multi-view compression pipelines with an additional pre-filtering step that bandlimits the multi-view signal to the display bandwidth. This imposes a shallow depth of field on the input images, thereby removing high frequency content. We show that this pre-filtering step leads to increased image quality compared to state-of-the-art multi-view coding at equal bitrate. We present results of an extensive user study that corroborate the benefits of our approach. Our work suggests that display pre-filtering will be a fundamental component in signal processing for 3D displays, and that any multi-view compression scheme will benefit from our pre-filtering technique.
Paper
Exploring Defocus Matting: Non-Parametric Acceleration, Super-Resolution, and Off-Center Matting
Joshi N, Matusik W, Avidan S, Pfister H, Freeman WT. Exploring Defocus Matting: Non-Parametric Acceleration, Super-Resolution, and Off-Center Matting. IEEE Computer Graphics & Applications, Special Issue on Computational Photography 2007;27:43-52.Abstract
Defocus matting is a fully automatic and passive method for pulling mattes from video captured with coaxial cameras that have different depths of field and planes of focus. Nonparametric sampling can accelerate the video-matting process from minutes to seconds per frame. In addition, a super-resolution technique efficiently bridges the gap between mattes from high-resolution video cameras and those from low-resolution cameras. Off-center matting pulls mattes for an external high-resolution camera that doesn't share the same center of projection as the low-resolution cameras used to capture the defocus matting data. In this article, we address these limitations and extend defocus matting in several important ways
Paper
Factored Time-Lapse Video
Sunkavalli K, Matusik W, Pfister H, Rusinkiewicz S. Factored Time-Lapse Video. In: ACM Trans. on Graphics (ACM SIGGRAPH 2007). ACM; 2007 p. 101-111.Abstract
We describe a method for converting time-lapse photography captured with outdoor cameras into Factored Time-Lapse Video (FTLV): a video in which time appears to move faster (i.e., lapsing) and where data at each pixel has been factored into shadow, illumination, and reflectance components. The factorization allows a user to easily relight the scene, recover a portion of the scene geometry (normals), and to perform advanced image editing operations. Our method is easy to implement, robust, and provides a compact representation with good reconstruction characteristics. We show results using several publicly available time-lapse sequences.
Paper
Multi-Scale Capture of Facial Geometry and Motion
Bickel B, Botsch M, R. Angst W, Pfister H, Gross M. Multi-Scale Capture of Facial Geometry and Motion. In: Transactions on Graphics (ACM SIGGRAPH 2007). ACM; 2007 p. 33-41.Abstract
We present a novel multi-scale representation and acquisition method for the animation of high-resolution facial geometry and wrinkles. We first acquire a static scan of the face including reflectance data at the highest possible quality. We then augment a traditional marker-based facial motion-capture system by two synchronized video cameras to track expression wrinkles. The resulting model consists of high-resolution geometry, motion-capture data, and expression wrinkles in 2D parametric form. This combination represents the facial shape and its salient features at multiple scales. During motion synthesis the motion-capture data deforms the high-resolution geometry using a linear shell-based mesh-deformation method. The wrinkle geometry is added to the facial base mesh using nonlinear energy optimization. We present the results of our approach for performance replay as well as for wrinkle editing.
Paper
Optical-Splitting Trees for High-Precision Monocular Imaging
McGuire M, Matusik W, Pfister H, Chen B, Hughes J, Nayar S. Optical-Splitting Trees for High-Precision Monocular Imaging. IEEE Computer Graphics & Applications, Special Issue on Computational Photography 2007;27:32-42.Abstract
In this article, we consider the design of monocular multiview optical systems that form optical splitting trees, where the optical path topology takes the shape of a tree because of recursive beam splitting. Designing optical splitting trees is challenging when it requires many views with specific spectral properties. We introduce a manual design paradigm for optical splitting trees and a computer-assisted design tool to create efficient splitting-tree cameras. The tool accepts as input a specification for each view and a set of weights describing the user's relative affinity for efficiency, measurement accuracy, and economy. An optimizer then searches for a design that maximizes these weighted priorities. Our tool's output is a splitting-tree design that implements the input specification and an analysis of the efficiency of each root-to-leaf path. Automatically designed trees appear comparable to those designed by hand; we even show some cases where they are superior. With the help of the optimizer, the system demonstrates high dynamic range, focusing, matting, and hybrid imaging implemented on a single, reconfigurable camera containing eight sensors
Paper
Point-Based Graphics
Gross M, H. Pfister (eds.) MK. Point-Based Graphics. Elsevier; 2007.
Small-Scale Reconfigurability for Improved Performance and Double-Precision in Graphics Hardware
Dale K, Sheaffer JW, Kumar VV, Luebke DP, Humphreys G, Skadron K. Small-Scale Reconfigurability for Improved Performance and Double-Precision in Graphics Hardware. International Journal of Electronics 2007;94:549-561.Abstract
We explore the application of small-scale reconfigurability (SSR) to graphics hardware. SSR is an architectural technique wherein functionality common to multiple subunits is reused rather than replicated, yielding high-performance reconfigurable hardware with reduced area requirements (Vijay Kumar and Lach “Designing, scheduling, and allocating flexible arithmetic components”, in Proceedings of the International Conference on Field Programmable Logic and Applications, 2003). We show that SSR can be used effectively in programmable graphics architectures to allow double-precision computation without affecting the performance of single-precision calculations and to increase fragment shader performance with a minimal impact on chip area.
Paper
Statistics of Infrared Images
Morris N, Avidan S, Matusik W, and Pfister H. Statistics of Infrared Images. In: Computer Vision and Pattern Recognition. CVPR '07. IEEE Conference on. Minneapolis, MN: 2007 p. 1-7.Abstract
The proliferation of low-cost infrared cameras gives us a new angle for attacking many unsolved vision problems by leveraging a larger range of the electromagnetic spectrum. A first step to utilizing these images is to explore the statistics of infrared images and compare them to the corresponding statistics in the visible spectrum. In this paper, we analyze the power spectra as well as the marginal and joint wavelet coefficient distributions of datasets of indoor and outdoor images. We note that infrared images have noticeably less texture indoors where temperatures are more homogenous. The joint wavelet statistics also show strong correlation between object boundaries in IR and visible images, leading to high potential for vision applications using a combined statistical model.
Paper

Pages