Publications

2016
J. Pan, D. Sun, M. - H. Yang, and H. Pfister, “Blind Image Deblurring Using Dark Channel Prior,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, 2016.Abstract
Abstract
We present a simple and effective blind image deblur-
ring method based on the dark channel prior. Our work is
inspired by the interesting observation that the dark chan-
nel of blurred images is less sparse. While most image
patches in the clean image contain some dark pixels, these
pixels are not dark when averaged with neighboring high-
intensity pixels during the blur process. This change in the
sparsity of the dark channel is an inherent property of the
blur process, which we both prove mathematically and val-
idate using training data. Therefore, enforcing the sparsity
of the dark channel helps blind deblurring on various sce-
narios, including natural, face, text, and low-illumination
images. However, sparsity of the dark channel introduces
a non-convex non-linear optimization problem. We intro-
duce a linear approximation of the
min
operator to com-
pute the dark channel. Our look-up-table-based method
converges fast in practice and can be directly extended to
non-uniform deblurring. Extensive experiments show that
our method achieves state-of-the-art results on deblurring
natural images and compares favorably methods that are
well-engineered for specific scenarios.
M. Piovarči, et al., “An Interaction-Aware, Perceptual Model For Non-Linear Elastic Objects,” ACM Transactions on Graphics 35(4) (Proc. SIGGRAPH 2016, Anaheim, California, USA). 2016.Abstract

Everyone, from a shopper buying shoes to a doctor palpating a growth, uses their sense of touch to learn about the world. 3D printing is a powerful technology because it gives us the ability to control the haptic impression an object creates. This is critical for both replicating existing, real-world constructs and designing novel ones. However, each 3D printer has different capabilities and supports different materials, leaving us to ask: How can we best replicate a given haptic result on a particular output device? In this work, we address the problem of mapping a real-world material to its nearest 3D printable counterpart by constructing a perceptual model for the compliance of nonlinearly elastic objects. We begin by building a perceptual space from experimentally obtained user comparisons of twelve 3D-printed metamaterials. By comparing this space to a number of hypothetical computational models, we identify those that can be used to accurately and efficiently evaluate human-perceived differences in nonlinear stiffness. Furthermore, we demonstrate how such models can be applied to complex geometries in an interaction-aware way where the compliance is influenced not only by the material properties from which the object is made but also its geometry. We demonstrate several applications of our method in the context of fabrication and evaluate them in a series of user experiments.

2015
VESICLE: Volumetric Evaluation of Synaptic Interfaces using Computer Vision at Large Scale
W. G. Roncal, et al., “VESICLE: Volumetric Evaluation of Synaptic Interfaces using Computer Vision at Large Scale,” in Proceedings of the British Machine Vision Conference (BMVC), 2015, pp. 81.1-81.13. Publisher's VersionAbstract


An open challenge at the forefront of modern neuroscience is to obtain a comprehensive mapping of the neural pathways that underlie human brain function; an enhanced understanding of the wiring diagram of the brain promises to lead to new breakthroughs in diagnosing and treating neurological disorders. Inferring brain structure from image data, such as that obtained via electron microscopy (EM), entails solving the problem of identifying biological structures in large data volumes. Synapses, which are a key communication structure in the brain, are particularly difficult to detect due to their small size and limited contrast. Prior work in automated synapse detection has relied upon time-intensive, error-prone biological preparations (isotropic slicing, post-staining) in order to simplify the problem. This paper presents VESICLE, the first known approach designed for mammalian synapse detection in anisotropic, non-poststained data. Our methods explicitly leverage biological context, and the results exceed existing synapse detection methods in terms of accuracy and scalability. We provide two different approaches - a deep learning classifier (VESICLE-CNN) and a lightweight Random Forest approach (VESICLE-RF), to offer alternatives in the performance-scalability space. Addressing this synapse detection challenge enables the analysis of high-throughput imaging that is soon expected to produce petabytes of data, and provides tools for more rapid estimation of brain-graphs. Finally, to facilitate community efforts, we developed tools for large-scale object detection, and demonstrated this framework to find ~50,000 synapses in 60,000 um^3 (220 GB on disk) of electron microscopy data.

PDF icon Paper.pdf
A Crowdsourced Alternative to Eye-tracking for Visualization Understanding
N. W. Kim, Z. Bylinskii, M. A. Borkin, A. Oliva, K. Z. Gajos, and H. Pfister, “A Crowdsourced Alternative to Eye-tracking for Visualization Understanding,” in CHI’15 Extended Abstracts, Seoul, Korea, 2015, pp. 1349-1354. Publisher's VersionAbstract

In this study we investigate the utility of using mouse clicks as an alternative for eye fixations in the context of understanding data visualizations. We developed a crowdsourced study online in which participants were presented with a series of images containing graphs and diagrams and asked to describe them. Each image was blurred so that the participant needed to click to reveal bubbles - small, circular areas of the image at normal resolution. This is similar to having a confined area of focus like the human eye fovea. We compared the bubble click data with the fixation data from a complementary eye-tracking experiment by calculating the similarity between the resulting heatmaps. A high similarity score suggests that our methodology may be a viable crowdsourced alternative to eye-tracking experiments, especially when little to no eye-tracking data is available. This methodology can also be used to complement eye-tracking studies with an additional behavioral measurement, since it is specifically designed to measure which information people consciously choose to examine for understanding visualizations.

PDF icon Paper
State-of-the-Art in GPU-Based Large-Scale Volume Visualization
J. Beyer, M. Hadwiger, and H. Pfister, “State-of-the-Art in GPU-Based Large-Scale Volume Visualization,” Computer Graphics Forum, 2015. Publisher's VersionAbstract

This survey gives an overview of the current state of the art in GPU techniques for interactive large-scale volume visualization. Modern techniques in this field have brought about a sea change in how interactive visualization and analysis of giga-, tera-, and petabytes of volume data can be enabled on GPUs. In addition to combining the parallel processing power of GPUs with out-of-core methods and data streaming, a major enabler for interactivity is making both the computational and the visualization effort proportional to the amount and resolution of data that is actually visible on screen, i.e., “output-sensitive” algorithms and system designs. This leads to recent output- sensitive approaches that are “ray-guided,” “visualization-driven,” or “display-aware.” In this survey, we focus on these characteristics and propose a new categorization of GPU-based large-scale volume visualization techniques based on the notions of actual output-resolution visibility and the current working set of volume bricks—the current subset of data that is minimally required to produce an output image of the desired display resolution. Furthermore, we discuss the differences and similarities of different rendering and data traversal strategies in volume rendering by putting them into a common context—the notion of address translation. For our purposes here, we view parallel (distributed) visualization using clusters as an orthogonal set of techniques that we do not discuss in detail but that can be used in conjunction with what we discuss in this survey.

PDF icon Paper
Large-Scale Automatic Reconstruction of Neuronal Processes from Electron Microscopy Images
V. Kaynig, et al., “Large-Scale Automatic Reconstruction of Neuronal Processes from Electron Microscopy Images,” Medical Image Analysis, vol. 22, no. 1, pp. 77-88, 2015.Abstract

Automated sample preparation and electron microscopy enables acquisition of very large image data sets. These technical advances are of special importance to the field of neuroanatomy, as 3D reconstructions of neuronal processes at the nm scale can provide new
insight into the fine grained structure of the brain. Segmentation of large-scale electron microscopy data is the main bottleneck in the analysis of these data sets. In this paper we present a pipeline that provides state-of-the art reconstruction performance while scaling
to data sets in the GB-TB range. First, we train a random forest classifier on interactive sparse user annotations. The classifier output is combined with an anisotropic smoothing prior in a Conditional Random Field framework to generate multiple segmentation
hypotheses per image. These segmentations are then combined into geometrically consistent 3D objects by segmentation fusion. We provide qualitative and quantitative evaluation of the automatic segmentation and demonstrate large-scale 3D reconstructions
of neuronal processes from a 27; 000 m3 volume of brain tissue over a cube of 30 m in each dimension corresponding to 1,000 consecutive image sections. We also introduce Mojo, a proofreading tool including semi-automated correction of merge errors
based on sparse user scribbles.

PDF icon Paper
Computational Design of Walking Automata
G. Bharaj, S. Coros, B. Thomaszewski, J. Tompkin, B. Bickel, and H. Pfister, “Computational Design of Walking Automata,” in ACM SIGGRAPH / Eurographics Symposium on Computer Animation, 2015. Publisher's VersionAbstract

Creating mechanical automata that can walk in stable and pleasing manners is a challenging task that requires both skill and expertise. We propose to use computational design to offset the technical difficulties of this process. A simple drag-and-drop interface allows casual users to create personalized walking toys from a library of pre-defined template mechanisms. Provided with this input, our method leverages physical simulation and evolutionary optimization to refine the mechanical designs such that the resulting toys are able to walk. The optimization process is guided by an intuitive set of objectives that measure the quality of the walking motions. We demonstrate our approach on a set of simulated mechanical toys with different numbers of legs and various distinct gaits. Two fabricated prototypes showcase the feasibility of our designs.

PDF icon Paper
Blind Video Temporal Consistency
N. Bonneel, J. Tompkin, K. Sunkavalli, D. Sun, S. Paris, and H. Pfister, “Blind Video Temporal Consistency,” ACM Transactions on Graphics (SIGGRAPH Asia), 2015. Webpage
Computational Design of Metallophone Contact Sounds
G. Bharaj, et al., “Computational Design of Metallophone Contact Sounds,” ACM Transactions on Graphics (SIGGRAPH Asia), 2015. Webpage
Context-guided Diffusion for Label Propagation on Graphs
K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, “Context-guided Diffusion for Label Propagation on Graphs,” International Conference on Computer Vision (ICCV), 2015. Webpage
Generalizing Wave Gestures from Sparse Examples for Real-time Character Control
H. Rhodin, et al., “Generalizing Wave Gestures from Sparse Examples for Real-time Character Control,” ACM Transactions on Graphics (SIGGRAPH Asia), 2015. Webpage
Joint 5D Pen Input for Light Field Displays
J. Tompkin, et al., “Joint 5D Pen Input for Light Field Displays,” ACM User Interface Software and Technology (UIST), 2015. Webpage
Local High-order Regularization on Data Manifolds
K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, “Local High-order Regularization on Data Manifolds,” IEEE Computer Vision and Pattern Recognition (CVPR). 2015. Webpage
Semi-supervised Learning with Explicit Relationship Regularization
K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, “Semi-supervised Learning with Explicit Relationship Regularization,” IEEE Computer Vision and Pattern Recognition (CVPR), 2015. Webpage
Beyond Memorability: Visualization Recognition and Recall
M. Borkin, et al., “Beyond Memorability: Visualization Recognition and Recall,” Visualization and Computer Graphics, IEEE Transactions on, vol. PP, pp. 1-1, 2015. Publisher's VersionAbstract

In this paper we move beyond memorability and investigate how visualizations are recognized and recalled. For this study we labeled a dataset of 393 visualizations and analyzed the eye movements of 33 participants as well as thousands of participantgenerated text descriptions of the visualizations. This allowed us to determine what components of a visualization attract people’s attention, and what information is encoded into memory. Our findings quantitatively support many conventional qualitative design guidelines, including that (1) titles and supporting text should convey the message of a visualization, (2) if used appropriately, pictograms do not interfere with understanding and can improve recognition, and (3) redundancy helps effectively communicate the message. Importantly, we show that visualizations memorable “at-a-glance” are also capable of effectively conveying the message of the visualization. Thus, a memorable visualization is often also an effective one.

PDF icon PaperPDF icon Supplemental MaterialFile VideoPDF icon Slides
NeuroBlocks - Visual Tracking of Segmentation and Proofreading for Large Connectomics Projects
A. K. Al-Awami, et al., “NeuroBlocks - Visual Tracking of Segmentation and Proofreading for Large Connectomics Projects,” IEEE Transactions on Visualization and Computer Graphics, vol. to appear, 2015.Abstract

In the field of connectomics, neuroscientists acquire electron microscopy volumes at nanometer resolution in order to reconstruct a detailed wiring diagram of the neurons in the brain. The resulting image volumes, which often are hundreds of terabytes in size, need to be segmented to identify cell boundaries, synapses, and important cell organelles. However, the segmentation process of a single volume is very complex, time-intensive, and usually performed using a diverse set of tools and many users. To tackle the associated challenges, this paper presents NeuroBlocks, which is a novel visualization system for tracking the state, progress, and evolution of very large volumetric segmentation data in neuroscience. NeuroBlocks is a multi-user web-based application that seamlessly integrates the diverse set of tools that neuroscientists currently use for manual and semi-automatic segmentation, proofreading, visualization, and analysis. NeuroBlocks is the first system that integrates this heterogeneous tool set, providing crucial support for the management, provenance, accountability, and auditing of large-scale segmentations. We describe the design of NeuroBlocks, starting with an analysis of the domain-specific tasks, their inherent challenges, and our subsequent task abstraction and visual representation. We demonstrate the utility of our design based on two case studies that focus on different user roles and their respective requirements for performing and tracking the progress of segmentation and proofreading in a large real-world connectomics project.

PDF icon PaperFile Video
2014
Time-Lapse Photometric Stereo and Applications
F. Shen, K. Sunkavalli, N. Bonneel, S. Rsinkiewicz, H. Pfister, and X. Tong, “Time-Lapse Photometric Stereo and Applications,” in Pacific Graphics, Seoul, Korea, 2014, 7th ed., vol. 33.Abstract
This paper presents a technique to recover geometry from time-lapse sequences of outdoor scenes. We build upon photometric stereo techniques to recover approximate shadowing, shading and normal components allowing us to alter the material and normals of the scene. Previous work in analyzing such images has faced two fundamental difficulties: 1. the illumination in outdoor images consists of time-varying sunlight and skylight, and 2. the motion of the sun is restricted to a near-planar arc through the sky, making surface normal recovery unstable. We develop methods to estimate the reflection component due to skylight illumination. We also show that sunlight directions are usually non-planar, thus making surface normal recovery possible. This allows us to estimate approximate surface normals for outdoor scenes using a single day of data. We demonstrate the use of these surface normal for a number of image editing applications including reflectance, lighting, and normal editing.
PDF icon Paper
Device Effect on Panoramic Video+Context Tasks
F. Pece, J. Tompkin, H. Pfister, J. Kautz, and C. Theobalt, “Device Effect on Panoramic Video+Context Tasks,” in European Conference on Visual Media Production (CVMP 2014), London, UK, 2014.Abstract

Panoramic imagery is viewed daily by thousands of people, and panoramic video imagery is becoming more common. This imagery is viewed on many different devices with different properties, and the effect of these differences on spatio-temporal task performance is yet untested on these imagery. We adapt a novel panoramic video interface and conduct a user study to discover whether display type affects spatio-temporal reasoning task performance across desktop monitor, tablet, and head-mounted displays. We discover that, in our complex reasoning task, HMDs are as effective as desktop displays even if participants felt less capable, but tablets were less effective than desktop displays even though participants felt just as capable. Our results impact virtual tourism, telepresence, and surveillance applications, and so we state the design implications of our results for panoramic imagery systems.

PDF icon PaperFile Video
J. W. Lichtman, H. Pfister, and N. Shavit, “The Big Data Challenges of Connectomics,” Nature Neuroscience, no. 17, pp. 1448–1454, 2014. Publisher's VersionAbstract

The structure of the nervous system is extraordinarily complicated because individual neurons are interconnected to hundreds or even thousands of other cells in networks that can extend over large volumes. Mapping such networks at the level of synaptic connections, a field called connectomics, began in the 1970s with a the study of the small nervous system of a worm and has recently garnered general interest thanks to technical and computational advances that automate the collection of electron-microscopy data and offer the possibility of mapping even large mammalian brains. However, modern connectomics produces 'big data', unprecedented quantities of digital information at unprecedented rates, and will require, as with genomics at the time, breakthrough algorithmic and computational solutions. Here we describe some of the key difficulties that may arise and provide suggestions for managing them.

PDF icon Paper
Efficient Learning of Image Super-resolution and Compression Artifact Removal with Semi-local Gaussian Processes
Y. Kwon, K. I. Kim, J. Tompkin, J. H. Kim, and C. Theobalt, “Efficient Learning of Image Super-resolution and Compression Artifact Removal with Semi-local Gaussian Processes,” Transactions on Pattern Analysis and Machine Intelligence, 2014.Abstract

Improving the quality of degraded images is a key problem in image processing, but the breadth of the problem leads to domain-specific approaches for tasks such as super-resolution and compression artifact removal. Recent approaches have shown that a general approach is possible by learning application-specific models from examples; however, learning models sophisticated enough to generate high-quality images is computationally expensive, and so specific per-application or per-dataset models are impractical. To solve this problem, we present an efficient semi-local approximation scheme to large-scale Gaussian processes. This allows efficient learning of task-specific image enhancements from example images without reducing quality. As such, our algorithm can be easily customized to specific applications and datasets, and we show the efficiency and effectiveness of our approach across five domains: single-image super-resolution for scene, human face, and text images, and artifact removal in JPEG- and JPEG 2000-encoded images.

PDF icon PaperPDF icon Supplemental

Pages