W. G. Roncal, et al., “VESICLE: Volumetric Evaluation of Synaptic Interfaces using Computer Vision at Large Scale,” in Proceedings of the British Machine Vision Conference (BMVC), 2015, pp. 81.1-81.13. Publisher's VersionAbstract

An open challenge at the forefront of modern neuroscience is to obtain a comprehensive mapping of the neural pathways that underlie human brain function; an enhanced understanding of the wiring diagram of the brain promises to lead to new breakthroughs in diagnosing and treating neurological disorders. Inferring brain structure from image data, such as that obtained via electron microscopy (EM), entails solving the problem of identifying biological structures in large data volumes. Synapses, which are a key communication structure in the brain, are particularly difficult to detect due to their small size and limited contrast. Prior work in automated synapse detection has relied upon time-intensive, error-prone biological preparations (isotropic slicing, post-staining) in order to simplify the problem. This paper presents VESICLE, the first known approach designed for mammalian synapse detection in anisotropic, non-poststained data. Our methods explicitly leverage biological context, and the results exceed existing synapse detection methods in terms of accuracy and scalability. We provide two different approaches - a deep learning classifier (VESICLE-CNN) and a lightweight Random Forest approach (VESICLE-RF), to offer alternatives in the performance-scalability space. Addressing this synapse detection challenge enables the analysis of high-throughput imaging that is soon expected to produce petabytes of data, and provides tools for more rapid estimation of brain-graphs. Finally, to facilitate community efforts, we developed tools for large-scale object detection, and demonstrated this framework to find ~50,000 synapses in 60,000 um^3 (220 GB on disk) of electron microscopy data.

A Crowdsourced Alternative to Eye-tracking for Visualization Understanding
N. W. Kim, Z. Bylinskii, M. A. Borkin, A. Oliva, K. Z. Gajos, and H. Pfister, “A Crowdsourced Alternative to Eye-tracking for Visualization Understanding,” in CHI’15 Extended Abstracts, Seoul, Korea, 2015, pp. 1349-1354 . Publisher's VersionAbstract

In this study we investigate the utility of using mouse clicks as an alternative for eye fixations in the context of understanding data visualizations. We developed a crowdsourced study online in which participants were presented with a series of images containing graphs and diagrams and asked to describe them. Each image was blurred so that the participant needed to click to reveal bubbles - small, circular areas of the image at normal resolution. This is similar to having a confined area of focus like the human eye fovea. We compared the bubble click data with the fixation data from a complementary eye-tracking experiment by calculating the similarity between the resulting heatmaps. A high similarity score suggests that our methodology may be a viable crowdsourced alternative to eye-tracking experiments, especially when little to no eye-tracking data is available. This methodology can also be used to complement eye-tracking studies with an additional behavioral measurement, since it is specifically designed to measure which information people consciously choose to examine for understanding visualizations.

State-of-the-Art in GPU-Based Large-Scale Volume Visualization
J. Beyer, M. Hadwiger, and H. Pfister, “State-of-the-Art in GPU-Based Large-Scale Volume Visualization,” Computer Graphics Forum, 2015. Publisher's VersionAbstract

This survey gives an overview of the current state of the art in GPU techniques for interactive large-scale volume visualization. Modern techniques in this field have brought about a sea change in how interactive visualization and analysis of giga-, tera-, and petabytes of volume data can be enabled on GPUs. In addition to combining the parallel processing power of GPUs with out-of-core methods and data streaming, a major enabler for interactivity is making both the computational and the visualization effort proportional to the amount and resolution of data that is actually visible on screen, i.e., “output-sensitive” algorithms and system designs. This leads to recent output- sensitive approaches that are “ray-guided,” “visualization-driven,” or “display-aware.” In this survey, we focus on these characteristics and propose a new categorization of GPU-based large-scale volume visualization techniques based on the notions of actual output-resolution visibility and the current working set of volume bricks—the current subset of data that is minimally required to produce an output image of the desired display resolution. Furthermore, we discuss the differences and similarities of different rendering and data traversal strategies in volume rendering by putting them into a common context—the notion of address translation. For our purposes here, we view parallel (distributed) visualization using clusters as an orthogonal set of techniques that we do not discuss in detail but that can be used in conjunction with what we discuss in this survey.

Large-Scale Automatic Reconstruction of Neuronal Processes from Electron Microscopy Images
V. Kaynig, et al., “,” Medical Image Analysis, vol. 22, no. 1, pp. 77-88, 2015.

Automated sample preparation and electron microscopy enables acquisition of very large image data sets. These technical advances are of special importance to the field of neuroanatomy, as 3D reconstructions of neuronal processes at the nm scale can provide new
insight into the fine grained structure of the brain. Segmentation of large-scale electron microscopy data is the main bottleneck in the analysis of these data sets. In this paper we present a pipeline that provides state-of-the art reconstruction performance while scaling
to data sets in the GB-TB range. First, we train a random forest classifier on interactive sparse user annotations. The classifier output is combined with an anisotropic smoothing prior in a Conditional Random Field framework to generate multiple segmentation
hypotheses per image. These segmentations are then combined into geometrically consistent 3D objects by segmentation fusion. We provide qualitative and quantitative evaluation of the automatic segmentation and demonstrate large-scale 3D reconstructions
of neuronal processes from a 27; 000 m3 volume of brain tissue over a cube of 30 m in each dimension corresponding to 1,000 consecutive image sections. We also introduce Mojo, a proofreading tool including semi-automated correction of merge errors
based on sparse user scribbles.

Computational Design of Walking Automata
G. Bharaj, S. Coros, B. Thomaszewski, J. Tompkin, B. Bickel, and H. Pfister, “Computational Design of Walking Automata,” in ACM SIGGRAPH / Eurographics Symposium on Computer Animation, 2015. Publisher's VersionAbstract

Creating mechanical automata that can walk in stable and pleasing manners is a challenging task that requires both skill and expertise. We propose to use computational design to offset the technical difficulties of this process. A simple drag-and-drop interface allows casual users to create personalized walking toys from a library of pre-defined template mechanisms. Provided with this input, our method leverages physical simulation and evolutionary optimization to refine the mechanical designs such that the resulting toys are able to walk. The optimization process is guided by an intuitive set of objectives that measure the quality of the walking motions. We demonstrate our approach on a set of simulated mechanical toys with different numbers of legs and various distinct gaits. Two fabricated prototypes showcase the feasibility of our designs.

Blind Video Temporal Consistency
N. Bonneel, J. Tompkin, K. Sunkavalli, D. Sun, S. Paris, and H. Pfister, “Blind Video Temporal Consistency,” ACM Transactions on Graphics (SIGGRAPH Asia), 2015. Webpage
Computational Design of Metallophone Contact Sounds
G. Bharaj, et al., “Computational Design of Metallophone Contact Sounds,” ACM Transactions on Graphics (SIGGRAPH Asia), 2015. Webpage
Context-guided Diffusion for Label Propagation on Graphs
K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, “Context-guided Diffusion for Label Propagation on Graphs,” International Conference on Computer Vision (ICCV), 2015. Webpage
Generalizing Wave Gestures from Sparse Examples for Real-time Character Control
H. Rhodin, et al., “Generalizing Wave Gestures from Sparse Examples for Real-time Character Control,” ACM Transactions on Graphics (SIGGRAPH Asia), 2015. Webpage
Joint 5D Pen Input for Light Field Displays
J. Tompkin, et al., “Joint 5D Pen Input for Light Field Displays,” ACM User Interface Software and Technology (UIST), 2015. Webpage
Local High-order Regularization on Data Manifolds
K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, “Local High-order Regularization on Data Manifolds,” IEEE Computer Vision and Pattern Recognition (CVPR). 2015. Webpage
Semi-supervised Learning with Explicit Relationship Regularization
K. I. Kim, J. Tompkin, H. Pfister, and C. Theobalt, “Semi-supervised Learning with Explicit Relationship Regularization,” IEEE Computer Vision and Pattern Recognition (CVPR), 2015. Webpage
Beyond Memorability: Visualization Recognition and Recall
M. Borkin, et al., “Beyond Memorability: Visualization Recognition and Recall,” Visualization and Computer Graphics, IEEE Transactions on, vol. PP, pp. 1-1, 2015. Publisher's VersionAbstract

In this paper we move beyond memorability and investigate how visualizations are recognized and recalled. For this study we labeled a dataset of 393 visualizations and analyzed the eye movements of 33 participants as well as thousands of participantgenerated text descriptions of the visualizations. This allowed us to determine what components of a visualization attract people’s attention, and what information is encoded into memory. Our findings quantitatively support many conventional qualitative design guidelines, including that (1) titles and supporting text should convey the message of a visualization, (2) if used appropriately, pictograms do not interfere with understanding and can improve recognition, and (3) redundancy helps effectively communicate the message. Importantly, we show that visualizations memorable “at-a-glance” are also capable of effectively conveying the message of the visualization. Thus, a memorable visualization is often also an effective one.

Paper Supplemental Material Video Slides
NeuroBlocks - Visual Tracking of Segmentation and Proofreading for Large Connectomics Projects
A. K. Al-Awami, et al., “NeuroBlocks - Visual Tracking of Segmentation and Proofreading for Large Connectomics Projects,” IEEE Transactions on Visualization and Computer Graphics, vol. to appear, 2015.Abstract

In the field of connectomics, neuroscientists acquire electron microscopy volumes at nanometer resolution in order to reconstruct a detailed wiring diagram of the neurons in the brain. The resulting image volumes, which often are hundreds of terabytes in size, need to be segmented to identify cell boundaries, synapses, and important cell organelles. However, the segmentation process of a single volume is very complex, time-intensive, and usually performed using a diverse set of tools and many users. To tackle the associated challenges, this paper presents NeuroBlocks, which is a novel visualization system for tracking the state, progress, and evolution of very large volumetric segmentation data in neuroscience. NeuroBlocks is a multi-user web-based application that seamlessly integrates the diverse set of tools that neuroscientists currently use for manual and semi-automatic segmentation, proofreading, visualization, and analysis. NeuroBlocks is the first system that integrates this heterogeneous tool set, providing crucial support for the management, provenance, accountability, and auditing of large-scale segmentations. We describe the design of NeuroBlocks, starting with an analysis of the domain-specific tasks, their inherent challenges, and our subsequent task abstraction and visual representation. We demonstrate the utility of our design based on two case studies that focus on different user roles and their respective requirements for performing and tracking the progress of segmentation and proofreading in a large real-world connectomics project.

Paper Video
Time-Lapse Photometric Stereo and Applications
F. Shen, K. Sunkavalli, N. Bonneel, S. Rsinkiewicz, H. Pfister, and X. Tong, “

Time-Lapse Photometric Stereo and Applications

,” in Pacific Graphics, Seoul, Korea, 2014, 7th ed., vol. 33.Abstract
This paper presents a technique to recover geometry from time-lapse sequences of outdoor scenes. We build upon photometric stereo techniques to recover approximate shadowing, shading and normal components allowing us to alter the material and normals of the scene. Previous work in analyzing such images has faced two fundamental difficulties: 1. the illumination in outdoor images consists of time-varying sunlight and skylight, and 2. the motion of the sun is restricted to a near-planar arc through the sky, making surface normal recovery unstable. We develop methods to estimate the reflection component due to skylight illumination. We also show that sunlight directions are usually non-planar, thus making surface normal recovery possible. This allows us to estimate approximate surface normals for outdoor scenes using a single day of data. We demonstrate the use of these surface normal for a number of image editing applications including reflectance, lighting, and normal editing.
Device Effect on Panoramic Video+Context Tasks
F. Pece, J. Tompkin, H. Pfister, J. Kautz, and C. Theobalt, “Device Effect on Panoramic Video+Context Tasks,” in European Conference on Visual Media Production (CVMP 2014), London, UK, 2014.Abstract

Panoramic imagery is viewed daily by thousands of people, and panoramic video imagery is becoming more common. This imagery is viewed on many different devices with different properties, and the effect of these differences on spatio-temporal task performance is yet untested on these imagery. We adapt a novel panoramic video interface and conduct a user study to discover whether display type affects spatio-temporal reasoning task performance across desktop monitor, tablet, and head-mounted displays. We discover that, in our complex reasoning task, HMDs are as effective as desktop displays even if participants felt less capable, but tablets were less effective than desktop displays even though participants felt just as capable. Our results impact virtual tourism, telepresence, and surveillance applications, and so we state the design implications of our results for panoramic imagery systems.

Paper Video
J. W. Lichtman, H. Pfister, and N. Shavit, “The Big Data Challenges of Connectomics,” Nature Neuroscience, no. 17, pp. 1448–1454, 2014. Publisher's VersionAbstract

The structure of the nervous system is extraordinarily complicated because individual neurons are interconnected to hundreds or even thousands of other cells in networks that can extend over large volumes. Mapping such networks at the level of synaptic connections, a field called connectomics, began in the 1970s with a the study of the small nervous system of a worm and has recently garnered general interest thanks to technical and computational advances that automate the collection of electron-microscopy data and offer the possibility of mapping even large mammalian brains. However, modern connectomics produces 'big data', unprecedented quantities of digital information at unprecedented rates, and will require, as with genomics at the time, breakthrough algorithmic and computational solutions. Here we describe some of the key difficulties that may arise and provide suggestions for managing them.

Efficient Learning of Image Super-resolution and Compression Artifact Removal with Semi-local Gaussian Processes
Y. Kwon, K. I. Kim, J. Tompkin, J. H. Kim, and C. Theobalt, “

Efficient Learning of Image Super-resolution and Compression Artifact Removal with Semi-local Gaussian Processes

,” Transactions on Pattern Analysis and Machine Intelligence, 2014.Abstract

Improving the quality of degraded images is a key problem in image processing, but the breadth of the problem leads to domain-specific approaches for tasks such as super-resolution and compression artifact removal. Recent approaches have shown that a general approach is possible by learning application-specific models from examples; however, learning models sophisticated enough to generate high-quality images is computationally expensive, and so specific per-application or per-dataset models are impractical. To solve this problem, we present an efficient semi-local approximation scheme to large-scale Gaussian processes. This allows efficient learning of task-specific image enhancements from example images without reducing quality. As such, our algorithm can be easily customized to specific applications and datasets, and we show the efficiency and effectiveness of our approach across five domains: single-image super-resolution for scene, human face, and text images, and artifact removal in JPEG- and JPEG 2000-encoded images.

Paper Supplemental
Interactive Intrinsic Video Editing
N. Bonneel, K. Sunkavalli, J. Tompkin, D. Sun, S. Paris, and H. Pfister, “

Interactive Intrinsic Video Editing

,” ACM Transactions on Graphics (Proc. SIGGRAPH), vol. 33, no. 6, 2014.Abstract
Separating a photograph into its reflectance and illumination intrinsicimages is a fundamentally ambiguous problem, and state-of-the-art algorithms combine sophisticated reflectance and illumination priors with user annotations to create plausible results. However, these algorithms cannot be easily extended to videos for two reasons: first, naıvely applying algorithms designed for single images to videos produce results that are temporally incoherent; second, effectively specifying user annotations for a video requires interactive feedback, and current approaches are orders of magnitudes too slow to support this. We introduce a fast and temporally consistent algorithm to decompose video sequences into their reflectance and illumination components. Our algorithm uses a hybrid formulation that separates image gradients into smooth illumination and sparse reflectance gradients using look-up tables. We use a multi-scale parallelized solver to reconstruct the reflectance and illumination from these gradients while enforcing spatial and temporal reflectance constraints and user annotations. We demonstrate that our algorithm automatically produces reasonable results, that can be interactively refined by users, at rates that are two orders of magnitude faster than existing tools, to produce high-quality decompositions for challenging real-world video sequences. We also show how these decompositions can be used for a number of video editing applications including recoloring, retexturing, illumination editing, and lighting-aware compositing.
Paper Video
Characterizing Cancer Subtypes using Dual Analysis in Caleydo
C. Turkay, A. Lex, M. Streit, H. Pfister, and H. Hauser, “

Characterizing Cancer Subtypes using Dual Analysis in Caleydo

,” IEEE Computer Graphics and Applications, vol. 34, no. 2, pp. 38-47, 2014. Publisher's VersionAbstract

The comprehensive analysis and characterization of cancer subtypes is an important problem to which significant resources have been devoted in recent years. In this paper we integrate the dual analysis method, which uses statistics to describe both the dimensions and the rows of a high dimensional dataset, into StratomeX, a Caleydo view tailored to cancer subtype analysis. We introduce significant difference plots for showing the elements of a candidate cancer subtype that differ significantly from other subtypes, thus enabling analysts to characterize cancer subtypes. We also enable analysts to investigate how samples relate to the subtype they are assigned and to the other groups. Our approach gives analysts the ability to create well-defined candidate subtypes based on statistical properties. We demonstrate the utility of our approach in three case studies, where we show that we are able to reproduce findings from a published cancer subtype characterization.