<p>NeuroLines: A Subway Map Metaphor for Visualizing Nanoscale Neuronal Connectivity</p>
Al-Awami AK, Beyer J, Strobelt H, Kasthuri N, Lichtman JW, Pfister H, Hadwiger M.

NeuroLines: A Subway Map Metaphor for Visualizing Nanoscale Neuronal Connectivity

. IEEE Transactions on Visualization and Computer Graphics. 2014;to appear.Abstract
We present NeuroLines, a novel visualization technique designed for scalable detailed analysis of neuronal connectivity at the nanoscale level. The topology of 3D brain tissue data is abstracted into a multi-scale, relative distance-preserving subway map visualization that allows domain scientists to conduct an interactive analysis of neurons and their connectivity. Nanoscale connectomics aims at reverse-engineering the wiring of the brain. Reconstructing and analyzing the detailed connectivity of neurons and neurites (axons, dendrites) will be crucial for understanding the brain and its development and diseases. However, the enormous scale and complexity of nanoscale neuronal connectivity pose big challenges to existing visualization techniques in terms of scalability. NeuroLines offers a scalable visualization framework that can interactively render thousands of neurites, and that supports the detailed analysis of neuronal structures and their connectivity. We describe and analyze the design of NeuroLines based on two real-world use-cases of our collaborators in developmental neuroscience, and investigate its scalability to large-scale neuronal connectivity data. 
Paper Video
<p>Design and Evaluation of Interactive Proofreading Tools for Connectomics</p>
Haehn D, Knowles-Barley S, Roberts M, Beyer J, Kasthuri N, Lichtman JW, Pfister H.

Design and Evaluation of Interactive Proofreading Tools for Connectomics

. IEEE Transactions on Visualization and Computer Graphics [Internet]. 2014:to appear. WebsiteAbstract
Proofreading refers to the manual correction of automatic segmentations of image data. In connectomics, electron microscopy data is acquired at nanometer-scale resolution and results in very large image volumes of brain tissue that require fully automatic segmentation algorithms to identify cell boundaries. However, these algorithms require hundreds of corrections per cubic micron of tissue. Even though this task is time consuming, it is fairly easy for humans to perform corrections through splitting, merging, and adjusting segments during proofreading. In this paper we present the design and implementation of Mojo, a fully-featured single-user desktop application for proofreading, and Dojo, a multi-user web-based application for collaborative proofreading. We evaluate the accuracy and speed of Mojo, Dojo, and Raveler, a proofreading tool from Janelia Farm, through a quantitative user study. We designed a between-subjects experiment and asked non-experts to proofread neurons in a publicly available connectomics dataset. Our results show a significant improvement of corrections using web-based Dojo even in comparison to fully manual expert segmentation, when given the same amount of time. In addition, all participants using Dojo reported better usability. We discuss our findings and provide an analysis of requirements for designing visual proofreading software.
Paper Video
<p>NeuroLines - A Subway Map Metaphor for Visualizing Nanoscale Neuronal Connectivity (best poster award)</p>
Al-Awami AK, Beyer J, Strobelt H, Kasthuri N, Lichtman JW, Pfister H, Hadwiger M.

NeuroLines - A Subway Map Metaphor for Visualizing Nanoscale Neuronal Connectivity (best poster award)

. Poster at the 4th Symposium on Biological Data Visualization. 2014.Abstract
We introduce NeuroLines, a novel tool designed for visualizing neuronal morphology and connectivity at the nanoscale level. NeuroLines uses a subway map metaphor to abstract the topology of 3D brain tissue data into a multi-scale, relative distance-preserving 2D visualization. This allows domain scientists to conduct an interactive analysis of neurons and their connectivity. Nanoscale connectomics attempts to reverse-engineer the wiring diagram of the brain. This task, coupled with the task of analyzing the detailed connectivity of neurites (axons, dendrites), is crucial to understanding the brain, its development and pathologies. However, the main challenge with such tasks is the enormous scale, complexity and visual clutter of nanoscale connectivity. This makes it difficult for existing visualization techniques to render such data in a meaningful way. NeuroLines offers a scalable visualization platform that can interactively render thousands of neurites in an uncluttered fashion, paired with interactive features to support the detail analysis of neuronal connectivity. 
Sun D, Liu C, Pfister H.

Local Layering for Joint Motion Estimation and Occlusion Detection

. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2014.Abstract
Most motion estimation algorithms (optical flow, layered models) cannot handle large amount of occlusion in textureless regions, as motion is often initialized with no occlusion assumption despite that occlusion may be included in the final objective. To handle such situations, we propose a local layering model where motion and occlusion relationships are inferred jointly. In particular, the uncertainties of occlusion relationships are retained so that motion is inferred by considering all the possibilities of local occlusion relationships. In addition, the local layering model handles articulated objects with self-occlusion. We demonstrate that the local layering model can handle motion and occlusion well for both challenging synthetic and real sequences.
<p>A Survey of GPU-Based Large-Scale Volume Visualization</p>
Beyer J, Hadwiger M, Pfister H.

A Survey of GPU-Based Large-Scale Volume Visualization

. Eurographics Conference on Visualization (EuroVis). 2014:to appear.Abstract
This survey gives an overview of the current state of the art in GPU techniques for interactive large-scale volume visualization. Modern techniques in this field have brought about a sea change in how interactive visualization and analysis of giga-, tera-, and petabytes of volume data can be enabled on GPUs. In addition to combining the parallel processing power of GPUs with out-of-core methods and data streaming, a major enabler for interactivity is making both the computational and the visualization effort proportional to the amount and resolution of data that is actually visible on screen, i.e., output-sensitive algorithms and system designs. This leads to recent output-sensitive approaches that are ray-guided, visualization-driven, or display-aware. In this survey, we focus on these characteristics and propose a new categorization of GPU-based large-scale volume visualization techniques based on the notions of actual output-resolution visibility and the current working set of volume bricks - the current subset of data that is minimally required to produce an output image of the desired display resolution. For our purposes here, we view parallel (distributed) visualization using clusters as an orthogonal set of techniques that we do not discuss in detail but that can be used in conjunction with what we discuss in this survey.
Bonneel N, Rabin J, Peyré G, Pfister H.

Sliced and Radon Wasserstein Barycenters of Measures

. Journal of Mathematical Imaging and Vision. 2014.Abstract
This article details two approaches to compute barycenters of measures using 1-D Wasserstein distances along radial projections of the input measures. The first method makes use of the Radon transform of the measures, and the second is the solution of a convex optimization problem over the space of measures. We show several properties of these barycenters and explain their relationship. We show numerical approximation schemes based on a discrete Radon transform and on the resolution of a non-convex optimization problem. We explore the respective merits and drawbacks of each approach on applications to two image processing problems: color transfer and texture mixing.
Facial Performance Enhancement Using Dynamic Shape Space Analysis
Bermano AH, Bradley D, Beeler T, Zünd F, Nowrouzezahrai D, Baran I, Sorkine O, Pfister H, Sumner RW, Bickel B, et al. Facial Performance Enhancement Using Dynamic Shape Space Analysis. In: ACM Transactions On Graphics . ; 2013.Abstract
The facial performance of an individual is inherently rich in subtle deformation and timing details. Although these subtleties make the performance realistic and compelling, they often elude both motion capture and hand animation. We present a technique for adding fine-scale details and expressiveness to low-resolution art-directed facial performances, such as those created manually using a rig, via marker-based capture, by fitting a morphable model to a video, or through Kinect reconstruction using recent faceshift technology. We employ a high-resolution facial performance capture system to acquire a representative performance of an individual in which he or she explores the full range of facial expressiveness. From the captured data, our system extracts an expressiveness model that encodes subtle spatial and temporal deformation details specific to that particular individual. Once this model has been built, these details can be transferred to low-resolution artdirected performances. We demonstrate results on various forms of input; after our enhancement, the resulting animations exhibit the same nuances and fine spatial details as the captured performance, with optional temporal enhancement to match the dynamics of the actor. Finally, we show that our technique outperforms the current state-of-the-art in example-based facial animation
Segmenting Planar Superpixel Adjacency Graphs w.r.t. Non-planar Superpixel Affinity Graphs
Andres B, Yarkony J, Manjunath BS, Kirchhoff S, Turetken E, Fowlkes CC, Pfister H. Segmenting Planar Superpixel Adjacency Graphs w.r.t. Non-planar Superpixel Affinity Graphs. Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR). 2013.Abstract
We address the problem of segmenting an image into a previously unknown number of segments from the perspective of graph partitioning. Specifically, we consider minimum multicuts of superpixel affinity graphs in which all affinities between non-adjacent superpixels are negative. We propose a relaxation by Lagrangian decomposition and a constrained set of re-parameterizations for which we can optimize exactly and efficiently. Our contribution is to show how the planarity of the adjacency graph can be exploited if the affinity graph is non-planar. We demonstrate the effectiveness of this approach in user-assisted image segmentation and show that the solution of the relaxed problem is fast and the relaxation is tight in practice.
<p>A Fully-Connected Layered Model of Foreground and Background Flow</p>
Sun D, Wulff J, Sudderth EB, Pfister H, Black MJ.

A Fully-Connected Layered Model of Foreground and Background Flow

. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2013:2451-2458. Paper
Exploring the Connectome - Petascale Volume Visualization of Microscopy Data Streams
Beyer J, Hadwiger M, Al-Awami A, Jeong W, Kasthuri N, Lichtman JW, Pfister H. Exploring the Connectome - Petascale Volume Visualization of Microscopy Data Streams. IEEE Computer Graphics and Applications. 2013;33(4):50-61.Abstract
Recent advances in high-resolution microscopy allow neuroscientists to acquire volume data of neural tissue of extreme size. However, the tremendous resolution and the high complexity of neural structures present big challenges to storage, processing, and visualization at interactive rates. We present a system for interactive exploration of petascale (petavoxel) volumes resulting from high-throughput electron microscopy data streams. Our system can concurrently handle multiple volumes, and also supports the simultaneous visualization of high-resolution voxel segmentation data. We employ a visualization-driven system design that allows us to restrict most computations to a small sub-set of the data. We employ a multi-resolution virtual memory architecture for better scalability than previous approaches and handling of incomplete data. We illustrate the real-world use of our system for a mouse cortex volume of one teravoxel in size, where several hundred neurites as well as synapses have been segmented and labeled.
ConnectomeExplorer: Query-Guided Visual Analysis of Large Volumetric Neuroscience Data
Beyer J, Al-Awami A, Kasthuri N, Lichtman JW, Pfister H, Hadwiger M. ConnectomeExplorer: Query-Guided Visual Analysis of Large Volumetric Neuroscience Data. IEEE Transactions on Visualization and Computer Graphics. 2013;19(12).Abstract
This paper presents ConnectomeExplorer, an application for the interactive exploration and query-guided visual analysis of large volumetric electron microscopy (EM) data sets in connectomics research. Our system incorporates a knowledge-based query algebra that supports the interactive specification of dynamically evaluated queries, which enable neuroscientists to pose and answer domain-specific questions in an intuitive manner. Queries are built step by step in a visual query builder, building more complex queries from combinations of simpler queries. Our application is based on a scalable volume visualization framework that scales to multiple volumes of several teravoxels each, enabling the concurrent visualization and querying of the original EM volume, additional segmentation volumes, neuronal connectivity, and additional meta data comprising a variety of neuronal data attributes. We evaluate our application on a data set of roughly one terabyte of EM data and 750 GB of segmentation data, containing over 4,000 segmented structures and 1,000 synapses. We demonstrate typical use-case scenarios of our collaborators in neuroscience, where our system has enabled them to answer specific scientific questions using interactive querying and analysis on the full-size data for the first time.
Paper Video
What Makes a Visualization Memorable?
Borkin MA, Vo AA, Bylinskii Z, Isola P, Sunkavalli S, Oliva A, Pfister H. What Makes a Visualization Memorable?. IEEE Transactions on Visualization and Computer Graphics (Proceedings of InfoVis 2013). 2013.Abstract
An ongoing debate in the Visualization community concerns the role that visualization types play in data understanding. In human cognition, understanding and memorability are intertwined. As a first step towards being able to ask questions about impact and effectiveness, here we ask: “What makes a visualization memorable?” We ran the largest scale visualization study to date using 2,070 single-panel visualizations, categorized with visualization type (e.g., bar chart, line graph, etc.), collected from news media sites, government reports, scientific journals, and infographic sources. Each visualization was annotated with additional attributes, including ratings for data-ink ratios and visual densities. Using Amazon’s Mechanical Turk, we collected memorability scores for hundreds of these visualizations, and discovered that observers are consistent in which visualizations they find memorable and forgettable. We find intuitive results (e.g., attributes like color and the inclusion of a human recognizable object enhance memorability) and less intuitive results (e.g., common graphs are less memorable than unique visualization types). Altogether our findings suggest that quantifying memorability is a general metric of the utility of information, an essential step towards determining how to design effective visualizations.
Paper Supplemental Material Experiment Demo Video Slides
Evaluation of Filesystem Provenance Visualization Tools
Borkin MA, Yeh CS, Boyd M, Macko P, Gajos KZ, Seltzer M, Pfister H. Evaluation of Filesystem Provenance Visualization Tools. IEEE Transactions on Visualization and Computer Graphics (Proceedings of InfoVis 2013). 2013.Abstract
Having effective visualizations of filesystem provenance data is valuable for understanding its complex hierarchical structure. The most common visual representation of provenance data is the node-link diagram. While effective for understanding local activity, the node-link diagram fails to offer a high-level summary of activity and inter-relationships within the data. We present a new tool, InProv, which displays filesystem provenance with an interactive radial-based tree layout. The tool also utilizes a new time-based hierarchical node grouping method for filesystem provenance data we developed to match the user’s mental model and make data exploration more intuitive. We compared InProv to a conventional node-link based tool, Orbiter, in a quantitative evaluation with real users of filesystem provenance data including provenance data experts, IT professionals, and computational scientists. We also compared in the evaluation our new node grouping method to a conventional method. The results demonstrate that InProv results in higher accuracy in identifying system activity than Orbiter with large complex data sets. The results also show that our new time-based hierarchical node grouping method improves performance in both tools, and participants found both tools significantly easier to use with the new time-based node grouping method. Subjective measures show that participants found InProv to require less mental activity, less physical activity, less work, and is less stressful to use. Our study also reveals one of the first cases of gender differences in visualization; both genders had comparable performance with InProv, but women had a significantly lower average accuracy (56%) compared to men (70%) with Orbiter.
Paper Video Slides
Entourage: Visualizing Relationships between Biological Pathways using Contextual Subsets
Lex A, Partl C, Kalkofen D, Streit M, Wasserman AM, Gratzl S, Schmalstieg D, Pfister H. Entourage: Visualizing Relationships between Biological Pathways using Contextual Subsets. IEEE Transactions on Visualization and Computer Graphics (InfoVis '13) [Internet]. 2013;19(12):2536-2545. Project WebsiteAbstract
Biological pathway maps are highly relevant tools for many tasks in molecular biology. They reduce the complexity of the overall biological network by partitioning it into smaller manageable parts. While this reduction of complexity is their biggest strength, it is, at the same time, their biggest weakness. By removing what is deemed not important for the primary function of the pathway, biologists lose the ability to follow and understand cross-talks between pathways. Considering these cross-talks is, however, critical in many analysis scenarios, such as judging effects of drugs. In this paper we introduce Entourage, a novel visualization technique that provides contextual information lost due to the artificial partitioning of the biological network, but at the same time limits the presented information to what is relevant to the analyst's task. We use one pathway map as the focus of an analysis and allow a larger set of contextual pathways. For these context pathways we only show the contextual subsets, i.e., the parts of the graph that are relevant to a selection. Entourage suggests related pathways based on similarities and highlights parts of a pathway that are interesting in terms of mapped experimental data. We visualize interdependencies between pathways using stubs of visual links, which we found effective yet not obtrusive. By combining this approach with visualization of experimental data, we can provide domain experts with a highly valuable tool. We demonstrate the utility of Entourage with case studies conducted with a biochemist who researches the effects of drugs on pathways. We show that the technique is well suited to investigate interdependencies between pathways and to analyze, understand, and predict the effect that drugs have on different cell types.
Paper Video
<p>LineUp: Visual Analysis of Multi-Attribute Rankings (best paper award)</p>
Gratzl S, Lex A, Gehlenborg N, Pfister H, Streit M.

LineUp: Visual Analysis of Multi-Attribute Rankings (best paper award)

. IEEE Transactions on Visualization and Computer Graphics (InfoVis '13) [Internet]. 2013;19(12):2277-2286. Publisher's VersionAbstract
Rankings are a popular and universal approach to structuring otherwise unorganized collections of items by computing a rank for each item based on the value of one or more of its attributes. This allows us, for example, to prioritize tasks or to evaluate the performance of products relative to each other. While the visualization of a ranking itself is straightforward, its interpretation is not, because the rank of an item represents only a summary of a potentially complicated relationship between its attributes and those of the other items. It is also common that alternative rankings exist which need to be compared and analyzed to gain insight into how multiple heterogeneous attributes affect the rankings. Advanced visual exploration tools are needed to make this process efficient. In our paper we present a comprehensive analysis of requirements for the visualization of multi-attribute rankings. Based on these considerations, we propose LineUp - a novel and scalable visualization technique that uses bar charts. This interactive technique supports the ranking of items based on multiple heterogeneous attributes with different scales and semantics. It enables users to interactively combine attributes and flexibly refine parameters to explore the effect of changes in the attribute combination. This process can be employed to derive actionable insights as to which attributes of an item need to be modified in order for its rank to change. Additionally, through integration of slope graphs, LineUp can also be used to compare multiple alternative rankings on the same set of items, for example, over time or across different attribute combinations. We evaluate the effectiveness of the proposed multi-attribute visualization technique in a qualitative study. The study shows that users are able to successfully solve complex ranking tasks in a short period of time.
Paper Video
Example-Based Video Color Grading
Bonneel N, Sunkavalli K, Paris S, Pfister H. Example-Based Video Color Grading. ACM Transactions on Graphics (Proc. ACM SIGGRAPH). 2013;32(4).Abstract
In most professional cinema productions, the color palette of the movie is painstakingly adjusted by a team of skilled colorists -- through a process referred to as color grading -- to achieve a certain visual look. The time and expertise required to grade a video makes it difficult for amateurs to manipulate the colors of their own video clips. In this work, we present a method that allows a user to transfer the color palette of a model video clip to their own video sequence. We estimate a per-frame color transform that maps the color distributions in the input video sequence to that of the model video clip. Applying this transformation naively leads to artifacts such as bleeding and flickering. Instead, we propose a novel differential-geometry-based scheme that interpolates these transformations in a manner that minimizes their curvature, similarly to curvature flows. In addition, we automatically determine a set of keyframes that best represent this interpolated transformation curve, and can be used subsequently, to manually refine the color grade. We show how our method successfully transfer color palettes between videos for a range of visual styles and a number of input video clips.
Paper Images Video
A Collaborative Digital Pathology System for Multi-Touch Mobile and Desktop Computing Platforms
Jeong W, Schneider J, Hansen A, Lee M, Faulkner-Jones BE, Hecht J, Najarian R, Yee E, Lichtrman J, Pfister H. A Collaborative Digital Pathology System for Multi-Touch Mobile and Desktop Computing Platforms. Computer Graphics Forum. 2013.Abstract
Collaborative slide image viewing systems are becoming increasingly important in pathology applications such as telepathology and E-learning. Despite rapid advances in computing and imaging technology, current digital pathology systems have limited performance with respect to remote viewing of whole slide images on desktop or mobile computing devices. In this paper we present a novel digital pathology client-server systems that supports collaborative viewing of multi-plane whole slide images over standard networks using multi-touch enabled clients. Our system is built upon a standard HTTP web server and a MySQL database to allow multiple clients to exchange image and metadata concurrently. We introduce a domain-specific image-stack compression method that leverages real-time hardware decoding on mobile devices. It adaptively encodes image stacks in a decorrelated color space to achieve extremely low bitrates (0.8 bpp) with very low loss of image quality. We evaluate the image quality of our compression method and the performance of our system for diagnosis with an in-depth user study.
Paper Video
<p>Video Snapshots: Creating High-Quality Images from Video Clips</p>
Sunkavalli K, Joshi N, Kang SB, Cohen M, Pfister H.

Video Snapshots: Creating High-Quality Images from Video Clips

. IEEE Transactions on Visualization and Computer Graphics. 2012;18(11):1868-1879.Abstract
We describe a unified framework for generating a single high-quality still image ("snapshot”) from a short video clip. Our system allows the user to specify the desired operations for creating the output image, such as super resolution, noise and blur reduction, and selection of best focus. It also provides a visual summary of activity in the video by incorporating saliency-based objectives in the snapshot formation process. We show examples on a number of different video clips to illustrate the utility and flexibility of our system.
Non-causal Temporal Prior for Video Deblocking
Sun D, Liu C. Non-causal Temporal Prior for Video Deblocking. In: European Conferenceon Computer Vision (ECCV). Springer; 2012.Abstract
Real-world video sequences coded at low bit rates suffer from compression artifacts, which are visually disruptive and can cause problems to computer vision algorithms. Unlike the denoising problem where the high frequency components of the signal are present in the noisy observation, most high frequency details are lost during compression and artificial discontinuities arise across the coding block boundaries. In addition to sparse spatial priors that can reduce the blocking artifacts for a single frame, temporal information is needed to recover the lost spatial details. However, establishing accurate temporal correspondences from the compressed videos is challenging because of the loss of high frequency details and the increase of false blocking artifacts. In this paper, we propose a non-causal temporal prior model to reduce video compression artifacts by propagating information from adjacent frames and iterating between image reconstruction and motion estimation. Experimental results on real-world sequences demonstrate that the deblocked videos by the proposed system have marginal statistics of high frequency components closer to those of the original ones, and are better input for standard edge and corner detectors than the coded ones.
Interactive Volume Exploration of Petascale Microscopy Data Streams Using a Visualization-Driven Virtual Memory Approach
Hadwiger M, Beyer J, Jeong W, Pfister H. Interactive Volume Exploration of Petascale Microscopy Data Streams Using a Visualization-Driven Virtual Memory Approach. IEEE Transactions on Visualization and Computer Graphics (SciVis'12). 2012;18(2):2285-2294.Abstract
This paper presents the first volume visualization system that scales to petascale volumes imaged as a continuous stream of high-resolution electron microscopy images. Our architecture scales to dense, anisotropic petascale volumes because it: (1) decouples construction of the 3D multi-resolution representation required for visualization from data acquisition, and (2) decouples sample access time during ray-casting from the size of the multi-resolution hierarchy. Our system is designed around a scalable multi-resolution virtual memory architecture that handles missing data naturally, does not pre-compute any 3D multi-resolution representation such as an octree, and can accept a constant stream of 2D image tiles from the microscopes. A novelty of our system design is that it is visualization-driven: we restrict most computations to the visible volume data. Leveraging the virtual memory architecture, missing data are detected during volume ray-casting as cache misses, which are propagated backwards for on-demand out-of-core processing. 3D blocks of volume data are only constructed from 2D microscope image tiles when they have actually been accessed during ray-casting. We extensively evaluate our system design choices with respect to scalability and performance, compare to previous best-of-breed systems, and illustrate the effectiveness of our system for real microscopy data from neuroscience.
Video Paper