Publications

2014
<p>A Survey of GPU-Based Large-Scale Volume Visualization</p>
Beyer J, Hadwiger M, Pfister H.

A Survey of GPU-Based Large-Scale Volume Visualization

. Eurographics Conference on Visualization (EuroVis). 2014:to appear.Abstract
This survey gives an overview of the current state of the art in GPU techniques for interactive large-scale volume visualization. Modern techniques in this field have brought about a sea change in how interactive visualization and analysis of giga-, tera-, and petabytes of volume data can be enabled on GPUs. In addition to combining the parallel processing power of GPUs with out-of-core methods and data streaming, a major enabler for interactivity is making both the computational and the visualization effort proportional to the amount and resolution of data that is actually visible on screen, i.e., output-sensitive algorithms and system designs. This leads to recent output-sensitive approaches that are ray-guided, visualization-driven, or display-aware. In this survey, we focus on these characteristics and propose a new categorization of GPU-based large-scale volume visualization techniques based on the notions of actual output-resolution visibility and the current working set of volume bricks - the current subset of data that is minimally required to produce an output image of the desired display resolution. For our purposes here, we view parallel (distributed) visualization using clusters as an orthogonal set of techniques that we do not discuss in detail but that can be used in conjunction with what we discuss in this survey.
paper107_camera_ready.pdf
Bonneel N, Rabin J, Peyré G, Pfister H.

 

Sliced and RadonWasserstein Barycenters of Measures

. Journal of Mathematical Imaging and Vision. 2014.Abstract
This article details two approaches to compute barycenters of measures using 1-D Wasserstein distances along radial projections of the input measures. The first method makes use of the Radon transform of the measures, and the second is the solution of a convex optimization problem over the space of measures. We show several properties of these barycenters and explain their relationship. We show numerical approximation schemes based on a discrete Radon transform and on the resolution of a non-convex optimization problem. We explore the respective merits and drawbacks of each approach on applications to two image processing problems: color transfer and texture mixing.
2013
Facial Performance Enhancement Using Dynamic Shape Space Analysis
Bermano AH, Bradley D, Beeler T, Zünd F, Nowrouzezahrai D, Baran I, Sorkine O, Pfister H, Sumner RW, Bickel B, et al. Facial Performance Enhancement Using Dynamic Shape Space Analysis. In: ACM Transactions On Graphics . ; 2013.Abstract
The facial performance of an individual is inherently rich in subtle deformation and timing details. Although these subtleties make the performance realistic and compelling, they often elude both motion capture and hand animation. We present a technique for adding fine-scale details and expressiveness to low-resolution art-directed facial performances, such as those created manually using a rig, via marker-based capture, by fitting a morphable model to a video, or through Kinect reconstruction using recent faceshift technology. We employ a high-resolution facial performance capture system to acquire a representative performance of an individual in which he or she explores the full range of facial expressiveness. From the captured data, our system extracts an expressiveness model that encodes subtle spatial and temporal deformation details specific to that particular individual. Once this model has been built, these details can be transferred to low-resolution artdirected performances. We demonstrate results on various forms of input; after our enhancement, the resulting animations exhibit the same nuances and fine spatial details as the captured performance, with optional temporal enhancement to match the dynamics of the actor. Finally, we show that our technique outperforms the current state-of-the-art in example-based facial animation
Image
Segmenting Planar Superpixel Adjacency Graphs w.r.t. Non-planar Superpixel Affinity Graphs
Andres B, Yarkony J, Manjunath BS, Kirchhoff S, Turetken E, Fowlkes CC, Pfister H. Segmenting Planar Superpixel Adjacency Graphs w.r.t. Non-planar Superpixel Affinity Graphs. Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR). 2013.Abstract
We address the problem of segmenting an image into a previously unknown number of segments from the perspective of graph partitioning. Specifically, we consider minimum multicuts of superpixel affinity graphs in which all affinities between non-adjacent superpixels are negative. We propose a relaxation by Lagrangian decomposition and a constrained set of re-parameterizations for which we can optimize exactly and efficiently. Our contribution is to show how the planarity of the adjacency graph can be exploited if the affinity graph is non-planar. We demonstrate the effectiveness of this approach in user-assisted image segmentation and show that the solution of the relaxed problem is fast and the relaxation is tight in practice.
Paper
<p>A Fully-Connected Layered Model of Foreground and Background Flow</p>
Sun D, Wulff J, Sudderth EB, Pfister H, Black MJ.

A Fully-Connected Layered Model of Foreground and Background Flow

. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2013:2451-2458. Paper
Exploring the Connectome - Petascale Volume Visualization of Microscopy Data Streams
Beyer J, Hadwiger M, Al-Awami A, Jeong W, Kasthuri N, Lichtman JW, Pfister H. Exploring the Connectome - Petascale Volume Visualization of Microscopy Data Streams. IEEE Computer Graphics and Applications. 2013;33(4):50-61.Abstract
Recent advances in high-resolution microscopy allow neuroscientists to acquire volume data of neural tissue of extreme size. However, the tremendous resolution and the high complexity of neural structures present big challenges to storage, processing, and visualization at interactive rates. We present a system for interactive exploration of petascale (petavoxel) volumes resulting from high-throughput electron microscopy data streams. Our system can concurrently handle multiple volumes, and also supports the simultaneous visualization of high-resolution voxel segmentation data. We employ a visualization-driven system design that allows us to restrict most computations to a small sub-set of the data. We employ a multi-resolution virtual memory architecture for better scalability than previous approaches and handling of incomplete data. We illustrate the real-world use of our system for a mouse cortex volume of one teravoxel in size, where several hundred neurites as well as synapses have been segmented and labeled.
Paper
ConnectomeExplorer: Query-Guided Visual Analysis of Large Volumetric Neuroscience Data
Beyer J, Al-Awami A, Kasthuri N, Lichtman JW, Pfister H, Hadwiger M. ConnectomeExplorer: Query-Guided Visual Analysis of Large Volumetric Neuroscience Data. IEEE Transactions on Visualization and Computer Graphics. 2013;19(12).Abstract
This paper presents ConnectomeExplorer, an application for the interactive exploration and query-guided visual analysis of large volumetric electron microscopy (EM) data sets in connectomics research. Our system incorporates a knowledge-based query algebra that supports the interactive specification of dynamically evaluated queries, which enable neuroscientists to pose and answer domain-specific questions in an intuitive manner. Queries are built step by step in a visual query builder, building more complex queries from combinations of simpler queries. Our application is based on a scalable volume visualization framework that scales to multiple volumes of several teravoxels each, enabling the concurrent visualization and querying of the original EM volume, additional segmentation volumes, neuronal connectivity, and additional meta data comprising a variety of neuronal data attributes. We evaluate our application on a data set of roughly one terabyte of EM data and 750 GB of segmentation data, containing over 4,000 segmented structures and 1,000 synapses. We demonstrate typical use-case scenarios of our collaborators in neuroscience, where our system has enabled them to answer specific scientific questions using interactive querying and analysis on the full-size data for the first time.
Paper Video
What Makes a Visualization Memorable?
Borkin MA, Vo AA, Bylinskii Z, Isola P, Sunkavalli S, Oliva A, Pfister H. What Makes a Visualization Memorable?. IEEE Transactions on Visualization and Computer Graphics (Proceedings of InfoVis 2013). 2013.Abstract
An ongoing debate in the Visualization community concerns the role that visualization types play in data understanding. In human cognition, understanding and memorability are intertwined. As a first step towards being able to ask questions about impact and effectiveness, here we ask: “What makes a visualization memorable?” We ran the largest scale visualization study to date using 2,070 single-panel visualizations, categorized with visualization type (e.g., bar chart, line graph, etc.), collected from news media sites, government reports, scientific journals, and infographic sources. Each visualization was annotated with additional attributes, including ratings for data-ink ratios and visual densities. Using Amazon’s Mechanical Turk, we collected memorability scores for hundreds of these visualizations, and discovered that observers are consistent in which visualizations they find memorable and forgettable. We find intuitive results (e.g., attributes like color and the inclusion of a human recognizable object enhance memorability) and less intuitive results (e.g., common graphs are less memorable than unique visualization types). Altogether our findings suggest that quantifying memorability is a general metric of the utility of information, an essential step towards determining how to design effective visualizations.
Paper Supplemental Material Experiment Demo Video Slides
Evaluation of Filesystem Provenance Visualization Tools
Borkin MA, Yeh CS, Boyd M, Macko P, Gajos KZ, Seltzer M, Pfister H. Evaluation of Filesystem Provenance Visualization Tools. IEEE Transactions on Visualization and Computer Graphics (Proceedings of InfoVis 2013). 2013.Abstract
Having effective visualizations of filesystem provenance data is valuable for understanding its complex hierarchical structure. The most common visual representation of provenance data is the node-link diagram. While effective for understanding local activity, the node-link diagram fails to offer a high-level summary of activity and inter-relationships within the data. We present a new tool, InProv, which displays filesystem provenance with an interactive radial-based tree layout. The tool also utilizes a new time-based hierarchical node grouping method for filesystem provenance data we developed to match the user’s mental model and make data exploration more intuitive. We compared InProv to a conventional node-link based tool, Orbiter, in a quantitative evaluation with real users of filesystem provenance data including provenance data experts, IT professionals, and computational scientists. We also compared in the evaluation our new node grouping method to a conventional method. The results demonstrate that InProv results in higher accuracy in identifying system activity than Orbiter with large complex data sets. The results also show that our new time-based hierarchical node grouping method improves performance in both tools, and participants found both tools significantly easier to use with the new time-based node grouping method. Subjective measures show that participants found InProv to require less mental activity, less physical activity, less work, and is less stressful to use. Our study also reveals one of the first cases of gender differences in visualization; both genders had comparable performance with InProv, but women had a significantly lower average accuracy (56%) compared to men (70%) with Orbiter.
Paper Video Slides
LineUp: Visual Analysis of Multi-Attribute Rankings
Gratzl S, Lex A, Gehlenborg N, Pfister H, Streit M. LineUp: Visual Analysis of Multi-Attribute Rankings. IEEE Transactions on Visualization and Computer Graphics (InfoVis '13) [Internet]. 2013;19(12):2277-2286. Project WebsiteAbstract
Rankings are a popular and universal approach to structuring otherwise unorganized collections of items by computing a rank for each item based on the value of one or more of its attributes. This allows us, for example, to prioritize tasks or to evaluate the performance of products relative to each other. While the visualization of a ranking itself is straightforward, its interpretation is not, because the rank of an item represents only a summary of a potentially complicated relationship between its attributes and those of the other items. It is also common that alternative rankings exist which need to be compared and analyzed to gain insight into how multiple heterogeneous attributes affect the rankings. Advanced visual exploration tools are needed to make this process efficient. In our paper we present a comprehensive analysis of requirements for the visualization of multi-attribute rankings. Based on these considerations, we propose LineUp - a novel and scalable visualization technique that uses bar charts. This interactive technique supports the ranking of items based on multiple heterogeneous attributes with different scales and semantics. It enables users to interactively combine attributes and flexibly refine parameters to explore the effect of changes in the attribute combination. This process can be employed to derive actionable insights as to which attributes of an item need to be modified in order for its rank to change. Additionally, through integration of slope graphs, LineUp can also be used to compare multiple alternative rankings on the same set of items, for example, over time or across different attribute combinations. We evaluate the effectiveness of the proposed multi-attribute visualization technique in a qualitative study. The study shows that users are able to successfully solve complex ranking tasks in a short period of time.
Paper Video
Entourage: Visualizing Relationships between Biological Pathways using Contextual Subsets
Lex A, Partl C, Kalkofen D, Streit M, Wasserman AM, Gratzl S, Schmalstieg D, Pfister H. Entourage: Visualizing Relationships between Biological Pathways using Contextual Subsets. IEEE Transactions on Visualization and Computer Graphics (InfoVis '13) [Internet]. 2013;19(12):2536-2545. Project WebsiteAbstract
Biological pathway maps are highly relevant tools for many tasks in molecular biology. They reduce the complexity of the overall biological network by partitioning it into smaller manageable parts. While this reduction of complexity is their biggest strength, it is, at the same time, their biggest weakness. By removing what is deemed not important for the primary function of the pathway, biologists lose the ability to follow and understand cross-talks between pathways. Considering these cross-talks is, however, critical in many analysis scenarios, such as judging effects of drugs. In this paper we introduce Entourage, a novel visualization technique that provides contextual information lost due to the artificial partitioning of the biological network, but at the same time limits the presented information to what is relevant to the analyst's task. We use one pathway map as the focus of an analysis and allow a larger set of contextual pathways. For these context pathways we only show the contextual subsets, i.e., the parts of the graph that are relevant to a selection. Entourage suggests related pathways based on similarities and highlights parts of a pathway that are interesting in terms of mapped experimental data. We visualize interdependencies between pathways using stubs of visual links, which we found effective yet not obtrusive. By combining this approach with visualization of experimental data, we can provide domain experts with a highly valuable tool. We demonstrate the utility of Entourage with case studies conducted with a biochemist who researches the effects of drugs on pathways. We show that the technique is well suited to investigate interdependencies between pathways and to analyze, understand, and predict the effect that drugs have on different cell types.
Paper Video
Example-Based Video Color Grading
Bonneel N, Sunkavalli K, Paris S, Pfister H. Example-Based Video Color Grading. ACM Transactions on Graphics (Proc. ACM SIGGRAPH). 2013;32(4).Abstract
In most professional cinema productions, the color palette of the movie is painstakingly adjusted by a team of skilled colorists -- through a process referred to as color grading -- to achieve a certain visual look. The time and expertise required to grade a video makes it difficult for amateurs to manipulate the colors of their own video clips. In this work, we present a method that allows a user to transfer the color palette of a model video clip to their own video sequence. We estimate a per-frame color transform that maps the color distributions in the input video sequence to that of the model video clip. Applying this transformation naively leads to artifacts such as bleeding and flickering. Instead, we propose a novel differential-geometry-based scheme that interpolates these transformations in a manner that minimizes their curvature, similarly to curvature flows. In addition, we automatically determine a set of keyframes that best represent this interpolated transformation curve, and can be used subsequently, to manually refine the color grade. We show how our method successfully transfer color palettes between videos for a range of visual styles and a number of input video clips.
Paper Images Video
A Collaborative Digital Pathology System for Multi-Touch Mobile and Desktop Computing Platforms
Jeong W, Schneider J, Hansen A, Lee M, Faulkner-Jones BE, Hecht J, Najarian R, Yee E, Lichtrman J, Pfister H. A Collaborative Digital Pathology System for Multi-Touch Mobile and Desktop Computing Platforms. Computer Graphics Forum. 2013.Abstract
Collaborative slide image viewing systems are becoming increasingly important in pathology applications such as telepathology and E-learning. Despite rapid advances in computing and imaging technology, current digital pathology systems have limited performance with respect to remote viewing of whole slide images on desktop or mobile computing devices. In this paper we present a novel digital pathology client-server systems that supports collaborative viewing of multi-plane whole slide images over standard networks using multi-touch enabled clients. Our system is built upon a standard HTTP web server and a MySQL database to allow multiple clients to exchange image and metadata concurrently. We introduce a domain-specific image-stack compression method that leverages real-time hardware decoding on mobile devices. It adaptively encodes image stacks in a decorrelated color space to achieve extremely low bitrates (0.8 bpp) with very low loss of image quality. We evaluate the image quality of our compression method and the performance of our system for diagnosis with an in-depth user study.
Paper Video
2012
<p>Video Snapshots: Creating High-Quality Images from Video Clips</p>
Sunkavalli K, Joshi N, Kang SB, Cohen M, Pfister H.

Video Snapshots: Creating High-Quality Images from Video Clips

. IEEE Transactions on Visualization and Computer Graphics. 2012;18(11):1868-1879.Abstract
We describe a unified framework for generating a single high-quality still image ("snapshot”) from a short video clip. Our system allows the user to specify the desired operations for creating the output image, such as super resolution, noise and blur reduction, and selection of best focus. It also provides a visual summary of activity in the video by incorporating saliency-based objectives in the snapshot formation process. We show examples on a number of different video clips to illustrate the utility and flexibility of our system.
Paper
Non-causal Temporal Prior for Video Deblocking
Sun D, Liu C. Non-causal Temporal Prior for Video Deblocking. In: European Conferenceon Computer Vision (ECCV). Springer; 2012.Abstract
Real-world video sequences coded at low bit rates suffer from compression artifacts, which are visually disruptive and can cause problems to computer vision algorithms. Unlike the denoising problem where the high frequency components of the signal are present in the noisy observation, most high frequency details are lost during compression and artificial discontinuities arise across the coding block boundaries. In addition to sparse spatial priors that can reduce the blocking artifacts for a single frame, temporal information is needed to recover the lost spatial details. However, establishing accurate temporal correspondences from the compressed videos is challenging because of the loss of high frequency details and the increase of false blocking artifacts. In this paper, we propose a non-causal temporal prior model to reduce video compression artifacts by propagating information from adjacent frames and iterating between image reconstruction and motion estimation. Experimental results on real-world sequences demonstrate that the deblocked videos by the proposed system have marginal statistics of high frequency components closer to those of the original ones, and are better input for standard edge and corner detectors than the coded ones.
Paper
Interactive Volume Exploration of Petascale Microscopy Data Streams Using a Visualization-Driven Virtual Memory Approach
Hadwiger M, Beyer J, Jeong W, Pfister H. Interactive Volume Exploration of Petascale Microscopy Data Streams Using a Visualization-Driven Virtual Memory Approach. IEEE Transactions on Visualization and Computer Graphics (SciVis'12). 2012;18(2):2285-2294.Abstract
This paper presents the first volume visualization system that scales to petascale volumes imaged as a continuous stream of high-resolution electron microscopy images. Our architecture scales to dense, anisotropic petascale volumes because it: (1) decouples construction of the 3D multi-resolution representation required for visualization from data acquisition, and (2) decouples sample access time during ray-casting from the size of the multi-resolution hierarchy. Our system is designed around a scalable multi-resolution virtual memory architecture that handles missing data naturally, does not pre-compute any 3D multi-resolution representation such as an octree, and can accept a constant stream of 2D image tiles from the microscopes. A novelty of our system design is that it is visualization-driven: we restrict most computations to the visible volume data. Leveraging the virtual memory architecture, missing data are detected during volume ray-casting as cache misses, which are propagated backwards for on-demand out-of-core processing. 3D blocks of volume data are only constructed from 2D microscope image tiles when they have actually been accessed during ray-casting. We extensively evaluate our system design choices with respect to scalability and performance, compare to previous best-of-breed systems, and illustrate the effectiveness of our system for real microscopy data from neuroscience.
Video Paper
A Practical Framework for Sharing and Rendering Real-World Bidirectional Scattering Distribution Functions
Ward G, Kurt M, Bonneel N. A Practical Framework for Sharing and Rendering Real-World Bidirectional Scattering Distribution Functions. Lawrence Berkeley National Laboratory Technical Report LBNL-5954E; 2012.Abstract
The utilization of real-world materials has been hindered by a lack of standards for sharing and interpreting measured data. This paper presents an XML representation and an Open Source C library to support bidirectional scattering distribution functions (BSDFs) in data-driven lighting simulation and rendering applications.The library provides for the efficient representation, query, and Monte Carlo sampling of arbitrary BSDFs in amodel-free framework. Currently, we support two BSDF data representations: one using a fixed subdivision of thehemisphere, and one with adaptive density. The fixed type has advantages for certain matrix operations, while theadaptive type can more accurately represent highly peaked data. We discuss advanced methods for data-drivenBSDF rendering for both types, including the proxy of detailed geometry to enhance appearance and accuracy.We also present an advanced interpolation method to reduce measured data into these standard representations.We end with our plan for future extensions and sharing of BSDF data.
Paper
Multi-Video Browsing and Summarization
Dale K, Shechtman E, Avidan S, Pfister H. Multi-Video Browsing and Summarization. International Workshop on Large-Scale Video Search and Mining. 2012.Abstract
We propose a method for browsing multiple videos with a common theme, such as the result of a search query on a video sharing website, or videos of an event covered by multiple cameras. Given the collection of videos we first align each video with all others. This pairwise video alignment forms the basis of a novel browsing interface, termed the Browsing Companion. It is used to play a primary video and, in addition as thumbnails, other video clips that are temporally synchronized with it. The user can, at any time, click on one of the thumbnails to make it the primary. We also show that video alignment can be used for other applications such as automatic highlight detection and multivideo summarization.
Paper
Visualization for the Physical Sciences
Lipsa DR, Laramee RS, Cox SJ, Roberts JC, Walker R, Borkin MA, Pfister H. Visualization for the Physical Sciences. Computer Graphics Forum. 2012;31:2317-2347.Abstract
Close collaboration with other scientific fields is an important goal for the visualization community. Yet engaging in a scientific collaboration can be challenging. The physical sciences, namely astronomy, chemistry, earth sciences and physics, exhibit an extensive range of research directions, providing exciting challenges for visualization scientists and creating ample possibilities for collaboration. We present the first survey of its kind that provides a comprehensive view of existing work on visualization for the physical sciences. We introduce novel classification schemes based on application area, data dimensionality and main challenge addressed, and apply these classifications to each contribution from the literature. Our survey helps in understanding the status of current research and serves as a useful starting point for those interested in visualization for the physical sciences.
Paper
Fabricating Articulated Characters from Skinned Meshes
Baecher M, Bickel B, James DL, Pfister H. Fabricating Articulated Characters from Skinned Meshes. ACM Transactions on Graphics (Proc. ACM SIGGRAPH). 2012;31.Abstract
Articulated deformable characters are widespread in computer animation. Unfortunately, we lack methods for their automatic fabrication using modern additive manufacturing (AM) technologies. We propose a method that takes a skinned mesh as input, then estimates a fabricatable single-material model that approximates the 3D kinematics of the corresponding virtual articulated character in a piecewise linear manner. We first extract a set of potential joint locations. From this set, together with optional, user-specified range constraints, we then estimate mechanical friction joints that satisfy inter-joint non-penetration and other fabrication constraints. To avoid brittle joint designs, we place joint centers on an approximate medial axis representation of the input geometry, and maximize each joint’s minimal cross-sectional area. We provide several demonstrations, manufactured as single, assembled pieces using 3D printers.
Paper Video