High-throughput Analysis of Large Microscopy Image Datasets on CPU-GPU Cluster Platforms

• Type of Material: Article
• Authors: George Teodoro, Emory University; Tony Pan, Emory University; Tahsin M. Kurc, Emory University; Jun Kong, Emory University; Lee Cooper, Emory University; Norbert Podhorszki, Oak Ridge National Laboratory; Scott Klasky, Oak Ridge National Laboratory; Joel Saltz, Emory University
• Language: English
• Date: 2013-05-01
• Publisher: IEEE Xplore
• Publication Version: Author Accepted Manuscript (After Peer Review)
• Copyright Statement: © 2013, IEEE.
• Final Published Version (URL): http://dx.doi.org/10.1109/IPDPS.2013.11
• Title of Journal or Parent Work: Proceedings of the International Parallel and Distributed Processing Symposium, IPDPS
• Volume: 2013
• Start Page: 103
• End Page: 114
• Grant/Funding Information: This work was supported in part by HHSN261200800001E from the National Cancer Institute, R24HL085343 from the National Heart Lung and Blood Institute, by R01LM011119-01 and R01LM009239 from the National Library of Medicine, RC4MD005964 from National Institutes of Health, and PHS UL1RR025008 from the Clinical and Translational Science Awards program. This research used resources of the Keeneland Computing Facility at the Georgia Institute of Technology, which is supported by the National Science Foundation under Contract OCI-0910735.
• Keywords: GPGPU; CPUGPU platforms; image segmentation pipelines
• Research Categories: Biology, Bioinformatics; Computer Science; Health Sciences, Radiology

Abstract

Analysis of large pathology image datasets offers significant opportunities for the investigation of disease morphology, but the resource requirements of analysis pipelines limit the scale of such studies. Motivated by a brain cancer study, we propose and evaluate a parallel image analysis application pipeline for high throughput computation of large datasets of high resolution pathology tissue images on distributed CPU-GPU platforms. To achieve efficient execution on these hybrid systems, we have built runtime support that allows us to express the cancer image analysis application as a hierarchical data processing pipeline. The application is implemented as a coarse-grain pipeline of stages, where each stage may be further partitioned into another pipeline of fine-grain operations. The fine-grain operations are efficiently managed and scheduled for computation on CPUs and GPUs using performance aware scheduling techniques along with several optimizations, including architecture aware process placement, data locality conscious task assignment, data prefetching, and asynchronous data copy. These optimizations are employed to maximize the utilization of the aggregate computing power of CPUs and GPUs and minimize data copy overheads. Our experimental evaluation shows that the cooperative use of CPUs and GPUs achieves significant improvements on top of GPU-only versions (up to 1.6×) and that the execution of the application as a set of fine-grain operations provides more opportunities for runtime optimizations and attains better performance than coarser-grain, monolithic implementations used in other works. An implementation of the cancer image analysis pipeline using the runtime support was able to process an image dataset consisting of 36,848 4Kx4K-pixel image tiles (about 1.8TB uncompressed) in less than 4 minutes (150 tiles/second) on 100 nodes of a state-of-the-art hybrid cluster system.