close
Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2016 Jun 15;32(12):i52-i59.
doi: 10.1093/bioinformatics/btw252.

Classifying and segmenting microscopy images with deep multiple instance learning

Oren Z Kraus  1 , Jimmy Lei Ba  2 , Brendan J Frey  1
Affiliations

Classifying and segmenting microscopy images with deep multiple instance learning

Oren Z Kraus et al. Bioinformatics. .

Abstract

Motivation: High-content screening (HCS) technologies have enabled large scale imaging experiments for studying cell biology and for drug screening. These systems produce hundreds of thousands of microscopy images per day and their utility depends on automated image analysis. Recently, deep learning approaches that learn feature representations directly from pixel intensity values have dominated object recognition challenges. These tasks typically have a single centered object per image and existing models are not directly applicable to microscopy datasets. Here we develop an approach that combines deep convolutional neural networks (CNNs) with multiple instance learning (MIL) in order to classify and segment microscopy images using only whole image level annotations.

Results: We introduce a new neural network architecture that uses MIL to simultaneously classify and segment microscopy images with populations of cells. We base our approach on the similarity between the aggregation function used in MIL and pooling layers used in CNNs. To facilitate aggregating across large numbers of instances in CNN feature maps we present the Noisy-AND pooling function, a new MIL operator that is robust to outliers. Combining CNNs with MIL enables training CNNs using whole microscopy images with image level labels. We show that training end-to-end MIL CNNs outperforms several previous methods on both mammalian and yeast datasets without requiring any segmentation steps.

Availability and implementation: Torch7 implementation available upon request.

Contact: oren.kraus@mail.utoronto.ca.

PubMed Disclaimer

Figures

Fig. 1.
Fig. 1.
Convolutional MIL model. g(·) is global pooling function that aggregates instance probabilities pij
Fig. 2.
Fig. 2.
MIL Pooling functions. Top, pooling function activations by ratio of feature map activated (pij¯). Bottom, activation functions learned by Noisy-AND a10 (nand_a = 10.0) for different classes of the breast cancer dataset
Fig. 3.
Fig. 3.
Class feature map probabilities for hand written digit sample. Class specific feature map probabilities (Pi) overlaid on a sample from the cluttered hand written digit dataset labelled as five. The model successfully classifies regions with fives, and is not sensitive to the background or distractors
Fig. 4.
Fig. 4.
Breast cancer screen. Left, sample full resolution image with epithelial MOA. Right, samples of segmented cells sampled from 12 MOA categories
Fig. 5.
Fig. 5.
Yeast protein localization screen. Left, sample full resolution image with budneck, budtip, cell periphery, and cytoplasm protein localizations. Right, segmented cells with protein localizations manually labelled in Chong et al. (2015)
Fig. 6.
Fig. 6.
Localizing cells with Jacobian maps. Top, yeast cells tagged with a protein that has cell cycle dependent localizations and corresponding Jacobian maps generated from class specific feature maps. Bottom, segmentation by thresholding Jacobian maps and de-noising with loopy bp
See this image and copyright information in PMC

References

    1. Altschuler S.J., Wu L.F. (2010) Cellular heterogeneity: do differences make a difference? Cell, 141, 559–563. - PMC - PubMed
    1. Breker M. et al. (2013) A novel single-cell screening platform reveals proteome plasticity during yeast stress responses. J. Cell Biol., 200, 839–850. - PMC - PubMed
    1. Carpenter A.E. et al. (2006) Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol., 7, R100. - PMC - PubMed
    1. Chen L.C. et al. (2014) Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv Preprint arXiv, 1412, 7062. - PubMed
    1. Chong Y.T. et al. (2015) Yeast proteome dynamics from single cell imaging and automated analysis. Cell, 161, 1413–1424. - PubMed