Image segmentation quality measurements

For determining how good a segmentation algorithm is, and to compare different algorithms, we need a metric. A common metric is the Jaccard Index, which is a measure of overlap between a reference segmentation and the segmentation an algorithm produced for example. If we work with label images, a fair method is to determine the overlap of every annotated object with the most overlapping object in the automatic segmentation. If we average this value over all annotated objects, we receive the Sparse Jaccard Index as defined in The Segmentation Game.

import os
import napari
import napari_segment_blobs_and_things_with_membranes as nsbatwm
from the_segmentation_game import metrics
from skimage.io import imread

To demonstate this, we pick a random example image from the BBBC007 dataset (Jones et al., Proc. ICCV Workshop on Computer Vision for Biomedical Image Applications, 2005), available from the Broad Bioimage Benchmark Collection [Ljosa et al., Nature Methods, 2012].

image_folder = "../../data/BBBC007_batch/"
sparse_annotation_folder = "../../data/BBBC007_sparse_instance_annotation/"

test_image_filename = "17P1_POS0013_D_1UL.tif"

For visualizing the image and the corresponding manual annotation, we use napari.

viewer = napari.Viewer()

The example image

image = imread(folder + test_image_filename)

viewer.add_image(image)

napari.utils.nbscreenshot(viewer)

The manual annotation

sparse_labels = imread(sparse_annotation_folder + test_image_filename)

viewer.add_labels(sparse_labels)

napari.utils.nbscreenshot(viewer)

# hide last labels layer
viewer.layers[-1].visible = False

The automatic segmentation

For demonstration purposes, we use background-subtraction and Voronoi-Otsu-Labeling to segment the nuclei in this image automatically.

def my_segmentation_algorithm(input_image):

    # background subtraction
    background_subtracted = nsbatwm.white_tophat(input_image, radius = 10)
    
    # instance segmenation / labeling
    labels_result = nsbatwm.voronoi_otsu_labeling(background_subtracted, spot_sigma=5, outline_sigma=1)

    return labels_result

labels = my_segmentation_algorithm(image)

viewer.add_labels(labels)

napari.utils.nbscreenshot(viewer)

Quality estimation: Sparse Jaccard Index

From the two label images loaded and produced above we can compute the sparse Jaccard Index.

metrics.jaccard_index_sparse(sparse_labels, labels)

0.8357392602053431

Exercise

Use the following for-loop and code snippets from above to compute the segmentation quality of all images in the folder. Provide the average quality over all images.

for image_filename in os.listdir(image_folder):
    print(image_folder + image_filename)

../../data/BBBC007_batch/17P1_POS0013_D_1UL.tif
../../data/BBBC007_batch/20P1_POS0005_D_1UL.tif
../../data/BBBC007_batch/20P1_POS0007_D_1UL.tif
../../data/BBBC007_batch/20P1_POS0010_D_1UL.tif
../../data/BBBC007_batch/A9 p7d.tif
../../data/BBBC007_batch/AS_09125_040701150004_A02f00d0.tif