Implements functions for ground truth estimation from the annotations of multiple experts. Based on SimpleITK.
exp_dir = Path('gt_tmp')
url = 'https://github.com/matjesg/deepflash2/releases/download/sample_data/'
download_sample_data(url, 'wue1_cFOS_expert_segmentation_samples.zip', exp_dir , extract=True)
files=['0004_mask.png', '0001_mask.png']
experts = [f'expert_{i}' for i in range(1,6)]
100.60% [229376/228011 00:00<00:00]

Helper Functions

Installing SimpleITK, which is not a dependency of deepflash2.

import_sitk[source]

import_sitk()

import_sitk();

Ground Truth Estimation

Simultaneous truth and performance level estimation (STAPLE)

The STAPLE algorithm considers a collection of segmentations and computes a probabilistic estimate of the true segmentation and a measure of the performance level represented by each segmentation.

Source: Warfield, Simon K., Kelly H. Zou, and William M. Wells. "Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation." IEEE transactions on medical imaging 23.7 (2004): 903-921

def staple(segmentations, foregroundValue = 1, threshold = 0.5):
    'STAPLE: Simultaneous Truth and Performance Level Estimation with simple ITK'
    sitk = import_sitk()
    segmentations = [sitk.GetImageFromArray(x) for x in segmentations]
    STAPLE_probabilities = sitk.STAPLE(segmentations)
    STAPLE = STAPLE_probabilities > threshold
    #STAPLE = sitk.GetArrayViewFromImage(STAPLE)
    return sitk.GetArrayFromImage(STAPLE)

staple_multi_label[source]

staple_multi_label(segmentations, label_undecided_pixel=1)

STAPLE: Simultaneous Truth and Performance Level Estimation with simple ITK

for f in files:
    for i in range(3,7):
        experts = [f'expert_{j}' for j in range(1,i)]
        segmentations = [_read_msk(exp_dir/exp/f) for exp in experts]
        out_staple = staple(segmentations)
        out_staple_multi = staple_multi_label(segmentations)
        test_eq(out_staple, out_staple_multi)

Majority Voting

Use majority voting to obtain the reference segmentation. Note that this filter does not resolve ties. In case of ties it will assign the backgtound label (0) to the result.

m_voting[source]

m_voting(segmentations, labelForUndecidedPixels=0)

Majority Voting from simple ITK Label Voting

GT Estimator

Class for ground truth estimation

class GTEstimator[source]

GTEstimator(exp_dir='expert_segmentations', config=None, path=None, cmap='viridis', verbose=1) :: GetAttr

Class for ground truth estimation

t = GTEstimator(exp_dir=exp_dir, config=Config(instance_segmentation_metrics=True));

Found 5 unique segmentation mask(s) from 5 expert(s)

t.show_data(files=files);

t.gt_estimation()
t.show_gt(files=files)

Starting ground truth estimation - STAPLE
100.00% [5/5 00:11<00:00] 
2022-06-08 08:31:38,730 [INFO] WRITING LOG OUTPUT TO /home/magr/.cellpose/run.log
dice_score mean_average_precision average_precision_at_iou_50
expert_1 0.610313 0.258657 0.466667
expert_2 0.857131 0.499589 0.739837
expert_3 0.682427 0.258915 0.471963
expert_4 0.822903 0.369849 0.770992
expert_5 0.850710 0.474451 0.782258
average 0.764697 0.372292 0.646343
dice_score mean_average_precision average_precision_at_iou_50
expert_1 0.615232 0.209793 0.384615
expert_2 0.831553 0.330434 0.531034
expert_3 0.701515 0.232453 0.425926
expert_4 0.854105 0.447582 0.727941
expert_5 0.800831 0.298439 0.638298
average 0.760647 0.303740 0.541563 
t.gt_estimation(method='majority_voting', save_dir=exp_dir/'mv_test')
t.show_gt(method='majority_voting', max_n=2)

Starting ground truth estimation - majority_voting
100.00% [5/5 00:05<00:00]
dice_score mean_average_precision average_precision_at_iou_50
expert_1 0.710698 0.314402 0.532110
expert_2 0.837439 0.398583 0.699187
expert_3 0.619365 0.169128 0.391509
expert_4 0.805809 0.416399 0.632353
expert_5 0.860788 0.450650 0.786885
average 0.766820 0.349832 0.608409
dice_score mean_average_precision average_precision_at_iou_50
expert_1 0.676638 0.341097 0.538462
expert_2 0.845505 0.469134 0.833333
expert_3 0.591542 0.169529 0.390698
expert_4 0.861491 0.543565 0.846154
expert_5 0.893066 0.647050 0.863636
average 0.773649 0.434075 0.694457 
shutil.rmtree(exp_dir)