torch_em.data.datasets.medical.prostatex
The PROSTATEx dataset contains annotations for prostate lesion and prostate zone segmentation in multi-parametric prostate MRI.
The MRI scans are from the SPIE-AAPM-NCI PROSTATEx challenge (346 patients) and are distributed as DICOM series on TCIA. The segmentation masks are from the third-party "PROSTATEx masks" repository (Cuocolo et al. 2021):
- Lesion masks for 299 lesions of 200 patients, on the axial T2-weighted and on the ADC images. The lesions are stored as instance labels, the instance id is the PROSTATEx finding id.
- Whole gland and zonal masks on the axial T2-weighted images for 204 patients. The zone labels are: 1: peripheral zone, 2: transition zone (the rest of the gland, i.e. transition zone, central zone and anterior fibromuscular stroma).
The label type 'zones_detailed' provides a second, finer set of zonal masks on the axial T2-weighted images
from the ProstateZones release (Hartman et al. 2024), which separates the zones the label type 'zones' merges
and adds the urethra, for 200 patients. Its label ids are 1: peripheral zone, 2: central zone,
3: transition zone, 4: anterior fibromuscular stroma, 5: urethra (see DETAILED_ZONE_IDS). 40 of these
patients were delineated by two readers independently; the second delineation is stored next to the first as
'zones_detailed/t2/labels_reader2'.
NOTE: The ProstateZones masks are distributed without the images they were drawn on and without their label legend. The series was identified from the list of series that its repository ships, and the label ids were assigned by measuring each annotation: the urethra is the smallest structure, the transition zone the largest, the anterior fibromuscular stroma the most anterior one and the central zone the most superior one.
NOTE: This requires the pynrrd python package to read the ProstateZones masks.
This module downloads only the annotated T2 and ADC series from TCIA, stacks them into volumes and stores them
together with the aligned masks in one hdf5 file per patient. The masks were drawn on NIfTI conversions of the DICOM
series (dcm2niix), so the module maps them back onto the DICOM grid and checks the alignment against the NIfTI images
shipped with the masks. The hdf5 groups are '
NOTE: This requires the pydicom python package.
The dataset is located at https://www.cancerimagingarchive.net/collection/prostatex/ and the masks at https://github.com/rcuocolo/PROSTATEx_masks.
This dataset is from the publications https://doi.org/10.1109/TMI.2014.2303821 (PROSTATEx) https://doi.org/10.1016/j.ejrad.2021.109647 (masks) and https://doi.org/10.1038/s41597-024-03945-2 (ProstateZones, CC BY 4.0). The data was released at https://doi.org/10.7937/K9TCIA.2017.MURS5CL. Please cite them if you use this dataset in your research.
1"""The PROSTATEx dataset contains annotations for prostate lesion and prostate zone segmentation 2in multi-parametric prostate MRI. 3 4The MRI scans are from the SPIE-AAPM-NCI PROSTATEx challenge (346 patients) and are distributed as DICOM series 5on TCIA. The segmentation masks are from the third-party "PROSTATEx masks" repository (Cuocolo et al. 2021): 6- Lesion masks for 299 lesions of 200 patients, on the axial T2-weighted and on the ADC images. 7 The lesions are stored as instance labels, the instance id is the PROSTATEx finding id. 8- Whole gland and zonal masks on the axial T2-weighted images for 204 patients. 9 The zone labels are: 1: peripheral zone, 2: transition zone (the rest of the gland, i.e. transition zone, 10 central zone and anterior fibromuscular stroma). 11 12The label type 'zones_detailed' provides a second, finer set of zonal masks on the axial T2-weighted images 13from the ProstateZones release (Hartman et al. 2024), which separates the zones the label type 'zones' merges 14and adds the urethra, for 200 patients. Its label ids are 1: peripheral zone, 2: central zone, 153: transition zone, 4: anterior fibromuscular stroma, 5: urethra (see `DETAILED_ZONE_IDS`). 40 of these 16patients were delineated by two readers independently; the second delineation is stored next to the first as 17'zones_detailed/t2/labels_reader2'. 18 19NOTE: The ProstateZones masks are distributed without the images they were drawn on and without their label 20legend. The series was identified from the list of series that its repository ships, and the label ids were 21assigned by measuring each annotation: the urethra is the smallest structure, the transition zone the largest, 22the anterior fibromuscular stroma the most anterior one and the central zone the most superior one. 23 24NOTE: This requires the pynrrd python package to read the ProstateZones masks. 25 26This module downloads only the annotated T2 and ADC series from TCIA, stacks them into volumes and stores them 27together with the aligned masks in one hdf5 file per patient. The masks were drawn on NIfTI conversions of the DICOM 28series (dcm2niix), so the module maps them back onto the DICOM grid and checks the alignment against the NIfTI images 29shipped with the masks. The hdf5 groups are '<label_type>/<sequence>' with the datasets 'raw' and 'labels' 30(e.g. 'lesions/t2/raw' and 'lesions/t2/labels'); the zones group additionally contains the binary 31whole-gland mask ('zones/t2/prostate'). Note that for some patients the lesion and the zone masks were drawn on 32different T2 series, hence the raw data is stored per group. 33 34NOTE: This requires the pydicom python package. 35 36The dataset is located at https://www.cancerimagingarchive.net/collection/prostatex/ 37and the masks at https://github.com/rcuocolo/PROSTATEx_masks. 38 39This dataset is from the publications https://doi.org/10.1109/TMI.2014.2303821 (PROSTATEx) 40https://doi.org/10.1016/j.ejrad.2021.109647 (masks) and 41https://doi.org/10.1038/s41597-024-03945-2 (ProstateZones, CC BY 4.0). 42The data was released at https://doi.org/10.7937/K9TCIA.2017.MURS5CL. 43Please cite them if you use this dataset in your research. 44""" 45 46import os 47import re 48import csv 49import json 50from glob import glob 51from tqdm import tqdm 52from natsort import natsorted 53from collections import defaultdict 54from typing import Union, Tuple, List, Literal 55 56import numpy as np 57import requests 58 59from torch.utils.data import Dataset, DataLoader 60 61import torch_em 62 63from .. import util 64 65 66MASKS_COMMIT = "21b9dfde9da4f7b719c206fe1ca00ae31d6f5cf3" 67 68URLS = { 69 "images": f"{util.NBIA_API_URL}getSeries?Collection=PROSTATEx", 70 "masks": f"https://github.com/rcuocolo/PROSTATEx_masks/archive/{MASKS_COMMIT}.zip", 71 "zones_detailed": "https://zenodo.org/records/10718469/files/ProstateZones.zip?download=1", 72 "series_list": ( 73 "https://raw.githubusercontent.com/UMU-DDI/ProstateZones/main/Support%20Files/list_of_PROSTATEx_files.txt" 74 ), 75} 76 77CHECKSUMS = { 78 "images": None, # The DICOM series are downloaded individually from TCIA. 79 "masks": None, # GitHub does not guarantee stable archive checksums. 80 "zones_detailed": "5a45816230b4bf94e88527a00754cc0b9329ec2c3459179a3931c11cb574e80a", 81 "series_list": None, # GitHub does not guarantee stable raw file checksums. 82} 83 84ZONE_IDS = {"peripheral_zone": 1, "transition_zone": 2} 85"""The zone ids of the 'zones' labels.""" 86 87DETAILED_ZONE_IDS = { 88 "peripheral_zone": 1, "central_zone": 2, "transition_zone": 3, 89 "anterior_fibromuscular_stroma": 4, "urethra": 5, 90} 91"""The zone ids of the 'zones_detailed' labels.""" 92 93 94def _load_dicom_volume(series_dir): 95 """Stack a DICOM series into a volume with axes (z, y, x) and slices sorted along the slice normal.""" 96 import pydicom 97 98 slices = [pydicom.dcmread(dcm_path) for dcm_path in natsorted(glob(os.path.join(series_dir, "*.dcm")))] 99 slices = [dcm for dcm in slices if hasattr(dcm, "ImagePositionPatient")] 100 orientation = np.array([float(v) for v in slices[0].ImageOrientationPatient]) 101 normal = np.cross(orientation[:3], orientation[3:]) 102 slices.sort(key=lambda dcm: np.dot(np.array([float(v) for v in dcm.ImagePositionPatient]), normal)) 103 return np.stack([dcm.pixel_array for dcm in slices]) 104 105 106def _nifti_to_dicom_grid(data): 107 """Map a NIfTI array (x, y, z; dcm2niix conversion of an axial series) onto the DICOM grid (z, y, x). 108 109 dcm2niix flips the row axis of axial series (DICOM rows run anterior -> posterior, the NIfTI y axis runs 110 posterior -> anterior), so the array is transposed and flipped along y. 111 """ 112 return np.asarray(data).transpose(2, 1, 0)[:, ::-1] 113 114 115def _load_nifti_on_dicom_grid(path): 116 import nibabel as nib 117 118 nifti = nib.load(path) 119 assert nib.aff2axcodes(nifti.affine) == ("L", "A", "S"), f"Unexpected axes for {path}" 120 return _nifti_to_dicom_grid(nifti.dataobj) 121 122 123def _parse_image_name(name): 124 """Parse a name like 'ProstateX-0000_t2_tse_tra_4' into the patient id and the DICOM series number.""" 125 patient_id, _, series_number = re.match(r"(ProstateX-\d+)_(.*)_(\d+)$", name.strip()).groups() 126 return patient_id, int(series_number) 127 128 129def _read_image_lists(mask_dir): 130 """Read the lists of the DICOM series the masks were drawn on. 131 132 Returns a dict {(label_type, sequence, patient_id): (series_number, nifti_image_path)}. 133 """ 134 image_series = {} 135 with open(os.path.join(mask_dir, "lesions", "Image_list.csv"), "r") as f: 136 for row in csv.DictReader(f): 137 for sequence in ("T2", "ADC"): 138 patient_id, series_number = _parse_image_name(row[sequence]) 139 image_path = os.path.join(mask_dir, "lesions", "Images", sequence, f"{row[sequence].strip()}.nii.gz") 140 image_series[("lesions", sequence.lower(), patient_id)] = (series_number, image_path) 141 with open(os.path.join(mask_dir, "prostate", "image_list.csv"), "r") as f: 142 for row in csv.DictReader(f): 143 patient_id, series_number = _parse_image_name(row["T2"]) 144 image_path = os.path.join(mask_dir, "prostate", "Images", f"{row['T2'].strip()}.nii.gz") 145 image_series[("zones", "t2", patient_id)] = (series_number, image_path) 146 return image_series 147 148 149def _nrrd_on_dicom_grid(path): 150 """Map a ProstateZones nrrd array (x, y, z) onto the DICOM grid (z, y, x). 151 152 These are stored with the geometry of the series they were drawn on, so unlike the dcm2niix NIfTI 153 conversions of the lesion and zone masks they only have to be transposed and not flipped. 154 """ 155 import nrrd 156 157 data, _ = nrrd.read(path) 158 return np.asarray(data).transpose(2, 1, 0) 159 160 161def _read_detailed_zone_series(path): 162 """Read the axial T2 series number of every patient from the ProstateZones series list. 163 164 Each line names the series of one patient, starting with the axial T2 series, e.g. 165 'ProstateX-0000:4.000000-t2tsetra-00702:3.000000-t2tsesag-87368:...'. 166 """ 167 series_numbers = {} 168 with open(os.path.join(path, "list_of_PROSTATEx_files.txt"), "r") as f: 169 for line in f: 170 fields = line.strip().split(":") 171 if len(fields) < 2: 172 continue 173 match = re.match(r"([\d.]+)-t2tsetra-", fields[1]) 174 if match is not None: 175 series_numbers[fields[0]] = int(float(match.group(1))) 176 return series_numbers 177 178 179def _get_detailed_zone_masks(zones_dir, patient_id): 180 """Get the ProstateZones masks of a patient, which are one file or one file per reader.""" 181 number = patient_id.split("-")[-1] 182 single = os.path.join(zones_dir, "Singles", f"Seg-{number}.nrrd") 183 if os.path.exists(single): 184 return [single] 185 duplicates = [os.path.join(zones_dir, "Duplicates", reader, f"Seg-{number}_{reader}.nrrd") 186 for reader in ("R1", "R2")] 187 return duplicates if all(os.path.exists(mask_path) for mask_path in duplicates) else [] 188 189 190def _load_detailed_zone_labels(mask_paths, shape): 191 """Load the ProstateZones masks of a patient, one per reader.""" 192 labels = [_nrrd_on_dicom_grid(mask_path) for mask_path in mask_paths] 193 if any(mask.shape != shape for mask in labels): 194 return None 195 return [mask.astype("uint8") for mask in labels] 196 197 198def _get_series_metadata(path, download): 199 """Get the metadata of all series in the collection from the NBIA REST API.""" 200 metadata_path = os.path.join(path, "prostatex_series.json") 201 if not os.path.exists(metadata_path): 202 if not download: 203 raise RuntimeError(f"Cannot find the data at {path}, but download was set to False.") 204 response = requests.get(URLS["images"]) 205 response.raise_for_status() 206 with open(metadata_path, "w") as f: 207 json.dump(response.json(), f, indent=2) 208 209 with open(metadata_path, "r") as f: 210 return json.load(f) 211 212 213def _find_mask(directory, patient_id, suffix=""): 214 """Find the mask of a patient in one of the folders with the gland and zone masks. 215 216 The file names are inconsistent in the mask repository (e.g. 'ProstateX-080.nii.gz' instead of 217 'ProstateX-0080.nii.gz' and 'VOLUME-0202_pz.nii.gz' instead of 'ProstateX-0202_pz.nii.gz'), 218 so the masks are matched via the patient number rather than via the file name. 219 """ 220 number = int(patient_id.split("-")[-1]) 221 for mask_path in natsorted(glob(os.path.join(directory, f"*{suffix}.nii.gz"))): 222 if int(re.search(r"(\d+)", os.path.basename(mask_path)).group(1)) == number: 223 return mask_path 224 return None 225 226 227def _get_mask_paths(mask_dir, label_type, sequence, patient_id): 228 """Get the mask files of a patient for the given label type and sequence.""" 229 if label_type == "lesions": 230 return natsorted(glob(os.path.join(mask_dir, "lesions", "Masks", sequence.upper(), f"{patient_id}-*"))) 231 232 prostate_dir = os.path.join(mask_dir, "prostate") 233 mask_paths = [ 234 _find_mask(os.path.join(prostate_dir, "mask_prostate"), patient_id), 235 _find_mask(os.path.join(prostate_dir, "mask_pz"), patient_id, "_pz"), 236 _find_mask(os.path.join(prostate_dir, "mask_tz"), patient_id, "_tz"), 237 ] 238 return mask_paths if all(mask_path is not None for mask_path in mask_paths) else [] 239 240 241def _find_series(dicom_dir, candidates, image_path, mask_shape): 242 """Find the downloaded DICOM series that matches the NIfTI image the masks were drawn on. 243 244 The NIfTI images that ship with the masks are used to identify the series: the stacked DICOM volume has to 245 match the NIfTI image voxel by voxel. For the few patients without a NIfTI image in the mask repository the 246 series is identified by the shape of the masks instead. 247 248 Returns the volume on the DICOM grid or None if no candidate series matches. 249 """ 250 reference = _load_nifti_on_dicom_grid(image_path) if os.path.exists(image_path) else None 251 for series in candidates: 252 series_dir = os.path.join(dicom_dir, series["SeriesInstanceUID"]) 253 if not os.path.exists(series_dir): 254 continue 255 volume = _load_dicom_volume(series_dir) 256 if reference is None: 257 if volume.shape == mask_shape: 258 return volume 259 elif volume.shape == reference.shape and np.array_equal(volume, reference.astype(volume.dtype)): 260 return volume 261 return None 262 263 264def _load_lesion_labels(mask_paths, patient_id, shape): 265 """Combine the per-lesion masks of a patient into an instance segmentation (id = finding id).""" 266 labels = np.zeros(shape, dtype="uint8") 267 pattern = re.compile(rf"{patient_id}-Finding(\d+)-.*ROI\.nii\.gz", re.IGNORECASE) 268 finding_ids = [int(pattern.match(os.path.basename(mask_path)).group(1)) for mask_path in mask_paths] 269 n_masks = 0 270 for mask_path, finding_id in zip(mask_paths, finding_ids): 271 # The finding ids start at 1, except for a single lesion in the current release of the masks 272 # (ProstateX-0005), which is called 'Finding0' and gets the next free id instead of the background id. 273 if finding_id == 0: 274 finding_id = max(finding_ids) + 1 275 mask = _load_nifti_on_dicom_grid(mask_path) 276 if mask.shape != shape: # A few masks were drawn on a different series and cannot be used. 277 continue 278 labels[mask > 0] = finding_id 279 n_masks += 1 280 return labels if n_masks > 0 else None 281 282 283def _load_zone_labels(mask_paths, shape): 284 """Combine the peripheral and transition zone masks into a semantic segmentation.""" 285 prostate, pz, tz = [_load_nifti_on_dicom_grid(mask_path) > 0 for mask_path in mask_paths] 286 if any(mask.shape != shape for mask in (prostate, pz, tz)): 287 return None, None 288 zones = np.zeros(shape, dtype="uint8") 289 zones[tz] = ZONE_IDS["transition_zone"] 290 zones[pz] = ZONE_IDS["peripheral_zone"] 291 return zones, prostate.astype("uint8") 292 293 294def _preprocess_prostatex(dicom_dir, mask_dir, zones_dir, series_metadata, image_series, preprocessed_dir): 295 import h5py 296 297 series_by_number = defaultdict(list) 298 for series in series_metadata: 299 series_by_number[(series["PatientID"], int(series["SeriesNumber"]))].append(series) 300 301 groups_per_patient = defaultdict(list) 302 for (label_type, sequence, patient_id), (series_number, image_path) in image_series.items(): 303 groups_per_patient[patient_id].append((label_type, sequence, series_number, image_path)) 304 305 os.makedirs(preprocessed_dir, exist_ok=True) 306 for patient_id, groups in tqdm(sorted(groups_per_patient.items()), desc="Preprocess PROSTATEx"): 307 out_path = os.path.join(preprocessed_dir, f"{patient_id}.h5") 308 # Groups that are already stored are kept, so that a file written by an earlier version of this 309 # module gains the groups it is missing instead of being recomputed. 310 if os.path.exists(out_path): 311 with h5py.File(out_path, "r") as f: 312 stored = {key for key in f if isinstance(f[key], h5py.Group)} 313 groups = [group for group in groups if group[0] not in stored] 314 if not groups: 315 continue 316 317 datasets = {} 318 for label_type, sequence, series_number, image_path in groups: 319 if label_type == "zones_detailed": 320 mask_paths = _get_detailed_zone_masks(zones_dir, patient_id) 321 else: 322 mask_paths = _get_mask_paths(mask_dir, label_type, sequence, patient_id) 323 if not mask_paths: # Some patients only have masks for one of the label types. 324 continue 325 if label_type == "zones_detailed": 326 mask_shape = _nrrd_on_dicom_grid(mask_paths[0]).shape 327 else: 328 mask_shape = _load_nifti_on_dicom_grid(mask_paths[0]).shape 329 candidates = series_by_number[(patient_id, series_number)] 330 volume = _find_series(dicom_dir, candidates, image_path, mask_shape) 331 if volume is None: 332 raise RuntimeError(f"No DICOM series matches the masks of {patient_id} ({label_type}, {sequence}).") 333 if label_type == "lesions": 334 labels = _load_lesion_labels(mask_paths, patient_id, volume.shape) 335 if labels is None: 336 continue 337 elif label_type == "zones_detailed": 338 reader_labels = _load_detailed_zone_labels(mask_paths, volume.shape) 339 if reader_labels is None: 340 continue 341 labels = reader_labels[0] 342 # 40 of the patients were delineated by two readers independently. 343 if len(reader_labels) > 1: 344 datasets[f"{label_type}/{sequence}/labels_reader2"] = reader_labels[1] 345 else: 346 labels, prostate = _load_zone_labels(mask_paths, volume.shape) 347 if labels is None: 348 continue 349 datasets[f"{label_type}/{sequence}/prostate"] = prostate 350 datasets[f"{label_type}/{sequence}/raw"] = volume 351 datasets[f"{label_type}/{sequence}/labels"] = labels 352 353 if not datasets: 354 continue 355 if os.path.exists(out_path): 356 with h5py.File(out_path, "a") as f: 357 for key, data in datasets.items(): 358 f.create_dataset(key, data=data, compression="gzip") 359 else: 360 tmp_path = out_path + ".tmp" 361 with h5py.File(tmp_path, "w") as f: 362 for key, data in datasets.items(): 363 f.create_dataset(key, data=data, compression="gzip") 364 os.rename(tmp_path, out_path) 365 366 367def get_prostatex_data(path: Union[os.PathLike, str], download: bool = False) -> str: 368 """Download the PROSTATEx dataset. 369 370 The download is resumable: series that were already downloaded and patients that were already converted are 371 skipped when the function is called again. 372 373 Args: 374 path: Filepath to a folder where the data is downloaded for further processing. 375 download: Whether to download the data if it is not present. 376 377 Returns: 378 Filepath where the preprocessed data is stored. 379 """ 380 os.makedirs(path, exist_ok=True) 381 preprocessed_dir = os.path.join(path, "preprocessed") 382 383 # Download the masks. 384 mask_dir = os.path.join(path, f"PROSTATEx_masks-{MASKS_COMMIT}", "Files") 385 if not os.path.exists(mask_dir): 386 zip_path = os.path.join(path, f"PROSTATEx_masks-{MASKS_COMMIT}.zip") 387 util.download_source(path=zip_path, url=URLS["masks"], download=download, checksum=CHECKSUMS["masks"]) 388 util.unzip(zip_path=zip_path, dst=path) 389 390 # Download the ProstateZones annotations and the list of the series they were drawn on. 391 zones_dir = os.path.join(path, "ProstateZones") 392 if not os.path.exists(os.path.join(zones_dir, "Singles")): 393 zip_path = os.path.join(path, "ProstateZones.zip") 394 util.download_source( 395 path=zip_path, url=URLS["zones_detailed"], download=download, checksum=CHECKSUMS["zones_detailed"] 396 ) 397 util.unzip(zip_path=zip_path, dst=zones_dir, remove=False) 398 util.download_source( 399 path=os.path.join(path, "list_of_PROSTATEx_files.txt"), url=URLS["series_list"], 400 download=download, checksum=CHECKSUMS["series_list"], 401 ) 402 403 image_series = _read_image_lists(mask_dir) 404 detailed_series = _read_detailed_zone_series(path) 405 for patient_id, series_number in detailed_series.items(): 406 if _get_detailed_zone_masks(zones_dir, patient_id): 407 # The ProstateZones masks ship without a reference image, so the series is matched by shape. 408 image_series[("zones_detailed", "t2", patient_id)] = (series_number, "") 409 410 n_patients = len({patient_id for _, _, patient_id in image_series}) 411 if len(glob(os.path.join(preprocessed_dir, "*.h5"))) == n_patients: 412 return preprocessed_dir 413 414 # Find the series the masks were drawn on and download them from TCIA. 415 series_metadata = _get_series_metadata(path, download) 416 series_numbers = {(patient_id, number) for (_, _, patient_id), (number, _) in image_series.items()} 417 series_uids = sorted( 418 series["SeriesInstanceUID"] for series in series_metadata 419 if (series["PatientID"], int(series["SeriesNumber"])) in series_numbers 420 ) 421 dicom_dir = os.path.join(path, "dicom") 422 if download: # Series that were downloaded already are skipped. 423 util.download_tcia_series(series_uids, dst=dicom_dir, csv_filename=os.path.join(path, "prostatex_series")) 424 elif not all(os.path.exists(os.path.join(dicom_dir, uid)) for uid in series_uids): 425 raise RuntimeError(f"Cannot find the data at {path}, but download was set to False.") 426 427 _preprocess_prostatex(dicom_dir, mask_dir, zones_dir, series_metadata, image_series, preprocessed_dir) 428 return preprocessed_dir 429 430 431def get_prostatex_paths( 432 path: Union[os.PathLike, str], 433 sequence: Literal["t2", "adc"] = "t2", 434 label_type: Literal["lesions", "zones", "zones_detailed"] = "lesions", 435 download: bool = False, 436) -> List[str]: 437 """Get paths to the PROSTATEx data. 438 439 Args: 440 path: Filepath to a folder where the data is downloaded for further processing. 441 sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'. 442 label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'. 443 download: Whether to download the data if it is not present. 444 445 Returns: 446 List of filepaths for the hdf5 files, which contain the image data ('<label_type>/<sequence>/raw') 447 and the label data ('<label_type>/<sequence>/labels'). 448 """ 449 import h5py 450 451 assert sequence in ("t2", "adc"), f"Invalid sequence: {sequence}." 452 assert label_type in ("lesions", "zones", "zones_detailed"), f"Invalid label type: {label_type}." 453 if label_type in ("zones", "zones_detailed") and sequence != "t2": 454 raise ValueError("The zone labels are only available for the 't2' sequence.") 455 456 data_dir = get_prostatex_data(path, download) 457 volume_paths = [] 458 for volume_path in natsorted(glob(os.path.join(data_dir, "*.h5"))): 459 with h5py.File(volume_path, "r") as f: 460 if f"{label_type}/{sequence}/labels" in f: 461 volume_paths.append(volume_path) 462 return volume_paths 463 464 465def get_prostatex_dataset( 466 path: Union[os.PathLike, str], 467 patch_shape: Tuple[int, ...], 468 sequence: Literal["t2", "adc"] = "t2", 469 label_type: Literal["lesions", "zones", "zones_detailed"] = "lesions", 470 resize_inputs: bool = False, 471 download: bool = False, 472 **kwargs 473) -> Dataset: 474 """Get the PROSTATEx dataset for prostate lesion or zone segmentation. 475 476 Args: 477 path: Filepath to a folder where the data is downloaded for further processing. 478 patch_shape: The patch shape to use for training. 479 sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'. 480 label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'. 481 resize_inputs: Whether to resize inputs to the desired patch shape. 482 download: Whether to download the data if it is not present. 483 kwargs: Additional keyword arguments for `torch_em.default_segmentation_dataset`. 484 485 Returns: 486 The segmentation dataset. 487 """ 488 volume_paths = get_prostatex_paths(path, sequence, label_type, download) 489 490 if resize_inputs: 491 resize_kwargs = {"patch_shape": patch_shape, "is_rgb": False} 492 kwargs, patch_shape = util.update_kwargs_for_resize_trafo( 493 kwargs=kwargs, patch_shape=patch_shape, resize_inputs=resize_inputs, resize_kwargs=resize_kwargs 494 ) 495 496 return torch_em.default_segmentation_dataset( 497 raw_paths=volume_paths, 498 raw_key=f"{label_type}/{sequence}/raw", 499 label_paths=volume_paths, 500 label_key=f"{label_type}/{sequence}/labels", 501 patch_shape=patch_shape, 502 is_seg_dataset=True, 503 **kwargs 504 ) 505 506 507def get_prostatex_loader( 508 path: Union[os.PathLike, str], 509 batch_size: int, 510 patch_shape: Tuple[int, ...], 511 sequence: Literal["t2", "adc"] = "t2", 512 label_type: Literal["lesions", "zones", "zones_detailed"] = "lesions", 513 resize_inputs: bool = False, 514 download: bool = False, 515 **kwargs 516) -> DataLoader: 517 """Get the PROSTATEx dataloader for prostate lesion or zone segmentation. 518 519 Args: 520 path: Filepath to a folder where the data is downloaded for further processing. 521 batch_size: The batch size for training. 522 patch_shape: The patch shape to use for training. 523 sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'. 524 label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'. 525 resize_inputs: Whether to resize inputs to the desired patch shape. 526 download: Whether to download the data if it is not present. 527 kwargs: Additional keyword arguments for `torch_em.default_segmentation_dataset` or for the PyTorch DataLoader. 528 529 Returns: 530 The DataLoader. 531 """ 532 ds_kwargs, loader_kwargs = util.split_kwargs(torch_em.default_segmentation_dataset, **kwargs) 533 dataset = get_prostatex_dataset(path, patch_shape, sequence, label_type, resize_inputs, download, **ds_kwargs) 534 return torch_em.get_data_loader(dataset, batch_size, **loader_kwargs)
The zone ids of the 'zones' labels.
The zone ids of the 'zones_detailed' labels.
368def get_prostatex_data(path: Union[os.PathLike, str], download: bool = False) -> str: 369 """Download the PROSTATEx dataset. 370 371 The download is resumable: series that were already downloaded and patients that were already converted are 372 skipped when the function is called again. 373 374 Args: 375 path: Filepath to a folder where the data is downloaded for further processing. 376 download: Whether to download the data if it is not present. 377 378 Returns: 379 Filepath where the preprocessed data is stored. 380 """ 381 os.makedirs(path, exist_ok=True) 382 preprocessed_dir = os.path.join(path, "preprocessed") 383 384 # Download the masks. 385 mask_dir = os.path.join(path, f"PROSTATEx_masks-{MASKS_COMMIT}", "Files") 386 if not os.path.exists(mask_dir): 387 zip_path = os.path.join(path, f"PROSTATEx_masks-{MASKS_COMMIT}.zip") 388 util.download_source(path=zip_path, url=URLS["masks"], download=download, checksum=CHECKSUMS["masks"]) 389 util.unzip(zip_path=zip_path, dst=path) 390 391 # Download the ProstateZones annotations and the list of the series they were drawn on. 392 zones_dir = os.path.join(path, "ProstateZones") 393 if not os.path.exists(os.path.join(zones_dir, "Singles")): 394 zip_path = os.path.join(path, "ProstateZones.zip") 395 util.download_source( 396 path=zip_path, url=URLS["zones_detailed"], download=download, checksum=CHECKSUMS["zones_detailed"] 397 ) 398 util.unzip(zip_path=zip_path, dst=zones_dir, remove=False) 399 util.download_source( 400 path=os.path.join(path, "list_of_PROSTATEx_files.txt"), url=URLS["series_list"], 401 download=download, checksum=CHECKSUMS["series_list"], 402 ) 403 404 image_series = _read_image_lists(mask_dir) 405 detailed_series = _read_detailed_zone_series(path) 406 for patient_id, series_number in detailed_series.items(): 407 if _get_detailed_zone_masks(zones_dir, patient_id): 408 # The ProstateZones masks ship without a reference image, so the series is matched by shape. 409 image_series[("zones_detailed", "t2", patient_id)] = (series_number, "") 410 411 n_patients = len({patient_id for _, _, patient_id in image_series}) 412 if len(glob(os.path.join(preprocessed_dir, "*.h5"))) == n_patients: 413 return preprocessed_dir 414 415 # Find the series the masks were drawn on and download them from TCIA. 416 series_metadata = _get_series_metadata(path, download) 417 series_numbers = {(patient_id, number) for (_, _, patient_id), (number, _) in image_series.items()} 418 series_uids = sorted( 419 series["SeriesInstanceUID"] for series in series_metadata 420 if (series["PatientID"], int(series["SeriesNumber"])) in series_numbers 421 ) 422 dicom_dir = os.path.join(path, "dicom") 423 if download: # Series that were downloaded already are skipped. 424 util.download_tcia_series(series_uids, dst=dicom_dir, csv_filename=os.path.join(path, "prostatex_series")) 425 elif not all(os.path.exists(os.path.join(dicom_dir, uid)) for uid in series_uids): 426 raise RuntimeError(f"Cannot find the data at {path}, but download was set to False.") 427 428 _preprocess_prostatex(dicom_dir, mask_dir, zones_dir, series_metadata, image_series, preprocessed_dir) 429 return preprocessed_dir
Download the PROSTATEx dataset.
The download is resumable: series that were already downloaded and patients that were already converted are skipped when the function is called again.
Arguments:
- path: Filepath to a folder where the data is downloaded for further processing.
- download: Whether to download the data if it is not present.
Returns:
Filepath where the preprocessed data is stored.
432def get_prostatex_paths( 433 path: Union[os.PathLike, str], 434 sequence: Literal["t2", "adc"] = "t2", 435 label_type: Literal["lesions", "zones", "zones_detailed"] = "lesions", 436 download: bool = False, 437) -> List[str]: 438 """Get paths to the PROSTATEx data. 439 440 Args: 441 path: Filepath to a folder where the data is downloaded for further processing. 442 sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'. 443 label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'. 444 download: Whether to download the data if it is not present. 445 446 Returns: 447 List of filepaths for the hdf5 files, which contain the image data ('<label_type>/<sequence>/raw') 448 and the label data ('<label_type>/<sequence>/labels'). 449 """ 450 import h5py 451 452 assert sequence in ("t2", "adc"), f"Invalid sequence: {sequence}." 453 assert label_type in ("lesions", "zones", "zones_detailed"), f"Invalid label type: {label_type}." 454 if label_type in ("zones", "zones_detailed") and sequence != "t2": 455 raise ValueError("The zone labels are only available for the 't2' sequence.") 456 457 data_dir = get_prostatex_data(path, download) 458 volume_paths = [] 459 for volume_path in natsorted(glob(os.path.join(data_dir, "*.h5"))): 460 with h5py.File(volume_path, "r") as f: 461 if f"{label_type}/{sequence}/labels" in f: 462 volume_paths.append(volume_path) 463 return volume_paths
Get paths to the PROSTATEx data.
Arguments:
- path: Filepath to a folder where the data is downloaded for further processing.
- sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'.
- label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'.
- download: Whether to download the data if it is not present.
Returns:
List of filepaths for the hdf5 files, which contain the image data ('
/ /raw') and the label data (' / /labels').
466def get_prostatex_dataset( 467 path: Union[os.PathLike, str], 468 patch_shape: Tuple[int, ...], 469 sequence: Literal["t2", "adc"] = "t2", 470 label_type: Literal["lesions", "zones", "zones_detailed"] = "lesions", 471 resize_inputs: bool = False, 472 download: bool = False, 473 **kwargs 474) -> Dataset: 475 """Get the PROSTATEx dataset for prostate lesion or zone segmentation. 476 477 Args: 478 path: Filepath to a folder where the data is downloaded for further processing. 479 patch_shape: The patch shape to use for training. 480 sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'. 481 label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'. 482 resize_inputs: Whether to resize inputs to the desired patch shape. 483 download: Whether to download the data if it is not present. 484 kwargs: Additional keyword arguments for `torch_em.default_segmentation_dataset`. 485 486 Returns: 487 The segmentation dataset. 488 """ 489 volume_paths = get_prostatex_paths(path, sequence, label_type, download) 490 491 if resize_inputs: 492 resize_kwargs = {"patch_shape": patch_shape, "is_rgb": False} 493 kwargs, patch_shape = util.update_kwargs_for_resize_trafo( 494 kwargs=kwargs, patch_shape=patch_shape, resize_inputs=resize_inputs, resize_kwargs=resize_kwargs 495 ) 496 497 return torch_em.default_segmentation_dataset( 498 raw_paths=volume_paths, 499 raw_key=f"{label_type}/{sequence}/raw", 500 label_paths=volume_paths, 501 label_key=f"{label_type}/{sequence}/labels", 502 patch_shape=patch_shape, 503 is_seg_dataset=True, 504 **kwargs 505 )
Get the PROSTATEx dataset for prostate lesion or zone segmentation.
Arguments:
- path: Filepath to a folder where the data is downloaded for further processing.
- patch_shape: The patch shape to use for training.
- sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'.
- label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'.
- resize_inputs: Whether to resize inputs to the desired patch shape.
- download: Whether to download the data if it is not present.
- kwargs: Additional keyword arguments for
torch_em.default_segmentation_dataset.
Returns:
The segmentation dataset.
508def get_prostatex_loader( 509 path: Union[os.PathLike, str], 510 batch_size: int, 511 patch_shape: Tuple[int, ...], 512 sequence: Literal["t2", "adc"] = "t2", 513 label_type: Literal["lesions", "zones", "zones_detailed"] = "lesions", 514 resize_inputs: bool = False, 515 download: bool = False, 516 **kwargs 517) -> DataLoader: 518 """Get the PROSTATEx dataloader for prostate lesion or zone segmentation. 519 520 Args: 521 path: Filepath to a folder where the data is downloaded for further processing. 522 batch_size: The batch size for training. 523 patch_shape: The patch shape to use for training. 524 sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'. 525 label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'. 526 resize_inputs: Whether to resize inputs to the desired patch shape. 527 download: Whether to download the data if it is not present. 528 kwargs: Additional keyword arguments for `torch_em.default_segmentation_dataset` or for the PyTorch DataLoader. 529 530 Returns: 531 The DataLoader. 532 """ 533 ds_kwargs, loader_kwargs = util.split_kwargs(torch_em.default_segmentation_dataset, **kwargs) 534 dataset = get_prostatex_dataset(path, patch_shape, sequence, label_type, resize_inputs, download, **ds_kwargs) 535 return torch_em.get_data_loader(dataset, batch_size, **loader_kwargs)
Get the PROSTATEx dataloader for prostate lesion or zone segmentation.
Arguments:
- path: Filepath to a folder where the data is downloaded for further processing.
- batch_size: The batch size for training.
- patch_shape: The patch shape to use for training.
- sequence: The MRI sequence, either 't2' or 'adc'. The zone labels are only available for 't2'.
- label_type: The label type, one of 'lesions', 'zones' or 'zones_detailed'.
- resize_inputs: Whether to resize inputs to the desired patch shape.
- download: Whether to download the data if it is not present.
- kwargs: Additional keyword arguments for
torch_em.default_segmentation_datasetor for the PyTorch DataLoader.
Returns:
The DataLoader.