Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Has Anybody Used

Has anybody used albumentations with pytorch in remote execution on clearml? The task is freezing during DataLoader initialization every time I put any augmentations there.

  
  
Posted 2 years ago
Votes Newest

Answers 6


When you run it locally it doesn't freeze?

  
  
Posted 2 years ago

` # dataset_class.py
from PIL import Image
from torch.utils.data import Dataset as BaseDataset

class Dataset(BaseDataset):

def __init__(
    self,
    images_fps,
    masks_fps,
    augmentation=None,
):
    self.augmentation = augmentation
    self.images_fps = images_fps
    self.masks_fps = masks_fps
    self.ids = len(images_fps)


def __getitem__(self, i):

    # read data
    img = Image.open(self.images_fps[i])
    mask = Image.open(self.masks_fps[i])

    # apply augmentations
    if self.augmentation:
        sample = self.augmentation(image=img, mask=mask)
        image_aug, mask_aug = sample["image"], sample["mask"]

    return image_aug, mask_aug

def __len__(self):
    return self.ids

training_script.py

from dataset_class import Dataset
import albumentations as albu
from torch.utils.data import DataLoader

usual clearml config etc

train_albs = [
albu.HorizontalFlip(p=0.5),
]
augs = albu.Compose(train_albs)

train_dataset = Dataset(images_fps, masks_fps, augmentations=augs)

train_loader_l = DataLoader(
train_dataset,
batch_size=16,
shuffle=True,
num_workers=0,
) `

  
  
Posted 2 years ago

We use albumentations with scripts that execute remotely and have no issues. Good question from CostlyOstrich36

  
  
Posted 2 years ago

we normally do something like that - not sure what why it's freezing for you without more info

  
  
Posted 2 years ago

When I run locally w/o clearml, it doesn't freeze

  
  
Posted 2 years ago

never tried to start it locally in clearml

  
  
Posted 2 years ago
670 Views
6 Answers
2 years ago
one year ago
Tags