Unanswered
Hi Folks. Wondering If Anyone Has Used Clearml In Conjunction With
Hi folks. Wondering if anyone has used ClearML in conjunction with Ray ? We currently use ClearML for artefact storage, logging and experiment tracking and are trying to introduce Ray to speed up our model training pipeline, but when we try to run any ClearML actions in the Ray workers (e.g. clearml.Task.upload_artifact
), it kills at least one of the Ray workers due to memory pressure (OOM). I wonder if it's possible to reconfigure Ray to avoid this, or if because of the way ClearML is threaded, this is somewhat unavoidable? Any help/experiences would be greatly appreciated. Thanks!
226 Views
0
Answers
2 months ago
2 months ago
Tags