Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Am Trying To Save My Trained Model Weights In S3 Bucket Instead Of Using Clearml Storage When Using Clearml-Task For Ml Training Remotely. I Tried To Use --Skip-Task-Init In Clearml-Task And Set Task.Init In My Scripts, But It Doesn'T Seem To Work.

Hi, I am trying to save my trained model weights in S3 bucket instead of using ClearML storage when using clearml-task for ml training remotely. I tried to use --skip-task-init in clearml-task and set task.init in my scripts, but it doesn't seem to work. The tags and output_uri doesn't apply to the remote training

clearml-task \
    --project fluoro-motion-detection \
    --name uniformer-train \
    --folder ~/algorithms-python \
    --script ~/algorithms-python/fluoro_motion_detection/src/run/train.py \
    --requirements ~/algorithms-python/fluoro_motion_detection/src/clearml/requirements.txt \
    --args hydra=$HYDRA_OVERRIDES \
    --docker mzhengtelos/algorithm-ml:poetry-project-cuda11.4.3-runtime-ubuntu20.04 \
    --docker_args "--env CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=$PYTHON_ENV_DIR --env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
    --skip-task-init\
    --queue test-gpu\
clearml_task = Task.init(project_name=cfg.clearml_project_name, task_name=cfg.clearml_task_name, tags=cfg.tags,
                         output_uri=cfg.paths.s3_clearml_dir)

got error when using --output-uri in clearml-task clearml.storage - ERROR - Failed uploading: ' LazyEval Wrapper ' object cannot be interpreted as an integer

  
  
Posted 7 months ago
Votes Newest

Answers 6


Hi @<1597762318140182528:profile|EnchantingPenguin77> ,settings in task.init() are only applied to a local run, and are ignored when running remotely. In the case of clearml-task you're essentially not running locally at all, only remotely. For that, you can use the --output-uri clearml-task parameter

  
  
Posted 7 months ago

@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I tried to use --output-uri in clearml-task but got the same error clearml.storage - ERROR - Failed uploading: ' LazyEval Wrapper ' object cannot be interpreted as an integer

  
  
Posted 7 months ago

@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I am using 1.12.0

  
  
Posted 7 months ago

What version of ClearML SDK are you using?

  
  
Posted 7 months ago

Can you try upgrading to the latest version (v1.12.2)? This is an issue that was solved in v1.12.1

  
  
Posted 7 months ago

Hi @<1523701087100473344:profile|SuccessfulKoala55> I was able to solve this issue after upgrade clearml to 1.12.2, but my training/val loss become nan after the update

  
  
Posted 7 months ago