Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Am Trying To Save My Trained Model Weights In S3 Bucket Instead Of Using Clearml Storage When Using Clearml-Task For Ml Training Remotely. I Tried To Use --Skip-Task-Init In Clearml-Task And Set Task.Init In My Scripts, But It Doesn'T Seem To Work.

Hi, I am trying to save my trained model weights in S3 bucket instead of using ClearML storage when using clearml-task for ml training remotely. I tried to use --skip-task-init in clearml-task and set task.init in my scripts, but it doesn't seem to work. The tags and output_uri doesn't apply to the remote training

clearml-task \
    --project fluoro-motion-detection \
    --name uniformer-train \
    --folder ~/algorithms-python \
    --script ~/algorithms-python/fluoro_motion_detection/src/run/train.py \
    --requirements ~/algorithms-python/fluoro_motion_detection/src/clearml/requirements.txt \
    --args hydra=$HYDRA_OVERRIDES \
    --docker mzhengtelos/algorithm-ml:poetry-project-cuda11.4.3-runtime-ubuntu20.04 \
    --docker_args "--env CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=$PYTHON_ENV_DIR --env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
    --skip-task-init\
    --queue test-gpu\
clearml_task = Task.init(project_name=cfg.clearml_project_name, task_name=cfg.clearml_task_name, tags=cfg.tags,
                         output_uri=cfg.paths.s3_clearml_dir)

got error when using --output-uri in clearml-task clearml.storage - ERROR - Failed uploading: ' LazyEval Wrapper ' object cannot be interpreted as an integer

  
  
Posted one year ago
Votes Newest

Answers 6


@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I am using 1.12.0

  
  
Posted one year ago

Can you try upgrading to the latest version (v1.12.2)? This is an issue that was solved in v1.12.1

  
  
Posted one year ago

What version of ClearML SDK are you using?

  
  
Posted one year ago

Hi @<1523701087100473344:profile|SuccessfulKoala55> I was able to solve this issue after upgrade clearml to 1.12.2, but my training/val loss become nan after the update

  
  
Posted one year ago

@<1523701087100473344:profile|SuccessfulKoala55> Hi Jake, I tried to use --output-uri in clearml-task but got the same error clearml.storage - ERROR - Failed uploading: ' LazyEval Wrapper ' object cannot be interpreted as an integer

  
  
Posted one year ago

Hi @<1597762318140182528:profile|EnchantingPenguin77> ,settings in task.init() are only applied to a local run, and are ignored when running remotely. In the case of clearml-task you're essentially not running locally at all, only remotely. For that, you can use the --output-uri clearml-task parameter

  
  
Posted one year ago