Hi @<1523701070390366208:profile|CostlyOstrich36> , any suggestion for this error?
Hi @<1523701070390366208:profile|CostlyOstrich36> , here it is
Hi @<1597762318140182528:profile|EnchantingPenguin77> , can you please add the full log?
Hi, I am trying to save my trained model weights in S3 bucket instead of using ClearML storage when using clearml-task for ml training remotely. I tried to use --skip-task-init in clearml-task and set task.init in my scripts, but it doesn't seem to work. The tags and output_uri doesn't apply to the remote training
clearml-task \
--project fluoro-motion-detection \
--name uniformer-train \
--folder ~/algorithms-python \
--script ~/algorithms-python/fluoro_motion_detection/src/run/train.py \
--requirements ~/algorithms-python/fluoro_motion_detection/src/clearml/requirements.txt \
--args hydra=$HYDRA_OVERRIDES \
--docker mzhengtelos/algorithm-ml:poetry-project-cuda11.4.3-runtime-ubuntu20.04 \
--docker_args "--env CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=$PYTHON_ENV_DIR --env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
--skip-task-init\
--queue test-gpu\
clearml_task = Task.init(project_name=cfg.clearml_project_name, task_name=cfg.clearml_task_name, tags=cfg.tags,
output_uri=cfg.paths.s3_clearml_dir)
got error when using --output-uri in clearml-task clearml.storage - ERROR - Failed uploading: '
LazyEval Wrapper
' object cannot be interpreted as an integer
Hi @<1523701070390366208:profile|CostlyOstrich36> , any suggestion for this error?
Hi @<1523701070390366208:profile|CostlyOstrich36> , here it is
Hi @<1597762318140182528:profile|EnchantingPenguin77> , can you please add the full log?