Hi SparklingHedgehong28 , can you please elaborate on the steps you take during the process + how you connect your config to the task?

Hi CostlyOstrich36 Thank you for reaching back and sorry for my embarrassingly long answer here.
We launch the job on a remote executor this way –
From a repository where the code lies we launch a job with clearml-task in this form: clearml-task --name projx-1-debug-$(date +%s) --project kbc --folder . --script projx/kbcrun.py --requirements custom-requirements.txt --docker python:3.9.13 --task-type training --queue aws-cpu-docker --args graph_dir=not/needed config_path=examples/transe_config_unfiltered.json
So the transe_config_unfiltered.json is the config we need to have uploaded. The engineer changes params in it before run. It’s normally a dynamic file not part of the repository. IF the file is tracked (previously added to the repo, even empty) the change is seen and is uploaded to the worker. If it’s not tracked in git, it’s not uploaded and we’re ought to have a placeholder in the repo, changing the file on every run of the job.

So it would be nice to have an option for the file to be uploaded but not be part of the repo, as the file contains dynamic parameters for launching the job. There are so many of them that putting then into --args alone is to heavyweight.

Hope that make sense.