I gave this it a try to switch from Task.create() to Task.init(). I think I am pretty close to switch to using init(). But still have issue of ModuleNotFoundError: No module named 'src' when using task.init().

My project setup look like this:

project_root/
    |--src/
    |--runbooks/
        |--run_task.py

So if I use Task.create(repo=xx, script="runbooks/run_task.py"), it works but if I switch to using Task.init() with the same repo setup (task.set_repo, and then follow by task.execute_remotely), it will complain about ModuleNotFoundError: No module named 'src'`

I suspect that it is about how the working_directory setup, from what I see in the signature of task.create and task.set_repo, they both state that Default: repository root folder (task.create signature), execution work directory will be the repository root folder (task.set_repo signature). But there must be something different about this two workflow since one work another doesn't

Any ideas?

that is just to compare the same functionality used in task.create() can be achieved when using task.init workflow. Not technically required

Confirmed that without task.set_repo, it come down to the same error:
ModuleNotFoundError: No module named 'src'

I mean that task.init will automatically detect the repo without specifying it

Why do you manually use set_repo ?

Hi ScrawnyCrocodile51 , You can set the repository using Task.set_repo - None

Although if you use Task.init it will automatically detect the repository from the script. If you don't want to execute the code on your machine you can use Task.execute_remotely - None

And finally, you can use Task.set_base_docker to set the docker image you want to use - None

I would advise using Task.init and it should make your life much easier 🙂

I can try taking it out see if it fix the issue. But i feel it is not the root cause

ScrawnyCrocodile51 how are you calling Tasl.init()?

Thanks for the tips! I will take them for a spin

Can you add a screenshot of the execution section of the experiment when you add task.set_repo and when you don't (Just add some sample script in the same folder and run task.init and print hello world)?

Executing task id [7605f1e5ce6b45e99e9302d93bc3bac6]:
repository = git@xxx
branch = xxx
version_num = 9dca88fa23ff93d446eb2ff7d615d7ade213c8aa
tag = 
docker_cmd = iocr.io/xxx
entry_point = clearml_init.py
working_dir = dev

Based on the logging,
working_dir = dev is the problem. I need to have a way to overwrite the working_dir.

Would it cause problem to manually set repo when using task.init()?

What do you mean by signature?

from clearml import Task

task = Task.init(,...)
print("hello world!")

task = Task.init(
        project_name=PROJECT_NAME,
        task_name=TASK_NAME,
        task_type=Task.TaskTypes.data_processing,
    )
    task.set_repo(
        repo="git@xxx.git",
        branch=branch
    )
    task.set_base_docker(
        docker_image="docker-image",
    )
    task.execute_remotely(queue_name=QUEUE_NAME)

This is how I use the task init

function's signature in the source code