AgitatedDove14 We are trying to create an interface for our DL team so that they can send local repo path, entrypoint to script, requirements, etc as inputs and the job will get added on the queue.
Hi StraightCoral86
When I run an experiment using
Task.create()
,
Use Task.init 🙂
Task.create is meant to create an extranl Task (i.e. Job) ins the system, Not to auto-gernerate a job from the running code. Make sense ?
I passed the path to local repo. all files in working directory other than entrypoint are missing.
I was able to successfully enqueue the task but only entrypoint script is visible to it and nothing else.
So you passed a repository link is it did not show on the Task ?
What exactly is missing and how the Task was created ?
clearML console
clearml_agent: ERROR: Can not run task without repository or literal script in script.diff
script to create job
from clearml import Task
task = Task.create(project_name=‘example’, task_name=‘test_5_2’.format(i), script=‘~/train-script-old/train1.py’, working_directory=‘~/train-script-old/’, add_task_init_call=False)
status = Task.enqueue(task, ‘test’)
print(status)
cleamrl sdk (i.e. python client)
The issue is that the Task.create did not add the repo, link (again as mentioned above, you need to pass the local folder or repo link to the repo argument of the Task.create function). I "think" it could automatically deduce the repo from the script entry point, but I'm not sure. hence my question on the clearml package version
p.s. StraightCoral86 I might be missing something here, please feel free to describe the entire execution scenario and what you are trying to achieve 🙂
Ohh StraightCoral86 did you check cleaml-task ? This is exactly what it does
(this is the CLI, from code you basically call Task.create & Task.enqueue)
Will this solve it ?
Right, you need to pass "repo" and direct it to the repository path
(BTW, what's the cleaml version)
Also, How do I make the files other than entry script visible to the job?
The assumption for clearml (regradless on how you create a Task) is that you code is either a standlone script (or jupyter notebook) or inside a git repository. In case of a git repository cleamrl-agent will clone the git repository of the code, apply the uncommitted changes and run your code.
I did that AgitatedDove14 . That is where I faced this issue. I was able to successfully enqueue the task but only entrypoint script is visible to it and nothing else.