Reputation
Badges 1
533 × Eureka!AgitatedDove14 sorry for the late reply,
It's right after executing all the steps. So we have the following block which determines whether we run locally or remotely
if not arguments.enqueue: pipe.start_locally(run_pipeline_steps_locally=True) else: pipe.start(queue=arguments.enqueue)
And right after we have a method that calls Task.current_task()
which returns None
I also ran it without $(pwd) on the Create Clearml task templates section, I added it because of CostlyOstrich36 's comments but it didn't help
Cool, now I understand the auto detection better
the level of configurability in this thing is one of the best I've seen
anyway, my ultimate goal is to create templates for other tasks... Is that possible in any other way through the CLI?
and also in the extra_vm_bash_script
variables, I ahve them under export TRAINS_API_ACCESS_KEY
and export TRAINS_API_SECRET_KEY
I only found Project ID, which I'm not sure what this refers to - I have the project name
Worth mentioning, nothing has changed before we executed this, it worked before and now after the update it breaks
Committing that notebook with changes solved it, but I wonder why it failed
I mean, I barely have 20 experiments
so basically - if she has new commits locally that werent pushed it won't work
But if she did not commit her latest changes, and now she enqueues - it will work?
What do you mean by submodules?
She did not push, I told her she does not have to push before executing as trains figures out the diffs.
When she pushes - it works
actually i was thinking about model that werent trained uaing clearml, like pretrained models etc
pgrep -af trains
shows that there is nothing running with that name
I was trying out the pipeline controller for the first time and I felt a bit of a burden that just for the sake of trying I had to launch an agent
and the machine I have is 10.2.
I also tried nvidia/cuda:10.2-base-ubuntu18.04 which is the latest
I guess the AMI auto updated
I'll check the version tomorrow, about the current_task call, I tried before and after - same result
TimelyPenguin76 I think our problem is that the agent is not using this environment, I'm not sure which one he does... Is there a way to hard-code the agent environment?
This error just keeps coming back... I already made the watermarks like 0.5gb
thx TimelyPenguin76
skimming over this, I can't find how to filter by project name or something similar