Thanks LethalCentipede31 , i think (3) is the most stable solution (as it doesn't require to add another package, and should work on any python version / OS)
This is actually what we do for downloads .
DO you know if there is a minimum required python requests version ?

Quite hard for me to try this right

👍
How do I reproduce it ?

replace the base-docker-image and it should work fine 🙂

LovelyHamster1
Also you can use pip freeze instead of the static code analysis , on your development machines set:
detect_with_pip_freeze: false
https://github.com/allegroai/clearml/blob/e9f8fc949db7f82b6a6f1c1ca64f94347196f4c0/docs/clearml.conf#L169

Hi PanickyLion56
Yep savefig also works, you can also do,
from clearml import Logger Logger.current_logger().report_matplotlib_figure(title="My Plot Title", series="My Plot Series", iteration=10, figure=plt)https://github.com/allegroai/clearml/blob/0c5d12b830987aa9bb8d44d81e92ff9198008f29/examples/frameworks/matplotlib/matplotlib_example.py#L25

Hi LazyFish41
Could it be some permission issue on /home/quetalasj/.clearml/cache/ ?

Very odd, I still can't reproduce. This is just the cleanup service running without anything else ?
What's the clearml version it is using ?

that machine will be able to pull and report multiple trials without restarting

What do you mean by "pull and report multiple trials" ? Spawn multiple processes with different parameters ?
If this is the case: the internals of the optimizer could be synced to the Task so you can access them, but this is basically the internal representation, which is optimizer dependent, which one did you have in mind?
Another option is to pull Tasks from a dedicated queue and use the LocalClearMLJob ...

The difference is that I want a single persistent machine, with a single persistent python script that can pull execute and report multiple tasks

So basically instead of using the agent, so simply spin a sub process ?

MysteriousBee56 what do you mean by "local repository"?
Like no git server, or local commit before pushing it ?

I "think" I have a clue on the issue that is lost here in the translation:
Specifically to me it all comes down to the definition of "pipeline"
From the clearml perspective:
Manual Task - code that is executed by the user (or any other mechanism Outside of the agent)
Remote Task - code that is executed by the Agent

Pipeline is a Task
Pipeline can be "manual task" but also "remote task"
Pipeline generates "remote tasks"
Task status (e.g. pipeline status as it is also a Task) can be: draft, a...

Thanks GloriousPanda26 !

You can try calling
task._update_repository()I'm still trying to figure out how to reproduce it...

and trains version

Meaning the node restarted (or actually moved)

hmmm I see...
It seems to miss the fact that your process do uses the GPU.
Maybe it only happens later, that the GPU is used?
Does that make sense ?

RoughTiger69 how did you end up with a Task with just "origin" in the repo field ?

because it should have detected it...
Did you see "Repository and package analysis timed out ..."

Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build/80754af9/attrs_1604765588209/work'Seems like pip failed creating a folder
Could it be you are out of space ?

https://stackoverflow.com/questions/47085458/why-is-multiprocessing-queue-get-so-slow

YEY!

My pleasure 🙂

ShallowCat10 Of course it is 🙂

This points to the wrong cu117 / driver - could that be?

Calling the script without the

PipelineDecorator.run_locally()

i.e. running the pipeline remotely still gives the

ModuleNotFoundError: No module named

Do you have the needed module listed on the pipeline controller Task ? (press on the details link, then go to Execution tab / "Installed Packages"

I added the link just in case anyway 

Smart move :)

DilapidatedDucks58 , Of course there is 🙂 actually with the latest pip 20.1 and the next RC it will be automatically detected and put into "installed package"

You can treat the "installed packages" just like you would any other "requirements.txt", just add:
git+ https://github.com/ ... and you are good to go

Hi SkinnyPanda43
This issue was fixed with clearml-agent 1.5.1, can you verify?

And this is with the latest pycharm plugin 1.1.0 ?

Hi @<1569496075083976704:profile|SweetShells3>
Are you using the standard docker-compose ? are using the default elastic container ?
What exactly changed ?

Hi WickedElephant66

Setting the pipeline controller with pipeline_execution_queue as None

is actually launching the pipeline controller on your "dev" machine, not sure why you have two of them?