👍

Good, so we narrowed it down. Now the question is how come it is empty ?

I have a theory

My bad, there is a mixture in terms.
"configuration object" is just a dictionary (or plain text) stored on the Task itself.
It has no file representation (well you could get it dumped to a file, but it is actually stored a s a blob of text on the Task itself, at the backend side)

Hi JuicyFox94
I think you are correct, this bug will explain the entire thing.
Basically what happens is that remote_execute stops the local run before the configuration is set on the Task. Then running remotely the code pull the configuration, sees that it is empty and does nothing.
Let me see if I can reproduce it...

btw it's on the filesystem

then I enqueue it and it's created but obv empty

yaya confirmed

maybe this can cause the issue?

what is?

got it

I finally found where the issue is, I opened an issue on gh so it's more manageable:
https://github.com/allegroai/clearml/issues/273

I also found a not so good (at least for me) behaviour:
https://github.com/allegroai/clearml/issues/272

and I will give you feedback here

\o/

a big ty for now

I mean this blob is then saved on the fs

It can if you do:
temp_file = task.connect_configuration('/path/to/config/file', name='configuration object is a config file')Then temp_file is actually a local copy of the text coming from the Task.
When running in manual mode the content of '/path/to/config/file' is stored on the Task When running remotely by the agent, the content from the Task is dumped into a temp file and the path to the file is returned in temp_file

No worries, just found it. Thanks!
I'll make sure to followup on the GitHub issue for better visibility 🙂

this configuration object is stored as a file in /root/.trains ?

today I'm in the middle of sprint plannings for my team so I will not be probably fast to help if needed, but feel free to ping me just in case (I will try to do my best)

If the manual execution (i.e. pycharm) was working it should have stored it on the Pipeline Task.

checking it 😄

I have multiple agents not sharing /root/.trains

maybe this can cause the issue?

Not likely.
In the original pipeline (the one executed from the Pycharm) do you see the "Pipeline" section under Configuration -> "Config objects" in the UI?

My pleasure

?

ok the issue must be there, After first creation nothing is there

I mean this blob is then saved on the fs

Sure thing 🙂

perfect, now the whole process is clear to me