I also found that you should have a deterministic ordering

before

you apply a fixed seed

Not sure I follow ?

DeliciousBluewhale87 could you send the full log of the Task?

MuddySquid7 you mean you are creating them with TB ? or are you uploading them as debug images ?
Specifically in the ClearML UI, do you have it under "plots" tab or "debug samples" tab ?

Okay, let me check

PompousBeetle71 cool, next RC will have the argparse exclusion feature :)

Hi @<1523701323046850560:profile|OutrageousSheep60>
What do you mean by "in clearml server" ? I do not see any reason a subprocess call from a Task will be an issue. What am I missing ?

make sure you follow all the steps :
https://clear.ml/docs/latest/docs/deploying_clearml/upgrade_server_linux_mac
(basically make sure you get the latest docker-compose.yml and the pull it
curl -o /opt/clearml/docker-compose.yml docker-compose -f /opt/clearml/docker-compose.yml pull docker-compose -f /opt/clearml/docker-compose.yml up -d

The default cleanup service should work with S3 with a correctly configured clearml service agent if I understand the workings correctly.

Yes I think you are correct

I am referring to the UI.

In that case, no 😞 . This is actually a backend server change (from the UI it should be relatively simple). Is this somehow a showstopper ?

👍

FileNotFoundError: [Errno 2] No such file or directoryCould it be the file you are trying to run is not in the repository ?
Are you running inside a docker ?
Any chance you can send the full log ?

I'm not, not sure about WickedElephant66 / DefiantHippopotamus88

Sounds good to me 🙂

Yes 🙂
BTW: do you guys do remote machine development (i.e. Jupyter / vscode-server) ?

My bad, there is a mixture in terms.
"configuration object" is just a dictionary (or plain text) stored on the Task itself.
It has no file representation (well you could get it dumped to a file, but it is actually stored a s a blob of text on the Task itself, at the backend side)

Do note that the needed module is just a local folder with scripts.

Oh that is the issue, is it in the git repo ?

SubstantialElk6 on the client side?

JoyousKoala59 what is the Trains server you have? the link you posted is to upgrade from v0.15 to v0.16, not from trains to clearml

at that point we define a queue and the agents will take care of training 

This is my preferred way as well :)

what is?

yes 🙂
But I think that when you get the internal_task_representation.execution.script you are basically already getting the API object (obviously with the correct version) so you can edit it in place and pass it too

With pleasure 🙂

WittyOwl57 that is odd there is a specific catch for SystemExit
https://github.com/allegroai/clearml/blob/51d70efbffa87aa41b46c2024918bf4c584f29cf/clearml/backend_interface/task/repo/scriptinfo.py#L773
How do I reproduce this issue/warning ?
Also: "Repository and package analysis timed out (300.0 sec), giving up" seriously ove 5 minutes ?! how large is the git repo?

Oh right, I missed the fact the helper functions are also decorated, yes it makes sense we add the tags as well.
Regarding nested pipelines, I think my main question is , are they independent or are we generating everything from the same code base?

Hi ShallowCat10
What's the TB your are using?
Is this example working correctly for you?
https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorboard_pr_curve.py

RoundMosquito25 do notice the agent is pulling the code from the remote repo, so you do need to push the local commits, but the uncommitted changes clearml will do for you. Make sense?

This workflow however is the only way I have found to easily fix my previous ‘Module not found’ errors

Hmm okay make sense,
Did you try to set these ?
or even hack the sys.path with something like
import sys, os sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)+"/../")

No worries 🙂
GiganticTurtle0 I'm glad it was solved 👍

WackyRabbit7 you can configure AWS autoscaler with two types of instances , with priority to one of them. So in theory you do not need two autoscaler processes, with that in mind I "think" single IAM should suffice

Ohh I see now, okay there are two entries on an artifact, the actual artifact (link to file somewhere) and the text preview of the artifact . I think the "preview" is the issue

So this is a UI visual issue?