now, I need to pass a variable to the Preprocess class

you mean for the construction ?

Hi PompousParrot44
You can check the cleanup service example.
It sleeps for 24 hours then spins up and does its thing.
You can always launch this service tasks on the services queue, its purpose is to run those services on the trains-server as additional CPU services. They will also be registered as service nodes, so you have visibility into which service is running.
In order to clone a task and wait for its completion.
Use the TrainsJob https://github.com/allegroai/trains/blob/65a4a...

Oh I see, that kind of make sense
I think this is the section you should use:
None
But instead of the clearml-services container you should use the regular container (or just have it installed as part of the entry-point on any ubuntu based container)
Notice the important parts here are:
[None](https://github.com/allegroai/clearml-server/blob/6a1fc04d1e8b112fb334c8743d...

in order to work with ssh cloning, one has to manually install openssh-client to the docker image, looks like that

Correct, you have to have SSH inside the container so that git can use it.
You can always install with the following setup inside your agent's clearml.conf:
extra_docker_shell_script: ["apt-get install -y openssh-client", ]
https://github.com/allegroai/clearml-agent/blob/73625bf00fc7b4506554c1df9abd393b49b2a8ed/docs/clearml.conf#L145

If the same Task is run with different parameters...

ShinyWhale52 sorry, I kind of missed that in the explanation
The pipeline will always* create a new copy (clone) of the original Task (step), then modify the step's inputs etc.
The idea is that you have the experiment management (read execution management) to create full transparancy into the pipelines and steps. Think of it as the missing part in a lot of pipelines platforms where after you executed the pipeline you need to furthe...

somehow set docker_args and docker_bash_setup_script equivalent??task.set_base_docker(...)
# somehow setup repo and branch to download to remote instance before runningThis is automatically detected based on your local commit/branch as well ass uncommitted changes

Also, I just wanted to say thanks for the tool! I'm managing a small data science practice and it's going to be really nice to have a view of all of the experiments we've got and know our GPU utilization, all without having to give every data scientist access to each box where the workflows are run. Incredibly stoked.

♥ ❤ ♥

` param = {'arg': value}
task.connect(param, section='new section')

create pipeline here

pipeline `

It seems like there is no way to define that a Task requires docker support from an agent, right?

Correct, basically the idea is you either have workers working in venv mode or docker.
If you have a mixture of the two, then you can have the venv agents pulling from one queue (say default_venv) and the docker mode agents pulling from a different queue (say default_docker). This way you always know what you are getting when you enqueue your Task

Hi MuddySquid7 issue is verified, v1.1.1 will be released in a few hours with a fix.
Thank you for noticing!

I was thinking mainly about AWS.

Meaning S3?

Hmm could it be this is on the "helper functions" ?

JitteryCoyote63 This seems like exactly what you are saying, elastic license issue...

make sure you follow all the steps :
https://clear.ml/docs/latest/docs/deploying_clearml/upgrade_server_linux_mac
(basically make sure you get the latest docker-compose.yml and the pull it
curl -o /opt/clearml/docker-compose.yml docker-compose -f /opt/clearml/docker-compose.yml pull docker-compose -f /opt/clearml/docker-compose.yml up -d

BTW: the agent will resolve pytorch based on the install CUDA version.

or at least stick to the requirements.txt file rather than the actual environment

You can also for it to log the requirements.txt with
Task.force_requirements_env_freeze(requirements_file="requirements.txt") task = Task.init(...)

What's the pip verison?

Ohh I see, could you copy paste what you put there (instead of the secret and key *** will do 🙂 )

Weird issue, I'll make sure we fix compatibility with python 3.9

MysteriousBee56 I see...
So yes, you can with the APIClient you have full RESTful access to the backend.
I think there was a similar discussion https://allegroai-trains.slack.com/archives/CTK20V944/p1593524144116300
HandsomeCrow5 how did you end up solving it? I think you had a similar use case?!

I see.
You can get the offline folder programmatically then copy the folder content (it's the same as the zip, and you can also pass a folder instead of zip to the import function)
task.get_offline_mode_folder()You can also have a soft link of the offline folder (if you are working on a linux machine:
ln -s myoffline_folder ~/.trains/cache/offline

doing some extra "services"

what do you mean by "services" ? (from the system perspective any Task that is executed by an agent that is running in "services-mode" is a service, there are no actual limitation on what it can do 🙂 )

No worries 🙂 glad to hear it worked out 🙂

Hi @<1547028074090991616:profile|ShaggySwan64>
I have to admit that personally I do not know pdm , could you share links, and help us understand what is the value over pip/poetry/conda ?

Hi LethalDolphin75
I think you are right there isn't one (although I remember a discussion about it...)
Anyhow it will be very easy to implement, just inherit from:
https://github.com/allegroai/clearml/blob/400c6ec103d9f2193694c54d7491bb1a74bbe8e8/clearml/automation/parameters.py#L111
And return the power of the parent value here:
https://github.com/allegroai/clearml/blob/400c6ec103d9f2193694c54d7491bb1a74bbe8e8/clearml/automation/parameters.py#L146
And
https://github.com/allegroai/...

This is a part of a bigger process which times quite some time and resources, I hope I can try this soon if this will help get to the bottom of this

No worries, if you have another handle on how/why/when we loose the current Task, please share 🙂

Hi VexedCat68

One of my steps just finds the latest model to use. I want the task to output the id, and the next step to use it. How would I go about doing this?

When you say "I want the task to output the id" do you mean to pass t to the next step:
Something like this one:
https://github.com/allegroai/clearml/blob/c226a748066daa3c62eddc6e378fa6f5bae879a1/clearml/automation/controller.py#L224

Hi ComfortableHorse5
Yes this is more of a suggestion that you should write them using the platform capabilities, the UI implementation is being worked on, as well as a few helpers classes, I thin you'll be able to see a few in the next release 🙂

MuddySquid7
are you saying that for some reason the models pick the artifacts ? Is that reproducible ? (they are two different things)
Can you see the df.pkl on the Models section of the Task (in the UI) ?

Okay, could you try to run again with the latest clearml package from github?
pip install -U git+