Hi DrabCockroach54

... and no logs for python script.

what do you mean by "no logs" , is it clearml logs? or k8s pod logs ?

JitteryCoyote63 hacky but sure 🙂
` from trains.config import config_obj

print(config_obj) `

Hi SmugOx94
Hmm are you creating the environment manually, or is it done by Task.init ?
(Basically Task.init will store the entire environment of conda, and if the agent is working with conda package manager it will use it to restore it)
https://github.com/allegroai/clearml-agent/blob/77d6ff6630e97ec9a322e6d265cd874d0ab00c87/docs/clearml.conf#L50

Generally speaking, for the exact reason if you are passing a list of files, or a folder, it will actually zip them and upload the zip file. Specifically to pipeline it should be similar. BTW I think you can change the number of parallel upload threads in StorageManager, but as you mentioned it is faster to zip into one file. Make sense?

SubstantialElk6 Ohh okay I see.
Let's start with background on how the agent works:
When the agent pulls a job (Task), it will clone the code based on the git credentials available on the host itself, or based on the git_user/git_pass configured in ~/clearml.conf
https://github.com/allegroai/clearml-agent/blob/77d6ff6630e97ec9a322e6d265cd874d0ab00c87/docs/clearml.conf#L18
The agent can work in two modes:
Virtual environment mode, where it will create a new venv for each experiment ba...

Hi SubstantialElk6
No need for that, you can use the helm chart (or spin them once with kubctl) then they take care of scheduling by themselves.
You can also use the k8s glue (basically spinning kubernetes pods automatically for you, based on the Tasks that you push into the ClearML queue)
https://github.com/allegroai/clearml-agent/blob/master/examples/k8s_glue_example.py

In short, two possible deployments
Static k8s pod running the agent (then the agent runs all the experiments inside t...

Hi RoughTiger69
A. Yes makes total sense . Basically you can use Task.export Task.import to do achieve this process (notice we assume the dataset artifacts links are available on both, usually this is the case)

B. The easiest way would be to use Process , then one subprocess is exporting from dev , where the credentials and configuration is passed with os environment. The another subprocess imports it to the prod server (again with os environment pointing to the prod server). Make sense?

Hi SubstantialElk6

saved in the files_server (indicated in ClearML.conf) instead of the indicated output_uri in the dataset.create argument

What's the clearml SDK version ? how are you specifying the output target?

Yes 🙂
BTW: do you guys do remote machine development (i.e. Jupyter / vscode-server) ?

It is way too much to pass on env variable 😞

👍

The easiest is to pass an entire trains.conf file

Nice!
is trainsConfig pure text blob ?

an implementation of this kind is interesting for you or do you suggest to fork

You mean adding a config map storing a default trains.conf for the agent?

Hi ReassuredOwl55

How would I find Tasks that have the same code with different inputs/parameters?

Assuming you have the git repo
you can do:
Task.query_tasks(..., task_filter={'_all_'=dict(fields=['script.repository'], pattern='github.com/user/repo'))wdyt?

Hi JitteryCoyote63
Is this close ?
https://github.com/allegroai/clearml/issues/283

and the step is "queued" or is it "queued" in the pipeline state (i.e. the visualization did not update) ?

WackyRabbit7 I might be missing something here, but the pipeline itself should be launched on the "pipelines" queue, is the pipeline itself running? or is it the step itself that is stuck in ""queued" state?

JuicyFox94
NICE!!! this is exactly what I had in mind.
BTW: you do not need to put the default values there, basically it reads the defaults from the package itself trains-agent/trains and uses the conf file as overrides, so this section can only contain the parts that are important (like cache location credentials etc)

Hi JuicyFox94
you pointed to exactly the issue 🙂
In your trains.conf
https://github.com/allegroai/trains/blob/f27aed767cb3aa3ea83d8f273e48460dd79a90df/docs/trains.conf#L94

I think this is great! That said, it only applies when you are spining agents (the default helm is for the server). So maybe we need another one? or an option?

and I install the tar

I think the only way to do that is add it into the docker bash setup script (this is a bash script executed before Task)

Ohh is this a conda installed ?

Gitlab has support for S3 based cache btw.

This might still be considered "slow" compared to local-dist/cluster mount

Would adding support for some sort of post task script help? Is something already there?

Interesting, can you expand on the use case? (currently there is only pre-task script, for setup)

But functionality is working

Awesome , I will wait with the merge until tested internally .
There is a resale coming out after the weekend, once it is out I expect we will merge it.

(or woman or in between, we are supportive as long as code is working 🙂 )

YEY!

Sounds good, I assumed that was the case but I was not sure.
Let's make sure that in the clearml.conf we write it in the comment above the use_credentials_chain option, so that when users look for IAM roles configuration they can quick search for it 🙂

😞 let me verify

Hi @<1610083503607648256:profile|DiminutiveToad80>

<h1>Request Entity Too Large</h1>

What's the size of the file? how are you running your clearml-server?