https://app.clear.ml 🙂

Hi ReassuredTiger98
I think you should have something like:

` @PipelineDecorator.component(task_type=TaskTypes.application, docker='clara_docker_container_if_we_need')
def step_one(param):
print('step_one')
import os
os.system('run me clara')
# I'm assuming we should wait?
return

@PipelineDecorator.component(task_type=TaskTypes.training)
def step_two(param):
print('step_two')
import something
somthing.to_do()
return

@PipelineDecorator.pipeline(name='c...

Task.init(..., output_uri='s3://...')

Hi BattyLion34
The windows issue seems like it is coming from missing QT installed on the Host machine
Check the pyqt5 version in your "Installed packages"
see here:
https://superuser.com/questions/1433913/qtpy-pythonqterror-no-qt-bindings-could-be-found

Regrading the linux, it seems your are missing the object_detection package, where do you usually install it from ?

BTW: Can you also please test with the latest clearml version , 1.7.2

Hm GiganticTurtle0 let me check quickly it

Should have worked, the error you are getting is docker-compose parsing the yml file
Is this exactly the one from the trains-server repo ?

RipeGoose2 models are automatically registered
i.e. added to the models artifactory, but it only points to where the files are stored
Only if you are passing the output_uri argument to the Task.init, they will be actually uploaded.
If you want to disable this behavior you can pass
Task.init(..., auto_connect_frameworks={'pytorch': False})

Hmm, #790 should be solved in 1.7.2
Yes, I always see the "model uploaded completed" for such stuck tasksAny chance this is reproducible ?
How many processes do you see running (i.e. ps -Af | grep python) ?
What is the training framework? is it multiprocess ? how are you launching the process itself? is it Linux OS? is it running inside a specific container ?

Does this mean the model weights are stored on the clearml-server file system?

By default they are just logged (i.e. the local path is stored, but the file is not uploaded). If you want to automatically store the model, pass output_uri=True to the Task.init , or any object store / shared folder (e.g. output_uri=' s3://bucket/folder ' ). ClearML will automatically create a subfolder for the Task, and upload all models/artifacts to it.
` task = Task.init(project_name='ex...

BTW: there is a fix to the priority thing:
https://github.com/allegroai/trains-agent/blob/216b3e21790659467007957d26172698fd74e075/docs/trains.conf#L77

https://www.geeksforgeeks.org/invalid-decimal-literal-in-python/

This is the warning hence my question

The problem is, the configuration is loaded at import time, so there is no "time" to pass anything other than environment variable.
That said if the only difference is server config you can use
Task.set_credentials

Are you sure trains-server not trains package (i.e. backend)

How can the first process corrupt the second

I think that something went wrong and both Agents are using the same "temp" folder to setup the experiment.

why doesn't this occur if I run pipeline from command line?

The services queue is creating new dockers with everything in them so they cannot step on each others toes (so to speak)

I run all the processes as administrator. However, I've tested running the pipeline from command line in non-administrator mode, it works fine....

Hi @<1724960464275771392:profile|DepravedBee82>
After

Starting Task Execution:

It will literally start the process running your code,
Can you send the full log of the Task? what is the code doing? which system is running the agent (i.e. Windows/Mac/Linux docker etc)

Hi SubstantialElk6

We can't seem to find a way for the end user to pass in their git credentials when they run their codes in both agent and non-agent scenarios. Any advice here?

The bottom line is the agent needs to have read-only access to all the repositories so it can launch any Task. I would recommend to create an agent git user with read-only credentials and configure the agent to use it. wdyt?

(I think it is the empty config file)

Interesting...
We could followup the .env configuration, and allow the clearml-task to add configuration files from cmd line. This will be relatively easy to add. We could expand the Environment support (that somewhat exists), and add the ability to read variables from .emv and Add them to an "hyperparemeter" section, named Environment. wdyt?

Interesting use case, do you already have the connect_configuration in the code? or do we need to somehow create it ?

Hi @<1727497172041076736:profile|TightSheep99>
I think you are correct! it will use the internal individual file upload retry but does not let you control it.
Could you please open a github issue so that we do not forget to add it?

Yes, the agent's mode is global, i.e. all tasks are either inside docker or in venv. In theory you can have two agents on the same machine one venv one docker listening to two diff queues

What about output_uri?

If you are using StorageManager directly, output_uri is not relevant

I can't think of any hack that will satisfy your IT other than than an actual vault...
wdyt?

I want to be able to delete only the logs since they are taking a lot of space in my case.

I see... I do not think this is possible 😞
You can disable the auto logging though ... pass auto_connect_streams=False to Task.init

Any idea where that could come from? Could we turn off the local logging as well - in these kinds of runs we don’t need it?

It is supposed to create it automatically... I tested with other examples (clearml version 1.7.3rc1) everything seems to work
What am I missing? how do we recreate the issue ? can you verify it is still not working with the latest RC?

I'm not familiar with this one, I think you should be able to control it with:
None

CLEARML_AGENT__API__HTTP__RETRIES__BACKOFF_FACTOR

Do people generally update the same model “entry”? That feels so wrong to me…how do you reproduce a older model version or do a rollback etc?

Correct, they do not 🙂 On the Task itself the output models will reflect the diff filenames you saved, usually ppl just add a running number.

Which means there will be atleast multiple published models entries of same model over time?

Only the specific one will be published (not all the Models the Task created)

sounds good, CheerfulGorilla72 could I ask you to open a github issue and suggest it? just so we do not forget ?