Hi SplendidToad10
In order to run a pipeline you first have to create the steps (i.e Tasks).
This is usually dont by running the code once (basically running any code with Task.init call will create a Task for that specific code, including the enviroement definition needed to reproduce it by the Agent)

Hi @<1523701295830011904:profile|CluelessFlamingo93>
from your log:

ImportError: cannot import name 'packaging' from 'pkg_resources' (/home/bat/.clearml/venvs-builds/3.9/lib/python3.9/site-packages/pkg_resources/__init__.py)

I'm guessing yolox/setuptools
None
Try adding to the "Installed packages"

setuptools==69.5.1

(Something about the `setup...

👍

JuicyFox94
NICE!!! this is exactly what I had in mind.
BTW: you do not need to put the default values there, basically it reads the defaults from the package itself trains-agent/trains and uses the conf file as overrides, so this section can only contain the parts that are important (like cache location credentials etc)

And this is with the latest pycharm plugin 1.1.0 ?

But that should not mean you cannot write to them, no?!

Can you also make sure you did not check "Disable local nachine git detection" in the clearml PyCharm plugin?

Hi GrievingTurkey78
First, I would look at the CLI clearml-data as a baseline for implementing such a tool:
Docs:
https://github.com/allegroai/clearml/blob/master/docs/datasets.md
Implementation :
https://github.com/allegroai/clearml/blob/master/clearml/cli/data/main.py
Regrading your questions:
(1) No, a new dataset version will only store the diff from the parent (if files are removed it stored the metadata that says the file was removed)
(2) Yes any get operation will downl...

CooperativeFox72
Could you try to run the docker and then inside the docker try to do:
su root whoami

If i point directly to the data.yaml the training starts without any problem

what do you mean? how do you know where the extracted file is?
basically:

data_path = Dataset.get(...).get_local_copy()

then you should be able to open your file with open(data_path + "/data.yaml", "rt")
doe that work?

Do people generally update the same model “entry”? That feels so wrong to me…how do you reproduce a older model version or do a rollback etc?

Correct, they do not 🙂 On the Task itself the output models will reflect the diff filenames you saved, usually ppl just add a running number.

obviously http://app.trains.domain.com will also work

The thing I don't understand is how come this DOES work on our linux setups

I do not think it actually works... I could not have find a code that will convert the ENV in the config string ...

I'll be happy to test it out if there's any commit available?

Please do, and feel free to PR it 😍
https://github.com/allegroai/clearml/blob/d3e986393ac8d1a1ea48302224962570ab8e6f9e/clearml/backend_api/session/session.py#L576
https://github.com/allegroai/clearml/blob/d3e98639...

What is the proper way to change a clearml.conf ?

inside a container you can mount an external clearml.conf, or override everything with OS environment
https://clear.ml/docs/latest/docs/configs/env_vars#server-connection

Hi NonchalantGiraffe17
You mean this documentation?
https://clear.ml/docs/latest/docs/references/api/tasks#post-tasksclone

CourageousLizard33 column order / specific selection is stored per user. If you press the share button you will have a link with all the definitions embedded on it.
Column resizing and order is in the next version release :)

the only problem with it is that it will start the task even if the task is completed

What is the criteria ?

I think it would make sense to have one task per run to make the comparison on hyper-parameters easier

I agree. Could you maybe open a GitHub issue on it, I want to make sure we solve this issue 🙂

It's a running number because PL is creating the same TB file for every run

Yes, I think the API is probably the easiest:
from clearml.backend_api.session.client import APIClient client = APIClient() project_list = client.projects.get_all() print(project_list)

Ohh RotundHedgehog76 this implies a single jupyter hub with multiple uses, is that correct ?
(if this is the case, then yes, clearml-session is definitely not the correct solution, I would look for a helm chart for jupyter hub)

BTW: UnevenDolphin73 you should never actually do "task = clearml.Task.get_task(clearml.config.get_remote_task_id())"
You should just do " Task.init() " it will automatically take the "get_remote_task_id" and do all sorts of internal setups, you will end up with the same object but in an ordered fashion

Yes even without any arguments give to Task.init() , it has everything from the server

Hi @<1523701323046850560:profile|OutrageousSheep60>
What do you mean by "in clearml server" ? I do not see any reason a subprocess call from a Task will be an issue. What am I missing ?

Can I delete logs from existing experiments on the ClearML server?

Only by resetting the Task (which would delete everything), or deleting the Task iteself.

You can also disable the auto console log, and report manually ?

Task.init(..., auto_connect_streams=False)

PunySquid88 do you want to test a fix?

UptightMouse31 You can add any metric (KPI) with "manual" logging
Logger.current_logger().report_scalar("KPI", "metric", iteration=0, value=1.1)This means you can later add a column KPI/metric to your experiment table.
Will this do the trick ?

I was thinking such limitations will exist only for published

Published Task could not me "marked started" even when with force flag

PompousBeetle71 , These are cuda versions, I'm looking for the nvidia driver version for example 440.xx or 418.xx .
The reason is, we set an OS environment for the driver, and I remember that old drivers did not support it . Basically they do not support NVIDIA_VISIBLE_DEVICES=all , so I'm trying to see if that's the case, then we could add fix .

Nicely found @<1595587997728772096:profile|MuddyRobin9> !

Nicely done DeterminedToad86 🙂
Wasn't this issue resolved by torch?