Hi @<1603198134261911552:profile|ColossalReindeer77>
When you select poetry as package manager the agent passes control to poetry, this means poetry needs to decide on hte correct torch wheel based on your cuda. I do not think poetry can do that, but I do think you can specify the extra index url to take the torch wheel from:
None

Yep the automagic only kick in with Task.init... The main difference and the advantage of using a Dataset object is the underlying Task resides in a specific structure that is used when searching based on project/name/version, but other than that, it should just work

(once you verify PR the fix, I'll make sure it is merged)

Hi @<1631102016807768064:profile|ZanySealion18>

I'm using SSH for authentication, however, known_hosts doesn't seem to be passed to the docker so it prompts for authentification/fingerprint. Any ideas?

Hmm it is supposed to automatically mount your ~/.ssh folder into the docker to solve for that.
First try to set force_git_ssh_protocol: true
None
If that does not he...

Hmm not sure, try the latest anyhow 🙂

instead of the one that I want or the one of the env which it is started from.

The default is the python that is used to run the agent.
agent.ignore_requested_python_version = true agent.python_binary = /my/selected/python3.8

Oh that is odd. Is this reproducible? @<1533620191232004096:profile|NuttyLobster9> what was the flow that required another task.init?

No sure what O'm seeing here

I was using clearml == 0.17.5 and I also had this issue

I think it was introduced when we moved to subprocess reporting, with 0.17.5
You can disable it with the following in clearml.conf:
sdk.development.report_use_subprocess = false

Hi SkinnyPanda43
Are you trying to access the same Task or an external one ?

Ohh "~/trains.conf" is root probably

So does that mean "origin" solves the issue ?

great 🙂
two things:
I'm not sure argparse supports dict as a type (I mean it will take anything but I'm not sure it will parse your arguments as dict) I know there was an issue with argparsing, but I think it was solvedbtw: Basically the way clearml-agent works, it does not actually pass the arguments in commandline but directly to the argparser at runtime
What happens if you clone the Task (the one with Args showing and without the explicit task.connect(_args) and send it to the age...

MysteriousBee56 I would do Task.create()
you can get the full Task internal representation with task.data
Then call task._edit(script={'repo': ...}) to edit/update all the Task entries.
You can check the dull details of the task object here: https://github.com/allegroai/trains/blob/master/trains/backend_api/services/v2_8/tasks.py#L954
BTW: when you have a sample script working, consider PR-ing it, I'm sure it will be useful for others 🙂 (also a great way to get us involved with debuggin...

How does ClearML select reference branch? Could it be that ClearML only checks "origin" branch?

Yes 😞 I think we can quickly fix that, I'm just trying to realize if there are down sides to running "git ls-remote --get-url" without origin

Oh that makes sense, This depends on how you setup the clearml k8s glue, (becuase the resource allocation is done by k8s) a good hack to limit the number of containers per GPU is to set a RAM limitation per pod, then k8s will know to limit the number of pods on the same GPU machine,
wdty?

I think this is the only mount you need:

Data persisted in every Kubernetes volume by ClearML will be accessible in /tmp/clearml-kind folder on the host.

SuccessfulKoala55 is this correct ?

Notice the error code:
Action failed <400/401: tasks.create/v1.0 (Invalid project id: id=first_attempt)>If that is the case, The project ID is incorrect (project id is not the project name)

Ephemeral Dataset, I like that! Is this like splitting a dataset for example, then training/testing, when done deleting. Making sure the entire pipeline is reproducible, but without storing the data long term?

will my datasets be stored on the same machine that hosts the clearml server?

By default yes, they will be stored to the files-server (but you can change it, this is an argument for both the CLI and the python interface)

so I guess this could be one reason to start about thinking upgrading ....

Wait you mean the clearml-server ? (there is no reason not to upgrade the python package)

Hi JitteryCoyote63
could you check if the problem exists in the latest RC?
pip install clearml==1.0.4rc1

CharmingBeetle38 try adding "General/" before the arguments. This means batch_size becomes General/batch_size. This is only because we are accessing the parameters externally, when the task is executed it is resolved automatically

Hi CharmingBeetle38
On the base task, do you see those arguments under the Configuration tab?
Also, if they are under Args section, you should add "Args/" prefix to the HP optimization (this is how you differentiate between the sections)

So the issue is that you have two reference branches on the local git, one to gitlab one to gitea and it fails to understand which on is the correct remote ...
I wonder if "git ls-remote --get-url" will always work ?!

CharmingStarfish14 can you check something from code, just to see if this would solve the issue?

Hi AttractiveCockroach17

. Many of these experiments appear with status running on clearml even though they have finish running,

Could it be their process just terminated? (i.e. not properly shutdown) ?
How are you running these multiple experiments?
BTW: if the server does not see any change in a Task for (I think the default is 2 hours) it will automatically mark these Task as aborted

You mean like for your internal support channel inside your company ?

Hi AttractiveCockroach17
when exactly are you seeing this error ?

What do you mean? every Model has a unique ID, what do you consider a version?