if project_name is None and Task.current_task() is not None: project_name = Task.current_task().get_project_name()This should have fixed it, no?

You mean Task.clone() API call failed ?

GreasyPenguin14 we never had troubles with Task.init (or any other clearml calls) and working with the pycharm debugger, we use it quite extensively ...
Actually on a very similar setup...
Could you send the full log?
Or maybe a code snippet to reproduce this behavior ?
(We did notice they fixed a few issues with the debugger in 2020.3.3 so it's worth upgrading)

GreasyPenguin14 could you test with the 0.17.5rc4 ?
Also what's the PyCharm / OS?

pip install clearml==0.17.5rc4

Not really 😞
Everyone can do everything, the idea is sharability and accessibility.
I do know that in the paid tier they have full access control roles SSO etc, but unfortunately its way too complicated for the open-source.
Basically what I'm saying is trust your fellow colleagues 🙂

sure no prob

DilapidatedDucks58 no don't say that, you are wonderful 😉

trains-agent --gpus 0 --queue my_queue -d
should create a worker machine:gpu0
Then you can do trains-agent --gpus 1 --queue my_queue -d which will create machine:gpu1

TRAINS_WORKER_NAME=first_agent trains-agent --gpus 0
and
TRAINS_WORKER_NAME=second_agent trains-agent --gpus 0

Ohhhh , okay as long as you know, they might fall on memory...

You mean why you have two processes ?

I think this one is on us, I don't think a search would have led you to the correct answer ...
I'll try to make sure they add something regrading the configuration 🙂

We should probably have a section on that (i.e. running two agents on the same GPU, then explain how top use it)

https://github.com/allegroai/trains-agent/blob/master/docs/trains.conf#L25

let me check

🙂

MysteriousBee56 , The agent is not running on the "server" it's running on its machine.
The server just reflects the fact he agent is up..
To actually take it down you need to SSH (or connect to that machine) and stop the actual trains-agent process.
What is exactly the scenario you had in mind?

not sure what is the "right way" 🙂
But I do pkill -f "trains-agent --gpus 0" This will kill a process that started "trains-agent --gpus 0" Notice it matches the cmd pattern so it has to match the way you executed the agent. You can check it with ps -Af | grep trains-agent

MysteriousBee56 what do you mean "delete a worker"
stop the agent running remotely ?

Ohh now I get it...
Wait a couple of hours, 0.16 is out today with trains-agent --stop flag 🙂

I know there is a aux cfg with key value pairs but how can use it in the python code?

This is actually for helping to configure Triton services, you cannot (I think) easily access it from the code

now, I need to pass a variable to the Preprocess class

you mean for the construction ?

it worked!!!!

YEY!

I pass the IDs to the docker container as environment variables, so this does need restart for the docker container but I guess we can live with that for now

So this would help you decide on which actual Model file to download ? (trying to understand how the argument is being used, meaning should we have it stored somewhere? there is meta-data on the Model itself so we can use that to store the data)

Hmm yes, that is a good point, maybe we should allow to specify a parameter on the model configuration to help with the actual type ...

I wonder if the try/except approach would work for XGboost load, could we just try a few classes one after the other?

DS, this way they only need to remember (and me only need to teach them where to find) one id.

Yes that's the point, this ID is the Model UID (as opposed to the Task ID), the reason I kind if "insist" on it is that the Model ID is built into the system meaning, this is how you register it, as opposed to the Task ID that somehow needs to be hacked/passed externally

TBH the main reason I went with our API is that because of the custom model loading, we need to use the "custom" framew...

and then in Preprocess:

self.model = get_model(task_id=os.environ['TASK_ID'], model_name=os.environ['MODEL_NAME'])That's the part I do not get, Models have their own entity (with UID), this is in contrast to artifacts that are only stored on Tasks.
The idea when you are registering a model with clearml-serving, you can specify the model ID, this should replace the need for the TASK_ID+model_name in your code, and the clearml-serving will basically bring it to you
Basically this fun...

Can the user be overwritten during task configuration (I don't see such an option in the documentation)?

Hm, not really 😞 this is tied with security feature on top.
That said,

stored them as a k8s secret and they are reused whenever anyone from our ML team starts a new ML model training

Does that mean you are running an Agent on the k8s cluster? what's exactly the flow that causes your k8s credentials to be used

Even before we had a chance to properly notice everyone 🙂
Thank you! All the details will follow in a dedicated post, for the time being, I can say that pushing a model with pre/post processing python code and full scalable inference solution has never been easier
https://github.com/allegroai/clearml-serving/tree/main/examples/sklearn

and those env variables are credentials for ClearML. Since they are taken from k8s secrets, they are the same for every user.

Oh ...

I can create secrets for every new user and set env variables accordingly, but perhaps you see a better way out?

So the thing is, if a User spins the k8s job, the user needs to pass their credentials (so the system knows who it is)... You could just pass the user's key/secret (not nice, but probably not a big issue, as everyone is an Admin anyhow,...