We have to do it in-premise.. Cloud providers are not allowed for the final implementation. Of course, now we use Cloud to test out our ideas.

So now you don’t have any failures but gpu usage issue?

I didnt run the hyper_parameter_optimzer.py, as I was thinking if there is already a problem with the base, no use with running the series of experiments

How about running the ClearML agent in docker mode?

Prev, we had our clearml-agent run in the bare-metal machine instead in docker formation. There wasnt any issue.. Though I havent tried with 0.17.2 version

It is like generating a report per Task level (esp for Training Jobs).. It's like packaging a report out per Training job..

we also might have some other steps incorporated for other tools. We intend to have Label-Studio upstream.. So defintely needed some orchestrator tool

Yes, I am already using a Pipeline.

2.  I have another project build using the Pipeline. The pipeline always loads the last commited dataset from the above Dataset project and run few other stuff.

Just not sure, how to make the Pipeline to listen to changes in the Dataset project.

Hi AgitatedDove14 , Now we prefer to run dynamic agents instead using
python3 k8s_glue_example.pyIn this case, is it still possible to pass --order-fairness at the queue level or this is more of a Enterprise edition feature.

However, I am able to get it to work, if I launch a clearml-agent outside the kubernetes ecosystem.

Nice, tks

Yeah, currently we are evaulating Seldon.. But was wondering whether clearml enterprise version wud do something similar ?

sure, I'll post some questions once I wrap my mind around it..

TimelyPenguin76 : Yup that's what I do now.. However, shld config to use some distributed storage later

Sure tks.. can

In our local setup, we use minio though ?

This is where I downloaed the log. Seems like some docker issue, though i cant seem to figure it out. As an alternative, I spawned a clearml-agent outside the k8 environment and it was able to execute well.

Is there any documentation on how, we can use this ports mode ? I didnt seem to find any.. Tks

The use case, is lets say i run
python k8s_glue_example.py --queue glue_qAnd some guys pushes an hyperparameterization job with over 100 experiments to the glue_q, one minute later, I push a simple training job to glue_q.. But I will be forced to wait for the 100 experiments to finish.

Hi, we choose prefect ...

Ah kk, it is ---laptop:0 worker is no more now.. But wrt to our original qn, I can see the agent(worker) in the clearml-server UI ..

Sound Perfect. 👍

Yeah, I restarted the deployment and sshed into the host machine also.. (Img below)

Hmm, unfortutenly it is still pending as in nothing is running

kkie..now I get it.. I set up the clearml-agent on an EC2 instance. and it works now.
Thanks

yeah, seems good for now.. Tks

Hi AgitatedDove14 , Thanks for the explanation .
python k8s_glue_example.py --queue high_priority_q --ports-mode --num-of-services 10 python k8s_glue_example.py --queue low_priority_q --ports-mode --num-of-services 2Would the above be a good way to simulate the below ?
clearml-agent daemon --queue high_priority_q low_priority_q

Hi AgitatedDove14 ,
At this point, Showing the url of the cleamltask might be sufficient. Unless in the future, someone wants it to be customised.
But the bigger question is if there is tool to aid with this workflow building ? We are currently experimenting with airflow/prefect.

RoughTiger69
So prefect tasks :
Loads Data into clearml-data Runs trainining in clearml Publish model (manual trigger required, so user publishes model) and return model url Seldon deploys the model ( model url passed in)

kkie, tks.. I will connect with you offline .

we can always use the latest Clearml.
We were thinking of a use case for a client, who has Sonatype Nexus in their environment ? Could we leverage on it, or would we need minio instead ?