No no, I mean now i can export a csv file into clearml-data. I was wondering if it possible to export directly from a sql database.

HI another qn,
dataset_upload_task = Task.get_task(task_id=args['dataset_task_id'])
iris_pickle = dataset_upload_task.artifacts['dataset'].get_local_copy()
How would I replicate the above for Dataset ? Like how to get the iris_pickle file. I did some hacking likewise below.
ds.get_mutable_local_copy(target_folder='data')
Subesequently, I have to load the file by name also.I wonder whether there is more elegant way

yup, i updated this in my local clearml.conf... Or should be updating this elsewhere as well

Thanks , Got it

AgitatedDove14 Full Log as requested.

Hi, sorry for the delayed response. Btw, all the pods are running all good.

This is my example. Iteration 10 so there are 10 runs. Looking at the 4th run, 60% of the jobs, 91% iteration, 94% time.. What does it mean ?

Github Issue : https://github.com/allegroai/clearml-agent/issues/50
AgitatedDove14 , Have added the github issue as requested. Thanks for the help. 👍

hi FriendlySquid61 , The clearml-agent got filled up due to values.yaml file. However, agentservices was empty so I filled it up manually..

` python3 k8s_glue_example.py --queue glue_high_q glue_low_q
usage: k8s_glue_example.py [-h] [--queue QUEUE] [--ports-mode] [--num-of-services NUM_OF_SERVICES] [--base-port BASE_PORT] [--base-pod-num BASE_POD_NUM] [--gateway-address GATEWAY_ADDRESS]
[--pod-clearml-conf POD_CLEARML_CONF] [--overrides-yaml OVERRIDES_YAML] [--template-yaml TEMPLATE_YAML] [--ssh-server-port SSH_SERVER_PORT] [--namespace NAMESPACE]
k8s_glue_example.py: error: unrecognized arguments: glue...

kkie.. Will try it .. tks

Is there any documentation on how, we can use this ports mode ? I didnt seem to find any.. Tks

Hi AgitatedDove14
I am still not very clear on using this, even after looking at k8s_glue_example.py 's code
Is it possible to give a sample usage of how this works ?
python k8s_glue_example.py --ports-mode --num-of-servicesAnother question, I am still not sure , how this resolves my original question.
https://github.com/allegroai/clearml-agent/issues/50#issuecomment-811554045
How will imposing an instance limit , prevent or allow --order-fairness feature for example, which ex...

Sure tks.. can

Yup, tried that.. Same error also

Nothing changed.. the clearml.conf is still as is (empty)

we also might have some other steps incorporated for other tools. We intend to have Label-Studio upstream.. So defintely needed some orchestrator tool

The above screenshot is from my local settings... My agents run in the k8s system (like in a pod)

Hi AgitatedDove14 , imho links are def better, unless someone decides to archive their Tasks.. Just wondering about the possibility only..

Hi SuccessfulKoala55 , kkie..
1)Actually, now i am using AWS. I am trying to set up Clearml server in K8. However, clearml-agents will be just another ec2-instance/docker image.
2) For phase 2, I will try Clearml AWS AutoScaler Service.
3) At this point, I think I will have a crack at JuicyFox94 's solution as well.

Thanks JuicyFox94 .
Not really from devops background, Let me try to digest this.. 🙏

We have to do it in-premise.. Cloud providers are not allowed for the final implementation. Of course, now we use Cloud to test out our ideas.

Our main goal, maybe I shld have stated prior. We are data scientists who need a mlops environment to track and also run our experiments..

sure, I'll post some questions once I wrap my mind around it..

nice.. this looks a bit friendly.. 🙂 .. Let me try it.. Thanks

Hi SuccessfulKoala55
I have set it True some time back already

Just to add on, I am using minikube now.

TimelyPenguin76 : Yup that's what I do now.. However, shld config to use some distributed storage later

TimelyPenguin76 :
from clearml import Dataset ds = Dataset.get(dataset_project="PROJ_1", dataset_name="dataset")