i mean they are the same...

This is from my k8 cluster. Using the clearml helm package, I was able to set this up.

Thanks.

TimelyPenguin76 :
from clearml import Dataset ds = Dataset.get(dataset_project="PROJ_1", dataset_name="dataset")

Nice tutorial.. Though personally, I prefer a more clean-cut presentation (without the Yays and muaks or the the turtle). 😄 But usually, as long as content is there, it shldnt matter...

Something is wierd.. It is showing workers which are not running now...

Ah kk.. Got it. Thanks..

nice... we need moarrrrrrrr !!!!!!!!
It wud be really helpful, if you cud do the next episode on setting up clearml in kubernetes.. 😇
In anyways, keep up the good work for the community

It'll be good if there was yaml file to deploy clearml-agents into the k8 system.

Hi, using the pipeline examples, with
step1_dataset_artifact.py, step2_data_processing.py, step3_train_model.py ==> pipeline_controller.pyIn the above example, the pipeline_controller is stringing together 3 python files, instead could it string together 3 containers instead. Of course, we can manually compile each into a docker image, but does clearml has some similar approach baked in.

Hi martin, i just untemplate-ed the
helm template clearml-server-chart-0.17.0+1.tgzI found this lines inside.
- name: CLEARML_AGENT_DOCKER_HOST_MOUNT value: /opt/clearml/agent:/root/.clearmlUpon ssh-ing into the folders in the both the physical node (/opt/clearml/agent) and the pod (/root/.clearml), it seems there are some files there.. So the mounting worked, it seems.
I am not sure, I get your answer. Should i change the values to something else ?
Thanks

kkie..now I get it.. I set up the clearml-agent on an EC2 instance. and it works now.
Thanks

Thanks guys.. perfect..

i ran this in my local machine..
clearml-task --project playground --name tensorboard_toy --script tensorboard_toy.py --requirements requirements.txt --queue myqueue

For the clearml-agent deployment file, I updated this line
python3 -m pip install clearml-agent==0.17.2rc4and restarted the deployment. However the conf file is still empty.

Should I also update the clearml-agent-services as well in the clearml-agent-services deployment file ?

I mean PROJ_1 gets updated from an external source...

yup, i updated this in my local clearml.conf... Or should be updating this elsewhere as well

Yeah, that worked.. As I was the running the agent in a different machine as our deployment of clearml was in k8s.

Hi AgitatedDove14 , I also fiddled around by changing this line and restarted the deployment. But this just causes it revert back 0.17.2rc4 again.
python3 -m pip install clearml-agent==0.17.2rc3

Ah kk, it is ---laptop:0 worker is no more now.. But wrt to our original qn, I can see the agent(worker) in the clearml-server UI ..

Ah, so in the future, we can add non-clearml code as a step in the pipeline controller.

what does a control plane do ? I cant understand this..
Like the serving engine, will get the user input, preprocess, infer it and send back the results..

Currently, in the diagram here.. Clearml File server is shown as a local storage drive. Our 2 primary concerns.
Is there any ways , we can scale this file server when our data volume explodes. Maybe it wouldnt be an issue in the K8s environment anyways. Or can it also be configured such that all data is stored in the hdfs (which helps with scalablity). Is there any security to protect this data in this storage ?

I just checked the /root/clearml.conf file and it just contains
sdk{ }

👌

Maybe more of data repository than a model repository...

AgitatedDove14 We too self host (on prem) the helm charts in our local k8s ecosystem.
Triggering - Will be nice feature indeed, currently we are using clearml.monitors to address these now
Is it the UI presenting the entire workflow? - This portion will also be nice. (Let's say someone uses a 1) clearmldataset -> 2) Pipeline Controller (Contains preprocessing, training, hyperparamter tuning) -> 3) clearml-serving ).. If they can see the entire thing, in one flow
We are using seldon f...

Hi, Some walk around I thought of.. Btw, I havent tried . AnxiousSeal95 , your comments

1 ) Attach a clearml-task id to each new dataset-id
So in the future, when new data comes in, get the last data commit from the project(Dataset) and get the clearml-task for it. Then clone the clearml-task, and pass in the new data. The only downside, is the need to clone the cleaml-task.
Or alternatively
2) Attach a gitsha-id of the processing code to each new dataset-id.
This can't give the exact code ...