Thank you for the help, it works now.

I've tried the ip of the ClearML Server and the IP of my local machine on which the agent is also running on and none of the two work.

this is it.

You can see there's no task bar on the left. basically I can't get any credentials to the server or check queues or anything.

Another question, in the parents sequence in pipe.add_step, we have to pass in the name of the step right?

Thank you for the help guys.

the storage is basically the machine the clearml server is on, not using s3 or anything

when you connect to the server properly, you're able to see the dashboard like this with menu options on the side.

Yeah

I normally just upload the data to the ClearML server and then remove it locally from my machine but I understand that isn't what you want. A quick hack was the only thing I could come up with at the moment xd. Anyway you're welcome. Hope you find a solution.

How about instead of uploading the entire dataset to the clearml server, upload a text file with the location of the dataset on the machine. I would think that should do the trick.

Yeah, that would ensure its clarity.

Good point

I understand that storing data outside ClearML won't ensure its immutability. I guess this can be built in as a feature into ClearML at some future point.

I understand your problem. I think you normally can specify where you want the data to be stored in a conf file somewhere. people here can better guide you. However in my experience, it kinda uploads the data and stores it in its own format.

AgitatedDove14 Your second option is somewhat like how shortcuts work right? Storing pointers to the actual data?

are there other packages other than venv required on the agent? Since I'm not sure exactly what packages do I need on the agent. Since the function normally wouldn't need venv. It just adds a number by 1

I feel like they need to add this in the documentation 😕

in case of a string and incase of an object?

Thank you

Also I just want to say thanks for all the help. And this tool is brilliant how it supports an end to end pipeline in this completely new space for MLOps. You guys have been incredibly helpful and what you've made is incredible.

They want to start integrating MLOps into the ML projects here at our company for reproducibility and continual training. ClearML popped up as a potential option so they want me to design a complete pipeline for one of our projects currently being worked on. They're ...