Good point

How would the two be different? Other than I can pass the directory to local mutable copy

Yeah, that would ensure its clarity.

I'll look into those 3. Do those files use step 1, step 2 and step 3 files though?

Also I made another thread regarding clear ml agent. can you respond to that? I'm gonna be trying to set up a clear ml server properly on a server machine. Want to test how to train models and enqueue tasks and automate this whole process with GPU training included.

I hope what I said was clear. Basically in reality they both seem mutable, with just the directory downloaded to being optional in one and in the other it's downloaded to the cache folder always.

How about instead of uploading the entire dataset to the clearml server, upload a text file with the location of the dataset on the machine. I would think that should do the trick.

is this the correct way to upload an artifact?

checkpoint.split('.')[0] is the name that I want it assigned and the second argument is the path to the file.

I want to serve using Nvidia Triton for now.

For anyone who's struggling with this. This is how I solved it. I'd personally not worked with GRPC so I instead looked at the HTTP docs and that one was much simpler to use.

My use case is that the code using pytorch saves additional info like the state dict when saving the model. I'd like to save that information as an artifact as well so that I can load it later.

I'm not sure about auto logging, since you might be using different datasets or you might get a dataset but might not use it based on specific conditions. However as a developer choosing to use such as ClearML who considers it more of an ecosystem instead of just a continuous training pipeline, I would want as many aspects of the MLOPS process and the information around the experiment to be able to be logged within the bounds of ClearML without having to use external databases or libraries.

Basically when I'm loading the model in InputModel, it loads it fine but I can't seem to get a local copy.

This here shows my situation. You can see the code on the left and the tasks called 'Cassava Training' on the right. They keep getting enqueued even though I only sent a trigger once. By that I mean I only published a dataset once.

Basically want the model to be uploaded to the server alongside the experiment results.

Thank you

I already have the dataset id as a hyperparameter. I get said dataset. I'm only handling one dataset right now but merging multiple ones is a simple task as well.

Also I'm not very experienced and am unsure what proposed querying is and how and if it works in ClearML here.

Let me share the code with you, and how I think they interact with eachother.

recursively.

I'm kind of at a point where I don't know a lot of what to even search for.

SuccessfulKoala55 Sorry to ping you like this. I have to ask. What's the minimum requirements for clear ml installation. Excluding requirements for databases or file server

I've tried the ip of the ClearML Server and the IP of my local machine on which the agent is also running on and none of the two work.

AnxiousSeal95 I just have a question, can you share an example of accessing an artifact of a previous step in the pre execute callback?

Well yeah, you can say that. In add function step, I pass in a function which returns something. And since I've written the name of the returned parameter in add_function_step, I can use it, but I can't seem to figure out a way to do something similar using a task in add_step

Wait is it possible to do what i'm doing but with just one big Dataset object or something?

AnxiousSeal95 I'm trying to access the specific value. I checked the type of task.artifacts and it's a ReadOnlyDict. Given that the return value I'm looking for is called merged_dataset_id, how would I go about doing that?

I checked and it seems when i an example from git, it works as it should. but when I try to run my own script, the draft is in read only mode.

Creating a new dataset object for each batch allows me to just publish said batches introducing immutability.

AgitatedDove14 Your second option is somewhat like how shortcuts work right? Storing pointers to the actual data?

I'm not using decorators. I have a bunch of function_steps followed by a normal task step, where I've passed a base_task_id.

I want to check the value of one of the functional steps, and if it holds true, I want to execute the task step otherwise I want the pipeline to end there, since the task step is the last one.