My use case is basically if I want to now access this dataset from somewhere else, shouldn't I be able to do so using its id?

Yeah

I'll take a look thank you

With online learning, my two main concerns are that the training would be completely stochastic in nature, I would not be able to split the data into train test splits, and that it would be very expensive and inefficient to train online.

Yeah, I kept seeing the message but I was sure there were files in the location.

I just realized, I hadn't worked with the Datasets api for a while and I forgot that I'm supposed to call add_files(location) and then upload, not upload(location). My bad.

Can you give me an example url for the api call to stop_many?

Should I just train for 1 epoch? Or multiple epochs? Given I'm only training on the new batch of data and not the whole dataset?

I'm currently installing nvidia docker on my machine, where the agent resides. I was also getting an error regarding gpu not being available in docker since the agent was running on docker mode. I'll share update in a bit. Trying to re run the whole set up

It says command not found.

I want to maybe have a variable in the simple-pipeline.py, which has the value returned by split_dataset

Creating a new dataset object for each batch allows me to just publish said batches introducing immutability.

Sorry for the late response. Agreed, that can work, although I would prefer a way to access the data by M number of batches added instead of a certain range, since these cases aren't interchangeable. Also a simple thing that can be done is that you can create an empty Dataset in the start, and then make it the parent of every dataset you add.

dataset._task.mark_started()

Even though I ended my schedulers and triggers, the anonymous tasks keep increasing.

I'm not sure what dataset task is. I mainly just created dataset using ClearML.Dataset.Create

I don't think so. Also I fixed it for now. Let me mention the fix. Gimme a bit

Can you please share the endpoint link?

I have the server running now and for now it seems that I'm able to get the dataset even in the other file. I'll mess around with it now to get a hang of it and see how it actually works

My main query is do I wait for it to be a sufficient batch size or do I just send each image as soon as it comes to train

I've also mentioned it on the issue I created but I had the issue even when I set the type to bool in parser.add_argument(type=bool)

Understandable.

I've finally gotten the triton engine to run. I'll be going through nvidia triton docs to find how to make an inference request. If you have an example inference request, I'll appreciate if you can share it with me.

For anyone reading this. I think I've gotten an understanding. I can add folders to a dataset so I'll be creating single dataset and will just keep adding folders to it. Then keep records of it in a database

I'm not sure about auto logging, since you might be using different datasets or you might get a dataset but might not use it based on specific conditions. However as a developer choosing to use such as ClearML who considers it more of an ecosystem instead of just a continuous training pipeline, I would want as many aspects of the MLOPS process and the information around the experiment to be able to be logged within the bounds of ClearML without having to use external databases or libraries.

Yeah

Is the only possible way to get a specific node, is to use one of the get_running_nodes or get_processed_nodes, and then checking every node in the list to see if the name matches the one we're looking for?

AgitatedDove14

Okay so I read the docs and the above questions are cleared now thank you. I just have one other question, how would I access the artifact of a previous step within the pre execute callback? Can you share an example?

Another question, in the parents sequence in pipe.add_step, we have to pass in the name of the step right?

It does to me. However I'm proposing a situation where a user gets N number of Datasets using Dataset.get, but uses m number of datasets for training where m < n. Would it make sense to only log the m datasets that were used for training? How would that be done?