I'll take a look at it, thank you.

after creating, I tried adding batch to it and got this error

On both the main ubuntu and the vm, I simply installed it in a conda environment using pip

I'm kind of new to developing end to end applications so I'm also learning how the predefined pipelines work as well. I'll take a look at the clear ml custom pipelines

do I just post the issue on the main clearml repo?

I just assumed it should only be triggered by dataset related things but after a lot of experimenting i realized its also triggered by tasks, if the only condition passed is dataset_project and no other specific trigger condition like on publish or on tags are added.

However, since a new task started in the project, it would again start a new task.

then I use trigger_scheduler.start()

For anyone facing a similar issue to mine and wanting the model to uploaded just like data is uploaded,

in the Task.init, set the output_uri = True.

This basically makes it use the default file server for clearml that you define in the clearml.conf file. Ty.

Basically the environment/container the agent is running in needs to have specific cuda installed. Is that correct CostlyOstrich36 ?

Basically want to be able to serve a model, and also send requests to it for inference.

Yeah exact same usage.

shouldn't checkpoints be uploaded immediately, that's the purpose of checkpointing isn't it?

It was nargs="?" I think

there are other parameters for add_task as well, I'm just curious as to how do I pass the folder and batch size in the schedule_fn=watch_folder part

I've also mentioned it on the issue I created but I had the issue even when I set the type to bool in parser.add_argument(type=bool)

Thanks for the help.

I'm both printing it and writing it to a file

Alright. Anyway I'm practicing with the pipeline. I have an agent listening to the queue. Only problem is, it fails because of requirement issues but I don't know how to pass requirements in this case.

It's an artifact.

I would normally like for it to install any requirements needed on its own.

It works this way. Thank you.

I need to both remove the artifact from the UI and the storage.

My current approach is, watch a folder, when there are sufficient data points, just move N of them into another folder and create a raw dataset and call the pipeline with this dataset.

It gets downloaded, preprocessed, and then uploaded again.

In the final step, the preprocessed dataset is downloaded and is used to train the model.

AgitatedDove14 CostlyOstrich36 I think that is the approach that'll work for me. I just need to be able to remove checkpoints I don't need given I know their name, from the UI and Storage.

Is there a difference? I mean my use case is pretty simple. I have a training and it basically creates a lot of checkpoints. I just want to keep the n best checkpoints and whenever there are more than N checkpoints, I'll delete the worst performing one. Deleted both locally and from the the task artifacts.

Sorry for the late response. Agreed, that can work, although I would prefer a way to access the data by M number of batches added instead of a certain range, since these cases aren't interchangeable. Also a simple thing that can be done is that you can create an empty Dataset in the start, and then make it the parent of every dataset you add.

Thank you, I found the solution to my issue, when I started reading at default output uri.

adding tags this way to a Dataset object works fine. This issue only occured when doing this to a model.

Thanks, I went through it and this seems easy