What about amount of storage required?

Here's the thread
https://clearml.slack.com/archives/CTK20V944/p1636613509403900

The question has been answered though you can take a look if I understood correctly there.

You could be right, I just had a couple of packages with this issue so I just removed the version requirement for now. Another issue that might be the case, might be that I'm on ubuntu some of the packages might've been for windows thus the different versions not existing

I just made a custom repo from the ultralytics yolov5 repo, where I get data and model using data id and model id.

Understandable. I mainly have regular image data, not video sequences so I can do the train test splits like you mentioned normally. What about the epochs though? Is there a recommended number of epochs when you train on that new batch?

Alright. Anyway I'm practicing with the pipeline. I have an agent listening to the queue. Only problem is, it fails because of requirement issues but I don't know how to pass requirements in this case.

Any way to make it automatically install any packages it finds that it requires? Or do I have to explicitly pass them in packages?

Considering I don't think the function itself requires Venv to run normally but in this case it says it can't find venv

Can you spot something here? Because to me it still looks like it should only create a new Dataset object if batch size requirement is fulfilled, after which it creates and publishes the dataset and empties the directory.

Once the data is published, a dataset trigger is activated in the checkbox_.... file. which creates a clearml-task for training the model.

Let me share the code with you, and how I think they interact with eachother.

I'll test it with the updated one.

I'll read the 3 examples now. Am I right to assume that I should drop Pipeline_Controller.py

I download the dataset and model, and load them. Before training them again.

I recall being able to pass a script to the agent using the command line along with a requirements file.

There's data when I manually went there. The directory was originally hidden my bad.

AgitatedDove14 Your second option is somewhat like how shortcuts work right? Storing pointers to the actual data?

I think maybe it does this because of cache or something. Maybe it keeps a record of an older login and when you restart the server, it keeps trying to use the older details maybe

Also, is clearml open source and accepting contributions or is it just a limited team working on it? Sorry for an off topic question.

Ok this worked. Thank you.

let me check

To me it still looks like the only difference is that the non mutable copy is downloaded to the cache folder while mutable copy downloads to the directory I want. I could delete files from both sets so it seems like it's up to the user to make sure not to mutate the non mutable download in the cache folder.

I followed the instructions here at https://allegro.ai/clearml/docs/docs/deploying_clearml/clearml_server_linux_mac.html

I feel like they need to add this in the documentation 😕

Big thank you though.

Finalizes locks the model and publish I assume publishes it to the server

os.listdir(location) shows nothing

How would the two be different? Other than I can pass the directory to local mutable copy

The scheduler is set to run once per hour but even now I've got around 40+ anonymous running tasks.

Should I not run the scheduler remotely if I'm monitoring a local folder?

Yeah