For anyone reading this. I think I've gotten an understanding. I can add folders to a dataset so I'll be creating single dataset and will just keep adding folders to it. Then keep records of it in a database

I'm kind of new to developing end to end applications so I'm also learning how the predefined pipelines work as well. I'll take a look at the clear ml custom pipelines

Finalizes locks the model and publish I assume publishes it to the server

I'm assuming

Quick follow up question. Once I parse args, should they be directly available for i even enque the project for the first time or will i be able to access hyperparameters after running it once?

Also my execution just completed and as of yet, I can only see the hyperparameters as a report. not in a configurable form. I've just started with ClearML and am having these issues.

My draft is View Only but the cloned toy task one is in normal Draft mode.

Basically when I have to re run the experiment with different hyperparameters, I should clone the previous experiment and change the hyperparameters then before putting it in the queue?

It seems that is the case. Thank you for all your help guys.

Understandable.

I'm using clear-ml agent right now. I just upload the task inside a project. I've used arg parse as well however as of yet, I have not been able find writable hyperparameters in the UI. Is there any tutorial video you can recommend that deals with this or something? I was following https://www.youtube.com/watch?v=Y5tPfUm9Ghg&t=1100s this one on youtube but I can't seem to recreate his steps as he sifts through his code.

I'll look into it. Thank you everyone.

My use case is basically if I want to now access this dataset from somewhere else, shouldn't I be able to do so using its id?

This works, thanks. Do you have any link to where I can also see the parameters of the Dataset class or was it just on git?

Yeah

dataset = Dataset.create(data_name, project_name)
            print('Dataset Created, Adding Files...')
            dataset.add_files(data_dir)
            print('Files added succesfully, Uploading Files...')
            dataset.upload(output_url=upload_dir, show_progress

I followed the instructions here at https://allegro.ai/clearml/docs/docs/deploying_clearml/clearml_server_linux_mac.html

I'll take a look at it, thank you.

For anyone reading this. apparently there aren't any credentials for my own custom server for now. I just ran it without credentials and it seems to work.

{"meta":{"id":"c3edee177ae348e5a92b65604b1c7f58","trx":"c3edee177ae348e5a92b65604b1c7f58","endpoint":{"name":"","requested_version":1.0,"actual_version":null},"result_code":400,"result_subcode":0,"result_msg":"Invalid request path /","error_stack":null,"error_data":{}},"data":{}}

I get the following error.

this is the console output

Thank you for the help.

I just copied the commands in order from the page and pasted them. All of the linux ones specifically.

I don't think I changed anything.

i think it downloads from the curl command

the one mentioned on the page.

If it helps, I can try and record my steps in a video.

Ok so update. It works now. last steps I did I can remember to fix it are.

set the host variable to the ip assigned to my laptop by the network.