And multiple agents can listen to the same queue right?

Also the repository is on bitbucket which is why I set git_host to that.

You mean I should set it to this?

I'll take a look thank you

Also the tutorial mentioned serving-engine-ip as a variable but I have no idea what the ip of the serving engine is.

https://clearml.slack.com/archives/CTK20V944/p1641282791308500?thread_ts=1640767505.214000&cid=CTK20V944

I actually just asked about this in another thread. Here's the link. Asking about the usage of the upload_artifact

I have the server running now and for now it seems that I'm able to get the dataset even in the other file. I'll mess around with it now to get a hang of it and see how it actually works

I was getting a different error when I posted this question. Now i'm just getting this connection error

Then I can use ClearML-Data with it properly.

keeps retrying and failing when I use Dataset.get

recursively.

https://github.com/ultralytics/yolov5

dataset._task.mark_started()

Wait is it possible to do what i'm doing but with just one big Dataset object or something?

Sorry for the late response. Agreed, that can work, although I would prefer a way to access the data by M number of batches added instead of a certain range, since these cases aren't interchangeable. Also a simple thing that can be done is that you can create an empty Dataset in the start, and then make it the parent of every dataset you add.

I'm currently installing nvidia docker on my machine, where the agent resides. I was also getting an error regarding gpu not being available in docker since the agent was running on docker mode. I'll share update in a bit. Trying to re run the whole set up

I understand your problem. I think you normally can specify where you want the data to be stored in a conf file somewhere. people here can better guide you. However in my experience, it kinda uploads the data and stores it in its own format.

Thank you.

Well I'm still researching how it'll work. I'm expecting it to not be very good and will make the model learning very stochastic in nature.

I plan to instead at the training stage, instead of just getting this model, use Dataset.squash, to get previous M datasets merged together.

This should introduce stability in the dataset.

Also this way, our model is trained on a batch of data multiple times but only for a few times before that batch is discarded. We keep the training data fresh for co...

I already have the dataset id as a hyperparameter. I get said dataset. I'm only handling one dataset right now but merging multiple ones is a simple task as well.

Also I'm not very experienced and am unsure what proposed querying is and how and if it works in ClearML here.

so code goes

Even though I ended my schedulers and triggers, the anonymous tasks keep increasing.

Can you please share the endpoint link?

As of yet, I can only select ones that are visible and to select more, i'll have to click on view more, which gets extremely slow.

Yeah, I kept seeing the message but I was sure there were files in the location.

I just realized, I hadn't worked with the Datasets api for a while and I forgot that I'm supposed to call add_files(location) and then upload, not upload(location). My bad.

Thanks for the help.

1.1.5

Retrying (Retry(total=239, connect=239, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb2191dcaf0>: Failed to establish a new connection: [Errno 111] Connection refused')': /auth.login
Retrying (Retry(total=238, connect=238, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb2191e10a0>: Failed to establish a new connection: ...

I think I understand now that I first need to have clearml server up and running.

Another issue I'm having is I ran a task using clearml-task and did it using a repo. It runs fine, when I clone said task however and run it on the same queue again, it throws an error from the code. I can't seem to figure out why its happening.