Let me tell you what I think is happening and you can correct me where I'm going wrong.

Under certain conditions at certain times, a Dataset is published, that activates a Dataset trigger. So if every day I publish one dataset, I activate a Dataset Trigger that day once it's published.

N publishes = N Triggers = N Anonymous Tasks, right?

SuccessfulKoala55 Sorry to ping you like this. I have to ask. What's the minimum requirements for clear ml installation. Excluding requirements for databases or file server

I then did what MartinB suggested and got the id of the task from the pipeline DAG, and then it worked.

Doesn't matter how many times I run this code, it'll always give this same output. The tag gets appended to the list but isn't saved. Unless there's something else I'm supposed to do as well.

So minimum would be 2 cores with 8 gigs for ram. I'm going to assume 4 cores and 16 gigs would be recommended.

Quick follow up question. Once I parse args, should they be directly available for i even enque the project for the first time or will i be able to access hyperparameters after running it once?

Like there are files in a specific folder on Machine A. A script on Machine A, creates a Dataset, adds files located in that folder, and publishes it. Now can you look at that dataset on the server machine? Not from the ClearML interface but inside normal directories, like in /opt/clearml etc. this directory mentioned is just an example.

I'm getting this error.

clearml_agent: ERROR: Failed cloning repository.

1. Make sure you pushed the requested commit:
2. Check if remote worker has valid credentials

I get what you're saying. I was considering training on just the new data to see how it works. To me it felt like that was the fastest way to deal with data drift. I understand that it may introduce instability however. I was curious how other developers who have successfully managed to set up continuous training deal with it. 100% new data, or a ratio between new and old data. And if it is the latter, what should be the case, which should be the majority, old data or new data?

Also my execution just completed and as of yet, I can only see the hyperparameters as a report. not in a configurable form. I've just started with ClearML and am having these issues.

adding tags this way to a Dataset object works fine. This issue only occured when doing this to a model.

Here they are. I've created and published the dataset. Then when I try to get a local copy, the code works but i'm not sure how to proceed to be able to use that data.

You're suggesting that the false is considered a string and not a bool? Am I understanding it correctly? Also, in that case, wouldn't this problem also occur when I originally create the task using clearml-task?

Or am I not understanding it clearly.

Not sure myself. I have a pipeline step now, that'll return either clearml dataset id or -1. I want to stop the pipeline execution if I get -1 in the output of that step but I'm not sure how to achieve that

I'm using clear-ml agent right now. I just upload the task inside a project. I've used arg parse as well however as of yet, I have not been able find writable hyperparameters in the UI. Is there any tutorial video you can recommend that deals with this or something? I was following https://www.youtube.com/watch?v=Y5tPfUm9Ghg&t=1100s this one on youtube but I can't seem to recreate his steps as he sifts through his code.

I'm on windows rn, and I work with clearml on ubuntu. I think it's 1.1.5rc4

I've been having this issue for a while now :((

So it won't work without clearml-agent? Sorry for the barrage of questions. I'm just very confused right now.

the one mentioned on the page.

Is this how I'm supposed to send the request to stop all running tasks, if task_ids is the list of task ids which are still running?

Ok so update. It works now. last steps I did I can remember to fix it are.

Also, the steps say that I should run the serving process on the default queue but I've run it on a queue I created called a serving queue and have an agent listening for it.

My draft is View Only but the cloned toy task one is in normal Draft mode.

Thank you, this is a big help. I'll give this a go now.

AnxiousSeal95 I'm trying to access the specific value. I checked the type of task.artifacts and it's a ReadOnlyDict. Given that the return value I'm looking for is called merged_dataset_id, how would I go about doing that?

AnxiousSeal95 Basically its a function step return. if I do, artifacts.keys(), there are no keys, even though the step prior to it does return the output

I checked the value is being returned, but I'm having issues accessing merged_dataset_id in the preexecute_callback like the way you showed me.

I'm both printing it and writing it to a file

It's an artifact.

Thank you