I'll look into it. Thank you everyone.

yeah.

I checked the value is being returned, but I'm having issues accessing merged_dataset_id in the preexecute_callback like the way you showed me.

I've tried the ip of the ClearML Server and the IP of my local machine on which the agent is also running on and none of the two work.

I'm both printing it and writing it to a file

I'm not using decorators. I have a bunch of function_steps followed by a normal task step, where I've passed a base_task_id.

I want to check the value of one of the functional steps, and if it holds true, I want to execute the task step otherwise I want the pipeline to end there, since the task step is the last one.

Good point

this is it.

AnxiousSeal95 I'm trying to access the specific value. I checked the type of task.artifacts and it's a ReadOnlyDict. Given that the return value I'm looking for is called merged_dataset_id, how would I go about doing that?

I normally just upload the data to the ClearML server and then remove it locally from my machine but I understand that isn't what you want. A quick hack was the only thing I could come up with at the moment xd. Anyway you're welcome. Hope you find a solution.

Basically when I'm loading the model in InputModel, it loads it fine but I can't seem to get a local copy.

AnxiousSeal95 Basically its a function step return. if I do, artifacts.keys(), there are no keys, even though the step prior to it does return the output

trigger_scheduler.add_dataset_trigger(schedule_task_id=TASK_ID, schedule_queue='default',
trigger_project='Cassava Leaf Disease Classification', name='start task on data - End Game')

AnxiousSeal95 I just have a question, can you share an example of accessing an artifact of a previous step in the pre execute callback?

My draft is View Only but the cloned toy task one is in normal Draft mode.

There's a whole task bar on the left in the server. I only get this page when i use the ip 0.0.0.0

This is the task scheduler btw which will run a function every 6 hours.

Alright. Can you guide me on how to edit the task configuration object? Is it done via the UI or programatically? Is there a config file and can it work with any config file I create or is it a specific config file? Sorry for the barrage of questions.

I initially wasn't able to get the value this way.

Thank you.

I would normally like for it to install any requirements needed on its own.

Let me share the code with you, and how I think they interact with eachother.

My use case is basically if I want to now access this dataset from somewhere else, shouldn't I be able to do so using its id?

So I got my answer, for the first one. I found where the data is stored in the server

When I try to access the server with the IP I set as CLEARML_HOST_IP, it looks like this. I set that IP to the ip assigned to me by the network

In the case of api call,

given that i have id of the task I want to stop, I would make a post request to [CLEARML_SERVER_URL]:8080/tasks.stop

with the request body set up like the one mentioned in the api?

Basically if I pass an arg with a default value of False, which is a bool, it'll run fine originally, since it just accepted the default value.

I get what you're saying. I was considering training on just the new data to see how it works. To me it felt like that was the fastest way to deal with data drift. I understand that it may introduce instability however. I was curious how other developers who have successfully managed to set up continuous training deal with it. 100% new data, or a ratio between new and old data. And if it is the latter, what should be the case, which should be the majority, old data or new data?

Understandable. I mainly have regular image data, not video sequences so I can do the train test splits like you mentioned normally. What about the epochs though? Is there a recommended number of epochs when you train on that new batch?

I'll create a github issue. Overall I hope you understand.