@<1523701070390366208:profile|CostlyOstrich36> ClearML: 1.10.1, I'm not self-hosting the server so whatever the current version is. Unless you mean the operating system?

@<1523701435869433856:profile|SmugDolphin23> Good to know.

@<1539780284646428672:profile|PoisedElephant79> Sorry for not getting back with this sooner. Dataset.get() doesn't work like you suggested. In the documentation it's clear:

Get a

specific

Dataset. If multiple datasets are found, the dataset with the highest semantic version is returned. If no semantic version is found, the most recently updated dataset is returned. This functions raises an Exception in case no dataset can be found and the

auto_create=True

...

This is odd, the ordering of the files is different and there appears to be some missing from the preview. But as far as I can tell the files aren't different. What am I missing here?

The original file sizes are the same but the compressed sizes seem to be different.

Thanks for the reply @<1523701070390366208:profile|CostlyOstrich36> !

It says in the documentation that:
Add a folder into the current dataset. calculate file hash, and compare against parent, mark files to be uploaded

It seems to recognize the dataset as another version of the data but doesn't seem to be validating the hashes on a per file basis. Also, if you look at the photo, it seems like some of the data does get recognized as the same as the prior data. It seems like it's the correct...

I have manually verified that the line-by-line content of the csv files is identical using hashlib.sha256(). Why would it be that the file content is the same, they are generated by the same process (literally just rerunning the same code twice) but ClearML treats them differently.

Alright, I deleted everything in the ClearML web-app waited a day tried again, it seems to be showing a configuration object in the configuration section of the scheduler task again. I honestly don't know what changed. Maybe some strange caching on the server side that got cleaned up.

@<1523701205467926528:profile|AgitatedDove14> Question: Does the schedule_function option in the TaskScheduler.add_task() method run at the time the task is scheduled to execute? So if I pass a functi...

Yes, I'm experimenting with this. I actually wrote my own process to do this so I just had to adapt it as a callable to pass to the scheduler. However, I'm running into an issue and I don't think this is a user error this time. When I start the scheduler, it starts running, shows up in the web-app, but then an error message in the web-app pops up Fetch parents failed and the Scheduler task disappears from the web-app. I can't even see an error log because the task is gone.

I'm running th...

Stability is for wimps. I live on the edge of brining down production at any moment, like a real developer. But thanks for the update! 🙃

Hi @<1523701205467926528:profile|AgitatedDove14> . I think I'm misunderstanding something here. I have the scheduler service running. Now that it's running how does one add a new task or remove an existing task from the scheduler? I get that I can add them before starting the scheduler service but once the service is running is there any way to connect to it and change the schedule?

I thought the advantage of this service would be we could schedule tasks just by connecting to the existing t...

Well, if I stop the cron service and start it back up I don't have to re-register each schedule. If, for instance, I start the TaskScheduler, register a task, and stop the scheduler, how do I restart the TaskScheduler in a way that re-register the tasks? Because, in theory, they could be registered from several users and I might be unaware of tasks that were previously scheduled. What is the best practices to preserve state?

I made a video of the Scheduler config error. You can see that the same code run locally works and doesn't on remote. (I just uploaded the video so the quality might suffer until YT finishes processing the higher resolution versions).

None

Strange, the code seems to work perfectly when I run it locally. To make it more confusing, the queue that I enqueue it to when I run it remotely is using agents from the same server that I'm running it locally from.

That's what I was getting at. It wasn't clear to me from the documentation that it saves the state.

Hi Jake 👍 ,

Maybe the content is cached? The repo isn't big. I didn't realize the log was missing content. I believe I copied everything but I'll double check in a moment.

Actually, clearing the cache on the other project might have fixed it. I just tested it out and it seems to be working.

Sure. I'm in Europe but we can also test things async.

The answer is simple but also not completely obvious to someone new to the platform. So you can inject new command line args that hydra will recognize. This is what the Hydra section of args is for. However, if you enable _allow_omegaconf_edit_: True , I think ClearML will “inject” the OmegaConf saved under the configuration object of the prior run, overwriting the overrides. I’ll experiment with this behavior a bit more to be sure.

I think this error occurred for me because when I first authenticated with the project I was using username/password and later I transitioned to using ssh keys. That's why clearing the cache worked.

Did you validate that branch exists on remote?

Ah, perfect.

Awesome! Did you managed to solve the tailscale issue with ClearML sessions? Sorry I wasn't active with that. I don't use sessions often and I found a suitable alternative in the short time. Any hopes of the changes making their way to a PR for the official release?

That's wonderful! I'll have to give it a try. ;)

I figured you'd say that so I went ahead with that PR. I got it working but I'm going to test it a bit further.

I had 2 datasets on archive and 0 unarchived. When I ran the following command:

Dataset.list_datasets(dataset_project=self.task.get_project_name(), only_completed=True)

It returned two entrees for the two datasets I had on archive.

Ah, that makes sense. What is supposed to be hidden changes depending on the section your in, which makes sense. Now there needs to a packman sprite easter egg hidden somewhere else.

Alright, I tried testing it out by commenting out the code for generating new csv's, so for successive runs the CSVs are identical. However, when I use dataset.add_files() it still generated a new version of the dataset.

# log the data to ClearML if a task is passed
        if self.task:
            self.clearml_dataset = Dataset.create(dataset_name="[LTV] Dataset")
            self.clearml_dataset.add_files(path=save_path, verbose=True)
            if self.tags is not None:
          ...

I see, that could work.

I'm not sure why the logs were incomplete. I think part of the reason it wasn't pulling from the repo was that it was pulling from cache. I cleared the clearml cache for that project and reran it. This should be the full log.

I will add a gh issue. Is this part open source? Could I make a PR?

In the mean time I still need to implement this with the current version of ClearML. So the only way would be to have one variable per parent? Is there any smarter way to work around it?

Is it possible the cached repository was cloned before you changed your agent settings?

Which settings are you referring to? I can't remember if I was using https auth when the project would have been first cached. Would that make a difference?

Also, did you set

agent.enable_git_ask_pass: true

?

The only instance of it in the config is commented out.

    # if set, use GIT_ASKPASS to pass user/pass when cloning / fetch repositories
    # it solves pas...