That's great! I look forward to trying this out.

Is there currently a way to bind the same GPU to multiple queues? I believe the agent complains last time I tried (which was a bit ago).

This doesn't really make a lot of sense. ClearML would be better served for tracking which version of the code you used for a corresponding task and you'd use something like github or gitlab to track code and host your code. You could use ClearML to help you reconstruct the environment and code from a task given it's being tracked by git and hosted somewhere you can access.

It hooks into the calls made by the code. If you never save the model to disk, add it to a tool like MLflow/Tensorboard, or manually add the artifact to ClearML, afaik it won't save the artifact.

Hyperdatasets are the only ones that require a premium. If you're using normal datasets it should be fine.

It sounds like you didn't set up your config. Did you ever initialize clearml?

What version of ClearML server are you using?

Are you self hosting a ClearML server?

I found I was having this issue as well. I don't have an alias defined in the pipeline but in a task and I get the same error. I'm not hosting my own server but using the free web service at the moment.

Provide a bit more detail. What framework are you using?

Will there be a recording posted?

Thanks Martin. I read this method as "getting the data associated with the model training" not "getting metadata for the model". This is what I'm looking for.

Thanks for your reply @<1523701070390366208:profile|CostlyOstrich36> Is there an example where a pipeline is built from existing tasks? I'd like to experiment with it and I don' t see any examples of what you describe with my (clearly lacking) google-fu. What happens if you wrap a function with a task.init() with a pipeline decorator or is that the process you're speaking of?

Interesting approach. I'll give that a try. Thanks for the reply!

Let me give that a try. Thanks for all the help.

Hi again @<1523701435869433856:profile|SmugDolphin23> ,

The approach you suggested seems to be working albeit with one issue. It does correctly identify the different versions of the dataset when new data is added, but I get an error when I try and finalize the dataset:

Code:

        if self.task:
            # get the parent dataset from the project
            parent = self.clearml_dataset = Dataset.get(
                dataset_name="[LTV] Dataset",
                dataset_project=...

It's even attempting to install omegaconf but not from the repo, likely because it's a dependency of hydra-colorlog.

Collecting omegaconf<2.4,>=2.2
  Using cached omegaconf-2.2.3-py3-none-any.whl (79 kB)
  Using cached omegaconf-2.2.2-py3-none-any.whl (79 kB)
  Using cached omegaconf-2.2.1-py3-none-any.whl (78 kB)

Actually this is not how it works, pip will install in any way it sees fit, and it is not consistent between versions (it has to do with dependency resolving)

Oh I see. What a pain. 🤣

You can configure the agent to first install specific packages, and only then others, just add the package names here:

That's an interesting solution. I'll keep that in mind as I work more with ClearML.

Thanks for your help Martin!

@<1523701087100473344:profile|SuccessfulKoala55> You wouldn't happen to know what's going on here. :D

It's verbatim from requirements as I pass that into ClearML.

Yes, it indeed appears to be a regex issue. If I run:

Dataset.list_datasets(
                dataset_project=self.task.get_project_name(),
                partial_name=re.escape('[LTV] Dataset Test'),
                only_completed=True,
            )

It works as expected. I'm not sure how raw you want to leave the partial_name features. I could create a PR to fix this but would you want me to re.escape at the list_datasets() level? Or go deeper and do it at `Task._query_task...

The plot thickens. It seems like there's something odd going on with the interaction between [LTV] and additional text. If I just search [LTV] it works, if I just search Dataset Test it works, but if I put them together it breaks the search. Now that I think about it, there's other oddities that seem to happen in the web interface that might be explained by some bugs around using brackets in names.

Why? That's not how I authenticate. Also, if it was simply an issue with authentication wouldn't there be some error message in the log?

Do you start the clearml agents on the server with the same user that has the credentials saved?

Alright, I fixed the issue with the scheduler eating itself. But now I'm still getting the same bug as two days ago. So the Scheduler process starts fine and doesn't "crash." But I don't get the config object in the web-app again. It seems to work if I run it locally.

To answer your earlier question, I'm using the app.clear.ml portal so

• WebApp: 3.20.1-1525
• Server: 3.20.1-1299
• API: 2.28
• And my Python ClearML version: 1.14

Maybe the sleep between scheduler.mark_completed() and scheduler.delete() is too short? But I don't get why deleting the old scheduler task would break the new scheduler. I'm going to try testing by running the scheduler locally.

Thanks again for the info. I might experiment with it to see first hand what the advantages are.

Yeah, it's because it's just hooking into the save operation and capturing the output, regardless of the parent call.

Thanks so much @<1523701205467926528:profile|AgitatedDove14> !

I'm using pro. Sorry, for the delay, I didn't notice I never sent the response.