Can you elaborate on the hidden project? Is this part of a task you created and something is not showing up?

That's pretty weird. I don't see any clear indications something is wrong, it simply doesn't execute the rest it would seem. Did it successfully run the first time before cloning it?

For pipelines there's currently no way to use different compressions. You can still use it when explicitly uploading https://clear.ml/docs/latest/docs/references/sdk/dataset/#upload

We're sorry about that, this seem like a bug indeed. Could you open a github issue?

Did you use --git-credentials ?
https://clear.ml/docs/latest/docs/apps/clearml_session#accessing-a-git-repository

You can get all tasks: https://clear.ml/docs/latest/docs/references/sdk/task#taskget_all

You can search tasks: https://clear.ml/docs/latest/docs/clearml_sdk/task_sdk#querying--searching-tasks

And you can get the status:
https://clear.ml/docs/latest/docs/references/sdk/task#get_status

Can you give me a bit more info what exactly you're trying to log and what framework you're using?

You can use the same project name and the same Taskname if that helps?

I don't think it's possible to set a custom ID if it doesn't exist yet. But I'll double check!

And in the logs?

Just checking, are you just trying to use a different docker image in a task? Because then you might want to use this: https://clear.ml/docs/latest/docs/apps/clearml_task/#docker
https://clear.ml/docs/latest/docs/clearml_agent#docker-mode

You can also use https://clear.ml/docs/latest/docs/references/sdk/task/#taskget_task since task.clone also accepts a task object

ThoughtfulBadger56 Have you uncommented the existing venvs_cache section in the config file?
https://clear.ml/docs/latest/docs/clearml_agent#virtual-environment-reuse

PIP can install from git repositories!
So you can point to your own repository or even a specific commit hash.

https://pip.pypa.io/en/stable/topics/vcs-support/

If you're looking for what docker volumes were used, that's in the docker compose file:
https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml

ExasperatedCrab78 do you know how this could be?

Can you try again with "-m 16g" ?

you can pass use the compression parameter in dataset.upload . The supported values are:
ZipFile.ZIP_STORED (no compression) ZipFile.ZIP_DEFLATED (requires zlib) ZipFile.ZIP_BZIP2 (requires bz2) ZipFile.ZIP_LZMA (requires lzma)Note that you need to import ZipFile beforehand: from zipfile import ZipFile
You're probably looking for ZIP_BZIP2 , but I'm not sure about that.

https://clear.ml/docs/latest/docs/webapp/webapp_exp_track_visual/#installed-packages

You can set where to store it via this config file: https://clear.ml/docs/latest/docs/fundamentals/artifacts/#setting-upload-destination

Or you can just load a config file or object: https://clear.ml/docs/latest/docs/references/sdk/task/#connect_configuration

If you run it, what does it say in experiment list -> experiments -> execution -> installed packages?

AnxiousSeal95 Do you know this one?

It looks like there is this option under Settings->Configuration->Show Hidden Projects

So for notebooks requirements are indeed not checked elsewhere.
You can however include them with using this line before Task.init

Task.force_requirements_env_freeze(requirements_file=requirements.txt)

I'm afraid what you're trying to do isn't a supported implementation.
You'll have to choose between using docker mode to have one virtual environment for everything or using the pip mode where you can used the cached virtual environments but you can't reuse the one you currently have.

Could you test the following:
Without reusing the virtual environment you made manually:
Can you run a task twice and see if the second run is at least reusing the virtual environment of the first run?

Does this help? https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server#what-is-clearml-server

I'd you've got a self hosted instance you can have a look yourself https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config
But other then that I'm not sure. AnxiousSeal95 any thoughts?

no this should work with this one. I'll double check if I'm remembering it correctly but I thought you should be able to start a task after loading your own configuration object, where can set the agent.package_manager.system_site_packages = true .

Could you try to see if it does work when you log those manually?
https://clear.ml/docs/latest/docs/clearml_sdk/model_sdk#manually-logging-models