I decided to use root container, now it is working.

thanks @<1715900788393381888:profile|BitingSpider17> for attaching the log it really helps/
Notice from the log:

'-v', '/home/clearml/.clearml/cache:/clearml_agent_cache'

and as expected we also get:

sdk.storage.cache.default_base_dir = /clearml_agent_cache

Yet I can see the error you pointed:

FileNotFoundError: [Errno 2] No such file or directory: '/clearml_agent_cache/storage_manager/datasets'

Now, could it be that the same folder is used for both root and Non-root containers?
What I suspect happened is a "root" container, created the sub-directory /clearml_agent_cache/storage_manager and now the non-root container is trying to access it and fails.
You can quickly verify that by running ls on the host machine cache folder ls -la /home/clearml/.clearml/cache
Of course it can be quickly fixed with sudo chmod 777 -R /home/clearml/.clearml/cache
My question here is are you mixing two types of non-root & root containers ?

@<1523701205467926528:profile|AgitatedDove14> . I rebuilt my image, so that user 'username' owns directory /clearml_agent_cache and tried to run with it, yet I still cannot run the experiment. Attaching the full log, 'I am username' text is python debug message to makes sure that the user is correct

Wait @<1715900788393381888:profile|BitingSpider17> are you passing it on a single Task? these values are read by the daemon (i.e. running on the host) which means it is not getting them from the Task context (which leads to zero effect on the mount points)
Notice that in new versions of the clearml-agent the SDK mount point was changed to: sdk_cache: "/clearml_agent_cache" exactly to solve for the non-root containers:
None
what's the version you are using?

It is formatting of slack, that removed double underscore, this is unformatted environment
-e CLEARML_CACHE_DIR=/home/username/clearml_agent_cache -e CLEARML_AGENT__AGENT__DOCKER_INTERNAL_MOUNTS__SDK_CACHE=/home/username/clearml_agent_cache -e CLEARML_AGENT__AGENT__VCS_CACHE__PATH=/home/username/.clearml/vsc-cache -e CLEARML_AGENT__AGENT__DOCKER_INTERNAL_MOUNTS__VCS_CACHE=/home/username/.clearml/vcs-cache -e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 -e CLEARML_AGENT__AGENT__DOCKER_INIT_BASH_SCRIPT="" -e LOCAL_PYTHON="/home/username/cv_workspace/libs/pipenv/.venvs/python_deps-KN9jOxaY/bin/python"

Hi @<1715900788393381888:profile|BitingSpider17>
Notice that you need __ (double underscore) for converting "." in the clearml.conf file,
this means agent.docker_internal_mounts.sdk_cache will be CLEARML_AGENT__AGENT__DOCKER_INTERNAL_MOUNTS__SDK_CACHE
None