can you provide some mode details please ? Do you intend to store your artefacts locally or remotely ?
Does the manual reporting also fails ?

If you could also give your clearml packages versions it could help

I store the artifacts on a minio server (in my LAN).
If I run the python script locally (i.e. no execute remotely() it works fine).
I use the latest clearml 1.6.2

Did you by any chance save the checkpoint without any file extention? Or with a weird name containing sl...

can you test what happens if you pass the credentials in the global scope as well, i.e. here:

That didn’t help

AgitatedDove14 ,
From the experiment’s console log:
` - boto3==1.16.2

• botocore==1.19.2 `

Thanks ExasperatedCrab78
AgitatedDove14 - attached

Oh wow AgitatedDove14 . Appreciate it!
Are you sure it’s just a matter of the python version?
The same experiment script, was working on the exact docker image in the past (with older clearml versions though…).
For example this experiment log:

AgitatedDove14 its running inside a docker based worker.
Are you interested in the full pip freeze of that docker?

Thanks AgitatedDove14 !
I’ll use clearml 1.4.1 until the fix is out.

Im afraid there's something with my account, since it won't login on other browser as well, but logging with a different email works

After signing with google, the login page is stuck at this

SweetBadger76
It’s not a credential issue, because I do upload artifacts manually with tsk.upload_artifact(...)
I’ll try changing the extension, but I have to admit that in the past (I havn’t used clearml for a while and updated it recently to the latest verion) it did got this file extension right

what does that actually mean?
2022-07-17 07:59:40,339 - clearml.storage - ERROR - Failed uploading: Parameter validation failed: Invalid type for parameter ContentType, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

Not sure I understand the purpose of this.
it meant pip will look for wheels at thus url?

CostlyOstrich36
I do get errors - failing to launch the clearml images.
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested standard_init_linux.go:228: exec user process caused: exec format errorWell, after diving into this, it seems like the clearml images were built usin amd64 (on top of amd64 base images...)

Thanks

Yes.

Its kwcoco

This is my conda env export :
name: clearml
channels:

• defaults
  dependencies:
• ca-certificates=2021.10.26=hecd8cb5_2
• certifi=2021.10.8=py39hecd8cb5_0
• libcxx=12.0.0=h2f01273_0
• libffi=3.3=hb1e8313_2
• ncurses=6.3=hca72f7f_2
• openssl=1.1.1l=h9ed2024_0
• pip=21.2.4=py39hecd8cb5_0
• python=3.9.7=h88f2d9e_1
• readline=8.1=h9ed2024_0
• setuptools=58.0.4=py39hecd8cb5_0
• sqlite=3.36.0=hce871da_0
• tk=8.6.11=h7bc2e8c_0
• tzdata=2021e=hda174b...

AgitatedDove14 Yes, thats correct.
It's in my local conda environment though.

AgitatedDove14 , here's the log

Thank for the great explanation! Now it makes much more sense.
You are right about the issue that 'kwcoco' isn't being detected, and Im actually running this as a single script, and the kwcoco not imported directly (but from within another package).
Ill try running it from a repo and see how it works.

Maybe I should understand better how this actually works.
It's not a rare use case to use conda environments, but install some packages using pip (when they're not available at any conda channel). Will that use case be supported on clearml?

No. But this isn't the only package that was installd using pip inside a conda env.
Compare it with Pandas as an example - was installed using pip nad it's actually instaled inside the worker as well (using pip).

AgitatedDove14
It was installed by 'pip install kwcoco' while my conda env was active.
Not sure if it answers your question..

So I run the same script as part of a git repo - but unfortunately the package is still missing.
I'm not sure if it matters but 'kwcoco' is being imported inside one of the repo's functions and not on the script's header.

AgitatedDove14 any ideas?

AgitatedDove14
I'm not sure.
In my case I'm not trying to reproduce a local environment in the agent, but to run a script inside a docker which already has the environment built in.
The environment is conda based.

AgitatedDove14
Yes, I'd like to point to a specific binary, which is in a conda environment.
(b.t.w how can I specify the python version on the Task?)

It didn't work either. Still same error.
conda command was already in PATH and the conda activate is executed properly, but it prompts to run conda init (i.e conda wasn't configures at that shell):
` You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:

• ba...

AgitatedDove14
It's still failing.
I updated clearml-agent to 1.2.0rc7 and also:
docker_setup_bash_script= [ "export PATH=""/workspace/miniconda/bin:$PATH", "export LOCAL_PYTHON=/workspace/miniconda/bin/python3", "conda activate"])But the conda activate (base env) returns:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.

I noticed that conda ...

Anyway, when I add the binary's path to PATH, it still won't work.
I call
task.set_base_docker(docker_image='my/docker/image', docker_setup_bash_script= ["export PATH=""/workspace/miniconda/bin:$PATH"])just after Task.init.
On execution, the agent installs all pip requirements with python binary at /usr/bin/python3.6
but eventually the task fails with, /workspace/miniconda/bin/python3: No module named clearml_agent

AgitatedDove14 .
Note that the actual error is /workspace/miniconda/bin/python3: No module named clearml_agent since all the packages (including clearml_agent) were already installed by the agent on the default (non conda) python binary.

That would be a very useful feature.
What is the status of that issue? I havn't found it on github.