It could be related to ClearML agent or server then. We temporarily upload a given .env file to internal S3 bucket (cache), then switch to remote execution. When the remote execution starts, it first looks for this .env file, downloads it using StorageManager, uses dotenv, and then continues the execution normally

We do not CostlyFox64 , but this is useful for the future 🙂 Thanks!
TimelyPenguin76 I'll have a look, one moment.

JitteryCoyote63 I use clearml.config.get_remote_task_id()

Not that I recall

That would be nice :)

I'm guessing that's not on pypi yet?

Thanks! That's what I thought, but then I get
2021-12-21 22:08:35,376 - clearml.storage - ERROR - Failed uploading: Parameter validation failed: Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

Alternatively, it would be good to specify both some requirements and auto-detect 🤔

Well the individual tasks do not seem to have the expected environment.

... and any way to define the VPC is missing too 🤔

https://clearml.slack.com/apps/A019MGGA2Q1-around ?

I have no idea what’s the difference, but it does not log the internal repository 😞 If I knew why, I would be able to solve it myself… hehe

Well you could start by setting the output_uri to True in Task.init .

JitteryCoyote63 yes exactly, sorry, I forgot to add the Task.get_task in my response. That's exactly what we do 😅

Happens with the latest version indeed.
I can’t share our code, but the gist of it is:

pipe = PipelineController(name=..., project=..., version=...)

pipe.add_function_step(...)  # Many calls

pipe.set_default_execution_queue(...)
pipe.start(queue=..., wait=True)

It seems that the agent uses the remote repository 's lock file. We've removed and renamed the file locally (caught under local changes), but it still installs from the remote lock file 🤔

Or do you mean the contents of the configuration, probably :face_palm: ... one moment

If I set the following:
"extra_clearml_conf": "sdk.aws.s3.credentials = [\n{\nhost: 'ip:9000'\nkey: 'xxx'\nsecret: 'xxx'\nmultipart: false\nsecure: false\n},\n{\nhost: 'ip2:9000'\nkey: 'xxx'\nsecret: 'xxx'\nmultipart: false\nsecure: false\n}\n]"I run into a weird furl error:
ValueError: Invalid port '9000''.

Of course now it's not there anymore 😆 If/when it happens again I'll ping you here 🙂

Let me know if you do; would be nice to have control over that 😁

I have no tried to download the entire file, but yes

I think so, it was just missing from the official documentation 🙂 Thanks!

I cannot, the instance is long gone... But it's not different to any other scaled instances, it seems it just took a while to register in ClearML

Could also be related to K8, so pinging JuicyFox94 just in case 😉

Yup, latest version of ClearML SDK, and we're deployed on AWS using K8s helm

Just a side note - the 1.1.1 notice keeps popping up even though the server is at 1.1.1 (and I've cleared browser cache etc)

I narrowed the bug down to the "fix" in 1.8.1, see my other post

Nice, that works 🙂 Thanks!

Illustrating the first bullet point

I wouldn't mind going the requests route if I could find the API end point from the SDK?