Yeah that works too. So one can override the queue ID but not the worker 🤔

My suspicion is that this relates to https://clearml.slack.com/archives/CTK20V944/p1643277475287779 , where the config file is loaded prematurely (upon import ), so our dotenv.load_dotenv() call has not yet registered.

Basically when running remotely, the first argument to any configuration (whether object or string, or whatever) is ignored, right?

Or if it wasn't clear, that chunk of code is from clearml's dataset.py

Well the individual tasks do not seem to have the expected environment.

We're using 1.1.5 at the moment -- I'll make sure everyone updates to 1.1.6 on Monday.

That solution does not work for us unfortunately -- the .env is an argument from argparse, and because we cannot attach non-git files to a remote task (again issue #395), we have to first download CLI arguments for remote execution and ensure they exist on the remote agent.

When I use the APIClient to fetch the tags for the project, I get an empty collection of system tags:

<projects.GetProjectTagsResponse: {
    "tags": [],
    "system_tags": []
}>

Any sneak preview? 😉 😁

One must then ask, of course, what to do if e.g. a text refers to a dictionary configuration object? 🤔

From the one you sent - None

Right, so where can one find documentation about it?
The repo just has the variables with not much explanations.

I commented on your suggestion to this on GH. Uploading the artifacts would happen via some SDK before switching to remote execution.
When cloning a task (via WebUI or SDK), a user should have an option to also clone these input artifacts or simply linking to the original. If linking to the original, then if the original task is deleted - it is the user's mistake.

Alternatively, this potentially suggests "Input Datasets" (as we're imitating now), such that they are not tied to the original t...

No I don't this at all

Yeah 🤔 🤔 they did. I'll give your suggested fix a go on Monday!

Full log:
` command: /usr/sbin/helm --version=4.1.2 upgrade -i --reset-values --wait -f=/tmp/tmp77d9ecye.yml clearml clearml/clearml
msg: |-
Failure when executing Helm command. Exited 1.
stdout:
stderr: W0728 09:23:47.076465 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0728 09:23:47.126364 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unava...

Not really, I've only been able to somewhat understand the scope of where it happens, and I'm not sure it's even a ClearML issue (maybe matplotlib)

AgitatedDove14 another option I thought would be nice is to actually self-sign the internal MinIO bucket, but then I get
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1076)Are you aware of any other way then (other than the secure: false flag?

Seems like Task.create is the correct use-case then, since again this is about testing flows using e.g. pytest, so the task is not the current process.

I've at least seen references in dataset.py 's code that seem to apply to offline mode (e.g. in Dataset.create there is if output_uri and not Task._offline_mode: , so someone did consider datasets in offline mode)

I guess I'll have to rerun the experiment without tags for this?

Yeah that works fine 😄 I just fetch it once to map argparse users to their IDs for later filtering.

The Slack Monitoring example should be updated btw, as they now use slack_sdk instead of slack (in the import statements)

No need for S3 or other resources?

We just inherit from logging.Handler and use that in our logging.config.dictConfig ; weird thing is that it still logs most of the tasks, just not the last one?

Interesting, why won’t it be possible? Quite easy to get the source code using e.g. dill .

Can I subscribe to updates? 😁

Ah. Apparently getting a task ID while it’s running can cause this behaviour 🤔

Running a self-hosted server indeed. It's part of a code that simply adds or uploads an artifact 🤔

Is there a way to accomplish this right now FrothyDog40 ? 🤔

That sounds about right FrothyDog40

Yes, exactly. I have not yet had a chance to try this out -- should it work?

It is. Let me see what else I have set up for MinIO in configs, one moment