I think the environment variables path might work for you then?
You'd set your config with
use_credentials_chain: ${CREDENTIALS_CHAIN} Then in Python you could os.environ['CREDENTIALS_CHAIN'] = True/False before you make any calls to ClearML?

StaleButterfly40 what use case are you looking for? I've used environment variables in the config file and then I can overwrite them in os.environ before ClearML loads the config

I thought this follows from our previous discussion SuccessfulKoala55 , where this is a built-in feature of pyhocon?

There's not much (or anything) in the log to provide...

` (.venv) 15:42 [0:user@server$~] CLEARML_CONFIG_FILE=~/agent_clearml.conf clearml-agent daemon --queue default on_prem --detached --order-fairness
Environment variables set from configuration: ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY', 'AWS_DEFAULT_REGION']
...

Sorry for the late reply Jake -- I was away on holidays -- it works perfectly now, thanks!

The agent also uses a different clearml.conf , so it should not matter?

Most of these are configurations (specific for an execution, but one such configuration defines multiple tasks). Some models might be uploaded if the user does not use our built-in link to ClearML model fetching 😄

The documentation is messy, I’ve complained about it the in the past too 🙈

I didn't mention code in #340 nor did I mention data here 😄 The idea was to package non git-specific files for remote execution

Honestly, this is all related to issue #340. The only reason we have this to begin with is because we need one separate "initializer" task that downloads the remote cache and prepares the agent environment for execution (downloading the configuration files, etc).
Otherwise it fits perfectly with pipelines, but we're not there yet.

In the local execution we don't have this initializer task, so we use Task.init() before starting to work on a model, and task.close() when we're done....

It does not 🙂
We started discussing it here - https://clearml.slack.com/archives/CTK20V944/p1640955599257500?thread_ts=1640867211.238900&cid=CTK20V944
You suggested this solution - https://clearml.slack.com/archives/CTK20V944/p1640973263261400?thread_ts=1640867211.238900&cid=CTK20V944
And I eventually found this solution to work - https://clearml.slack.com/archives/CTK20V944/p1641034236266500?thread_ts=1640867211.238900&cid=CTK20V944

Thanks CostlyOstrich36 !

Hm, I did not specify any specific versions previously. What was the previous default?

And yes, our flow would break anyway with the internal references within the yaml file. It would be much simpler if we could specify the additional files

The new task is not running inside a new subprocess. Our platform trains several models, and we'd like each of them to be tracked in their own Task . When running locally, this is "out of the box", as we can init and close before and after each model.
When running remotely, one cannot close the main task (since it is what orchestrates everything), and so this workaround was needed.

I am; it seems like maybe a couple of hours?

But since this has come up a lot recently, any updates on #340? 😍

@<1523701087100473344:profile|SuccessfulKoala55> could you provide some instructions?

Thanks!

I'll give it a shot. Honestly, the SDK documentation for both InputModel and OutputModel is (sorry) horrible ...

Can't wait for the documentation revamping.

Just a side note - the 1.1.1 notice keeps popping up even though the server is at 1.1.1 (and I've cleared browser cache etc)

Opened a matching feature request issue for this -> https://github.com/allegroai/clearml/issues/418

With clearml-agent 1.5.1

I'll kill the agent and try again but with the detached mode 🤔

Erm, one sec, let me find that thread...

It seems that the agent uses the remote repository 's lock file. We've removed and renamed the file locally (caught under local changes), but it still installs from the remote lock file 🤔

For now we've monkey-patched it to our usecase:

` Dataset._Dataset__hidden_tag = "active"

    def foo(cls, dataset_project, dataset_name):
        dataset_project = dataset_project or "Datasets"
        return dataset_project, dataset_project.rpartition("/")[0]

    Dataset._build_hidden_project_name = foo `

The poetry.lock in github, it seems 🤔

Here's an example where poetry.lock is removed, and still the console reads:
url: .... branch: HEAD commit: 22fffaf8d5f377b7f10140e642a7f6f26b72ffaa root: /.../.clearml/venvs-builds/3.10/task_repository/... Applying uncommitted changes Poetry Enabled: Ignoring requested python packages, using repository poetry lock file! Creating virtualenv ds-platform in /.../.clearml/venvs-builds/3.10/task_repository/.../.venv Updating dependencies Resolving dependencies...

Still happening with 1.4.1 SuccessfulKoala55