https://github.com/allegroai/clearml-agent/pull/98 AgitatedDove14 😁

Hm, just a small update - I just verified and it does indeed work on linux:

` import clearml
import dotenv

if name == "main":
dotenv.load_dotenv()
config = clearml.backend_api.Config.load() # Success, parsed with environment variables `

Yeah 🤔 🤔 they did. I'll give your suggested fix a go on Monday!

Another side effect btw is that some of our log files (we add a file handler to the logger) end up at 0 bytes. This specifically happens with Ray and ClearML and does not reproduce locally

Yeah, and just thinking out loud what I like about the numpy/pandas documentation

Example configuration -
` version: 1
disable_existing_loggers: true
formatters:
simple:
format: '%(asctime)s %(levelname)-9s %(name)-24s: %(message)s'
filters:
brackets:
(): ccutils.logger.BracketFilter
handlers:
console:
class: ccmlp.utils.TqdmStreamHandler
level: INFO
formatter: simple
filters: [brackets]
loggers: # Set logging levels for specific packages
urllib3:
level: WARNING
matplotlib:
level: WARNING
...

1.1.4

If that's the case, wouldn't it apply across the board? This happens in a single task within ray - the other tasks (I have many in a single run) are fine

That gives us the benefit of creating "local datasets" (confined to the scope of the project, do not appear in Datasets tabs, but appear as normal tasks within the project)

Why not give ClearML read-only access credentials to the repository?

Or do you mean the contents of the configuration, probably :face_palm: ... one moment

AFAICS it's quite trivial implementation at the moment, and would otherwise require parsing the text file to find some references, right?
https://github.com/allegroai/clearml/blob/18c7dc70cefdd4ad739be3799bb3d284883f28b2/clearml/task.py#L1592

Alternatively, it would be good to specify both some requirements and auto-detect 🤔

BTW AgitatedDove14 following this discussion I ended up doing the regex way myself to sync these, so our code has something like the following. We abuse the object description here to store the desired file path.

` config_path = task.connect_configuration(configuration=config_path, name=config_fname)
included_files = find_included_files_in_source(config_path)
while included_files:
file_to_include = included_files.pop()
sub_config = task.connect_configuration(
configurat...

I'll try upgrading to 1.1.5, one moment

Yeah that works fine 😄 I just fetch it once to map argparse users to their IDs for later filtering.

The Slack Monitoring example should be updated btw, as they now use slack_sdk instead of slack (in the import statements)

That's fine as well - the code simply shows the name of the environment variable, not it's value, since that's taken directly from the agent listening to the services queue (and who's then running the scaler)

I think this is about maybe the credential.helper used

I can indeed.

Maybe it's the missing .bashrc file actually. I'll look into it.

Yes; I tried running it both outside venv and inside a venv. No idea why it uses 2.7?

Holy crap this was a light-bulb moment, is this listed somewhere in the docs?
It solves so much of my issues xD

I think you're interested in the Monitor class:)

It's self-hosted TimelyPenguin76
SuccessfulKoala55 WebApp: 1.4.0-175 • Server: 1.4.0-175 • API: 2.18

That could work, given that:
Could we add a preview section? One reason I don't like using the configuration section is that it makes debugging much much harder. Will the clearml-agent download and unzip the files, placing them into the same local folder as needed for execution? What if we want to include non-configuration objects? (i.e. the model case I listed)

AFAIK that's the only way right now (see my comment here - https://clearml.slack.com/archives/CTK20V944/p1657720159903739?thread_ts=1657699287.630779&cid=CTK20V944 )
Or then if you have the ClearML paid service, I believe there is a "vaults" service, right AgitatedDove14 ?

https://clear.ml/docs/latest/docs/references/sdk/services_monitor

Then you can run this as a task, see also this example https://clear.ml/docs/latest/docs/guides/services/slack_alerts

I'm not too worried about the dataset appearing (or not) in the Datasets tab. I would like it (the original task ) to to not disappear from the original project I assigned it to

I'll try with 1.1.5 first, then 1.1.6rc0

Thanks LazyFish41 , that explains it 👍