The logs are on the bucket, yes.
The default file server is also set to s3://ip:9000/clearml

Is it currently broken? 🤔

Maybe. When the container spins, are there any identifiers regarding the task etc available? I create a folder on the bucket per python train.py so that the environment variables files doesn't get overwritten if two users execute almost-simultaneously

I'm not sure I follow, how would that solution look like?

I'm trying to decide if ClearML is a good use case for my team 🙂
Right now we're not looking for a complete overhaul into new tools, just some enhancements (specifically, model repository, data versioning).

We've been burnt by DVC and the likes before, so I'm trying to minimize the pain for my team before we set out to explore ClearML.

If everything is managed with a git repo, does this also mean PRs will have a messy metadata file attached to them?

I should maybe mention that the security regarding this is low, since this is all behind a private VPN server anyway, I'm mostly just interested in having the credentials used for backtracking purposes

Thanks AgitatedDove14 , I'll first have to prove viability with the free version :)

I mean, I know I could connect_configuration({k: os.environ.get(k) for k in [...]}) , but then those environment variables would be exposed in the ClearML UI, which is not ideal (the environment variables in question hold usernames and passwords, required for DB access)

Maybe this is part of the paid version, but would be cool if each user (in the web UI) could define their own secrets, and a task could then be assigned to some user and use those secrets during boot?

The S3 bucket credentials are defined on the agent, as the bucket is also running locally on the same machine - but I would love for the code to download and apply the file automatically!

Bump SuccessfulKoala55 ?

I think ClearML boots up only afterwards, so those environment variables may not be available yet.
You should set them manually in the bootstrap code unfortuantely.

I guess I'll have to rerun the experiment without tags for this?

Opened this - https://github.com/allegroai/clearml/issues/594

Sure! It looks like this

Also something we are very much interested in (including the logger-based scatter plots etc)

Each user creates a .env file for their needs or exports them in the shell running the python code. Currently I copy the environment variables to an S3 bucket and download it from there.

Also I appreciate the time youre taking to answer AgitatedDove14 and CostlyOstrich36 , I know Fridays are not working days in Israel, so thank you 🙂

Would be good if that's mentioned explicitly in the docs 😄 Thanks!

That could be a solution for the regex search; my comment on the pop-up (in the previous reply) was a bit more generic - just that it should potentially include some information on what failed while fetching experiments 😄

Also, creating from functions allows dynamic pipeline creation without requiring the tasks to pre-exist in ClearML, which is IMO the strongest point to make about it

Let me verify a hypothesis...

Yes; I tried running it both outside venv and inside a venv. No idea why it uses 2.7?

I know, that should indeed be the default behaviour, but at least from my tests the use of --python ... was consistent, whereas for some reason this old virtualenv decided to use python2.7 otherwise 🤨

Still failing with 1.2.0rc3 😞 AgitatedDove14 any thoughts on your end?

I also tried switching to dockerized mode now, getting the same issue 🤔

I'm using 1.1.6 (upgraded from 1.1.6rc0) - should I try 1.1.7rc0 or smth?

I'll have yet another look at both the latest agent RC and at the docker-compose, thanks!

There was no "default" services agent btw, just the queue, I had to launch an agent myself (not sure if it's relevant)

AgitatedDove14
I'll make a PR for it now, but the long story is that you have the full log, but the virtualenv version is not logged anywhere (the usual output from virtualenv just says which Python version is used, etc).