Especially coming from the standpoint of a team leader or other kind of supervision (or anyone who wants to view the experiment which is not the code author), when looking at an experiment you want to see the actual code

Thanks 🙂

I mean the code in whatever form it is - I'm working with git specifically, but if i have diffs I'd like to see the code with the diffs applied

eventually i think it should display the contents of the script executed in the most straightforward manner regardless of version control

No absolutely not. Yes I do have a GOOGLE_APPLICATION_CREDENTIALS environment variable set, but nowhere do we save anything to GCS. The only usage is in the code which reads from BigQuery

Let's take a step back. Let's remove the clearml-services from the docker compose for a second, and run it manually (then you can control everything). Once you have it running manually, let's try to replicate the setup back to the docker compose, make sense ?

I'd prefer not to docker-compose down as researchers are actively working on it, what do you say that I will manually kill the services agent and launch one myself?

AgitatedDove14
So I couldn't kill the service agent myself (permission denied, I'm not sudo). What I did is I docker-compose down ed, commented out only the environment variable of GOOGLE_APPLICATION_CREDENTIALS from the clearml services agent service and upped the docker-compose again. I enqueued the Cleanup Service and now it works. Really weird, looks like the setting of GOOGLE_APPLICATION_CREDENTIALS causes an error when set even though I'm 100% is it not used for storag...

google store package could be the cause, because indeed we have the env var set, but we don't use the google storage package

AgitatedDove14 clearml version on the Cleanup Service is 0.17.0

actually i was thinking about model that werent trained uaing clearml, like pretrained models etc

to fix it, I excluded this var entirely from the docker-compose

Will try this out and report

I don't think the problem is setting that variable, I think it has something to do with it but not that obvious... Because it did work for me in the past, since then we docker-compose up/downed a few times, changed some other things etc... Can't figure out what made it get to this point

🤔 is the "installed packages" part editable? good to know

Isn't it a bit risky manually changing a package version? what if it won't be compatible with the rest?

I'm saying that because in the task under "INSTALLED PACKAGES" this is what appears

I was here, but I can't find info for the questions I mentioned

AgitatedDove14 ?

Manual model registration?

the path to the JSON file

How can I change the version of the Cleanup Service?

AgitatedDove14 sorry for delayed reply - where do I read the version the Cleanup Service is using?

AgitatedDove14 I still can't get it to work... I couldn't figure out how can I change the clearml version in the runtime of the Cleanup Service as I'm not in control of the agent that executes it

the link to manual model registry doesn't work

2021-10-11 10:07:19 ClearML results page: `
2021-10-11 10:07:20
Traceback (most recent call last):
File "tasks/hpo_n_best_evaluation.py", line 256, in <module>
main(args, task)
File "tasks/hpo_n_best_evaluation.py", line 164, in main
trained_models = get_models_from_task(task=hpo_task)
File "tasks/hpo_n_best_evaluation.py", line 72, in get_models_from_task
with open(pickle_path, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/elior/.clearml/c...

what if i want it to use ssh creds?

This error just keeps coming back... I already made the watermarks like 0.5gb

Or should I change all three of them?

I see it's a plotly plot, even though I report a matplotlib one

Can I force it to output the static image?

Gotcha, didn't think of an external server as Service Containers are part of Github's offering, I'll consider that