Reputation
Badges 1
662 × Eureka!So kind of the ability to have more artifact types in "Artifacts" tab, other than Other
and OutputModels
, etc
That's a nice work around of course - I'm sure it works and I'll give it a shot momentarily. I'm just wondering if ClearML could automatically recognize image files in upload_artifact
(and other well known suffixes) and do that for me.
Sure! It's a bit intricate as it accommodates many of our different plotting functionalities, but this consists of the important bits (I realize we have some bad naming here, but fig[0]
is actually a Figure object, and fig[1]
is an Axes object):
` plt.switch_backend('agg')
sns.set_theme(...)
fig = plt.subplots(...)
sns.histplot(data, ax=fig[1], ...)
fig[1].set_xlim(...)
fig[1].set_ylim(...)
fig[1].legend(loc='best')
fig[1].set_xlabel(xlabel)
fig[1].set_ylabel(ylabel)
fig[1].set_...
It seems that the agent uses the remote repository 's lock file. We've removed and renamed the file locally (caught under local changes), but it still installs from the remote lock file 🤔
Okay, I'll test it out by trying to downgrade to 4.0.0 and then upgrade to 4.1.2
Just to make sure, the chart_ref
is allegroai/clearml
right? (for some reason we had clearml/clearml
and it seems like it previously worked?)
The Task.init
is called at a later stage of the process, so I think this relates again to the whole setup process we've been discussing both here and in #340... I promise to try ;)
This happened again 🤔
How many files does ClearML touch? :shocked_face_with_exploding_head:
Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC
Should this be under the clearml
or clearml-agent
repo?
CostlyOstrich36 That looks promising, but I don't see any documentation on the returned schema (i.e. workers.worker_stats
is not specified anywhere?)
I think ClearML boots up only afterwards, so those environment variables may not be available yet.
You should set them manually in the bootstrap code unfortuantely.
Is there some default Docker image you ship with ClearML that you'd recommend, or can/should we use our own? 🙂
I believe that happens natively thanks to pyhocon? No idea why it fails on mac
It does (root in a docker container); it shouldn't touch /run/systemd/generator/systemd-networkd.service
anyway though
I think this is about maybe the credential.helper
used
I'd like to set up both with and without GPUs. I can use any region, preferably some EU one.
Yes exactly, but I guess I could've googled for that 😅
Copy the uncommitted changes captured by ClearML using the UI, write to changes.patch
, run git apply changes.patch
👍
Maybe it's the missing .bashrc
file actually. I'll look into it.
Thanks CostlyOstrich36 !
Say I upload each of these yamls as a configuration object (as with the above). Once I try to load bar.yaml remotely it will crash, since foo.yaml is missing (and is instead a clearml configuration object).
Does that make sense?
i.e. It does not process tasks on its own?
Anything else you’d recommend paying attention to when setting the clearml-agent helm chart?
@<1523701205467926528:profile|AgitatedDove14> this
SuccessfulKoala55 TimelyPenguin76
After looking into it, I think it's because our AMI does not have docker, and that the default instance suggested by ClearML auto scaler example is outdated
https://clear.ml/docs/latest/docs/references/sdk/services_monitor
Then you can run this as a task, see also this example https://clear.ml/docs/latest/docs/guides/services/slack_alerts
nevermind! Found and answered (solution in the issue linked above)