Reputation
Badges 1
662 × Eureka!When is the next release expected? π
(in the current version, that is, weβd very much like to use them obviously :D)
You don't even need to set the CLEARML_WORKER_ID, it will automatically assign one based on the machine's name
It does, but I don't want to guess the json structure (what if ClearML changes it or the folder structure it uses for offline execution?). If I do this, I'm planning a test that's reliant on ClearML implementation of offline mode, which is tangent to the unit test
Or well, because it's not geared for tests, I'm just encountering weird shit. Just calling task.close()
takes a long time
This is with:Task.set_offline_mode(True) task = Task.init(..., auto_connect_streams=False)
I'm working on the config object references π
Either one would be nice to have. I kinda like the instant search option, but could live with an ENTER to search.
I opened this meanwhile - https://github.com/allegroai/clearml-server/issues/138
Generally, it would also be good if the pop-up presented some hints about what went wrong with fetching the experiments. Here, I know the pattern is incomplete and invalid. A less advanced user might not understand what's up.
I'm trying to decide if ClearML is a good use case for my team π
Right now we're not looking for a complete overhaul into new tools, just some enhancements (specifically, model repository, data versioning).
We've been burnt by DVC and the likes before, so I'm trying to minimize the pain for my team before we set out to explore ClearML.
Hm, that seems less than ideal. I was hoping I could pass some CSV locations. I'll try and find a workaround for that. Thanks!
Okay trying again without detached
See e None @<1523701087100473344:profile|SuccessfulKoala55>
Now my extra_vm_bash_script
looks like so:
` deactivate
apt-get install -y gfortran libopenblas-dev liblapack-dev libpq-dev python-is-python3 python3-pip python3-dev proj-bin libgraphviz-dev graphviz graphviz-dev libgdal-dev
apt-get install software-properties-common -y
add-apt-repository ppa:deadsnakes/ppa -y
apt update
apt install python3.7 python3.8 python3.9 python3.7-distutils python3.8-distutils python3.9-distutils python3.10-distutils python3.7-dev python3.8-dev python3.9-dev pyt...
I'll try a hacky-way around it with sed -i 's/include-system-site-packages = false/include-system-site-packages = true/g' clearml_agent_venv/pyvenv.cfg
and report back.
But to be fair, I've also tried with python3.X -m pip install poetry
etc. I get the same error.
That's fine for the current use-case I believe.
Once the team is happy with the logging functionality, we'll move on to remote execution and things will update.
If I add the bucket to that (so CLEARML_FILES_HOST=
s3://minio_ip:9000/minio/bucket ), I then get the following error instead --
2021-12-21 22:14:55,518 - clearml.storage - ERROR - Failed uploading: SSL validation failed for
... [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)
I guess the thing that's missing from offline execution is being able to load an offline task without uploading it to the backend.
Or is that functionality provided by setting offline mode and then importing an offline task?
Nope, no other config files
Ah, uhhhh whatever is in the helm/glue charts. I think itβs the allegroai/clearml-agent-k8s-base
, but since I hadnβt gotten a chance to try it out, itβs hard to say with certainty which would be the best for us π
I'll give it a shot. Honestly, the SDK documentation for both InputModel and OutputModel is (sorry) horrible ...
Can't wait for the documentation revamping.
Is it CLEARML_CONFIG_FILE
? (I had to dig this from the GH code π
)
Or if it wasn't clear, that chunk of code is from clearml's dataset.py
That could be a solution for the regex search; my comment on the pop-up (in the previous reply) was a bit more generic - just that it should potentially include some information on what failed while fetching experiments π
Is there a preferred way to stop the agent?
That will come at a later stage
I've also followed https://clearml.slack.com/archives/CTK20V944/p1628333126247800 but it did not help
Anyway sounds good! π
I will TIAS, but maybe worthwhile to also mention if it has to be the absolute path or if relative path is fine too!