Still crashing, I think that may not be the correct virtual environment to edit 🤔
It's the one created later down the line
I'll try a hacky-way around it with sed -i 's/include-system-site-packages = false/include-system-site-packages = true/g' clearml_agent_venv/pyvenv.cfg and report back.
I'll try that in a bit (that requires some access control changes). Any idea how can I modify the dynamically created virtualenv?
` Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
The currently activated Python version 3.10.6 is not supported by the project (~3.8.0).
Trying to find and use a compatible version.
Using python3.8 (3.8.16)
Creating virtualenv ... in /root/.clearml/venvs-builds/3.10/task_repository/...git/.venv
Installing dependencies from ...
Created this for follow up, SuccessfulKoala55 ; I'm really stumped. Spent the entire day on this 🥹
https://github.com/allegroai/clearml-agent/issues/134
Anyway sounds good! 🙂
AgitatedDove14 for future reference this is indeed a PEP-610 related bug, fixed in https://python-poetry.org/blog/announcing-poetry-1.2.0a1/ . I see we can choose the pip version in the config, can we also set the poetry version used? Or is it updated from the lock file itself, or...?
Hmmm maybe 🤔 I thought that was expected behavior from poetry side actually
Haha, I've opened so many issues these past few days... Sure, np!
Does that make sense CostlyOstrich36 ? Any thoughts on how to treat this? For the time being I'm also perfectly happy to include something specific to extra_clearml_conf , but I'm not sure how to set the sdk.aws.s3.credentials to be a list of dictionaries as needed
Oh and clearml-agent==1.1.2
Sorry, been away for a while!
I have no additional information, since it was a bug in my model that I have since eliminated...
Maybe it was just a matplotlib error and can be dropped for now. I'll let you know if it pops up again!
Trying now with 1.4.1, but I believe the changes you're referring to SuccessfulKoala55 were also introduced in 1.4.0, right?
Basically when there are occasionally extreme values (i.e. most values fall in [0, 50] range, and one value suddenly falls in 50e+12 range), the plotting library (matplotlib or ClearML, unsure) hangs for a really long time
Hm, this didn't happen until now; I'd be happy to try again with a new version, but something with 1.4.0 broke our StorageManager, so we reverted to 1.3.2
AFAICS it's quite trivial implementation at the moment, and would otherwise require parsing the text file to find some references, right?
https://github.com/allegroai/clearml/blob/18c7dc70cefdd4ad739be3799bb3d284883f28b2/clearml/task.py#L1592
Not sure if @<1523701087100473344:profile|SuccessfulKoala55> or @<1523701827080556544:profile|JuicyFox94> maybe knows?
I was thinking of using the --volume settings in clearml.conf to mount the relevant directories for each user (so it's somewhat customizable). Would that work?
It would be amazing if one can specify specific local dependencies for remote execution, and those would be uploaded to the file server and downloaded before the code starts executing
Any updates on this? We can't do anything with our K8s since this 404...
Thanks for the reply @<1523701827080556544:profile|JuicyFox94> ! I'll debug more and let you know
And last but not least, for dictionary for example, it would be really cool if one could do:my_config = task.connect_configuration(my_config, name=name) my_other_config = task.connect_configuration(my_other_config, name=other_name) my_other_config['bar'] = my_config # Creates the link automatically between the dictionaries