SuccessfulKoala55 it does not

Ultimately we're trying to avoid docker in AWS autoscaler (virtualization on top of virtualization seems redundant), and instead we maintain an AMI for a faster boot sequence.
We had no issues when we used pip , but now when trying to work with poetry all these issues came up.
The way I understand poetry to work, is that it is expected there is one system-wide installation that is used for virtual environment creation and manipulation. So at least it may be desired that the ...

I've tried also e.g. setting gent.package_manager.priority_packages = ["poetry"] , and/or agent.package_manager.poetry_version = ">1.2.0" , and other flags, but these affect only the main /clearml_agent_venv environment, and not the one actually generated by the clearml-agent when executing the task

This happened again 🤔
How many files does ClearML touch? :shocked_face_with_exploding_head:

Let me know if there's any additional information that can help SuccessfulKoala55 !

SuccessfulKoala55 no that did not solve the issue 😞

Let me have a quick look.

I'll try a hacky-way around it with sed -i 's/include-system-site-packages = false/include-system-site-packages = true/g' clearml_agent_venv/pyvenv.cfg and report back.

Hi SuccessfulKoala55 !
Could you elaborate on how best to delete these from the database?

It is

Created this for follow up, SuccessfulKoala55 ; I'm really stumped. Spent the entire day on this 🥹
https://github.com/allegroai/clearml-agent/issues/134

It failed on some missing files in my remote_execution, but otherwise seems fine now

Hi AgitatedDove14 !

Ah, thanks! I'll use the artifacts for linking.

We've forgone the "use current task" already because it indeed made things even more difficult (the task that was used is then automatically hidden by this automatic renaming of dataset tasks).
The current implementation (since 1.6.3 I think) creates the issues in the linked comment (with images to visualize).

That's a nice work around of course - I'm sure it works and I'll give it a shot momentarily. I'm just wondering if ClearML could automatically recognize image files in upload_artifact (and other well known suffixes) and do that for me.

Indeed. I'll open an issue, sure!

I guess I'll have to rerun the experiment without tags for this?

So the ..data referenced in the example above are part of the git repository?
What about setting the working_directory to the user working directory using Task.init or Task.create ?

https://clearml.slack.com/archives/CTK20V944/p1652350709961419

I commented on your suggestion to this on GH. Uploading the artifacts would happen via some SDK before switching to remote execution.
When cloning a task (via WebUI or SDK), a user should have an option to also clone these input artifacts or simply linking to the original. If linking to the original, then if the original task is deleted - it is the user's mistake.

Alternatively, this potentially suggests "Input Datasets" (as we're imitating now), such that they are not tied to the original t...

That gives us the benefit of creating "local datasets" (confined to the scope of the project, do not appear in Datasets tabs, but appear as normal tasks within the project)

DeterminedCrab71 not in this scenario, but I do have it occasionally, see my earlier thread asking how to increase session timeout time

I was thinking of using the --volume settings in clearml.conf to mount the relevant directories for each user (so it's somewhat customizable). Would that work?

It would be amazing if one can specify specific local dependencies for remote execution, and those would be uploaded to the file server and downloaded before the code starts executing

Not really, I've only been able to somewhat understand the scope of where it happens, and I'm not sure it's even a ClearML issue (maybe matplotlib)

If relevant, I'm using Chrome Version 101.0.4951.41 (Official Build) (64-bit)

Uhhh, but pyproject.toml does not necessarily entail poetry... It's a new Python standard

Hm, that seems less than ideal. I was hoping I could pass some CSV locations. I'll try and find a workaround for that. Thanks!

I understand, but then the toml file needs to be parsed to ensure poetry is used. It's just a tool entry in the pyproject.toml.

Not really - it will just show the string. A preview would be more like a low-res version of the uploaded image or similar.

I just used this to create the dual_gpu queue:
clearml-agent daemon --queue dual_gpu --create-queue --gpus 0,1 --detached

Let me know if you do; would be nice to have control over that 😁