Here is the log from the instance

super, thanks a lot 🙂

because at some point it introduces too much overhead I guess

Hi TimelyPenguin76 ,
trains-server: 0.16.1-320
trains: 0.15.1
trains-agent: 0.16

Hi SuccessfulKoala55 , How can I now if I log in in this free access mode? I assume it is since in the login page I only see login field, not password field

cool, thanks AgitatedDove14 !

AgitatedDove14 Yes exactly! it is shown in the recording above

I ended up dropping omegaconf altogether

Also tried task.get_logger().report_text(str(task.data.hyperparams))
-> AttributeError: 'Task' object has no attribute 'hyperparams'

yes what happens in the case of the installation with pip wheels files?

Well, as long as you’re using a single node, it should indeed alleviate the shard disk size limit, but I’m not sure ES will handle that too well. In any case, you can’t change that for existing indices, you can modify the mapping template and reindex the existing index (you’ll need to index to another name, delete the original and create an alias to the original name as the new index can’t be renamed...)

Ok thanks!

Well, as long as you use a single node, multiple shards offer no sca...

I made sure before deleting the old index that the number of docs matched

I created a snapshot of both disks

Yes I am 🙂

When running my training code

AgitatedDove14 SuccessfulKoala55 I just saw that clearml-server 1.4.0 was released, congrats 🚀 🙌 Was this bug fixed with this new version?

Also, from https://lambdalabs.com/blog/install-tensorflow-and-pytorch-on-rtx-30-series/ :

As of 11/6/2020, you can't pip/conda install a TensorFlow or PyTorch version that runs on NVIDIA's RTX 30 series GPUs (Ampere). These GPUs require CUDA 11.1, and the current TensorFlow/PyTorch releases aren't built against CUDA 11.1. Right now, getting these libraries to work with 30XX GPUs requires manual compilation or NVIDIA docker containers.

But what wheel is downloading trains in that case?

Very nice! Maybe we could have this option as a toggle setting in the user profile page, so that by default we keep the current behaviour, and users like me can change it 😄 wdyt?

/opt/clearml/data/fileserver does not appear anywhere, sorry for the confusion - It’s the actual location where the files are stored

oh seems like it is not synced, thank you for noticing (it will be taken care immediately)

Thank you!

does not contain a specific wheel for cuda117 to x86, they use the pip defualt one

Yes so indeed they don't provide support for earlier cuda versions on latest torch versions. But I should still be able to install torch==1.11.0+cu115 even if I have cu117. Before that is what the clearml-agent was doing

That would be awesome 🎉

There is a pinned github thread on https://github.com/allegroai/clearml/issues/81 , seems to be the right place?

That said, v1.3.1 is already out, with what seems like a fix:

So you mean 1.3.1 should fix this bug?

This is new right? it detects the local package, uninstalls it and reinstalls it?

yes, that's also what I thought

I think my problem is that I am launching an experiment with python3.9 and I expect it to run in the agent with python3.8. The inconsistency is from my side, I should fix it and create the task with python3.8 with:
task.data.script.binary = "python3.8" task._update_script(convert_task.data.script)Or use python:3.9 when starting the agent

both are repos for python modules (experiment one and dependency of the experiment)

(I am not part of the awesome ClearML team, just a happy user 🙂 )

No 🙂

I get the same error when trying to run the task using clearml-agent services-mode with docker, so weird