Hi @<1535069219354316800:profile|PerplexedRaccoon19> , it depends on how you run the agent that is listening to your queue. If you set it to run with multiple gpus you'd be able to utilize multiple gpus in your sessions 🙂
Hi FlutteringWorm14 , what happens if you try to delete those tasks manually?
Hi @<1535069219354316800:profile|PerplexedRaccoon19> , the agent will try to use the relevant python version according to what the experiment ran on originally. In general, it's best to run inside dockers with a docker image specified per experiment 🙂
Hi @<1625666182751195136:profile|MysteriousParrot48> , how did you run the original experiment? Can you add the full log and also the agent configuration?
Hi @<1582904426778071040:profile|SteepBat69> , that's an interesting question. In theory I think it should be possible
SarcasticSparrow10 , please note that during the upgrade you do NOT copy /opt/clearml/data/mongo
into /opt/clearml/data/mongo_4
, you create the folder like in the instructions: sudo mkdir /opt/clearml/data/mongo_4
This is the reason that it is giving out errors - You've got old mongo data in your mongo 4 folder...
Please follow the instructions to the letter - this should work 🙂
The cool thing is that you can also configure this from code as well 🙂
One example is:
https://clear.ml/docs/latest/docs/references/sdk/task#taskinitTask.init(...,output_uri="<URL_TO_BUCKET>")
You can configure it in ~/clearml.conf
at api.files_server
Yeah I think you're right. You're basically looking for a capability to segregate users into groups so they won't have visibility between themselves?
Role based access controls are supported only in Scale/Enterprise versions.
Hi @<1749602873152376832:profile|EncouragingSquid4> , Welcome!
To address your questions:
- I think you can do routing for them but you'd have to set it up.
- I assume the models are being saved to the files server. Can you share a screenshot?
I think so, yes
Hi FierceRabbit20 , I don't think there is such an option out of the box but you can simply add it to your startup of the machine or create a Cron job
Hi UnevenDolphin73 , can you please elaborate on what do you mean by what CPU does the queue consume?
You can pull all machine usage statistics using the API. Is there something specific you're looking for?
Hi @<1523704157695905792:profile|VivaciousBadger56> , can you provide some screenshots of what you're seeing?
Hi ThankfulHedgehong21 ,
What versions of ClearML & ClearML-Agent are you using?
Also, can you provide a small code snippet to play with?
I think this might be what you're looking for:
https://clear.ml/docs/latest/docs/references/api/workers
https://clear.ml/docs/latest/docs/references/api/queues
You can access all reports through the REST API
I'll try and see if it reproduces on my side, thanks! 🙂
ExtensiveCamel16 , hi! Hope you give us a shout out 🙂
The sample script you posted runs fine on server 1.6.0. I did however comment out from machine_learning.clearml_client import Task
and used from clearml import Task
Can you please try with the regular import?
Hi @<1523704157695905792:profile|VivaciousBadger56> , can you elaborate on this error please?
2023-02-14 13:06:44,336 - clearml.Task - WARNING - Failed auto-detecting task repository: [WinError 123] Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch: '[...]\\<input>'
Hi @<1529271098653282304:profile|WorriedRabbit94> , I'll ask the guys to take a look at this and what is required for it.
Can you verify in the INFO section of an individual step to what queue it is enqueued into? Can you see them in the Queues page?
I think this can give you more information:
https://stackoverflow.com/questions/51279711/what-does-1000-mean-in-chgrp-and-chown
This means it assigns the owner as the first linux user on that machine.
There aren't any specific functions for this. But all of this information sits on the task object. I suggest running dir(task)
to see where this attribute is stored
I think that something like that exists, it appears to be done in the paid version called hyper-datasets. The documentation is open for all apparently 🙂
Hopefully will have updates soon
EcstaticGoat95 , thanks a lot! Will take a look 🙂
What version of clearml
and clearml-agent
are you using, what OS? Can you add the line you're running for the agent?