This is exactly what the build command is for. I suggest reviewing the documentation
I think you need to configure api.files_server in ~clearml.conf as your s3 bucket
Hi @<1523701295830011904:profile|CluelessFlamingo93> , I think you can also control the agent sampling rate (to sample queue every 10 or 20 seconds instead of 5 for example)
Hi! Hmmm, good question. I think it's asynchronous since most of the uploading processes are usually async. Is there a specific use case you're thinking of?
RattyLouse61 , looks like a bug, but I didn't have a chance to play with it myself yet. Maybe open a github issue to follow up on this?
Strange. Can you add your clearml.conf from the agent machine? Please make sure to obscure all secrets 🙂
GreasyPenguin14 , Hi 🙂
I'm guessing that it tries to communicate during task.init()
Try running the function when you initialize the Task object
Hmmm interesting. According to bytes it looks like 2GB. What type is the file?
Hi @<1539780284646428672:profile|PoisedElephant79> , please post in the same thread you started, no need to spam the main channel 🙂
Regarding your issue - it looks like you have some issue with authentication. How are you spinning the server?
CUDA is the driver itself. The agent doesn't install CUDA but installs a compatible torch assuming that CUDA is properly installed.
I wasn't able to reproduce it on my side. Can you try the following?
In clearml/examples/reporting/mode_config.py
Under line 45:OutputModel().update_weights('my_best_model.bin')
Add the following:output_model = task.models['output'][-1]output_model.tags=['deployed']
And check in the UI if you get a tag on the model
SubstantialElk6 , the agent is designed to re-run in an environment as close as possible to the original. Can you please provide logs of the two experiments so we can compare? I'm not sure what the issue is. Do both computers have the same python versions?
Hi @<1749965229388730368:profile|UnevenDeer21> , I think this is what you're looking for
None
Hi SuperiorPanda77 ,how are the tasks running? Locally or via agent? What does the log show?
Can you add a screenshot of how you see them currently?
Hey ItchySeahorse94 , I think this might be what you're looking for 🙂
https://github.com/allegroai/clearml-serving
Hi @<1670964680270548992:profile|SuperiorOctopus47> , you can manually create experiments and log metrics into them via the REST API - None
You basically have some older runs on your tensorboard that you want to import to ClearML?
Hi @<1523701868901961728:profile|ReassuredTiger98> , you can fetch the task object, there one of the attributes of the task is it's worker. This way you can see on what machine it is running 🙂
Hi @<1572032849320611840:profile|HurtRaccoon43> are you referring to Dima's request?
Hi @<1533619716533260288:profile|SmallPigeon24> , is it possible the experiment wasn't run on a worker? In what state is the task?
@<1523701553372860416:profile|DrabOwl94> , I would suggest restarting the elastic container. If that doesn't help, check the ES folder permissions - maybe something changed
Hi SucculentWoodpecker18 , I don't think there is an updated roadmap currently. You can see updates and releases here: https://clearml.slack.com/archives/C03E7MNDG3C
Is there some specific feature you're looking for?
If you get GPU-hours per project stats it would be really cool if you added this as a pull request
Hi @<1540142651142049792:profile|BurlyHorse22> , it looks like an error in your code that is bringing the traceback. What is happening during the traceback?
Hi, SkinnyPanda43 , from what version did you upgrade to which version?
Hi UnevenDolphin73 ,
If I look at a specific experiment (say, the Artifacts tab), and then click on another experiment in the experiment list, it used to automatically show the newly selected experiment's Artifacts tab. It still does this, but it now shows a blank page. I have to choose a different tab and switch back.I think they fixed it in the next version that should be released soon.
(Not sure if by design) When selecting an experiment in a (new) project, it used to automatically swit...
OddShrimp85 Hi 🙂
I think ClearML detects the packages that were in use during the script's run. Regarding the global packages, that's what the docker image is for, so it all comes pre-installed