
Reputation
Badges 1
25 × Eureka!Okay let me check if I can test on this git version.
The docker crashes and I want to be abel to debug it exactly as it is run by the agent
On your machine (any machine)
pip install clearml-agent
clearml-agent build --id <taskID> --docker "local_mydocker_name"
docker run -it local_mydocker_name bash
@<1571308003204796416:profile|HollowPeacock58> seems like an internal issue copying this object config.model
This is a complex object, and it seems that for some reason
None
As a workaround just do not connect this object. it seems you cannot pickle it / copy it (see GH issue)
Hi MysteriousBee56 , do you have Trains installed from the git?
Another question, you mentioned "it breaks my execution", I'm assuming you mean trains-agent?!
If that is the case, there is a fix for trains-agent install 0.15.2rc0
- Artifacts and models will be uploaded to the output URI, debug images are uploaded to the default file server. It can be changed via the Logger.
- Hmm is this like a configuration file?
You can do.
local_text_file = task.connect_configuration('filenotingit.txt')
Then open the 'local_text_file' it will create a local copy of the data in runtime, and the content will be stored on the Task itself. - This is how the agent installs the python packages, but if the docker already contactains th...
It looks somewhat familiar ... π
SuccessfulKoala55 any idea?
Oh you can definitely use the RestAPI, but in this specific case, I'm not sure there is something better.
(BTW: Look for APIClient it a pythonic interface for the RestAPI)
Hi @<1791277437087125504:profile|BrightDog7>
Seems like mostly proportion change, the data is the same but the layout on the web is wider hence the change (btw you can download the data from the web UI as json to double check)
Notice that it tries to convert it to "interactive" data points for easier zooming etc, and that's probably the cause of the proportion change.
You can force an image (like what you get directly from matplotlib):
logger.report_matplotlib_figure(
title='NLLs...
I just set
agent.enable_git_ask_pass: true
in the config of the clearml agent (v1.5.1) and the task is still stuck at asking username when trying to get the private dependency.
Hmm that should not happen, could you delete the cache and retry? maybe?
Now that we have the free tier (a.k.a community server) we might change the default behavior.
The idea is always to allow an easy way to on-board and test the system.
ReassuredTiger98
BTW: what's the scenario where your machine reverted to the default configuration (i.e. no configuration file) ?
But that should not mean you cannot write to them, no?!
Quick update, I might have been able to reproduce the issue ( GreasyPenguin14 working "offline" is a great hack to accelerate debugging this issue, thank you!)
It seems it is related to the known and very annoying Python forking issue (and this is why changing to "spawn" method solves the issue):
https://bugs.python.org/issue6721
Long story short, in some cases when forking (i.e. ProcessPoolExecutor), python can copy locks in a "bad" state, this means that you can end up with a lock acquir...
Hi DeliciousBluewhale87
I think you are correct, there is no way to pass it.
As TimelyPenguin76 mentioned you can either set a default output_uri on the agent's config file, or edit the created Task in the UI.
What is the specific use case ? Maybe we should add this ability, wdyt?
So obviously that is the problem
Correct.
ShaggyHare67 how come the "installed packages" are now empty ?
They should be automatically filled when executing locally?!
Any chance someone mistakenly deleted them?
Regrading the python environment, trains-agent
is creating a new clean venv for every experiment, if you need you can set in your trains.conf
:agent.package_manager.system_site_packages: true
https://github.com/allegroai/trains-agent/blob/de332b9e6b66a2e7c67...
Thanks StaleKangaroo85 bug is verified. Let me check to see where exactly is the bug.
Two points
Notice that x_labels should be the size of the histogram It seems that you have to pass the labels as well (otherwise you get the trace-0), so if you add labels=['random histogram']
and labels=['random histogram2']
, you'll get the correct legend.Anyhow I'll make sure we also fix it in code so it is automatically labels are [series] if not specified, thanks!
Can you verify this example is not working for you?
https://github.com/allegroai/clearml/blob/master/examples/frameworks/hydra/hydra_example.py
We are here if you need further help π
Closing the data doesnt work: dataset.close() AttributeError: 'Dataset' object has no attribute 'close'
Hi @<1523714677488488448:profile|NastyOtter17> could you send he full exception ?
SmallAnt76
see https://clear.ml/pricing/ , under "What plan should I choose?"
what you are looking for is the first column "open-source". make sense ?
This is the prerequisites of the docker service installed on the host machine (where the agent is running)
Basically follow: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
https://docs.docker.com/compose/gpu-support/
I understand but how do you launch the cleaml-agent
itself:clearml-agent daemon --detached --queue default --docker
HealthyStarfish45 what exactly did you have in mind, in terms of the widget ?
JitteryCoyote63 Great to hear π
BTW:
Would it be possible to extendΒ
Task.init
Β with aΒ
force_reuse
Β that would enforce reusing these tasks
You can pass continue_last_task=True
I think it should be equivalent to what you suggest
It's just another flag when running the trains-agent
You can have multiple service-mode instances, there is no actual limit π
BoredHedgehog47 if you are running it on K8s, then the setup script is running before everything else, even before an agent appears on the machine, unfortunately this means the output is not logged yet, hence the missing console lines (I think the next version of the glue will fix that)
In order to test you can do:export TEST_ME
then inside your code you will be able to see itos.environ['TEST_ME']
Make sense ?
Weird ?!, I see this in the code:
https://github.com/allegroai/clearml/blob/382d361bfff04cb663d6d695edd7d834abb92787/clearml/automation/controller.py#L2871
how to put or handle this configuration and where?
In your clearml.conf on the machine with the agent just add at the bottom of the file agent.venvs_cache.path=~/.clearml/venvs-cache