![Profile picture](https://clearml-web-assets.s3.amazonaws.com/scoold/avatars/JumpyPig73.png)
Reputation
Badges 1
57 × Eureka!We have run experiments in the past (before I put ClearML into my code) which has logged scalars, plots etc. to local tensorboard. Is there any way to import this data to ClearML cloud for tracking, visualization and comparison?
I'm signed up for Pro. Is there some restricted docs site for pro users CostlyOstrich36 ?
Thanks! I'll check for this locally and get back
so there's no way to do that when running in pip or conda mode?
This is great! Thanks for the example Martin, much appreciated!
Ok. I think I misunderstood what you said. I thought you meant you've already opened a bug ticket. If that's not the case, do you want to me create one on github?
Thanks! Do you have a public bug tracker? If yes, are you able to share the issue number so I can follow it?
I need to put it into my code, so will be eagerly waiting for the fix
I haven't had much time to look into this but ran a quick debug and it seems like the exception
on the __exit_hook
variable is None
even though the process failed. So seems like hydra maybe somehow preventing the hook callback from executing correctly. will dig in a bit more next week
Also tagged you SuccessfulKoala55
Thanks for the quick support!
Thanks for getting back Martin. The hydra example fails when i try to queue it to my local withStarting Task Execution: Traceback (most recent call last): File "hydra_example.py", line 10, in <module> @hydra.main(config_path="config_files", config_name="config") AttributeError: module 'hydra' has no attribute 'main'
AgitatedDove14 finally had a chance to properly look into it and I think I know what's going on
When running any task with hydra, hydra wraps the called method in its own https://github.com/facebookresearch/hydra/blob/a559aa4bf6807d5e3a82e065987825fa322351e2/hydra/_internal/utils.py#L211 . When the task throws any exception, it triggers the except
block of this method which handles the exception.
CML marks a task as failed only if the whatever exception the task generated was not ha...
Will try this. Thanks for promptly looking into this. Much appreciated!
Do you want me to try running it manually?
For hydra-core:
` ...
- humanfriendly==10.0
- hydra==2.5
- idna==3.3
... `
Thanks! I'll give the RC a shot.
Aah I see it only says Image
. Somehow I hit tunnel vision on Base Docker Image
as stated in the docs and couldn't identify both to mean the same thing 😅 thanks
the state of the Task changes immediately when it crashes ?
I think so. It goes from running to completed immediately on crash
I thought the agent created a new conda env and installed all packages, recorded during initial task run, from scratch (except for caching with venv). Is that not the case?
Would you happen to have a timeline for when the feature might become available?
I tried using 1.2.0rc1
but it doesn't work as expected. We have a bunch of options for fire in the entrypoint, but irrespective of whichever I enter on the command line, fire still just executes the first command that was defined in my dictionary under fire.Fire({...})
. It however routes to the correct command if I use 1.1.6
which tells me that this is being caused by some issue with 1.2.0rc1
(the one created when you executed the code on your laptop
I haven't executed the task myself at all. I just cloned it from the examples that are available in the SaaS console upon account creation - specifically hyper-parameters example
under the ClearML Examples
project.
OS - Ubuntu 20.04
Conda - 4.10.3
The agent is running in a conda env with python==3.9.7
Is this the info you were looking for?
I think there's some confusion here. I'm not running the server. My metrics are getting logged to the CML cloud.
No, we currently don't handle it gracefully. It just crashes. But we do use hydra which does sort of arrests that exception first. I'm wondering if it's Hydra causing this issue. I'll look into it later today