Reputation
Badges 1
53 × Eureka!We can't really know (possibly ever 🙂 ), but if the bug happens again I'll be sure to report it here.
The UI shows the log as is (and as pasted above). In the console I'm getting correct output (a single tqdm progress line):
` [2021-09-17 13:29:51,860][pytorch_lightning.utilities.distributed][INFO] - GPU available: True, used: True
[2021-09-17 13:29:51,862][pytorch_lightning.utilities.distributed][INFO] - TPU available: False, using: 0 TPU cores
[2021-09-17 13:29:51,862][pytorch_lightning.utilities.distributed][INFO] - IPU available: False, using: 0 IPUs
[2021-09-17 13:29:51,866][pytorch_ligh...
Not sure how to check that tbh. Does this help:root@aea5d96a8ed3:/usr/agent# clearml-agent --version CLEARML-AGENT version 1.0.0
Would be nice to display this info maybe somewhere inhere:
In case anyone is interested, the minimum effort workaround I found is to edit pytorch_lightning/callbacks/progress.py
and change all occurrences of dynamic_ncols=True
to dynamic_ncols=False
in the calls to tqdm
. One could of course implement a custom callback inheriting from their ProgressBar
class.
Interesting, I don't get newlines in any of my consoles:ClearML Task: overwriting (reusing) task id=38cc10401fcc43cfa432b7ceed7df0cc 2021-10-08 14:57:53,704 - clearml.Task - INFO - No repository found, storing script code instead ClearML results page:
`
...
Try updating to 1.1.0?
I don't control tqdm, (otherwise I would have already gone for Stef's suggestion) - pytorch-lightning does in this particular script 😞 .
Hi Jake, thanks for the reply. I've tried the account key method, works fine - but unfortunately clearml expects an old version of azure-storage-blob
(<2.1), which is incompatible with the recent versions (^12.). Any clues of how we could work around this one? Thanks again.