
Reputation
Badges 1
53 × Eureka!you think that simply adding origin will fix this - I really don't mind doing that
Not sure how to check that tbh. Does this help:root@aea5d96a8ed3:/usr/agent# clearml-agent --version CLEARML-AGENT version 1.0.0
Would be nice to display this info maybe somewhere inhere:
Sorry, I meant the "origin" part. The warning is no more.
Hi AgitatedDove14
this is how our calls look like:
` from pytorch_lightning.loggers import TensorBoardLogger
logger = TensorBoardLogger(save_dir=".", name="debug plotting", 1)
logger.experiment.add_histogram(f"A", data[data.by == 0])
logger.experiment.add_histogram(f"B", data[data.by == 1]) `the result of which is shown in my post above.
This is some test data, and how we'd like things to look:
` def make_data(size: int=10000, n: int=5) -> pd.DataFrame:
x = np.abs(np.random.normal(siz...
AgitatedDove14 Yes! That would be exactly what I want (i.e. get_configuration_as_dict
.) Alas, no such thing exists in 1.4.1. Is that supposed to come in a next version?
CostlyOstrich36
` {"meta":{"id":"3cceedbbc65d480096ebb02b5aba5902","trx":"3cceedbbc65d480096ebb02b5aba5902","endpoint":{"name":"tasks.get_configurations","requested_version":"2.17","actual_version":"1.0"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"configurations"...
Hi SweetBadger76 , thanks, I think I've made it work. The main point of confusion was between dealing with different type of Task
objects (i.e. clearml.backend_api.services.v2_13.tasks.Task
returned by get_all
, which don't have any of those methods).
Interestingly, set_parameters
didn't just work as expected, I had to flatten the dicts myself (which clearml apparently does on its own when I call set_parameters
on a new task.
Thank you all. 🙏
'scikit' worked nicely, thanks again
Interesting, I don't get newlines in any of my consoles:ClearML Task: overwriting (reusing) task id=38cc10401fcc43cfa432b7ceed7df0cc 2021-10-08 14:57:53,704 - clearml.Task - INFO - No repository found, storing script code instead ClearML results page:
`
...
This is how a configuration item looks like:<tasks.ConfigurationItem: { "name": "filter", "value": "inference = [{\n type = \"StreamFilter\"\n params {\n context = \"full\"\n op = \"or\"\n lower_bounds {\n key = 16\n mouse = 32\n }\n }\n }]\ntrain {\n users {\n op = \"and\"\n lower_bounds {\n min_sessions = 32\n }\n }\n}", "type": "dictionary" }>
The value
is a string that prints pretty but I'm not sure how to p...
I'm been overly optimistic it seems (based on the hints from this issue: https://stackoverflow.com/questions/59455268/how-to-disable-progress-bar-in-pytorch-lightning ). It really looked like it worked. But I was mislead by the fact that the issue has a peculiar pattern to it: it doesn't appear at the beginning of each epoch. There seem to be a couple of extra cr / flushes at the beginning and then a (puzzling) extra "Validation" log line seems to trigger the continuous scrolling. Anyway, my...
New to lightning too, but I'm suspecting that since your args don't mention a specific logger, the pl trainer will instantiate the default one. Excerpt from the trainer docstring:logger: Logger (or iterable collection of loggers) for experiment tracking. A ``True`` value uses the default ``TensorBoardLogger``. ``False`` will disable logging. If multiple loggers are provided and the
save_dir` property of that logger is not set, local files (check...
Hi again. After looking into the matter a little bit, I realise I'd have liked having the option of using a StoreManager
ABC which I would implement myself using whatever storage provider I happen to use and whatever package versions happened to support it. To put it differently, instead of you implementing managers for gcs, azure, aws, etc, it would be a much nicer alternative (for me, and I suspect eventually for you too) for clearml's store manager to wrap whatever object the user pr...
If we decide go forward with clearml we'll probably do just that 🙂
I don't control tqdm, (otherwise I would have already gone for Stef's suggestion) - pytorch-lightning does in this particular script 😞 .
I found out that the lightning trainer has a progress_bar_refresh_rate
argument (default set to 1) which produces the spamming logs. If I set that to 10, I get 1/10th of the spam (but a janky progress bar in the console). I could set it to 0 to disable it, but that's not really a fix. What I'd really want is the same behaviour in the console (one smooth progress bar) and one line per epoch in the logs; high hopes, right? 😊
In case anyone is interested, the minimum effort workaround I found is to edit pytorch_lightning/callbacks/progress.py
and change all occurrences of dynamic_ncols=True
to dynamic_ncols=False
in the calls to tqdm
. One could of course implement a custom callback inheriting from their ProgressBar
class.
Hi Martin, it is a tqdm parameter (the default ProgressBar
in pytorch lightning is unfortunately relying on tqdm). This is from the tqdm docs:dynamic_ncols : bool, optional If set, constantly alters
ncolsand
nrows` to the
environment (allowing for window resizes) [default: False].
nrows : int, optional
The screen height. If specified, hides nested bars outside this
bound. If unspecified, attempts to use environment...
Hi Martin, to expand on my previous comments: the template for _Driver
already exists; I'm suggesting to make it public. Consequently, StorageHelper
should accept a driver
parameter to __init__
, defaulting to None
. Only when its value is not provided by the user should the library go out of its way to do the right thing and check all the known storage providers, fetch credentials, what not - stuff that will not work for most users, most of the time (even if you ...
Hi Jake, thanks for the reply. I've tried the account key method, works fine - but unfortunately clearml expects an old version of azure-storage-blob
(<2.1), which is incompatible with the recent versions (^12.). Any clues of how we could work around this one? Thanks again.
` # Development mode worker
worker {
# Status report period in seconds
report_period_sec: 2
# ping to the server - check connectivity
ping_period_sec: 30
# Log all stdout & stderr
log_stdout: true
# Carriage return (\r) support. If zero (0) \r treated as \n and flushed to backend
# Carriage return flush support in seconds, flush consecutive line feeds (\r) every X (default: 10) s...
Unfortunately there doesn't seem to be any out-of-the-box functionality for ridgeline plots (joyplots) in plotly. They are certainly doable ( https://www.python-graph-gallery.com/ridgeline-graph-plotly , or https://chart-studio.plotly.com/~empet/14632/plotly-joyplotridgelines/#/ ) but I'd guess this won't happen any time soon 🤭 . We'd be happy with also having functionality similar to the one from the Scalars tab: first isolating one iteration (the latest by default) and grouping togeth...
Hi AgitatedDove14 , I deleted everything in /opt/clearml as per the docs. Should I delete anything else?