Reputation
Badges 1
53 × Eureka!I'm been overly optimistic it seems (based on the hints from this issue: https://stackoverflow.com/questions/59455268/how-to-disable-progress-bar-in-pytorch-lightning ). It really looked like it worked. But I was mislead by the fact that the issue has a peculiar pattern to it: it doesn't appear at the beginning of each epoch. There seem to be a couple of extra cr / flushes at the beginning and then a (puzzling) extra "Validation" log line seems to trigger the continuous scrolling. Anyway, my...
Try updating to 1.1.0?
Thanks AppetizingMouse58 . I managed to fix it by removing docker completely and reinstalling it.
I actually put all the commands in a script. The failure mode is exactly the same. I have no idea what to do next.
` #!/bin/bash
clearml_root=$1
if [[ $# -gt 0 ]]; then
echo Using "$1" as root
else
echo No root argument was provided, using /datadrive1
clearml_root=/datadrive1
fi
clearml="$clearml_root/clearml"
rm -R "$clearml"
mkdir -p "$clearml"/data/elastic_7
mkdir -p "$clearml"/data/mongo_4/db
mkdir -p "$clearml"/data/mongo_4/configdb
mkdir -p "$clearml"/data/redis
mkdir -p "$cl...
I didn't add that to the script since the effect is persistent (i.e. it only needs to be done once, right?) In any case, I checked that multiple times and it was as expected.
New to lightning too, but I'm suspecting that since your args don't mention a specific logger, the pl trainer will instantiate the default one. Excerpt from the trainer docstring:logger: Logger (or iterable collection of loggers) for experiment tracking. A ``True`` value uses the default ``TensorBoardLogger``. ``False`` will disable logging. If multiple loggers are provided and the
save_dir` property of that logger is not set, local files (check...
Unfortunately it still happens 😞 :
` Epoch 51: 100%|███████████████████████████████████████████████████████████| 361/361 [02:52<00:00, 2.10it/s, loss=0.169, v_num=9-29]
2021-09-17 09:58:22,253 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
2021-09-17 10:03:22,254 - clearml.Task - INFO - Repository and package analysis timed out (300.0 sec), giving up
2021-09-17 10:03:22,313 - clearml.Task - WARNING - Failed auto-det...
I'll let you know asap
Yes, I find myself trying to select "points" on the overview tab. And I find myself wanting to see more interesting info in the tootip.
The problem appears to be related to the lack of an "origin" remote. However, I think the configuration is perfectly legitimate, so maybe there's a better alternative than relying on the behaviour of git ls-remote --get-url
.
Hi SweetBadger76 , thanks, I think I've made it work. The main point of confusion was between dealing with different type of Task
objects (i.e. clearml.backend_api.services.v2_13.tasks.Task
returned by get_all
, which don't have any of those methods).
Interestingly, set_parameters
didn't just work as expected, I had to flatten the dicts myself (which clearml apparently does on its own when I call set_parameters
on a new task.
Thank you all. 🙏
OK I can confirm that those display ok for me too. My problem is, they don't for my experiments, which is I care about..
I apologize 😳 . Turns out I have all those things. Again, my apologies 😞 .
Sorry, I meant the "origin" part. The warning is no more.
✦ ❯ git remote -v github git@github.com:biocatchltd/volt.git (fetch) github git@github.com:biocatchltd/volt.git (push)
Take a guess 😂 (it's "origin")
there is - it's called "github"
✦2 ❯ git remote show
github
Must be something else foul at play here..
We can't really know (possibly ever 🙂 ), but if the bug happens again I'll be sure to report it here.
The template appears to be <alias> <url> <fetch|push>
.
The .git/config
file has sections for each remote too. Example:[remote "github"] url = git@github.com:biocatchltd/volt.git fetch = +refs/heads/:refs/remotes/github/
Would be nice to report which remote the checked out branch actually tracks.
` radu on vm-aimbrain-01 in experiments/runners/all via 🐍 v3.8.5 via C volt
❯ git ls-remote --get-url github
github
radu on vm-aimbrain-01 in experiments/runners/all via 🐍 v3.8.5 via C volt
❯ git ls-remote --get-url
fatal: No remote configured to list refs from.
radu on vm-aimbrain-01 in experiments/runners/all via 🐍 v3.8.5 via C volt
❯ git --version
git version 2.17.1 `
Hi AgitatedDove14 , I deleted everything in /opt/clearml as per the docs. Should I delete anything else?
I forgot to say I've set up a local server - we are still testing phase. I've created credentials for them because they couldn't generate them for themselves (they did clearml-init, and have eacha clearml.conf file but the ADD CRENDENTIALS part didn't show up for them).