Reputation
Badges 1
59 × Eureka!Here is the developer tool Network screen capture after refreshing the page and trying to login.
Woohoo! ๐
The instructions in the https://superuser.com/questions/278948/clear-cache-for-specific-domain-name-in-chrome/444881#444881 were not accurate, but they brought me close enough.
Here is the exact sequence of operations:
F12 --> Applications tab --> Storage --> Clear site data --> refresh login screen
Thanks everyone for your help!
TimelyMouse69 , yes, I ran successfully the first time before cloning it.
Yes, I create the experiment by calling Task.init.
As you suggested, in the experiment tab I define the script path and the working directory.
Again, the task only created the environment and after that reported 'completed' without running my code.
Attaching the log of the last run, with the setting of the script and the folder.
The only thing I need to do is clone my experiment. Can you help me make this happen?
As you suggested, I tried with a git repository. Got a completely different error. Attached is the log file. Any idea what's wrong?
I don't get the error any longer and the experiments get deleted as expected. So no complains on my side...
I am running my own server. Those are not example experiments.
Where do I see the agent print outs?
I am using an old version. It's a trains server of version 0.16.3.
I clicked Fetch/XHR and got the following (after another reboot)
Many errors :white_frowning_face: . Any idea what they mean?
I did not upgrade anything and did not do docker pull.
I am having a temporary network issue . Will send the output of the โ docker inspectโ as soon as I can reconnect to my server.
No other error messages but the dashboard screen is blank.
However, there is a breakthrough: I can run the dashboard from Safari (Mac browser). So the problem is only in Chrome.
Here is a snapshot of the blank screen:
I can enter my user name but even the button underneath it is blank (see below). Once clicking it, the whole screen is blank as in the 1st image that I sent.
I am using a self hosted server.
I suspect that maybe the server gets stuck when I compare a large number of experiments (~10). Can that be possible?
SuccessfulKoala55 , here is the output of "docker inspect trains-webserver" (attached).
No. I put a break point in my python script, and examined os.environ. The only environment variable with 'CLEARML' in its name is CLEARML_PROC_MASTER_ID, whose value is '16188:' (maybe it means something to you?)
I get an empty list for the 'XHR' filter.
Could it be the file you are trying to run is not in the repository ?
It is unclear what file is missing. The only hint is "Keyerror: '.'" and I am not sure what that refers to. All my code files are in the repository. Maybe the problem is with some installed package file?
Are you running inside a docker ?
No, I am running inside a conda environment.
Any chance you can send the full log ? (edited)
What I sent is the full agent daemon log. If you are asking for the console...
I don't see a cache related to clearml:(base) sigalr@rack-bermano-g03:~$ find . -name *cache* -not -name __pycache* ./.pycharm_helpers/python_stubs/cache ./.cache ./.conda/pkgs/cache
The 1st and last are obviously unrelated, and the middle one contains files related to python:(base) sigalr@rack-bermano-g03:~$ ls .cache/ matplotlib motd.legal-displayed pip
ok, so ~/clearml.conf points to ~/.clearml/cache, and such a file does not exist.
Thanks, I will give it a try
Who/What created the initial experiment ?
I created the initial experiment from command-line, with either "python folder/script.py" or "python -m folder.script".
Both end up with the experiment not running. I am attaching an agent daemon log where the initial experiment was called with "python folder/script.py".
Why isn't the entry point just the python script?
The entry point is folder.script and not just the script because I need the 'current' folder while running the script ...
AgitatedDove14 , thank you so much for your help.
I had a long video session today with the Israeli clearml engineers. There were plenty of things I had to do, and the two major ones were to define the environment variable CLEARML_AGENT_SKIP_PIP_VENV_INSTALL so it points to my conda environment python, and to call 'import clearml' from the top of my file (it was called from inside a method).
So now I can clone ๐
Attached are the agent log and the task log