BTW: latest PyCharm plugin with 2022 support was just released:
https://github.com/allegroai/clearml-pycharm-plugin/releases/tag/1.1.0

hmm DeliciousKoala34
what are you getting if you put this at the top of your code (the one you are running in the remote docker)
import os print([(k, os.environ[k]) for k in os.environ if k.startswith("CLEARML_")])

Hmm, could it be that the working dir is outside of the git repo?

Can you also make sure you did not check "Disable local nachine git detection" in the clearml PyCharm plugin?

And this is with the latest pycharm plugin 1.1.0 ?

In your code, can you print the following:
import os print(os.environ.keys())There should be a few keys the Pycharm plugin is sending from the local machine, pointing to the git repo

Hi HarebrainedBear62
What's the type of data ?

TenseOstrich47 FYI:
This might what you are looking for 🙂
https://github.com/allegroai/clearml-agent/blob/822984301889327ae1a703ffdc56470ad006a951/docs/clearml.conf#L61

So the thing is clearml automatically detects the last iteration of the previous run, my assumption you also add it hence the double shift.
SourOx12 could that be it?

It analyses the script code itself, going over all imports and adding only the directly imported packages

Then we can figure out what can be changed so CML correctly registers process failures with Hydra

JumpyPig73 quick question, the state of the Task changes immediately when it crashes ? are you running it with an agent (that hydra triggers) ?

If this is vanilla clearml with Hydra runners, what I suspect happens is Hydra is overriding the signal callback hydra adds (like hydra clearml needs to figure out of the process crashed), then what happens is that clearml's callback is never cal...

python package

Okay. And 

110

 means 11.1 and not 11.0? (edited)

110 means 11.0, the odd thing is, it actually installed 11.1, and from the pytorch website this is exactly how they suggest to install with conda...
Let me know if forcing the CUDA version changes anything

Hmm good question, I'm actually not sure if you can pass 24GB (this is not a limit on the GPU memory, this affects the memblock size, I think)

Hi @<1571308003204796416:profile|HollowPeacock58>

parameters = task.connect(config, name='config_params')

It seems that your DotDict does not support the python copy operator?
i.e.

from copy import copy
copy(DotDict())

fails ?

SlipperyDove40 following on the missing section name, this seems like backwards compatibility issue. Try calling with backwards_compatibility=False
my_params = Task.get_parameters(backwards_compatibility=False)This should always add the section name prefix.

CurvedHedgehog15 is it plots or scalars you are after ?

SmarmyDolphin68 , All looks okay to me...
Could you verify you still get the plot on debug samples as image with the latest trains RC
pip install trains==0.16.4rc0

FYI 🙂
https://www.tensorflow.org/guide/keras/custom_callback

And only the main one ?

https://allegro.ai/docs/webapp/webapp_project/#customizing-the-experiments-table

SmallDeer34 I have to admit this reference is relatively old, maybe we should update to auther http://clearml.ml (would that make sense ?)

Hi AstonishingWorm64
I think you are correct, there is external interface to change the docker.
Could you open a GitHub issue so we do not forget to add an interface for that ?
As a temp hack, you can manually clone "triton serving engine" and edit the container image (under the execution Tab).
wdyt?

Bottom line the driver version in the host machine does not support the CUDA version you have in the docker container

@<1523705099182936064:profile|GrievingDeer61> if you see them in your Tensorboard they will be captured in ClearML as well.
wdyt?

EDIT
examples (tensorboard or tensorboardX):
None
None

adding the functionality to clearml-task sounds very attractive!

Hmm, what do you think?
parser.add_argument('--configuration', type=str, default=None, help='Specify local configuration file' ) parser.add_argument('--configuration-name', type=str, default=None, help='configuration section name' ) ... with open(args.configuration, 'rt') as f: create_populate.task.set_configuration_object(args.name, config_text=f.read())Add h...

trains-server 3.6

you mean 0.16?

Hi @<1572395184505753600:profile|GleamingSeagull15>
Try adjusting:
None
to 30 sec
It will reduce the number of log reports (i.e. API calls)

? Do you have a link how to setup a task scheduler to run in service mode in k8s?

basically spin the agent pod and add an argument to the agent itself (this is the --service-mode)
https://clear.ml/docs/latest/docs/clearml_agent#services-mode

Hi SkinnyPanda43

I realized that the params are not being saved anymore

Could you test with clearml==1.0.4 ?