Hi again CostlyOstrich36 ,

I just wanted to share what ended up working for me. Basically I worked it out both for Hydra (thanks CurvedHedgehog15 ) and for PytorchLightningCLI.

So, for PL-CLI, I used this construct so we don't have to modify our training scripts based on our experiment tracker

` from pytorch_lightning.utilities.cli import LightningCLI
from clearml import Task

class MyCLI(LightningCLI):
def before_instantiate_classes(self) -> None:
# init the task
tas...

Hi CurvedHedgehog15 , thanks for replying!
I guess that one could modify the config with variable interpolation (similar to how it's done in YAML, e.g. ${encoder.layers} ) - however, it seems to be quite invasive to specify that in our trainer script 😞

It's running v7.17.18 @<1722061389024989184:profile|ResponsiveKoala38>

The lightning folks won't include new loggers anymore (since mid-2022, see None ) 🙂

Sure. I'll give it a few minor releases and then try again 🙂 Thanks for the responses @<1722061389024989184:profile|ResponsiveKoala38> !

Sorry for the late reply @<1722061389024989184:profile|ResponsiveKoala38> . So this is the diff between my local version (hosted together on a single server with docker-compose). Does anything spring to mind?

Do you mean to the Web UI?

Yes that's what I meant, sorry I'm still coming to terms with ClearML terminology 😅 . Is it possible to store the web app cloud access token serverside so we don't have to input it in the Web UI? 🙂

On the server or the client? :)

Hi CostlyOstrich36
I have created a base task on which I'm optimizing hyperparameters. With clearml-param-search I could use --params-override to set a static parameter, which should not be optimized, e.g. changing the number of epochs for all experiments. It seems to me that this capability is not present in HyperParameterOptimizer . Does that make sense?

From the example on https://clear.ml/docs/latest/docs/apps/clearml_param_search/ :
` clearml-param-search {...} --p...

Hey SweetBadger76 , thanks for answering. I'll check it out! Does that correspond to filling out azure.storage in the clearml.conf file?

And how do I ensure that the server can access the files from the blob storage?

I just tried and the result is the same. The other method only triggers on exceptions

This is on clearml v1.16.4

Yes, I tried updating recently, it costed me a full days work of rolling back versions until I found something that worked 😅

Which version of the server are you running?

@<1523701070390366208:profile|CostlyOstrich36> just opened an issue on this: None

I've tried setting the output_uri on Task.init, but that seems to only affect model checkpoints and artifacts

Hi @<1523701070390366208:profile|CostlyOstrich36> , yeah we figured as much. Is there a setting in the server that limits logging - or disables it completely?

Perfect! Thanks SuccessfulKoala55 , that would be an acceptable workaround until setup_upload also supports Azure 🙂 🙌

Hi @<1523701070390366208:profile|CostlyOstrich36> , the task is being aborted via the web UI - I have another method that catches local interrupts (exceptions like keyboard interrupts and crashes). The case is equal for running tasks via agents or just local cli

Any tips on how to check if we are storing data on deleted tasks? Maybe @<1722061389024989184:profile|ResponsiveKoala38> knows? Is there a field on each scalar that I can cross check with ClearML?

diff --git a/docker-compose.yml b/docker-compose.diff.yml
index c6b49e1..07f7f43 100644
--- a/docker-compose.yml
+++ b/docker-compose.diff.yml
@@ -5,7 +5,7 @@ services:
     command:
     - apiserver
     container_name: clearml-apiserver
-    image: allegroai/clearml:1.15.0
+    image: allegroai/clearml:latest
     restart: unless-stopped
     volumes:
     - /opt/clearml/logs:/var/log/clearml
@@ -19,17 +19,18 @@ services:
     environment:
       CLEARML_ELASTIC_SERVICE_HOST: elastics...

Sure. Really, I'm just using the default client:
# ClearML SDK configuration file
api {
web_server: http://server.azure.com:8080
api_server: http://server.azure.com:8008
files_server: http://server.azure.com:8081
credentials {
"access_key" = "..."
"secret_key" = "..."
}

}
sdk {
# ClearML - default SDK configuration

storage {
    cache {
        # Defaults to system temp folder / cache
        default_base_dir: "~/.clearml/c...

@<1722061389024989184:profile|ResponsiveKoala38> cool, thanks! I guess it will then be straightforward to script then.

What is your gut feeling regarding the size of the index? Is 87G a lot for an elastisearch index?

@<1590514584836378624:profile|AmiableSeaturtle81> this was last time i tried: https://clearml.slack.com/archives/CTK20V944/p1725534932820309

@<1590514584836378624:profile|AmiableSeaturtle81> that’s the service we are using :-)

How much RAM have you assigned to your elastic service?

Hi CostlyOstrich36
What I'm seeing is expected behavior:

In my toy example, I have a VAE which is defined by a YAML config file and parsed with PytorchLightning CLI. Part of the config defines the latent dimension (n_latents) and the number of input channels of the decoder (in_channels). These two values needs to be the same. When I just use the Lightning CLI, I can use variable interpolation with OmegaConf like this:
` class_path: mymodel.VAE
init_args:
{...}
bottleneck:
class_pat...

SuccessfulKoala55 Thanks for the help. I've setup my client to use my blob storage now, and it works wonderfully.

I've also added a token to my server, so now I can access the audio samples from the server.
Is there a way to add a common token serverside so the other members of the team don't have to create a token?

I also struggle a bit with report_matplotlib_figure() in which plots does not appear in the web ui. I have implemented the following snippet in my pytorch lightning logger:
` @...

Yeah, that makes sense. The only drawback is that you'll get a single point that all lines will go through in the Parallel Coordinates plot when the optimization finishes 🙂

@<1523701070390366208:profile|CostlyOstrich36> any thoughts? Are the model files themselves easier to serve?

How does it look in the Web UI?

I just had a look, and they are visible under debug samples, but not under plots, as I had expected.
I thought that by using report_matplotlib_figure it would get grouped under plots? 🙂