Yes, exactly! Unfortunately I am not so familiar with the internals of the library but I could take a look and figure that out.

Sure! For torch I have:

torch==2.0.1
    # via
    #   monai
    #   pytorch-lightning
    #   torchio
    #   torchmetrics

Using detect_with_pip_freeze: true runs into package version not found for some of the ones I have locally.

Pigar is capturing different versions that the ones I have installed on my local machine (not a problem except for one). I just want to force the version of that package in a way that I don’t have to manually change it from the UI for every experiment.

What additional context do you need?

@<1523701070390366208:profile|CostlyOstrich36> Thanks for the help! It ended being a mistake on my side. Misconfigured the VM's memory and it had only 3.75 G. Failed when installing torch.

Any idea why this could happen?

Nice catch AgitatedDove14 ! Sure I’ll open the issue right now.

So should I set them all with a default value? The working dir is the project one, the one that contains the module package

TimelyPenguin76 I found out its just one package that is causing the error ( cloudpickle breaks everything). Is there a way to use Pigar but force a single package to have a version?

No, I have all the packages with a version. I just want to know if there is a way to override the requirements versions detected by Pigar when using detect_with_pip_freeze: false . I have locally cloudpickle==1.4.1 but when running the code and sending the task to the node the environment uses cloudpickle==1.6.0 . I have to manually change the version on the UI. Is there a way to force this single package to have a version? Maybe on the requirments.txt or something similar

I configured a firewall rule that opened the ports for the instance (not 100% sure if this is the right way) using network tags. Yes, the whole screen is black and no trains logo show up: Safari can’t open the page because the server where this page is located isn’t responding.

So I would have to disconnect pytorch? And then upload the model at the end

It works perfectly! AgitatedDove14 There is something weird on my side 😢

Also, should I allow 8080 , 8008 , and 8081 on ingress and egress on GCP or is only egress enough?

Hi AgitatedDove14 thanks for your reply, with the dashboard I meant the Web-App (UI) . I am trying to access http://<External IP>:8080 but unfortunately nothing shows up.

Hey CostlyOstrich36 sorry to ping you! Let's say I enqueue multiple experiments on a couple of agents and one of them fails. Is it possible to restart the experiment from the UI using the latest checkpoint? What if the experiment gets assigned to the other agent? I am not sure how the continue_last_task flag would help in this case.

Thanks AgitatedDove14 ! seems to be subclassed model + extension

I enabled both https and http

Thanks SuccessfulKoala55 !

Yes! What env variables should I pass

It’s file://

I get the URL to the checkpoint/weights can I use this to download the weights?

How can I check that Martin?

Thanks 🙌

Yes! How can I help? AgitatedDove14

Let me double check!

AgitatedDove14 Well I have a loss function which is something like:
class MyLoss(...): def forward(...): weights = self.compute_weights(...) return (weights * (target-preds)).mean()There seems to be a problem on certain batch when computing the weights. What would be the best way to log the batch that causes the problem, along with the weights being computed.

AgitatedDove14 Thanks! Im trying to figure out how to create a minimum working example! I am also working with Hydra so that may be a thing. The extension is whats causing it to fail (haven’t figured out why).

Hi AgitatedDove14 ! Do you have any updates on this?