Yes AgitatedDove14 , I am not sure what they use by default. Here is a simple working example:
` from typing import Optional

import torch
from clearml import Task
from pytorch_lightning import LightningDataModule, LightningModule
from pytorch_lightning.utilities.cli import LightningCLI
from torch.utils.data import DataLoader, Dataset, Subset

class RandomDataset(Dataset):
def init(self, size, length):
self.len = length
self.data = torch.randn(length, size)

def ...

AgitatedDove14 Thanks! Im trying to figure out how to create a minimum working example! I am also working with Hydra so that may be a thing. The extension is whats causing it to fail (haven’t figured out why).

Hey AgitatedDove14 after playing around seems that if the callback filepath points to an hdf5 file it is not uploaded.

Side note: When running src.train as a module the server gets the command as src and has to be modified to be src.train

I changed it to point to a folder and it shows up

AgitatedDove14 I am not sure why the packages get different versions, maybe since the package is not directly imported in my code it is possible to get a different version to what I have locally (?). Should all the libraries versions match exactly between local and the code that runs in the agent? The Task.add_requirements(package_name, package_version=None) workaround works perfectly! I just add the previous version that doesn’t break the code. Yes, definitely a force flag could help ...

Using detect_with_pip_freeze: true runs into package version not found for some of the ones I have locally.

Thanks SuccessfulKoala55 I’ll give it a try!

SuccessfulKoala55 just to let you know: since I opened the link straight from the GCP console it was using https on the address instead of http hence the error. Thanks a lot for your help!

It is the latest RC, I get the following:
` Executing Conda: /opt/conda/bin/conda install -p /home/ramon/.clearml/venvs-builds/3.8 -c pytorch -c conda-forge -c defaults 'pip<20.2' --quiet --json
Pass
Trying pip install: /home/ramon/.clearml/venvs-builds/3.8/task_repository/my-rep.git/requirements.txt
Executing Conda: /opt/conda/bin/conda install -p /home/ramon/.clearml/venvs-builds/3.8 -c pytorch -c conda-forge -c defaults numpy==1.20.3 --quiet --json
Pass
Warning, could not locate PyTorch to...

Thanks 🙌

Yes! I will take a look at it!

I configured a firewall rule that opened the ports for the instance (not 100% sure if this is the right way) using network tags. Yes, the whole screen is black and no trains logo show up: Safari can’t open the page because the server where this page is located isn’t responding.

AgitatedDove14 its on the checkpoint

Hi AgitatedDove14 ! Do you have any updates on this?

With pip I get the first error I showed, I tried conda and it starts running but at some point crashes with:
clearml_agent: ERROR: 'NoneType' object has no attribute 'lower'

Thanks!

My bad :man-facepalming: It was just specifying weights_path=dirpath since the first argument is weights_filename

Sure! I enqueue the experiment from my local machine:
python -m src.train model=my_model loss=my_loss dataset=my_dataset

Then I go to the server and run the experiment and create a copy to run with a new model. On the copy, I go to the script path and modify it to be:
-m src.train model=my_other_model loss=my_loss dataset=my_dataset

The new experiment, even though the script path has my_new_model default, starts training using my_model .

I can also see ...

Hey CostlyOstrich36 sorry to ping you! Let's say I enqueue multiple experiments on a couple of agents and one of them fails. Is it possible to restart the experiment from the UI using the latest checkpoint? What if the experiment gets assigned to the other agent? I am not sure how the continue_last_task flag would help in this case.

Nice catch AgitatedDove14 ! Sure I’ll open the issue right now.

Thanks AgitatedDove14 🙌

I am using pytorch_lightning , I'll try to create a snippet I can share! Thanks 🙌

Pigar is capturing different versions that the ones I have installed on my local machine (not a problem except for one). I just want to force the version of that package in a way that I don’t have to manually change it from the UI for every experiment.

Also, should I allow 8080 , 8008 , and 8081 on ingress and egress on GCP or is only egress enough?

AgitatedDove14 Thanks! I’ll give it a try! Makes sense 👌

Thanks AgitatedDove14

Thats really cool! But I would still prefer avoid using pip_freeze, is there a way?

Thanks AgitatedDove14 🙌

No, I have all the packages with a version. I just want to know if there is a way to override the requirements versions detected by Pigar when using detect_with_pip_freeze: false . I have locally cloudpickle==1.4.1 but when running the code and sending the task to the node the environment uses cloudpickle==1.6.0 . I have to manually change the version on the UI. Is there a way to force this single package to have a version? Maybe on the requirments.txt or something similar