Yes the workaround it's working 🙂

As an example, in Task.create() there is the possibility to install packages using a requirements.txt, and if not specified, it uses the requirements.txt of the repository. I'd like something like for Task.init() if possible

Nice, I didn't know that 🙂

Does it work if I launch the clearml-agent on a docker and pip doesn't know the packages to install?

My problem right now is that Pytorch Lightning need the s3fs package to store model checkpoint into s3 buckets, but in my "installed packages" is not imported and I get an import error

Because at the moment I'm having a problem with the s3fs package where I have it in my requirements.txt but the import manager at the entry point doesn't install it

Yes it does 👍 Btw, at the moment I added import(s3fs) in my entry point and it's working, thank you!

Make sure you have the S3 credentials in your agent's clearml.conf

Ok this could be a problem, as right now I'm using ec2-instances with a instance-profile (I use it in the autoscaler) so they have by the default the right s3 permissions. But I'll try it anyway

Hi AgitatedDove14 , I'm interested in this feature to run the agent and force it to install packages from requirements.txt. Is it available?

Hi TimelyPenguin76 , I used api_client.tasks.create and It works, thank you!

Actually I had the same issue even with that value set to False

Yep I changed it

Ok now I noticed that If I change the value of the port inside the Hydra parameters section ( not the overrides) It does actually change also in the experiment. The overrides doesn't seem to be working

Also, if I want to modify another parameter, e.g. ui.height I have this problem:

Yes I think it's only related to the UI. Do you think It can be fixed somehow? It would be the easiest way to launch new experiments with a different configuration

Hi AgitatedDove14 , sorry for the late reply. Btw, I tried with the latest RC and the issue is still there. So if I clone an experiment, modify an overrides params eg ['training.max_epochs=10'] my experiment run the old configuration. Therefore it seems that it doesn't change the OmegaConf configuration.

` # ClearML - Hydra Example

from clearml import Task

from dataclasses import dataclass

import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import OmegaConf

@dataclass
class MySQLConfig:
host: str = "localhost"
port: int = 3306

cs = ConfigStore.instance()

Registering the Config class with the name 'config'.

cs.store(name="config", node=MySQLConfig)

@hydra.main(config_name="config")
def my_app(cfg: MySQLConfig) -> None:
# type (DictConfig) -> None
...

Thank you!!

Hi AgitatedDove14 , FriendlySquid61 ! I managed to grant permission to the AWS autoscaler to spin instances using the instance profile as suggested by FriendlySquid61 . The instances are created and terminated correclty, however the new instances don't executed the queued task and shutdown immediately. I noticed that the clearml credential at
self.web_server = Session.get_app_server_host()
self.api_server = Session.get_api_server_host()
` self.files_server = S...

Nice, I'll try also with the extra_bash_script, thank you!

It's working correctly, thank you!

FriendlySquid61 Your solution seems to have solved the problem. But only after I removed the export CLEARML_API_HOST={api_server}
export CLEARML_WEB_HOST={web_server}
export CLEARML_FILES_HOST={files_server}
command from the bash script executed when the instance is launched

I'm using Hydra 1.0.6

Hi TimelyPenguin76 , I tried your approach and it works, thank you! However it's a bit different to what I was trying to do: instead of cloning an existing task I'd like to specify the repository and a specific commit tag to use as it is done in Task.create. If this is possible with the API client it would be perfect

Hi AgitatedDove14 , I noticed that in the Hydra parameters section it is not possible to add as parameters keys string with dots: .(dot) $(dollar) and space are not allowed in parameter key. However, it's very useful to add parameters with the dot to change something in a sub-configuration as, for example, training.max_epochs=10 . Do you think it's possible to allow this?

I created this toy example so you don't need any external conf files. Btw if I first launch the task as python example.py port=80 than the task will print the message "Is this a webserver" correctly. If then in the UI I clone the same task, overrides the port with ['port=43'] , for example, and run the experiment, I will still get the message "Is this a webserver" so the port didn't change

AgitatedDove14 that seems like the best option. Once the aws autoscaler is inside a docker container I can deploy it inside a kube pod or a job. This, however, requires that I slightly modify the clearml helm chart with the aws-autoscaler deployment, right?

` from clearml import Task

from dataclasses import dataclass

import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import OmegaConf

@dataclass
class MySQLConfig:
host: str = "localhost"
port: int = 3306

@dataclass
class UserInterface:
title: str = "My app"
width: int = 1024
height: int = 768

@dataclass
class MyConfig:
db: MySQLConfig = MySQLConfig()
ui: UserInterface = UserInterface()

cs = ConfigStore.instance()
cs.store(name="config", n...

if in the "installed packages" I have all the packages installed from the requirements.txt than I guess I can clone it and use "installed packages"

Hi AgitatedDove14 , you can try with this toy example. If i run the task with python example.py ui.width=2048 the task will run correctly and print Title=My app, size=2048x768 pixels . However, in the UI I'm not allowed to change the ui.width in the Hydra parameters section: the 'Save' button is frozen