Hi @<1523707653782507520:profile|MelancholyElk85> ! I left you a comment on the PR

Hi @<1545216070686609408:profile|EnthusiasticCow4> !

So you can inject new command line args that hydra will recognize.

This is true.

However, if you enable _allow_omegaconf_edit_: True, I think ClearML will "inject" the OmegaConf saved under the configuration object of the prior run, overwriting the overrides

This is also true.

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Could you please share some code that could help us reproduced the issue? I tried cloning, changing parameters and running a decorated pipeline but the whole process worked as expected for me.

Hi @<1558986821491232768:profile|FunnyAlligator17> ! There are a few things you should consider:

• Artifacts are not necessarily pickles. The objects you upload as artifacts can be serialized in a variety of ways. Our artifacts manager handles both serialization and deserialization. Because of this, you should not pickle the objects yourself, but specify artifact_object as being the object itself.
• To get the deserialized artifact, just call task.artifacts[name].get() (not get_local...

Hi DangerousDragonfly8 ! Sorry for the late reply. I'm taking a look and will come back to you shortly

I left another comment today. It’s about something raising an exception when creating a set from the file entries

Hi @<1570583237065969664:profile|AdorableCrocodile14> ! get_local_copy will always copy/download external files to a folder. To get the external files, there is property on the dataset called link_entries which returns a list of LinkEntry objects, which contain a link attribute, and each such link should point to a extrenal file (in this case, your local paths prefixed with file:// )

@<1675675705284759552:profile|NonsensicalAnt77> Can you try using None to setup the credentials? Maybe there is an issue parsing/finding the conf file

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! At the moment, we don't support default arguments that are typed via a class implemented in the same module as the function.
The way pipelines work is: we copy the code of the function steps (eventually their decorator's as well if declared in the same file), then we copy all the imports in the module. Problem is, we don't copy classes.
You could have your enum in a separate file, import it and it should work

@<1523701949617147904:profile|PricklyRaven28> Can you please try clearml==1.16.2rc0 ? We have released a fix that will hopefully solve your problem

how does your model look like?

@<1545216070686609408:profile|EnthusiasticCow4> I believe you are correct. Can you try another optimization method while we look into this?

Can you share a snippet?

DangerousDragonfly8 you can try to start the pipeline like this:
pipe.start(step_task_completed_callback=callback)where callback has the signature:
def callback(pipeline, node, parameters): print(pipeline, node, parameters)Note that even tho the parameter name is step_task_completed_callback , it is actually ran before the task is started. This is actually a bug...
We will need to review the callbacks, but I think you can work with this for now...

That makes sense, yeah it would be nice to have a way to exclude some files when calling sync_folder

Hi @<1657918706052763648:profile|SillyRobin38> ! If it is compatible with http/rest, you could try setting api.files_server to the endpoint or sdk.storage.default_output_uri in clearml.conf (depending on your use-case).

@<1675675705284759552:profile|NonsensicalAnt77> have you tried setting secure: true and host: storage.yandexcloud.net:443 ?

DangerousDragonfly8 I'm pretty sure you can use pre_execute_callback or post_execute_callback for this. you get the PipelineController in the callback and the Node . Then you can modify the next step/node. Note that you might need to access the Task object directly to change the execution_queue and docker_args . You can get it from node.job.task https://clear.ml/docs/latest/docs/references/sdk/automation_controller_pipelinecontroller#add_funct...

Sorry about that 😅

Hi @<1590514584836378624:profile|AmiableSeaturtle81> ! What function are you using to upload the data?

Also, do you need to close the task? It will close automatically when the program exits

UnevenDolphin73 looks like we clear all loggers when a task is closed, not just clearml ones. this is the problem

Hi RoundMole15 ! Are you able to see a model logged when you run this simple example?
` from clearml import Task
import torch.nn.functional as F
import torch.nn as nn
import torch
class TheModelClass(nn.Module):
def init(self):
super(TheModelClass, self).init()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, 120)
self.fc2 = nn.Linear(120, 84)
s...

I am honestly not sure if it will work, but we do have a http driver that could query your endpoint. None
It's worth to give it a try

There might be something wrong with the agent using ubuntu:22.04 . Anyway, good to know everything works fine now

@<1654294828365647872:profile|GorgeousShrimp11> Any change your queue is actually named megan-testing and not megan_testing ?

Hi @<1523701279472226304:profile|SoreHorse95> ! add_external_files will only stores the links. If the file changes and you don't have a dataset with updated links, I would expect that some caching mechanisms will break, resulting in some files to not be cached/not be downloaded again in the cache after getting the dataset.

Hi @<1780043419314294784:profile|LargeHamster21> ! Looks like you are using python3.11 (agent.default_python=3.11), while Pyro4 is incompatible with this python version: None
I would suggest trying to downgrade the python version or migrate to Pyro5

Maybe you want to use some other functions then the ones I quoted, so feel free to read the docs, you should be able to do this

can you share the logs of the controller?