Thanks GrievingTurkey78
Sure just PR (should work with any Python/Hydra version):
kwargs['config']=config kwargs['task_function']=partial(PatchHydra._patched_task_function, task_function,) result = PatchHydra._original_run_job(*args, **kwargs)

Ohh sorry you will also need to fix the def _patched_task_function
The parameter order is important as the partial call relies on it.

Yey!

Hi ConfusedPig65
Any keras model will be automatically uploaded if you pass an upload url to the Task init:
task = Task.init('examples', 'keras upload test', output_uri=" ")(You can also pass to output_uri s3://buckket/folder or change the default output_uri in the clearml.conf file)
After this line any keras model will be automatically uploaded (you will see it under the Artifacts Tab)
Accessing models from executed tasks:
` trains_task = Task.get_task('task_uid_here')
last_check...

If you are using the latest RC:
pip install clearml==0.17.5rc5You can pass True it will use the "files_server" as configured in your clearml.conf
I used the http link as a filler to point to the files_server.
Make sense ?

So I have a task that just loads a model, but I don't see it as an artifact in the UI

You should see it under Artifacts, Input model if you are calling Keras load function (or similar)

It does not upload, the default behavior is to log the artifact (so you know where you stored, but not enforce unnecessary uploads)
If you were to change:
task = Task.init(project_name='examples', task_name='Keras with TensorBoard example')to:
task = Task.init(project_name='examples', task_name='Keras with TensorBoard example', output_uri=" ")It would also upload the model

Maybe the configuration file changed?
None
The logic is if the name and project are the same, and there are no artifacts/models, and the last time it was created was under 72 hours, reuse the Task

okay, let me know if it works

I think poetry should somehow return error if toml is "empty" then we can detect it...

SmallBluewhale13
And the Task.init registers 0.17.2 , even though it prints (while running the same code from the same venv) 0.17.2 ?

Let me check

I'm glad it worked out, thanks SmallBluewhale13 🙂

You can always log it manually:
from clearml import InputModel input_model = InputModel.import_model(weights_url='/tmp/keras_example/weight.6.hdf5')

ohh sorry, weights_url=path
Basically url can be the local path to the weights file 🙂

Looks good!

I pass my dataset as parameter of pipeline:

@<1523704757024198656:profile|MysteriousWalrus11> I think you were expecting the dataset_df dataframe to be automatically serialized and passed, is that correct ?
If you are using add_step, all arguments are simple types (i.e. str, int etc.)
If you want to pass complex types, your code should be able to upload it as an artifact and then you can pass the artifact url (or name) for the next step.

Another option is to use pipeline from dec...

so does the container install anything,

The way the agent works with dockers:
spin the docker Install the base stuff inside the docker (like git and make sure it has python etc) Create a new venv inside the docker, inheriting everything from the docker's system wide python packages, this means that if you have the "object_detection" package installed, it will be available inside the new venv. Install specific python package your Task requires (inside the venv). This allows you to over...

yes

argument saying always create from code

can be helpful

@<1523701523954012160:profile|ShallowCormorant89> any chance you can open a github issue on that, just so we do not forget ?

if we can edit the configuration objects of a pipeline, that can be beneficial too. which we're unable to do from UI

Actually you already can, after you clone the pipeline, you can press on details then go to configuration Tab, and edit the pipeline object. The format is HOCON (...

Here are my extra_docker_arguments that make the thing working:

GentleSwallow91 Nice!
BTW: in theory there should not need to be any need to add the specific: "-v","/home/nino/.ssh:/home/testuser/.ssh", the agent should do that automatically

Hi PanickyMoth78 , an RC is out with a fix.
pip install clearml==1.6.3rc0
Thank you for noticing the graph issue.
Btw do notice that since data is being changed inside the controller loop the parents are still kind of odd, because it is not clear to the logic the source of the data so it assumes it depends on the current state (i.e. all the leaves)

Hi ReassuredTiger98
Good point, since the user actually "running" the code is the agent, all the api calls are registered under its name, including the Model creation.
This is a good point, though ...
I know the enterprise tiers add "impersonate" as part of the security layer, meaning that the agent is Not actually running the code but the creating "user" is, which solve this problem. I'm not sure what actually can be done without this feature... thoughts?

you can run md5 on the file as stored in the remote storage (nfs or s3)

s3 is implementation specific (i.e. minio weka wassaby etc, might not support it) and I'm actually not sure regrading nfs (I mean you can run it, but it actually means you are reading the data, that said, nfs by definition I'm assuming is relatively fast access)
wdyt?

Hi CloudySwallow27

This error occurs randomly during training (in other words training does successfully start).

What's the cleamrl-agent version you are using, and the clearml version ?

2023-02-15 12:49:22,813 - clearml - WARNING - Could not retrieve remote configuration named 'SSH'

This is fine, it means it uses the default identity keys

The thing is - when I try to connect with normal SSH there are no issues

Now I'm lost, so when exactly do you see the issue ?

tried it and restarted the agent, but not working properly

What do you mean not working? can you provide logs ?

Do I set the 

CLEARML_FILES_HOST

 to the end point instead of an s3 bucket?

Yes you are right this is not straight forward:
CLEARML_FILES_HOST=" s3://minio_ip:9001 "
Notice you must specify "port" , this is how it knows this is not AWS. I would avoid using an IP and register the minio as a host on your local DNS / firewall. This way if you change the IP the links will not get broken 🙂

${PWD} works!

This will be resolved every call to Task.init (so I would recommend against it), how about "$HOME/" ?

Which means there will be atleast multiple published models entries of same model over time?

Only the specific one will be published (not all the Models the Task created)

👍