SmugOx94 Yes, we just introduced it 🙂 with 0.16.3
Discussion was here (I'll make sure to update the issue that the version is out)
https://github.com/allegroai/trains/issues/222
In your trains.conf add the following line:
sdk.development.store_code_diff_from_remote = trueIt will store the diff from the remote HEAD instead of the local one.

https://app.clear.ml 🙂

I would expect that after calling Task.enqueue(exit=True), the local task is closed and no processes related to it is running

Ohh my apologies, I did not understand that.
Are you saying that locally you call task.remote_execute(exit_process=True) and it does not leave the local process ?

Hi TrickyFox41

is there a way to cache the docker containers used by the agents

You mean for the apt get install part? or the venv?
(the apt packages themselves are cached on the host machine)
for the venv I would recommend turning on cache here:
https://github.com/allegroai/clearml-agent/blob/76c533a2e8e8e3403bfd25c94ba8000ae98857c1/docs/clearml.conf#L131

JitteryCoyote63 I think that with 0.17.2 we stopped mounting the venv build to the host machine. Which means it is all stored inside the docker.

Not yet 😞
It should not be complex to implement,
The actual aws auto scaler class is implementing just two functions:

def spin_up_worker(self, resource, worker_id_prefix, queue_name):
https://github.com/allegroai/clearml/blob/e9f8fc949db7f82b6a6f1c1ca64f94347196f4c0/clearml/automation/auto_scaler.py#L104

def spin_down_worker(self, instance_id):
https://github.com/allegroai/clearml/blob/e9f8fc949db7f82b6a6f1c1ca64f94347196f4c0/clearml/automation/auto_scaler.py#L...

it worked!!!!

YEY!

I pass the IDs to the docker container as environment variables, so this does need restart for the docker container but I guess we can live with that for now

So this would help you decide on which actual Model file to download ? (trying to understand how the argument is being used, meaning should we have it stored somewhere? there is meta-data on the Model itself so we can use that to store the data)

Yes

JitteryCoyote63 There is a basic elastic license that should always be there. If for some reason it was deleted/expired then the following command should fix it:
curl -XPOST ' http://localhost:9200/_xpack/license/start_basic '

Correct 🙂

2 and 3 - I want to manage access control over the RestAPI

Long story short, put a load-balancer in front of the entire thing (see the k8s setup), and have the load-balancer verify JWT token as authentication (this is usually the easiest)

1- Exactly, custom code

Yes, we need to add a custom example there (somehow forgotten)
Could you open an Issue for that?
in the meantime:
` #

Preprocess class Must be named "Preprocess"

No need to inherit or to implement all methods

lass P...

It can also work by running on multiple known nodes.

Horovod sits on top of openmpi that needs ssh to open multiple nodes, I'm not sure how one would connect it without passing the SSH keys from one node to the other, and making sure they can directly communicate. (Not saying it is not possible, but just a few things to configure before it works, the enterprise edition remove the need for the direct SSH connection between the nodes)

How would i add a glue for multinode?

Basic...

I love the new docs layout!

Thank you and thank docusaurus, they rock!

RoughTiger69

Apparently,

, doesn’t populate that dict with

any keys that don’t already exist in it

.

Are you saying new entries are not added to the Dict even if they are on the Task (i.e. only entries that already exist on the dict are populated ?
But you already have all the entries defined here:
https://github.com/allegroai/clearml/blob/721569bb77d89d89e5b4f32a0ed98311c4574650/examples/services/aws-autoscaler/aws_autoscaler.py#L22

Since all this is ha...

Would love to just cap it at a fixed amount for a month for API calls.

Try the timeout configuration, I think this shoud solve all your issues, and will be fairly easy to set for everyone

Hmm I would recommend passing it as an artifact, or returning it's value from the decorated pipeline function. Wdyt?

Hi ObnoxiousStork61

but unfortunately I can't fetch them from my local computer,
is this intended?

By default ClearML will only log the wights files.
It can also automatically upload them, if you pass a destination for storage at Task.init/
For example, to store on the files server:
Task.init(..., output_uri=True)To store on S3 (sub folders will be created automatically based on the Task ID
Task.init(..., output_uri=' ')

Oh right, I missed the fact the helper functions are also decorated, yes it makes sense we add the tags as well.
Regarding nested pipelines, I think my main question is , are they independent or are we generating everything from the same code base?

Hi @<1730396272990359552:profile|CluelessMouse37>

However, the caching doesn't seem to be working correctly. Despite not changing the configuration, the first step runs every time.

How are you creating the cached component?
is this a standalone script or a git repo link?

These parameters are dictionaries of specific configurations (dict of dict) that are the same but might not be taken into account properly by the caching mechanism.

hmm for the component to be cached (or reuse...

Thank you EnviousStarfish54 !
This is very helpful!
I'm looking at Kedro and the project you shared, and a few thoughts came to mind:
I very much like the idea of using functions as "nodes" (and to extend, using notebook cells with tags as nodes). This got me thinking, I'm pretty sure we could have a similar imlmentation with ClearML. My thinking is using inspect or dill to convert the functions/cells into plain text code, automatically analyze the runtime requirements, and creat...

Checkout the trains-agent repo https://github.com/allegroai/trains-agent
It is fairly straight forward.

PompousParrot44 I think the website should address that:
https://allegro.ai/
But the TD;DR is the enterprise version adds Full Dataset Versioning on top, with end-to-end integration from code to DLOps (e.g.. data sampling , database query capabilities, data visualization, multi-site support, permission etc,)

Instead you can do: TRAINS_WORKER_NAME = "trains-agent":$DYNAMIC_INSTANCE_ID
Then the Worker ID will running instance appended to the worker name. This means that even if you use the same $DYNAMIC_INSTANCE_ID twice, you will not have two agent registering on the same name.

Hi JitteryCoyote63
I think that what happens is that the agent are registered on the same name (id). How many agent do you see in the "Workers" tab?

Hi JitteryCoyote63

cleanup_service task in the DevOps project: Does it assume that the agent in services mode is in the trains-server machine?

It assumes you have an agent connected to the "services" queue 🙂
That said, it also tries to delete the tasks artifacts/models etc, you can see it here:
https://github.com/allegroai/trains/blob/c234837ce2f0f815d3251cde7917ab733b79d223/examples/services/cleanup/cleanup_service.py#L89
The default configuration will assume you are running i...

Thanks!

VexedCat68 make sense, we could also (if implementing this feature) add a special Tag to the dataset , so you know it contains "external" links, wdyt?

Yes, consider VexedCat68 txt file the Dataset "content" , this will enable ypu to safely get the list of files, and then you can use the StorageManager to download them extend this concept and have it built into the Dataset itself, i.e. allow you to add files as links and make sure it will just download them. The caveat here is that the Dataset at the end, returns a folder with the files, when you specify links, you have to also specify the target location locally (at the end you want a fol...

Hi SkinnyPanda43

I realized that the params are not being saved anymore

Could you test with clearml==1.0.4 ?

Hi VexedCat68

One of my steps just finds the latest model to use. I want the task to output the id, and the next step to use it. How would I go about doing this?

When you say "I want the task to output the id" do you mean to pass t to the next step:
Something like this one:
https://github.com/allegroai/clearml/blob/c226a748066daa3c62eddc6e378fa6f5bae879a1/clearml/automation/controller.py#L224