So obviously the straight forward solution is to report normalize the step value when reporting to TB, i.e. int(step/batch_size). Which makes sense as I suppose the batch size is known and is part of the hyper-parameters. Normalization itself can be done when comparing experiments in the UI, and in the backend can do that, if given the correct normalization parameter. I think this feature request should actually be posted on GitHub, as it is not as simple as one might think (the UI needs to a...

Ohh then use the AWS autoscaler, basically it what you want, spin an EC2 and set an agent there, then if the EC2 goes down (for example if this is a spot), it will spin it up again automatically with the running Task on it.
wdyt?

Hi, I would like to understand how I can set the pip cache location for my agent,

ClumsyElephant70 by default the pip cache (and all other cache folders) are mounted back into the host itself ~/.clearml/
I'm assuming the idea is shared cache, if this is the case, do:
docker_pip_cache = ~/my_shared_nfs/pip-cachehttps://github.com/allegroai/clearml-agent/blob/e3e6a1dda81bee2dd20a64d09746568e415f1823/docs/clearml.conf#L139

FlatOctopus65

In my local environment

pipeline_package

is installed in development mode

In order to install the package you need to specify the git repo of the package, this is how the pipeline would know where to bring it from.
Either install it locally with "pip install git+ https://github.com/ ...." or add tp the packages argument of the Pipeline wrapper packages = ["git+ https://github.com/ "] `
wdyt?

Task status change to "completed" is set after all artifacts upload is completed.
JitteryCoyote63 that seems like the correct behavior for your scenario

I mean test with:
pipe.start_locally(run_pipeline_steps_locally=False)This actually creates the steps as Tasks and launches them on remote machines

In terms of creating dynamic pipelines and cyclic graphs, the decorator approach seems the most powerful to me.

Yes that is correct, the decorator approach is the most powerful one, I agree.

E.g. I might need to have different N-numbers for the local and remote (ClearML) storage.

Hmm yes, that makes sense

That'd be a great solution, thanks! I'll create a PR shortly

Thank you! 🙏 🤩

I am writing quite a bit of documentation on the topic of pipelines. I am happy to share the article here, once my questions are answered and we can make a pull request for the official documentation out of it.

Amazing please share once done, I will make sure we merge it into the docs!

Does this mean that within component or add_function_step I cannot use any code of my current directories code base, only code from external packages that are imported - unless I add my code with ...

I think it's supposed to be out early Nov 🙂

Hi @<1569858449813016576:profile|JumpyRaven4>

task.add_requirements()

This is the problem, if you look closely this is a class method, meant for helping the Task.init better capture python packages, it does Not change the task requirements.
To do that, use " task.set_packages "

Ohh then you do docker sibling:
Basically you map the docker socket into the agent's docker , that lets the agent launch another docker on the host machine.
You cab see an example here:
https://github.com/allegroai/clearml-server/blob/6434f1028e6e7fd2479b22fe553f7bca3f8a716f/docker/docker-compose.yml#L144

I have to specify the full uri path ?

No it should be something like " s3://bucket "

the model files management is not fully managed like for the datasets ?

They are 🙂

Hi SmugDog62

My guess is that there's an issue with the git repo detector.

Seems like you are correct
Can are you getting on the execution tab?
Is the repo correct?
Do you see the notebook in the uncommited changes ?

ohh, could it be a 32bit version of python ?

Hi ArrogantBlackbird16

but it returns a task handle even after the Task has been closed.

It should not ... That is a good point!
Let's fix that 🙂

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-create-https-ssl-load-balancer.html
🙂

PunyBee36 to get https add an aws elb before the server , the elb will add the https to any outside connection

Hmm, as a quick solution you can use the custom example and load everything manually:
https://github.com/allegroai/clearml-serving/blob/219fa308df2b12732d6fe2c73eea31b72171b342/examples/custom/preprocess.py
But you have a very good point, I'm not sure how one could know what's the xgboost correct class, do you?

👍

it worked!!!!

YEY!

I pass the IDs to the docker container as environment variables, so this does need restart for the docker container but I guess we can live with that for now

So this would help you decide on which actual Model file to download ? (trying to understand how the argument is being used, meaning should we have it stored somewhere? there is meta-data on the Model itself so we can use that to store the data)

Hi @<1529633468214939648:profile|CostlyElephant1>

Is it possible to get user ID of the current user

On the Task.data object itself there should be a filed named " user " that's the user ID of the owner (creator) of the Task.
You can filter based on this id with

Tasks.get_tasks(..., task_filter={'user': ["user-id-here"]})

wdyt?

I think it is on the JWT token the session gets from the server
a bit of a hack but should work 🙂

session = task.session # or Task._get_default_session() 
my_user_id = session.get_decoded_token(session.token)['identity']['user']

LudicrousDeer3 when using Logger you can provide 'iteration' argument, is this what you are looking for?

Nice!!!

EmbarrassedSpider34 I can update that an RC should be out later today with a fix 🙂

Hi LittleShrimp86

just to login into your clearml app (demo or server) so I can run python files related to clearml.

I think this amounts to creating a Task and enqueueing it, am I understanding correctly ?

Hi DrabCockroach54

... and no logs for python script.

what do you mean by "no logs" , is it clearml logs? or k8s pod logs ?

WackyRabbit7 you can configure AWS autoscaler with two types of instances , with priority to one of them. So in theory you do not need two autoscaler processes, with that in mind I "think" single IAM should suffice

In order to facilitate the multiple credentials one must use the Clearml SDK obviously.

Yes 🙂