Reputation
Badges 1
383 × Eureka!just that the task itself is still Running state
Planning to exec into the container and run it in a loop and see what happens
How can a task running like this know its own project name?
It’s essentially this now:
from clearml import Task print(Task.current_task())
Yes, I have no experience with triton does it do lazy loading? Was wondering how it can handle 10s, 100s of models. If we load balance across a set of these engine containers with say 100 models and all of these models get traffic but distribution is not even, each of those engine container will load all those 100 models?
Got it. Never ran GPU workload in EKS before. Do you have any experience and things to watch out for?
Thanks AgitatedDove14 . Have removed Task.current_task() usage for this now. Think I can do without it
Ok it’s not implemented right, that’s what I was asking
Yeah meant the 1.0+ release as I don’t think the chart has been updated
Ok, but doesn't work for me though. Can you or AgitatedDove14 help me in linking to relevant code so that I can see what's wrong?
The Optimizer task is taking a lot of time to complete. Is it doing something here:
You mean the job with the exact same arguments ?
Yes
One thing I am looking at is nbdev from fastai folks
This worked well:
if project_name is None and Task.current_task() is not None: project_name = Task.current_task().get_project_name()
So packages have to be installed and not just be mentioned in requirements / imported?
I had given some feedback previously, hope they can be addressed 🤞
No we don’t need temp keys to work this way
Can I switch off git diff (change detection?)
In this case, particularly because of pickle protocol version between 3.7 and 3.8
Ah, just saw from the example that even that is doing the config pbtxt stuff - https://github.com/allegroai/clearml-serving/blob/main/examples/keras/keras_mnist.py#L51
For different workloads, I need to habe different cluster scaler rules and account for different gpu needs
Do let me know when task scheduler docs are up 🙂
I have my notebooks in a git repo and partially use nbdev for scripts