I would think that a combination of kubernetes (I believe the preferred way to support multiple users at once, but open to being wrong) and individual queue's is probably the solution here.

for example; in kubernetes you could setup an agent to listen to bob-queue and another agent to listen to alice-queue. In the kubernetes dashboard you could assign a certain amount of cpu/memory and if using taints, gpu or not.

@<1687643893996195840:profile|RoundCat60> you set it once, inside the docker-compose itself.. it will affect all docker containers but, to be honest, docker tends to log everything

I may have to let someone with more know-how on the histogram and graphing answer if there is anyway to change graph layout though

So those datasets are the same but.. yes.. it's squashed/warped

one last tiny thing TrickySheep9 .. please do let us know how you get on, good or bad.. and if you bump into anything unexpected then please do scream and let us know 🙂

(when I say fancy, you are free to substitute whatever adjective you wish instead 🙂

honestly.. I think google are "fine" with it.. there are plenty of other (more egregious) abuses of their colab and they haven't screamed yet.

understand. Are you comfortable with docker ? If so, I would probably suggest doing a docker run -it <identifier> bash and seeing if that folder does, indeed, exist in the docker image

howdy Tim, I have tried to stay out of this, because a lot is going over my head (I am not a smart man 🙂 but, one thing I wanted to ask, are you doing the swapping in and out of code to do a/b testing with your models ?! Is this the reason for doing this ? Because if so, I would be vastly more inclined to try and think of a good way to do that. Again, this maybe wrong, I am trying to understand the use case for swapping in and out code. 🙂

I have a strange theory, that if the code is in classes, then you could include both in one .py file and then ENV["use_model"]="a" or ENV["use_model"]="b" to select between them .. in that way, you would clone the experiment and change the config and re-run

so yes indeedly ..

sudo find /var/lib/ -type d -exec du -s -x -h {} \; | grep G | more

seems to give saner results.. of course, in your case, you may also want to grep M for megabyte

there is a --docker flag for clearml-agent that will build containers :)

from the look of those two graphs, the underlying data is totally different 😕

can you show me the complete output from 'docker-compose ps' please ? 🙂

huh.. that example animation of automated driving on their homepage https://plotly.com/ is pretty awesome.

Hey Slava, I don't mean to be "that guy" but, I am interested in what do you think a feature store means/implies/should do. The term is still (to my mind) very open to interpretation.. so I would honestly love to hear from you (and others)

The enterprise feature store we have should probably be more named as "data store but with advanced search/update capabilities" but.. that's not as nice sounding.

If you mean feature store as 'data ingestion via a DSL with type checking' then this is no...

Hey there waves

Not sure about plans to automate this in the future, as this is more how docker behaves and not really clearml, especially with the overlay2 filesystem. The biggest offender usually is your json logfiles. have a look in /var/lib/docker/containers/ for *.log

assuming this IS the case, you can tell docker to only log upto a max-size .. I have mine set to 100m or some such

container_name:
  logging:
    options:
      max-size: 10m

Howdy and Morning @<1687643893996195840:profile|RoundCat60> .. docker when using overlay2 doesn't have it's mount points show up in a 'df' btw, they will only appear in a 'df -a', mostly because since they are simply 'overlays', they don't (technically) consume any space (I mean, the files are still in the /var/lib but not for the space counting practices used by df)

this is why I was suggesting a find, maybe with a 'du' .. actually.. let me try that here.. 2s

Never a problem Tim.. although it does prompt me to try and figure out a/b model testing myself ... I see everything as a "potential blog post" 😄 😄

Hey Paul, glad to see you in here but sorry that you're having issues. Jst a quick clarification if I can, you said 'set up using docker'.. do you mean you did a 'docker-compose up' or some other commands such as only starting specific parts/docker containers only ?

Hey Federico, since you are doing this from inside python, you could always call the 'get_parameters_as_dict' from the Task you have cloned, merge/update whichever ones you want to (or not), and then call ' set_parameters_as_dict ' .. I believe that should get you where you want to go 🙂

Shameless plug here ; https://clear.ml/blog/jupyter-notebooks-used-as-clearml-workers/

this whole area is a WIP of course, but I am trying to capture some of the really interesting Q and A from here so that they don't jst disappear into the void 🙂

obviously 🙂

but of course, this is all largely dependent on your code and structure etc

Stupid question Tim (and I understand that maybe your code is under NDA etc but) can you show the python code that you need to a/b against ?

you will probably want to find the culprit, so a find should work wonders. I probably suspect elasticsearch first. It tends to go nuts 😕

Sorry.. we are currently swamped with ODSC East requests and presentations etc 😄

clearml-deploy is clearml-serving but with other parts more intwined such as ci/cd prompts/callbacks, if you think clearml-deploy has a bit more love given to it, I believe that will put you on the right track, but at it's core, it's the same idea Sir.

the hyper datasets have always been there in the enterprise offering. It allows you to query datasets and perform functions such as updating labels on an image without an entire re-batching. I think we are trying to find a way to bring this to...

I assume that clearml's is on the right ?