the one on the right, for example, has no data points at around the 19 mark

I assume that clearml's is on the right ?

the part that I am concerned on is that the first pair of graphs you showed, the dataset (even from jst looking at it) are very different 😕

I may have to let someone with more know-how on the histogram and graphing answer if there is anyway to change graph layout though

Never a problem Tim.. although it does prompt me to try and figure out a/b model testing myself ... I see everything as a "potential blog post" 😄 😄

Stupid question Tim (and I understand that maybe your code is under NDA etc but) can you show the python code that you need to a/b against ?

howdy Tim, I have tried to stay out of this, because a lot is going over my head (I am not a smart man 🙂 but, one thing I wanted to ask, are you doing the swapping in and out of code to do a/b testing with your models ?! Is this the reason for doing this ? Because if so, I would be vastly more inclined to try and think of a good way to do that. Again, this maybe wrong, I am trying to understand the use case for swapping in and out code. 🙂

Howdy Jevgeni, that's .. strange. I am using google colab (free edition 🙂 and doing exactly the same as you, but I don't see any uncommited changes.. hrrm.. can you try this on colab maybe ? I am wondering if it's your jupyter notebook's version of python or some other notebook extension maybe

adding this to the ever expanding list of "nice to have" features 👍

I know the storage can be swapped out to using S3 (obviously)

I believe that's correct, yes

(when I say fancy, you are free to substitute whatever adjective you wish instead 🙂

I have a strange theory, that if the code is in classes, then you could include both in one .py file and then ENV["use_model"]="a" or ENV["use_model"]="b" to select between them .. in that way, you would clone the experiment and change the config and re-run

you will probably want to find the culprit, so a find should work wonders. I probably suspect elasticsearch first. It tends to go nuts 😕

Hey there waves

Not sure about plans to automate this in the future, as this is more how docker behaves and not really clearml, especially with the overlay2 filesystem. The biggest offender usually is your json logfiles. have a look in /var/lib/docker/containers/ for *.log

assuming this IS the case, you can tell docker to only log upto a max-size .. I have mine set to 100m or some such

hhrrmm.. in the initial problem, you mentioned that the /var/lib/docker/overlay2 was growing large in size.. but.. 4GB seems "fine" for docker images.. I wonder .. does your nvme0n1p1 ever report like 85% or 90% used or do you think that the 4GB is a lot ? when you restart the server, does the % used noticeably drop ? that would suggest tmp files inside the docker image itself which.. is possible with docker (weird but, possible)

The way I read that is if you have exposed your clearml-server via k8s ingress then you can, from outside the k8s, say to clearml-session this is the k8s ingress/ip

hey @<1687643893996195840:profile|RoundCat60> .. did you ever get the problem sorted ?

hello Emanuel 👋
I assume you are going to use python, in which case, inside each ClearML Task there is a method called get_reported_scalars that should have all the data I think.

you may want to read the warning at https://allegro.ai/clearml/docs/rst/references/clearml_python_ref/task_module/task_task.html#clearml.task.Task.get_reported_scalars on this.. and cache yourself as appropriate.. actually, the docs for the API are pretty thorough.. so if this isn't the exact itch you ne...

I also take it you were thinking of fargate for the clearml-agent's .. which would be awesome but.. oi.. not sure how much luck you would have there 🙂 🙂

huh.. that example animation of automated driving on their homepage https://plotly.com/ is pretty awesome.

I believe you can set it on a 'per container' way as well.

Hello.. I don't think so. Code of ethics can obviously vary from one job to another, and of course, so can legal compliance. You obviously have something very specific in mind, if you can expand on what you are looking for specifically, we maybe able to help.

usually though, genearlly speaking, a tools ethics and legality are set by the business side - not really something software would enforce on you. I hope I understand your question.

clearml-deploy is clearml-serving but with other parts more intwined such as ci/cd prompts/callbacks, if you think clearml-deploy has a bit more love given to it, I believe that will put you on the right track, but at it's core, it's the same idea Sir.

the hyper datasets have always been there in the enterprise offering. It allows you to query datasets and perform functions such as updating labels on an image without an entire re-batching. I think we are trying to find a way to bring this to...

but of course, this is all largely dependent on your code and structure etc

❤ 👍

container_name:
  logging:
    options:
      max-size: 10m

The takeaway from the pricing page, I think, is that clearml is free as in speech. If you want super duper support that may cost $ but the folks in the community here do an awesome job in the meantime.

I am guessing it could be but.. I don't feel that k8s is clearml-session's main focus/push

understand. Are you comfortable with docker ? If so, I would probably suggest doing a docker run -it <identifier> bash and seeing if that folder does, indeed, exist in the docker image