(when I say fancy, you are free to substitute whatever adjective you wish instead 🙂

aaahhh.. I will wager good money Sir that you are then using ipython in vscode which is probably trying to do something "fancy" with the interpreter

You may also want to brush up on the security and firewalls for AWS.. those always seem to be voodoo as far as I can tell 😄

you can cut steak with a spoon, but that doesn't mean it's a good idea 😉 😉

There is already a pre-built AMI fwiw 🙂

I know the storage can be swapped out to using S3 (obviously)

open to being totally wrong 🙂

👍 good luck!!!

Hey Leandro, I don't think it will work with replacing elasticserver with the AWS service type 😕

Shameless plug here ; https://clear.ml/blog/jupyter-notebooks-used-as-clearml-workers/

this whole area is a WIP of course, but I am trying to capture some of the really interesting Q and A from here so that they don't jst disappear into the void 🙂

you're not going to get the same performance as you would from your own dual xeon with 128gb of ram etc 🙂

obviously 🙂

Hello.. I don't think so. Code of ethics can obviously vary from one job to another, and of course, so can legal compliance. You obviously have something very specific in mind, if you can expand on what you are looking for specifically, we maybe able to help.

usually though, genearlly speaking, a tools ethics and legality are set by the business side - not really something software would enforce on you. I hope I understand your question.

the one you are in 🙂

Howdy Jevgeni, that's .. strange. I am using google colab (free edition 🙂 and doing exactly the same as you, but I don't see any uncommited changes.. hrrm.. can you try this on colab maybe ? I am wondering if it's your jupyter notebook's version of python or some other notebook extension maybe

@<1687643893996195840:profile|RoundCat60> you set it once, inside the docker-compose itself.. it will affect all docker containers but, to be honest, docker tends to log everything

Hey there waves

Not sure about plans to automate this in the future, as this is more how docker behaves and not really clearml, especially with the overlay2 filesystem. The biggest offender usually is your json logfiles. have a look in /var/lib/docker/containers/ for *.log

assuming this IS the case, you can tell docker to only log upto a max-size .. I have mine set to 100m or some such

I believe you can set it on a 'per container' way as well.

hey @<1687643893996195840:profile|RoundCat60> .. did you ever get the problem sorted ?

Howdy and Morning @<1687643893996195840:profile|RoundCat60> .. docker when using overlay2 doesn't have it's mount points show up in a 'df' btw, they will only appear in a 'df -a', mostly because since they are simply 'overlays', they don't (technically) consume any space (I mean, the files are still in the /var/lib but not for the space counting practices used by df)

this is why I was suggesting a find, maybe with a 'du' .. actually.. let me try that here.. 2s

hhrrmm.. in the initial problem, you mentioned that the /var/lib/docker/overlay2 was growing large in size.. but.. 4GB seems "fine" for docker images.. I wonder .. does your nvme0n1p1 ever report like 85% or 90% used or do you think that the 4GB is a lot ? when you restart the server, does the % used noticeably drop ? that would suggest tmp files inside the docker image itself which.. is possible with docker (weird but, possible)

so yes indeedly ..

sudo find /var/lib/ -type d -exec du -s -x -h {} \; | grep G | more

seems to give saner results.. of course, in your case, you may also want to grep M for megabyte

container_name:
  logging:
    options:
      max-size: 10m

clearml-deploy is clearml-serving but with other parts more intwined such as ci/cd prompts/callbacks, if you think clearml-deploy has a bit more love given to it, I believe that will put you on the right track, but at it's core, it's the same idea Sir.

the hyper datasets have always been there in the enterprise offering. It allows you to query datasets and perform functions such as updating labels on an image without an entire re-batching. I think we are trying to find a way to bring this to...

laughs okily dokily Sir.. noted and noted 👍
I must admit, I have been using plotly and matplotlib for years and.. I have never used animations once. I am old school though 🙂

the brain surface viewer (more dash than anything) .. jst.. wow

The way I read that is if you have exposed your clearml-server via k8s ingress then you can, from outside the k8s, say to clearml-session this is the k8s ingress/ip

you will probably want to find the culprit, so a find should work wonders. I probably suspect elasticsearch first. It tends to go nuts 😕

Hey Manoj, I am not sure how clearml-session would know how to setup kube-proxy, if that's your intent.

Personally, I would run the clearml-server and agents on the k8s cluster, and then expose the server endpoints via kube proxy or some other nicer ingress. Then you can run jupyter locally and you should be good. Jupyter session remotely running on k8s would be a logistical nightmare 🙂

I believe that's correct, yes