I also want to stress that these don't need to be happy-path interviews/results, although those are infinitely nicer to do 🙂 So I hear you with also noting what does not work as much as what did 👍

Hello.. I don't think so. Code of ethics can obviously vary from one job to another, and of course, so can legal compliance. You obviously have something very specific in mind, if you can expand on what you are looking for specifically, we maybe able to help.

usually though, genearlly speaking, a tools ethics and legality are set by the business side - not really something software would enforce on you. I hope I understand your question.

Hey Federico, since you are doing this from inside python, you could always call the 'get_parameters_as_dict' from the Task you have cloned, merge/update whichever ones you want to (or not), and then call ' set_parameters_as_dict ' .. I believe that should get you where you want to go 🙂

Evening Geoffrey, sorry for getting to this later in the day, I have been rather swamped today. All good though, all good.

What you raise is a good question. A very good question.

One of things that we have been thinking of around these parts is doing deep dives and interviews with users on how they came to ClearML, what setup they are using, key technologies and languages. In short, a sort of interview which will lead to a recipe book in the spirit of cooking (I would say CookBook but I th...

obviously 🙂

Howdy Jevgeni, that's .. strange. I am using google colab (free edition 🙂 and doing exactly the same as you, but I don't see any uncommited changes.. hrrm.. can you try this on colab maybe ? I am wondering if it's your jupyter notebook's version of python or some other notebook extension maybe

to be perfectly honest, I think I stopped investigating all the stuff plotly and friends can do these days.. I am sitting here with my mouth wide open.. some of their examples are awesome eye candy 😄

the brain surface viewer (more dash than anything) .. jst.. wow

can you show me the complete output from 'docker-compose ps' please ? 🙂

that's... a very good question. When I was using Feast, it was that more than one person was interested in using the ingested data, so it became that 'single source of truth'. From then on, ClearML was used to do the actual pipeline flow and training/testing/serving runs and, since it's all python shop, it worked pretty well. We used it offline, since we didn't care about online with having features at inference time. I should probably write up something about this when I have the time come t...

hello Emanuel 👋
I assume you are going to use python, in which case, inside each ClearML Task there is a method called get_reported_scalars that should have all the data I think.

you may want to read the warning at https://allegro.ai/clearml/docs/rst/references/clearml_python_ref/task_module/task_task.html#clearml.task.Task.get_reported_scalars on this.. and cache yourself as appropriate.. actually, the docs for the API are pretty thorough.. so if this isn't the exact itch you ne...

speaking as a google colab/jupyter notebooks person, I know we are missing some tutorials/docs there .. noted on the full blown example/testcase 👍

Ohhh... that makes sense.. use best of breed in areas where we don't overlap.

hhrrmm.. in the initial problem, you mentioned that the /var/lib/docker/overlay2 was growing large in size.. but.. 4GB seems "fine" for docker images.. I wonder .. does your nvme0n1p1 ever report like 85% or 90% used or do you think that the 4GB is a lot ? when you restart the server, does the % used noticeably drop ? that would suggest tmp files inside the docker image itself which.. is possible with docker (weird but, possible)

Hey there waves

Not sure about plans to automate this in the future, as this is more how docker behaves and not really clearml, especially with the overlay2 filesystem. The biggest offender usually is your json logfiles. have a look in /var/lib/docker/containers/ for *.log

assuming this IS the case, you can tell docker to only log upto a max-size .. I have mine set to 100m or some such

@<1687643893996195840:profile|RoundCat60> you set it once, inside the docker-compose itself.. it will affect all docker containers but, to be honest, docker tends to log everything

I believe you can set it on a 'per container' way as well.

hey @<1687643893996195840:profile|RoundCat60> .. did you ever get the problem sorted ?

you will probably want to find the culprit, so a find should work wonders. I probably suspect elasticsearch first. It tends to go nuts 😕

container_name:
  logging:
    options:
      max-size: 10m

adding this to the ever expanding list of "nice to have" features 👍

so yes indeedly ..

sudo find /var/lib/ -type d -exec du -s -x -h {} \; | grep G | more

seems to give saner results.. of course, in your case, you may also want to grep M for megabyte

I agree with Martin.B, it appears to be a CUDA mismatch. The version of torch is trying to use cuda 10.2 but you have
agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04that should probably be
agent.default_docker.image = nvidia/cuda:10.2-runtime-ubuntu18.04

❤ 👍

that's pretty darned awesome!! I didn't know we could do that 😄 😄

this obviously doesn't stop bad actors from assigning (say) an alice job over to bobs-queue but.. I am not sure of anyway to solve that.

Hey Slava, I don't mean to be "that guy" but, I am interested in what do you think a feature store means/implies/should do. The term is still (to my mind) very open to interpretation.. so I would honestly love to hear from you (and others)

The enterprise feature store we have should probably be more named as "data store but with advanced search/update capabilities" but.. that's not as nice sounding.

If you mean feature store as 'data ingestion via a DSL with type checking' then this is no...

loving it .. thanks for the suggestion.. I am trying to build up a roadmap, so thank you for the idea

that is one of the things I am working away on, even as we speak! If you have any items that you want to see sooner rather than later, please let me know 👍

honestly, I don't think the feature store we have would suit your needs. It is much closer to a data store in functionality with some nice to haves, rather than a feature store that is missing some bits.
Personally, I have used Feast before with a client, but only because it's a "pip install" to get it into place. It's a much lower barrier to entry than most of the others (again, bear in mind, I am a pythonista)