Hi @<1634001100262608896:profile|LazyAlligator31>

Is this because the code repo is being recreated in this directory?

Yes this is correct 🙂
Basically the entire code base + venv is installed there, to make sure it does not intyerfere with the "system" preinstalled environment
(it also allows for caching on the host machine 🙂 )

Hi @<1631102016807768064:profile|ZanySealion18>

ClearML doesn't pick up model checkpoints automatically.

What's the framework you are using?
BTW:

Task.add_requirements("requirements.txt")

if you want to specify Just your requirements.txt, do not use add_requirements use:

Task.force_requirements_env_freeze(requirements_file="requirements.txt")

(add requirements with a filename does the same thing, but this is more readable)

Which version? is this reproducible in this example?
None
(can you try with the latest clearml version 1.13.2?)

ShortElephant92 yep, this is definitely enterprise feature 🙂
But you can configure user/pass on the open source, even store as hasedh the passwords if you need.

. And I saw that it upload the notebook it self as notebook. Does it is normal? There is a way to disable it?

Hi FriendlyElk26
Yes this is normal, it backups your notebook as well as converts it into python code (see "Execution - uncommitted changes" so that later the clearml-agent will be able to run it for you on remote machines.
You can also use task.connect({"param": "value") to expose arguments to use in the notebook so that later you will be able to change them from the U...

GentleSwallow91 notice that on the Task you have "Installed Packages" this is the equivalent of requirments.txt , you can edit it and add a missing package, or programatically add it in code (though usually directly imported packages are automatically registered, how come this one is missing?)

to add a package in code:
Task.add_requirements(package_name="my_package", package_version=">=1") task = Task.init(...)

base docker image but clearML has not determined it during the script ru...

Because by definition the Task already exists

AFAICS it's quite trivial implementation at the moment, and would otherwise require parsing the text file to find some references, right?

Yes, but the main issue is the parsing, it needs to have a specific standard. We use HOCON because it is great to read and edit (basically JSON would be a subset of HOCON)

the original pyhocon does support include statements as you mentioned -

Correct, my thinking was to expand them into "@configuration_section.key" or something of that nature

In our case, we have a custom YAML instruction

!include

, i.e.

Hmm interesting, in theory this might work since configuration encoding (when passing dicts), is handled with HOCON which does support referencing.
That said currently it is not aware of "remote configurations" only ENV variables and local files.
It will be cool to add, do we have a github issue on that? (would you like to see if you can PR such a thing?)

Hi FantasticPig28

or does every individual user have to configure their own minio credentials?

You can configure the clients files entry in the clearml.conf (or use an OS environment)
files_server: " "https://github.com/allegroai/clearml/blob/12fa7c92aaf8770d770c8ed05094e924b9099c16/docs/clearml.conf#L10
Notice to make sure you also provide credentials here:
https://github.com/allegroai/clearml/blob/12fa7c92aaf8770d770c8ed05094e924b9099c16/docs/clearml.conf#L97

https://stackoverflow.com/questions/47085458/why-is-multiprocessing-queue-get-so-slow

Ok the doc needs fix (edited)

suggestion?

Ephemeral Dataset, I like that! Is this like splitting a dataset for example, then training/testing, when done deleting. Making sure the entire pipeline is reproducible, but without storing the data long term?

Is task.parent something that could help?

Exactly 🙂 something like:
# my step is running here the_pipeline_task = Task.get_task(task_id=task.parent)

ComfortableShark77 are you saying you need "transformers" in the serving container?
CLEARML_EXTRA_PYTHON_PACKAGES: "transformers==x.y"https://github.com/allegroai/clearml-serving/blob/6005e238cac6f7fa7406d7276a5662791ccc6c55/docker/docker-compose.yml#L97

How can i get loaded model in Preporcess class in ClearML Serving?

ComfortableShark77
You mean your preprocess class needs a python package or is it your own module ?

AbruptHedgehog21 what exactly do you store as a Mode file ? is this a python object pickled ?

Hi WackyRabbit7
Yes, we definitely need to work on wording there ...
"Dynamic" means you register a pandas object that you are constantly logging into while training, think for example the image files you are feeding into the network. Then Trains will make sure it is constantly updated & uploaded so you have a way to later verify/compare different runs and detect dataset contemplation etc.
"Static" is just, this is my object/file upload and store it as an artifact for me ...
Make sense ?

Hi AdventurousRabbit79
Try:
"extra_clearml_conf" : "aws { s3 {key: A, secret : B, region: C, }} ",Generally speaking no need for the quotes on the secret/key
You also need the comma to separate between keys.
You can test if it is working by adding the same string to your local clearml.conf and importing the cleaml package

this results at the end of an experiment in an object to be saved under a given name regardless if it was dynamic or not?

Yes, at the end the name of the artifact is what it will be stored under (obviously if you reuse the name you basically overwrites the artifact)

Hi JitteryCoyote63 a few implementation details on the services-mode, because I'm not certain I understand the issue.
The docker-agent (running in services mode) will pick a Task from the services queue, then it will setup the docker for it spin it and make sure the Task starts running inside the docker (once it is running inside the docker you will see the service Task registered as additional node in the system, until the Task ends) once that happens the trains-agent will try to fetch the...

Hi NastyOtter17

"Project" is so ambiguous

LOL yes, this is something GCP/GS is using:
https://googleapis.dev/python/storage/latest/client.html#module-google.cloud.storage.client

Hi ProudMosquito87 trains-agent will automatically clone your code into the docker, no need to worry about it 🙂 make sure you configure the https://github.com/allegroai/trains-agent/blob/master/docs/trains.conf#L16 or the trains-agent machine contains the git ssh keys in the home folder of the user executing the trains-agent

StaleKangaroo85 check https://demoapp.trains.allegro.ai/projects/0e152d03acf94ae4bb1f3787e293a9f5/experiments/193ac2bced184c49a57658fceb4bd7f9/info-output/metrics/plots?columns=type&columns=name&columns=status&columns=project.name&columns=user.name&columns=started&columns=last_update&columns=last_iteration&order=last_update on the demo server, seems okay to me...

Hi DilapidatedDucks58 ,
I'm not aware of anything of this nature, but I'd like to get a bit more information so we could check it.
Could you send the web-server logs ? either from the docker or the browser itself.

Please hit Ctrl-F5 refresh the entire page, see if it is till empty....

Not intentional! When I launched the AMI it was running an older version

I think this is exactly the reason they decided to change the location 🙂 so you will have to manually upgrade, reasoning is we changed directory names (maybe a few more things)
Yes shutdown the current docker copse curl the new docker compose rename folder spin it up againFull instructions here:
https://allegro.ai/clearml/docs/docs/deploying_clearml/clearml_server_aws_ec2_ami.html#upgrading

however setting up the interpertier on pycharm is different on mac for some reason, and the video just didnt match what I see

MiniatureCrocodile39 Are you running on a remote machine (i.e. PyCharm + remote ssh) ?

Hi RipeGoose2 all PR's are welcome, feel free to submit :)

Ohh you mean ~/clearml.conf ?