Seems the apiserver is out of connections, this is odd...
SuccessfulKoala55 do you have an idea ?

Yeah that makes sense, I mean it will probably be a bit more than that per month when it's up but half when it's down (just fyi, when AWS instances are down you still pay for the EBS storage).
If you are trying o save a buck here, double check on that otherwise you will end at the same cost level but after spending resource on migrating.
If you want a good hack you can always download the data and then just store it locally (i.e. half the migration job) and just reduce the number of users whe...

Hi SourSwallow36
What do you man by Log each experiment separately ? How would you differentiate between them?

FileNotFoundError: [Errno 2] No such file or directory: 'tritonserver': 'tritonserver'

This is oddd.
Can you retry with the latest from the github ?
pip install git+

I want to build a real time data streaming anomaly detection service with clearml-serving

Oh, so the way it currently works clearml-serving will push the data in real-time into Prometheus (you can control the stats/input/out), then you can build the anomaly detection in grafana (for example alerts on histograms over time is out-of-the-box, and clearml creates the histograms overtime).
Would you also need access to the stats data in Prometheus ? or are you saying you need to process it ...

Please attach the log 🙂

In theory it should have worked.
Can you send me the full Task log? (with cache and everything?)
I suspect since these are not the default folders, something is misconfigured / missing
(you can DM the log, so it won't end on a public the channel))

That makes total sense, this is exactly an OS scenario for signal 9 🙂

Ohh I see.
In your web app, look for the "?" icon (bottom left corner), click on it, it should open the full platform documentation

Hi JealousParrot68
no need for decorators, you can just pass the function to schedule_function=<function goes here> 🙂
See scheduler here
https://github.com/allegroai/clearml/blob/8708967a5ef4d8529a1a5ea417672e3ebbb258d7/clearml/automation/scheduler.py#L485
And triggers here:
https://github.com/allegroai/clearml/blob/8708967a5ef4d8529a1a5ea417672e3ebbb258d7/clearml/automation/trigger.py#L193
https://github.com/allegroai/clearml/blob/8708967a5ef4d8529a1a5ea417672e3ebbb258d7/clea...

VexedCat68 actually a few users already suggested we auto log the dataset ID used as an additional configuration section, wdyt?

yes, looks like. Is it possible?

Sounds odd...
Whats the exact project/task name?
And what is the output_uri?

(Do notice that even though you can spin two agents on the same GPU, the nvidia drivers cannot share allocated GPU memory, so if one Task consumes too much memory the other will not have enough free GPU memory to run)

Basically the same restriction as manually launching two processes using the same GPU

When you install using pip <filename> you should end up with something like:
minerva @ file://... or minerva @ https://...

Hi VexedCat68
Are we talking youtubes ? docs? courses ?

Hi LazyTurkey38
Configuring these folders will be pushed later today 🙂
Basically you'll have in your clearml.conf
` agent {
docker_internal_mounts {
sdk_cache: "/clearml_agent_cache"
apt_cache: "/var/cache/apt/archives"
ssh_folder: "/root/.ssh"
pip_cache: "/root/.cache/pip"
poetry_cache: "/root/.cache/pypoetry"
vcs_cache: "/root/.clearml/vcs-cache"
venv_build: "/root/.clearml/venvs-builds"
pip_download: "/root/.clearml/p...

another option is the download fails (i.e. missing credentials on the client side, i.e. clearml.conf)

Any chance you can share the Log?
(feel free to DM it so it will not end up public)

Interesting...
We could followup the .env configuration, and allow the clearml-task to add configuration files from cmd line. This will be relatively easy to add. We could expand the Environment support (that somewhat exists), and add the ability to read variables from .emv and Add them to an "hyperparemeter" section, named Environment. wdyt?

Hmm you mean how long it takes for the server to timeout on registered worker? I'm not sure this is easily configured

ReassuredTiger98 both are running with pip as package manager, I thought you mentioned conda as package manager, no?
agent.package_manager.type = pipAlso the failed execution is looking for "ruamel_yaml_conda" but it is nowhere to be found on the original one?! how is that possible ?

Here is a nice hack for you:
Task.add_requirements( package_name='carla', package_version="> 0 ; python_version < '2.7' # this hack disables the pip install")This will essentially make sure the agent will skip the installation of the package, but at least you will know it is there.

Hi SmallDeer34
Can you try with the latest RC , I think we fixed something with the jupyter/colab/vscode support
!pip install clearml==1.0.3rc1

How did you define the decorator of "train_image_classifier_component" ?
Did you define:
@PipelineDecorator.component(return_values=['run_model_path', 'run_tb_path'], ...Notice two return values

SoreDragonfly16 notice that if in the web UI you aborting a task it will do exactly what you described, print a message and quit the process. Any chance someone did that?

Yes, the left side is the location of the file on the host machine, the right side is the location of the file inside the docker. in our case it is the same location

Everything seems correct...
Let's try to set it manually.
create a file ~/trains.conf , then copy paste the credentials section from the UI, it should look something like:
api { web_server: http:127.0.0.1:8080 api_server: http:127.0.0.1:8008 files_server: http:127.0.0.1:8081 credentials { "access_key" = "access" "secret_key" = "secret" } }Let's see if that works

SarcasticSparrow10 LOL there is a hack around it 🙂
Run your code with python -O
Which basically skips over all assertion checks

So it should cache the venvs right?

Correct,

path: /clearml-cache/venvs-cache

Just making sure, this is the path to the host cache folder

ClumsyElephant70 I think I lost track of the current issue 😞 what's exactly not being cached (or working)?

Can you send the full log as attachment?