how can I start up the clearml agent using the clearml-agent image instead of SDK?

Not sure I follow, what do you mean instead of the SDK? and what is the "clearml-agent image" ?

Hi JumpyPig73

import data from old experiments into the dashboard.

what do you mean by "old experiments" ?

EnviousStarfish54 data versioning on the open source leverages the artifacts and storage and caching capabilities of Trains.
A simple workflow

1. Upload data
   https://github.com/allegroai/events/blob/master/odsc20-east/generic/dataset_artifact.py
2. Preprocessing data
   https://github.com/allegroai/events/blob/master/odsc20-east/generic/process_dataset.py
3. Using data
   https://github.com/allegroai/events/blob/master/odsc20-east/scikit-learn/sklearn_jupyter.ipynb

New RC hopefully solves it @<1643060801088524288:profile|HarebrainedOstrich43> could you check if it works for you now?

pip install clearml==1.14.0rc0

Also, I would upgrade the backend 0.15.1 a few bugs were fixed since 0.14.x some have to do with the plots...

LOl my pleasure - I guess we should have a link in the doc string of add_requirements to set_packages , I will tell the guys

I just cloned it from the examples that are available in the SaaS console upon account creation

Ohhh! that would explain it. Maybe it is broken there?! let me check a second

None
Change to:

CLEARML_AGENT_GIT_USER: ${CLEARML_AGENT_GIT_USER:my_git_user_here}

and the same for the password.
You can also just set the environment variables before launching docker-compose, whatever is more convenient for you

Hi UnsightlySeagull42

But now I need the hyperparameters in every python file.

You can always get the Task from anywhere?
main_task = Task.current_task()

For reporting the console logs you can use :
logger.report_text("my log line here", print_console=False)https://github.com/allegroai/clearml/blob/b4942321340563724bc16f60ea5dd78c9161778d/clearml/logger.py#L120

Still not supported 😞

Hi LackadaisicalOtter14

However, whenever we spin up a session, 

 always gets run and overwrites our configs

what do you mean by that?
The what config are being overwritten? (generally speaking, it just add the OS environment it needs to for the setup process)

As we use a custom CUDA image, we do not want this running on user login, and get ugly error messages about missing symlinks.

You can customize the startup bash script (running inside Any container) here:
https://github.com/allegroai/clearml-agent/blob/bf07b7f76d3236c1118b81730c6d9718705a795a/docs/clearml.conf#L145
LackadaisicalOtter14 Would that help?

Hi AgitatedTurtle16 could you verify you can access the API server with curl?

Any chance you can open a GitHub issue so we do not forget this feature ?

BTW: could it be the Task.init is Not called on the "module.name" entry point, but somewhere internally ?

Regarding the agent - No particular reason. Can you point me on how to do it?

This is a good place to start
https://clear.ml/docs/latest/docs/getting_started/mlops/mlops_first_steps

We need the automagic... 

This is one of the great benefits of using clearml 

😄
Sure, try this one:
Task.debug_simulate_remote_task('reused_task_id') task = Task.init(...)Notice it will take the arguments form the cleaml-task itself (e.g. override argparse arguments with what ...

Hi ChubbyLouse32
If I understand correctly you can relatively easy take a clearml Task and launch it on LSF, an integration would be something like:

` from clearml import Task
from clearml.backend_api.session.client import APIClient

while True:
result = client.queues.get_next_task(queue=q_id)
if not result or not result.entry:
sleep(5)
continue
task_id = result.entry.task

here is where we create the LSF job, this is just a pseudo code

os.system("lsf-launch-cmd 'clearml...

Yey!

clearml will register conda packages that cannot be installed if clearml-agent is configured to use pip. So although it is nice that a complete package list is tracked, it makes it cumbersome to rerun the experiment.

Yes mixing conda & pip is not supported by clearml (or conda or pip for that matter)
Even python package numbers might not exist on both.
We could add a flag not to update back the pip freeze, it's an easy feature to add. I'm just wondering on the exact use case

What's the trains version / trains-server version ?

hi ElegantCoyote26

but I can't see any documentation or examples about the updates done in version 1.0.0

So actually the docs are only for 1.0... https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving

Hi there, are there any plans to add better documentation/example

Yes, this is work in progress, the first Item on the list is custom model serving example (kind of like this one https://github.com/allegroai/clearml-serving/tree/main/examples/pipeline )

about...

It reverts back, but it cannot "delete" the last reported iteration value.
Make sense ?

🤞

What are you getting with:

curl http://<ip>:8008/auth.login

sdk.storage.cache.size.cleanup_margin_percent

Hi ReassuredTiger98
This is actually future proofing the cache mechanism and allowing it be "smarter" i.e. clean based on cache folder size instead of cache folder entries, this is currently not available

sdk.storage.cache

 parameters for the agent?

For both local execution and with an agent

When are datasets deleted if I run local execution?

When you hit the cache entry limit (100 if I recall). This can a...

I am trying to use the

configuration vault

option but it doesn't seem to apply the variables I am using.

Hi EmbarrassedSpider34 I think this is an enterprise feature...

Manged to make the credentials attached to the configuration when the task is spinned,

I'm assuming env variables ?

Ssh is used to access the actual container, all other communication is tunneled on top of it. What exactly is the reason to bind to 0.0.0.0 ? Maybe it could be a flag that you, but I'm not sure in what's the scenario and what are we solving, thoughts?

Hi ThoughtfulBadger56

If I clone and enqueue the cloned task on the webapp, does the clearml server execute the whole cmd above?

You mean agent will execute it? Do you have Task.init inside your code ?

Finally managed; you keep saying "all projects" but you meant the "All Experiments" project instead. That's a good start 

 Thanks!

Yes, my apologies you are correct: "all experiments"