Still not supported 😞

Hmm let me check something

Hi LackadaisicalOtter14

However, whenever we spin up a session, 

 always gets run and overwrites our configs

what do you mean by that?
The what config are being overwritten? (generally speaking, it just add the OS environment it needs to for the setup process)

As we use a custom CUDA image, we do not want this running on user login, and get ugly error messages about missing symlinks.

You can customize the startup bash script (running inside Any container) here:
https://github.com/allegroai/clearml-agent/blob/bf07b7f76d3236c1118b81730c6d9718705a795a/docs/clearml.conf#L145
LackadaisicalOtter14 Would that help?

Hi AgitatedTurtle16 could you verify you can access the API server with curl?

Any chance you can open a GitHub issue so we do not forget this feature ?

BTW: could it be the Task.init is Not called on the "module.name" entry point, but somewhere internally ?

Ohh now I get it...
Wait a couple of hours, 0.16 is out today with trains-agent --stop flag 🙂

Regarding the agent - No particular reason. Can you point me on how to do it?

This is a good place to start
https://clear.ml/docs/latest/docs/getting_started/mlops/mlops_first_steps

We need the automagic... 

This is one of the great benefits of using clearml 

😄
Sure, try this one:
Task.debug_simulate_remote_task('reused_task_id') task = Task.init(...)Notice it will take the arguments form the cleaml-task itself (e.g. override argparse arguments with what ...

Hi ChubbyLouse32
If I understand correctly you can relatively easy take a clearml Task and launch it on LSF, an integration would be something like:

` from clearml import Task
from clearml.backend_api.session.client import APIClient

while True:
result = client.queues.get_next_task(queue=q_id)
if not result or not result.entry:
sleep(5)
continue
task_id = result.entry.task

here is where we create the LSF job, this is just a pseudo code

os.system("lsf-launch-cmd 'clearml...

Yey!

clearml will register conda packages that cannot be installed if clearml-agent is configured to use pip. So although it is nice that a complete package list is tracked, it makes it cumbersome to rerun the experiment.

Yes mixing conda & pip is not supported by clearml (or conda or pip for that matter)
Even python package numbers might not exist on both.
We could add a flag not to update back the pip freeze, it's an easy feature to add. I'm just wondering on the exact use case

You can switch to docker-mode for better control over cuda drivers, or use conda and specify cudatoolkit (this feature will be part of the next RC, meanwhile it will install the cudatoolkit based on the global cuda_version).

What's the trains version / trains-server version ?

hi ElegantCoyote26

but I can't see any documentation or examples about the updates done in version 1.0.0

So actually the docs are only for 1.0... https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving

Hi there, are there any plans to add better documentation/example

Yes, this is work in progress, the first Item on the list is custom model serving example (kind of like this one https://github.com/allegroai/clearml-serving/tree/main/examples/pipeline )

about...

It reverts back, but it cannot "delete" the last reported iteration value.
Make sense ?

🤞

Hi @<1593051292383580160:profile|SoreSparrow36>
Of course fully integrated, here's a link to the docs https://clear.ml/docs/latest/docs/clearml_agent/clearml_agent_deployment/#slurm
The main advantage is the ability to launch and control jobs from outside the slurm cluster, from simple pipeline to logging the console outputs the performance and the ability to abort jobs directly from clearml as well as sto...

Hi AttractiveShrimp45
Well, I would use the Task.connect to add a section with any configuration your are using. for example
Task.current_task().connect(my_dict_with_conf_for_data, name="dataset51")wdyt?

looks like service-writing-time for me!

Nice!

persist/restore state so that tasks are restartable?

You mean if you write preemption-ready training code ?

What are you getting with:

curl http://<ip>:8008/auth.login

sdk.storage.cache.size.cleanup_margin_percent

Hi ReassuredTiger98
This is actually future proofing the cache mechanism and allowing it be "smarter" i.e. clean based on cache folder size instead of cache folder entries, this is currently not available

sdk.storage.cache

 parameters for the agent?

For both local execution and with an agent

When are datasets deleted if I run local execution?

When you hit the cache entry limit (100 if I recall). This can a...

1. Artifacts and models will be uploaded to the output URI, debug images are uploaded to the default file server. It can be changed via the Logger.
2. Hmm is this like a configuration file?
   You can do.
   local_text_file = task.connect_configuration('filenotingit.txt')
   Then open the 'local_text_file' it will create a local copy of the data in runtime, and the content will be stored on the Task itself.
3. This is how the agent installs the python packages, but if the docker already contactains th...

I am trying to use the

configuration vault

option but it doesn't seem to apply the variables I am using.

Hi EmbarrassedSpider34 I think this is an enterprise feature...

Manged to make the credentials attached to the configuration when the task is spinned,

I'm assuming env variables ?

Ssh is used to access the actual container, all other communication is tunneled on top of it. What exactly is the reason to bind to 0.0.0.0 ? Maybe it could be a flag that you, but I'm not sure in what's the scenario and what are we solving, thoughts?

Hi ThoughtfulBadger56

If I clone and enqueue the cloned task on the webapp, does the clearml server execute the whole cmd above?

You mean agent will execute it? Do you have Task.init inside your code ?

Finally managed; you keep saying "all projects" but you meant the "All Experiments" project instead. That's a good start 

 Thanks!

Yes, my apologies you are correct: "all experiments"

now i cant download neither of them

would be nice if address of the artifacts (state and zips) was assembled on the fly and not hardcoded into db.

The idea is this is fully federated, the server is not actually aware of it, so users can manage multiple storage locations in a transparent way.

if you have any tips how to fix it in the mongo db that would be great ....

Yes that should be similar, but the links would be in artifact property on the Tasks object
not exactly...

Hi UnsightlySeagull42

does anyone know how this works with git ssh credentials?

These will be taken from the host ~/.ssh folder

Thank you @<1523720500038078464:profile|MotionlessSeagull22> always great to hear 🙂
btw, if you feel like sharing your thoughts with us, consider filling our survey , it should not take more than 5min