Hi UnevenDolphin73

Does ClearML somehow

remove

any loggers from

logging

module? We suddenly noticed that we have some handlers missing when running in ClearML

I believe it adds a logger, it should not remove any loggers,
What's the clearml version you are using ?

Hi @<1541954607595393024:profile|BattyCrocodile47>

Can you help me make the case for ClearML pipelines/tasks vs Metaflow?

Based on my understanding

• Metaflow cannot have custom containers per step (at least I could not find where to push them)
• DAG only execution. I.e. you cannot have logic driven flows
• cannot connect git repositories to different component in the pipeline
• Visualization of results / artifacts is rather limited
• Only Kubernetes is supported as underlying prov...

Maybe there is setting in docker to move the space used in a different location?

No that I know of...

I can simply increase the storage of the first disk, no problem with that

probably the easiest 🙂

But as you described 

 it looks like an edge case, so I don’t mind 

🙂

MysteriousBee56 I would do Task.create()
you can get the full Task internal representation with task.data
Then call task._edit(script={'repo': ...}) to edit/update all the Task entries.
You can check the dull details of the task object here: https://github.com/allegroai/trains/blob/master/trains/backend_api/services/v2_8/tasks.py#L954
BTW: when you have a sample script working, consider PR-ing it, I'm sure it will be useful for others 🙂 (also a great way to get us involved with debuggin...

Nice 🙂
@<1523710674990010368:profile|GreasyPenguin14> for future reference the agent part in the clearml.conf is only created when you call clearml-agent init (no need for it for the python SDK). Full default configuration is here:
None

None
Change to:

CLEARML_AGENT_GIT_USER: ${CLEARML_AGENT_GIT_USER:my_git_user_here}

and the same for the password.
You can also just set the environment variables before launching docker-compose, whatever is more convenient for you

Make sense 🙂
Just make sure you configure the git user/pass in the docker-compose so the agent has your credentials for the repo clone.

What's up @<1523708288686886912:profile|DeliciousBluewhale87>

@<1523710674990010368:profile|GreasyPenguin14> make sure it to uses https not ssh:
edit ~/clearml.conf

force_git_ssh_protocol: false

and that you have both git_user & git_pass set in your clearml.conf

This is exactly what I did here, and it is working 😞
https://demoapp.demo.clear.ml/projects/0e919ea1cc5c499b99e1ab85004b6e97/experiments/887edef09d4549e88b829a34c87d4d5b/output/execution

@<1523710674990010368:profile|GreasyPenguin14> If I understand correctly you can use tokens as user/pass (it's basically the same interface from the git client perspective, meaning from ClearML

git_user = gitlab-ci-token
git_pass = <the_actual_toke>

WDYT?

. So to conclude: it has to be executed manually first, then with trains agent?

Yes, that said, as you mentioned, you can always edit the "installed packages" once manually, from that point you are basically cloning the experiment, including the "installed packages" so it should work if the original worked.
Make sense ?

I think we added it somewhere in 0.14, anyhow I just checked the Logger doc, it is there now 🙂

Okay how do I reproduce it ?

Hi @<1523722267119325184:profile|PunySquid88> I guess it's a good thing we talk, because I believe that what you are looking for is already available :)

Logger.current_logger().report_media('title', 'series', iteration=1337, local_path='/tmp/bunny.mp4')

This will actually work on any file, that said, the UI might display the wrong icon (which will be fixed in the next version).

We usually think of artifacts as data you want to reuse, so all the files uploaded there are accessibl...

👍

Hi @<1547028031053238272:profile|MassiveGoldfish6>
hmm yeah you need to remove the "hidden" system_tag from the project

from clearml.backend_api.session.client import APIClient

c = APIClient()

print(c.projects.get_by_id("PROJECT_ID_HERE").to_dict())

c.projects.update(project="PROJECT_ID_HERE", system_tags=["test"])

print(c.projects.get_by_id("PROJECT_ID_HERE").to_dict())

Notice you can get the project ID from the URL
`/projects/1974af8ccdac454b836c47349c4e826e/experiments/84...

Apparently the error comes when I try to access from

get_model_and_features

the pipeline component

load_model

. If it is not set as pipeline component and only as helper function (provided it is declared before the components that calls it (I already understood that and fixed, different from the code I sent above).

ShallowGoldfish8 so now I'm a bit confused, are you saying that now it works as expected ?

Good news a dedicated class for exactly that will be out in a few days 🙂
Basically task scheduler and task trigger scheduler, running as a service cloning/launching tasks either based on time (cron alike) or based on a trigger).
wdyt?

That is awesome!
If you feel like writing a bit about the use-case and how you solved it, I think AnxiousSeal95 will be more than happy to publish something like that 🙂

I think EmbarrassedSpider34 is correct.
When you pass the requirements to clearml-task, actually the agent depending on how it was configured (conda / pip) will do the installation.
That said, maybe it is worth adding support to provide the env.yml in the CLI ?
(Notice that adding specific channels needs to be configured on the agent, they are not stored per Task)
AlertCamel57 wdyt?

How does this work in the context of a pipeline?

Is your pipeline from functions / decorators ? or is it from Tasks ?
(if this is Tasks then just changing the entry point in the overides)
In case of functions or decorators, you have to do that manually (i.e. your function needs to do "accelerate launch"

from accelerate.commands.launch import launch_command, launch_command_parser
parser = launch_command_parser()
args = parser.parse_args("-command -here".split())
launch_command(arg...

Interesting, i wasn't aware of this python module for executing accelerate. I'll try to use that.

It's essentially the cmd line:
None
None

We used subprocess for it, ...

Popen? os.system? fork?

SubstantialElk6 on the client side?

If nothing specific comes to mind i can try to create some reproducible demo code (after holiday vacation)

Yes please! 🙏
In the mean time see if the workaround is a valid one

I still don't get resource logging when I run in an agent.

@<1533620191232004096:profile|NuttyLobster9> there should be no difference ... are we still talking about <30 sec? or a sleep test? (no resource logging at all?)

have a separate task that is logging metrics with tensorboard. When running locally, I see the metrics appear in the "scalars" tab in ClearML, but when running in an agent, nothing. Any suggestions on where to look?

This is odd and somewhat consistent with actu...

with tensorboard logging, it works fine when running from my machine, but not when running remotely in an agent.

This is odd, could you send the full Task log?

And after having called

Task.init()

the second time, the automatic logging of resources and tensorboard plots works as well. I would recommend adding explanation to the docs for

Oh yeah! you always need to call Task.init first, Task,current_task should be called from anywhere you like but after the Task.init was called.

Hi ShortElephant92
No, this is opt-in, so other then checking for updates once in a while, no traffic at all