How often does Trains record the usage? I tried running a dummy programming which sleeps for 20 seconds and I don't see any plot.

Is this configurable?

@<1523701087100473344:profile|SuccessfulKoala55> Thx! I swear I saw it before, but somehow I just overlooked it.

I also get this from the logging

TRAINS Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring

Sorry for late reply AgitatedDove14

The code that init Task is put inside the first node. https://github.com/noklam/allegro_test/blob/6be26323c7d4f3d7e510e19601b34cde220beb90/src/allegro_test/pipelines/data_engineering/nodes.py#L51-L52

https://demoapp.demo.clear.ml/projects/6484f0ac373b4186a75aae68404aadd4/experiments/a7b5a19976864aa28e6ddfb589cbb6a5/output/log

repo: https://github.com/noklam/allegro_test
commit: https://github.com/noklam/allegro_test/commit/6be26323c7d4f3d7e51...

Could u give me some pointers where ClearML auto-capture log/stdout? I suspect as Kedro has configuration on logging and somehow ClearML fail to catch it.

This will cause a redundant Trains session, I guess.

EnviousStarfish54  quick update, regardless of the  logging.config.dictConfig   issue, I will make sure that even when the logger is removed, the clearml logging will continue to function  🙂
The commit will be synced after the weekend

Will the new fix avoid this issue and does it still requires the incremental flag?

From the logging documentation

Thus, when the incremental key of a configuration dict is present and is True,

the system will completely igno...

AgitatedDove14 Let me share the exact code and commit and entry point to you later. Thanks!

I mean, once I add environment variable, can trains.conf overwrite it? I am guessing environment variable will have a higher hierarchy.

The things that I want to achieve is:
Block user to access to public server If they configure trains.conf, then it's fine

import os os.environ["TRAINS_API_HOST"] = "YOUR API HOST " os.environ["TRAINS_WEB_HOST"] = "YOUR WEB HOST " os.environ["TRAINS_FILES_HOST"] = "YOUR FILES HOST "

is that possible?

Hi, just to be clear, self hosted option is still available right? I need to know this as we have spent some effort on integrating Trains internally and expect to continue the development for a while.

Wow! Just need this, I am surprised that I don't need to configure on server side

is it possible to overwrite if trains.conf did exist

good to know~

I don't want to mess with the standard setup.

I want a reliable way, so I don't want to hardcode to check if trains.conf exist in a certain path

Not sure why my elasticsearch & mongodb crashed. I have to remove and recreate all the dockers. Then clearml-agent works fine too

This is the only log I see.

Yes, i did use foreground.

I tested in a older "trains" server, it will show up log like this if no job is pick up. While my new "clearml-agent" shows nothing

No tasks in queue bb1bb1673f224fc98bbc8f86779be802
No tasks in Queues, sleeping for 5.0 seconds

I am running on Window 10 Machine, is this not compatible?

https://demoapp.demo.clear.ml/projects/7e29b23583fd46f1a85dee28a480eff2/experiments/7f3f58a33dd1476abb4582a6df3f28fe/output/log

I tried to step in the debugger, I can't quite see the clearml handlers in logging._handlers, the dict is empty, where is the clearml handler stored? AgitatedDove14

https://github.com/allegroai/clearml/commit/164fa357ed01704b11db67b8a7ac19791fbc49d1

This works. So it is still in master and should be included in 1.0.5?

I think it's related to the fix that use "incremental: true", this seems to fix 1 problem, but at the same time it will ignore all other handlers.

AgitatedDove14 Yes, as I found as Kedro's pipeline start running, the log will not be sent to the UI Console anymore. I tried both Task.init before/after the start of kedro pipeline and the result is the same. The log is missing, but the Kedro logger is print to sys.stdout in my local terminal.

Hi, sorry for really late update.
I finally get some more information about the logging issue. After diving in Kedro source code, I found there was a line that link to the log issue.

logging.config.dictConfig(conf_logging)This line will cause ClearML fail to pick up the log, any idea? AgitatedDove14 SuccessfulKoala55

A snippets to test is available, try commenting the line:6, the log will pop up, when line:6 is enable, event print() is not shown up.

` import logging
from clearml im...

Any commits/PR I can reference to the fix? Thanks.

FYR, I have shared the information to Kedro team.

https://github.com/quantumblacklabs/kedro/issues/792
It seems that they have their own justification too, maybe there are better config to make both work, I don't know yet.

Ok, will prepare a PR and script to reproduce the error

Ok, then maybe it can be still used as a data versioning solution. Except that I have to manually track the task id (those generate artifact) for versioning myself.

How do I confirm this?