Also what do you have in the "Configuration" section of the serving inference Task?

I did change the

instead of 8080?

So this is the issue

Hi @<1523701523954012160:profile|ShallowCormorant89>
This means the system did not detect any "iteration" reporting (think scalars) and it needs a time-series axis for the monitoring, so it just uses seconds from start

I have no idea what string reference could be used when steps come from Task?

Oh I see, you are correct, when it comes to Tasks the assumption is your are passing strings (with selectors on the strings, i.e. the curly brackets) but there is no fancy serialization/deserialization as you have with pipelines from decorators / functions. The reason for that is that the Task itslef is a standalone, there is no way for the pipeline logic to actually "pull data" from it and "pass" it to the o...

where is the port? why https ?

that might be it.
Is the web UI working properly ?
What ports are you using?

can I mix steps with Task and Function?

Hmm interesting question, I think that in theory you should be able to, I have to admit that I have not tried yet, but it should work

None

Are these fields of ClearML Task?

correct

okay, just so I understand, this is what you have on your client that can connect with the server:
api { api_server: web_server: files_server: credentials {"access_key": "KEY", "secret_key": "SECRET"} }

@<1523707653782507520:profile|MelancholyElk85> what are you trying to change ? maybe there is a better way?
BTW: if you do step_base_task.export_task() you can use the parts that you need in the dict and pass them to:
task_overrides argument in add_step (you might need to flatten the nested arguments with '.' , and thinking about it, maybe we should do that automatically?!)

okay so the error should have been:

trains_agent: ERROR: Connection Error: it seems api_server is misconfigured. Is this the TRAINS API server http://<IP>:8008 ?

Not https nor 8010 ?!

Yes the clearml-server AMI - we want to be able to back it up and encrypt it on our account

I think the easiest and safest way for you is to actually have full control over the AMI, and recreate once from scratch.
Basically any ubuntu/centos + docker and docker-compose should do the trick, wdyt ?

Could you disable the windows anti-virus firewall and test?

pip install clearml==1.0.3rc1

Ahha! Nice work!

👏

now it has log, but only the initial one

So the subprocesses are not logged ?

How do I reproduce it ? (all the processes are on the same machine?)

And only the main one ?

https://hub.docker.com/layers/nvidia/cuda/10.1-cudnn7-runtime-ubuntu18.04/images/sha256-963696628c9a0d27e9e5c11c5a588698ea22eeaf138cc9bff5368c189ff79968?context=explore
the docker image is missing the cudnn which is a must for TF to work 🙂

Hmm so the Task.init should be called on the main process, this way the subprocess knows the Task is already created (you can call Task.init twice to get the task object). I wonder if we somehow can communicate between the sub processes without initializing in the main one...

This is a horrible setup, it means no authentication will pass, it will literally break every JWT authentication scheme

What do you have in the .netrc in the machine section

ElegantKangaroo44 it seems to work here?!
https://demoapp.trains.allegro.ai/projects/0e152d03acf94ae4bb1f3787e293a9f5/experiments/48907bb6e870479f8b230e6b564cd52e/output/metrics/plots

I think I found something relating to the issue on the subprocess not logging. Let me check if we can share something quickly

Wouldn't it make sense to use a single one ?

Why are you initializing 3 diff Tasks ?

Let me check that for a sec

Releasing an RC