nfs version 3

That's the thing, NFS will automatically set file access and flags based on the mount options you cannot change them post mount.
How about creating a new user just for the agent, it makes sense from security / credentials perspective

SubstantialElk6
Notice if you are using a manual setup the default is "secure: false" you have to change it to "secure: true":
https://github.com/allegroai/clearml-agent/blob/176b4a4cdec9c4303a946a82e22a579ae22c3355/docs/clearml.conf#L251

Hi CheekyFox58
If you are running the HPO+training on your own machine, it should work just fine in the Free tier

The HPO with the UI and everything, is designed to run the actual training on remote machines, and I think this makes it a Pro feature.

I am just about to move house, which is stressful enough without a global pandemic(!), so until that's completed I won't commit to anything.

Sure man 🙂 no rush, I appreciate the gesture regardless of the outcome
Many thanks!

Looking at the 

supervisor

 method of the base 

AutoScaler

 class, where are the worker IDs kept.
Is it in the class attribute 

queues

 ?

Actually the supervisor is passing a fixed prefix, then it asks the clearml-server on workers starting with this name.
This way we can have a fixed init script for all agents, while we still can differentiate them from the other agent instances in the system. Make sense ?

Thanks SubstantialElk6 !
Happy new year 🎉 🍺 🍾 🎇

Hopefully once things calm down at work I will find more time.

Sounds good 🙂

What we would like ideally, is a system where development, training, and deployment are almost one and the same thing, to reduce the lead time from development code to production models.

This is very aligned with the goals of ClearML 🙂
I would to understand more on what is currently missing in ClearML so we can better support this approach

my inexperience in using them a lot until recently. I can see how that is a better solution

I think I failed in explaining my self, I me...

What about output_uri?

If you are using StorageManager directly, output_uri is not relevant

Hi UnsightlySeagull42
Could you test with the latest RC
pip install clearml==1.0.4rc0Also could you provide some logs?

No worries 🙂

Hi SubstantialElk6
ClearML-Data doesn't actually "load" the data, it brings it locally and returns a folder with all your data files, from that point onward, it's up to your code to load it to the framework. Make sense ?

hmmm I see...
It seems to miss the fact that your process do uses the GPU.
Maybe it only happens later, that the GPU is used?
Does that make sense ?

available agent, i.e. not running anything else.
I mean how long would instance 1 wait until instance 2 of the experiment is up and running?
In other words what happens of all the nodes/agents are working and we still "need" additional instance.
This is basically like "pre-allocating" the nodes, only they wait in real-time until the additional node joins them.
Agent A pulls the 3 node Task, the Task clones itself (Task B) and enqueues on "very high priory queue" Task A wait until Task B is ru...

Ohh okay something seems to half work in terms of configuration, the agent has enough configuration to register itself, but fails to pass it to the task.
Can you test with the latest agent RC:
0.17.2rc4

Hi DeliciousBluewhale87
clearml-agent 0.17.2 was just release with the fix, let me know if it works

pip install clearml==0.17.5rc4

CooperativeFox72 yes 20 experiments in parallel means that you always have at least 20 connection coming from different machines, and then you have the UI adding on top of it. I'm assuming the sluggishness you feel are the requests being delayed.
You can configure the API server to have more process workers, you just need to make sure the machine has enough memory to support it.

GreasyPenguin14 thank you! that will make our life a lot easier 🙂

What do you mean cache files ? Cache is machine specific and is set in the clearml.conf file.
Artifacts / models are uploaded to the files server (or any other object storage solution)

I was hoping that there's a universal flag somewhere. Asking this because I want all the Models and Artifacts to be stored in one place and the users shouldn't have to edit their configuration files.

You mean like make sure all models/artifacts are always uploaded?

Hmm, so the way the configuration works is it loads the default configuration (equivalent to the example in the docs) then it adds the ~/clearml.conf on top. That means that you can tell your users to just copy paste the credentials from the UI into a template you make. How is that ?

Thanks MagnificentSeaurchin79 ! This code snippet is exactly what I needed, let me check if I can reproduce it.

Thanks!
I think this one will cover both case (the issue is with files on the root of the dataset)
if not (fnmatch(k, path) and fnmatch(k if '/' in k else '/{}'.format(k), '*/' + wildcard))}

You are doing great 🙂 don't worry about it

BattyLion34 Okay, I'll try to see if we can solve the multi-instance issue on Windows (because obviously it should be automatic)

Hi @<1716987933514272768:profile|SuccessfulPuppy43>

How to make remote ClearML agent do

pip install -e .

in theory there is no need to do that clearml-agent adds the repo root folder to the python path.
If you insist on actually installing it, try to add to your "installed packages" section a "requirement.txt" compatible line:

-e .

self.task.upload_artifact('trend_step', self.trend_step + 1)Out of curiosity why would every request generate an artifact ? Wouldn't it be better to report as part of the statistics ?
What would be the size / type of the matrix X (i.e. np.size / np.dtype) ?

Hi HealthyStarfish45

2. is there an advantage in using tensorboard over your reporting?

Not unless your code already uses TB or has some built in TB loggers.

html reporting looks powerfull, can one inject some javascript inside?

As long as the JS is self contained in the html script, anything goes :)

there was a problem with index order when converting from pytorch tensor to numpy array

HealthyStarfish45 I'm assuming you are sending numpy to report_image (which makes sense) if you want to debug it, you can also test tensorboard add_image or matplotlib imshow. both will send debug images