Hi @<1727497172041076736:profile|TightSheep99>
Yes it can, it will upload the meta-data as well as the files (it will also do de-dup and will not upload files that already exist in the dataset based on the hash of teh file content)

Hi RoundSeahorse20
Try the following , let me know if it worked.
clear_logger = logging.getLogger('clearml.metrics') clear_logger.setLevel(logging.ERROR)

Hi ScaryKoala63
Sure, add the following to your clearml.conf:
sdk.storage.cache.default_cache_manager_size = 400I think you are correct, it seems like for some reason you hit the cache limit, and a previous entry was deleted

@<1523710674990010368:profile|GreasyPenguin14> If I understand correctly you can use tokens as user/pass (it's basically the same interface from the git client perspective, meaning from ClearML

git_user = gitlab-ci-token
git_pass = <the_actual_toke>

WDYT?

If the only issue is this line
task.execute_remotely(..., exit_process=True)It has to finish the static analysis of the entire repository (which usually happens in the background but now we have to wait for it). If the repo is large this could actually take 20sec (depending on CPU/drive of the machine itself)

Wait I might be completely off.
Is this line "hangs" ?

task.execute_remotely(..., exit_process=True)

PanickyMoth78 RC is out
pip install clearml==1.6.3rc1🤞

Let me check if we can hack something...

Hi SubstantialElk6

32 CPU cores, 64GB ram

Should be plenty, this sounds like network bottle neck issue, I can't imagine the server is actually CPU bounded

We are using k8s glue to spawn the job. ...

I think this is actual network latency, nothing to do with the jobs, could it be the server is very far away?
What happens when you manually start a Task from your machine ?
Is the latency fixed? Is it just when starting a new Task?

Hi ItchyHippopotamus18
The iteration reporting is automatically detected if you are using tensorboard, matplotlib, or explicitly with trains.Logger
I'm assuming there were no reports, so the monitoring falls back to report every 30seconds where the iterations are seconds from start" (the thing is, this is a time series, so you have to have an X axis...)
Make sense ?

Ohhhh , okay as long as you know, they might fall on memory...

Ohh, I see now, yes that should be fixed as well 🙂

Hi @<1566596960691949568:profile|UpsetWalrus59>
Could it be the two experiments have the exact name ?
(I sounds like a bug in the UI, but I'm trying to make sure, and also understand how to reproduce)
What's your clearml-server version ?

Or you want to generate it from a previously executed run?

There may be cases where failure occurs before my code starts to run (and, perhaps, after it completes)

Yes that makes sense, especially from IT failure perspective

Hi @<1649221394904387584:profile|RattySparrow90>

: Are the models I defined to be served e.g. via the CLI downloaded to the serving pod

Yes this is done automatically and online (i.e. when you update the using CLI/API) , based on the models/endpoints you set

So that they are physically lying there as a file I can see in the filesystem?

They are, and cached there

Or is it more the case that the pod gets the model when needed/when an API call for this model is incoming?

I...

Hi @<1578555761724755968:profile|GrievingKoala83>

Is it possible to overrided the parameters through the configuration file when restarting the pipeline from ui?

The parameters of the Pipeline are overridden from the UI, not the pipeline components,
you can to use the pipeline parameters as is as the pipeline components parameters

Is your pipeline built from Tasks, or decorators over functions ?

Long story short, not any longer (in previous versions of k8s it was possible, but after the runtime container change it is not supported)

Could you please add it, I really do not want to miss it 🙂

Thank you ElegantCoyote26 for catching that! 😍

Hi ReassuredTiger98

I do not want to share with the clearml-agent workstations.

Long story short, no 😞
The agent is responsible to spin all jobs, regardless of users, basically it has to have a read-only user for all the repositories. I "think" the enterprise version has a vault feature, that allows you to store these kind of secrets on the User itself.
What exactly is the use case?

Awesome 🙂

Hi Team, I'm currently trying to install ClearML-Server on a Powerpc server with RedHat7.

You are a brave man LividCrab90 !

s there dockerfiles for the ClearML-Server stack somewhere ?

The main issue is replacing the DB containers, do you have elastic/mongo/redis for powerpc ?

Hi @<1572395181150310400:profile|DeterminedHare56>
Yes Slack is not the best for knowledge sharing, but it is the easiest for users to communicate over, and it is the easiest to setup and scale.
Specifically you can find historical log of the Slack channel here: None
Which we hoped google will index, but seems like this is still not working as expected, if you have any inputs it will be great to improve it

Hi JitteryCoyote63

I would like to switch to using a single auth token.

What is the rationale behind to that ?

Try:
task.update_requirements('\n'.join([".", ])) 

BTW: generally speaking the default source dir inside a docker will be:
/root/.trains/venvs-builds/<python_version>/task_repository/<repository_name>/
for example:
/root/.trains/venvs-builds/3.6/task_repository/trains.git/

DeliciousBluewhale87 this is exactly how it works,
The glue puts a k8s job with the requested docker image (the one on the Task), the job itself (k8s job) starts the agent inside the requested docker, then the agent inside the docker will install all the required packages.

Thanks MuddyCrab47 !!!
I found it!
It turns out the artifact upload will always upload from stream (aka no multi-upload). I will make sure we fix it in the next RC (I think the plan is to have it out this weekend)