But I would need to reindex everything right? Is that a expensive operation?

Something like that?
` curl "localhost:9200/events-training_stats_scalar-adx3r00cad1bdfvsw2a3b0sa5b1e52b/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must": [
{
"match": {
"variant": "loss_model"
}
},
{
"match": {
"task": "8f88e4b8cff84f23bde74ed4b7213ec6"
}
}
]
}
},
"aggs": {
"series": {
"terms": { "field": "iter" }
}
}
}...

There is no way to filter on long types? I can’t believe it

So it looks like it tries to register a batch of 500 documents

Ha nice, good one! Thanks!

(Just to know if I should wait a bit or go with the first solution)

Yea again I am trying to understand what I can do with what I have 😄 I would like to be able to export as an environment variable the runtime where the agent is installing, so that one app I am using inside the Task can use the python packages installed by the agent and I can control the packages using clearml easily

So previous_task actually ignored the output_uri

(Btw the instance listed in the console has no name, it it normal?)

Yes 🙂 Thanks!

BTW, is there any specific reason for not upgrading to clearml?

I just didn't have time so far 🙂

ok, thanks SuccessfulKoala55 !

For the moment this is what I would be inclined to believe

btw CostlyOstrich36 , I can see in Profile > Version: 1.1.1-135 • 1.1.1 • 2.14 . What these numbers correspond to?

trains==0.16.4

yes, exactly: I run python my_script.py , the script executes, creates the task, calls task.remote_execute(exit_process=True) and returns to bash. Then, in the bash console, after some time, I see some messages being logged from clearml

SuccessfulKoala55 I tried to setup in a different machine the clearml-agent and now I get a different error message in the logs:
Warning: could not locate requested Python version 3.6, reverting to version 3.6 clearml_agent: ERROR: Python executable with version '3.6' defined in configuration file, key 'agent.default_python', not found in path, tried: ('python3.6', 'python3', 'python')

Ho and also use the colors of the series. That would be a killer feature. Then I simply need to match the color of the series to the name to check the tags

Ho the object is actually available in previous_task.artifacts

yes SparklingHedgehong28 🙂

Thanks @<1523701087100473344:profile|SuccessfulKoala55> ! Are alive workers sending ping to notify the server that they are alive or does the server infers that they are alive based on the last communication?

Hi CostlyOstrich36 , I mean insert temporary access keys

Ok, I am asking because I often see the autoscaler starting more instances than the number of experiments in the queues, so I guess I just need to increase the max_spin_up_time_min

Thanks a lot AgitatedDove14 !

I am confused now because I see in the master branch, the clearml.conf file has the following section:
# Or enable credentials chain to let Boto3 pick the right credentials. # This includes picking credentials from environment variables, # credential file and IAM role using metadata service. # Refer to the latest Boto3 docs use_credentials_chain: falseSo it states that IAM role using metadata service should be supported, right?

yes

But I can do:
` $ python

import torch
torch.cuda.is_available()
True
torch.backends.cudnn.version()
8005 `

I can ssh into the agent and:
source /trains-agent-venv/bin/activate (trains_agent_venv) pip show pyjwt Version: 1.7.1

so most likely one hard requirement installs the version 2 of pyjwt while setting up the experiment

hooo now I understand, thanks for clarifying AgitatedDove14 !