Cool thanks 🤩

Couldn't find any logic on which tasks fail and why... all the lines are exactly the same, only different parameters

When I said not the expected behavior, I meant that following the instructions on the docs, should lead to downloading the latest version

I double checked that the task object is the correct one

so when I'm on the machine through SSH

In the larger context I'd look on how other object stores treat similar problems, I'm not that advanced in these topics.

But adding a simple force_download flag to the get_local_copy method could solve many cases I can think of, for example I'd set it to true in my case as I don't mind the times it will re-download when not necessary as it is quite small (currently I always delete the local file, but it looks pretty ugly)

found it here:
https://github.com/allegroai/trains/blob/98ea965e6d0a39019b31e5fec146ab55da0d52e8/trains/storage/manager.py#L41

I mean usually it would read if cached_file: return cached_file

one sec

I might, I'll look at the internals later cause at a glance I didn't really get the logic inside get_local_copy ... the if there is ending with if ... not cached_file: return cached_file which from reading doesn't make much sense

Legit, if you have a cached_file (i.e. exists and accessible), you can return it to the caller

I agree, so shouldn't it be if cached_file: return cached_file instead of if not cached_file: return cached_file

-_- why there isn't a link to source on the docs?

So once I enqueue it is up? Docs says I can configure the queues that the auto scaler listens to in order to spin up instances, inside the auto scale task - I wanted to make sure that this config has nothing to do to where the auto scale task was enqueued to

BTW is the if not cached_file: return cached_file is legit or a bug?

👍

nope - I'm talking about here

https://github.com/allegroai/trains/blob/98ea965e6d0a39019b31e5fec146ab55da0d52e8/trains/storage/manager.py#L41

` # define pipeline
pipe = clearml.PipelineController(
name=TASK_NAME,
project=PROJECT_NAME,
version='0.0.1',
add_pipeline_tags=False,
)
pipe.set_default_execution_queue('default')

Adding steps

pipe.add_step(name=f'{start_date_train}_{end_date_train}_choose_best',
base_task_project=CHOOSE_PROJECT_NAME,
base_task_name=CHOOSE_TASK_NAME,
parameter_override=params_override,
...

I'll share some screenshots

AgitatedDove14 sorry for the late reply,

It's right after executing all the steps. So we have the following block which determines whether we run locally or remotely

if not arguments.enqueue: pipe.start_locally(run_pipeline_steps_locally=True) else: pipe.start(queue=arguments.enqueue)
And right after we have a method that calls Task.current_task() which returns None

😄

it works without any problem

🤞

Also being able to separate their configurations files would be good (maybe there is and I don't know?)

No I don't have trains anywhere in my code

the ones in the export are like you said

FriendlySquid61

Just updating, I still haven't touched this.... I did not consider the time it would take me to set up the auto scaling, so I must attend other issues now, I hope to get back to this soon and make it work

that will require restarting the agent again?

btw my site packages is false - should it be true? You pasted that but I'm not sure what it should be, in the paste is false but you are asking about true

SuccessfulKoala55 this actually doesn't work

` apiserver_conf = ConfigFactory.parse_file(API_SERVER_CONF_PATH)

POINT 1

conf_content = HOCONConverter.to_hocon(config=ConfigFactory.from_dict(apiserver_conf.as_plain_ordered_dict()),
compact=False,
level=0, indent=2)

apiserver_conf['auth']['fixed_users']['users'].append(
ConfigFactory.from_dict({'username': username, 'password': password, 'name': name}))

##...