I mean to use a function decorated with 

PipelineDecorator.pipeline

 inside another pipeline decorated in the same way.

Ohh... so would it make sense to add "helper_functions" so that a function will be available in the step's context ?
Or maybe we need a new to support "standalone" decorator?! Currently to actually "launch" the function step, you have to call it from the "pipeline" main logic function, but, at least in theory, one could do without the Pipeline itself.....

GreasyPenguin14 I think this is what you are looking for
Task.get_project_id('project_name')

Would be cool to let it get untracked as well, especially if we want to as an option

How would you decide what should be tracked?

Happy to hear 🙂

Hi RoughTiger69
Interesting question, maybe something like:

` @PipelineDecorator.component(...)
def process_sub_list(things_to_do=[0,1,2]):
r = []
for i in things_to_do:
print("doing", i)
r.append("done{}".format(i))
return r

@PipelineDecorator.pipeline(...)
def pipeline():

create some stuff to do:

results = []
for step in range(10):
r = process_sub_list(list(range(step*10, (step+1)*10)))
results.append(r)

push into one list with all result, this will ac...

Hi @<1535793988726951936:profile|YummyElephant76>
None
None
None

This is very odd ... let me check something

Did you run clearml-init after the pip install ?

Thanks ContemplativePuppy11 !

How would you pass data/args between one step of the pipeline to another ?
Or are you saying the pipeline class itself stores all the components ?

SourOx12
Hmmm. So if last iteration was 75, the next iteration (after we continue) will be 150 ?

We have tried to manually restart tasks reloading all the scalars from a dead task and loading latest saved torch model.

Hi ThickKitten19
how did you try to restart them ? how are you monitoring dying instances ? where . how they are running?

Correct 🙂

Something like the TYPE_STRING that Triton accepts.

I saw the github issue, this is so odd , look at the triton python package:
https://github.com/triton-inference-server/client/blob/4297c6f5131d540b032cb280f1e[…]1fe2a0744f8e1/src/python/library/tritonclient/utils/init.py

IrritableOwl63 in the profile page, look at the bottom right corner

and since the update the docs seem to be a bit off but I think I got it

Working on a whole new site 😉

Would you have an example of this in your code blogs to demonstrate this utilisation?

Yes! I definitely think this is important, and hopefully we will see something there 🙂 (or at least in the docs)

  is the "installed packages" part editable? good to know

Of course it is, when you clone a Task everything is Editable 🙂

Isn't it a bit risky manually changing a package version?

worst case it will crash quickly, and you reset/edit/enqueue 🙂
(Should work though)

Let me check the API reference
https://clear.ml/docs/latest/docs/references/api/endpoints#post-tasksget_all
So not straight query, but maybe:
https://clear.ml/docs/latest/docs/references/api/endpoints#post-tasksget_all_ex
all section might do the trick.
SuccessfulKoala55 any chance you have an idea on what to pass there ?

Depending on your security restrictions, but generally yes.

AttractiveCockroach17 can I assume you are working with the hydra local launcher ?

PompousBeetle71 the code is executed without arguments, in run-time trains / trains-agent will pass the arguments (as defined on the task) to the argparser. This means you that you get the ability to change them and also type checking 🙂

PompousBeetle71 if you are not using argparser how do you parse the arguments from sys.argv? manually?
If that's the case, post parsing, you can connect a dictionary to the Task and you will have the desired behavior
` task.connect(dict_with_arguments...

Yes, actually ensuring pip is there cannot be skipped (I think in the past it cased to many issues, hence the version limit etc.)
Are you saying it takes a lot of time when running? How long is the actual process that the Task is running (just to normalize times here)

Funny it's the extension "h5" , it is a different execution path inside keras...
Let me see what can be done 🙂

Yea the "-e ." seems to fit this problem the best.

👍

It seems like whatever I add to

docker_bash_setup_script

is having no effect.

If this is running with the k8s glue, there console out of the docker_bash_setup_script ` is currently Not logged into the Task (this bug will be solved in the next version), But the code is being executed. You can see the full logs with kubectl, or test with a simple export test

docker_bash_setup_script
` export MY...

Hi @<1572395184505753600:profile|GleamingSeagull15>
Try adjusting:
None
to 30 sec
It will reduce the number of log reports (i.e. API calls)

Thanks VivaciousPenguin66 !
BTW: if you are running the local code with conda, you can set the agent to use conda as well (notice that if you are running locally with pip, the agent's conda env will use pip to install the packages to avoid version mismatch)

WickedGoat98 sure that will not be complicated:
try something along the lines of :
agent: networks: - backend container_name: clearml-agent image: allegroai/clearml-agent:latest restart: unless-stopped privileged: true environment: CLEARML_HOST_IP: ${CLEARML_HOST_IP} CLEARML_WEB_HOST: ${CLEARML_WEB_HOST:-} CLEARML_API_HOST: `
CLEARML_FILES_HOST: ${CLEARML_FILES_HOST:-}
CLEARML_API_ACCESS_KEY: ${CLEARML_API_ACCESS_KEY:-}
...

Basically the links to the file server are saved in both mongo and elastic, so as long as these are host:ip based, at least in theory it should work

clearml-agent daemon --detached --queue manual_jobs automated_jobs --docker --gpus 0

If the user running this command can run "docker run", then you should ne fine

JitteryCoyote63 hacky but sure 🙂
` from trains.config import config_obj

print(config_obj) `