Hi StickyMonkey98

a 

very

 large number of running and pending tasks, and doing that kind of thing via the web-interface by clicking away one-by-one is not a viable solution.

Bulk operations are now supported , upgrade the clearml-server to 1.0.2 🙂

Is it possible to fetch a list of tasks via Task.get_tasks,

Sure:
Task.get_tasks(project_name='example', task_filter=dict(system_tags=['-archived']))

Hi DeliciousBluewhale87
I think you are correct, there is no way to pass it.
As TimelyPenguin76 mentioned you can either set a default output_uri on the agent's config file, or edit the created Task in the UI.
What is the specific use case ? Maybe we should add this ability, wdyt?

@<1535793988726951936:profile|YummyElephant76> oh you mean like jupyter server was running, then inside the notebook you would start a new venv, in that venv "notebook" package was missing, hence it failed detecting the notebook ?

apologies @<1798887585121046528:profile|WobblyFrog79> somehow I missed your reply,

My workflow is based around executing code that lives in the same repository, so it’s cumbersome having to specify repository information all over the place, and changing commit hash as I add new code.

It automatically infers the repo if the original as long as the pipeline code itself is inside the repo, by that I mean the pipeline logic, when you run it the first time (think development etc), if it s...

SubstantialElk6
Notice if you are using a manual setup the default is "secure: false" you have to change it to "secure: true":
https://github.com/allegroai/clearml-agent/blob/176b4a4cdec9c4303a946a82e22a579ae22c3355/docs/clearml.conf#L251

so that you can get the latest artifacts of that experiment

what do you mean by " the latest artifacts "? do you have multiple artifacts on the same Task or s it the latest Task holding a specific artifact?

@<1651395720067944448:profile|GiddyHedgehong81> just to be clear, Dataset.get_local_copy returns a path to your files,
You have to Manually add the additional path to the specific files you need to use. It does Not know that in advance.
That was the initial issue you had, and I assume it is the same one here. does that make sense ?

Anyhow if the StorageManager.upload was fast, the upload_artifact is calling that exact function. So I don't think we actually have an issue here. What do you think?

I think it would make sense to have one task per run to make the comparison on hyper-parameters easier

I agree. Could you maybe open a GitHub issue on it, I want to make sure we solve this issue 🙂

What if I register the artifact manually?

task.upload_artifact('local folder', artifact_object=' ')This one should be quite quick, it's updating the experiment

if fails during 

add_step

 stage for the very first step, because 

task_overrides

 contains invalid keys

I see, yes I guess it it makes sense to mark the pipeline as Failed 🙂
Could you add a GitHub issue on this behavior, so we do not miss it ?

Great to hear!

Hi NonchalantGiraffe17
You mean this documentation?
https://clear.ml/docs/latest/docs/references/api/tasks#post-tasksclone

However, the pipeline experiment is not visible in the project experiment list.

I mean press on the "full details" in the pipeline page

Well that depends on how you think about the automation. If you are running your experiments manually (i.e. you specifically call/execute them), then at the beginning of each experiment (or function) call Task.init and when you are done call Task.close . This can be done in parallel if you are running them from separate processes.
If you want to automate the process, you can start using the trains-agent which could help you spin those experiments on as many machines as you l...

I believe AnxiousSeal95 is.
ElatedFish50 any specific reason for the question?

Hi JitteryCoyote63
Show running experimentsIt doesn't?
Have the legend clickable, to hide/show experiments based on their status:+1:
Have a line connecting points that are SOTA (example in https://paperswithcode.com/sota/image-generation-on-cifar-10 )I like that, how is that selected? (I know FE are thinking of replacing this entire graph library, so maybe good timing in terms of what to look at)

Hi @<1523702969063706624:profile|PoisedShark13>

However, INSTALLED PACKAGES of my task is misses many of installed packages (any idea why?)

It automatically detects the directly imported packages, literally analyzing your code base and looking for imports
The derivative packages (i.e. the one that any of the "main" packages need, will be listed after the first time the agent installs everything)
If something specific is missing, you can manually add it with:

Task.add_requiremen...

This port is odd, it should be 8008
8015

So basically a list of Path objects ?

I can share some code

Please do 🙂

Thank you @<1689446563463565312:profile|SmallTurkey79> !!!

Hmm that is a good question, are you mounting the clearml.conf somehow ?

repeat it until they are all dead 🙂

Hi ApprehensiveFox95
I think this is what you are looking for:
step1 = Task.create( project_name='examples', task_name='pipeline step 1 dataset artifact', repo=' ` ',
working_directory='examples/pipeline',
script='step1_dataset_artifact.py',
docker='nvcr.io/nvidia/pytorch:20.11-py3'
).id

step2 = Task.create(
project_name='examples', task_name='pipeline step 2 process dataset',
repo=' ',
working_directory='examples/pipeline',
script='step2_data_pr...

🤞

No worries, and I will make sure we output a warning if section names are not used 🙂

Okay, let's take a step back and I'll explain how things work.
When running the code (initially) and calling Task.init
A new experiment is created on the server, it automatically stores the git repo link, commit ID, and the local uncommitted changes . these are all stored on the experiment in the server.
Now assume the trains-agent is running on a different machine (which is always the case even if it is actually on the same machine).
The trains-agent will create a new virtual-environmen...

Wait I might be completely off.
Is this line "hangs" ?

task.execute_remotely(..., exit_process=True)

OutrageousGrasshopper93 is "--gpus all" working ?