I see..
Generally speaking If that is the case, I would think it might be better to use the docker mode, it offers way more stable environment, regardless on the host machine runinng the agent. Notice there is no need to use custom containers, as the agent will basically run the venv process, only inside a container, allowing you to reuse offf the shelf containers.

If you were to add this, where would you put it? I can use a modified version of 

clearml-agent

Yep, that would b...

JitteryCoyote63 How is it so far ?

Okay, some progress, so what is the difference ?
Any chance the issue can be reproduced with a small toy code ?
Can you run the tqdm loop inside the code that exhibits the CR issue ? (maybe some initialization thing that is causing it to ignore the value?!)

Hi JitteryCoyote63
I think there is a GitHub issue (request on it), this is not very trivial to build (basically you need the agent to first temporary pull the git, apply changes, build docker, remove temp build, and restart with the new image)
Any specific reason for not pushing a docker, or using the extra docker bash script on the Task itslef?

create a new file, copy paste to the new file these lines, and run it inside vscode, what are you getting in the console?

from clearml import Task Task.add_requirements("tensorflow") task = Task.init(project_name="debug", task_name="requirements") print("done")

An easier fix for now will probably be some kind of warning to the user that a task is created but not connected

That is a good point, maybe if you do not have a "main" Task, then we print the warning (with some flag to disable the warning) ?

PompousParrot44 unfortunately not yet 😞
But the gist is :
MongoDB stores experiment data (i.e. execution parameters, git ref etc.)
ElasticSearch stores results (i.e. metrics console logs, debug image links etc.)
Does that help?

Hi FloppyDeer99
Since this thread is a bit old, I might have missed something 🙂
Are we saying the links are not working in the UI ?
(notice the links themselves are generated by the clearml package, so if there was a bug, still not sure here, then old links will remain invalid until manually fixed) Can you verify that the latest clearml generates working links?

would I have to execute each task in the pipeline locally(but still connected to trains),

Somehow you have to have the pipeline step Task in the system, you can import it from code, or you can run it once, then the pipeline will clone it and reuse it. Am I missing something ?

I did nothing to generate a command-line. Just cloned the experiment and enqueued it. Used the server GUI.

Who/What created the initial experiment ?

I noticed that if I run the initial experiment by "python -m folder_name.script_name"

"-m module" as script entry is used to launch entry points like python modules (which is translated to "python -m script")
Why isn't the entry point just the python script?
The command line arguments are passed as arguments on the Args section of t...

Hi @<1610083503607648256:profile|DiminutiveToad80>
Yes, it does. They are also cached by default (on the machine with the agent)
None

Uninstall the current clearml-agent and reinstall this wheel, I hacked it to have ==, let's see if that works

Hi CooperativeSealion8
Seems like your NoScript addon is blocking the site :)

Hi JitteryCoyote63 ,
These properties are usually not available on the UI and are used internal, hence the lack of documentation. Regrading parent property, it will hold a parent Task.id (str) , that said it has no real effect on the Task itself. You can however search for Tasks with a specific parent ID (For examples, this is how the the hyper parameter class is using this property)

Is this like a local minio?
What do you have under the sdk/aws/s3 section ?

I still do not get why this leads to some 0.5 values when in my plot there should only be 0 and 1.

Smart sub-sampling (lowpass filter before, aka averaging on a window)

I think that is good enough

Hi @<1523701260895653888:profile|QuaintJellyfish58>

Is there a way or a trigger to detect when the number of workers in a queue reaches zero?

You mean to spin them down? what's the rational ?

I’d like to implement a notification system that alerts me when there are no workers left in the queue.

How are they "dropping" ?

Specifically to your question, let me check I'm sure there is an API that get's that data becuase you can see it in the UI 🙂

Just making sure, the machine that you were running the "trains-init" on can access the API server ?

Hi FlutteringWorm14

Is there some way to limit that?

What do you mean by that? are you referring to the Free tier ?

Thus, the return data from step 2 needs to be available somewhere to be used in step 3.

Yep 🙂

It will serialize the data on the dict?

I thought it will just point to a local file location where you have the data 🙂

I didn’t know that each steps runs in a different process

Actually ! you can run them as functions as well, try:
if __name__ == '__main__': PipelineDecorator.debug_pipeline() # call pipeline function hereIt will just run them as functions (ret...

Try adding this environment variable:
export TRAINS_CUDA_VERSION=0

Of course, I used "localhost"

Do not use "localhost" use your IP then it would be registered with a URL that points to the IP and then it will work

I think the only way is using the API, with task.query_tasks and filter, would that have helped?

Yeah the doctring is always the most updated 🙂

containing the

Extension

module

Not sure I follow, what is the Extension module ? what were you running manually that is not just pip install /opt/keras-hannd ?

SubstantialElk6 whats the command line you are using ?

PompousBeetle71 let me know if it solves your problem

I see...
Current (and this will change soon) the entire delta is stored in a single file, so there is no real way to download a "subset" of the data, only a parent version 😞

Lets say that this small dataset has a ID ....

Yes this would be exactly the way to do so:

` param ={'dataset': small_train_dataset_id_here}
task.connect(param)

dataset_folder = Dataset.get(param['dataset']).get_local_copy()
... Locally it will use the small_train_dataset_id_here ` , then whe...

How can I ensure that additional tasks aren’t created for a notebook unless I really want to?

TrickySheep9 are you saying two Tasks are created in the same notebook without you closing one of them ?
(Also, how is the git diff warning there with the latest clearml, I think there was some fix related to that)