You can get a mutable copy of the entire dataset (original version), with get_mutable_copy() Then change the files on the returned directory, then create a new Dataset with the parent dataset as the original verison, then sync the folder.
You can also just update the specific file (without needing to download the entire original version)

ssh: Could not resolve hostname

: Name or service not known

@<1542316991337992192:profile|AverageMoth57> so is this the main issue? this seems unrelated to the Gerrit thing, just missing configuration of the .ssh on the agent machine, is that correct?

The pipeline itself is also a task, so this line works in a pipeline. Task.current_task is a class method that returns the running task (pipeline in our case), then then the usual interface. BTW what are you having in the conf file ?

PanickyAnt52 when the docker is loaded, it will search for the highest python version to use for the agent. Then when it is launching the Task itself, it will first try to match the python version requested by the Task. It does so by looking for "python3.7" ,
what are you getting when running "which python3.7" inside the docker ? Could it be you have a venv inside the docker with the diff python version ?

Hi FierceHamster54
Sure just do
dataset = Dataset.get(dataset_project="project", dataset_name="name")This will by default fetch the latest version

Hi MistakenDragonfly51

Is it possible to use it without using the clearml agent system?

Yes it is, which would mean everything is executed locally
basically:
an_optimizer.start_locally()instead of this line
https://github.com/allegroai/clearml/blob/51af6e833ddc5a8ba1efaaf75980f58616b25e85/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py#L121

Since pytorch is a special example (the agent will pick the correct pytorch based on the installed CUDA) , the agent will first make sure the file is downloaded, and then pass the resolving for pip to decide if it necessary to install. (bottom line, we downloaded the torch for no reason but it is cached so no real harm done) It might be the second package needs a specific numpy version... this resolving is don't by pip, not the agent specifically. Anyhow --system-site-packages is applicable o...

Maybe we should add it to Storage Manager? What do you think?

GrievingTurkey78 did you open the 8008 / 8080 / 8081 ports on your GCP instance (I have to admit I can't remember where exactly in the admin panel you do that, but I can assure you it is there :)

YEY! 🚀 🎉

current task fetches the good Task

Assuming you fork the process than the gloabl instance" is passed to the subprocess. Assuming the sub-process was spawned (e.g. POpen) then an environement variable with the Task's unique ID is passed. then when you call the "Task.current_task" it "knows" the Task was already created and it will fetch the state from the clearml-server and create a new Task object for you to work with.
BTW: please use the latest RC (we fixed an issue with exactly this...

FlatStarfish45

In the parent task, the libs appear installed.

What do you mean by "parent Task"? Is this the base task we are optimizing (i.e. the experiment / model we are optimizing) ?
Or is it the "Optimization Task" itself?

Hi TenseOstrich47
Does the .ssh folder on the user running the agent contain the correct credentials ?
Basically from the user running the agent on the agent's machine can you clone the repo with:
ssh://git@github.com/15gifts/py-db.git

CurvedHedgehog15 there is not need for :
task.connect_configuration( configuration=normalize_and_flat_config(hparams), name="Hyperparameters", )Hydra is automatically logged for you, no?!

How do you report it?

hurray

And after having called

Task.init()

the second time, the automatic logging of resources and tensorboard plots works as well. I would recommend adding explanation to the docs for

Oh yeah! you always need to call Task.init first, Task,current_task should be called from anywhere you like but after the Task.init was called.

I still have name

my_name

, but the project name

my_project/.datasets/my_name

rather than

my_project/.datasets

Yes, this is the expected behavior

And I don't see any new projects / subprojects where that dataset creation Task is stored

They are marked "hidden" hence by default you cannot see them in the UI (so they will only appear in the Dataset page),
you can turn the UI hidden flag by going to your settings page and selecting "Con...

Good point!
I'll make sure we do 🙂

Can you clone the git with the .ssh credentials on the host machine ?
If so, can you do the same manually inside a docker (i.e. spin a docker with mount -v /home/hostuser/.ssh:/root/.ssh) ?

fyi: hot fix for 1.3.0 (smoothing graphs) was just released see v1.3.1

I am actually considering rolling back to 1.1.0,

Can you share why?
JitteryCoyote63 notice from the release notes of 1.2:

Important Note!
This release requires a MongoDB migration from previous versions. Please see

for more information.

I'm not sure you can downgrade that easily ...

link with "localhost" in it Oo

Hmm I think this is the main issue, for some reason the dataset default upload destination is "localhost", what do you have configured in your clearml.conf under files server?

Hi WittyOwl57

I'm guessing clearml is trying to unify the histograms for each iteration, but the result is in this case not useful.

I think you are correct, the TB histograms are actually a 3d histograms (i.e. 2d histograms over time, which would be the default for kernel;/bias etc.)

is there a way to ungroup the result by iteration, and, is it possible to group it by something else (e.g. the tags of the two plots displayed below side by side).

Can you provide a toy example...

Oh, yes, that might be (threshold is 3 minutes if no reports) but you can change that:
task.set_resource_monitor_iteration_timeout(seconds_from_start=10)

I should mention this is run within a TF v1 session context

This should not be connected.

everything gets stored as intended (to clearML dashboard)

So in jupyter it works? But from command line it does not ? what's the difference ?

and config is empty ?

Yes I think you are correct

ShallowCat10 so you mean like meta-data on top of the image? or another level of title series ?
because, the iteration field itself is an integer...

LazyFish41 just making sure, you built a container from the docker file, and used it as base docker image for the Task, is that correct ?
Also notice the cleaml-agent will not change the entry point of the docker meaning if the entry point does not end with plain bash, it will not actually run anything

but I belive it should have work with 0.14.1 as well

Correct