Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Just Encountered A Really Frightening Bug. Best I Can Explain What Happened Was This: Data Scientist Created New Venv, Installed Clearml==1.11.0 Instead Of Clearml[S3]==1.11.1, And Upon Re-Running A Pipeline From Cli, The Entire Project "Disappeared" (W

I just encountered a really frightening bug. Best I can explain what happened was this:
Data scientist created new venv, installed clearml==1.11.0 instead of clearml[s3]==1.11.1, and upon re-running a pipeline from CLI, the entire project "disappeared" (when I enabled hidden files, I can see it). She can still create new tasks under the project name but the pipeline is now unable to find the project.

Why did the project become "hidden" and is it possible to unhide it? I don't see any such option from the UI and googling around is really not yielding anything (neither are the github issues).

Context:
docker-compose stack for clearml-server deployment. Had been using 1.11.1.
I have tried shutting down and restarting the stack.
I have also hit my computer with a shoe.

Any help would be greatly appreciated.

  
  
Posted 9 months ago
Votes Newest

Answers 19


Hi @<1593051292383580160:profile|SoreSparrow36> , can I assume you're running a self hosted server? Is there any chance you were either using a very old SDK or old backend?

The default behavior now is to create pipeline tasks as hidden and only show them as part of the pipelines UI section.

  
  
Posted 9 months ago

yeah let's step through this, i'm having her execute these steps as we speak.

create a task with the new project name. its created as a draft. can see it in the UI under the new project.

pipeline script is updated with new project name for. execute script to create pipeline. now see in UI under this new project name. nothing hidden.

the pipeline is running. when the queue is default (only serviced by one container with agent in it ( clearml-agent==1.5.2 ). abort it. everything is still in place, nothing hidden.

clearml[s3]==1.11.1 on the system. and let's see the docker containers from the stack...

the apiserver container is running 1.10.4

  
  
Posted 9 months ago

the project wasn't hidden before. I'm aware of the pipeline tasks being hidden, that makes sense for organization. but the actual project itself as an entirety has a ghost icon.

she created a new project and started working in there, it was visible in the UI... and just now it disappeared again. it's kind of like running the pipeline makes it disappear.

  
  
Posted 9 months ago

Can you please add a stand alone code snippet that reproduces this?

Just to make sure I understand the flow - you run an experiment and create it inside project 'my_example'

Afterwards you run a pipeline and you specify the controller 'my_example'.

This will make 'my_example' into a hidden project

Am I getting it right?

  
  
Posted 9 months ago

one note is that it happened after I tried deploying a set of workers to a new queue, which she tried to use to run the tasks in parallel instead of our default queue which is only serviced by one worker (a container i built)

  
  
Posted 9 months ago

Doesn't seem to reproduce for me (just ran pipeline and nothing changed about my project)

  
  
Posted 9 months ago

Also, can you please specify all the versions of agent/sdk/backend you're using?

  
  
Posted 9 months ago

then back to CLI, updated the pipeline to point the tasks to the new queue. run it, shows up in the UI (same container as default worker, just replicated w a new docker-compose and CMD to point to the new queue).

  
  
Posted 9 months ago

waiting now to see if they disappear.
any problems you may have spotted with the versions used?

project hasn't disappeared just yet. but it's happened twice now

  
  
Posted 9 months ago

so when the task completed successfully (changed the queue to default and let it finish instead of aborting), the project disappeared.

  
  
Posted 9 months ago

Can you please provide stand alone code snippets that reproduce this behavior?

  
  
Posted 9 months ago

maybe an important note: I mounted the same cache directory for the agents.

  
  
Posted 9 months ago

i will attempt to start that now.

  
  
Posted 9 months ago

didnt disappear this time.

  
  
Posted 9 months ago

and now its gone.

  
  
Posted 9 months ago

the dataset, task, and pipeline were under the same project name. i'm seeing what happens if the dataset project name was different ( f"{project_name}_data" ). which project would get deleted... the dataset one or the project of the task that kicked it off?

and the answer is...
the project is preserved, the dataset's project hidden.

so ... empty dataset names due to a small typo in parameter override + the choice for the dataset to have the same project name as the task that created it (which felt reasonable...) led to a frankenbug where the project disappeared.

thank you SO much for letting the UI have a toggle to show hidden stuff, without that I would have never had a chance of finding it!

and thank you for replying so quickly. it was clear that "projects becoming hidden" was not ever intentional.

  
  
Posted 9 months ago

i think we may have found the frankenbug?

the argument to the dataset name was not being overridden correctly (mistyped), so the default value of an empty string (instead of a placeholder like "CHANGE_ME") in the parent task caused the dataset to basically get created with an empty name, and somehow that hid the whole project, despite hundreds of existing tasks in it.

and no way to un-hide it as far as I can tell?

  
  
Posted 9 months ago

I opened github.com/allegroai/clearml/pull/1083 as an attempt to help catch this.

  
  
Posted 9 months ago

Must have missed this:

I have also hit my computer with a shoe.

Might need bigger shoe 😄

  
  
Posted 9 months ago
511 Views
19 Answers
9 months ago
9 months ago
Tags