Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, Guys! I’M Trying To Connect Clearml To My Task And Getting Strange Error: After

Hi, guys! I’m trying to connect clearml to my task and getting strange error: after
CUDA_VISIBLE_DEVICES=1 python train.py dataset_config=visual_tags_dataset_default_config dataset_config/preprocessors=preprocessors_config_default
I got
File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/task.py", line 529, in init task._connect_argparse(parser=parser, parsed_args=parsed_args) File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/task.py", line 2191, in _connect_argparse parser, args=args, namespace=namespace, parsed_args=parsed_args) File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/backend_interface/task/args.py", line 186, in copy_defaults_from_argparse __parameters_types=task_defaults_types File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/backend_interface/task/task.py", line 976, in update_parameters self._set_parameters(*args, __update=True, **kwargs) File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/backend_interface/task/task.py", line 918, in _set_parameters type=param_type, File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/backend_api/services/v2_9/tasks.py", line 1003, in __init__ self.description = description File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/backend_api/services/v2_9/tasks.py", line 1067, in description self.assert_isinstance(value, "description", six.string_types) File "/home/yulia/ys-ds-visual-insights/venv/lib/python3.7/site-packages/trains/backend_api/session/datamodel.py", line 116, in assert_isinstance raise TypeError("Expected %s of type %s, got %s" % (field_name, expected, type(value).__name__)) TypeError: Expected description of type (<class 'str'>,), got LazyCompletionHelpdoes anyone know how to fix it?

  
  
Posted 3 years ago
Votes Newest

Answers 30


DepressedChimpanzee34
I might have an idea , based on the log you are getting LazyCompletionHelp in stead of str
Could it be you installed hyrda bash completion ?
https://github.com/facebookresearch/hydra/blob/3f74e8fced2ae62f2098b701e7fdabc1eed3cbb6/hydra/_internal/utils.py#L483

  
  
Posted 3 years ago

DepressedChimpanzee34 I cannot find cfg.py here
https://github.com/allegroai/clearml/tree/master/examples/frameworks/hydra/config_files
(or anywhere else)

  
  
Posted 3 years ago

Yes, I think we just found out it breaks clearml πŸ™‚
could you test with the latest stable, just in case ?
(I'll make sure we have an RC that supports the hydra dev version)

  
  
Posted 3 years ago

could you test with the latest stable, just in case - unfortunately, no, it’s not that easy to make my code working with that hydra(

thank you for your help!

  
  
Posted 3 years ago

let me check when a fix can be deployed for Hydra...

  
  
Posted 3 years ago

it happens on task init

  
  
Posted 3 years ago

Hi AgitatedDove14 , I am using the pre release of hydra together with the latest clearml, and previous clearml server version (I didn't have time yet to update)..
I'm getting similar error to the mentioned in the post:
2021-05-06 15:48:07,359 - root - WARNING - 2021-05-06 15:48:07,359 - root - WARNING - File "/home/user/.local/lib/python3.8/site-packages/clearml/backend_api/services/v2_9/tasks.py", line 1067, in description self.assert_isinstance(value, "description", six.string_types) 2021-05-06 15:48:07,359 - root - WARNING - 2021-05-06 15:48:07,359 - root - WARNING - File "/home/user/.local/lib/python3.8/site-packages/clearml/backend_api/session/datamodel.py", line 116, in assert_isinstance raise TypeError("Expected %s of type %s, got %s" % (field_name, expected, type(value).__name__)) 2021-05-06 15:48:07,360 - root - WARNING - 2021-05-06 15:48:07,360 - root - WARNING - TypeError: Expected description of type (<class 'str'>,), got LazyCompletionHelp 2021-05-06 15:48:07,360 - root - WARNING - 2021-05-06 15:48:07,360 - root - WARNING - Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. Process finished with exit code 1any ideas how to solve this issue?

  
  
Posted 3 years ago

DepressedChimpanzee34
What's the hydra version ?
I tested with 1.1.0dev3 and it worked for me

  
  
Posted 3 years ago

Hmm let me rerun (offline mode right ?)

  
  
Posted 3 years ago

the offline error is different

  
  
Posted 3 years ago

no, online

  
  
Posted 3 years ago

I get it with the simplest config if I define it as a dataclass, using the example you share as a basis:
config_files/cfg.py
` from hydra.core.config_store import ConfigStore
from dataclasses import dataclass

@dataclass
class MasterConfig:
test: str = 'test'

cs = ConfigStore.instance()
cs.store(name="config", node=MasterConfig) and for the main I had to make some small changes to connect my local server (I'm sharing them just in case): hydra_example.py: # ClearML - Hydra Example

import hydra

from omegaconf import OmegaConf

from clearml import Task
from dataclasses import dataclass
from config_files import cfg

@dataclass
class CLEARML_SERVER_CONFIG:
API_HOST: str
WEB_HOST: str
FILE_SERVER: str
OUTPUT_URI: str
KEY: str
SECRET: str
# HOST: str

@hydra.main(config_name="config")
def my_app(cfg):
# type (DictConfig) -> None

CLEARML_LOCAL = CLEARML_SERVER_CONFIG(API_HOST=' ` ` ',
                                      WEB_HOST=' ` ` ',
                                      FILE_SERVER=' ` ` ',
                                      OUTPUT_URI=' ` ` ',
                                      KEY='2CBR16L9T9R6ANYTAA5K',
                                      SECRET='pK)i-(ImwRPyL(Eu1nKB5cdy(Y+9_XkGxLrpZmWw0^Abyc7&+b')
# if credentials are not set should offline doesn't work
Task.set_credentials(api_host=CLEARML_LOCAL.API_HOST, web_host=CLEARML_LOCAL.WEB_HOST,
                     files_host=CLEARML_LOCAL.FILE_SERVER, key=CLEARML_LOCAL.KEY,
                     secret=CLEARML_LOCAL.SECRET)
task = Task.init(project_name="examples", task_name="hydra configuration")

if name == "main":
my_app() `and the output error is the same as I shared above

  
  
Posted 3 years ago

Why do you need to have the configuration added manually ? isn't the cleaml.conf easier ? If not I think OS environments are easier no?- well at this point I'm not sure it is still essential, we have 3 run-modes offline, local-server, cloud-sever and this option made it work for all of them.. can be that it is not required anymore and its just legacy..
I run run above code, everything worked with no exception/warning...- this is strange.. you ran it with the dataclass config I added?
What is the try/except solves exactly ? - it solved the issue I reported
TypeError: Expected description of type (<class 'str'>,), got LazyCompletionHelp

  
  
Posted 3 years ago

I found a super weird fix for this error.. no idea why it works but maybe it can help with debugging it..
try: task = Task.init(project_name="examples", task_name="hydra configuration", reuse_last_task_id=False) except: task = Task.init(project_name="examples", task_name="hydra configuration", reuse_last_task_id=True)

  
  
Posted 3 years ago

if I remove the import and use the path I get this:
raise MissingConfigException(hydra.errors.MissingConfigException: Cannot find primary config 'config'. Check that it's in your config search path.

  
  
Posted 3 years ago

its the module where the cfg is defined, like in the example you shared config_files/cfg.py

  
  
Posted 3 years ago

and the path it shows is correct..

  
  
Posted 3 years ago

from the example I shared above
config_files/cfg.py
` from hydra.core.config_store import ConfigStore
from dataclasses import dataclass

@dataclass
class MasterConfig:
test: str = 'test'

cs = ConfigStore.instance()
cs.store(name="config", node=MasterConfig) `

  
  
Posted 3 years ago

Any chance you can zip the entire folder? I can't figure out what's missing, specifically "from config_files" , i.e. I have no packages nor file named config_files

  
  
Posted 3 years ago

DepressedChimpanzee34
I have sad news, it is working just fine for me 😞
Python3.6 Ubuntu hydra-core==1.1.0.dev6

  
  
Posted 3 years ago

πŸ˜• well I have py38 and debian.. strange I'll give it another look

  
  
Posted 3 years ago

its only these two files.. nothing else

  
  
Posted 3 years ago

thanks AgitatedDove14 , Not that I know I didn't do any special setup other than the straightforward pip install hydra-core --pre

  
  
Posted 3 years ago

AgitatedDove14 , the hydra people say its not on "their side".. so it is still a mystery to me

  
  
Posted 3 years ago

Python3.8 I can quickly check, give me a minute

  
  
Posted 3 years ago

Noooooooooo, it is still working πŸ™‚

  
  
Posted 3 years ago

well at this point I'm not sure it is still essential, we have 3 run-modes offline, local-server, cloud-sever and this option made it work for all of them.. can be that it is not required anymore and its just legacy..

LOL, sure if you have so many setups, that makes sense πŸ™‚

this is strange.. you ran it with the dataclass config I added?

Yes but I had to remove the:
from config_files import cfgand instead used:
@hydra.main(config_path="config_files", config_name="config") def my_app():I need help in reproducing it.

  
  
Posted 3 years ago

Can you test with the hydra example? if the example works, any chance you can send a toy to reproduce it ?
https://github.com/allegroai/clearml/tree/master/examples/frameworks/hydra

  
  
Posted 3 years ago

Hi DepressedChimpanzee34
Why do you need to have the configuration added manually ? isn't the cleaml.conf easier ? If not I think OS environments are easier no? I run run above code, everything worked with no exception/warning... What is the try/except solves exactly ?

  
  
Posted 3 years ago

dev6

  
  
Posted 3 years ago
615 Views
30 Answers
3 years ago
one year ago
Tags
Similar posts