Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys

Hi Guys 🙂
We have setup a ClearnML environement and launched our experiment tracking.
Everything get tracked properly but when trying to reproduce locally the experiment through the queue and ML Agent we are facing this error:

Traceback (most recent call last):
  File "/home/illuin/anaconda3/envs/emotion-classification/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/illuin/anaconda3/envs/emotion-classification/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/illuin/.clearml/venvs-builds/3.10/task_repository/emotion-classification.git/emotions_classification/__main__.py", line 4, in <module>
    cli()
  File "/home/illuin/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/illuin/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/illuin/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/click/core.py", line 1652, in invoke
    assert cmd is not None
AssertionError
/home/illuin/.clearml/venvs-builds/3.10/bin/python -u -m emotions_classification.__main__ lightning train -c configs/logging.yml -c configs/lightning_module/multiclass-transformer.yml -c configs/data_module/fr-emotions-data-module.yml -c configs/trainer/debug.yml -c configs/common/camembert-base.yml trainer.logger.name=debug

If we take the command line and run it, everything works but through ML Agents / Queues, it does not.

Anyone can help us ? 🙂

  
  
Posted one year ago
Votes Newest

Answers 12


Hello 🙂
here is the full trace:
None

  
  
Posted one year ago

Yeah sure 🙂
Will share it, let me get it ^^

  
  
Posted one year ago

It seems the task initialization process is successful, can you share a snippet of the code, how click is used and what are you doing in the main part of the code (including Task.init() etc.)?

  
  
Posted one year ago

Here are the code.
When using ClearML Agent it does not work but if we copy paste the command line specified in the log as Entry point it works

"entry_point = -m emotions_classification.__main__  lightning train -c  configs/logging.yml -c  configs/lightning_module/multiclass-transformer.yml -c  configs/data_module/fr-emotions-data-module.yml -c  configs/trainer/debug.yml -c  configs/common/camembert-base.yml trainer.logger.name=debug
working_dir = .
  
  
Posted one year ago

Hey @<1564422650187485184:profile|ScaryDeer25> , we just released clearml==1.11.1rc2 which should solve the compatibility issues for lightning >= 2.0. Can you install it and check whether it solves your problem?

  
  
Posted one year ago

Hi @<1564422650187485184:profile|ScaryDeer25> , is thie from a task run by an agent?

  
  
Posted one year ago

Hi 🙂
Thanks for your reply, I will find it out

  
  
Posted one year ago

Yeah, this is the ouput of the run from the agent

  
  
Posted one year ago

The issue may be related to the fact that right now we have some edge cases when working with lightning >= 2.0, we should have better support in the upcoming release

  
  
Posted one year ago

Can you please share the entire task log? Also, how are you running the agent?

  
  
Posted one year ago

Any news guys about this issue ? 🙂

  
  
Posted one year ago

Hello 🙂
This is the snipper using Click

@define()
class ClearMLConfig:
    task_name: str = "First experiment"
    task_type: Task.TaskTypes = Task.TaskTypes.training
    reuse_last_task_id: bool = True
    output_uri: Optional[str] = None


clearml_config = ClearMLConfig()
task = Task.init(project_name='Illuin Emotions',
                 task_name=clearml_config.task_name,
                 reuse_last_task_id=clearml_config.reuse_last_task_id,
                 task_type=clearml_config.task_type,
                 output_uri=clearml_config.output_uri)


@click.group()
def cli() -> None:
    if os.path.isfile(".env"):
        logger.info("Loading .env file")
        load_dotenv(".env")


cli.add_command(lightning_cli, name="lightning")
cli.add_command(huggingface_cli, name="huggingface")

And to log it we use:
Task.current_task().connect(attrs.asdict(config))

  
  
Posted one year ago
635 Views
12 Answers
one year ago
one year ago
Tags
Similar posts