Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Guys

Hi Guys 🙂
We have setup a ClearnML environement and launched our experiment tracking.
Everything get tracked properly but when trying to reproduce locally the experiment through the queue and ML Agent we are facing this error:

Traceback (most recent call last):
  File "/home/illuin/anaconda3/envs/emotion-classification/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/illuin/anaconda3/envs/emotion-classification/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/illuin/.clearml/venvs-builds/3.10/task_repository/emotion-classification.git/emotions_classification/__main__.py", line 4, in <module>
    cli()
  File "/home/illuin/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/illuin/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/illuin/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/click/core.py", line 1652, in invoke
    assert cmd is not None
AssertionError
/home/illuin/.clearml/venvs-builds/3.10/bin/python -u -m emotions_classification.__main__ lightning train -c configs/logging.yml -c configs/lightning_module/multiclass-transformer.yml -c configs/data_module/fr-emotions-data-module.yml -c configs/trainer/debug.yml -c configs/common/camembert-base.yml trainer.logger.name=debug

If we take the command line and run it, everything works but through ML Agents / Queues, it does not.

Anyone can help us ? 🙂

  
  
Posted 11 months ago
Votes Newest

Answers 12


It seems the task initialization process is successful, can you share a snippet of the code, how click is used and what are you doing in the main part of the code (including Task.init() etc.)?

  
  
Posted 11 months ago

Hi @<1564422650187485184:profile|ScaryDeer25> , is thie from a task run by an agent?

  
  
Posted 11 months ago

Can you please share the entire task log? Also, how are you running the agent?

  
  
Posted 11 months ago

Hello 🙂
here is the full trace:
None

  
  
Posted 11 months ago

Yeah, this is the ouput of the run from the agent

  
  
Posted 11 months ago

Yeah sure 🙂
Will share it, let me get it ^^

  
  
Posted 11 months ago

Hello 🙂
This is the snipper using Click

@define()
class ClearMLConfig:
    task_name: str = "First experiment"
    task_type: Task.TaskTypes = Task.TaskTypes.training
    reuse_last_task_id: bool = True
    output_uri: Optional[str] = None


clearml_config = ClearMLConfig()
task = Task.init(project_name='Illuin Emotions',
                 task_name=clearml_config.task_name,
                 reuse_last_task_id=clearml_config.reuse_last_task_id,
                 task_type=clearml_config.task_type,
                 output_uri=clearml_config.output_uri)


@click.group()
def cli() -> None:
    if os.path.isfile(".env"):
        logger.info("Loading .env file")
        load_dotenv(".env")


cli.add_command(lightning_cli, name="lightning")
cli.add_command(huggingface_cli, name="huggingface")

And to log it we use:
Task.current_task().connect(attrs.asdict(config))

  
  
Posted 11 months ago

Here are the code.
When using ClearML Agent it does not work but if we copy paste the command line specified in the log as Entry point it works

"entry_point = -m emotions_classification.__main__  lightning train -c  configs/logging.yml -c  configs/lightning_module/multiclass-transformer.yml -c  configs/data_module/fr-emotions-data-module.yml -c  configs/trainer/debug.yml -c  configs/common/camembert-base.yml trainer.logger.name=debug
working_dir = .
  
  
Posted 11 months ago

Hi 🙂
Thanks for your reply, I will find it out

  
  
Posted 11 months ago

Any news guys about this issue ? 🙂

  
  
Posted 11 months ago

The issue may be related to the fact that right now we have some edge cases when working with lightning >= 2.0, we should have better support in the upcoming release

  
  
Posted 11 months ago

Hey @<1564422650187485184:profile|ScaryDeer25> , we just released clearml==1.11.1rc2 which should solve the compatibility issues for lightning >= 2.0. Can you install it and check whether it solves your problem?

  
  
Posted 10 months ago
506 Views
12 Answers
11 months ago
10 months ago
Tags
Similar posts