Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey Everyone, I Have Been Trying To Get The Pytorch Lightning Cli To Work With Remote Task Execution, But It Just Won'T Work. I Took The

Hey everyone, I have been trying to get the pytorch lightning CLI to work with remote task execution, but it just won't work. I took the https://github.com/allegroai/clearml/blob/master/examples/frameworks/jsonargparse/pytorch_lightning_cli.py and extended it like this:
` task = Task.init(project_name="example", task_name="pytorch_lightning_jsonargparse")

  • task.execute_remotely(queue_name="single-gpu") # that's our internal queue
    LightningCLI(ImageClassifier, seed_everything_default=42, save_config_overwrite=True, run=True) When running this experiment I get the following error: Traceback (most recent call last):
    File "/home/clearml/.clearml/venvs-builds.1/3.9/task_repository/clearml.git/examples/frameworks/jsonargparse/pytorch_lightning_cli.py", line 120, in <module>
    LightningCLI(ImageClassifier,
    File "/home/clearml/.clearml/venvs-builds.1/3.9/lib/python3.9/site-packages/pytorch_lightning/utilities/cli.py", line 157, in init
    super().init(*args, **kwargs)
    File "/home/clearml/.clearml/venvs-builds.1/3.9/lib/python3.9/site-packages/pytorch_lightning/cli.py", line 346, in init
    self.subcommand = self.config["subcommand"] if run else None
    File "/home/clearml/.clearml/venvs-builds.1/3.9/lib/python3.9/site-packages/jsonargparse/namespace.py", line 200, in getitem
    leaf_key, parent_ns, _ = self._parse_required_key(key)
    File "/home/clearml/.clearml/venvs-builds.1/3.9/lib/python3.9/site-packages/jsonargparse/namespace.py", line 161, in _parse_required_key
    raise KeyError(f'Key "{key}" not found in namespace.')
    KeyError: 'Key "subcommand" not found in namespace.' `
    Running locally is no problem at all.

To me it seems like the command line args are not passed when running the script remotely.
Any ideas how to fix this?

  
  
Posted 2 years ago
Votes Newest

Answers 9


I commented this and a bit more on an existing issue: https://github.com/allegroai/clearml/issues/403#issuecomment-1316689416

  
  
Posted 2 years ago

HomelyShells16 Thanks for the detailed write-up and minimal example. I'm running it now too

  
  
Posted 2 years ago

Awesome, thanks! Make sure to check out the linked issue, as I have added more findings there

  
  
Posted 2 years ago

I'm able to reproduce, but your workaround seems to be the best one for now. I tried launching with clearml-task command as well, but we have the same issue there: only argparse arguments are allowed.
AgitatedDove14 any better workaround for this, other than waiting for the jsonargparse issue to be fixed?

  
  
Posted 2 years ago

I found a hack! 🎉 See the issue for details: https://github.com/allegroai/clearml/issues/403#issuecomment-1317362482

  
  
Posted 2 years ago

Hi HomelyShells16 How about doing things this way? does it work for you?
` class ClearmlLightningCLI(LightningCLI):
def init(self, *args, **kwargs):
Task.add_requirements("requirements.txt")
self.task = Task.init(
project_name="example",
task_name="pytorch_lightning_jsonargparse",
)
super().init(*args, **kwargs)

def instantiate_classes(self, *args, **kwargs):
    super().instantiate_classes(*args, **kwargs)
    self.task.execute_remotely(queue_name="Eugene")

if name == "main":
ClearmlLightningCLI(ImageClassifier, seed_everything_default=42, save_config_overwrite=True, run=True) `

  
  
Posted 2 years ago

Thanks SmugDolphin23 for the suggestion.
Sadly this does not solve the problem of having Namespace s inside the config, which cannot be parsed by jsonargparse when running remotely. The Error message is the same as shown https://github.com/allegroai/clearml/issues/403#issuecomment-1316689416 .

  
  
Posted 2 years ago

HomelyShells16 looks like some changes have been made to jsonargparse and pytorch_lightning since we released this binding feature. could you try with jsonargparse==3.19.4 and pytorch_lightning==1.5.0 ? (no namespace parsing hack should be needed with these versions I believe)

  
  
Posted 2 years ago

Also, feel free to open a GH ticket (thank you for reporting this issue :d)

  
  
Posted 2 years ago
1K Views
9 Answers
2 years ago
one year ago
Tags
Similar posts