Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello All, Thanks For This Really Cool Software And Community! I Have A Question On

Hello all, thanks for this really cool software and community!
I have a question on importing local modules when using the Pipelines from Decorators.
Starting from the official example (condensed below), I tried importing a function from a python file in the same directory as the script, and cannot get it to work.
I keep bumping against the ModuleNotFoundError: No module named exception. I’ve tried adding the repo parameter to the component in various ways, and importing the function in various ways but am stuck.
Pipelines from Tasks seem like a second option, but I’d prefer to use decorators if possible.

  
  
Posted one year ago
Votes Newest

Answers 20


If I use the PipelineDecorator.debug_pipeline() everything works as expected

  
  
Posted one year ago

Thanks a lot! I’m still in the process of setting up, so running on a remote worker has not been successful yet, but I’ll report back on this issue if that fixes it!

  
  
Posted one year ago

Hi ObedientDolphin41

I keep bumping against the

ModuleNotFoundError: No module named

exception.

Import the package inside the component function (the one you decorated), it will make sure it lists it in the requirements section automatically.
You can also set it manually by passing it to as the "packages" argument on the decorator function:

https://github.com/allegroai/clearml/blob/7016138c849a4f8d0b4d296b319e0b23a1b7bd9e/clearml/automation/controller.py#L3239

  
  
Posted one year ago

Good to know! Thanks 🙌

  
  
Posted one year ago

It seems to be working now, by running the Pipeline locally with PipelineDecorator.run_locally() and running the script using the following command:
PYTHONPATH="fill_in_your_current_dir" python pipeline.py
Cloning this in the UI and enqueueing now also allows remote execution.
Calling the script without the PipelineDecorator.run_locally() i.e. running the pipeline remotely still gives the ModuleNotFoundError: No module named

  
  
Posted one year ago

Just checked and it’s not there, even for the successfully-remotely-ran pipeline. Do note that the needed module is just a local folder with scripts. The differences between the successful pipeline (ran locally and cloned in the UI) vs the errored pipeline (ran remotely) are also very hard to spot to me, they have the exact same Installed Packages and execution details

  
  
Posted one year ago

Do note that the needed module is just a local folder with scripts.

Oh that is the issue, is it in the git repo ?

  
  
Posted one year ago

So you think maybe this is functionality that only works when running with an agent? Interesting

  
  
Posted one year ago

Calling the script without the

PipelineDecorator.run_locally()

i.e. running the pipeline remotely still gives the

ModuleNotFoundError: No module named

Do you have the needed module listed on the pipeline controller Task ? (press on the details link, then go to Execution tab / "Installed Packages"

  
  
Posted one year ago

I had the same issues too on some of my components and I had to specify them in the packages=["package-1", "package-2", ...] in my @PipelineDecorator.component() decorator parameters

  
  
Posted one year ago

Does it happens for all your packages or for a specific one ?

  
  
Posted one year ago

THat make sense since this function executes your component as classic pythonic functions

  
  
Posted one year ago

I found this thread https://clearml.slack.com/archives/CTK20V944/p1662456550871399?thread_ts=1662453504.528979&cid=CTK20V944 but unsure how this is applicable when running the pipeline locally

  
  
Posted one year ago

Thanks to both! Unfortunately the same error occurs with the following code snippet. (Jean, instead of the component parameter you mean packages , right? I could not find the former 🙂 )
@PipelineDecorator.component(..., packages=['someutils']) def step_one(): from someutils import someutilfunc someutilfunc(32)

  
  
Posted one year ago

In any case, I’m happy it’s fully running now 🙂

  
  
Posted one year ago

Hmm that makes sense, btw the PYTHONPATH set by the agent would be the working dir listed under the Task, But if you set the agent.force_git_root_python_path the agent would also add the root git repo to the python path

  
  
Posted one year ago

Yes, also present in the git repo (hosted on gitlab and seemed to correctly retrieve it, couldn’t find any errors about this in the logs)

  
  
Posted one year ago

Does your current running environment has all the requuired packages, cause your pipeline controller has the run_locally() option and I'm not sure if the pipeline orchestrator will follow the same logic of installing all your component's imports as dependencies on remote workers if you do not execute on a distant agent but in locall using that option

  
  
Posted one year ago

I would try to not run it locally but in your execution queues on a remote worker, if that's not it it is likely a bug

  
  
Posted one year ago

Happens to all! Importing of local packages in these decorated pipelines hasn’t really worked yet (except when running via Pycharm, which seems to make sure that the location of the original code is always in the path)

  
  
Posted one year ago
621 Views
20 Answers
one year ago
one year ago
Tags