Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello All, Thanks For This Really Cool Software And Community! I Have A Question On

Hello all, thanks for this really cool software and community!
I have a question on importing local modules when using the Pipelines from Decorators.
Starting from the official example (condensed below), I tried importing a function from a python file in the same directory as the script, and cannot get it to work.
I keep bumping against the ModuleNotFoundError: No module named exception. I’ve tried adding the repo parameter to the component in various ways, and importing the function in various ways but am stuck.
Pipelines from Tasks seem like a second option, but I’d prefer to use decorators if possible.

  
  
Posted 6 months ago
Votes Newest

Answers 20


I had the same issues too on some of my components and I had to specify them in the packages=["package-1", "package-2", ...] in my @PipelineDecorator.component() decorator parameters

  
  
Posted 6 months ago

Thanks to both! Unfortunately the same error occurs with the following code snippet. (Jean, instead of the component parameter you mean packages , right? I could not find the former 🙂 )
@PipelineDecorator.component(..., packages=['someutils']) def step_one(): from someutils import someutilfunc someutilfunc(32)

  
  
Posted 6 months ago

Happens to all! Importing of local packages in these decorated pipelines hasn’t really worked yet (except when running via Pycharm, which seems to make sure that the location of the original code is always in the path)

  
  
Posted 6 months ago

It seems to be working now, by running the Pipeline locally with PipelineDecorator.run_locally() and running the script using the following command:
PYTHONPATH="fill_in_your_current_dir" python pipeline.py
Cloning this in the UI and enqueueing now also allows remote execution.
Calling the script without the PipelineDecorator.run_locally() i.e. running the pipeline remotely still gives the ModuleNotFoundError: No module named

  
  
Posted 6 months ago

Calling the script without the

PipelineDecorator.run_locally()

i.e. running the pipeline remotely still gives the

ModuleNotFoundError: No module named

Do you have the needed module listed on the pipeline controller Task ? (press on the details link, then go to Execution tab / "Installed Packages"

  
  
Posted 6 months ago

I would try to not run it locally but in your execution queues on a remote worker, if that's not it it is likely a bug

  
  
Posted 6 months ago

THat make sense since this function executes your component as classic pythonic functions

  
  
Posted 6 months ago

If I use the PipelineDecorator.debug_pipeline() everything works as expected

  
  
Posted 6 months ago

Yes, also present in the git repo (hosted on gitlab and seemed to correctly retrieve it, couldn’t find any errors about this in the logs)

  
  
Posted 6 months ago

Good to know! Thanks 🙌

  
  
Posted 6 months ago

Thanks a lot! I’m still in the process of setting up, so running on a remote worker has not been successful yet, but I’ll report back on this issue if that fixes it!

  
  
Posted 6 months ago

Just checked and it’s not there, even for the successfully-remotely-ran pipeline. Do note that the needed module is just a local folder with scripts. The differences between the successful pipeline (ran locally and cloned in the UI) vs the errored pipeline (ran remotely) are also very hard to spot to me, they have the exact same Installed Packages and execution details

  
  
Posted 6 months ago

So you think maybe this is functionality that only works when running with an agent? Interesting

  
  
Posted 6 months ago

Do note that the needed module is just a local folder with scripts.

Oh that is the issue, is it in the git repo ?

  
  
Posted 6 months ago

In any case, I’m happy it’s fully running now 🙂

  
  
Posted 6 months ago

I found this thread https://clearml.slack.com/archives/CTK20V944/p1662456550871399?thread_ts=1662453504.528979&cid=CTK20V944 but unsure how this is applicable when running the pipeline locally

  
  
Posted 6 months ago

Does your current running environment has all the requuired packages, cause your pipeline controller has the run_locally() option and I'm not sure if the pipeline orchestrator will follow the same logic of installing all your component's imports as dependencies on remote workers if you do not execute on a distant agent but in locall using that option

  
  
Posted 6 months ago

Hmm that makes sense, btw the PYTHONPATH set by the agent would be the working dir listed under the Task, But if you set the agent.force_git_root_python_path the agent would also add the root git repo to the python path

  
  
Posted 6 months ago

Does it happens for all your packages or for a specific one ?

  
  
Posted 6 months ago

Hi ObedientDolphin41

I keep bumping against the

ModuleNotFoundError: No module named

exception.

Import the package inside the component function (the one you decorated), it will make sure it lists it in the requirements section automatically.
You can also set it manually by passing it to as the "packages" argument on the decorator function:

https://github.com/allegroai/clearml/blob/7016138c849a4f8d0b4d296b319e0b23a1b7bd9e/clearml/automation/controller.py#L3239

  
  
Posted 6 months ago
121 Views
20 Answers
6 months ago
4 months ago
Tags