I found this thread https://clearml.slack.com/archives/CTK20V944/p1662456550871399?thread_ts=1662453504.528979&cid=CTK20V944 but unsure how this is applicable when running the pipeline locally
Hi ObedientDolphin41
I keep bumping against the
ModuleNotFoundError: No module named
exception.
Import the package inside the component function (the one you decorated), it will make sure it lists it in the requirements section automatically.
You can also set it manually by passing it to as the "packages" argument on the decorator function:
I had the same issues too on some of my components and I had to specify them in the packages=["package-1", "package-2", ...] in my @PipelineDecorator.component() decorator parameters
Thanks to both! Unfortunately the same error occurs with the following code snippet. (Jean, instead of the component parameter you mean packages , right? I could not find the former 🙂 )@PipelineDecorator.component(..., packages=['someutils']) def step_one(): from someutils import someutilfunc someutilfunc(32)
Does it happens for all your packages or for a specific one ?
Happens to all! Importing of local packages in these decorated pipelines hasn’t really worked yet (except when running via Pycharm, which seems to make sure that the location of the original code is always in the path)
Does your current running environment has all the requuired packages, cause your pipeline controller has the run_locally() option and I'm not sure if the pipeline orchestrator will follow the same logic of installing all your component's imports as dependencies on remote workers if you do not execute on a distant agent but in locall using that option
So you think maybe this is functionality that only works when running with an agent? Interesting
If I use the PipelineDecorator.debug_pipeline() everything works as expected
THat make sense since this function executes your component as classic pythonic functions
I would try to not run it locally but in your execution queues on a remote worker, if that's not it it is likely a bug
Thanks a lot! I’m still in the process of setting up, so running on a remote worker has not been successful yet, but I’ll report back on this issue if that fixes it!
It seems to be working now, by running the Pipeline locally with PipelineDecorator.run_locally() and running the script using the following command:PYTHONPATH="fill_in_your_current_dir" python pipeline.py
Cloning this in the UI and enqueueing now also allows remote execution.
Calling the script without the PipelineDecorator.run_locally() i.e. running the pipeline remotely still gives the ModuleNotFoundError: No module named
Calling the script without the
PipelineDecorator.run_locally()
i.e. running the pipeline remotely still gives the
ModuleNotFoundError: No module named
Do you have the needed module listed on the pipeline controller Task ? (press on the details link, then go to Execution tab / "Installed Packages"
Just checked and it’s not there, even for the successfully-remotely-ran pipeline. Do note that the needed module is just a local folder with scripts. The differences between the successful pipeline (ran locally and cloned in the UI) vs the errored pipeline (ran remotely) are also very hard to spot to me, they have the exact same Installed Packages and execution details
In any case, I’m happy it’s fully running now 🙂
Do note that the needed module is just a local folder with scripts.
Oh that is the issue, is it in the git repo ?
Yes, also present in the git repo (hosted on gitlab and seemed to correctly retrieve it, couldn’t find any errors about this in the logs)
Hmm that makes sense, btw the PYTHONPATH set by the agent would be the working dir listed under the Task, But if you set the agent.force_git_root_python_path the agent would also add the root git repo to the python path