Hi @<1545216070686609408:profile|EnthusiasticCow4>
The auto detection of clearml is based on the actual imported packages, not the requirements.txt of your entire python environment. This is why some of them are missing.
That said you can always manually add them

Task.add_requirements("hydra-colorlog") # optional add version="1.2.0"
task = Task.init(...)

(notice to call before Task.init)

That make sense. I was confused what the source was.

is there a limit to the search depth for this?

i’ve got a training script that imports local package files and those items import other local package files. ex:

train.py

from local_package.callbacks import Callbacks

local_package/callbacks.py

from local_package.analysis import Analysis

local_package/analysis.py

import pandas as pd

the original task only lists the following as installed packages:

clearml == 1.9.1rc0
pytorch_lightning == 1.8.6
torchvision == 0.14.1

i’m also using conda, but am using conda for package management, and the local package is installed via poetry.

name: training_env
channels:
  - conda-forge
  - clearml
  - defaults
dependencies:
  - python=3.10
  - black
  - pylint
  - tqdm
  - pyyaml
  - click
  - matplotlib
  - pip
  - poetry
  - pytest
  - scipy
  - scikit-image
  - scikit-learn
  - clearml
  - pip:
      - sox
      - torch
      - torchvision
      - torchaudio
      - pytorch-lightning
      - pandas

i can of course add dependencies to the task manually as shown above, but this feels like a bug? maybe?

actually its missing imports from the second level too

@<1523712386849050624:profile|NastyFox63>

is there a limit to the search depth for this?

Yes, the Task.init auto package listing is Only the first depth (i.e. directly imported),
the reason is that the derivative packages should be resolved by pip, when the agent remotely executes that Task.
Now when the Agent is installing the task the Entire python environment is stored, so that it is always fully reprpoducible,
Make sense ?

okay, so my problem is actually that using a “local” package is not supported—ie i need to pip install the code i’m running and that must correctly specify its dependencies

How so? Installing a local package should work, what am I missing?

if i have code that’s just in a git repo but is not installed in any way, it runs fine if i invoke the entrypoint in a shell. but clearml will not find dependencies in secondary imports (as described above) if the agent someone just clones the repo but does not install the python package in some way.

that using a “local” package is not supported

I see, I think the issue is actually pulling the git repo of the second local package, is that correct ?
(assuming you add the requirement manually, with Task.add_requirements) , is that correct ?