I guess what I'm confused about is that the final resolved environment is different between the first manual execution and the reproduced one -- the first runs perfectly fine, the second crashes and fails to make the environment.
the first runs perfectly fine,
Just making sure, running in an agent?
the second crashes
Running inside the same container as the first one ?
Hi @<1523701797800120320:profile|SteadySeagull18>
...the job -> requeue it from the GUI, then a different environment is installed
The way that it works is, in the "originating" (i.e. first manual) execution only the directly imported packages are listed (no derivative packages that re required by the original packages)
But when the agent is reproducing the job, it creates a whole clean venv for the experiment, installs the required packages, then pip resolves the derivatives, and then the agent Freezes the entire venv to be later reproduced exactly as it was.
make sense ?
Yes that was a tricky one, basically always blame pip 🙂
One machine (original parent)
agent.package_manager.type = pip
agent.package_manager.pip_version =
Which would not upgrade the pip and use the preinstalled Unpacking python-pip-whl (20.0.2-5ubuntu1.10)
The other one has:
agent.package_manager.type = pip
agent.package_manager.pip_version.0 = <20.2 ; python_version < '3.10'
agent.package_manager.pip_version.1 = <22.3 ; python_version >\= '3.10'
and it installed - Successfully installed pip-20.1.1
which for some reason tried to install "pycairo" even though it was not listed anywhere
One suggestion is to make sure all agents have the same configuration. Another is to add pip into the "installed packages" section.
(Notice that in the next release we will specifically include it there, to avoid these kind of scenarios)
Woo, what a doozy. Thanks for the debug @<1523701205467926528:profile|AgitatedDove14> ! Will move forward with your suggestions.
Woo, what a doozy.
yeah those "broken" pip versions are making our life hard ...