SuccessfulKoala55 i want agent should take packages which is there in my system.how can i do that?
Hi SuccessfulKoala55 ,No its taking different version of pacakges if u compare with local execution.
ExasperatedCrab78 ,in my system i have SQLAlchemy version 1.4.39,i have run locally and it is is running fine,when i am trying to run through agent it taking sqlalchemy version 2.0.0 which is not there in my system.and i have provided in requirement.txt SQLAlchemy==1.4.39.
SuccessfulKoala55 I have copied code from None , there is one file pipeline.py in which we have two function pipe.start_locally(run_pipeline_steps_locally=True) and pipe.start(). when i am running code with pipe.start_locally its working fine with package SQLAlchemy version of 1.4.39.when i am trying to run with pipe.strat(), along with running agent it creating virtula environment with SQLAIchemy of version 2. i want 1.4.39 how can i setup?see the attach snap i am getting error for this version
Hi SuperiorCockroach75 , the clearml experiment manager will try to detect your package requirements from its original environment. Meaning that if you run the code and it imports e.g. SQLAlchemy, then it will log the exact version of SQLAlchemy you have installed locally.
When you run only get_data,py locally and have the experiment manager track it, can you then look at the task that is made in the clearml webUI and check the installed packages section? That will be the packages that will be used when executing the task remotely later.
It should be exactly the same version as your local package, which you can check using pip freeze | grep SQL . In this case it could mean your local installed version is 2 and you should downgrade locally, so clearml will pick up the correct version.
If you're using a requirements.txt file like above, be sure to set the version as fixed! e.g.SQLAlchemy==x.x.xinstead of justSQLAlchemy` . In the latter case, python will install the latest available version of the package, which may lead to version 2.
Also, please be aware that by default the agent will use a specific python version (depending on which dockerfile it's using to run tasks in). This could be a different version than your local one and lead to mismatches in package versions.
SmugDolphin23 ,i have added pacakges in pipeline componenet,but it is not taking that version.
Note: i am running this code using agent.
SuperiorCockroach75 did you add specific packages with constraints? Can you share the code?
SuccessfulKoala55 Waiting for your response.
SuccessfulKoala55 I have shared above.
Hi SuperiorCockroach75 Try setting packages in your pipline component to your requirements.txt or simply add the list of packages (with the specific versions). None
ExasperatedCrab78 can you also give brief descriptionhow the agent reuse environment?
I must say I don't really know where this comes from. As far as I understand the agent should install the packages exactly as they are saved on the task itself. Can you go to the original experiment of the pipeline step in question (You can do this by selecting the step and clicking on Full Details" in the info panel), there under the execution tab you should see which version the task detected.
The task itself will try to autodetect the packages, but you can override these if needed using the packages parameter like SmugDolphin23 has said. Can you please send me a screenshot with the correct SQLAlchemy version set using this method? Thanks!
Hi SuperiorCockroach75 , I assume you mean having a clearml-agent running your task remotely. In that case, the task (after being run once locally) already has the python packages requirements embedded into it, so the agent can use these in order to find out the correct packages to install
