Hi SuperiorCockroach75 , I assume you mean having a clearml-agent running your task remotely. In that case, the task (after being run once locally) already has the python packages requirements embedded into it, so the agent can use these in order to find out the correct packages to install
Hi @<1523701087100473344:profile|SuccessfulKoala55> ,No its taking different version of pacakges if u compare with local execution.
@<1523701087100473344:profile|SuccessfulKoala55> i want agent should take packages which is there in my system.how can i do that?
Hi @<1533257278776414208:profile|SuperiorCockroach75> , the clearml experiment manager will try to detect your package requirements from its original environment. Meaning that if you run the code and it imports e.g. SQLAlchemy, then it will log the exact version of SQLAlchemy you have installed locally.
When you run only get_data,py locally and have the experiment manager track it, can you then look at the task that is made in the clearml webUI and check the installed packages section? That will be the packages that will be used when executing the task remotely later.
It should be exactly the same version as your local package, which you can check using pip freeze | grep SQL . In this case it could mean your local installed version is 2 and you should downgrade locally, so clearml will pick up the correct version.
If you're using a requirements.txt file like above, be sure to set the version as fixed! e.g.SQLAlchemy==x.x.xinstead of justSQLAlchemy` . In the latter case, python will install the latest available version of the package, which may lead to version 2.
Also, please be aware that by default the agent will use a specific python version (depending on which dockerfile it's using to run tasks in). This could be a different version than your local one and lead to mismatches in package versions.
@<1533257278776414208:profile|SuperiorCockroach75> did you add specific packages with constraints? Can you share the code?
Hi @<1533257278776414208:profile|SuperiorCockroach75> Try setting packages in your pipline component to your requirements.txt or simply add the list of packages (with the specific versions). None
@<1523701087100473344:profile|SuccessfulKoala55> Waiting for your response.
@<1523701435869433856:profile|SmugDolphin23> ,i have added pacakges in pipeline componenet,but it is not taking that version.
Note: i am running this code using agent.
@<1523701118159294464:profile|ExasperatedCrab78> ,in my system i have SQLAlchemy version 1.4.39,i have run locally and it is is running fine,when i am trying to run through agent it taking sqlalchemy version 2.0.0 which is not there in my system.and i have provided in requirement.txt SQLAlchemy==1.4.39.
Hi @<1533257278776414208:profile|SuperiorCockroach75>
I must say I don't really know where this comes from. As far as I understand the agent should install the packages exactly as they are saved on the task itself. Can you go to the original experiment of the pipeline step in question (You can do this by selecting the step and clicking on Full Details" in the info panel), there under the execution tab you should see which version the task detected.
The task itself will try to autodetect the packages, but you can override these if needed using the packages parameter like @<1523701435869433856:profile|SmugDolphin23> has said. Can you please send me a screenshot with the correct SQLAlchemy version set using this method? Thanks!
@<1523701087100473344:profile|SuccessfulKoala55> I have shared above.
@<1523701118159294464:profile|ExasperatedCrab78> can you also give brief descriptionhow the agent reuse environment?
@<1523701087100473344:profile|SuccessfulKoala55> I have copied code from None , there is one file pipeline.py in which we have two function pipe.start_locally(run_pipeline_steps_locally=True) and pipe.start(). when i am running code with pipe.start_locally its working fine with package SQLAlchemy version of 1.4.39.when i am trying to run with pipe.strat(), along with running agent it creating virtula environment with SQLAIchemy of version 2. i want 1.4.39 how can i setup?see the attach snap i am getting error for this version