ignore the indentation, didn't align when I copy over
Any followup on this question? Recap:
Task,add_requirements() doesn't seem to do install the package from my experiment
Additionally, as alternative of add_requirements() if I can't get it working, is there an example of using docker bash init script
you can point me to
Hi @<1523701070390366208:profile|CostlyOstrich36> , I tried out Task.add_requirements way to add packages, but it doesn't seem to be working as I expected. here is the snippet i used to setup this up:
Task.force_store_standalone_script()
add_packages = ["fastparquet"]
for pkg in add_packages:
Task.add_requirements(pkg)
task = Task.init(project_name=project_name, task_name=task_name)
task.set_base_docker(docker_arguments="--env CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=0")
task.execute_remotely(queue_name=queue)
...
master_df.to_parquet(dataset_path) -> this is what I use to test whether fastparquet is installed properly or not
And I would get this complain from pandas:
File "/usr/local/lib/python3.8/dist-packages/pandas/io/parquet.py", line 60, in get_engine
raise ImportError(
ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
Hi @<1797800418953138176:profile|ScrawnyCrocodile51> , you can use Task.add_requirements
to add any packages. Additionally, you can also install packages with the docker bash init script