Unanswered
Hi Clearml Community, Is There A Way To Install Additional Packages On Top Of Base Docker Env When Using
Hi @<1523701070390366208:profile|CostlyOstrich36> , I tried out Task.add_requirements way to add packages, but it doesn't seem to be working as I expected. here is the snippet i used to setup this up:
Task.force_store_standalone_script()
add_packages = ["fastparquet"]
for pkg in add_packages:
Task.add_requirements(pkg)
task = Task.init(project_name=project_name, task_name=task_name)
task.set_base_docker(docker_arguments="--env CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=0")
task.execute_remotely(queue_name=queue)
...
master_df.to_parquet(dataset_path) -> this is what I use to test whether fastparquet is installed properly or not
And I would get this complain from pandas:
File "/usr/local/lib/python3.8/dist-packages/pandas/io/parquet.py", line 60, in get_engine
raise ImportError(
ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
58 Views
0
Answers
4 months ago
4 months ago