So the config loading is not deferred until execution 😞
You don't load the configuration during from clearml import Task config is loaded during Task.init() So you can make all your configuration additions up to the point Task.init() is run
I see, okay that already clarifies some stuff, I'll dig a bit more into this then! Thanks!
Hi UnevenDolphin73 , basically the configuration loading is deferred, but once it's loaded it's not re-loaded. As long as you won't call any ClearML SDK method before setting the env vars, I think you should be fine
SuccessfulKoala55 CostlyOstrich36 actually it is the import statement, just finally got around to the traceback:
File "/home/.../ccmlp/configs/mlops.py", line 4, in <module> from clearml import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/__init__.py", line 4, in <module> from .task import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/task.py", line 31, in <module> from .backend_interface.metrics import Metrics File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/__init__.py", line 2, in <module> from .task import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/task/__init__.py", line 1, in <module> from .task import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/task/task.py", line 31, in <module> from ...binding.artifacts import Artifacts File "/home/.../.venv/lib/python3.8/site-packages/clearml/binding/artifacts.py", line 23, in <module> from ..backend_interface.metrics.events import UploadEvent File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/metrics/__init__.py", line 2, in <module> from .interface import Metrics File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/metrics/interface.py", line 11, in <module> from ..base import InterfaceBase File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/base.py", line 12, in <module> from ..config import config_obj File "/home/.../.venv/lib/python3.8/site-packages/clearml/config/__init__.py", line 13, in <module> config_obj = load_config(Path(__file__).parent) # noqa: F405 File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_api/config/__init__.py", line 15, in load config.load_relative_to(this_module_path, *additional_module_paths) File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 131, in load_relative_to self.reload() File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 192, in reload self.replace(self._reload()) File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 175, in _reload config = functools.reduce( File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 178, in <lambda> self._read_single_file(file_path, verbose=self._verbose), File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 312, in _read_single_file return ConfigFactory.parse_file(file_path) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 100, in parse_file return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 153, in parse_string return ConfigParser().parse(content, basedir, resolve, unresolved_value) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 443, in parse has_unresolved = cls.resolve_substitutions(config, allow_unresolved) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 623, in resolve_substitutions is_optional_resolved, resolved_value = cls._resolve_variable(config, substitution) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 470, in _resolve_variable raise ConfigSubstitutionException( clearml.utilities.pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${CLEARML_ENDPOINT} (line: 2, col: 25)
Setting the endpoint will not be the only thing missing though, so unfortunately that's insufficient 😞
Would it also be loaded on e.g. Task.running_locally() ?
Hmmm Regarding your issue you can use the following env vars to define your endpoint
https://clear.ml/docs/latest/docs/configs/env_vars/#server-connection
What is your usecase? Do you want to change the endpoint mid run?
We load the endpoint (and S3 credentials) from a .env file, so they're not immediately available at the time of from clearml import Task .
It's a convenience thing, rather than exporting many environment variables that are tied together.
That's what I thought too, it should only look for the CLEARML_TASK_ID environment variable?
From looking at the code it should be fine, I think (i.e. won't load the configuration)
Hey SuccessfulKoala55 ! Is the configuration file needed for Task.running_locally() ? This is tightly related with issue #395, where we need additional files for remote execution but have no way to attach them to the task other then using the StorageManager as a temporary cache.