So the config loading is not deferred until execution 😞
SuccessfulKoala55 CostlyOstrich36 actually it is the import statement, just finally got around to the traceback:
File "/home/.../ccmlp/configs/mlops.py", line 4, in <module> from clearml import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/__init__.py", line 4, in <module> from .task import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/task.py", line 31, in <module> from .backend_interface.metrics import Metrics File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/__init__.py", line 2, in <module> from .task import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/task/__init__.py", line 1, in <module> from .task import Task File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/task/task.py", line 31, in <module> from ...binding.artifacts import Artifacts File "/home/.../.venv/lib/python3.8/site-packages/clearml/binding/artifacts.py", line 23, in <module> from ..backend_interface.metrics.events import UploadEvent File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/metrics/__init__.py", line 2, in <module> from .interface import Metrics File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/metrics/interface.py", line 11, in <module> from ..base import InterfaceBase File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_interface/base.py", line 12, in <module> from ..config import config_obj File "/home/.../.venv/lib/python3.8/site-packages/clearml/config/__init__.py", line 13, in <module> config_obj = load_config(Path(__file__).parent) # noqa: F405 File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_api/config/__init__.py", line 15, in load config.load_relative_to(this_module_path, *additional_module_paths) File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 131, in load_relative_to self.reload() File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 192, in reload self.replace(self._reload()) File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 175, in _reload config = functools.reduce( File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 178, in <lambda> self._read_single_file(file_path, verbose=self._verbose), File "/home/.../.venv/lib/python3.8/site-packages/clearml/backend_config/config.py", line 312, in _read_single_file return ConfigFactory.parse_file(file_path) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 100, in parse_file return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 153, in parse_string return ConfigParser().parse(content, basedir, resolve, unresolved_value) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 443, in parse has_unresolved = cls.resolve_substitutions(config, allow_unresolved) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 623, in resolve_substitutions is_optional_resolved, resolved_value = cls._resolve_variable(config, substitution) File "/home/.../.venv/lib/python3.8/site-packages/clearml/utilities/pyhocon/config_parser.py", line 470, in _resolve_variable raise ConfigSubstitutionException( clearml.utilities.pyhocon.exceptions.ConfigSubstitutionException: Cannot resolve variable ${CLEARML_ENDPOINT} (line: 2, col: 25)
That's what I thought too, it should only look for the CLEARML_TASK_ID environment variable?
From looking at the code it should be fine, I think (i.e. won't load the configuration)
Hey SuccessfulKoala55 ! Is the configuration file needed for Task.running_locally() ? This is tightly related with issue #395, where we need additional files for remote execution but have no way to attach them to the task other then using the StorageManager as a temporary cache.
Hi UnevenDolphin73 , basically the configuration loading is deferred, but once it's loaded it's not re-loaded. As long as you won't call any ClearML SDK method before setting the env vars, I think you should be fine
Would it also be loaded on e.g. Task.running_locally() ?
I see, okay that already clarifies some stuff, I'll dig a bit more into this then! Thanks!
You don't load the configuration during from clearml import Task config is loaded during Task.init() So you can make all your configuration additions up to the point Task.init() is run
Setting the endpoint will not be the only thing missing though, so unfortunately that's insufficient 😞
We load the endpoint (and S3 credentials) from a .env file, so they're not immediately available at the time of from clearml import Task .
It's a convenience thing, rather than exporting many environment variables that are tied together.
Hmmm Regarding your issue you can use the following env vars to define your endpoint
https://clear.ml/docs/latest/docs/configs/env_vars/#server-connection
What is your usecase? Do you want to change the endpoint mid run?