Reputation
Badges 1
662 × Eureka!Any updates @<1523701087100473344:profile|SuccessfulKoala55> ? 🙂
Should this be under the clearml
or clearml-agent
repo?
I'm working on the config object references 😉
From the log you shared, the task is picked up by the
worker_d1bd92a3b039400cbafc60a7a5b1e52b_4e831c4cbaf64e02925b918e9a3a1cf6_<hostname>:gpu0,1
worker
I can try and target the default one if it helps..?
It's pulled from the remote repository, my best guess is that the uncommitted changes apply only after the environment is set up?
Follow up on this btw, from the WebUI/Server POV, I see there's an "Admin" role, etc. Do those have additional views available, such as users etc?
I'll try it out, but I would not like to rewrite that code myself maintain it, that's my point 😅
Or are you suggesting I Task.import_offline_session
?
Now, the original pyhocon does support include statements as you mentioned - https://github.com/chimpler/pyhocon
Sure SuccessfulKoala55 , and thanks for looking into it.
As an alternative (for now, or in general), we could consider reverting back to pip. The issue we encounter is that we have a monorepo, so frozen requirements should specify relative paths, but pip freeze
does not seem to do that, so ClearML also fails in pip
mode
Thanks @<1537605940121964544:profile|EnthusiasticShrimp49> ! That’s definitely the route I was hoping to go, but the create_function_task
is still a bit of a mystery, as I’d like to use an entire class with relevant logic and proper serialization for inputs, and potentially I’ll need to add more “helper functions” (as in the case of DataTransformationStep
, for example). Any thoughts on that? 🤔
Just a side note - the 1.1.1 notice keeps popping up even though the server is at 1.1.1 (and I've cleared browser cache etc)
Yes; I tried running it both outside venv and inside a venv. No idea why it uses 2.7?
I also tried switching to dockerized mode now, getting the same issue 🤔
Hmmm maybe 🤔 I thought that was expected behavior from poetry side actually
Don't even need to specify json=...
😉 Thanks!
It could be related to ClearML agent or server then. We temporarily upload a given .env file to internal S3 bucket (cache), then switch to remote execution. When the remote execution starts, it first looks for this .env file, downloads it using StorageManager, uses dotenv, and then continues the execution normally
Thanks! I'll wait for the release note/docs update 😁
But since this has come up a lot recently, any updates on #340? 😍
Opened a matching feature request issue for this -> https://github.com/allegroai/clearml/issues/418
It's not exactly "debugging", but rather a description of the generated model/framework (generated with pygraphviz).
Right so it uses whatever version is available on the agent.
Yeah it would be nice to have either a poetry_version
(a-la https://github.com/allegroai/clearml-agent/blob/5afb604e3d53d3f09dd6de81fe0a494dacb2e94d/docs/clearml.conf#L62 ), rename the latter to manager_version
, or just install from the captured environment, etc? 🤔
Oh nono, more like:
- Create a pipeline
- Add N steps to it
- Run the pipeline
- It fails/succeeds, the user does something with the output
- The user would like to add/modify some steps based on the results now (after closer inspection).I wonder if at (5), do I have to recreate the pipeline every time? 🤔
We have an internal mono-repo and some of the packages are required - they’re all available correctly for the controller, only some are required for the individual tasks, but the “magic” doesn’t happen 😞
That is, the controller does not identify them as a requirement, so they’re not installed in the tasks environment.
We're using self hosted account
I’ve tracked it down further, it seems the pigar utility does not apply any smart logic there.
The case we have is the following -
- We have a monorepo, but all modules/libs share a common namespace
foo
; so e.g. working on modulemod
, we usefrom foo.mod import …
- This then looks for a module called
foo
, even though it’s just a namespace - In the dist-info requirement, it seems any hyphen, dot, etc are swapped for an underscore, so our site-packages represents this as `foo_m...