I'm signed up for Pro. Is there some restricted docs site for pro users CostlyOstrich36 ?

(the one created when you executed the code on your laptop

I haven't executed the task myself at all. I just cloned it from the examples that are available in the SaaS console upon account creation - specifically hyper-parameters example under the ClearML Examples project.

Will try this. Thanks for promptly looking into this. Much appreciated!

Would you happen to have a timeline for when the feature might become available?

Ok. I think I misunderstood what you said. I thought you meant you've already opened a bug ticket. If that's not the case, do you want to me create one on github?

agent default python is set to 3.9.7

no problem. Thanks for the information Erez!

so there's no way to do that when running in pip or conda mode?

Sorry for the delay CostlyOstrich36 here's the relevant lines from the console:
` ...
File "/home/binoyloaner/miniconda3/envs/DS974/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/binoyloaner/miniconda3/envs/DS974/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 103, in forward
return F.linear(input, self.weight, self.bias)
File "/home/binoyloaner/miniconda3/envs/DS974/lib/python3....

I haven't had much time to look into this but ran a quick debug and it seems like the exception on the __exit_hook variable is None even though the process failed. So seems like hydra maybe somehow preventing the hook callback from executing correctly. will dig in a bit more next week

The toy task is marked "Failed".

Yes

Could it be the script itself is using vanilla sys.argv and not Argparser ? (edited)

Thanks for bringing this up. Our code uses fire to parse command line args and then sort of hands off to hydra, so yes it does use sys.argv initially. Is this a possible issue?

I think the fire + hydra combination is not an issue anymore. We're going to separate the 2 out, and I tried it last night and argument modification and passing worked fine with hydra only.
In any case, thanks for you help Martin!

Yes I believe it's hydra too, so just learning how CML determines process status will be really helpful

I didn't check with the toy task, I thought the error codes might be an issue here so was just looking for the difference. I'll check for that too.
But for my hydra task, it's always marked completed, never failed

Thanks for confirming AgitatedDove14 . Do you have an approximate timeline as to when the RC might be out? I'm asking cause I'm going to write a workaround for it tomorrow and I'm wondering if I should just wait for the RC to come out.

Got it. Thanks for clearing that up!

yep

Just general company policy - the team should own the workspace, not a single IC on the team so anyone can make changes in case of lets say personnel changes for example

No, we currently don't handle it gracefully. It just crashes. But we do use hydra which does sort of arrests that exception first. I'm wondering if it's Hydra causing this issue. I'll look into it later today

Thanks for getting back Martin. The hydra example fails when i try to queue it to my local with
Starting Task Execution: Traceback (most recent call last): File "hydra_example.py", line 10, in <module> @hydra.main(config_path="config_files", config_name="config") AttributeError: module 'hydra' has no attribute 'main'

https://github.com/allegroai/clearml-server/issues/120

OS - Ubuntu 20.04
Conda - 4.10.3
The agent is running in a conda env with python==3.9.7
Is this the info you were looking for?

I thought the agent created a new conda env and installed all packages, recorded during initial task run, from scratch (except for caching with venv). Is that not the case?

re you running it with an agent (that hydra triggers) ?

you mean clearml-agent? then no, I've been running the process manually up until now

This is great! Thanks!
If I have access to the logs, python env and git commits, is there an API to log those to the experiments too?