run how? by using a queue monitored by the autoscaler when running the clearml-session command line tool?
I did not touch the interactive session code at all.
I installed clearml-session
using pip and ran the above command with a task id from a task I'd already run.
I have managed to connect. Our EC2 instances run in a private subnet so the ssh connection was not working for that reason I believe. Once I connected to my VPN it now worked.
(apologies for delay @<1523701087100473344:profile|SuccessfulKoala55> , we got called into meetings. Really appreciate your reactivity!)
No problem 🙂 Once you’ve merged it, what do we need to do to get the updated version please?
I think there's a bug with debugging_session, I'll take a look at it
I'll try to release an RC as soon as possible
It was a debugging session. We haven’t yet tried a “Standard” non-debugging clearml session.
@<1546665634195050496:profile|SolidGoose91> commented there again, but just to make this discussion quicker - did you call Task.init() somewhere? Did you modify the interactive session code in any way? (I saw edits in the uncommitted changes)?
No particular reason. This was our first time trying it and it seemed the quickest way to get off the ground. When I try without I have a similar error trying to connect although that could be due to the instance.
I ran again without the debug mode option and got this error:
>
> Starting Task Execution:
>
>
> Traceback (most recent call last):
> File "/root/.clearml/venvs-builds/3.6/code/interactive_session.py", line 377, in <module>
> from tcp_proxy import TcpProxy
> ModuleNotFoundError: No module named 'tcp_proxy'
>
> Process failed, exit code 1
keep in mind this is not a file that can import other modules since its entire contents (and the contents of other files, tcp_proxy.py for example) is being injected into the session
@<1523701087100473344:profile|SuccessfulKoala55> I think you’ve been tagged in the PR 🙂
We were just trying to run a plain clearml-session on the AWS autoscaler.
@<1546665634195050496:profile|SolidGoose91> this is actually not trivial (I'll check it out now and explain in the PR)
Tagging @<1529271085315395584:profile|AmusedCat74> my colleague with whom we ran into this issue.
no, quite the opposite. I think the issue is with using the debugging session option - why did you use that?
Thanks for the ping, for some reason I didn't see the email 😞
@<1546665634195050496:profile|SolidGoose91> can you share a way to reproduce that? From what I see this should be working...
I did not run clearml-session from within the git repo.