I want in my CI tests to reproduce a run in an agent
you mean to run it on the CI machine ?
because the env changes and some things break in agents and not locally
That should not happen, no? Maybe there is a bug that needs fixing on clearml-agent ?
I want in my CI tests to reproduce a run in an agent because the env changes and some things break in agents and not locally
JitteryCoyote63 of course there is 🙂Task.debug_simulate_remote_task(task_id="<task_id_here>")
Ho wow! is it possible to not specify a remote task? (If i am working with Task.set_offline(True))
JitteryCoyote63 if this is simulating an agent, the assumption is that the Task was already created, hence the task ID.
If i am working with Task.set_offline(True)
How would the two combine ? I mean off-line is be definition not executed by an agent, what am I missing ?
you mean to run it on the CI machine ?
yes
That should not happen, no? Maybe there is a bug that needs fixing on clearml-agent ?
It just to test that the logic being executed in if not Task.running_locally() is correct
Since I'm assuming there is no actual task to run, and you do not need to setup the environment (is that correct?)
you can do:$ CLEARML_OFFLINE_MODE=1 python3 my_main.pywdyt?
ClearML maintains a github action that sets up a dummy clearml-server,
You have one, it's the http://app.clear.ml (not a dummy one, but for this purpose it will work)
thoughts ?
It's the same but done from outside, you want the same and "offline" as well right?
is it different from Task.set_offline(True)?
Actually I think I am approaching the problem from the wrong angle
In my github action, I should just have a dummy clearml server and run the task there, connecting to this dummy clearml server
so that any error that could arise from communication with the server could be tested
even if I move the Github workers internally where they could have access to the prod server, I am not sure I would like that, because it would pile up test data in the prod server that is not necessary
Because it lives behind a VPN and github workers don’t have access to it
oh, if this is the case, why not use the "main" server?
Because it lives behind a VPN and github workers don’t have access to it
makes sense
If this is the case, I have to admit that combining offline-mode and remote execution makes sense, no?
I’d like to move to a setup where I don’t need these tricks
So there is a hack for it:CLEARML_OFFLINE_MODE=1 python3 my_main.pyWhich is the same as calling Task.set_offline
Then inside the code After the Task.init call:
` task = Task.init(...)
not sure what the if here is?!
Task.debug_simulate_remote_task(task_id="offline-1") `This will make things act as if this is running remotely , i.e. your logic Task.running_remotely() will be called.
Do notice that in remote mode, all the arguments / data is read from the clearml-server into the code, and since this is an offline mode, I'm not sure what you'll be getting. (it seems to be workign as expected, i.e. you get what you started with, but I would make sure again)
Make sense ?
and just run the same code I run production
That said , if you could open a github issue and explain the idea behind it, I think a lot of people will be happy to have such process , i.e. CI process verifying code. And I think we should have a "CI" flag doing exactly what we have in the "hack" wdyt?
It does, tested 🙂 but you should as well
I think the best case scenario would be that ClearML maintains a github action that sets up a dummy clearml-server, so that anyone can use it as a basis to run their tests, so that they just have to change to URL of the server to the local one executed in the github action and they can test seamlessly all their code, wdyt?
mmmh good point actually, I didn’t think about it