![Profile picture](https://clearml-web-assets.s3.amazonaws.com/scoold/avatars/AmusedCat74.png)
Reputation
Badges 1
29 × Eureka!Yep that's correct. If I have a task which runs every 5 minutes, I don't want a new task every 5 minutes as that will create a lot of tasks over a day. It would be better if I had just one task.
Solved for me as well now.
👍 thanks for clearing that up @<1523701087100473344:profile|SuccessfulKoala55>
Ah, didn’t know that. Yes in that case that would work 👍
@<1523701087100473344:profile|SuccessfulKoala55> Just following up as I figured out what was happening here and could be useful for the future.
The prefilled value for Number of GPUs
in the GCP Autoscaler is 1
.
When one ticks Run in CPU mode (no gpus)
it hides the GPU Type
and Number of GPUs
fields. However, the value which was these fields are still submitted in the API Request (I'm guessing here) when the Autoscaler is launched.
Hence, to get past this, you need to...
I cannot ping api.clear.ml on Ubuntu. Works fine on Mac though.
I have just encountered this. I believe it is because of the clearml-agent 1.7.0 release which added this as default: agent.enable_git_ask_pass: true
To fix, add in agent.enable_git_ask_pass: false
to your config.
@<1537605940121964544:profile|EnthusiasticShrimp49> How do I specify to not attach a gpu? I thought ticking 'Run in CPU Mode' would be sufficient. Is there something else I'm missing?
According to the documentation users.user
should be a valid endpoint?
I don't think there's really a way around this because AWS Lambda doesn't allow for multiprocessing.
Instead, I've resorted to using a clearml Scheduler which runs on a t3.micro instance for jobs which I want to run on a cron.
I’ve had some issues with clearml sessions. I’d be interested in seeing a PR. Would you mind posting a link please?
I believe this was an example report I made for a demo and I've since deleted the tasks which generated it 👍
Is there documentation for this as I was not able to figure this out unfortunately.
If a Task is in the 'Completed' I think the only option is to 'Reset' it (see image). You do clear the previous run execution but I think for a repetitive task this is fine.
Maybe this should only be the case if it is in a 'Completed' state rather than 'Failed'. I can see that in this case you would not want to clear the execution because you would want to see why it Failed. Thoughts?
👍 Thanks for getting back to me.
Another issue I found was that I could only use vpc subnets from the google project I am launching the VMs in.
I cannot use shared vpc subnets from another project. This would be a useful feature to implement as GCP recommends segmenting the cloud estate so that the vpc and VMs are in different projects.
I did not touch the interactive session code at all.
I installed clearml-session
using pip and ran the above command with a task id from a task I'd already run.
The code is quite nested by I've tried to extract out the important parts ( summmary_writer
is a tensorboard logger).
self.figure, (ax1, ax2, axc) = plt.subplots(1, 3, figsize=(total_width, total_height), facecolor="white")
self.summary_writer = self.tb_logger.experiment
self.summary_writer.add_figure(Partition.TRAINING.value, train_plot.figure, global_step=self.current_epoch + 1)
The train_plot.figure
is a matplotlib figure created using seaborn.
Let me know if this...
Is there a way I can do this with the python APIClient or even with the requests library?
I ran again without the debug mode option and got this error:
>
> Starting Task Execution:
>
>
> Traceback (most recent call last):
> File "/root/.clearml/venvs-builds/3.6/code/interactive_session.py", line 377, in <module>
> from tcp_proxy import TcpProxy
> ModuleNotFoundError: No module named 'tcp_proxy'
>
> Process failed, exit code 1
Furthermore, when using APIClient()
, users
is not a valid endpoint at all.
class APIClient(object):
auth = None # type: Any
queues = None # type: Any
tasks = None # type: Any
workers = None # type: Any
events = None # type: Any
models = None # type: Any
projects = None # type: Any
This is taken from clearml/backend_api/session/client/client.py
Thanks Jake. Do you know how I set the GPU count to 0?
I am using ClearML version 1.9.1. In code, I am creating a plot using matplotlib. I am able to see this in Tensorboard but it is not available in ClearML Plots
@<1673863823901069312:profile|BraveToad81>
@<1523701070390366208:profile|CostlyOstrich36> Thank you. Which docker image do you use with this machine image?