Reputation
Badges 1
35 × Eureka!Hi @<1523701205467926528:profile|AgitatedDove14> , I already tried to check manually in the web UI for some anomalous file, i.e. by downloading the log files or exporting the metrics plots, but I couldn't find anything that takes more than 100KB, and I'm already at 300MB of usage with just 15 tasks. It's not possible to get more info using some python APIs?
Yes, or even just something like task.get_size()
So the longest experiments I have takes ~800KB in logs. I have tens of plotly plots logged manually, how are they stored internally? I tried to export them to json and they don't take more than 50KB each, but maybe they take more memory internally?
What is not clear to me is how you would use the callbacks to run the step locally. Are there some properties that needs to be set in the task? I see that there is a start_controller_locally
option for the main @PipelineDecorator.pipeline
, but I don't see it for @PipelineDecorator.component
I think I found a solution using pipeline_task.move_to_project(new_project_name=f'{config.project_id}/.pipelines/{config.run_name}',
system_tags
=['hidden', 'pipeline'])