Yes I was thinking a separate branch.
The main issue with telling git to skip submodules is that it will be easily forgotten and will break stuff. BTW the git repo itself is cached so the second time there is no actual pull. Lastly it's not clear on where one could pass a git argument per task. Wdyt?
None of these submodules are required for the tasks, they are there for a different part of the project dealing with data generation.
So even having them fetched (even when cached) is quite the delay on the actual task.
Because submodules inside a git are basically a requirement for a git repo to run. Skipping over a few or selecting manually will break the agent. That said maybe shallow clone might be easier or faster. Regardless it should be an environment passed per Task. Feel free to add a GH issue request, if this is not a unique edge case we will add it
Hi @<1694157594333024256:profile|DisturbedParrot38>
You mean how to tell the agent to pull only some submodules of your git?
If this is the case you can actually remove them on your git branch, submodule is a file with a soft link. Wdyt?
Yea, but even though it's cached, it takes quite a long time, because my project has really alot of submodules, due to the submodules having their own submodules as well.
I don't really understand why fetching the submodules is the default.
If there's some or any mechanism that would allow me to constrain what the task sees, it would really help me alot.
just to give an idea about the scale of the problem on my side.
These are the submodules...
`Fetching submodule lvgl_ui_generator
2024-04-27 20:45:34
Fetching submodule lvgl_ui_generator/lv_drivers
Fetching submodule lvgl_ui_generator/lvgl
Fetching submodule lvgl_ui_generator_v2
Fetching submodule lvgl_ui_generator_v2/lv_micropython
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/asf4
2024-04-27 20:45:40
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/axtls
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/berkeley-db-1.xx
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/btstack
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/cyw43-driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/fsp
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/libffi
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/libhydrogen
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/lv_bindings
2024-04-27 20:45:45
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/lv_bindings/lvgl
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/lv_bindings/pycparser
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/lwip
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/mbedtls
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/micropython-lib
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/mynewt-nimble
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/nrfx
2024-04-27 20:45:51
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/nxp_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/btstack
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/cyw43-driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/lwip
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/mbedtls
2024-04-27 20:45:57
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/allwinner
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/bridgetek/ft9xx/ft90x-sdk
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/broadcom
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/broadcom/firmware.wiki
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/gd/nuclei-sdk
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/infineon/mtb-xmclib-cat3
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/microchip
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/mindmotion/mm32sdk
2024-04-27 20:46:03
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/nordic/nrfx
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/nuvoton
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/nxp/lpcopen
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/nxp/mcux-sdk
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/nxp/nxp_sdk
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/raspberry_pi/Pico-PIO-USB
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/renesas/rx
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/silabs/cmsis-dfp-efm32gg12b
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/sony/cxd56/spresense-exported-sdk
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_f0
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_f1
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_f2
2024-04-27 20:46:09
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_f3
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_f4
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_f7
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_g0
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_g4
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_h7
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_l0
2024-04-27 20:46:15
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_l1
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_l4
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_l5
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_u5
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/cmsis_device_wb
2024-04-27 20:46:20
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32f0xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32f1xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32f2xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32f3xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32f4xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32f7xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32g0xx_hal_driver
2024-04-27 20:46:26
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32g4xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32h7xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32l0xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32l1xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32l4xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32l5xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32u5xx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/st/stm32wbxx_hal_driver
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/ti
2024-04-27 20:46:32
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/hw/mcu/wch/ch32v307
Fetching submodule lvgl_ui_generator_v2/lv_micropython/lib/pico-sdk/lib/tinyusb/lib/CMSIS_5
... the list continues ...
Can't I adjust the environment passed to the task, so that it only includes the main repository without the submodules?
The code that is run in regards to clearml is really small.
If there's no mechanism on side of clearml, I might consider just putting that codebase into it's own submodule, making it a different repo without knowledge of the others.
You mean a seperate branch to work in without the submodules linked?
Not really sure how I'd go about doing that.
I'd be more happy with an option to say 'pull_submodules=False'
That is quite neat! You can also put a soft link from the main repo to the submodule for better visibility
I double checked the code it's always being passed 😞
Omg that's a lot of submodules!
It has nothing with what the tasks sees if you are inside a git repo you will have to cone it on the remote machine. Let me check in the code maybe you have a workaround
Well.. I'll guess I'll do the workaround then of putting the main code into a submodule and have everything run from there