=fil-profile= Preparing to write to fil-result/2021-08-19T20:23:30.905
=fil-profile= Wrote memory usage flamegraph to fil-result/2021-08-19T20:23:30.905/out-of-memory.svg
=fil-profile= Wrote memory usage flamegraph to fil-result/2021-08-19T20:23:30.905/out-of-memory-reversed.svg
FiercePenguin76 in the Tasks execution tab, under "script path", change to "-m filprofiler run catboost_train.py".
It should work (assuming the "catboost_train.py" is in the working directory).
and I have no way to save those as clearml artifacts
You could do (at the end of the codetask.upload_artifact('profiler', Path('./fil-result/'))
wdyt?
[.]$ /root/.clearml/venvs-builds/3.8/bin/python -u '/root/.clearml/venvs-builds/3.8/code/-m filprofiler run catboost_train.py'
not a full log yet (will have to inspect it to not have any non-public info), but something potentially interesting
the task is running, but no log output from fil-profiler (when ran totally locally, then it does some logging at the very beginning)
but this will be invoked before fil-profiler starts generating them
I thought it will flush in the background đ
You can however configure the profiler to a specific folder, then mount the folder to the host machine:
In the "base docker args" section add -v /host/folder/for/profiler:/inside/container/profile
Does it had any errors in the local run up to the task.execute_remotely
call?
You can try hack it, in the UI, under EXECUTION tab, add this prefix (-m scalene) to the script path, something like: - scalene my_clearml_task.py
, can you try with it? (make sure you install scalene or have it under your installed packages)
nope, I need to contact devops team for that, that can happen not earlier than Monday
[.]$ /root/.clearml/venvs-builds/3.8/bin/python -u '/root/.clearml/venvs-builds/3.8/code/-m filprofiler run catboost_train.py'
doesnât look good
Adding venv into cache: /root/.clearml/venvs-builds/3.8
Running task id [8c65e88253034bd5a8dba923062066c1]:
[pipelines]$ /root/.clearml/venvs-builds/3.8/bin/python -u -m filprofiler run catboost_train.py
now the problem is: fil-profiler persists the reports and then exits
I guess thatâs the only option, thanks for your help
how do you run it locally? the same?
so probably, my question can be transformed into: âCan I have control over what command is used to start my script on clearml-agentâ
But here you can see why it didnât succeed
but this time they were all present, and the command was run as expected:
for some reason, when I ran it previous time, then repo, commit and working dir were all empty
Adding venv into cache: /root/.clearml/venvs-builds/3.8 Running task id [aa2aca203f6b46b0843699d1da373b25]: [.]$ /root/.clearml/venvs-builds/3.8/bin/python -u '/root/.clearml/venvs-builds/3.8/code/-m filprofiler run catboost_train.py'
hmm that is odd.
Can you send the full log ?
No worries, I'll see if I can replicate it anyhow
btw, you can also run using python -m filprofiler run my_clearml_task.py
âassuming the âcatboost_train.pyâ is in the working directoryâ - maybe I get this part wrong?
So maybe the path is related to the fact I have venv caching on?
hmmm could be...
Can you quickly disable the caching and try ?
but this will be invoked before fil-profiler starts generating them
So maybe the path is related to the fact I have venv caching on?
and I have no way to save those as clearml artifacts