BTW: submodule diff should always get stored, can you provide some error logs on fail cases?

It's hard to pinpoint, I have a repository with several submodules, some changes are found, others not. I've added untracked files, but sometimes it simply doesn't pick them. Running git diff --submodule=diff (which is the same command in the GitDetector class) always return the correct results

Before manually modifying the diff:
If you have local commits (i.e. un-pushed) this might fail the diff apply, in that case you can set the following in your clearml.conf

I have both:
store_uncommitted_code_diff: true
and
store_code_diff_from_remote: true
set

weird, the other versions work just fine

*Actually looking at the code, when you call Task.create(...) it will always store the diff from the remote server.
Could that be the issue?

To edit the Task's diff:
task.update_task(dict(script=dict(diff='DIFF TEXT HERE')))

additionally, I found is that clearml==1.0.5 package is able to find these partial changes, newer versions find nothing at all, maybe it's because it's always comparing against remote

AgitatedDove14 I want to schedule bulk tasks to run via agents, so I'm running create

# pip install clearml==1.0.6rc2 Collecting clearml==1.0.6rc2 Downloading clearml-1.0.6rc2-py2.py3-none-any.whl (1.0 MB) |████████████████████████████████| 1.0 MB 12.1 MB/s Requirement already satisfied: pyjwt<3.0.0,>=1.6.4 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.1.0) Requirement already satisfied: PyYAML>=3.12 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (5.3.1) Requirement already satisfied: numpy>=1.10 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (1.19.2) Requirement already satisfied: Pillow>=4.1.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (8.0.1) Requirement already satisfied: attrs>=18.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (20.3.0) Requirement already satisfied: humanfriendly>=2.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (10.0) Requirement already satisfied: jsonschema>=2.6.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (3.2.0) Requirement already satisfied: psutil>=3.4.2 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (5.7.2) Requirement already satisfied: python-dateutil>=2.6.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.8.1) Requirement already satisfied: six>=1.11.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (1.15.0) Requirement already satisfied: requests>=2.20.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.24.0) Requirement already satisfied: furl>=2.0.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.1.3) Requirement already satisfied: pyparsing>=2.0.3 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.4.7) Requirement already satisfied: urllib3>=1.21.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (1.25.11) Requirement already satisfied: future>=0.16.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (0.18.2) Requirement already satisfied: pathlib2>=2.3.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.3.5) Requirement already satisfied: setuptools in /opt/anaconda/lib/python3.8/site-packages (from jsonschema>=2.6.0->clearml==1.0.6rc2) (50.3.1.post20201107) Requirement already satisfied: pyrsistent>=0.14.0 in /opt/anaconda/lib/python3.8/site-packages (from jsonschema>=2.6.0->clearml==1.0.6rc2) (0.17.3) Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.0.6rc2) (2020.6.20) Requirement already satisfied: chardet<4,>=3.0.2 in /opt/anaconda/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.0.6rc2) (3.0.4) Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.0.6rc2) (2.10) Requirement already satisfied: orderedmultidict>=1.0.1 in /opt/anaconda/lib/python3.8/site-packages (from furl>=2.0.0->clearml==1.0.6rc2) (1.0.1) Installing collected packages: clearml Attempting uninstall: clearml Found existing installation: clearml 1.0.5 Uninstalling clearml-1.0.5: Successfully uninstalled clearml-1.0.5 Rolling back uninstall of clearml Moving to /opt/anaconda/bin/clearml-data from /tmp/pip-uninstall-l1yaec_v/clearml-data Moving to /opt/anaconda/bin/clearml-init from /tmp/pip-uninstall-l1yaec_v/clearml-init Moving to /opt/anaconda/bin/clearml-task from /tmp/pip-uninstall-l1yaec_v/clearml-task Moving to /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/clearml/ from /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/~learml Moving to /opt/anaconda/lib/python3.8/site-packages/clearml-1.0.5.dist-info/ from /opt/anaconda/lib/python3.8/site-packages/~learml-1.0.5.dist-info ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/opt/anaconda/lib/python3.8/site-packages/clearml/__init__.py'

it seems it's following the path of the script i'm using to task.create, eg:

The folder it should run it is the script path you are passing (i.e. "script=ep_fn," )
Wrong path would imply that is it not finding the correct repository, is that the case ?

one final addition, this flag: store_code_diff_from_remote don't seem to change anything in regards of this issue

i'll be using the update_task, that worked just fine, thanks 🙂

That makes, sense, will investigate, thanks

 I want to schedule bulk tasks to run via agents, so I'm running 

create

I see, that makes sense.

specially when dealing with submodules,

BTW: submodule diff should always get stored, can you provide some error logs on fail cases?

Before manually modifying the diff:
If you have local commits (i.e. un-pushed) this might fail the diff apply, in that case you can set the following in your clearml.conf
store_code_diff_from_remote: truehttps://github.com/allegroai/clearml/blob/1f82b0c4010799be6157f5c845c7f6ac48e71c0c/docs/clearml.conf#L157

Thank you!

one thing i noticed is that it's not able to find the branch name on >=1.0.6x , while on 1.0.5 it can

That might be it! let me check the code again...

ShakyJellyfish91 what exactly are you passing to Task.create?
Could it be you are only passing script= and leaving repo= None ?

Can you see the repo itself ? the commit id ?

additionally, I found is that clearml==1.0.5 package is able to find these partial changes, newer versions find nothing at all, maybe it's because it's always comparing against remote

Hmm it was always from remote...
it is actually doing the following:
git rev-parse --abbrev-ref --symbolic-full-name @{u}Then with the branch name output,
git diff --submodule=diff <add_branch_name_here>

Thanks ShakyJellyfish91 ! please let me know what you come up with, I would love for us to fix this issue.

Thanks ShakyJellyfish91 this really helps to narrow it down!
Let me see what I can find

nm, i restarted the instance, now install works, let me see the diffs

Change to add_missing_installed_packages=False, here, and see if you end up with git diff
https://github.com/allegroai/clearml/blob/1f82b0c4010799be6157f5c845c7f6ac48e71c0c/clearml/backend_interface/task/populate.py#L158

ShakyJellyfish91 can you check if version 1.0.6rc2 can find the changes ?

AgitatedDove14 hm, that fails to install for me

Installing collected packages: clearml Attempting uninstall: clearml Found existing installation: clearml 1.0.5 Uninstalling clearml-1.0.5: Successfully uninstalled clearml-1.0.5 Rolling back uninstall of clearml Moving to /opt/anaconda/bin/clearml-data from /tmp/pip-uninstall-tcd44rxv/clearml-data Moving to /opt/anaconda/bin/clearml-init from /tmp/pip-uninstall-tcd44rxv/clearml-init Moving to /opt/anaconda/bin/clearml-task from /tmp/pip-uninstall-tcd44rxv/clearml-task Moving to /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/clearml/ from /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/~learml Moving to /opt/anaconda/lib/python3.8/site-packages/clearml-1.0.5.dist-info/ from /opt/anaconda/lib/python3.8/site-packages/~learml-1.0.5.dist-info ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/opt/anaconda/lib/python3.8/site-packages/clearml/__init__.py'

no diff still

one thing i noticed is that it's not able to find the branch name on >=1.0.6x , while on 1.0.5 it can

so,
1.0.5 = yes diffs
1.0.6rc1 = no diffs
1.0.6rc2 = no diffs

1.0.5 vs 1.0.6+

pip install clearml==1.0.6rc2Did not work?!

Eg, i'm creating a task using 

clearml.Task.create

 , often it doesn't properly get the git diff correctly,

ShakyJellyfish91 Task.create does not store any "git diff" automatically, is there a reason not to use Task.init ?

store_code_diff_from_remote

 don't seem to change anything in regards of this issue

Correct, it is always from remote

i'll be using the update_task, that worked just fine, thanks 

 (edite

Sure thing.

ShakyJellyfish91 , I took a quick look at the diff between the versions can you hack a non working version (preferably the latest) and verify the issue for me?

AgitatedDove14

I'm creating it like this
task = clearml.Task.create( task_name=task_name, project_name=settings.project, repo=settings.tps_repo, script=ep_fn, docker=settings.image, docker_args="-e TRAIN_CONFIG={} -e VALIDATION_CONFIG={} -e EXP={} -e EXP_CWD={} -e EXP_GIT={} {}".format( settings.train, settings.validation, settings.exp, settings.cwd, settings.cwd_git, settings.docker_args ), add_task_init_call=False )
it might be that the problem is the path where clearml is doing the git diff, it seems it's following the path of the script i'm using to task.create, eg:

env PATH=/path/to/clearml_run.py
cd /repo/base/path
python3 clerarml_run.py <args>

clearml will try to find repos at /path/to/clearml_run.py