Thank you!
one thing i noticed is that it's not able to find the branch name on >=1.0.6x , while on 1.0.5 it can
That might be it! let me check the code again...
Can you see the repo itself ? the commit id ?
Thanks ShakyJellyfish91 this really helps to narrow it down!
Let me see what I can find
Change to add_missing_installed_packages=False, here, and see if you end up with git diff
https://github.com/allegroai/clearml/blob/1f82b0c4010799be6157f5c845c7f6ac48e71c0c/clearml/backend_interface/task/populate.py#L158
# pip install clearml==1.0.6rc2 Collecting clearml==1.0.6rc2 Downloading clearml-1.0.6rc2-py2.py3-none-any.whl (1.0 MB) |████████████████████████████████| 1.0 MB 12.1 MB/s Requirement already satisfied: pyjwt<3.0.0,>=1.6.4 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.1.0) Requirement already satisfied: PyYAML>=3.12 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (5.3.1) Requirement already satisfied: numpy>=1.10 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (1.19.2) Requirement already satisfied: Pillow>=4.1.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (8.0.1) Requirement already satisfied: attrs>=18.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (20.3.0) Requirement already satisfied: humanfriendly>=2.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (10.0) Requirement already satisfied: jsonschema>=2.6.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (3.2.0) Requirement already satisfied: psutil>=3.4.2 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (5.7.2) Requirement already satisfied: python-dateutil>=2.6.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.8.1) Requirement already satisfied: six>=1.11.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (1.15.0) Requirement already satisfied: requests>=2.20.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.24.0) Requirement already satisfied: furl>=2.0.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.1.3) Requirement already satisfied: pyparsing>=2.0.3 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.4.7) Requirement already satisfied: urllib3>=1.21.1 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (1.25.11) Requirement already satisfied: future>=0.16.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (0.18.2) Requirement already satisfied: pathlib2>=2.3.0 in /opt/anaconda/lib/python3.8/site-packages (from clearml==1.0.6rc2) (2.3.5) Requirement already satisfied: setuptools in /opt/anaconda/lib/python3.8/site-packages (from jsonschema>=2.6.0->clearml==1.0.6rc2) (50.3.1.post20201107) Requirement already satisfied: pyrsistent>=0.14.0 in /opt/anaconda/lib/python3.8/site-packages (from jsonschema>=2.6.0->clearml==1.0.6rc2) (0.17.3) Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.0.6rc2) (2020.6.20) Requirement already satisfied: chardet<4,>=3.0.2 in /opt/anaconda/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.0.6rc2) (3.0.4) Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.0.6rc2) (2.10) Requirement already satisfied: orderedmultidict>=1.0.1 in /opt/anaconda/lib/python3.8/site-packages (from furl>=2.0.0->clearml==1.0.6rc2) (1.0.1) Installing collected packages: clearml Attempting uninstall: clearml Found existing installation: clearml 1.0.5 Uninstalling clearml-1.0.5: Successfully uninstalled clearml-1.0.5 Rolling back uninstall of clearml Moving to /opt/anaconda/bin/clearml-data from /tmp/pip-uninstall-l1yaec_v/clearml-data Moving to /opt/anaconda/bin/clearml-init from /tmp/pip-uninstall-l1yaec_v/clearml-init Moving to /opt/anaconda/bin/clearml-task from /tmp/pip-uninstall-l1yaec_v/clearml-task Moving to /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/clearml/ from /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/~learml Moving to /opt/anaconda/lib/python3.8/site-packages/clearml-1.0.5.dist-info/ from /opt/anaconda/lib/python3.8/site-packages/~learml-1.0.5.dist-info ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/opt/anaconda/lib/python3.8/site-packages/clearml/__init__.py'
AgitatedDove14
I'm creating it like thistask = clearml.Task.create( task_name=task_name, project_name=settings.project, repo=settings.tps_repo, script=ep_fn, docker=settings.image, docker_args="-e TRAIN_CONFIG={} -e VALIDATION_CONFIG={} -e EXP={} -e EXP_CWD={} -e EXP_GIT={} {}".format( settings.train, settings.validation, settings.exp, settings.cwd, settings.cwd_git, settings.docker_args ), add_task_init_call=False )
it might be that the problem is the path where clearml is doing the git diff, it seems it's following the path of the script i'm using to task.create, eg:
env PATH=/path/to/clearml_run.py
cd /repo/base/path
python3 clerarml_run.py <args>
clearml will try to find repos at /path/to/clearml_run.py
so,
1.0.5 = yes diffs
1.0.6rc1 = no diffs
1.0.6rc2 = no diffs
additionally, I found is that clearml==1.0.5 package is able to find these partial changes, newer versions find nothing at all, maybe it's because it's always comparing against remote
Hmm it was always from remote...
it is actually doing the following:git rev-parse --abbrev-ref --symbolic-full-name @{u}Then with the branch name output,git diff --submodule=diff <add_branch_name_here>
I want to schedule bulk tasks to run via agents, so I'm running
create
I see, that makes sense.
specially when dealing with submodules,
BTW: submodule diff should always get stored, can you provide some error logs on fail cases?
Before manually modifying the diff:
If you have local commits (i.e. un-pushed) this might fail the diff apply, in that case you can set the following in your clearml.confstore_code_diff_from_remote: truehttps://github.com/allegroai/clearml/blob/1f82b0c4010799be6157f5c845c7f6ac48e71c0c/docs/clearml.conf#L157
it seems it's following the path of the script i'm using to task.create, eg:
The folder it should run it is the script path you are passing (i.e. "script=ep_fn," )
Wrong path would imply that is it not finding the correct repository, is that the case ?
ShakyJellyfish91 what exactly are you passing to Task.create?
Could it be you are only passing script= and leaving repo= None ?
*Actually looking at the code, when you call Task.create(...) it will always store the diff from the remote server.
Could that be the issue?
To edit the Task's diff:task.update_task(dict(script=dict(diff='DIFF TEXT HERE')))
store_code_diff_from_remote
don't seem to change anything in regards of this issue
Correct, it is always from remote
i'll be using the update_task, that worked just fine, thanks
(edite
Sure thing.
ShakyJellyfish91 , I took a quick look at the diff between the versions can you hack a non working version (preferably the latest) and verify the issue for me?
pip install clearml==1.0.6rc2Did not work?!
weird, the other versions work just fine
Thanks ShakyJellyfish91 ! please let me know what you come up with, I would love for us to fix this issue.
ShakyJellyfish91 can you check if version 1.0.6rc2 can find the changes ?
AgitatedDove14 hm, that fails to install for me
Installing collected packages: clearml Attempting uninstall: clearml Found existing installation: clearml 1.0.5 Uninstalling clearml-1.0.5: Successfully uninstalled clearml-1.0.5 Rolling back uninstall of clearml Moving to /opt/anaconda/bin/clearml-data from /tmp/pip-uninstall-tcd44rxv/clearml-data Moving to /opt/anaconda/bin/clearml-init from /tmp/pip-uninstall-tcd44rxv/clearml-init Moving to /opt/anaconda/bin/clearml-task from /tmp/pip-uninstall-tcd44rxv/clearml-task Moving to /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/clearml/ from /opt/anaconda/envs/cvedia_env/lib/python3.8/site-packages/~learml Moving to /opt/anaconda/lib/python3.8/site-packages/clearml-1.0.5.dist-info/ from /opt/anaconda/lib/python3.8/site-packages/~learml-1.0.5.dist-info ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/opt/anaconda/lib/python3.8/site-packages/clearml/__init__.py'
AgitatedDove14 I want to schedule bulk tasks to run via agents, so I'm running create
no diff still
one thing i noticed is that it's not able to find the branch name on >=1.0.6x , while on 1.0.5 it can
nm, i restarted the instance, now install works, let me see the diffs
additionally, I found is that clearml==1.0.5 package is able to find these partial changes, newer versions find nothing at all, maybe it's because it's always comparing against remote
That makes, sense, will investigate, thanks
one final addition, this flag: store_code_diff_from_remote don't seem to change anything in regards of this issue
i'll be using the update_task, that worked just fine, thanks 🙂
Eg, i'm creating a task using
clearml.Task.create
, often it doesn't properly get the git diff correctly,
ShakyJellyfish91 Task.create does not store any "git diff" automatically, is there a reason not to use Task.init ?
BTW: submodule diff should always get stored, can you provide some error logs on fail cases?
It's hard to pinpoint, I have a repository with several submodules, some changes are found, others not. I've added untracked files, but sometimes it simply doesn't pick them. Running git diff --submodule=diff (which is the same command in the GitDetector class) always return the correct results
Before manually modifying the diff:
If you have local commits (i.e. un-pushed) this might fail the diff apply, in that case you can set the following in your clearml.conf
I have both:store_uncommitted_code_diff: true
andstore_code_diff_from_remote: true
set