clearml==0.17.4

Great, and as you pointed changes are detected.

If I make some changes in the original file it gets tracked (picture below)... but what if I use a completely new and git-untracked py script?

Basically it will capture the output of "git diff" so if you have a new file but it is Not added, then no changes will appear.
That said, this is usually the exception, most files that are not added to the git should not be logged, hence the logic.
Anyhow the idea is that the Agent will be able to replicate your entire code base, without you having to worry about constantly committing your code (not that it is bad, but we all tend to forget)