|
Obviously run by Professor Frink[^]! 
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Yea, I know this isn't the place to ask technical questions about this annoying 'feature' but can anyone recommend a good place to ask and actually get a meaningful answer from someone who knows? It seems to be a bug or a 'misfeature'.
For those who are interested, here is the issue:-
This is Windows 10 2004 and I added a path to the Windows Search search scope. The path contains archived dev projects. Let's say it's "D:\Data\Devstuff\Archive\". It contains subfolders (40-50 folders, one subfolder per project). Mostly C# but Python, TypeScript and other stuff. Most of the folders do not have ".git" subfolders but a few do.
I recently noticed that all the folders that have a ".git" subfolder (e.g. like this: "D:\Data\Devstuff\Archive\PythonHelloWorld\.git\") have been excluded from the Windows search search scope. So "D:\Data\Devstuff\Archive\" is included but "D:\Data\Devstuff\Archive\PythonHelloWorld\" (and any other subfolder of "D:\Data\Devstuff\Archive\" that has a ".git" subfolder) is listed in the excluded section. It wasn't me who did this; Windows did it all by itself at some point fairly recently! Not sure when, though.
After some experimentation:
(1) It's definitely the existence of a .git subfolder that causes its immediate parent folder to be excluded from the search/index scope.
(2) The ".vs" subfolder has no such effect.
(3) It's not a path length issue. The same thing happens no matter how deep the offending folder is.
(4) Manually removing the exclusions simply results in them being added back in by Windows.
(5) Rebuilding Windows Search (forcing a complete rebuild of its database and settings) from scratch doesn't cure the problem.
(6) I tried adding an exclusion rule via the registry to exclude "file:///*\.git\" (i.e. all ".git" folders) in the hope that this would allow the parent folder to be indexed but it had no effect. The parent folder was still removed from the search scope.
(7) By fiddling around with the registry to manually force folders with ".git" subfolders to be included, it is possible to hang the Windows Search service. The only cure is to allow Windows Search to remove the folders containing ".git" subfolders (usually be letting it rebuild from scratch).
On the face of it this seems to be a strangely hardcoded attempt to prevent ".git" folders being needlessly indexed but it seems to have gone awry in that it excludes the parent folders that simply contain the ".git" subfolders (and of course it then excludes any other subfolders, including the ones containing code that I want to be indexed).
Any ideas?
modified 15-Oct-20 9:41am.
|
|
|
|
|
Quote: .gitignore file anywhere? not sure if you mentioned that in your post. I know you can ignore files and folders using a .gitignore file in a visual studio solution, etc.
All the folders that contain a ".git" subfolder have a .gitignore file but such files have no meaning to Windows Search.
Just to confirm, I deleted the .gitignore file from "D:\Data\Devstuff\Archive\PythonHelloWorld\" and when I added "D:\Data\Devstuff\Archive\PythonHelloWorld\" to the Windows Search search scope via the UI, it just disappeared. I looked in the registry and the folder is now listed as a non-included folder. In other words, Windows Search saw me add it and then just disappeared it, and will not now index or search it.
|
|
|
|
|
markrlondon wrote: it excludes the parent folders that simply contain the ".git" subfolders (and of course it then excludes any other subfolders, including the ones containing code that I want to be indexed).
In a folder such as "D:\Data\Devstuff\Archive\PythonHelloWorld\" (which has a ".git" subfolder), if I add the "D:\Data\Devstuff\Archive\PythonHelloWorld\PythonHelloWorld\" subfolder (where all the code is located) manually to the Windows Search search scope then it is indexed and searched correctly. This is because the ".git" folder is at "D:\Data\Devstuff\Archive\PythonHelloWorld\.git\" and so Windows Search does not perceive it to be a problem for "D:\Data\Devstuff\Archive\PythonHelloWorld\PythonHelloWorld\".
|
|
|
|
|
If these are just archives, why not rename the .git things to .gat or something. Then, if you ever need to unarchive for use, restore the correct name.
Or maybe Microsoft will fix the problem. Someday. At a later date. TBD 
|
|
|
|
|
GenJerDan wrote: If these are just archives, why not rename the .git things to .gat or something. Then, if you ever need to unarchive for use, restore the correct name.
A good idea. That does seem to work.
In my earlier experimentation I tried temporarily removing the ".git" folder but I didn't think to rename it. Doh. And, indeed, renaming it does allow the folder in which it lives to be indexed and searched. So it seems from this that it is a hardcoded issue to do with ".git" folders.
GenJerDan wrote: Or maybe Microsoft will fix the problem. Someday. At a later date. TBD
To be fair, I expect they will fix it, if only because it will annoy their own in-house users of Git! 
However, it would help to be able to bring it to their attention. I just wish I knew where to do this in a manner that will allow it to be understood and treated seriously.
|
|
|
|
|
markrlondon wrote: In my earlier experimentation I tried temporarily removing the ".git" folder but I didn't think to rename it. Doh. And, indeed, renaming it does allow the folder in which it lives to be indexed and searched. So it seems from this that it is a hardcoded issue to do with ".git" folders.
Ah, after some further experimentation it's not just any ".git" folder: It would seem that the ".git" folder must have some real Git files in it, although I've not gone to the effort of figuring out which ones.
So it looks like Windows Search is trying (and failing) to do hardcoded clever things with the contents of ".git" folders. Which, I know from experience, is entirely unnecessary since no knowledge of Git is needed to correctly index the current status of a project managed by Git.
|
|
|
|
|
One lovable thing about git is that you never know which files will be in your directories. That in none of your business, sort of 
For tl;dr people: Indexing makes very little sense in a directory tree where git pulls away the rug under your feet, maybe a dozen of times (or more) every working day.
A little more detail:
Open a command window in your git project directory. Checkout branch xxx, list the files. Checkout branch yyy, list the files. The two sets of files may be completely disjunct, non-overlapping (or only partially overlapping). You haven't done any cd at all, not deleted or created any file. Yet the dir command gives two very different results.
Open another command window, and cd to the same project directory. Which set of files will you see there, if you do a dir command?
Go back to the first command window, and checkout branch xxx. Return to the second command window and repeat the dir command. Some, maybe all the files you just saw a few seconds ago, have disappeared, and another set has appeared. In that window, there is no indication of any command causing the change.
It would look just the same if you - rather than checking out another git branch - in the first window had actually deleted some files and created some new ones. Usually you would consider that as really deleting info, and building new info, a semantically significant thing. But in git, when you check out another branch, the info is not deleted, it is just hidden. No new info is created, it is just pulled out from hiding.
How should the indexing handle this? Indexing is performed - on which files? Those that git has hidden? After indexing is complete, you check out another branch, so some of those files that were visible are now hiddden. How would you handle a search that gives a hit in now hidden files? As if they were deleted? Or as if they are visible? How should this hit be reported to the user?
Consider git project directories to be managed by git, neither by yourself nor Windows. (It took me quite some frustration before I learned to never have two command windows open in the same git project directory, in particular if you work on 2+ branches!)
If the git delevelopers want to provide an indexing and searching mechanisms tailored to the .git direcory and file strucutres, so it can report a hit e.g. indicating which branches the file are found in (whether checked out or not): Fine. But don't expect that every OS shall interpret .git structures (nor any other application's privately defined structures).
|
|
|
|
|
I'm far from being a Git expert so thanks for your rather enlightening comment.
However, even though Git does its own thing, one thing that can be guaranteed at any one time is that the contents of the subfolder that Git is managing will always correctly contain whatever is the current state of the current branch of the project. It has to do this in order for the IDE or editor to be able to find things where it expects them, and this applies to indexing too.
So Windows Search definitely can correctly index and search folders managed by Git with no difficulty or special uncertainty. It automatically re-indexes if Git (or the IDE or editor) changes the current state of the project. It really does just work without the index/search tool knowing anything whatsoever about Git.
From the perspective of Windows Search, Git is no different to an editor or IDE: It's just a thing that changes files, and WS responds to that by dynamically re-indexing them.
To respond to your specific question:
trønderen wrote: How should the indexing handle this? Indexing is performed - on which files? Those that git has hidden? After indexing is complete, you check out another branch, so some of those files that were visible are now hiddden.
There is no difficulty at all. The indexer just indexes whatever is the current state of the files. When files change they are dynamically re-indexed. Honestly, it's an absolute non-problem in practice and, prior to Microsoft apparently mucking around with it, it worked very well in my experience and provided the answers that I expected.
trønderen wrote: How would you handle a search that gives a hit in now hidden files? As if they were deleted? Or as if they are visible? How should this hit be reported to the user?
The same way as any other search when a file has either not yet been indexed or the file has changed but the index has not yet been updated (which is in fact extremely rare once indexing is initially completed).
modified 15-Oct-20 11:24am.
|
|
|
|
|
(unrelated rant - nothing to do with exclusions that someone might've explicitly coded in for search results)
I've been logging into my various Win10 VMs to get them to install Tuesday's updates. To quickly get there, I hit the Start menu and type in 'update'. On some machines, I immediately get the shortcut (which is exactly how things should be, if we're expected to search for everything rather than navigating through a complex menu hierarchy). On other systems, it thinks and thinks and thinks about it, and the results list stays blank. At other times--those same systems--it might come back with the correct results.
This sh*t is broken. There's no other way to put it. And honestly this is a rare case where I, as a developer, don't care about the reasons behind that inconsistent behavior. Just fix it, MS.
|
|
|
|
|
It's one of the reasons I have used Agent Ransack for the past 12+ years.
Not being able to find certain file types with Windows search is nothing new.
I know you are not asking for a recommendation, but if you use Windows search as your main method to search for files and file contents and your work policy allows it - use Agent Ransack, I don't think you will regret moving from Windows search to Agent Ransack.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
|
|
|
|
|
I've frequently resorted to using VS's Search in Files to find stuff I know I've just edited, but Windows Search can't find.
It's never been intended to be particularly fast...but it works.
|
|
|
|
|
GuyThiebaut wrote: Not being able to find certain file types with Windows search is nothing new.
This is a common statement but I'm very familiar with Windows Search and it's not normally a problem. I've previously never failed to find something that I knew should be in the index. The problem with .git subfolders that I've now found seems to be entirely new.
I know that Microsoft have been working on Windows Search in recent months and it looks to me that they're trying to add some sort of hardcoded intelligence to Windows Search where it finds a Git-managed folder but it's not yet working correctly.
This is why I'd like to speak to someone at Microsoft who can understand what is going on, and I'd be willing to beta test the changes.
GuyThiebaut wrote: I know you are not asking for a recommendation, but if you use Windows search as your main method to search for files and file contents and your work policy allows it - use Agent Ransack, I don't think you will regret moving from Windows search to Agent Ransack.
I've tested a great many search tools and Windows Search is actually the only thing that means I'm still running Windows and not Linux! Windows Search's UI integration is exceptionally good, as is its extensibility. There's nothing wrong with Agent Ransack but overall (current issues notwithstanding) it does not suit my requirements as well as Windows search does.
|
|
|
|
|
I use 'Everything'. voidtools.com.
|
|
|
|
|
markrlondon wrote: Any ideas?
Windows search does not index cloud files. This may be causing your issue.
The best part of living in the Clouds is that you never know when those files are stored locally or 4000 miles away. A good implementation will be completely transparent to the end user.
ProjFS[^]
VFS for Git: Git at Enterprise Scale[^]
Virtual File System for Git[^]
Best Wishes,
-Tom_and_Frank or maybe just Tom, or Frank
|
|
|
|
|
Tom_and_Frank wrote: Windows search does not index cloud files. This may be causing your issue.
The best part of living in the Clouds is that you never know when those files are stored locally or 4000 miles away. A good implementation will be completely transparent to the end user.
No, this is nothing whatsoever to do with cloud storage. The files are stored locally.
** edit **
And the files are in NTFS, not Microsoft's new Git FS.
modified 17-Oct-20 7:19am.
|
|
|
|
|
I believe Windows search now skips over files marked IO_REPARSE_TAG_PROJFS or IO_REPARSE_TAG_FILE_PLACEHOLDER or various other reparse tags[^].
You would probably argue with the guy who researched and invented the technology.
|
|
|
|
|
This isn't the issue here.
To the best of my knowledge, Windows Search has never properly indexed the targets of symlinks or other reparse points. Back in Windows Desktop Search days it was possible to force WDS to do an initial crawl on the targets of symlinks (by adding the symlink to the index scope) but it could not then dynamically update the index when the target files changed since the file system watcher API could not detect changes.
Nowadays with WS in Windows 10, symlinks and other reparse points do not even appear in the UI to include paths in the index scope so that this source of confusion is eliminated.
Note also that the Git FS is not in use here, nor are cloud placeholders; it's all normal NTFS local files.
modified 17-Oct-20 7:32am.
|
|
|
|
|
Looks like it was a recent change: Supercharging Windows Search[^]
Quote: For our developers, we also made a change where the indexer no longer covers popular source repositories, like Git. This was partly because of the sheer size of these repositories, and also because the tools developers use to interact with their repos usually have their own search tools. We also worked with our Visual Studio partners to exclude their project folders, which resulted in a quick 30% improvement in disk usage, for an even better developer experience.
TTFN - Kent
|
|
|
|
|
Kent Sharkey wrote: For our developers, we also made a change where the indexer no longer covers popular source repositories, like Git. This was partly because of the sheer size of these repositories, and also because the tools developers use to interact with their repos usually have their own search tools. We also worked with our Visual Studio partners to exclude their project folders, which resulted in a quick 30% improvement in disk usage, for an even better developer experience.
Thanks for finding this.
What the flying frag were they thinking!  The tool I use to search my archived projects is Windows Search! The files are in my file system so as to be indexed! There's no option to disable this -- it's all hardcoded.
They seem to me to have lost touch with reality.
To say that I am disappointed by this bug (and that's what it is as far as I can see) is an understatement.
Trying to appease those who complain that their Windows Search is too slow by hobbling its capabilities seems... bizarre. Indeed, in my experience, Windows Search's ability to adjust CPU and disk usage so as to avoid impacting the users is fantastic.
Users can trivially exclude folders from Windows Search if they don't need them to be indexed. No one has ever been forced to index a massive Git project. This means that there is no good reason whatsoever to hardcode Windows Search to force it to exclude indexing of folders that have been explicitly *included* by the user (as is the case for me) just because they happen to have a .git subfolder. It's braindead behaviour.
modified 16-Oct-20 14:17pm.
|
|
|
|
|
... and some are still wondering why I have put it on my to-do list to change my entire home computer system to move away from the Windows ecosystem, even if it means that I have to adapt a different desktop that has a substantial number of its own bugs. But I have made the decision that I will do it, slowly, to move away from exactly this type of interference with my ownership of my own PC.
|
|
|
|
|
Martijn Smitshoek wrote: . and some are still wondering why I have put it on my to-do list to change my entire home computer system to move away from the Windows ecosystem, even if it means that I have to adapt a different desktop that has a substantial number of its own bugs. But I have made the decision that I will do it, slowly, to move away from exactly this type of interference with my ownership of my own PC.
Somewhat ironically, it is only Windows Search that has kept me on Windows. The alternatives on Linux are all very poorly integrated into the UI and have never offered the feature set that I'd need to use them seriously.
As a user of Windows Search for decades, I've never understand why people don't like it. Excluding this new Git issue (where it seems they have intentionally hobbled it!), I've always thought that Windows Search was one of the best, fastest, smoothly running, fully integrated, search tools. However, I have noticed that Microsoft have been further dumbing down the UI in File Manager in recent Windows versions. This has negatively impacted its usability.
But I still can't move to Linux-based OSs since they're UI integration is so poor and also, in some cases, lacking in capabilities.
modified 16-Oct-20 13:38pm.
|
|
|
|
|
LOL, who is using Windows Search in 2020 ? (Which can be extended to "who uses windows explorer in 2020 ?")
|
|
|
|
|
Giving alternatives for people who read it could be a good idea. 
At best posted in: Free Tools Discussion Boards[^] where they are easy to be found by all users of the site
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
I am. It is an exceptionally good integrated search tool (which I accept is an ironic thing to say when it now seems that Microsoft have intentionally hobbled it!).
I have never found anything that works as well or as quickly as Windows Search together with its fantastic UI integration and that includes Agent Ransack, Google Desktop Search (when it existed), Copernic, and others on both Windows and Linux. Recoll on Linux is good but lacks UI integration. The other desktop search tools on Linux (e.g. Tracker and Baloo) suffer from extraordinarily poor UI integration and also, in Baloo's case, lack many of the capabilities of Windows Search on Windows.
modified 16-Oct-20 13:45pm.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
