Inspired by Insiders – How Windows Insiders helped track down a tough Windows Search problem

Greetings from Adam Wilson, Lilian Wang  and the rest of the Windows Search team.

Anyone hanging around a Windows forum for too long eventually sees the same thing. Something is going wrong on a Windows PC and the accepted recommendation to fix it is by disabling the Windows Search service, also called the indexer or search indexer.

For us on the Windows Search team, the real questions that matter are: Are people really following these instructions to turn off the Windows Search service? And if they do, what problem are they trying to solve?

There have always been theories here in Redmond. Up until recently, though — there was no way to test these theories. Reading through the forums hasn’t provide enough information about what is going on. Calling or surveying Windows users hasn’t worked either because such a small percentage of people turn off Window Search that it didn’t give a large enough sample.

Still, the problem remained: Despite significant changes across several Windows 10 releases, there were enough people turning it off that we feared we might be missing a major underlying issue. But how to track it down?

Windows Insiders to the rescue

To get to the heart of the matter, we turned to Windows Insiders. During a three-month campaign, if someone disabled Windows Search, a ‘toast’ would pop up on the screen asking why.

We were hoping for maybe 100 reports when the campaign went out, which would be enough to start drawing some conclusions. However, once again, Windows Insiders surpassed our wildest imagination. Over the 3 months we gathered almost 700 different pieces feedback from Insiders all over the world.

And it was awesome data. Lilian, our data expert, and I sat in for hours together reading over every single piece of feedback that came back and making sure it was correctly sorted. Pouring over feedback from a dozen different languages and users with all sorts of different machines we put together the following chart:

Reason for Disabling Windows Search Instances
1. Excessive disk usage 165
2. Proactively because of general past perf issues 142
3. It doesn’t provide enough value 100
4. Done without user action or intent 90
5. To protect SSD from wear 71
6. Excessive CPU usage 69
7. All Other Reasons 55
 Total 692

Making Things Better

First, I’d to thank all you Windows Insiders for your feedback during our campaign. I can’t stress enough how much it has helped us. The Windows Search team is committed to addressing each issue you’ve raised. And as a first step in keeping that commitment, I want to talk about some of the things that we’ve done already to start resolving issues — and provide some quick fixes you can try right away.

Issue 1: Excessive Disk Usage

We’d seen this issue in previous surveys but never ranked as high. Consequently, it’s the issue we are going to work the hardest to improve. While disk usage is never going to go to zero, there are a few features that we’ve added to Windows 10 that will make things better:

First, when the system detects high disk usage, Windows Search will slow down any background activity. The new algorithm is not perfect, but it is a bit better. Second, to improve the throttling logic long term Insider machines running Insider Preview builds will log the number of times a day the Search service causes high disk usage and for how long. This extra data will allow us to iterate on the throttling algorithm and see how changes are performing on your machines.

Quick Fix: If the search service is causing you issues with disk usage, you can pause it for 15 minutes at a time. Just go to Control Panel > Indexing Options > Pause. This will pause the indexer for you and send a signal back to us that something is going seriously wrong. [1]

Issue 2: Disabled Proactively because of past performance issues

You folks are the hardest to please — but that doesn’t mean we are going to forget about you. Most of the Insiders falling under this bucket have had previously bad experiences with Windows Search. Something hadn’t worked, or Search had gotten in the way of an experience you value more.

Quick Fix: As I mentioned earlier, we are continually making improvements to Windows Search. So, let’s make a deal: Give Windows Search a try in the new release — for just one week. If you don’t like it or something bothers you — log feedback in the Feedback Hub (or upvote an existing issue) and disable it until the next release. Note: The first day after re-enabling search will be rough as it builds the database, but I believe that after a week it will win you over.

Issue 3: Disabled because it doesn’t add enough value

This group of Windows Insiders was one of my favorites — they did not mince their words — just straight up told us that Windows Search didn’t do what they wanted it to. And there was not a single comment that I disagreed with, they were all well-reasoned arguments about why Windows Search might not be for you. That’s said, it’s worth providing a quick summary of the benefits of Windows Search and why you should think twice before turning it off:

  • It powers the file search and enumeration behind Cortana, File Explorer, and many UWPs.
  • It powers email search for millions of Outlook users around the world
  • It provides suggestions from your personal web history when you are typing in the Edge address bar
  • It is used for the sync and change tracking logic in files on-demand
  • It gives you full control of where all this data is stored. In a privacy conscious era, having a search engine that stores its database on your computer is a wonderful thing to have. The search index for Windows Search is stored on your machine and no data about your files will ever leave your machine. [2]
  • It powers the new Timeline view. [3]

For more on Windows Search, check out our New FAQ article to learn more about search and indexing.

Issue 4: Disabled Without User Action

This was the biggest surprise to me — there are programs that will disable the indexer without a direct user action. Some of them are anti-virus while others are privacy protection software that claim to reduce the data that is being sent to Microsoft.

Quick Fix: As I mentioned earlier, your personal files or data will not be sent from Windows Search Service to Microsoft. The database is going to be stored locally on your machine and can be deleted at any time. However, if you are already running one of these programs and suspect Windows Search might be disabled, the steps to re-enable it are:

  1. Windows Key + R > services.msc
  2. Scroll down and double click Windows Search
  3. Confirm the following values are correct
    1. Service Status should be: Running
    2. Startup Type: Automatic (Delayed Start)

Check back in a couple days after enabling search to make sure that it is still working. Some programs will undo this on a schedule. If you see this happening check your anti-virus or other programs that are running in the background as they could be undoing your efforts.

Issue 5: Protect SSD from wear

This is the category of people that I completely identify with — my laptop and desktop both have an SSD and if there is anything possible to make it last longer then I’m going to do it.

Quick Fix: One option — which I do on my desktop — is to move the search index from your primary SSD to a secondary HDD. This means that I get all the benefits of fast search, timeline, and Cortana from Windows Search without risking my primary SSD.  To set up your machine like this, go to Control Panel > Indexing Options > Advanced > Index location. From here you can move the index to another drive.

It is a longer-term issue however that we are continuing to look at. Wear leveling on SSDs is getting better which will help, the indexer improving its compression logic reduces the writes and therefore the wear, but there is still work to do.

Issue 6: Excessive CPU Usage

This is a problem that we’ve been working on almost non-stop for 4 years and there is a still a lot of work that we can do. As a part of the work to log disk violations, we are also going to log cases where the indexer is using too much CPU.

We’ve also made a few tweaks to the CPU throttling logic. Look for other improvements as we head closer to the ship dates for the next version of Windows.

Issue 7: Other Reasons

Insiders have an uncanny ability to weed out bugs in Insider Preview builds — and Windows Search has been no exception. In fact, there is already a bug fix for an issue that an Insider mentioned during our survey:

  • Bug 15917485: Indexer can’t run because of incorrect permissions on the directory —running troubleshooter fails

As well there are a few more open bugs on our plate that we are going to investigate.

Quick Fix: If you are having issues with search not finding your files, not finding your drive, or finding a drive you don’t want it to search — look at Control Panel à Indexing Options. It will show which drives and folders Windows Search is going to look at. You can configure all the different folders that it will use.

Next Steps

We’re not at the happily ever after part. Heck — we are just out of the starting blocks here. There is still a ton of work to do — and we’re delighted to have Windows Insiders along for the ride. Along with ongoing experiments and surveys, we invite you to log feedback in the Feedback Hub when you can. We may not be able to reply to every Feedback item, but we’ll certainly read them. It’s all part of our never-ending quest to make Windows Search even better.

Until next time — keep searching.

Notes

  1. The button press will be logged in telemetry for Windows Insiders only
  2. And if you want to be super sure of this, the zero-exhaust mode of Windows — designed to not exchange any packets outside of web browsing — still includes Window Search. Highly recommended to not enable this on your own machine. It disables such helpful things as incoming VoIP calls, security updates, settings sync, email syncing, and access to Insider builds.
  3. So why do we say things like “Don’t turn off Windows Search if you want to use Timeline”? It seems like searching for your file and seeing the last 10 activities on your device shouldn’t be the same thing. The reason that they are linked can be explained by the simple principle: Listing all the items in a group is a search for everything in that group.

    For example, when you are viewing the most recent activities in your Timeline you are seeing the output of a search for all the Timeline items, sorted by date. We do this because it makes it very easy to filter and shape what you are seeing on the screen as you select various options. If you click to see only a single day of Timeline items, the search just changes to filter out the results that don’t match the target date.Looking at the Timeline UI, here are three different views on the timeline information that take only slightly different queries against the indexer.

    Looking at the Timeline UI, here are three different views on the timeline information that take only slightly different queries against the indexer.

    First, we have all the activities grouped by day, sorted by Top (a special property we calculate using magic and NinjaCats – or time spent on the activity, but basically the same thing):Activities grouped by day

    Next, we have the same data, now filtered by Top, grouped by hour and sorted by time:
    Activities filtered by Top, grouped by hour and sorted by time

    And finally, we have a more traditional search. The activities are filtered to ones matching the word cat, grouped by day, and ordered by date:
    Activities filtered to ones matching the word cat, grouped by day, and ordered by date

    This is hugely powerful for our developers and designers on Windows. They can change what information is being shown in various views quickly by just altering a query instead of having to rewrite their entire data store.

    The downside is that, for Windows Search, there can be cases where it is doing work (such as adding new Timeline items to the database) that don’t make sense. Why would searchindexer.exe be running when I’m just reading blogs in Edge? Well if you attached a debugger it would be because Edge is adding new entries to Timeline, which in turn causes the search service to start, but that is hard to tell without a lot of debugging.

What are your thoughts? Would you be interested in seeing more about what the search service is doing while it is running? Let us know in your feedback and comments in the Feedback Hub.

SHAPE THE FUTURE OF WINDOWS — BECOME AN INSIDER

PREVIEW THE NEXT EVOLUTION OF WINDOWS