Typing lags when there is a gigantic index to search through

[dansgithubuser]

I often open my root directory in atom to get easy access to all the files I don't open often enough to open in their own project. This makes for a large index (300K files), and slow search times. For a full search, this is unavoidable, expected, and fine.

What's unnacceptable is that the slowdown is on the UI thread. I currently do not use fuzzy finder because of this.

If you're thinking of writing this off as innappropriate use, then fuzzy finder still needs to tell users not to attempt this and give them the UI thread back.

The obvious solution is to take the heavy operations off the UI thread, but there might be more to milk here. Just by going through fuzzy finder's issues, it seems there's a lot of clever heuristics that have been evolved over time. Maybe it's possible (natural?) to use these such that the most likely thing the user is looking for shows up quickly? Then we actually save the user a lot of time (on average), even when the entire operation would take a long time.

I originally approached this through #156 but chatting with @lee-dohm, he suggested I create a new issue that concentrates on the problem, not a possible solution.

[dansgithubuser]

I wanted to attach a profile per lee's request, but when I save the file, an empty file comes out. I do not want to get off topic. I'll attach a screenshot. This is me typing asdf with slightly more than 50 ms between each letter.

[driskell]

I'm experiencing this issue with a large repository I work with. It used to respond extremely quickly. I'm not sure exactly when it started but it now lags for well over a second between characters. Even using up/down key press to scroll through causes well over a second lag between movements (even though mouse wheel scrolling is instant.) It seems also that the project is reindexed every time I open the panel, even though nothing has changed. The label shows for a few seconds. Even waiting for the indexing to finish and then typing do I get the exact same problem of seconds of delays between characters.

Profile snapshot:

I am using the latest Atom-Beta though so maybe it's a known problem?

[lee-dohm]

@driskell Is the repository you're working with public so that we can use it to reproduce the problem?

[driskell]

Unfortunately, it's private. I just inspected and the item list is approximately 89000 items.

I did locate the problem causing the long timeout at the left and right side of the profile above. This is when I was using just arrow keys and it was lagging just with that. It appears this is due to file-icons package refresh being called within 20ms of every key press, and it runs a normalisePath and generates icons for all 89000 items instead of just those visible in the list... Disabling for now fixes the lag on arrow keys, but still it is very slow to filter the list.

However, maybe with having 89000 items it is just to live with. With file-icons disabled though it opens much quicker (before it would hang immediately on open in that timeout segment left of profile.)

[driskell]

With file-icons disabled, if the scoring could be done on a background thread with a maximum of say, 5000 items or something per tick, this would leave a responsive UI and I can just wait and wouldn't mind. I think main issue at the moment is I start typing "framework" and I will only see "f", wait a few seconds, and then I'd see entries for "f", then wait a few seconds, and see entries for "framework". If it would wait until I'd typed it would be much more responsive.

Looking at profile it seems most time is spent in computeScore and pathScorer, and a tiny bit in sort. I did shave maybe a tiny amount off the processing time (didn't have time to measure) though by replacing the fuzzaldrin scoredCandidates sort, pluck and slice with a quickselect implementation to pull the top maxResults entries to start of scoredCandidates, then slice, pluck and sort. Of course, I then had to set maxResults to 10 in fuzzy-finder as it looks like it doesn't set a max results.

Wasn't sure on how to improve anything else, though I did wonder if there was a way to early skip entries when computing score if it was obvious early on it wouldn't meet a minimum score, then once it had maxResults items of score 9 or higher it could fast skip some obvious failures. But this is just speculation and probably not possible.

Background searching seems the way to go I think.

[fserb]

@lee-dohm I work on Chrome (so it's a public repo, if you feel like fetching it: https://www.chromium.org/developers/how-tos/get-the-code), and I get the exact problem as described by @driskell.

I'd be happy if fuzzy-finder at least allowed me to say "don't search for less than 3 characters" or something. Although I'm not sure this would help much (assuming that it only searches on the subset it already encountered).

Anyways, making the UI cancel the current search sounds a good idea in general, given that will make the interface more responsive.

[danielduan]

I recently noticed that the fuzzy search is getting slower and slower for me as well. The project I work on is massive as well but somehow sublime manages just fine.

It seems to always reindex every time the search is brought up and lag on every keystroke. I'd second the effort to cancel the current search or have a delay timer to trigger the search after typing has stopped.

There's a portion of my project that is currently symlinked and that seemed to slow down the fuzzy search quite a bit vs. if I remove that directory.

Would really appreciate any work being done on this. Thanks!

[umpirsky]

Damn file-icons, thanks, it works blazing fast after disabling it!

[Alhadis]

@umpirsky JSYK, I'm painfully aware of how file-icons is affecting scalability. That's one of the reasons #281 needs to be merged, because the performance overhead is the direct result of committing heinous crimes against decent programming principles.

In other words, it has to monkey-patch a crapload of package code and monitor the state of each opened list to dynamically reconstruct stuff. The hack got even worse after jQuery was ditched, and it wasn't as easy to find a foothole.

Soo... yeah, this isn't paradise for me either. :p

[Alhadis]

@driskell Could you do me a favour?

1. Re-enable/reinstall file-icons
2. Clone and dev-link the fork that hosts my topic branch for Consume File-Icons' element-icon service #281
3. Restart Atom, and open the massive project folder you mentioned earlier

Does file-icons still impose that much overhead on usability...?

[jeancroy]

@lee-dohm

I guess from context that this issue is on development deck or not far.
I think the filters we see on the bottom of the flame graph are mine (with alternate scoring)
I would have a request for the dev team, to discuss about the possibility of an async version of fuzzaldrin.filter

There are a lot of things that are computed to immediately be discarded and that bother me a bit.
I was thinking of doing a chunked / cancelable async api but there's work to be done to consume such an api on atom side.

I've also seen in another issue the dev team consider to move some fuzzy filtering to c++ side (at least regarding large file & autocomplete)

[lee-dohm]

@jeancroy Thanks for the update 👍 I'll bring it up to the team.

[davidcelis]

I'm still experiencing a lot of slowness with the fuzzy-finder, even in moderately sized projects. This happens whether or not I've installed file-icons. Just to be safe, here are two CPU profiles without file-icons installed:

CPU-Profiles.zip

[Alhadis]

I opened one of @davidcelis's .cpuprofile profiles, but I can't say the plain-text format is terribly insightful: 😢

{"nodes":[{"id":1,"callFrame":{"functionName":"(root)","scriptId":"0","url":"","lineNumber":-1,"columnNumber":-1},"hitCount":0,"children":[2,3,5,6,11,15,16,39,47,49,66,74,75,94,102,236,284,379,382,423,432,642,643,777,2291,2310,2316,2319,2366,2372,2634,2653,2681,2732,2737,2819,2823,2909,2940]},{"id":2,"callFrame":{"functionName":"(program)","scriptId":"0","url":"","lineNumber":-1,"columnNumber":-1},"hitCount":1971},{"id":3,"callFrame":{"functionName":"module.exports.WindowEventHandler.handleWindowFocus","scriptId":"24","url":"<embedded>","lineNumber":18558,"columnNumber":67},"hitCount":1,"children":[4],"positionTicks":[{"line":18561,"ticks":1}]},{"id":4,"callFrame":{"functionName":"remove","scriptId":"0","url":"","lineNumber":-1,"columnNumber":-1},"hitCount":2,"positionTicks":[{"line":18561,"ticks":2}]},{"id":5,"callFrame":{"functionName":"(idle)","scriptId":"0","url":"","lineNumber":-1,"columnNumber":-1},"hitCount":22872},{"id":6,"callFrame":{"functionName":"","scriptId":"24","url":"<embedded>","lineNumber":63360,"columnNumber":25},"hitCount":0,"children":[7],"deoptReason":"Bad value context for arguments

What's the correct (or best) way of evaluating this sort of data? I always get audits and profiling and memory heap records and whatnot all confused. :(

But I need to know just how much of an impact file-icons is still having on activation time, even with full integration with core-services. I have an inkling that, once atom/find-and-replace#974 is merged and shipped in the next stable release, I can begin ripping massive chunks of redundant and bloated code, and focus on anything that's slowing us down.,

[winstliu]

Load it in the Developer Tools.

[p2kmgcl]

Here you have a public example to reproduce: https://github.com/liferay/liferay-portal
I had to stop working with Atom because of this, the editor becomes laggy :c

[tlorens]

Working on a Symfony 4 project. I ignored ./app, ./node_modules, and ./vendor So only my project files are searched. HUGE performance improvement. I don't care to index 3rd party libraries and projects-- Shouldn't be editing those anyhow.

[yingqian18]

I had the same issue when using atom 1.25.1 and macOS Sierra 10.12.6. It's painful because the UI is not responsive at all. I saw there are tons of people complaining about this problem, while this issue remains unresolved for over two year maybe even longer. However, Sublime don't have this problem at all.

[bartek-szymanski]

For me fuzzy finder got faster after adding "node_modules" to the "Ignored Names" list in:

Preferences -> Core -> Ignored Names

You can also switch on "Hide Ignored Names" in:
Preferences -> Packages -> Tree View

https://discuss.atom.io/t/hide-specific-folders-from-tree-view-fuzzy-search/1190/2

[johnhabibi]

@EpsilonEridani, that does the trick for me as well!

[slicksammy]

It was probably node_modules for me.

Preferences -> Core -> Exclude VCS Ignored Paths (enabled)