1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-23 05:50:55 +01:00
phorge-phorge/resources
epriestley 1de130c9f5 Allow the Ferret engine to remove "common" ngrams from the index
Summary:
Ref T13000. This adds support for tracking "common" ngrams, which occur in too many documents to be useful as part of the ngram index.

If an ngram is listed in the "common" table, it won't be written when indexing documents, or queried for when searching for them.

In this change, nothing actually writes to the "common" table. I'll start writing to the table in a followup change.

Specifically, I plan to do this:

  - A new GC process updates the "common" table periodically, by writing ngrams which appear in more than X% of documents to it, for some value of X, if there are at least a minimum number of documents (maybe like 4,000).
  - A new GC process deletes ngrams that have been added to the common table from the existing indexes.

Hopefully, this will pare down the ngrams index to something reasonable over time without requiring any manual tuning.

Test Plan:
  - Ran some queries and indexes.
  - Manually inserted ngrams `xxx` and `yyy` into the ngrams table, searched and indexed, saw them ignored as viable ngrams for search/index.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13000

Differential Revision: https://secure.phabricator.com/D18672
2017-10-03 13:27:42 -07:00
..
builtin Add more repo images 2017-08-08 17:51:15 -07:00
celerity Make PHUITwoColumnView a little more printable 2017-09-25 19:56:22 +00:00
cows Use PHP implementation of Cowsay for cowsay rule 2015-09-13 12:27:30 -07:00
emoji Add some sort of sort to Emoji Autocomplete 2017-01-24 20:21:06 -08:00
figlet/custom Include "Figlet" and PEAR "Text_Figlet" in externals 2015-09-13 12:30:48 -07:00
font Made Meme Generator 2013-01-19 18:43:43 -08:00
sprite Uh, update Phabricator login image 2017-08-11 13:37:26 -07:00
sql Allow the Ferret engine to remove "common" ngrams from the index 2017-10-03 13:27:42 -07:00
sshd Drop interactive login from sshd example 2017-06-27 12:51:46 -07:00