Manticore Search 7.0.0

Version 7.0.0

Released: Jan 30th 2025

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

📢📢📢Announcement blog post📢📢📢

Major changes

• Issue #1497 Added new Fuzzy Search and Autocomplete functionality for easier searching.
• Issue #1500 Integration with Kafka.
• Issue #1928 Introduced secondary indexes for JSON.
• Issue #2361 Updates and searches during updates are no longer blocked by chunk merging.
• Issue #2787 Automatic disk chunk flush for RT table to improve performance; now, we automatically flush a RAM chunk to a disk chunk, preventing performance issues caused by the lack of optimizations in RAM chunks, which could sometimes lead to instability depending on chunk size.
• Issue #2811 Scroll option for easier pagination.
• Issue #931 Integration with Jieba for better Chinese tokenization.

Minor changes

• ⚠️ BREAKING Issue #1111 Fixed support for global_idf in RT tables. Requires table recreation.
• ⚠️ BREAKING Issue #2103 Removed Thai characters from internal cjk charset. Update your charset definitions accordingly: if you have cjk,non_cjk and Thai characters are important for you, change it to cjk,thai,non_cjk, or cont,non_cjk, where cont is the new designation for all continuous-script languages (i.e., cjk + thai). Modify existing tables using ALTER TABLE.
• ⚠️ BREAKING Issue #2458 Incorporated zlib support in Windows packages.
• ⚠️ BREAKING Issue #2468 CALL SUGGEST / QSUGGEST now compatible with distributed tables. This increases the master/agent protocol version. If you are running Manticore Search in a distributed environment with multiple instances, make sure to first upgrade the agents, then the masters.
• ⚠️ BREAKING Issue #2889 Changed column name from Name to Variable name for PQ SHOW META.
• ⚠️ BREAKING Issue #879 Introduced per-table binlog with new options: binlog_common, binlog for create table / alter table. You need to make a clean shutdown of the Manticore instance before upgrading to the new version.
• ⚠️ BREAKING Issue #1789 Fixed incorrect error message when a node joins a cluster with the wrong replication protocol version. This change updates the replication protocol. If you are running a replication cluster, you need to:
  • First, cleanly stop all your nodes
  • Then, start the node that was stopped last with --new-cluster, using the tool manticore_new_cluster in Linux.
  • Read about restarting a cluster for more details.
• ⚠️ BREAKING Issue #2308 Added support for multiple tables in ALTER CLUSTER ADD and DROP. This change also affects the replication protocol. Refer to the previous section for guidance on handling this update.
• Issue #2997 Fixed dlopen issue on Macos.
• Commit 4954 Changed the default cutoff for OPTIMIZE TABLE on tables with KNN indexes to improve search performance.
• Commit cfc8 Added COUNT(DISTINCT) support for ORDER BY in FACET and GROUP BY.
• Issue #1103 Improved clarity in logging chunk merging.
• Issue #1130 Added support for DBeaver.
• Issue #1546 Implemented secondary indexes for POLY2D()/GEOPOLY2D() functions.
• Issue #1630 HTTP requests now support Content-Encoding: gzip.
• Issue #1831 Added SHOW LOCKS command.
• Issue #2187 Allowed Buddy request to daemon to bypass the searchd.max_connections constraint.
• Issue #2208 Added support for joining tables via the JSON HTTP interface.
• Issue #2235 Log successfully processed queries via Buddy in their original form.
• Issue #2249 Added special mode to run mysqldump for replicated tables.
• Issue #2268 Improved external files renaming on copy for CREATE TABLE and ALTER TABLE statements.
• Issue #2402 Updated searchd.max_packet_size default to 128MB.
• Issue #2419 Added support for IDF boost modifier in JSON "match".
• Issue #2430 Enhanced binlog writing synchronization to prevent errors.
• Issue #2479 Added support for SHOW TABLE INDEXES command.
• Issue #2485 Set session metadata for Buddy replies.
• Issue #2490 Millisecond resolution for aggregations at compatibility endpoints.
• Issue #2500 Changed error messages for cluster operations when replication fails to start.
• Issue #2584 New performance metrics in SHOW STATUS: min/max/avg/95th/99th percentile per query type for the last 1, 5, and 15 minutes.
• Issue #2639 Replaced all instances of index with table in requests and responses.
• Issue #2643 Added distinct column to HTTP /sql endpoint aggregation results.
• Issue #268 Implemented autodetection of data types imported from Elasticsearch.
• Issue #2744 Added collation support for string JSON field comparison expressions.
• [Issue #2752](https://githu...

Manticore Search 6.3.8

Manticore Search 6.3.8

Released: November 22nd 2024

➡️ INSTALL ⬅️

Version 6.3.8 continues the 6.3 series and includes only bug fixes.

Bug fixes

• PR #2777 Fixed the calculation of available threads when query concurrency is limited by the threads or max_threads_per_query settings.

Manticore Search 6.3.6

Manticore Search 6.3.6

Released: August 2nd 2024

➡️ INSTALL ⬅️

Version 6.3.6 continues the 6.3 series and includes only bug fixes.

Bug fixes

• Issue #2477 Fixed a crash introduced in version 6.3.4, which could occur when dealing with expressions and distributed or multiple tables.
• Issue #2352 Fixed a daemon crash or internal error upon early exit caused by max_query_time when querying multiple indexes.

Manticore Search 6.3.4

Version 6.3.4

Released: July 31st 2024

➡️ INSTALL ⬅️

Version 6.3.4 continues the 6.3 series and includes only minor improvements and bug fixes.

Minor changes

• Issue #1130 Added support for DBeaver.
• Issue #2146 Improved escaping of delimiters in word forms and exceptions.
• Issue #2268 Improved external files renaming on copy for CREATE and ALTER TABLE statements.
• Issue #2315 Added string comparison operators to SELECT list expressions.
• Issue #2363 Added support for null values in Elastic-like bulk requests.
• Issue #2374 Added support for mysqldump version 9.
• Issue #2375 Improved error handling in HTTP JSON queries with JSON path to the node where the error occurs.

Bug fixes

• Issue #2280 Fixed performance degradation in wildcard queries with many matches when disk_chunks > 1.
• Issue #2332 Fixed crash in MVA MIN or MAX SELECT list expressions for empty MVA arrays.
• Issue #2339 Fixed incorrect processing of Kibana's infinite range request.
• Issue #2342 Fixed join filter on columnar attributes from the right table when the attribute is not in the SELECT list.
• Issue #2343 Fixed duplicate 'static' specifier in Manticore 6.3.2.
• Issue #2344 Fixed LEFT JOIN returning non-matching entries when MATCH() over the right table is used.
• Issue #2350 Fixed saving of disk chunk at RT index with hitless_words.
• Issue #2364 The 'aggs_node_sort' property can now be added in any order among other properties.
• Issue #2368 Fixed error on full-text vs filter order in the JSON query.
• Issue #2376 Fixed bug related to incorrect JSON response for long UTF-8 requests.
• Issue #2684 Fixed calculation of presort/prefilter expressions that depend on joined attributes.
• Issue #301 Changed the method of calculating data size for metrics to read from the manticore.json file instead of checking the entire size of the data directory.
• Issue #302 Added handling of validation requests from Vector.dev.

Manticore Search 6.3.2

Version 6.3.2

Released: June 26th 2024

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Version 6.3.2 continues the 6.3 series and includes several bug fixes, some of which were discovered after the release of 6.3.0.

Breaking changes

• ⚠️Issue #2305 Updated aggs.range values to be numeric.

Bug fixes

• Commit c51c Fixed grouping by stored check vs rset merge.
• Commit 0e85 Fixed a crash in the daemon when querying with wildcard characters in an RT index using a CRC dictionary and local_df enabled.
• Issue #2200 Fixed a crash in JOIN on count(*) without GROUP BY.
• Issue #2201 Fixed JOIN not returning a warning when attempting grouping by a full-text field.
• Issue #2230 Addressed issue where adding an attribute via ALTER TABLE did not take KNN options into account.
• Issue #2231 Fixed failure in removing manticore-tools Redhat package in version 6.3.0.
• Issue #2242 Corrected issues with JOIN and multiple FACET statements returning incorrect results.
• Issue #2250 Fixed ALTER TABLE producing an error if the table is in a cluster.
• Issue #2252 Fixed the original query being passed into buddy from the SphinxQL interface.
• Issue #2267 Improved wildcard expansion in the CALL KEYWORDS for RT index with disk chunks.
• Issue #271 Fixed hanging of incorrect /cli requests.
• Issue #274 Resolved issues where concurrent requests to Manticore could get stuck.
• Issue #275 Fixed hanging of drop table if exists t; create table t via /cli.

Replication-related

• Issue #2270 Added support for cluster:name format in the /_bulk HTTP endpoint.

Manticore Search 6.3.0

Version 6.3.0

Released: May 23rd 2024

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

📢📢📢Announcement blog post📢📢📢

Major changes

• Issue #839 Implemented float_vector data type; implemented vector search.
• Issue #1673 INNER/LEFT JOIN (beta stage).
• Issue #1744 Implemented autodetection of date formats for timestamp fields.
• Issue #1720 Changed Manticore Search license from GPLv2-or-later to GPLv3-or-later.
• Commit 7a55 Running Manticore in Windows now requires Docker to run Buddy.
• Issue #1541 Added a REGEX full-text operator.
• Issue #2091 Ubuntu Noble 24.04 support.
• Commit 514d Revamp of time operations for better performance and new date/time functions:
  • CURDATE() - Returns current date in local timezone
  • QUARTER() - Returns the integer quarter of the year from a timestamp argument
  • DAYNAME() - Returns the weekday name for a given timestamp argument
  • MONTHNAME() - Returns the name of the month for a given timestamp argument
  • DAYOFWEEK() - Returns the integer weekday index for a given timestamp argument
  • DAYOFYEAR() - Returns the integer day of the year for a given timestamp argument
  • YEARWEEK() - Returns the integer year and the day code of the first day of current week for a given timestamp argument
  • DATEDIFF() - Returns the number of days between two given timestamps
  • DATE() - Formats the date part from a timestamp argument
  • TIME() - Formats the time part from a timestamp argument
  • timezone - Timezone used by date/time-related functions.
• Commit 30e7 Added range, histogram, date_range, and date_histogram aggregates to the HTTP interface and similar expressions into SQL.

Minor changes

• Issue #1285 Support of Filebeat versions 8.10 - 8.11.
• Issue #1771 ALTER TABLE ... type='distributed'.
• Issue #1788 Added the ability to copy tables using the CREATE TABLE ... LIKE ... WITH DATA SQL statement.
• Issue #2072 Optimized the table compacting algorithm: Previously, both manual OPTIMIZE and automatic auto_optimize processes would first merge chunks to ensure the count did not exceed the limit, and then expunge deleted documents from all other chunks containing deleted documents. This approach was sometimes too resource-intensive and has been disabled. Now, chunk merging occurs solely according to the progressive_merge setting. However, chunks with a high number of deleted documents are more likely to be merged.
• Commit ce6c Added protection against loading a secondary index of a newer version.
• Issue #1417 Partial replace via REPLACE INTO ... SET.
• Commit 7c16 Updated default merge buffer sizes: .spa (scalar attrs): 256KB -> 8MB; .spb (blob attrs): 256KB -> 8MB; .spc (columnar attrs): 1MB, no change; .spds (docstore): 256KB -> 8MB; .spidx (secondary indexes): 256KB buffer -> 128MB memory limit; .spi (dictionary): 256KB -> 16MB; .spd (doclists): 8MB, no change; .spp (hitlists): 8MB, no change; .spe (skiplists): 256KB -> 8MB.
• Issue #1859 Added composite aggregation via JSON.
• Commit 216b Disabled PCRE.JIT due to issues with some regex patterns and no significant time benefit.
• Commit 55cd Added support for vanilla Galera v.3 (api v25) (libgalera_smm.so from MySQL 5.x).
• Commit 86f9 Changed metric suffix from _rate to _rps.
• Commit c0c1 Improved docs about balancer HA support.
• Commit d1d2 Changed index to table in error messages; fixed bison parser error message fixup.
• Commit fd26 Support manticore.tbl as table name.
• Issue #1105 Support for running indexer via systemd (docs). ❤️ Thank you, @subnix for the PR.
• Issue #1294 Secondary indexes support in GEODIST().
• Issue #1394 Simplified SHOW THREADS.
• Issue #1424 Added support for the default values (agent_connect_timeout and agent_query_timeout) for create distributed table statement.
• Issue #1442 Added expansion_limit search query option that overrides searchd.expansion_limit.
• Issue #1448 Implemented ALTER TABLE for int->bigint conversion.
• Issue #146 Meta information in MySQL response.
• Issue #1494 SHOW VERSION.
• Issue #1582 Support of deleting documents by id array via JSON.
• Issue #1589 Improve error "unsupported value type".
• Issue #1634 Added Buddy ve...

Manticore Search 6.2.12

Manticore Search 6.2.12

Released: Aug 23rd 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Version 6.2.12 continues the 6.2 series and addresses issues discovered after the release of 6.2.0.

Bugfixes

• ❗Issue #1351 "Manticore 6.2.0 doesn't start via systemctl on Centos 7": Modified TimeoutStartSec from infinity to 0 for better compatibility with Centos 7.
• ❗Issue #1364 "Crash after upgrading from 6.0.4 to 6.2.0": Added replay functionality for empty binlog files from older binlog versions.
• PR #1334 "fix typo in searchdreplication.cpp": Corrected a typo in searchdreplication.cpp: beggining -> beginning.
• Issue #1337 "Manticore 6.2.0 WARNING: conn (local)(12), sock=8088: bailing on failed MySQL header, AsyncNetInputBuffer_c::AppendData: error 11 (Resource temporarily unavailable) return -1": Lowered the verbosity level of the MySQL interface warning about the header to logdebugv.
• Issue #1355 "join cluster hangs when node_address can't be resolved": Improved replication retry when certain nodes are unreachable, and their name resolution fails. This should resolve issues in Kubernetes and Docker nodes related to replication. Enhanced the error message for replication start failures and made updates to test model 376. Additionally, provided a clear error message for name resolution failures.
• Issue #1361 "No lower case mapping for "Ø" in charset non_cjk": Adjusted the mapping for the 'Ø' character.
• Issue #1365 "searchd leaves binlog.meta and binlog.001 after clean stop": Ensured that the last empty binlog file is removed properly.
• Commit 0871: Fixed the Thd_t build issue on Windows related to atomic copy restrictions.
• Commit 1cc0: Addressed an issue with FT CBO vs ColumnarScan.
• Commit c6bf: Made corrections to test 376 and added a substitution for the AF_INET error in the test.
• Commit cbc3: Resolved a deadlock issue during replication when updating blob attributes versus replacing documents. Also removed the rlock of the index during commit because it's already locked at a more basic level.

Minor changes

• Commit 4f91 Updated info on /bulk endpoints in the manual.

MCL

• Support of Manticore Columnar Library v2.2.4

Manticore Search 6.2.0

Manticore Search 6.2.0

Released: Aug 4th 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Major changes

• The query optimizer has been enhanced to support full-text queries, significantly improving search efficiency and performance.
• Integrations with:
  • mysqldump - to make logical backups using mysqldump
  • Apache Superset and Grafana to visualize data stored in Manticore
  • HeidiSQL and DBForge for easier development with Manticore
• We've started using GitHub workflows, making it simpler for contributors to utilize the same Continuous Integration (CI) process that the core team applies when preparing packages. All jobs can be run on GitHub-hosted runners, which facilitates seamless testing of changes in your fork of Manticore Search.
• We've started using CLT to test complex scenarios. For example, we're now able to ensure that a package built after a commit can be properly installed across all supported Linux operating systems. The Command Line Tester (CLT) provides a user-friendly way to record tests in an interactive mode and to effortlessly replay them.
• Significant performance improvement in count distinct operation by employing a combination of hash tables and HyperLogLog.
• Enabled multithreaded execution of queries containing secondary indexes, with the number of threads limited to the count of physical CPU cores. This should considerably improve the query execution speed.
• pseudo_sharding has been adjusted to be limited to the number of free threads. This update considerably enhances the throughput performance.
• Users now have the option to specify the default attribute storage engine via the configuration settings, providing better customization to match specific workload requirements.
• Support for Manticore Columnar Library 2.2.0 with numerous bug fixes and improvements in Secondary indexes.

Minor changes

• Buddy #153: The /pq HTTP endpoint now serves as an alias for the /json/pq HTTP endpoint.
• Commit 0bf1: We've ensured multi-byte compatibility for upper() and lower().
• Commit 2bb9: Instead of scanning the index for count(*) queries, a precalculated value is now returned.
• Commit 3c84: It's now possible to use SELECT for making arbitrary calculations and displaying @@sysvars. Unlike before, you are no longer limited to just one calculation. Therefore, queries like select user(), database(), @@version_comment, version(), 1+1 as a limit 10 will return all the columns. Note that the optional 'limit' will always be ignored.
• Commit 6aca: Implemented the CREATE DATABASE stub query.
• Commit 9dc1: When executing ALTER TABLE table REBUILD SECONDARY, secondary indexes are now always rebuilt, even if attributes weren't updated.
• Commit 46ed: Sorters utilizing precalculated data are now identified before using CBO to avoid unnecessary CBO calculations.
• Commit 102a: Implementing mocked and utilizing of the full-text expression stack to prevent daemon crashes.
• Commit 979f: A speedy code path has been added for match cloning code for matches that don't use string/mvas/json attributes.
• Commit a073: Added support for the SELECT DATABASE() command. However, it will always return Manticore. This addition is crucial for integrations with various MySQL tools.
• Commit bc04: Modified the response format of the /cli endpoint, and added the /cli_json endpoint to function as the previous /cli.
• Commit d70b: The thread_stack can now be altered during runtime using the SET statement. Both session-local and daemon-wide variants are available. Current values can be accessed in the show variables output.
• Commit d96e: Code has been integrated into CBO to more accurately estimate the complexity of executing filters over string attributes.
• Commit e77d: The DocidIndex cost calculation has been improved, enhancing overall performance.
• Commit f3ae: Load metrics, similar to 'uptime' on Linux, are now visible in the SHOW STATUS command.
• Commit f3cc: The field and attribute order for DESC and SHOW CREATE TABLE now match that of SELECT * FROM.
• Commit f3d2: Different internal parsers now provide their internal mnemonic code (e.g., P01) during various errors. This enhancement aids in identifying which parser caused an error and also obscures non-essential internal details.
• Issue #271 "Sometimes CALL SUGGEST does not suggest a correction of a single letter typo": Improved SUGGEST/QSUGGEST behaviour for short words: added the option sentence to show the entire sentence
• Issue #696 "Percolate index does not search properly by exact phrase query when stemming enabled": The percolate query has been modified to handle an exact term modifier, improving search functionality.
• Issue #829 "DATE FORMATTING methods": added the date_format() select list expression, which exposes the strftime() function.
• Issue #961 "Sorting buckets via HTTP JSON API": introduced an optional sort property for each bucket of aggregates in the HTTP interface.
• Issue #1062 "Improve error logging of JSON insert api failure - "unsupported value type"": The /bulk endpoint reports information regarding the number of processed and non-processed strings (documents) in case of an error.
• Issue #1070 "CBO hints don't support multiple attributes": Enabled index hints to handle multiple attributes.
• Issue #1106 "Add tags to http search query": Tags have been added to HTTP PQ responses.
• Issue #1301 "buddy should not create table in parallel": Resolved an issue that was causing failures from parallel CREATE TABLE operations. Now, only one CREATE TABLE operation can run at a time.
• Issue #1303 "add support of @ to column names".
• Issue #1316 "Queries on taxi dataset are slow with ps=1": The CBO logic has been refined, and the default histogram resolution has been set to 8k for better accuracy on attributes with randomly distributed values.
• Issue #1317 "Fix CBO vs fulltext on hn dataset": Enhanced logic has been implemented for determining when to use bitmap iterator intersection and when to use a priority queue.
• Issue #1318 "columnar: change iterator interface to single-call" : Columnar iterators now use a single Get call, replacing the previous two-step AdvanceTo + Get calls to retrieve a value.
• Issue #1319 "Aggregate calc speedup (remove CheckReplaceEntry?)": The CheckReplaceEntry call was removed from...

Manticore Search 6.0.4

Manticore Search 6.0.4

Released: Mar 15 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

New features

• Improved integration with Logstash, Beats etc. including:
  • Support for Logstash versions >= 7.13.
  • Auto-schema support.
  • Added handling of bulk requests in Elasticsearch-like format.
• Buddy commit ce90 Log Buddy version on Manticore start.

Bugfixes

• Issue #588, Issue #942 Fixed bad character at the search meta and call keywords for bigram index.
• Issue #1027 Lowercase HTTP headers are rejected.
• ❗Issue #1039 Fixed memory leak at daemon on reading output of the Buddy console.
• Issue #1056 Fixed unexpected behavior of question mark.
• Issue #1064 - Fixed race condition in tokenizer lowercase tables causing a crash.
• Commit 59bb Fixed bulk writes processing in the JSON interface for documents with id explicitly set to null.
• Commit 7b6b Fixed term statistics in CALL KEYWORDS for multiple same terms.
• Commit f381 Default config is now created by Windows installer; paths are no longer substituted in runtime.
• Commit 6940, Commit cc5a Fixed replication issues for cluster with nodes in multiple networks.
• Commit 4972 Fixed /pq HTTP endpoint to be an alias of the /json/pq HTTP endpoint.
• Commit 3b53 Fixed daemon crash on Buddy restart.
• Buddy commit fba9 Display original error on invalid request received.
• Buddy commit db95 Allow spaces in backup path and add some magic to regexp to support single quotes also.

Manticore Search 6.0.2

Manticore Search 6.0.2

Released: Feb 10 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Bugfixes

• Issue #1024 crash 2 Crash / Segmentation Fault on Facet search with larger number of results
• ❗Issue #1029 - WARNING: Compiled-in value KNOWN_CREATE_SIZE (16) is less than measured (208). Consider to fix the value!
• ❗Issue #1032 - Manticore 6.0.0 plain index crashes
• ❗Issue #1033 - multiple distributed lost on daemon restart