2.1.0

Features

Documentation for Elasticsearch.Net and NEST 2.x

• #1935 Add support for the Profile API
• #1924 Add support for render search template
• #1923 Add support for indices shard stores
• #1922 Add support for force merge
• #1921 Add support for cat snapshots
• #1920 Add support for cat repositories
• #1919 Add support for cat segments

Bugs/Fixes

• #1863 Sort failing due to serialization error
• #1929 Handle null _score inside ConcreteTypeConverter
• #1928 and #1952 Capture the Response bytes and HTTP status code when a BadAuth Response occurs and do not throw a serialization exception when attempting to deserialize (the response may not be json e.g. could be using a proxy for auth)
• #1947 Remove RethrowKeepingStackTrace in favour of simply throwing the exception; the former was an artifact of the older TPL-based async implementation which is unnecessary with the async/await async implementation
• #1931 Fix bug in Filter Aggregation with script
• #1930 Fix bug in TermsQueryDescriptor where passing an IEnumerable<T> to Terms<T>() without specifying T, results in the params T[] overload being called
• #1917 Fix bug in FieldStats when less than or equal to comparison used on Index constraint
• #1870 allow .Fields(Fields) everywhere

2.0.5

#1856 Fix DisableDirectStreaming not taking on error responses
#1860 Fix deserialization exception on GetIndex() when refresh_interval == -1
#1868 Add support for term filtering on _termvector and _mtermvector
#1880 Fix handling of the month/year intervals and serialization fixes for the Time class
#1887 Fix handling of return types when using the low-level client (ty @rossbrower)
#1901 Fix deserialization of known errors even if the response isn't valid JSON
#1904 Fix OnRequestCompleted not invoked when an exception is thrown

1.8.0

Upgraded to Json.NET 8

1.7.2

#1861 Limit NEST dependency to Elasticsearch.Net < 2.0.0
#1781 Raise upper limit on Newtonsoft.Json dependency
#1734 DynamicTemplates: Unable to specify FieldIndexOption.NotAnalyzed (generic mapping descriptor missing enum overload for index())
#1682 Add ability to specify type selector on scroll (ty @bytenik)
#1680 Raw query serialization fixes
#1634 Fix NRE in SniffingConnectionPool when running on Mono
#1626 DateHistogram aggregation: parse dates using DateTimeKind.Utc
#1614 Add ability to pass a TextWriter to EnableTrace (ty @ammachado)
#1609 Fix serialization when indexing a JObject (don't treat it as an IEnumerable<object>)
#1600 Mark MultiField property descriptor obsolete (ty @TheFireCookie)
#1595 SuggestOption missing Highlighted field
#1591 Fix missing search and index routing from _cat aliases response (ty @robertlyson)
#1589 Fix premature disposal of request state when using async
#1587 Add support for detect_noop to the update API (ty @robertlyson)
#1570 #1655 Fix serialization of PercolateDescriptor and add missing _score to PercolateResponse (ty @mikelazell)
#1568 Fix minor type in xmldocs (ty @SeanKillen)
#1566 Filter aggregation did not support raw JSON
#1563 Fix serialization of protected_words_path in the word delimitor token filter (ty @robertlyson)
#1599 Add missing RandomScore to FunctionScoreFunctionsDescriptor
#1553 Add HighlightQuery to HighlightFieldDescriptor (ty @Settler)

2.0.4

#1869 Improvements to DebugInformation and NRE fix when dispatch failed
#1867 Fixed meta data serialization of register percolator
#1866 Add missing values to multi term vectors response
#1845 Fixed missing information (CausedBy) from ElasticsearchClientException

2.0.3

#1852 Overriding auto mapped properties caused duplicates and ensure AutoMap is an idempotent operation
#1836 GeoUnit and DistanceType ignored when sorting
#1826 Fix NullReferenceException in attachment mapping
#1819 Boost factor on fields wasn't being serialized
#1857 #1858 Enum cache should be concurrent and cache all code paths
#1814 GeoLocation implicit conversion fixes (ty @niemyjski)
#1810 StickyConnectionPool implementation (ty @Romanx)

2.0.2

Bug Fixes

#1817 Do not serialize null or conditionless queries within collections
#1820 Fix NullReferenceException that occurred when serializing a conditionless BoolQuery
#1821 Max recursion depth was ignored when using AutoMap() from the put mapping API

2.0.1

Patch release to fix a sniffer parsing bug (#1811 as reported by @joelse)

GA Release 2.0.0

This marks the first GA release of our 2.0 client with well over a 1000 commits since 1.7.1 (the currently last released NEST version in the 1.x range).

Improvements

• #1788 Rename IElasticsearchClient to IElasticLowLevelClient to better specify intent
• #1795 Add SuffixExpressionVisitor for appending suffixes to lambda expressions
• #1801 .ConfigureAwait(false) all async/await methods
• #1794 add DebugInformation to responses and exceptions to aid in understanding problems with client calls

Bug Fixes

• #1789 Move highlight_query to HighlightField
• #1787 Fix generated XML comment documentation
• #1786 Move detect_noop back to the request body for an update request
• #1784 When specifying a format and extended_bounds on a date histogram aggregation, include the date_optional_time format to allow Elasticsearch to deserialize the bounds

Back to the drawing board

We took some time to go back to the drawing board. NEST is quite old (started in 2010) and not all of the choices that have accumulated in the code base make sense anymore.
So we stepped back to properly formalize how we see the lifetime of a call and worked off of that. Armed with the following diagram, we completely rewrote NEST's internals; The old Task Parallel Library (TPL) based code is now replaced with async/await, we have a much saner approach to exceptions and errors, and we expose enough information as an audit trail so you don't ever have to guess what went down during a call.

Our internals now also reflect this:

• IElasticClient exposes all the Elasticsearch API endpoints e.g client.Search this calls into ITransport's 2 methods Request and RequestAsync
• the default ITransport uses the passed in IRequestPipelineFactory to create an RequestPipeline which implements IPipeline.

This pipeline now handles all of the failover/sniffing/pinging logic and directly reflects the flow diagram.

We also simplified IConnection down just 2 methods. This means the outer edges are clean (ITransport and IConnection) and implementing your own should be really really simple. All of these (and also IMemoryStreamProvider and IDateTimeProvider) can be injected on the constructor of the client.

Test Framework

Another huge endeavour is the rework of our test framework. NEST 1.x was always well tested but used 5 different test projects and 5 years worth of changing our minds as how best to write tests and assertions, thus becoming a big hodgepodge of nunit assertions, fluent assertions, FakeItEasy, Moq combined with several different ways to compare json with object graphs and vice-versa. Trying to write a new test quickly became an exercise in yak shaving because there was no clear cut way on how best to write said test.

So the first thing we did as part of our 2.0 branch was to completely delete all of our tests. This act of insanity gave us carte blanche during our rewrite.

As of 2.0 we have one test project Tests and all of the tests are written in such a way that they can be run in unit test mode and integration test mode. Write once, run differently.
All of the API endpoint tests test all 4 request variations - two DSL's (fluent and object initializer) with both synchronous and asynchronous variations. We also test all of the moving parts of the Elasticsearch DSL (Aggregations, Sorting, IndexSettings, etc...) in the same way.

In addition to the more formal unit and integration tests, we also implemented a thing we dubbed Literate Testing to allow us to write tests in a more story telling form, with multi-line comments serving as the narrative for our asciidoc documentation while using the Roslyn compiler to pick out the interesting bits of code. This gives us the benefit of always compiling our documentation in addition to having only one place where we document, test and assert how a piece of code is supposed to work.

Another huge component of our testing framework is the Virtual Cluster that allows us to write tests for any situation and how we expect the client to behave.

/** we set up a 10 node cluster with a global request time out of 20 seconds.
* Each call on a node takes 10 seconds. So we can only try this call on 2 nodes
* before the max request time out kills the client call.
*/
var audit = new Auditor(() => Framework.Cluster
  .Nodes(10)
  .ClientCalls(r => r.FailAlways().Takes(TimeSpan.FromSeconds(10)))
  .ClientCalls(r => r.OnPort(9209).SucceedAlways())
  .StaticConnectionPool()
  .Settings(s => s.DisablePing().RequestTimeout(TimeSpan.FromSeconds(20)))
);

audit = await audit.TraceCalls(
  new ClientCall {
    { BadResponse, 9200 }, //10 seconds
    { BadResponse, 9201 }, //20 seconds
    { MaxTimeoutReached }
  },
  /**
  * On the second client call we specify a request timeout override to 80 seconds
  * We should now see more nodes being tried.
  */
  new ClientCall(r => r.RequestTimeout(TimeSpan.FromSeconds(80)))
  {
    { BadResponse, 9203 }, //10 seconds
    { BadResponse, 9204 }, //20 seconds
    { BadResponse, 9205 }, //30 seconds
    { BadResponse, 9206 }, //40 seconds
    { BadResponse, 9207 }, //50 seconds
    { BadResponse, 9208 }, //60 seconds
    { HealthyResponse, 9209 },
  }
);

This showcases the Virtual Cluster tests combined with Literate Tests and the extensive audit trail information available on each response (or exception).

I'm pleased to say we are back at a decent coverage rate (60%) and will continue to iterate and improve this.

Exception handling

Another big change in NEST 2.0 is how we deal with exceptions.

In NEST 1.x, the client threw a multitude of exceptions: MaxRetryException, ElasticsearchAuthException, ElasticsearchServerException, DslException, etc.. This made it challenging for users to handle exceptions and invalid responses, and understand the root cause of errors. On top of that, the types of exceptions thrown depended on what kind of IConnectionPool was injected, in order to maintain maximum backwards compatibility with NEST 0.x.

In NEST 2.x, exceptions are much more deterministic. The former ThrowOnElasticsearchServerExceptions() setting has been replaced with the more succinct ThrowExceptions(), which determines whether the client should ever throw an exception or not (client side and server exceptions). Furthermore, the types of exceptions have been reduced and simplified down to three types of exceptions:

ElasticsearchClientException: These are known exceptions, either an exception that occurred in the request pipeline(such as max retries or timeout reached, bad authentication, etc...) or Elasticsearch itself returned an error (could not parse the request, bad query, missing field, etc...). If it is an Elasticsearch error, the ServerError property on the response will contain the the actual error that was returned. The inner exception will always contain the root causing exception.

UnexpectedElasticsearchClientException: These are unknown exceptions, for instance a response from Elasticsearch not properly deserialized. These are usually bugs in the client and we encourage you to report them. This exception also inherits from ElasticsearchClientException so an additional catch block isn't necessary to handle but can be helpful in distinguishing between the two.

Runtime exceptions: These are CLR exceptions like ArgumentException, NullArgumentException etc. that are thrown when an API in the client is misused.

Breaking Changes

Even though a lot of work went into the interior, the exterior did not escape unscathed! On top of the many breaking changes that Elasticsearch 2.0 introduces, there are more then a few NEST 2.0 introduces.
We revalidated all the request and response domain objects against Elasticsearch 2.0.

A pretty complete list of breaking changes are available:

• Low level client changes, Elasticsearch.NET
• high level client changes, NEST

Elasticsearch 2.x support

NEST 2.0 supports all the new features in Elasticsearch 2.0 including pipeline aggregations. New features from Elasticsearch 2.2 have not yet been mapped.

Here we'll just highlight a couple features that are reflected in the NEST changes to whet your appetite!

Filters are Gone Gone Gone

In Elasticsearch 2.0, query and filter constructs have merged into one concept called queries and NEST 2.0 reflects this. So if you were previously using the AndFilter, this is now called the AndQuery; beware that some of these filters have been obsoleted and chances are high you were using them wrong!.

Isolated descriptors

In NEST 1.x we took pride in being a 1 to 1 mapping with the Elasticsearch API. In some cases however, this purity hid the real depth of parameters. As an example,

in NEST 1.x you could add sorts on Search() using:

client.Search<Project>(s=>s
  .SortAscending(...)
  .SortScript(...)
)

in NEST 2.0 you have to stoop down a level first in order to access the same functionality

client.Search<Project>(s=>s
  .Sor...

2.0.0-rc1

Release Candidate

This is the release candidate for NEST 2.x! Many improvements and fixes went into this release, including providing support for two new target frameworks, dotnet5.1 (.NETPlatform5.1) and net46 (.NETFramework4.6). As always, your feedback on this release are very welcome!

Improvements

• #1697 SUPPORT .NET CORE WITH DNX TOOLING
• #1089 Refactor to use System.Linq.Expressions.ExpressionVisitor
• #1686 Support other_bucket and other_bucket_key in filters aggregations
• #1705 AutoMap/Infer TimeSpan properties as long ticks
• #1742 Support looping of named aggregation buckets
• #1755 Add .NET 4.6 as a target framework

Bug Fixes

• #985 Reindex supports _parent and _routing
• #1433 Reindex supports reindexing one/many/all types within the from index
• #1490 Update now supports fields and is not generic
• #1612 Unify the methods and overloads available on Query<T>
• #1634 Ensure consistent purified Uris across all supported platforms
• #1708 Property Name should resolve to the last token
• #1729 Add Aggregation Metadata
• #1731 Use custom JsonConverters when supplied on derive JsonNetSerializer
• #1763 Add _timestamp to IHit<T>
• #1763 Add _ttl to IHit<T>
• #1766 when passed an a null reference for string, return null reference for DateMath
• #1767 Fields should not be null on response when specified on request
• #1772 Fix deficiencies in the dnx tooling for supporting dependency version ranges
• #1778 Align request/response type names
• #1776 Rename types in line with Naming Conventions
• #1773 Make ConnectionSettings protected on JsonNetSerializer