Skip to content

Commit

Permalink
FIX: replaces all ES with Elasticsearch (fixes #202) (#205)
Browse files Browse the repository at this point in the history
  • Loading branch information
bernardbeckerman authored Mar 10, 2020
1 parent 5d110e7 commit 073b613
Show file tree
Hide file tree
Showing 12 changed files with 49 additions and 49 deletions.
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ You may notice that this strategy means that `uptasticsearch` is tested for back

When developing on this package, you may want to run Elasticsearch locally to speed up the testing cycle. We've provided some gross bash scripts at the root of this repo to help!

To run the code below, you will need [Docker](https://www.docker.com/). Note that I've passed an argument to `setup_local.sh` indicating the major version of ES I want to run. If you don't do that, this script will just run the most recent major version of Elasticsearch. Look at the source code of `setup_local.sh` for a list of the valid arguments.
To run the code below, you will need [Docker](https://www.docker.com/). Note that I've passed an argument to `setup_local.sh` indicating the major version of Elasticsearch I want to run. If you don't do that, this script will just run the most recent major version of Elasticsearch. Look at the source code of `setup_local.sh` for a list of the valid arguments.

```
# Start up Elasticsearch on localhost:9200 and seed it with data
Expand Down
18 changes: 9 additions & 9 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@

## Features

### Added support for ES7.x
- [#161](https://github.com/uptake/uptasticsearch/pull/161) Added support for ES7.x. The biggest changes between that major version and 6.x were the removal of `_all` as a way to reference all indices, changing the response format of `hits.total` into an object like `{"hits": {"total": 50}}`, and restricting all indices to have a single type of document. More details can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html.
### Added support for Elasticsearch 7.x
- [#161](https://github.com/uptake/uptasticsearch/pull/161) Added support for Elasticsearch 7.x. The biggest changes between that major version and 6.x were the removal of `_all` as a way to reference all indices, changing the response format of `hits.total` into an object like `{"hits": {"total": 50}}`, and restricting all indices to have a single type of document. More details can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html.

# uptasticsearch 0.3.1

Expand All @@ -26,20 +26,20 @@

## Features

### Full support for ES6.x
- [#64](https://github.com/uptake/uptasticsearch/pull/64) added support for ES6.x. The biggest change between that major version and v5.x is that as of ES6.x all requests issued to the ES HTTP API must pass an explicit `Content-Type` header. Previous versions of ES tried to guess the `Content-Type` when none was declared
- [#66](https://github.com/uptake/uptasticsearch/pull/66) completed support for ES6.x. ES6.x changed the supported strategy for issuing scrolling requests. `uptasticsearch` will now hit the cluster to try to figure out which version of ES it is running, then use the appropriate scrolling strategy.
### Full support for Elasticsearch 6.x
- [#64](https://github.com/uptake/uptasticsearch/pull/64) added support for Elasticsearch 6.x. The biggest change between that major version and v5.x is that as of Elasticsearch 6.x all requests issued to the Elasticsearch HTTP API must pass an explicit `Content-Type` header. Previous versions of Elasticsearch tried to guess the `Content-Type` when none was declared
- [#66](https://github.com/uptake/uptasticsearch/pull/66) completed support for Elasticsearch 6.x. Elasticsearch 6.x changed the supported strategy for issuing scrolling requests. `uptasticsearch` will now hit the cluster to try to figure out which version of Elasticsearch it is running, then use the appropriate scrolling strategy.

## Bugfixes

### `get_fields()` when your index has no aliases
- previously, `get_fields()` broke on some legacy versions of Elasticsearch where no aliases had been created. The response on the `_cat/aliases` endpoint has changed from major version to major version. [#66](https://github.com/uptake/uptasticsearch/pull/66) fixed this for all major versions of ES from 1.0 to 6.2
- previously, `get_fields()` broke on some legacy versions of Elasticsearch where no aliases had been created. The response on the `_cat/aliases` endpoint has changed from major version to major version. [#66](https://github.com/uptake/uptasticsearch/pull/66) fixed this for all major versions of Elasticsearch from 1.0 to 6.2

### `get_fields()` when your index has multiple aliases
- previously, if you had multiple aliases pointing to the same physical index, `get_fields()` would only return one of those. As of [#73](https://github.com/uptake/uptasticsearch/pull/73), mappings for the underlying physical index will now be duplicated once per alias in the table returned by `get_fields()`.

### bad parsing of ES major version
- as of [#64](https://github.com/uptake/uptasticsearch/pull/64), `uptasticsearch` attempts to query the ES host to figure out what major version of Elasticsearch is running there. Implementation errors in that PR led to versions being parsed incorrectly but silently passing tests. This was fixed in [#66](https://github.com/uptake/uptasticsearch/pull/66). NOTE: this only impacted the dev version of the library on Github.
### bad parsing of Elasticsearch major version
- as of [#64](https://github.com/uptake/uptasticsearch/pull/64), `uptasticsearch` attempts to query the Elasticsearch host to figure out what major version of Elasticsearch is running there. Implementation errors in that PR led to versions being parsed incorrectly but silently passing tests. This was fixed in [#66](https://github.com/uptake/uptasticsearch/pull/66). NOTE: this only impacted the dev version of the library on Github.

### `ignore_scroll_restriction` not being respected
- In previous versions of `uptasticsearch`, the value passed to `es_search()` for `ignore_scroll_restriction` was not actually respected. This was possible because an internal function had defaults specified, so we never caught the fact that that value wasn't getting passed through. [#66](https://github.com/uptake/uptasticsearch/pull/66) instituted the practice of not specifying defaults on function arguments in internal functions, so similar bugs won't be able to silently get through testing in the future.
Expand Down Expand Up @@ -77,7 +77,7 @@
## Features

### Main function
- `es_search()` executes an ES query and gets a data.table
- `es_search()` executes an Elasticsearch query and gets a data.table

### Parse raw JSON into data.table
- `chomp_aggs()` converts a raw aggs JSON to data.table
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ The examples presented here pertain to a fictional Elasticsearch index holding s

### Example 1: Get a Batch of Documents <a name="example1"></a>

The most common use case for this package will be the case where you have an ES query and want to get a data frame representation of many resulting documents.
The most common use case for this package will be the case where you have an Elasticsearch query and want to get a data frame representation of many resulting documents.

In the example below, we use `uptasticsearch` to look for all survey results in which customers said their satisfaction was "low" or "very low" and mentioned food in their comments.

Expand Down
2 changes: 1 addition & 1 deletion docs/index.html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 9 additions & 9 deletions docs/news/index.html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions docs/reference/es_search.html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/reference/index.html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions py-pkg/uptasticsearch/fetch_all.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Functions for Pulling data from ES and unpacking into a table
"""Functions for Pulling data from Elasticsearch and unpacking into a table
"""

import pandas as pd
Expand All @@ -14,7 +14,7 @@ def es_search(es_host, es_index, query_body="{}", size=10000, max_hits=None,
Args:
es_host (string): a url of the elasticsearch cluster e.g. http://localhost:9200
es_index (string): the name of the ES index
es_index (string): the name of the Elasticsearch index
query_body (json): json query
size (int): the number of hits per page. Note: the size will not affect max_hits, but it will affect the time to return the max_hits.
max_hits (int, None): the total number of hits to allow. If None, no limit
Expand Down
Loading

0 comments on commit 073b613

Please sign in to comment.