You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The ElasticsearchIO testId is currently defined as
// remove the mapperFactory from the test id
override def testId: String =
s"ElasticsearchIO(${esOptions.nodes}, ${esOptions.usernameAndPassword})"
In addition, a JobTest pipeline output must be unique and only written to once. Combine, this makes testing a pipeline that writes to Elasticsearch more than once challenging. I suggest allowing for an Elasticsearch index name in the ElasticsearchOptions case class that would be used by testId as well as passed down to IndexBuilder.
I'm unsure how common this ES access pattern is in Scio Pipelines. My use case requires replicating BigQuery tables to Elasticsearch. To work with the existing constraints I could parameterize the job and replicate 1 bq table to 1 es index at a time rather than N bq tables to M es indexes in one shot.
The text was updated successfully, but these errors were encountered:
The ElasticsearchIO testId is currently defined as
In addition, a JobTest pipeline output must be unique and only written to once. Combine, this makes testing a pipeline that writes to Elasticsearch more than once challenging. I suggest allowing for an Elasticsearch index name in the ElasticsearchOptions case class that would be used by testId as well as passed down to IndexBuilder.
I'm unsure how common this ES access pattern is in Scio Pipelines. My use case requires replicating BigQuery tables to Elasticsearch. To work with the existing constraints I could parameterize the job and replicate 1 bq table to 1 es index at a time rather than N bq tables to M es indexes in one shot.
The text was updated successfully, but these errors were encountered: