[Serverless] Support testing against development serverless projects

[kpollich]

Ref draft PR #1766
Ref previous PR #1231

To support package development for serverless, elastic-package should support spinning up Elasticsearch and Kibana in serverless mode in its local stack, e.g.

elastic-package stack up --serverless

We could also add serverless: true as a profile setting instead of using a CLI argument.

[kpollich]

cc @jsoriano @ruflin

I created this issue to capture discussion as it's not clear what the intended result is here based on the existing PR's. I believe elastic-package stack can already be used to manage serverless projects, they're just not local Docker containers. @jsoriano do we have any docs on that process?

[jsoriano]

@jsoriano do we have any docs on that process?

Umm, unfortunately it looks like we only have some documentation about the variables that can be used with the serverless provider (here), but not about this provider itself. @mrodm do you remember if we have some documentation about stack providers?

[jsoriano]

FTR, serverless support was added in #1374.

[mrodm]

@mrodm do you remember if we have some documentation about stack providers?

AFAIK I think there is no documentation about the stack providers available in elastic-package (e.g. serverless) :( Just the list of options that you already linked.

[ruflin]

elastic-package stack up is used for local development of integrations but also for testing latest snapshot builds quickly. Like it is done today for stateful, it should be as simple to setup a stateless setup (at least view, feature set) on your local machine to do the testing.

[jsoriano]

I gave another try to start the serverless containers locally, managed by elastic-package and docker-compose, with no success. It seems to have certain complexity, it is not very documented beyond the scripts in the Kibana developer environment, and failures found are quite dark.
Given the complexity and the uncertainty of its maintenance costs, I am not sure if it is worth to maintain another way to deploy serverless, given also that elastic-package has support for actual serverless, and to reuse any other stack via environment variables.

So I am also considering other options:

• Add a provider for devenv, so we delegate the complexity to the official tool for serverless development environments, as we did with rally for benchmarks.
• Add a provider for the kibana development environment. Though I have neither been able to start it yet.
• Improve support to reuse other environments, like adding some feature to "import" stack configurations to profiles or increase support of authentication methods (like Support API Key as environment variable #1633).

From these options I think that the one that would provide more value is the last one. In my opinion package developers should use the existing serverless provider in most of the cases. If they, or some kibana developer, are working at the same time on some kibana feature, they will likely have a kibana developer environment, and it would be nice to make it easier for elastic-package to leverage this existing environment, and same thing with serverless development and devenv. Improvements in this area would also give us better foundations to support other environments that might appear in the future.

@ruflin @kpollich wdyt?

cc @mrodm

[jsoriano]

adding some feature to "import" stack configurations to profiles

Created issue about this, as we had yesterday another conversation about how to use elastic-package with the kibana developer environment: #1999