pack: skip unavailable configs from etcd and tcs #1066

patapenka-alexey · 2024-12-16T14:05:55Z

Closes #1038

@TarantoolBot document
Title: pack: skip unavailable configs from etcd and tcs

This patch fixes the tt pack error if etcd or tcs are unavailable during package creation.

test/integration/pack/test_bundles/vshard_app/test_app/router.lua

cli/running/running.go

oleg-jukovec

Please, add an entry into CHANGELOG.md.

elhimov · 2024-12-18T17:45:04Z

cli/pack/opts.go

@@ -49,7 +49,7 @@ func initAppsInfo(cliOpts *config.CliOpts, cmdCtx *cmdcontext.CmdCtx, packCtx *P
 	}
 	packCtx.AppList = appList
 	packCtx.AppsInfo, err = running.CollectInstancesForApps(packCtx.AppList, cliOpts,
-		cmdCtx.Cli.ConfigDir, cmdCtx.Integrity, true)
+		cmdCtx.Cli.ConfigDir, cmdCtx.Integrity, false)


It looks like the same approach is to be applied to both instances scripts and cluster config. So both of them are required or both of them are optional and it seems wrong in case of tt pack. I'm not sure about cluster config, maybe it's correct that we treat it as configuration and thus able to drop from package, but it seems we are not able to treat instances scripts in the same way because they contain business logic and thus look like the essential part of the package.

And regardless of our ability to drop some part of package while preparing it, it looks quite weird that content of the package depends on current availability of some items. What if next time one pack the same application but etcd/tcs config is available? As the content is changed we will get another package for the same application. Do we expect such a behavior?

test/integration/pack/test_pack.py

CHANGELOG.md

cli/cmd/replicaset.go

cli/running/running.go

oleg-jukovec

We still not skip a config load, but load a config and skip an error.

@TarantoolBot

@TarantoolBot document Title: pack: skip configs from etcd and tcs This patch fixes the `tt pack` error if `etcd` or `tcs` are unavailable during package creation. Part of #1038

From now on only the `check` and `start` commands do not skip errors during cluster config generation from local, etcs/tcs and env sources. Closes #1038

patapenka-alexey · 2025-01-11T06:21:55Z

I will update the second commit message after we discussed final solution.

elhimov · 2025-01-13T08:04:55Z

cli/running/running.go

@@ -126,6 +126,14 @@ type providerImpl struct {
 	instanceCtx *InstanceCtx
 }

+type SkipClusterConfig int


This type name confuses me. It will affect not only cluster config but also instances scripts. I would use for type name something more general, like LoadMode or similar and specific name for certain enum values (see below).

elhimov · 2025-01-13T08:23:07Z

cli/running/running.go

+	SkipConfig       SkipClusterConfig = iota // Skip cluster config loading at all.
+	SkipConfigErrors                          // Skip errors at cluster config loading.
+	LoadConfig                                // Load cluster config, trigger an errors if not.


Suggested change

SkipConfig SkipClusterConfig = iota // Skip cluster config loading at all.

SkipConfigErrors // Skip errors at cluster config loading.

LoadConfig // Load cluster config, trigger an errors if not.

SkipScriptsAndClusterConfig LoadMode = iota // Skip loading of instances scripts and cluster config.

IgnoreClusterConfigErrors // Skip errors at cluster config loading.

LoadScriptsAndClusterConfig // Load instances scripts and cluster config, trigger an errors if not.

elhimov · 2025-01-13T09:36:58Z

Please, mark the PR as ready for review (it is draft now).

oleg-jukovec · 2025-01-13T10:09:35Z

cli/running/running.go

@@ -126,6 +126,14 @@ type providerImpl struct {
 	instanceCtx *InstanceCtx
 }

+type SkipClusterConfig int


Suggested change

type SkipClusterConfig int

type ConfigLoad int

oleg-jukovec · 2025-01-13T10:10:26Z

cli/running/running.go

+	SkipConfig       SkipClusterConfig = iota // Skip cluster config loading at all.
+	SkipConfigErrors                          // Skip errors at cluster config loading.
+	LoadConfig                                // Load cluster config, trigger an errors if not.


The idiomatic way to name a enums in Go:

Suggested change

SkipConfig SkipClusterConfig = iota // Skip cluster config loading at all.

SkipConfigErrors // Skip errors at cluster config loading.

LoadConfig // Load cluster config, trigger an errors if not.

ConfigSkip SkipClusterConfig = iota // Skip cluster config loading at all.

ConfigLoadWIthErrors // Skip errors at cluster config loading.

ConfigLoadAlways // Load cluster config, trigger an errors if not.

LoadMode

LoadConfigSkip LoadConfigSkipErrors LoadConfigAlways

oleg-jukovec · 2025-01-13T10:23:47Z

cli/cmd/clean.go

@@ -104,7 +104,7 @@ func internalCleanModule(cmdCtx *cmdcontext.CmdCtx, args []string) error {
 	}

 	var runningCtx running.RunningCtx
-	if err := running.FillCtx(cliOpts, cmdCtx, &runningCtx, args, false); err != nil {
+	if err := running.FillCtx(cliOpts, cmdCtx, &runningCtx, args, running.SkipConfig); err != nil {


load/skip scripts?

oleg-jukovec · 2025-01-13T10:26:12Z

cli/cmd/connect.go

@@ -140,7 +140,8 @@ func resolveConnectOpts(cmdCtx *cmdcontext.CmdCtx, cliOpts *config.CliOpts,
 	newArgs = args[1:]
 	// FillCtx returns error if no instances found.
 	var runningCtx running.RunningCtx
-	if fillErr := running.FillCtx(cliOpts, cmdCtx, &runningCtx, args, false); fillErr == nil {
+	if fillErr := running.FillCtx(
+		cliOpts, cmdCtx, &runningCtx, args, running.SkipConfigErrors); fillErr == nil {


oleg-jukovec · 2025-01-13T10:26:28Z

cli/cmd/log.go

@@ -133,7 +133,7 @@ func internalLogModule(cmdCtx *cmdcontext.CmdCtx, args []string) error {

 	var err error
 	var runningCtx running.RunningCtx
-	if err = running.FillCtx(cliOpts, cmdCtx, &runningCtx, args, false); err != nil {
+	if err = running.FillCtx(cliOpts, cmdCtx, &runningCtx, args, running.SkipConfig); err != nil {


Load - instance scripts?

oleg-jukovec · 2025-01-13T10:27:59Z

cli/cmd/play.go

@@ -99,7 +99,8 @@ func internalPlayModule(cmdCtx *cmdcontext.CmdCtx, args []string) error {
 	// FillCtx returns error if no instances found.
 	var runningCtx running.RunningCtx

-	if err := running.FillCtx(cliOpts, cmdCtx, &runningCtx, []string{args[0]}, false); err == nil {
+	if err := running.FillCtx(
+		cliOpts, cmdCtx, &runningCtx, []string{args[0]}, running.SkipConfig); err == nil {


oleg-jukovec · 2025-01-13T10:28:40Z

cli/cmd/replicaset.go

@@ -518,7 +519,8 @@ func replicasetFillCtx(cmdCtx *cmdcontext.CmdCtx, ctx *replicasetCtx, args []str
 				}
 				// Re-fill context for an application.
 				ctx.InstName = instName
-				err := running.FillCtx(cliOpts, cmdCtx, &ctx.RunningCtx, []string{appName}, false)
+				err := running.FillCtx(


Load, expected replicaset status.

oleg-jukovec · 2025-01-13T10:29:57Z

cli/instances/list.go

@@ -31,7 +31,7 @@ func ListInstances(cmdCtx *cmdcontext.CmdCtx, cliOpts *config.CliOpts) error {
 	fmt.Printf("instances enabled directory: %s\n", cliOpts.Env.InstancesEnabled)

 	applications, err := running.CollectInstancesForApps(appList, cliOpts, cmdCtx.Cli.ConfigDir,
-		cmdCtx.Integrity, false)
+		cmdCtx.Integrity, running.LoadConfig)


Skip/instance script required?

oleg-jukovec · 2025-01-13T10:30:30Z

cli/replicaset/rebootstrap.go

@@ -60,7 +60,7 @@ func cleanDataFiles(instCtx running.InstanceCtx) error {
 // and starting it again.
 func Rebootstrap(cmdCtx cmdcontext.CmdCtx, cliOpts config.CliOpts, rbCtx RebootstrapCtx) error {
 	apps, err := running.CollectInstancesForApps([]string{rbCtx.AppName}, &cliOpts,
-		cmdCtx.Cli.ConfigDir, cmdCtx.Integrity, true)
+		cmdCtx.Cli.ConfigDir, cmdCtx.Integrity, running.SkipConfig)


oleg-jukovec · 2025-01-13T10:31:45Z

cli/running/running_test.go

@@ -35,7 +35,7 @@ func Test_CollectInstances(t *testing.T) {
 	instances, err := CollectInstances("script", instancesEnabledPath,
 		integrity.IntegrityCtx{
 			Repository: &mockRepository{},
-		}, true)
+		}, LoadConfig)


Tests with a new functionality are missed.

change configuration loading logic

patapenka-alexey changed the title ~~pack: skip configs from etcd and tcs~~ pack: skip unavailable configs from etcd and tcs Dec 16, 2024

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from ddf6386 to c75c205 Compare December 16, 2024 14:07

patapenka-alexey requested review from elhimov and oleg-jukovec December 16, 2024 14:27

tecl1s reviewed Dec 17, 2024

View reviewed changes

test/integration/pack/test_bundles/vshard_app/test_app/router.lua Show resolved Hide resolved

oleg-jukovec reviewed Dec 17, 2024

View reviewed changes

cli/running/running.go Outdated Show resolved Hide resolved

oleg-jukovec reviewed Dec 17, 2024

View reviewed changes

elhimov reviewed Dec 18, 2024

View reviewed changes

patapenka-alexey mentioned this pull request Dec 23, 2024

tt create vshard_cluster command generate incorrect router role #1071

Open

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from c75c205 to 7b9cd77 Compare December 23, 2024 07:55

dmyger reviewed Dec 23, 2024

View reviewed changes

test/integration/pack/test_pack.py Outdated Show resolved Hide resolved

test/integration/pack/test_pack.py Outdated Show resolved Hide resolved

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch 3 times, most recently from 30ff7e3 to b64c71a Compare December 23, 2024 13:15

oleg-jukovec added the full-ci Enables full ci tests label Dec 23, 2024

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from b64c71a to bfb7321 Compare December 26, 2024 08:18

patapenka-alexey requested review from elhimov, dmyger and oleg-jukovec December 26, 2024 13:05

dmyger approved these changes Dec 26, 2024

View reviewed changes

oleg-jukovec requested changes Dec 27, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

cli/cmd/replicaset.go Outdated Show resolved Hide resolved

cli/cmd/replicaset.go Outdated Show resolved Hide resolved

cli/running/running.go Outdated Show resolved Hide resolved

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from bfb7321 to 460b4fa Compare December 27, 2024 12:12

patapenka-alexey requested a review from oleg-jukovec December 27, 2024 13:23

oleg-jukovec requested changes Dec 28, 2024

View reviewed changes

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch 2 times, most recently from bcefd0a to f729a21 Compare January 3, 2025 12:12

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from f729a21 to 48559b6 Compare January 8, 2025 11:06

patapenka-alexey removed the full-ci Enables full ci tests label Jan 9, 2025

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from 48559b6 to bf7ba6c Compare January 9, 2025 11:08

patapenka-alexey marked this pull request as draft January 9, 2025 11:09

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from bf7ba6c to f7e5f77 Compare January 9, 2025 11:58

pack: skip unavailable configs from etcd and tcs

e31707e

@TarantoolBot document Title: pack: skip configs from etcd and tcs This patch fixes the `tt pack` error if `etcd` or `tcs` are unavailable during package creation. Part of #1038

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch 4 times, most recently from 503ded7 to d4f2418 Compare January 10, 2025 13:00

patapenka-alexey added the full-ci Enables full ci tests label Jan 10, 2025

cli: skip cluster config for the few commands

22ada6a

From now on only the `check` and `start` commands do not skip errors during cluster config generation from local, etcs/tcs and env sources. Closes #1038

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from d4f2418 to 22ada6a Compare January 10, 2025 13:19

patapenka-alexey requested a review from oleg-jukovec January 10, 2025 15:41

elhimov reviewed Jan 13, 2025

View reviewed changes

patapenka-alexey marked this pull request as ready for review January 13, 2025 09:38

oleg-jukovec reviewed Jan 13, 2025

View reviewed changes

rename and fix linter warnings

94c464a

change configuration loading logic

patapenka-alexey force-pushed the patapenka-alexey/gh-1038-tt-pack-error-etcd branch from 17c32a0 to 94c464a Compare January 14, 2025 06:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pack: skip unavailable configs from etcd and tcs #1066

pack: skip unavailable configs from etcd and tcs #1066

patapenka-alexey commented Dec 16, 2024 •

edited

Loading

oleg-jukovec left a comment

elhimov Dec 18, 2024 •

edited

Loading

elhimov Dec 19, 2024

oleg-jukovec left a comment

patapenka-alexey commented Jan 11, 2025

elhimov Jan 13, 2025

elhimov Jan 13, 2025

elhimov commented Jan 13, 2025

oleg-jukovec Jan 13, 2025

oleg-jukovec Jan 13, 2025 •

edited

Loading

oleg-jukovec Jan 13, 2025

oleg-jukovec Jan 13, 2025 •

edited

Loading

oleg-jukovec Jan 13, 2025

oleg-jukovec Jan 13, 2025 •

edited

Loading

oleg-jukovec Jan 13, 2025

oleg-jukovec Jan 13, 2025

oleg-jukovec Jan 13, 2025 •

edited

Loading

oleg-jukovec Jan 13, 2025

oleg-jukovec Jan 13, 2025

pack: skip unavailable configs from etcd and tcs #1066

Are you sure you want to change the base?

pack: skip unavailable configs from etcd and tcs #1066

Conversation

patapenka-alexey commented Dec 16, 2024 • edited Loading

oleg-jukovec left a comment

Choose a reason for hiding this comment

elhimov Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-jukovec left a comment

Choose a reason for hiding this comment

patapenka-alexey commented Jan 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elhimov commented Jan 13, 2025

Choose a reason for hiding this comment

oleg-jukovec Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-jukovec Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-jukovec Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleg-jukovec Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patapenka-alexey commented Dec 16, 2024 •

edited

Loading

elhimov Dec 18, 2024 •

edited

Loading

oleg-jukovec Jan 13, 2025 •

edited

Loading

oleg-jukovec Jan 13, 2025 •

edited

Loading

oleg-jukovec Jan 13, 2025 •

edited

Loading

oleg-jukovec Jan 13, 2025 •

edited

Loading