Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DB-15227] Populate the datatype in the UNSUPPORTED_DATATYPE_LIVE_MIGRATION / FF_FB fields in Assessment Issues #2298

Merged
merged 2 commits into from
Feb 6, 2025
+15 −16
Merged

[DB-15227] Populate the datatype in the UNSUPPORTED_DATATYPE_LIVE_MIGRATION / FF_FB fields in Assessment Issues #2298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d9ecb13
[DB-15227] Populate the datatype in the UNSUPPORTED_DATATYPE_LIVE_MIG…
sanyamsinghal Feb 6, 2025
7440b9b
minor miss
sanyamsinghal Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 13 additions & 14 deletions migtests/scripts/functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -874,7 +874,7 @@ normalize_json() {
local input_file="$1"
local output_file="$2"
local temp_file="/tmp/temp_file.json"
# local temp_file2="/tmp/temp_file2.json"
local temp_file2="/tmp/temp_file2.json"

# Normalize JSON with jq; use --sort-keys to avoid the need to keep the same sequence of keys in expected vs actual json
jq --sort-keys 'walk(
Expand Down Expand Up @@ -909,23 +909,22 @@ normalize_json() {
end
)' "$input_file" > "$temp_file"

# Sample code: how to ignore nested fields with a specific parent in a json
# TODO: For fields like SqlStatement or others we can more explicit regarding changing them
# jq --sort-keys 'walk(
# if type == "object" and (.UnsupportedFeatures?|type) == "array" then
# .UnsupportedFeatures |= map(
# .FeatureName? = "IGNORED"
# )
# else
# .
# end
# )' "$temp_file" > "$temp_file2"
# Second pass: for AssessmentIssue objects, if Category is "migration_caveats" and the Type is one
# of the two specified values, then set ObjectName to "IGNORED".
jq 'walk(
if type == "object" and
.Category == "migration_caveats" and
(.Type == "UNSUPPORTED_DATATYPE_LIVE_MIGRATION" or .Type == "UNSUPPORTED_DATATYPE_LIVE_MIGRATION_WITH_FF_FB")
then .ObjectName = "IGNORED"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it required to ignore the object name for these Type?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think just a temporary ignore until we fix it properly :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, plan is to have individual issue for each datatype.
Updating the test for this temporary fix will be a big change, a lot of test has the corresponding issues including osm, omnibus

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh okay, got it!

else .
end
)' "$temp_file" > "$temp_file2"

# Remove unwanted lines
sed -i '/Review and manually import.*uncategorized.sql/d' "$temp_file"
sed -i '/Review and manually import.*uncategorized.sql/d' "$temp_file2"

# Move cleaned file to output
mv "$temp_file" "$output_file"
mv "$temp_file2" "$output_file"
}


Expand Down
4 changes: 2 additions & 2 deletions yb-voyager/cmd/assessMigrationCommand.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1503,7 +1503,7 @@ func addMigrationCaveatsToAssessmentReport(unsupportedDataTypesForLiveMigration
Impact: constants.IMPACT_LEVEL_1, // Caveat - we don't know the migration is offline/online;
Description: UNSUPPORTED_DATATYPES_FOR_LIVE_MIGRATION_DESCRIPTION,
ObjectType: constants.COLUMN,
ObjectName: fmt.Sprintf("%s.%s.%s", colInfo.SchemaName, colInfo.TableName, colInfo.ColumnName),
ObjectName: fmt.Sprintf("%s.%s.%s (%s)", colInfo.SchemaName, colInfo.TableName, colInfo.ColumnName, colInfo.DataType), // TODO (fix): adding datatype here is temporary fix
DocsLink: UNSUPPORTED_DATATYPE_LIVE_MIGRATION_DOC_LINK,
})
}
Expand All @@ -1524,7 +1524,7 @@ func addMigrationCaveatsToAssessmentReport(unsupportedDataTypesForLiveMigration
Impact: constants.IMPACT_LEVEL_1,
Description: UNSUPPORTED_DATATYPES_FOR_LIVE_MIGRATION_WITH_FF_FB_DESCRIPTION,
ObjectType: constants.COLUMN,
ObjectName: fmt.Sprintf("%s.%s.%s", colInfo.SchemaName, colInfo.TableName, colInfo.ColumnName),
ObjectName: fmt.Sprintf("%s.%s.%s (%s)", colInfo.SchemaName, colInfo.TableName, colInfo.ColumnName, colInfo.DataType), // TODO (fix): adding datatype here is temporary fix
DocsLink: UNSUPPORTED_DATATYPE_LIVE_MIGRATION_DOC_LINK,
})
}
Expand Down