Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: artificial package deduplication #309

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat: artificial package deduplication #309

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f4411da
add the option
jkowalleck Nov 11, 2022
2983707
prep demo
jkowalleck Nov 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions HISTORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,13 @@ All notable changes to this project will be documented in this file.

## unreleased

* Added
* new CLI option `--deduplicate-components` ... write something ([#306] via [#309])
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO

* Misc
* Added test for flattened results (via [#312])

[#306]: https://github.com/CycloneDX/cyclonedx-node-npm/issues/306
[#309]: https://github.com/CycloneDX/cyclonedx-node-npm/pull/309
[#312]: https://github.com/CycloneDX/cyclonedx-node-npm/pull/312

## 1.6.0 - 2022-11-12
Expand Down
11 changes: 8 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,15 +77,20 @@ Options:
This might be used, if "npm install" was run with "--force" or "--legacy-peer-deps".
(default: false)
--package-lock-only Whether to only use the lock file, ignoring "node_modules".
This means the output will be based only on the few details in and the tree described by the "npm-shrinkwrap.json" or "package-lock.json", rather than the contents of "node_modules" directory.
Enabling this feature means the output will be based only on the few details in and the tree described by the "npm-shrinkwrap.json" or "package-lock.json", rather than the contents of "node_modules" directory.
(default: false)
--omit <type...> Dependency types to omit from the installation tree.
(can be set multiple times)
(choices: "dev", "optional", "peer", default: "dev" if the NODE_ENV environment variable is set to "production", otherwise empty)
--flatten-components Whether to flatten the components.
This means the actual nesting of node packages is not represented in the SBOM result.
Enabling this feature means the actual nesting of node packages is not represented in the SBOM result, which causes a massive information loss.
(default: false)
--deduplicate-components Whether to artificially de-duplicate the node packages.
Enabling this feature means the actual multiple/parallel installed instances of a packages are displayed as one component, which causes a massive information loss.
Enabling this feature implies option "--flatten-components=true"
(default: false)
--short-PURLs Omit all qualifiers from PackageURLs.
This causes information loss in trade of shorter PURLs, which might improve digesting these strings.
Enabling this feature causes information loss in trade of shorter PURLs, which might improve digesting these strings.
(default: false)
--spec-version <version> Which version of CycloneDX spec to use.
(choices: "1.2", "1.3", "1.4", default: "1.4")
Expand Down
10 changes: 10 additions & 0 deletions demo/gen-boms.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,16 @@ do
--output-format "$format" \
--output-file "$result_dir/flat/bom.$spec.$format" \
"$package"

echo ">>> $result_dir $spec $format deduplicated"
mkdir -p "$result_dir/deduplicated"
node -- "$BIN_CDX_N" \
--deduplicate-components \
--spec-version "$spec" \
--output-reproducible \
--output-format "$format" \
--output-file "$result_dir/deduplicated/bom.$spec.$format" \
"$package"
done
done
done
3 changes: 3 additions & 0 deletions docs/component_deduplication.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Component De-duplication

TODO
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO

6 changes: 6 additions & 0 deletions src/builders.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ interface BomBuilderOptions {
omitDependencyTypes?: Iterable<OmittableDependencyTypes>
reproducible?: BomBuilder['reproducible']
flattenComponents?: BomBuilder['flattenComponents']
deduplicateComponents?: BomBuilder['deduplicateComponents']
shortPURLs?: BomBuilder['shortPURLs']
}

Expand All @@ -54,6 +55,7 @@ export class BomBuilder {
omitDependencyTypes: Set<OmittableDependencyTypes>
reproducible: boolean
flattenComponents: boolean
deduplicateComponents: boolean
shortPURLs: boolean

console: Console
Expand All @@ -77,6 +79,7 @@ export class BomBuilder {
this.omitDependencyTypes = new Set(options.omitDependencyTypes ?? [])
this.reproducible = options.reproducible ?? false
this.flattenComponents = options.flattenComponents ?? false
this.deduplicateComponents = options.deduplicateComponents ?? false
this.shortPURLs = options.shortPURLs ?? false

this.console = console_
Expand Down Expand Up @@ -251,6 +254,9 @@ export class BomBuilder {
bom.components.add(component)
}
}
if (this.deduplicateComponents) {
// TODO
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO

}
}

// endregion components
Expand Down
21 changes: 17 additions & 4 deletions src/cli.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ interface CommandOptions {
omit: Omittable[]
specVersion: Spec.Version
flattenComponents: boolean
deduplicateComponents: boolean
shortPURLs: boolean
outputReproducible: boolean
outputFormat: OutputFormat
Expand All @@ -67,7 +68,7 @@ function makeCommand (process: NodeJS.Process): Command {
new Option(
'--package-lock-only',
'Whether to only use the lock file, ignoring "node_modules".\n' +
'This means the output will be based only on the few details in and the tree described by the "npm-shrinkwrap.json" or "package-lock.json", rather than the contents of "node_modules" directory.'
'Enabling this feature means the output will be based only on the few details in and the tree described by the "npm-shrinkwrap.json" or "package-lock.json", rather than the contents of "node_modules" directory.'
).default(false)
).addOption(
new Option(
Expand All @@ -86,13 +87,20 @@ function makeCommand (process: NodeJS.Process): Command {
new Option(
'--flatten-components',
'Whether to flatten the components.\n' +
'This means the actual nesting of node packages is not represented in the SBOM result.'
'Enabling this feature means the actual nesting of node packages is not represented in the SBOM result, which causes a massive information loss.'
).default(false)
).addOption(
new Option(
'--deduplicate-components',
'Whether to artificially de-duplicate the node packages.\n' +
'Enabling this feature means the actual multiple/parallel installed instances of a packages are displayed as one component, which causes a massive information loss.\n' +
'Enabling this feature implies option "--flatten-components=true"'
).default(false)
).addOption(
new Option(
'--short-PURLs',
'Omit all qualifiers from PackageURLs.\n' +
'This causes information loss in trade of shorter PURLs, which might improve digesting these strings.'
'Enabling this feature causes information loss in trade of shorter PURLs, which might improve digesting these strings.'
).default(false)
).addOption(
new Option(
Expand All @@ -107,7 +115,7 @@ function makeCommand (process: NodeJS.Process): Command {
new Option(
'--output-reproducible',
'Whether to go the extra mile and make the output reproducible.\n' +
'This requires more resources, and might result in loss of time- and random-based-values.'
'Enabling this feature requires more resources, and might result in loss of time- and random-based-values.'
).env(
'BOM_REPRODUCIBLE'
)
Expand Down Expand Up @@ -178,6 +186,10 @@ export function run (process: NodeJS.Process): void {
program.parse(process.argv)

const options: CommandOptions = program.opts()
if (options.deduplicateComponents && !options.flattenComponents) {
myConsole.info('INFO | Found option --deduplicate-components=true - therefore, forced option --flatten-components=true')
options.flattenComponents = true
}
myConsole.debug('DEBUG | options: %j', options)

const packageFile = resolve(process.cwd(), program.args[0] ?? 'package.json')
Expand Down Expand Up @@ -221,6 +233,7 @@ export function run (process: NodeJS.Process): void {
omitDependencyTypes: options.omit,
reproducible: options.outputReproducible,
flattenComponents: options.flattenComponents,
deduplicateComponents: options.deduplicateComponents,
shortPURLs: options.shortPURLs
},
myConsole
Expand Down