Merge branch '20240703-semgrep-configurable' of https://github.com/pu…

…rajit/pants into 20240703-semgrep-configurable

docs/docs/contributions/development/setting-up-pants.mdx

@@ -16,9 +16,9 @@ Pants requires a more modern OpenSSL version than the one that comes with macOS.
 
 ```bash
 $ brew install openssl
-$ echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bashrc
-$ echo 'export LDFLAGS="-L/usr/local/opt/openssl/lib"' >> ~/.bashrc
-$ echo 'export CPPFLAGS="-I/usr/local/opt/openssl/include"' >> ~/.bashrc
+$ echo 'export PATH="$(brew --prefix)/opt/openssl/bin:$PATH"' >> ~/.bashrc
+$ echo 'export LDFLAGS="-L$(brew --prefix)/opt/openssl/lib"' >> ~/.bashrc
+$ echo 'export CPPFLAGS="-I$(brew --prefix)/opt/openssl/include"' >> ~/.bashrc
 ```
 
 (If you don't have `brew` installed, see [https://brew.sh](https://brew.sh))

docs/docs/javascript/_category_.json

@@ -0,0 +1,4 @@
+{
+  "label": "Javascript",
+  "position": 13
+}

docs/docs/javascript/overview/_category_.json

@@ -0,0 +1,4 @@
+{
+  "label": "Javascript overview",
+  "position": 1
+}

docs/docs/javascript/overview/enabling-javascript-support.mdx

@@ -0,0 +1,71 @@
+---
+    title: Enabling Javascript support
+    sidebar_position: 0
+---
+
+How to enable Pants's bundled Javascript backend package.
+
+---
+
+:::note Example Javascript repository
+See [here](https://github.com/pantsbuild/example-javascript) for examples of Pants's Javascript functionality.
+
+:::
+
+### Configuring the repository
+
+Enable the experimental Javascript [backend](../../using-pants/key-concepts/backends.mdx) like this:
+
+```toml title="pants.toml"
+[GLOBAL]
+...
+backend_packages = [
+    "pants.backend.experimental.javascript"
+]
+```
+
+Pants uses [`package_json`](../../../reference/targets/package_json.mdx) targets to model a NodeJS package.
+Further, [`javascript_source`](../../../reference/targets/javascript_source.mdx) and
+[`javascript_tests`](../../../reference/targets/javascript_test.mdx) targets are used to know which Javascript files to
+run on and to set any metadata.
+
+You can generate these targets by running [`pants tailor ::`](../../getting-started/initial-configuration.mdx#5-generate-build-files).
+
+```
+❯ pants tailor ::
+Created project/BUILD:
+  - Add javascript_sources target project
+  - Add javascript_tests target tests
+```
+
+
+### Setting up node
+Pants will by default download a distribution of `node` according to the
+[`nodejs` subsystem](../../../reference/subsystems/nodejs) configuration. If you wish to instead use a locally installed
+ version of, for example, 18.0.0 using `nvm` and its `.nvmrc` file, the following will get you there:
+
+```toml tab={"label": "pants.toml"}
+[nodejs]
+known_versions = [] # Assign this to the empty list to ensure Pants never downloads.
+version = "v18.0.0"
+search_path = ["<NVM_LOCAL>"]
+
+```
+
+```txt tab={"label": ".nvmrc"}
+v18.0.0
+
+```
+
+### Setting up a package manager
+To set a package manager project wide, do the following:
+
+```toml title="pants.toml"
+[nodejs]
+package_manager = "pnpm" # or yarn, or npm.
+
+```
+
+you can instead opt to use the [`package.json#packageManager`](./package.mdx#package-manager) field for this setting.
+Regardless of setting, pants uses the [`corepack`](https://github.com/nodejs/corepack) version distributed with the Node
+version you have chosen to install and manage package managers.

docs/docs/javascript/overview/index.mdx

@@ -0,0 +1,29 @@
+---
+    title: Javascript overview
+    sidebar_position: 0
+---
+
+Pants's support for Javascript.
+
+:::caution Javascript support is beta stage
+We are done implementing most functionality for Pants's Javascript support
+([tracked here](https://github.com/pantsbuild/pants/labels/backend%3A%20javascript)).
+However, there may be use cases that we aren't yet handling.
+:::
+
+The Javascript and NodeJS ecosystem has a seemingly endless amount of frameworks and tooling,
+all orchestrated via package managers.
+
+Pants employs a wrapping approach with a thin caching layer applied on top of current supported package managers:
+`npm`, `pnpm` and `yarn`.
+
+Features for Javascript:
+
+- Caching the results of your test scripts and build scripts,
+ making the latter available in your Pants workflows as [`resources`](../../reference/targets/resource) and
+ package artifacts.
+- A consistent interface for all languages/tools in your repository,
+ such as being able to run `pants fmt lint check test package`.
+- [Remote execution and remote caching](../../using-pants/remote-caching-and-execution/index.mdx).
+- [Advanced project introspection](../../using-pants/project-introspection.mdx),
+ such as finding all code that transitively depends on a certain package.

docs/docs/javascript/overview/lockfiles.mdx

@@ -0,0 +1,65 @@
+---
+    title: Lockfiles
+    sidebar_position: 2
+---
+
+Package manager lockfile integration
+
+---
+
+Third-party dependencies are specified in the package.json fields.
+All package managers vendors a lockfile format specific for the package manager you are using. Pants knows of this
+lockfile and models it as a "resolve".
+
+Resolves is the only way to deal with dependencies within pants, and no extra configuration is required.
+
+You can however name your resolves/lockfiles. The resolve name is otherwise auto-generated.
+
+```toml title="pants.toml"
+[GLOBAL]
+backend_packages.add = [
+    "pants.backend.experimental.javascript"
+]
+
+[nodejs.resolves]
+package-lock.json = "my-lock"
+
+```
+
+You generate the lockfile as follows:
+
+```shell title="Bash"
+$ pants generate-lockfiles
+19:00:39.26 [INFO] Completed: Generate lockfile for my-lock
+19:00:39.29 [INFO] Wrote lockfile for the resolve `my-lock` to package-lock.json
+```
+
+
+## Using lockfiles for tools
+
+To ensure that the same version of tooling you have specified in `package.json` is used with a NodeJS powered tool,
+specify the resolve name for the tool.
+E.g., for the Prettier linter:
+
+```toml tab={"label": "pants.toml"}
+[GLOBAL]
+backend_packages.add = [
+    "pants.backend.experimental.javascript",
+    "pants.backend.experimental.javascript.lint.prettier",
+]
+
+[prettier]
+install_from_resolve = "nodejs-default"
+
+```
+```json tab={"label": "package.json"}
+{
+    "name": "@my-company/pkg",
+    "devDependencies": {
+        "prettier": "^2.6.2"
+    }
+}
+```
+```python tab={"label": "BUILD"}
+    package_json(name="pkg")
+```

docs/docs/javascript/overview/package.mdx

@@ -0,0 +1,67 @@
+---
+    title: package.json
+    sidebar_position: 3
+---
+
+package.json parsing and scripts integration
+
+---
+
+As mentioned in the [overview introduction](./index.mdx), Pants's approach to enable support for Javascript is to
+be a thin caching layer on top of your current tooling.
+
+Refer to the [example repository](https://github.com/pantsbuild/example-javascript) for example usage.
+
+### Package manager
+
+Pants uses `corepack` to manage package manager versions and installation. Like `corepack`, Pants respects
+the experimental "packageManager" feature in `package.json` files.
+
+```json title="package.json"
+{
+    "name": "@my-company/pkg",
+    "packageManager": "[email protected]"
+}
+```
+
+this setting will ensure that all scripts invoked for this package.json, and any
+[workspaces managed by this package.json](./workspaces.mdx) will use this particular version of `yarn`.
+It can be more convenient to define a project level
+[package manager](./enable-javascript-support.mdx#setting-up-a-package-manager).
+
+:::tip Choosing between `pants.toml` or `package.json` for package manager version configuration
+In general, if your team runs all tooling via Pants, using `pants.toml` reduces boilerplate in cases where you maintain
+multiple packages. If your team mixes usage of Pants and "bare" package manager invocations, package.json#packageManager
+is the safer option.
+:::
+
+### Testing
+
+By default Pants assumes a `package_json` target mapping a `package.json` includes a test script, e.g.
+
+```json title="package.json"
+{
+    "name": "@my-company/pkg",
+    "scripts": {
+        "test": "jest"
+    },
+    "devDependencies": {
+        "jest": "^29.5.0"
+    }
+}
+```
+
+and will use this script to execute your tests when running `pants test ::`.
+See [Goal arguments](../../using-pants/key-concepts/goals.mdx#goal-arguments) for the normal techniques for telling Pants what to
+run on.
+
+To enable configurability, the build symbol [`node_test_script`](../../../reference/build-file-symbols/node_test_script)
+contains options for changing the entry point from "test", and to enable coverage reporting.
+
+### Packaging
+
+Similarly, build scripts can be introduced to Pants via the
+[`node_build_script`](../../../reference/build-file-symbols/node_build_script) build symbol. This is intended to be used
+as a way to introduce artifacts generated via bundlers and/or compilers installed and ran via your package manager.
+The result can the be consumed by other targets as either `resource`-targets that can be depended on,
+or as a package for the [docker backend](../../docker/index.mdx).

docs/docs/javascript/overview/workspaces.mdx

@@ -0,0 +1,22 @@
+---
+    title: Monorepo workspaces
+    sidebar_position: 1
+---
+Package manager workspace management
+
+---
+
+Modern versions of package managers introduce similar
+concepts of "workspaces", a list of Nodejs packages all contained within the same code repository.
+
+Pants support all three flavors of package managers and understands the configuration
+settings specific for each tool:
+
+- [pnpm](https://pnpm.io/workspaces) pnpm-workspaces.yaml
+- [yarn](https://yarnpkg.com/features/workspaces) and [npm](https://docs.npmjs.com/cli/v10/using-npm/workspaces)
+    "workspaces" setting in package.json
+
+:::tip Use workspaces!
+It is encouraged by the Pants team to utilize this project setup to only have to deal with and maintain
+one [resolve](./lockfiles.mdx). Pants aims to provide full integration with these monorepo settings.
+:::

docs/notes/2.23.x.md

@@ -150,7 +150,11 @@ Fixed pulling `helm_artifact`s from OCI repositories.
 Added `workspace_invalidation_sources` field to `adhoc_tool` and `shell_command` target types. This new field allows declaring that these targets depend on files without bringing those files into the execution sandbox, but that the target should still be re-executed if those files change. This is intended to work with the `workspace_environment` support where processes are executed in the workspace and not in a separate sandbox.
 
 #### Semgrep
-Semgrep now allows configuring the config directory, the semgrepignore config file, and will recursively discover all rules in the directory.
+Semgrep now allows configuring config file discovery via [the new `config_name` option](https://www.pantsbuild.org/2.23/reference/subsystems/semgrep#config_name). In addition, it will now recursively discover all rules within a config directory, not just the immediate children.
+
+#### Workunit logger
+
+The `pants.backend.experimental.tools.workunit_logger` backend will now create directory specified by [the `logdir` option](https://www.pantsbuild.org/2.23/reference/subsystems/workunit-logger#logdir) if it doesn't already exist.
 
 ### Plugin API changes
 

src/python/pants/backend/tools/workunit_logger/rules.py

@@ -15,6 +15,7 @@
 from pants.engine.unions import UnionRule
 from pants.option.option_types import BoolOption, StrOption
 from pants.option.subsystem import Subsystem
+from pants.util.dirutil import safe_open
 
 logger = logging.getLogger(__name__)
 
@@ -66,7 +67,7 @@ def __call__(
             self._completed_workunits[wu["span_id"]] = wu
         if finished:
             filepath = f"{self.wulogger.logdir}/{context.run_tracker.run_id}.json"
-            with open(filepath, "w") as f:
+            with safe_open(filepath, "w") as f:
                 json.dump(just_dump_map(self._completed_workunits), f)
                 logger.info(f"Wrote log to {filepath}")