fix python packages logic to account for the fact that ls-files doesn…

…'t list directories (#25045)

this was an oversight on my part because I thought `git ls-files .` would include directories. I thought I had verified the resulting packages but I guess was confused because we still traverse explicit packages that are listed. This fixes like in https://app.graphite.dev/github/pr/dagster-io/internal/12014/alerts-fix-build-by-adding-logging-statement-back where we weren't running dagit tests after backend changed

# Test Plan

```➜ CLOUD_PROD_ELEMENTL_AGENT_TOKEN=x BUILDKITE_BRANCH=test-testing BUILDKITE_MESSAGE=x BUILDKITE_DOCKER_QUEUE=blah BUILDKITE_MEDIUM_QUEUE=blah BUILDKITE_WINDOWS_QUEUE=blah python .buildkite/pipeline.py | tee /tmp/steps.yaml
From github.com:dagster-io/dagster
 * branch                  master     -> FETCH_HEAD
2024-10-03 16:06:38 INFO     Changed files between origin/master (b52d314) and HEAD (e51c353):
2024-10-03 16:06:38 INFO       - python_modules/dagster-webserver/dagster_webserver/graphql.py
2024-10-03 16:06:38 INFO     Finding Python packages:
2024-10-03 16:06:38 INFO       - REPO_NAME_PLACEHOLDER
2024-10-03 16:06:38 INFO       - assets_dbt_python
2024-10-03 16:06:38 INFO       - assets_dynamic_partitions
2024-10-03 16:06:38 INFO       - assets_modern_data_stack
2024-10-03 16:06:38 INFO       - assets_pandas_pyspark
2024-10-03 16:06:38 INFO       - assets_pandas_type_metadata
2024-10-03 16:06:38 INFO       - assets_smoke_test
2024-10-03 16:06:38 INFO       - assets_yaml_dsl
2024-10-03 16:06:38 INFO       - automation
2024-10-03 16:06:38 INFO       - builtin-blueprints
2024-10-03 16:06:38 INFO       - custom-blueprints
2024-10-03 16:06:38 INFO       - dagit
2024-10-03 16:06:38 INFO       - dagster
2024-10-03 16:06:38 INFO       - dagster
2024-10-03 16:06:38 INFO       - dagster-airbyte
2024-10-03 16:06:38 INFO       - dagster-airflow
2024-10-03 16:06:38 INFO       - dagster-airlift
2024-10-03 16:06:38 INFO       - dagster-aws
2024-10-03 16:06:38 INFO       - dagster-azure
...
2024-10-03 16:06:41 INFO     Building dagster-cloud-dagit because of changes to {PythonPackage(dagster-webserver)}
2024-10-03 16:06:41 INFO     Building dagster-cloud-dagit because of changes to {PythonPackage(dagster-webserver)}
2024-10-03 16:06:41 INFO     Building dagster-cloud-dagit because of changes to {PythonPackage(dagster-webserver)}
2024-10-03 16:06:41 INFO     Building dagit because we're building dagster-cloud-dagit
```

.buildkite/dagster-buildkite/dagster_buildkite/python_packages.py

@@ -30,7 +30,7 @@ def __init__(self, setup_path: Path):
             # distribution if our setup.py doesn't implement setup() correctly
             reload(distutils_core)
 
-            distribution = distutils_core.run_setup(str(setup_path), stop_after="init")
+            distribution = distutils_core.run_setup(str(setup_path / "setup.py"), stop_after="init")
 
             self._install_requires = distribution.install_requires  # type: ignore[attr-defined]
             self._extras_require = distribution.extras_require  # type: ignore[attr-defined]
@@ -52,7 +52,7 @@ def __init__(self, setup_path: Path):
                 self._extras_require = {}
             else:
                 self.name = project["name"]
-                self._install_requires = project["dependnecies"]
+                self._install_requires = project["dependencies"]
                 self._extras_require = project.get("optional-dependencies", {})
 
     @property
@@ -146,13 +146,18 @@ def load_from_git(cls, git_info: GitInfo) -> None:
             ["git", "ls-files", "."],
             cwd=str(git_info.directory),
         ).decode("utf-8")
+        processed = set()
         packages = []
         for file in output.split("\n"):
-            path = git_info.directory / Path(file)
-            if path.is_dir() and (
-                (path / "setup.py").exists() or (path / "pyproject.toml").exists()
-            ):
-                packages.append(PythonPackage(path))
+            if not file:
+                continue
+            path_dir = (git_info.directory / Path(file)).parents[0]
+            if str(path_dir) in processed:
+                continue
+            processed |= {str(path_dir)}
+            assert path_dir.is_dir()
+            if (path_dir / "setup.py").exists() or (path_dir / "pyproject.toml").exists():
+                packages.append(PythonPackage(path_dir))
 
         for package in sorted(packages):
             logging.info("  - " + package.name)