Skip to content
This repository has been archived by the owner on Jul 18, 2024. It is now read-only.

Github Graphql improvements #281

Merged
merged 25 commits into from
Nov 20, 2023
Merged

Github Graphql improvements #281

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4b3bc46
ingest repository topics
RonaldEAM Nov 1, 2023
a4bfab7
update/simplify tests
RonaldEAM Nov 1, 2023
e2466fd
add batch queries (collaborators, branchProtectionRules)
RonaldEAM Nov 3, 2023
d9d36a2
batch queries for vulnAlerts
RonaldEAM Nov 3, 2023
7c1324a
batch team repos queries
RonaldEAM Nov 6, 2023
8e3de1a
batch queries for issues
RonaldEAM Nov 6, 2023
35a1ff0
batch commits/labels/reviews queries
RonaldEAM Nov 6, 2023
08e8f7e
Merge branch 'main' into optimization-dh
RonaldEAM Nov 6, 2023
074fe83
batch team members query
RonaldEAM Nov 7, 2023
d747aec
handle github timeout
RonaldEAM Nov 7, 2023
dc24167
add/fix tests
RonaldEAM Nov 8, 2023
2ff048d
change log.info back to log.debug
RonaldEAM Nov 8, 2023
88cc7ab
batch topics and tags queries
RonaldEAM Nov 8, 2023
82247d0
use max limit (100) on batched queries
RonaldEAM Nov 8, 2023
e42384e
send logger in withBatching
RonaldEAM Nov 14, 2023
5af9aa7
batch prs requests
RonaldEAM Nov 15, 2023
388f2bb
batch prs requests
RonaldEAM Nov 15, 2023
6315c1f
fix 1st iteration of dev test
RonaldEAM Nov 16, 2023
7276cb1
disable secret scanning alerts step (#282)
RonaldEAM Nov 15, 2023
b32d9a9
Bump version to: 3.10.0 [skip ci]
j1-internal-automation Nov 15, 2023
eb799f2
set step concurrency to 1 instead of using dependsOn
RonaldEAM Nov 16, 2023
97ef170
remove comments
RonaldEAM Nov 17, 2023
ff90cfb
Merge pull request #284 from JupiterOne/optimization-dh-prs
RonaldEAM Nov 17, 2023
325a523
remove unwanted alert about scope (#283)
RonaldEAM Nov 17, 2023
c763802
Bump version to: 3.10.1 [skip ci]
j1-internal-automation Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 51 additions & 53 deletions docs/jupiterone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,64 +155,62 @@ https://github.com/JupiterOne/sdk/blob/main/docs/integrations/development.md

The following entities are created:

| Resources | Entity `_type` | Entity `_class` |
| ----------------------------- | -------------------------------- | --------------- |
| Account | `github_account` | `Account` |
| GitHub Code Scanning Alerts | `github_code_scanning_finding` | `Finding` |
| GitHub Env Secret | `github_env_secret` | `Secret` |
| GitHub Secret Scanning Alert | `github_secret_scanning_finding` | `Finding` |
| GitHub Vulnerability Alert | `github_finding` | `Finding` |
| Github App | `github_app` | `Application` |
| Github Branch Protection Rule | `github_branch_protection_rule` | `Rule` |
| Github Environment | `github_environment` | `Configuration` |
| Github Issue | `github_issue` | `Issue` |
| Github Org Secret | `github_org_secret` | `Secret` |
| Github Pull Request | `github_pullrequest` | `PR` |
| Github Repo | `github_repo` | `CodeRepo` |
| Github Repo Secret | `github_repo_secret` | `Secret` |
| Github Team | `github_team` | `UserGroup` |
| Github User | `github_user` | `User` |
| Resources | Entity `_type` | Entity `_class` |
| ----------------------------- | ------------------------------- | --------------- |
| Account | `github_account` | `Account` |
| GitHub Code Scanning Alerts | `github_code_scanning_finding` | `Finding` |
| GitHub Env Secret | `github_env_secret` | `Secret` |
| GitHub Vulnerability Alert | `github_finding` | `Finding` |
| Github App | `github_app` | `Application` |
| Github Branch Protection Rule | `github_branch_protection_rule` | `Rule` |
| Github Environment | `github_environment` | `Configuration` |
| Github Issue | `github_issue` | `Issue` |
| Github Org Secret | `github_org_secret` | `Secret` |
| Github Pull Request | `github_pullrequest` | `PR` |
| Github Repo | `github_repo` | `CodeRepo` |
| Github Repo Secret | `github_repo_secret` | `Secret` |
| Github Team | `github_team` | `UserGroup` |
| Github User | `github_user` | `User` |

### Relationships

The following relationships are created:

| Source Entity `_type` | Relationship `_class` | Target Entity `_type` |
| --------------------- | --------------------- | -------------------------------- |
| `github_account` | **INSTALLED** | `github_app` |
| `github_account` | **HAS** | `github_org_secret` |
| `github_account` | **OWNS** | `github_repo` |
| `github_account` | **HAS** | `github_team` |
| `github_account` | **HAS** | `github_user` |
| `github_app` | **OVERRIDES** | `github_branch_protection_rule` |
| `github_env_secret` | **OVERRIDES** | `github_org_secret` |
| `github_env_secret` | **OVERRIDES** | `github_repo_secret` |
| `github_environment` | **HAS** | `github_env_secret` |
| `github_pullrequest` | **CONTAINS** | `github_pullrequest` |
| `github_repo` | **HAS** | `github_branch_protection_rule` |
| `github_repo` | **HAS** | `github_code_scanning_finding` |
| `github_repo` | **USES** | `github_env_secret` |
| `github_repo` | **HAS** | `github_environment` |
| `github_repo` | **HAS** | `github_finding` |
| `github_repo` | **HAS** | `github_issue` |
| `github_repo` | **USES** | `github_org_secret` |
| `github_repo` | **HAS** | `github_pullrequest` |
| `github_repo` | **HAS** | `github_repo_secret` |
| `github_repo` | **USES** | `github_repo_secret` |
| `github_repo` | **HAS** | `github_secret_scanning_finding` |
| `github_repo` | **ALLOWS** | `github_team` |
| `github_repo` | **ALLOWS** | `github_user` |
| `github_repo_secret` | **OVERRIDES** | `github_org_secret` |
| `github_team` | **OVERRIDES** | `github_branch_protection_rule` |
| `github_team` | **HAS** | `github_user` |
| `github_user` | **MANAGES** | `github_account` |
| `github_user` | **OVERRIDES** | `github_branch_protection_rule` |
| `github_user` | **ASSIGNED** | `github_issue` |
| `github_user` | **CREATED** | `github_issue` |
| `github_user` | **APPROVED** | `github_pullrequest` |
| `github_user` | **OPENED** | `github_pullrequest` |
| `github_user` | **REVIEWED** | `github_pullrequest` |
| `github_user` | **MANAGES** | `github_team` |
| Source Entity `_type` | Relationship `_class` | Target Entity `_type` |
| --------------------- | --------------------- | ------------------------------- |
| `github_account` | **INSTALLED** | `github_app` |
| `github_account` | **HAS** | `github_org_secret` |
| `github_account` | **OWNS** | `github_repo` |
| `github_account` | **HAS** | `github_team` |
| `github_account` | **HAS** | `github_user` |
| `github_app` | **OVERRIDES** | `github_branch_protection_rule` |
| `github_env_secret` | **OVERRIDES** | `github_org_secret` |
| `github_env_secret` | **OVERRIDES** | `github_repo_secret` |
| `github_environment` | **HAS** | `github_env_secret` |
| `github_pullrequest` | **CONTAINS** | `github_pullrequest` |
| `github_repo` | **HAS** | `github_branch_protection_rule` |
| `github_repo` | **HAS** | `github_code_scanning_finding` |
| `github_repo` | **USES** | `github_env_secret` |
| `github_repo` | **HAS** | `github_environment` |
| `github_repo` | **HAS** | `github_finding` |
| `github_repo` | **HAS** | `github_issue` |
| `github_repo` | **USES** | `github_org_secret` |
| `github_repo` | **HAS** | `github_pullrequest` |
| `github_repo` | **HAS** | `github_repo_secret` |
| `github_repo` | **USES** | `github_repo_secret` |
| `github_repo` | **ALLOWS** | `github_team` |
| `github_repo` | **ALLOWS** | `github_user` |
| `github_repo_secret` | **OVERRIDES** | `github_org_secret` |
| `github_team` | **OVERRIDES** | `github_branch_protection_rule` |
| `github_team` | **HAS** | `github_user` |
| `github_user` | **MANAGES** | `github_account` |
| `github_user` | **OVERRIDES** | `github_branch_protection_rule` |
| `github_user` | **ASSIGNED** | `github_issue` |
| `github_user` | **CREATED** | `github_issue` |
| `github_user` | **APPROVED** | `github_pullrequest` |
| `github_user` | **OPENED** | `github_pullrequest` |
| `github_user` | **REVIEWED** | `github_pullrequest` |
| `github_user` | **MANAGES** | `github_team` |

### Mapped Relationships

Expand Down
13 changes: 6 additions & 7 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@jupiterone/graph-github",
"version": "3.9.0",
"version": "3.10.1",
"description": "A JupiterOne Integration",
"repository": {
"type": "git",
Expand Down Expand Up @@ -50,18 +50,17 @@
"@octokit/request": "^5.6.3",
"@octokit/rest": "^19.0.3",
"date-fns": "^2.29.3",
"p-map": "^4.0.0",
"lodash": "^4.17.21",
"semver": "^7.3.7"
},
"peerDependencies": {
"@jupiterone/integration-sdk-core": "^11.0.3"
"@jupiterone/integration-sdk-core": "^11.2.0"
},
"devDependencies": {
"@jupiterone/integration-sdk-core": "^11.0.3",
"@jupiterone/integration-sdk-dev-tools": "^11.0.3",
"@jupiterone/integration-sdk-testing": "^11.0.3",
"@jupiterone/integration-sdk-core": "^11.2.0",
"@jupiterone/integration-sdk-dev-tools": "^11.2.0",
"@jupiterone/integration-sdk-testing": "^11.2.0",
"@octokit/graphql-schema": "^13.4.0",
"@types/lodash.omit": "^4.5.6",
"@types/node": "^20.8.2",
"@types/semver": "^7.3.13",
"auto": "^10.36.5"
Expand Down
142 changes: 71 additions & 71 deletions src/actions/__recordings__/partialIngest_1257640542/recording.har
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 8 additions & 2 deletions src/actions/partialIngestActionHandler.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import {
IntegrationError,
IntegrationLogger,
} from '@jupiterone/integration-sdk-core';
import { PullRequestResponse } from '../client/GraphQLClient';

interface EntityToIngest {
_type: string; // e.g. - github_pullrequest
Expand Down Expand Up @@ -104,7 +105,7 @@ const buildEntity = async (
await buildUserLoginMaps(client, repoName);

return toPullRequestEntity({
pullRequest,
pullRequest: pullRequest as unknown as PullRequestResponse,
commits: pullRequest.commits ?? [],
reviews: pullRequest.reviews ?? [],
labels: pullRequest.labels ?? [],
Expand All @@ -121,7 +122,12 @@ const buildUserLoginMaps = async (client: APIClient, repoName: string) => {
// There's not a good way to fetch a single member from an organization
// We can use a REST endpoint, but it pulls back different data.
await client.iterateOrgMembers((member) => {
memberByLoginMap.set(member.login, toOrganizationMemberEntity(member)._key);
if (member.login) {
memberByLoginMap.set(
member.login,
toOrganizationMemberEntity(member)._key,
);
}
});

await client.iterateRepoCollaborators(repoName, (collaborator) => {
Expand Down
Loading