Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial benchmark run #2256

Merged
merged 7 commits into from
Oct 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/actions/get_exclude_dirs/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,17 +39,17 @@ runs:
') }}"

SLOW_EXCLUSIONS="${{ format('
tests/end_to_end/candid_rpc/functional_syntax/ckbtc
tests/end_to_end/candid_rpc/class_syntax/bitcoin
tests/end_to_end/http_server/large_files
tests/end_to_end/http_server/open_value_sharing
tests/end_to_end/candid_rpc/class_syntax/stable_structures
tests/end_to_end/candid_rpc/functional_syntax/bitcoin
tests/end_to_end/candid_rpc/functional_syntax/composite_queries
tests/end_to_end/candid_rpc/functional_syntax/ckbtc
tests/end_to_end/candid_rpc/functional_syntax/cross_canister_calls
tests/end_to_end/candid_rpc/functional_syntax/management_canister
tests/end_to_end/candid_rpc/functional_syntax/stable_structures
tests/end_to_end/http_server/autoreload
tests/end_to_end/http_server/large_files
tests/end_to_end/http_server/open_value_sharing
') }}"

EXCLUDE_DIRS=""
Expand Down
14 changes: 4 additions & 10 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,28 +86,22 @@ jobs:
name: 'Property Class',
directories: './tests/property/candid_rpc/class_api'
}
- {
name: 'Property Functional',
directories: './tests/property/candid_rpc/functional_api'
}
- {
name: 'Property IC API',
directories: './tests/property/ic_api'
}
uses: ./.github/workflows/benchmark_parallel.yml
secrets:
GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LASTMJS_GITHUB_TOKEN: ${{ secrets.LASTMJS_GITHUB_TOKEN }}
with:
base-branch: ${{ needs.create-branch-prefix.outputs.base-branch }}
branch-prefix: ${{ needs.create-branch-prefix.outputs.branch-prefix }}
directories: ${{ matrix.benchmark_group.directories }}
exclude-dirs: ${{ needs.get-exclude-dirs.outputs.exclude-dirs }}
branch-prefix: ${{ needs.create-branch-prefix.outputs.branch-prefix }}

squash-branches:
needs: [run-benchmarks, create-branch-prefix]
uses: ./.github/workflows/squash_branches.yml
with:
base-branch: ${{ needs.create-branch-prefix.outputs.base-branch }}
branch-prefix: ${{ needs.create-branch-prefix.outputs.branch-prefix }}
commit-message: 'run benchmarks'
secrets:
Expand All @@ -128,7 +122,7 @@ jobs:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr create \
--base ${{ github.ref_name }} \
--base main \
--head ${{ needs.create-branch-prefix.outputs.base-branch }} \
--title "Benchmark Results for ${{ needs.create-branch-prefix.outputs.base-branch }}" \
--body "Automated PR for benchmark results"
13 changes: 9 additions & 4 deletions .github/workflows/benchmark_parallel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,19 @@ name: Parallel Benchmark
on:
workflow_call:
inputs:
base-branch:
required: true
type: string
branch-prefix:
required: true
type: string
directories:
required: true
type: string
exclude-dirs:
required: false
type: string
default: ''
branch-prefix:
required: true
type: string

secrets:
GPG_SIGNING_KEY:
Expand All @@ -29,6 +32,8 @@ jobs:
test-infos: ${{ steps.get-test-infos.outputs.test-infos }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.base-branch }}

- id: get-test-infos
uses: ./.github/actions/get_test_infos
Expand All @@ -50,7 +55,7 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.ref }}
ref: ${{ inputs.base-branch }}
token: ${{ secrets.LASTMJS_GITHUB_TOKEN || github.token }}

- uses: ./.github/actions/setup_node
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,6 @@ jobs:
release-version: ${{ inputs.release-version }}
test-infos: ${{ steps.get-test-infos.outputs.test-infos }}
steps:
- name: Print branch name
run: echo "${{ github.ref }}"

- uses: actions/checkout@v4
with:
ref: ${{ github.ref }}
Expand Down Expand Up @@ -115,6 +112,7 @@ jobs:
needs: [prepare-release, update-test-files-for-release-commit]
uses: ./.github/workflows/squash_branches.yml
with:
base-branch: release--${{ needs.prepare-release.outputs.release-version }}
branch-prefix: 'update--${{ needs.prepare-release.outputs.release-version }}-'
commit-message: 'Update test files for all tests and examples ${{ needs.prepare-release.outputs.release-version }}'
secrets:
Expand Down Expand Up @@ -159,7 +157,7 @@ jobs:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr create \
--base ${{ github.ref_name }} \
--base main \
--head release--${{ needs.prepare-release.outputs.release-version }} \
--title "Release ${{ needs.prepare-release.outputs.release-version }}" \
--body "Automated PR for release ${{ needs.prepare-release.outputs.release-version }}"
5 changes: 4 additions & 1 deletion .github/workflows/squash_branches.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ name: Squash Branches
on:
workflow_call:
inputs:
base-branch:
required: true
type: string
branch-prefix:
required: true
type: string
Expand All @@ -26,7 +29,7 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.ref }}
ref: ${{ inputs.base-branch }}
token: ${{ secrets.LASTMJS_GITHUB_TOKEN }}
fetch-depth: 0

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "azle",
"version": "0.25.0",
"version": "0.25.0-pre-bifurcation",
"description": "TypeScript and JavaScript CDK for the Internet Computer",
"scripts": {
"typecheck": "tsc --noEmit",
Expand Down
2 changes: 1 addition & 1 deletion test/benchmarks/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ async function updateBenchmarksForCanisters(
[canisterName]: {
previous: shouldUpdatePrevious
? acc[canisterName]?.current ?? {
version,
version: 'No previous benchmarks',
benchmarks: []
}
: acc[canisterName]?.previous,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"audio_recorder": {
"previous": { "version": "No previous benchmarks", "benchmarks": [] },
"current": {
"version": "0.25.0-pre-bifurcation",
"benchmarks": [
{
"instructions": { "__bigint__": "11206500" },
"method_name": "createUser",
"timestamp": { "__bigint__": "1730324763454588684" }
},
{
"instructions": { "__bigint__": "30853993" },
"method_name": "createRecording",
"timestamp": { "__bigint__": "1730324765362717539" }
},
{
"instructions": { "__bigint__": "43555749" },
"method_name": "deleteRecording",
"timestamp": { "__bigint__": "1730324767521567584" }
},
{
"instructions": { "__bigint__": "30641515" },
"method_name": "createRecording",
"timestamp": { "__bigint__": "1730324769463568493" }
},
{
"instructions": { "__bigint__": "29769073" },
"method_name": "deleteUser",
"timestamp": { "__bigint__": "1730324771508584739" }
}
]
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Benchmarks for audio_recorder

## Current benchmarks Azle version: 0.25.0-pre-bifurcation

| Id | Method Name | Instructions | Cycles | USD | USD/Million Calls |
| --- | --------------- | ------------ | ---------- | ------------- | ----------------- |
| 0 | createUser | 11_206_500 | 5_072_600 | $0.0000067449 | $6.74 |
| 1 | createRecording | 30_853_993 | 12_931_597 | $0.0000171948 | $17.19 |
| 2 | deleteRecording | 43_555_749 | 18_012_299 | $0.0000239504 | $23.95 |
| 3 | createRecording | 30_641_515 | 12_846_606 | $0.0000170817 | $17.08 |
| 4 | deleteUser | 29_769_073 | 12_497_629 | $0.0000166177 | $16.61 |

## Baseline benchmarks Azle version: No previous benchmarks

No benchmarks reported

---

**Note on calculations:**

- Cycles are calculated using the formula: base_fee + (per_instruction_fee \* number_of_instructions) + (additional_fee_per_billion \* floor(number_of_instructions / 1_000_000_000))
- base_fee: 590_000 cycles
- per_instruction_fee: 0.4 cycles
- additional_fee_per_billion: 400_000_000 cycles per billion instructions
- USD value is derived from the total cycles, where 1 trillion cycles = 1 XDR, and 1 XDR = $1.329670 (as of October 24, 2024)

For the most up-to-date XDR to USD conversion rate, please refer to the [IMF website](https://www.imf.org/external/np/fin/data/rms_sdrv.aspx).
For the most current fee information, please check the [official documentation](https://internetcomputer.org/docs/current/developer-docs/gas-cost#execution).
Loading
Loading