Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DAP/Whisper]Add initial documents for Whisper Preprocessor. #313

Merged
merged 9 commits into from
Jul 15, 2024
Merged

[DAP/Whisper]Add initial documents for Whisper Preprocessor. #313

merged 9 commits into from
Jul 15, 2024

Conversation

taiqzheng
Copy link
Contributor

@taiqzheng taiqzheng commented May 24, 2024 •
edited
Loading

Currently, it is in draft form with some remaining work to be completed.
However, it is now capable of being used to build an end-to-end Whisper model.

Work should be done in this PR:

  • Correct format for data
    Theclang-formattool offers different strategies for formatting data. Please check file ExtandDAPPass.cpp in “Files changed”.
  • Add an example
    The example has been designed, but the file location needs to be changed.
  • Correct the relationship with DAP Pass
    Update file locations for the extand-dap pass and also change the operation name.

Future work, not in this PR:

  • Update pass pipeline
    The one-shot-bufferize pass has conflicts with some operations.
  • Update example
    Currently only one case is supported, where its audio data is directly written into the example file. To support more audio cases, we need an audio library to read data from audio files.
  • Extract operators
    Some operators need to be extracted so that they can be reused in other algorithms.
  • Optimize the algorithm
    After extracting the operators, design vectorization pattern for each one.

@taiqzheng taiqzheng marked this pull request as draft May 29, 2024 08:20
@taiqzheng taiqzheng force-pushed the preprocess branch 4 times, most recently from 5d08f61 to e433100 Compare July 1, 2024 14:07
@taiqzheng taiqzheng marked this pull request as ready for review July 2, 2024 03:33
zhanghb97
zhanghb97 approved these changes Jul 15, 2024
View reviewed changes
Copy link
Member

@zhanghb97 zhanghb97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@zhanghb97 zhanghb97 merged commit a1a0384 into buddy-compiler:main Jul 15, 2024
1 check passed
6somehow added a commit to 6somehow/buddy-mlir that referenced this pull request Sep 26, 2024
@6somehow @WuXintong123
@zhanghb97 @FloatingcloudKnight @breezejh @effrey-liu @imkiva @taiqzheng @Avimitin @ShiHaoGao
[Midend] Enhancements and Optimizations for batch matmul and convolut…
94d7db2 
…ion [Examples] Added MLIRLinalg Examples for Various Optimization Options. (#1)

* [examples] Add mobilenet example.

Co-authored-by: zhanghb97 <[email protected]>
Co-authored-by: qingqing12138 <[email protected]>

* [examples] Add examples for transformer-based model optimization.

* [examples] Add convolution optimization examples (buddy-compiler#333)


---------

Co-authored-by: FloatingcloudKnight <‘[email protected]’>

* bmm2mm 0.0

* [examples] Add attention loop and fusion example.

* [BuddyWhisper] Add whisper model example and Conv1d operation.

Co-authored-by: zhanghb97 <[email protected]>

* [examples] Add rtclock to attention fusion examples.

* [thirdparty] Add riscv-gnu-toolchain as an submodule.

* [RVV] Add RVV environment guide and update examples.

* [examples] Annotate the llama mlir code. (buddy-compiler#337)

* [thirdparty] Remove legacy RISC-V toolchain.

* [frontend] Update convolution groups feature. (buddy-compiler#338)

* [examples] Update Whisper README doc.

* [NFC] Make buddy tests and examples depends on mlir-cpu-runner (buddy-compiler#342)

* [examples] Add vector iteration example.

* [DAP/Whisper]Add Whisper Preprocessor. (buddy-compiler#313)

* [examples] Add MLIR CF example.

* [examples] Fix cf-iteration-exit filecheck.

* [example] Fix cf-iteration-exit example.

* [examples] Add sigmoid and rope case.

* tiling batch matmul

* tiling batch matmul

* [frontend] Add missing dependencies into DAP target. (buddy-compiler#357)

Ninja introduced different build graph algorithm in version v1.12.0,
which will required project to explicitly specify dependencies. Without
this commit, ninja v1.12.0 will not link mlir-translate and llc before the DAP
target and causing build failure.

* bmm tile try to remove redundant subview

* pass check

* bmm tile try to remove redundant subview

* bmm tile to vector.load/store

* buddy opt add(fake rvv version)

* bmm fuse for loop

* bmm m n border control add

* [Container] Use integrated audio decoding method.

* [examples] Use container to read audio file.

Co-authored-by: taiqzheng <[email protected]>

* [NFC] Fix warnings.

* [build system] use variable from LLVMConfig (buddy-compiler#359)

* [build system] use variable from LLVMConfig

In the previous build script, we always assumed that the binary and the
library are relative to the MLIR_DIR variable. But this prevents
buddy-mlir from properly packaging by the system. We should use the
LLVMConfig.cmake file to have a more flexible build system.

Signed-off-by: Avimitin <[email protected]>

* [python] add install target for buddy python modules

* [nix] update buddy-mlir derivation

Signed-off-by: Avimitin <[email protected]>

---------

Signed-off-by: Avimitin <[email protected]>

* [chore] fix typo in pass manager (buddy-compiler#360)

Signed-off-by: Avimitin <[email protected]>

* [container] Add initial standalone image container.

* remove rvv

* add int support

* [examples] Add initial GPU matmul transform example.

* [examples] Add conv2d-nhwc-fhwc manual vectorization.

* [Container] Add encoder for wav audio format. (buddy-compiler#366)

* [examples] Fix rvv intrinsic.

* [midend] Fix batch matmul vectorization pass.

* [example] Fix typo.

* [examples] LeNet E2E pipeline uses batchmatmul-optimize pass.

* [examples] Fix LeNet E2E pipeline.

* [examples] Add memref type generation example.

* [NFC] Fix typo.

* [Docs] Add Python Virtual Environment Setup Guide. (buddy-compiler#373)

* [examples] Update module generation example.

* add conv nhwc

* add conv nhwc

* oc bug repair

* [Examples] Adapt Audio Container for dap examples. (buddy-compiler#369)

* [Examples] Adapt the new Audio Container for 'buddy-whisper-preprocess' example.

* [Examples] Adapt the new Audio container for 'buddy-biquad' example.

* [Examples] Adapt the new Audio Container for 'buddy-fir' example.

* [Examples] Adapt the new Audio Container for 'buddy-iir-scalar' example.

* [Examples] Adapt the new Audio Container for 'buddy-iir-vectorization' example.

* [DAP] Merge 'BuddyLibDAPVectorization' library into 'BuddyLibDAP' library.

* [Container] Update the Audio container to support converting a MemRef (base class) object to an Audio (derived class) object.

* [Examples] Adapt the new constructor in the Audio Container to facilitate the conversion of a MemRef object to an Audio object.

* [Container] Handle corner case for NaN. Reset NaN to 1.

* conv2d +dilation,strides

* conv2d +tilling

* fixed int float determine

* conv2d to forall

* clear useless commits

* conv2d pass float test

* add bmm scf

* add depthwise

* add depthwise correct

* [DAP/Whisper] Extract RFFT operation from 'dap.whisper_preprocess'. (buddy-compiler#379)

* [DAP/Whisper] Remove 'memref.copy' operation in 'dap.whisper_preprocess'.

* [DAP] Extract RFFT400Op from 'dap.whisper_preprocess'.

* for dev merge

* [examples] add MLIRLinalg example for options: 1.conv-nhwc-fhwc-optimize 2.conv-nhwc-fhwc-tile-optimize  3.depthwise-conv-nhwc-hwc-optimize 4.batchmatmul-tile-optimize 5.batchmatmul-scf-optimize . Example mlir:   batchmatmul conv2d_nhwc_fhwc depthwise_conv_2d_nhwc_hwc

* [examples] add MLIRLinalg example for options: 1.conv-nhwc-fhwc-optimize 2.conv-nhwc-fhwc-tile-optimize  3.depthwise-conv-nhwc-hwc-optimize 4.batchmatmul-tile-optimize 5.batchmatmul-scf-optimize . Example mlir:   batchmatmul conv2d_nhwc_fhwc depthwise_conv_2d_nhwc_hwc

* Update .gitmodules

* Update .gitignore

* [Midend] Enhancements and Optimizations for batch matmul and convolution  [Examples] Added MLIRLinalg Examples for Various Optimization Options

* [Midend] Enhancements and Optimizations for batch matmul and convolution [Examples] Added MLIRLinalg Examples for Various Optimization Options. fixed thirdparty.

* [Examples] Added MLIRLinalg Examples for Various Optimization Options.  linalg-batch-matmul-dync.mlir fixed .

---------

Signed-off-by: Avimitin <[email protected]>
Co-authored-by: WuXintong123 <[email protected]>
Co-authored-by: zhanghb97 <[email protected]>
Co-authored-by: qingqing12138 <[email protected]>
Co-authored-by: FloatingcloudKnight <[email protected]>
Co-authored-by: FloatingcloudKnight <‘[email protected]’>
Co-authored-by: Weijia <[email protected]>
Co-authored-by: effrey-liu <[email protected]>
Co-authored-by: Wu Xintong <[email protected]>
Co-authored-by: Kiva <[email protected]>
Co-authored-by: Taiqi Zheng <[email protected]>
Co-authored-by: Jiongjia Lu <[email protected]>
Co-authored-by: taiqzheng <[email protected]>
Co-authored-by: ShiHaoGao <[email protected]>
ShiHaoGao pushed a commit to ShiHaoGao/buddy-mlir that referenced this pull request Oct 18, 2024
@taiqzheng
[DAP/Whisper]Add Whisper Preprocessor. (buddy-compiler#313)
6899dcc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhanghb97 zhanghb97 zhanghb97 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@taiqzheng @zhanghb97