add pass for linalg matmultransposeb op #377

zhxzh-2001 · 2024-08-22T03:31:25Z

add Linalg MatMultransposebpass do the vectorization to speed up llama

zhanghb97

Giving quick feedback. Additionally, please add test and example for your vectorization pass.

zhanghb97 · 2024-08-27T11:52:55Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+#include <mlir/Pass/Pass.h>
+
+#include "Utils/Utils.h"
+#include <iostream>


Is iostream library necessary here?

zhanghb97 · 2024-08-27T11:54:19Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+//===----------------------------------------------------------------------===//
+
+namespace {
+class MatMul_TransposeB_VecPattern : public ConversionPattern{


MatMul_TransposeB_VecPattern -> MatMulTransposeBVecPattern

zhanghb97 · 2024-08-27T11:55:12Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+        SmallVector<Value,8> lowerBounds(2,c0);
+        SmallVector<Value,8> uperBounds {aRow,bRow/*bCol*/};
+        SmallVector<int64_t,8> steps{1,1};
+        //TODO


Please provide more detailed information about the TODO.

zhanghb97 · 2024-08-27T11:56:15Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+        affine::buildAffineLoopNest(
+            rewriter,loc,lowerBounds,uperBounds,steps,
+            [&](OpBuilder &builder,Location loc,ValueRange ivs){
+                //Value sum_0 = builder.create<mlir::arith::ConstantOp>(


If we do not need this line, please remove it.

zhanghb97 · 2024-08-27T11:57:02Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+                builder.create<affine::AffineStoreOp>(loc,sum.getResult(0),C,ValueRange{ivs[0],ivs[1]});
+            }
+        );
+        // clang-format on


Did not find the matched clang-format off.

zhanghb97 · 2024-08-27T11:57:27Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+        return success();   
+    }
+private:
+    int64_t vecsize;


vecsize -> vecSize

zhanghb97 · 2024-08-27T11:58:05Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+        :public PassWrapper<MatMul_TransposeB_VecPass,OperationPass<ModuleOp>>{
+public:
+    MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(MatMul_TransposeB_VecPass)
+    StringRef getArgument() const final{ return "matul-vectorization2"; }


matul-vectorization2 is not a good name for the pass.

zhanghb97 · 2024-08-27T11:58:31Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+  PassRegistration<MatMul_TransposeB_VecPass>();
+}
+} // namespace buddy
+} // namespace mlir


Please add an empty line here.

zhanghb97

Put the Python/Graph part in a separate PR. This PR only adds matmul transpose vectorization.

examples/BuddyMatmul/linalg-transposematmulb-f32.mlir

zhanghb97 · 2024-10-12T01:13:51Z

examples/BuddyMatmul/makefile

@@ -35,3 +35,22 @@ linalg-batchmatmul-f32-run:
 		-reconcile-unrealized-casts | \
 	${MLIR_CPU_RUNNER} ${OPT_FLAG} -e main -entry-point-result=void \
 		-shared-libs=${MLIR_RUNNER_UTILS} -shared-libs=${MLIR_C_RUNNER_UTILS}
+
+linalg-transposematmulb-f32-run:


Split transpose, matmul, and b. (transposematmulb -> matmul-transpose-b)

zhanghb97 · 2024-10-12T01:14:28Z

examples/BuddyMatmul/makefile

+
+linalg-transposematmulb-f32-run:
+	@${BUDDY_OPT} ./linalg-transposematmulb-f32.mlir\
+		-transpose_matmul_bvectorization \


-transpose_matmul_bvectorization -> -matmul_transpose_b_vectorization

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

zhanghb97 · 2024-10-12T01:18:50Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+public:
+    MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(MatMulTransposeBVecPass)
+    StringRef getArgument() const final{ return "transpose_matmul_bvectorization"; }
+    StringRef getDescription() const final { return "MatMul Vectorization second version.MatMul receive tensortype oprands."; }


"MatMul Vectorization second version.MatMul receive tensortype oprands." This is not a good description for this pass.

zhanghb97 · 2024-10-12T01:19:19Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+        :public PassWrapper<MatMulTransposeBVecPass,OperationPass<ModuleOp>>{
+public:
+    MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(MatMulTransposeBVecPass)
+    StringRef getArgument() const final{ return "transpose_matmul_bvectorization"; }


transpose_matmul_bvectorization -> matmul_transpose_b_vectorization

xlinsist · 2024-10-15T13:36:51Z

examples/BuddyMatmul/linalg-transposematmulb-f32.mlir

@@ -0,0 +1,79 @@
+// RUN: buddy-opt %s \
+// RUN:     -matmul_transpose_b_vectorization \


Recommend to change -matmul_transpose_b_vectorization to -matmul-transpose-b-vectorization

xlinsist · 2024-10-15T13:40:50Z

I encountered this error after using ninja && ninja check-buddy in the build directory. Could you reproduce it in your machine?

xlinsist · 2024-10-21T10:14:32Z

examples/BuddyMatmul/linalg-transposematmulb-f32.mlir

+  call @printMemrefF32(%printed_m5) : (memref<*xf32>) -> ()
+
+  return
+}


Add a new line at the end of the file.

xlinsist · 2024-10-21T10:20:44Z

tools/buddy-opt/buddy-opt.cpp

@@ -80,6 +80,10 @@ void registerLowerSchePass();
 void registerFuncBufferizeDynamicOffsetPass();
 void registerConvertMemcpyToGPUPass();
 void registerLegalizeShmemOutliningPass();
+void registerMatMul_TransposeB_VecPass();


What is the difference between registerMatMul_TransposeB_VecPass and registerMatMulTransposeBVecPass? Does this code have any particular meaning?

sorry,i typed an extra function.there is no pass named "registerMatMul_TransposeB_VecPass"

I'll delete it

xlinsist · 2024-10-21T10:23:13Z

midend/lib/Conversion/MatMulOptimization/MatMulTransposeBVec.cpp

+        Value B = op->getOperand(1);
+        Value C = op->getOperand(2);
+
+        // Get shape of input and output


typo: add period

xlinsist · 2024-10-21T10:26:25Z

frontend/Python/ops/linalg.py

@@ -1968,6 +1986,7 @@ def gt_op(node: GtOp, symbol_table):

 ops_registry = {
    "MatmulOp": matmul_op,
+    "transpose_Matmul_fusedOp": transpose_matmul_fused_op,


Recommend naming "TransposeMatmulFusedOp"

I'll put some changes to the graph in the next commit.So I'll delete it for now.

xlinsist · 2024-10-21T10:28:29Z

frontend/Python/graph/transform/fuse_ops.py

Is there any way I can verify the changes here, or will this part of the changes be verified in another related PR?

zhxzh-2001 · 2024-10-21T13:03:17Z

yes, it could be verified in my next pr.This pass is for an op which will be used after the next modification of graph,and will help with llama.

zhanghb97 reviewed Aug 27, 2024

View reviewed changes

zhanghb97 reviewed Oct 12, 2024

View reviewed changes

xlinsist reviewed Oct 15, 2024

View reviewed changes

“username” added 8 commits October 16, 2024 10:06

add pass for linalg matmultransposeb op

550f1b0

add example for linalg matmultransposeb op and check the code layout

2f030a2

check the code layout

834bab9

vector.transfer_read -> vector.load

76aa30f

fuse tanspose+matmul

cbf0a66

fix problems about pass MatMulTransposeBVec

9a3c051

deal with conflicts

e166353

change pass name

9e0980f

zhxzh-2001 force-pushed the main branch from 51dced4 to 9e0980f Compare October 16, 2024 02:20

fix problem for check-buddy

1730830

xlinsist reviewed Oct 21, 2024

View reviewed changes

correct the format,and removed some unnecessary changes to the pass.

8a0989d

xlinsist merged commit 5d9d52b into buddy-compiler:main Oct 22, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add pass for linalg matmultransposeb op #377

add pass for linalg matmultransposeb op #377

zhxzh-2001 commented Aug 22, 2024

zhanghb97 left a comment

zhanghb97 Aug 27, 2024

zhanghb97 Aug 27, 2024

zhanghb97 Aug 27, 2024

zhanghb97 Aug 27, 2024

zhanghb97 Aug 27, 2024

zhanghb97 Aug 27, 2024

zhanghb97 Aug 27, 2024

zhanghb97 Aug 27, 2024

zhanghb97 left a comment

zhanghb97 Oct 12, 2024

zhanghb97 Oct 12, 2024

zhanghb97 Oct 12, 2024

zhanghb97 Oct 12, 2024

xlinsist Oct 15, 2024

xlinsist commented Oct 15, 2024

xlinsist Oct 21, 2024

xlinsist Oct 21, 2024

zhxzh-2001 Oct 21, 2024

zhxzh-2001 Oct 21, 2024

xlinsist Oct 21, 2024

xlinsist Oct 21, 2024

zhxzh-2001 Oct 21, 2024

xlinsist Oct 21, 2024

zhxzh-2001 commented Oct 21, 2024

		@@ -0,0 +1,79 @@
		// RUN: buddy-opt %s \
		// RUN: -matmul_transpose_b_vectorization \

add pass for linalg matmultransposeb op #377

add pass for linalg matmultransposeb op #377

Conversation

zhxzh-2001 commented Aug 22, 2024

zhanghb97 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhanghb97 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xlinsist commented Oct 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhxzh-2001 commented Oct 21, 2024