Extend TestingGuide (adds notes on formatting) #216

banach-space · 2025-02-19T13:14:55Z

No description provided.

joker-eph

Thanks!

joker-eph · 2025-02-19T13:33:47Z

website/content/getting_started/TestingGuide.md

+* Follow existing conventions.
+
+By applying these best practices, we leverage available tools (e.g., test
+function names) to make tests easier to discover and maintain.


This a good addition: I would still add some documentation in comment to a test: it's not always clear just from a test name what's going on, and the same logic as in code applies (document the why, the context, the "what's no obvious from the code").

Yes, comments are also super important, thanks for raising this!

I have some ideas on "comments", so let me treat this as encouragement to extend this PR with a dedicated section on documentation :)

jpienaar

Thanks! Yes discoverability is important here.

website/content/getting_started/TestingGuide.md

rengolin · 2025-02-19T14:11:57Z

Awesome addition. All good practices that we've been telling each other for years but always too lazy to encode in a document. :)

kuhar · 2025-02-19T14:49:15Z

website/content/getting_started/TestingGuide.md

+
+When adding new tests, strive to follow these two key rules:
+
+1. **Follow the existing naming and whitespace style.**


When you say 'existing style', what scope do you have in mind? It could be that the style of a specific directory is inconsistent with the style used in majority of other dialects etc. IE, do we want to make things globally or locally consistent, or something else?

I’m deliberately avoiding specifying this :)

Making things globally consistent would be great for... consistency. However, that assumes one size fits all, which isn't always realistic. Given the current state of things, I also feel this would be an impractical goal.
The same applies to per-file vs per-directory consistency. For example, "mlir/test/Dialect/Vector" has around 50 files - enforcing 100% consistency across them would require significant effort and churn.

My view is that contributors and reviewers should make the right call on a case-by-case basis while being mindful of:

Reinventing the wheel in terms of formatting.

New tests ignoring existing conventions (whether at the file or directory level).

Ultimately, I see this as a gradual process. Let’s start with these guidelines to help "correct the course." Over time, we can refine them - possibly transitioning from guideline to requirement if it makes sense.

WDYT?

I've worked on a few different codebases that approached this differently: some preferred local consistency, some global (e.g., google3). I agree that this doesn't have to one or the other and there's some room for interpretation, but at the same time I'd like to have some codified incentive that will push us towards higher quality tests/code.

For example, we could say we prefer global consistency unless the local standards are either higher or more specialized. If a PR adheres to the global standards but sticks like a sore thumb compared to other local code, I'd think that in most cases sending a separate clean up PR would be a reasonable request.

@banach-space ☝️

Thanks, Jakub! Sorry for the delayed reply—I haven’t had much time for this PR today. 😅

My overall sentiment is that we should approach defining these guidelines as an incremental process, avoiding overly strict or prescriptive rules.

For example, we could say we prefer global consistency unless the local standards are either higher or more specialized.

I personally like that, but I worry that others may disagree, and we might get stuck discussing it. Also, if we ask for global consistency, who is actually going to help us get there with what we have today?

We shouldn't require new contributions to be fully consistent with everything we already have, given how... inconsistent our tests are. 😅

Proposed Levels of Consistency

How about defining different levels of consistency?

Per-PR consistency

When adding new Ops, documentation, and tests, they should use an identical style.

Mandatory.

Per-file consistency

If a file is already consistent, maintain that.

If a file is inconsistent, create a GitHub issue to address it.

Mandatory.

Per-directory consistency

This might require a lot of churn, so I’m unsure about enforcing it strictly.

Nice-to-have.

Per-project (MLIR-wide) consistency

Achieving this would require volunteers to refactor existing tests.

Nice-to-have.

This is just a rough idea - suggestions are welcome!

My overall sentiment is that we should approach defining these guidelines as an incremental process, avoiding overly strict or prescriptive rules.
I personally like that, but I worry that others may disagree, and we might get stuck discussing it.

This is a very good point, let's revisit this once the initial version lands.

Also, if we ask for global consistency, who is actually going to help us get there with what we have today?
We shouldn't require new contributions to be fully consistent with everything we already have, given how... inconsistent our tests are.

Maybe 'consistency' is not the best term here. The idea would be that we would be getting closer to the agreed on 'testing guidelines' (similar to the coding standards) incrementally as we land new code, and refactor old tests along the way. Having the coding standards codified and linkable is a fantastic tool in code reviews because it allows you to point out the best practices without making it feel like you are making one-of arguments against someone's implementation. I'd hope that with testing guidelines we can accomplish something similar and gradually steer tests towards higher maintainability/quality/readability.

IE, my proposed rule of thumb would be something along the lines: 'Adhere to the testing guidelines while following the local naming and formatting style.'

website/content/getting_started/TestingGuide.md

kuhar · 2025-02-19T14:51:38Z

website/content/getting_started/TestingGuide.md

+// CHECK-LABEL:   func @maskedload_regression_3(
+// CHECK-SAME:        %[[A0:.*]]: memref<16xf32>,
+// CHECK-SAME:        %[[A1:.*]]: vector<16xf32>) -> vector<16xf32> {
+//      CHECK:    return %[[A1]] : vector<16xf32>
+func.func @maskedload_regression_3(%arg0: memref<16xf32>, %arg1: vector<16xf32>) -> vector<16xf32> {
+  %c0 = arith.constant 0 : index
+  %vec_i1 = vector.constant_vec_i1 [0] : vector<16xi1>
+
+  %ld = vector.maskedload %arg0[%c0], %vec_i1, %arg1
+    : memref<16xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+
+  return %ld : vector<16xf32>
+}


Do we need all 3 examples to drive your point or could we limit this to just two?

"Generalization Takes Three Examples" :)

It allows me to go into a bit of nuance in this paragraph (i.e. to identify the commonalities):

##### Step 2: Improve Test Naming Instead of using "regression" (which doesn't add unique information), rename tests based on key attributes: * All examples test the `vector.maskedload` to `vector.load` lowering. * The first test uses a _dynamically_ shaped `memref`, while the others use _static_ shapes. * The mask in the first two examples is "all true" (`vector.constant_mas [16]`), while it is "all false" (`vector.constant_mask [0]`) in the thir example. * The first and the third tests use `i32` elements, whereas the second uses `i8`.

That's not something I feel super strongly about. Usually its tricky to see the "optimal" solution on first iteration and I guess that's the case here.

website/content/getting_started/TestingGuide.md

adam-smnk · 2025-02-19T16:19:47Z

website/content/getting_started/TestingGuide.md

+* The mask in the first two examples is "all true" (`vector.constant_mas
+  [16]`), while it is "all false" (`vector.constant_mask [0]`) in the thir
+  example.
+* The first and the third tests use `i32` elements, whereas the second uses


The key attributes should reflect what's important for the tested transform, in this case having different data types is good for completeness but they don't trigger any special/separate logic.
It would be sufficient to highlight one case that uses different data type with, for example, a suffix and omit data type for others.

I know that's not the intention behind this example but it can be misused to push for "enterprisification" (classic java gem: https://projects.haykranen.nl/java/) of naming with combinatorial explosion of all the little details.

These are good points and I have come across this myself (without a good solution).

One specific example are Vector ops that can represent "scalars" as:

vector<1xf32> vs vector<f32> vs f32.

To differentiate tests, we started adding things like @name_from_vec0d_to_vec1d, @name_from_vec1d_to_vec0d ... It spirals very quickly.

I don't have a solution for this, but will definitely incorporate your suggestion to ... not forget about the common sense 😅

Added a short paragraph in this commit. Let me know what you think - do we need more?

Great clause 👍
It conveys the spirit of these guidelines and defends against too pedantic rule lawyers.

adam-smnk · 2025-02-19T16:27:15Z

website/content/getting_started/TestingGuide.md

+
+When adding new tests, strive to follow these two key rules:
+
+1. **Follow the existing naming and whitespace style.**


nit: Should the existing bad style (test_1, test_2, ...) prevent naming new test cases more reasonably?

Great point, thanks! Let me add a dedicated paragraph on that.

website/content/getting_started/TestingGuide.md

Fix formatting, add paragraph on what to do when there's no (good) pre-exisiting style to follow.

Add a section on documentaiton

banach-space · 2025-02-20T11:22:41Z

Thank you for the reviews 🙏🏻 I've addressed most of your comments + added a section on "commenting"/"documenting".

Let me know what you think and I will happily iterate based on the feedback.

website/content/getting_started/TestingGuide.md

dcaballe · 2025-02-22T01:46:58Z

Thanks a lot! We definitely needed some specific guidelines here. Thanks for putting in the time!

I think it might be helpful to clarify the level of enforcement we expect for these guidelines and the degree of flexibility we should allow. I fully agree with the motivation points highlighted in the PR but I'm also worried that we make things painfully rigid. If we set the bar too high, there’s a risk that testing becomes tedious enough to potentially impact actual coverage. I’m also worried that some of the renaming/formatting aspects of the guidelines take the focus away in the code reviews from other aspects that should have more attention.

To alleviate some of this, esp. if we are going after a high level of enforcement, I think we should put some effort into automating some of the renaming/formatting aspects. For example, we could extend generate-test-checks.py (or implement a new tool) for MLIR so that it generates more meaningful CHECK variable names by default, derived from the dialect or op name, for example. That could greatly enhance consistency by default while minimizing manual effort.

I also feel consistency is open to interpretation. For example, if an existing test uses V_INPUT_0 as a CHECK variable name and a new test is added with a INPUT0 variable name, should we enforce a change? That name seems valid to me and we don’t enforce such a level of naming consistency on the C++ code. I think we may want to allow the same level of flexibility on the testing side.

Anyways, some food for thoughts and probably unpopular opinions I wanted to share 😊. Great writeup and ideas overall. Thanks!

banach-space · 2025-02-23T14:21:45Z

Thank you for the generous feedback, Diego! These are some great points.

Before addressing specific comments, I see the introduction of new testing standards as a three-step process:

Codify expectations (this PR).
Endorse & embrace—let the community adapt and refine the guidelines.
Enforce—once broadly adopted, introduce tooling to assist with enforcement.

This is merely Step 1. Right now, we have neither guidelines nor a clear place to document best practices.

I think it might be helpful to clarify the level of enforcement we expect for these guidelines and the degree of flexibility we should allow.

This might be tricky, and I'd rather leave it under-specified. Many LLVM guidelines are also loosely defined, and I think that’s fine.

I’m also worried that some of the renaming/formatting aspects of the guidelines take the focus away in the code reviews from other aspects that should have more attention.

Agreed - there needs to be a healthy balance. However, IMHO, not enough effort is put into future-proofing and maintaining our tests.

My hope is that these guidelines will streamline discussions and reduce time spent debating formats in reviews. Ultimately, this should free up time to focus on the actual code.

If we are going after a high level of enforcement, I think we should put some effort into automating some of the renaming/formatting aspects.

I don’t think we should automate this just yet.

For example, we could extend generate-test-checks.py (or implement a new tool) for MLIR so that it generates more meaningful CHECK variable names by default, derived from the dialect or op name.

I fully agree that better tooling would help, and this is something we should aim for in the future. However, there’s little point in improving tooling if we haven’t yet agreed on what we want to standardize.

First, we need to (1) codify best practices.
Once the community has (2) embraced these guidelines, we can explore ways to automate (3) enforcement.

I’ve considered extending generate-test-checks.py, but I believe we will always need human input to provide meaningful naming based on semantics.

Below is an automatically generated test check using today's version:

// Automatically generated - today’s version
// CHECK-LABEL:   func.func @bitcast_2d(
// CHECK-SAME:                          %[[VAL_0:[0-9]+|[a-zA-Z$._-][a-zA-Z0-9$._-]*]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK:           %[[VAL_1:.*]] = vector.bitcast %[[VAL_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK:           %[[VAL_2:.*]] = vector.bitcast %[[VAL_1]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK:           return %[[VAL_2]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
  %0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
  %1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
  return %1 : vector<2x16xi8>
}

For a mild improvement, we could extend generate-test-checks.py to rename VAL_0 and VAL_1 to BITCAST_1 and BITCAST_2:

// Automatically generated - possible near-future version (mild improvement)
// CHECK-LABEL:   func.func @bitcast_2d(
// CHECK-SAME:                          %[[ARG_0]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK:           %[[BITCAST_1:.*]] = vector.bitcast %[[ARG_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK:           %[[BITCAST_2.:*]] = vector.bitcast %[[BITCAST_1]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK:           return %[[BITCAST_2]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
  %0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
  %1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
  return %1 : vector<2x16xi8>
}

However, ideally, we’d want something context-aware, such as:

// Human-curated, context-aware naming
// CHECK-LABEL:   func.func @bitcast_2d(
// CHECK-SAME:                          %[[ARG_0]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK:           %[[UPCAST:.*]] = vector.bitcast %[[ARG_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK:           %[[DOWNCAST.:*]] = vector.bitcast %[[UPCAST]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK:           return %[[DOWNCAST]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
  %0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
  %1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
  return %1 : vector<2x16xi8>
}

To achieve this level of naming automation, we’d likely need LSP-based tooling, which is beyond my current capacity to explore.

I also feel consistency is open to interpretation.

Yes, I deliberately left "consistency" under-specified. I feel it's something where we should trust our judgment. See also this comment from Jakub (my reply).

Ultimately, we could reduce it to a simple question:

"Are you re-using an existing style?"
- If yes, then "You are maintaining a level of consistency, thank you!"
- If not, then "Please provide rationale." (e.g. "There's no pre-exisitng style to follow.")
Most importantly, "Please be mindful of consistency when contributing tests 🙏🏻 "

If an existing test uses V_INPUT_0 as a CHECK variable name and a new test is added with INPUT0, should we enforce a change?

In this case, assuming V_INPUT_0 appears in every test in a given file, I would ask the contributor: 

Why not keep things simple and consistent? Following the existing style makes it easier to compare test cases and identify edge cases. If you choose to diverge, please provide a rationale.

Ultimately, we should also trust our judgment - both as reviewers and contributors.

Final point

The actual guidelines/requirements boil down to:

Be consistent and, where possible, follow the existing style.
Document your tests using comments, meaningful function names, and descriptive MLIR/LIT variable names.

The examples I included in this PR are just that - examples. I don’t think the general principles are sufficient without concrete illustrations, but again, these are just suggestions to inspire contributors rather than rigid rules.

Thanks for considering this!

Fix typos/formatting, add paragraph on common sense

website/content/getting_started/TestingGuide.md

adam-smnk · 2025-02-24T11:10:57Z

website/content/getting_started/TestingGuide.md

+/// The permutation map was replaced with vector.transpose
+// CHECK-NOT:       permutation_map


This example is also made by the second example. Could they be collapsed?

Also, while the previous 3 examples at least are driven by more detailed narrative, I feel these verbose examples are overkill for a message that is summarized with two bullet points (just to be clear - I mean smaller example is more readable, not adding more description ;) ).

Or perhaps the key parts could use some emphasis (bold etc.). Or the actual test details could be omitted for documentation clarity.
Blob of IR triggers tl;dr before I filter out the message.

Same for the convolution example.

Agreed, let me trim these examples. Let me know what you think.

Or perhaps the key parts could use some emphasis (bold etc.).

Is it possible in code blocks in Markdown?

Same for the convolution example.

Yes, it's quite long. It's a bit tricky to find good examples that are also short 🤔 But I totally agree that examples in this guide should be of highest possible quality. No pressure! 😅

Or perhaps the key parts could use some emphasis (bold etc.).

Is it possible in code blocks in Markdown?

Not certain but most likely no (in standard Markdown).
Docs and examples like these are always art of tradeoffs. Do these the examples need to be complete and self-contained? Or are they primarily a vehicle to guide (reader's gaze) through key concepts?

I like the cleanups for Test Documentation Best Practices section. Each code block has is a self contained example that showcases a concept with matching description.

In case of Example: Improving Test Readability & Naming, I feel it's hard to follow changes done with each step. Especially as each code block fills up my screen, I have to jump between current block, previous block, and the step description.

Since formatting is not really an option, maybe emphasis by omission of the common elements could help? Sth like:
Base case:

// CHECK-LABEL: func @maskedload_regression_1( // CHECK-SAME: %[[A0:.*]]: memref<?xf32>, // CHECK-SAME: %[[A1:.*]]: vector<16xf32> func.func @maskedload_regression_1(%arg0: memref<?xf32>, %arg1: vector<16xf32>) { ... // CHECK: %[[LOAD:.*]] = vector.load %[[A0]][%[[C]]] : memref<?xf32>, vector<16xf32> %ld = vector.maskedload %arg0[%c0], %vec_i1, %arg1 ... }

Step 1 - Use Consistent Variable Names:

// CHECK-LABEL: func @maskedload_regression_1( // CHECK-SAME: %[[BASE:.*]]: memref<?xf32>, // CHECK-SAME: %[[PASS_THRU:.*]]: vector<16xf32> func.func @maskedload_regression_1(%base: memref<?xf32>, %pass_thru: vector<16xf32>) { ... // CHECK: %[[LOAD:.*]] = vector.load %[[BASE]][%[[C]]] %ld = vector.maskedload %base[%c0], %mask, %pass_thru ... }

Step 2 - Improve Test Naming

// CHECK-LABEL: func @maskedload_to_load_dynamic_i32_all_true( // CHECK-SAME: %[[BASE:.*]]: memref<?xf32>, // CHECK-SAME: %[[PASS_THRU:.*]]: vector<16xf32> func.func @maskedload_to_load_dynamic_i32_all_true(%base: memref<?xf32>, %pass_thru: vector<16xf32>) { ... }

The doc section could open with IR blob with the original three test cases, show these reduced steps, and close with all three test cases shown again with all these steps applied.

As usual, different tradeoffs - just treat it as yet another possibility.
And of course, it is biased toward my style of reading where I prefer to do quick visual diff than read text 😉

website/content/getting_started/TestingGuide.md

javedabsar1 · 2025-02-24T20:45:49Z

website/content/getting_started/TestingGuide.md

+When adding new tests, strive to follow these two key rules:
+
+1. **Follow the existing naming and whitespace style.**
+   - This applies when modifying existing test files.


s/when modifying existing test files/when modifying existing test files that already contain a number of tests following a particular convention that likely fits in that context/

I like your suggestion, thanks! Shall we also ask people to create GitHub issues when a test file does not follow a consistent style?

EDIT: Let me add it to #### What if there is no pre-existing style to follow? and keep this section short.

CC @kuhar (I mentioned something similar in another thread with you)

javedabsar1 · 2025-02-24T20:52:30Z

website/content/getting_started/TestingGuide.md

+2. **Consistently document the edge case being tested.**
+   - Clearly state what makes this test unique and how it complements other
+     similar tests.
+


Orthogonal tests
Do not test the same things again and again. When writing tests, decide what are the categories C1,C2, you want to test (e.g. number of loops, data types). That gives a natural naming loop_depth2_int. Struggling with good naming is sometimes reflective of absence of regular pattern in the set of tests being written.

Names and Comments
Test names should reflect what is being tested. The why shouldnt be attempted to code in the name and unless obvious from code/context, must be added as comment but avoiding over-explainng..

These are great, thanks @javedabsar1 ! I'd like to keep this particular section very short (and limit the guidelines to two very high level points). Is it OK if I add these points in #### What if there is no pre-existing style to follow? instead?

These are great, thanks @javedabsar1 ! I'd like to keep this particular section very short (and limit the guidelines to two very high level points). Is it OK if I add these points in #### What if there is no pre-existing style to follow? instead?

Sounds good.

website/content/getting_started/TestingGuide.md

javedabsar1 · 2025-02-24T21:03:51Z

Thanks @banach-space for doing this work. much appreciated.

dcaballe · 2025-02-25T01:22:11Z

Thanks, Andrzej. Yes, I mostly agree with all that you said! I think the example you provided above illustrates my specific concern:

For a mild improvement, we could extend generate-test-checks.py to rename VAL_0 and VAL_1 to BITCAST_1 and BITCAST_2:

// Automatically generated - possible near-future version (mild improvement)
// CHECK-LABEL: func.func @bitcast_2d(
// CHECK-SAME: %[[ARG_0]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK: %[[BITCAST_1:.]] = vector.bitcast %[[ARG_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK: %[[BITCAST_2.:]] = vector.bitcast %[[BITCAST_1]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK: return %[[BITCAST_2]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
%0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
%1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
return %1 : vector<2x16xi8>
}
However, ideally, we’d want something context-aware, such as:

// Human-curated, context-aware naming
// CHECK-LABEL: func.func @bitcast_2d(
// CHECK-SAME: %[[ARG_0]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK: %[[UPCAST:.]] = vector.bitcast %[[ARG_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK: %[[DOWNCAST.:]] = vector.bitcast %[[UPCAST]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK: return %[[DOWNCAST]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
%0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
%1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
return %1 : vector<2x16xi8>
}

Someone might find that BITCAST_1 is perfect, while someone else might find that it’s not capturing enough information. Or the other way around! I’d personally argue that for a test with three CHECK rules, even VAL_1 or A would be just fine. This is the kind of arguments that concerns me. As you well said, these guidelines are subject to interpretation and opinion so I think it's important that we somewhat define where the "acceptable band" is.

Regarding the automation, honestly, I think having a tool that generates variables based on the dialect/op names would be a fantastic starting point and help define where the acceptable bar is. Of course, I'm not saying you should implement such a tool :)

banach-space · 2025-02-25T18:47:30Z

Someone might find that BITCAST_1 is perfect, while someone else might find that it’s not capturing enough information. Or the other way around! I’d personally argue that for a test with three CHECK rules, even VAL_1 or A would be just fine.

I agree with you - for that small example, basic variable names would be totally fine (e.g. VAL_1). But note that that example was just to illustrate the limitations of generate-test-checks.py. When writing that, I had this much more involved example in mind:

https://github.com/llvm/llvm-project/blob/60cc3af0d93ecb8bfc9d6bebc6cbc395df3bb4b6/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-non-atomic.mlir#L85-L120

This is the kind of arguments that concerns me. As you well said, these guidelines are subject to interpretation and opinion so I think it's important that we somewhat define where the "acceptable band" is.

I've added a note on "common sense" in this commit. Perhaps I can extend that to capture your feedback (which I agree with).

…ting) Address comments from Javed

banach-space · 2025-02-26T14:42:59Z

Update 26/2/24

The latest commit addresses comments from @javedabsar1 . Thanks Javed, great suggestions!

website/content/getting_started/TestingGuide.md

kuhar · 2025-02-26T14:55:47Z

website/content/getting_started/TestingGuide.md

+A good rule of thumb is to think of yourself six months from now and ask:
+"What might be difficult to decipher without comments?"
+
+If you expect something to be tricky for future-you, it’s likely to be tricky


Suggested change

If you expect something to be tricky for future-you, it’s likely to be tricky

If you expect something to be tricky for the future-you, it’s likely to be tricky

To me, 'future you' sounds more natural than 'the future you.'

A quick Google search (as a non-scientific way to gauge popularity) also suggests that 'future you' is more commonly used.

But English is my second language, so I’m happy to be corrected :) Perhaps lets see whether anyone else comments?

After I found a few typos I copy-pasted the whole doc to google docs and it gave me blue squiggles under this expression and suggested adding 'the'. I'm no authority either.

"the future-you" sounds odd to me if you haven't already referred to the future-you previously in the sentence.

But if you want to sidestep all this, rephrase to something like "6 months from now" and use more plain, albeit verbose, ways to say this.

kuhar · 2025-02-26T14:56:53Z

website/content/getting_started/TestingGuide.md

+
+* `@{negative_}?maskedload_to_load_{static|dynamic}_{i32|i8}_{all_true|all_false|mixed}`.
+
+#### What if there is no pre-existing style to follow?


I like this section, it aligns pretty close with what I had in mind in one of the comment threads. Thanks!

kuhar · 2025-02-26T14:57:38Z

website/content/getting_started/TestingGuide.md

+
+If the test file you are modifying lacks a clear style and instead has mixed,
+inconsistent styles, try to identify the dominant one and follow it. Even
+better, consider refactoring the file to adopt a single, consistent style —


I wonder if it would be worth spelling out that such refactoring should generally go to a separate PR, or if that's obvious.

This is something I'd leave under-specified.

I agree with you that such changes should be submitted as a separate PR. At least in general. However, there are downsides:

more churn + more PR traffic.

Also, sometimes the noise level is fairly low and sending a dedicated PR might be too much. So, I'd trust our judgement on case-by-case basis. Is that OK? We can always refine later.

This is the generic advice that I added recently - https://llvm.org/docs/Contributing.html#how-to-submit-a-patch

Its focus is saving new contributor's time, but it does not disagree with what you've written here.

… formatting) Address latest comments from Jakub

banach-space · 2025-02-26T16:48:33Z

(2nd) Update 26/2/24

The latest commit addresses the latest comments from @kuhar . Thanks Jakub! 🙏🏻

Btw, apologies for taking my time responding to folks! There's a lot of great suggestion to go over - thank you all! I want to make sure that I carefully parse and incorporate all your great ideas :)

kuhar

LGTM

…otes on formatting) Address comments from Adam

…(adds notes on formatting) Add a note on creating seperate PRs

banach-space · 2025-02-27T10:38:40Z

Update 27/2/24

@adam-smnk , this commit addresses your most recent comments. Thanks for the suggestions 🙏🏻

Also, in this commit I've re-fined the guidelines on creating dedicated "refactoring PRs" (CC @kuhar ). Having re-read that section, it seemed like a useful addition.

kuhar · 2025-02-27T15:32:42Z

website/content/getting_started/TestingGuide.md

+In many cases, it’s best to create a separate PR for test refactoring to reduce
+per-PR noise. However, this depends on the scale of changes — reducing PR
+traffic is also important. Work with reviewers to use your judgment and decide
+the best approach.


This is great!

DavidSpickett

General comment: avoid as many contractions as you can.

This is part of some style guides but I know it's (!) not a universally liked rule. So I'm saying this because personally even I find it slightly clearer when contractions are not used and if you're writing a new document, the cost to remove them is low.

Otherwise, as someone who has spent 0 time in MLIR, I would find this document helpful. It'll have a lot of corner cases of its own but it gives me something to cite to justify my starting points.

website/content/getting_started/TestingGuide.md

DavidSpickett · 2025-02-28T11:27:44Z

website/content/getting_started/TestingGuide.md

+
+If the test file you are modifying lacks a clear style and instead has mixed,
+inconsistent styles, try to identify the dominant one and follow it. Even
+better, consider refactoring the file to adopt a single, consistent style —


This is the generic advice that I added recently - https://llvm.org/docs/Contributing.html#how-to-submit-a-patch

Its focus is saving new contributor's time, but it does not disagree with what you've written here.

website/content/getting_started/TestingGuide.md

DavidSpickett · 2025-02-28T12:13:32Z

website/content/getting_started/TestingGuide.md

+* Use high-level block comments to describe patterns being tested.
+* Think about maintainability - comments should help future developers
+  understand tests at a glance.
+* Comments should assist, not replace reading the code. Avoid over-explaining.


The last part seems like a bad transition, not quite a non-sequitur though. Make that into its own point?

Avoid redundant explanations.

?

This does feel a bit like:

do a lot of this

but don't do too much

got it?

But this is inherent to the process, and will come down to reviewer preference so I think this is just a feature of communication in general.

It's very hard to find the right balance, yes. In fact, one of my goals is to hint that:

Finding the right balance w.r.t. comments might be your most challenging task.

Let me rephrase this a bit to improve the flow. Perhaps I should re-write it altogether, but I need a bit more time to think about it. Alternatively, wait and see how people use these guidelines in practice 🤷🏻

website/content/getting_started/TestingGuide.md

…gGuide (adds notes on formatting) Addressed comments from David (1/N)

banach-space · 2025-02-28T18:40:29Z

Update 28/2/25

Incorporated most of the suggestions from @DavidSpickett , thanks! (see this commit). I've "resolved" the corresponding threads and left the remaining ones for later.

banach-space · 2025-03-02T15:19:22Z

Update 2/3/25

Minor updates to address feedback from David: commit

… TestingGuide (adds notes on formatting) Incorporate suggestions from David (2/N)

javedabsar1

LGTM. Thanks.

DavidSpickett · 2025-03-03T10:58:30Z

Nothing was that misleading, so whatever you do with my remaining suggestions is fine with me.

… Extend TestingGuide (adds notes on formatting) Address the remaining comments from David

banach-space · 2025-03-05T20:18:38Z

Update 5/3/25

This commit addresses the remaining comments from David, thank you!

Extend TestingGuide (adds notes on formatting)

b5a8a35

joker-eph reviewed Feb 19, 2025

View reviewed changes

jpienaar reviewed Feb 19, 2025

View reviewed changes

website/content/getting_started/TestingGuide.md Show resolved Hide resolved

kuhar reviewed Feb 19, 2025

View reviewed changes

adam-smnk reviewed Feb 19, 2025

View reviewed changes

umangyadav reviewed Feb 19, 2025

View reviewed changes

website/content/getting_started/TestingGuide.md Outdated Show resolved Hide resolved

banach-space added 2 commits February 20, 2025 11:09

fixup! Extend TestingGuide (adds notes on formatting)

aa27c9d

Fix formatting, add paragraph on what to do when there's no (good) pre-exisiting style to follow.

fixup! fixup! Extend TestingGuide (adds notes on formatting)

f362ae2

Add a section on documentaiton

banach-space force-pushed the andrzej/add_gudelines_testing branch from 8d25ae8 to f362ae2 Compare February 20, 2025 11:21

banach-space mentioned this pull request Feb 21, 2025

[mlir][vector] Add a check to ensure input vector rank equals target shape rank llvm/llvm-project#127706

Merged

dcaballe reviewed Feb 22, 2025

View reviewed changes

website/content/getting_started/TestingGuide.md Outdated Show resolved Hide resolved

website/content/getting_started/TestingGuide.md Outdated Show resolved Hide resolved

fixup! fixup! fixup! Extend TestingGuide (adds notes on formatting)

6f7b769

Fix typos/formatting, add paragraph on common sense

adam-smnk reviewed Feb 24, 2025

View reviewed changes

javedabsar1 reviewed Feb 24, 2025

View reviewed changes

dcaballe mentioned this pull request Feb 26, 2025

[mlir][vector] VectorLinearize: ub.poison support llvm/llvm-project#128612

Open

fixup! fixup! fixup! fixup! Extend TestingGuide (adds notes on format…

3aa153c

…ting) Address comments from Javed

kuhar reviewed Feb 26, 2025

View reviewed changes

fixup! fixup! fixup! fixup! fixup! Extend TestingGuide (adds notes on…

79c6f22

… formatting) Address latest comments from Jakub

kuhar approved these changes Feb 26, 2025

View reviewed changes

dcaballe approved these changes Feb 26, 2025

View reviewed changes

fixup! fixup! fixup! fixup! fixup! fixup! Extend TestingGuide (adds n…

e64268c

…otes on formatting) Address comments from Adam

banach-space force-pushed the andrzej/add_gudelines_testing branch from 737ddf9 to e64268c Compare February 27, 2025 10:22

fixup! fixup! fixup! fixup! fixup! fixup! fixup! Extend TestingGuide …

5240dda

…(adds notes on formatting) Add a note on creating seperate PRs

kuhar reviewed Feb 27, 2025

View reviewed changes

adam-smnk approved these changes Feb 28, 2025

View reviewed changes

DavidSpickett reviewed Feb 28, 2025

View reviewed changes

fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Extend Testin…

8430470

…gGuide (adds notes on formatting) Addressed comments from David (1/N)

fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Extend…

0e6e62f

… TestingGuide (adds notes on formatting) Incorporate suggestions from David (2/N)

banach-space force-pushed the andrzej/add_gudelines_testing branch from f9fafae to 0e6e62f Compare March 2, 2025 15:22

banach-space mentioned this pull request Mar 2, 2025

[mlir] Testing infra improvements ideas llvm/llvm-project#129443

Open

javedabsar1 approved these changes Mar 3, 2025

View reviewed changes

fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup!…

ddb155d

… Extend TestingGuide (adds notes on formatting) Address the remaining comments from David


		When adding new tests, strive to follow these two key rules:

		1. Follow the existing naming and whitespace style.

		/// The permutation map was replaced with vector.transpose
		// CHECK-NOT: permutation_map

	If you expect something to be tricky for future-you, it’s likely to be tricky
	If you expect something to be tricky for the future-you, it’s likely to be tricky


		* `@{negative_}?maskedload_to_load_{static\|dynamic}_{i32\|i8}_{all_true\|all_false\|mixed}`.

		#### What if there is no pre-existing style to follow?

Extend TestingGuide (adds notes on formatting) #216

Are you sure you want to change the base?

Extend TestingGuide (adds notes on formatting) #216

Conversation

banach-space commented Feb 19, 2025

joker-eph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpienaar left a comment

Choose a reason for hiding this comment

rengolin commented Feb 19, 2025

kuhar Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Proposed Levels of Consistency

Choose a reason for hiding this comment

kuhar Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

kuhar Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banach-space commented Feb 20, 2025

dcaballe commented Feb 22, 2025

banach-space commented Feb 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banach-space Feb 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javedabsar1 commented Feb 24, 2025

dcaballe commented Feb 25, 2025

banach-space commented Feb 25, 2025

banach-space commented Feb 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banach-space commented Feb 26, 2025 • edited Loading

kuhar left a comment

Choose a reason for hiding this comment

banach-space commented Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

DavidSpickett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banach-space commented Feb 28, 2025 • edited Loading

banach-space commented Mar 2, 2025 • edited Loading

javedabsar1 left a comment

Choose a reason for hiding this comment

DavidSpickett commented Mar 3, 2025

banach-space commented Mar 5, 2025

kuhar Feb 19, 2025 •

edited

Loading

kuhar Feb 25, 2025 •

edited

Loading

kuhar Feb 19, 2025 •

edited

Loading

banach-space Feb 26, 2025 •

edited

Loading

banach-space commented Feb 26, 2025 •

edited

Loading

banach-space commented Feb 27, 2025 •

edited

Loading

banach-space commented Feb 28, 2025 •

edited

Loading

banach-space commented Mar 2, 2025 •

edited

Loading