Skip to content

Latest commit

 

History

History
475 lines (338 loc) · 23.7 KB

Exerciser.md

File metadata and controls

475 lines (338 loc) · 23.7 KB

PCIe EXERCISER DOCUMENTATION

This document gives details of the various PCIe capabilities that exerciser device supports and how the exerciser is supposed to behave.

Contents

  • Introduction to PCIe Exerciser Endpoint Device

  • Execution model and flow control for following PCIe Features:

    • DMA Transaction
    • MSI Generation and Service
    • Legacy Interrupt Generation and Service
    • ATS Transaction
    • Transaction Monitoring

Introduction to PCIe Exerciser Endpoint Device

PCIe Exerciser is a client device wrapped up by PCIe Endpoint. This device was created to meet BSA (Base System Architecture) requirements for various PCIe capability validation tests. Exerciser's BAR0 space contains following control and support registers to support various PCIe capabilities in exerciser:

 Register Name                   Description                                       Address offset
 -------------                   -----------                                       --------------

 MSI Control Register            Configure and trigger MSIs.                             0x0

 Legacy Interrupt Control        Configure and trigger legacy interrupts.                0x4
 Register

 DMA Control Register            Configure and trigger DMA transactions(DMA_CTL_REG)     0x8

 DMA Offset Register             Configure DMA transaction access offset (in bytes)      0xc
                                 from base of exerciser memory.

 Bus Address Register            Configure bus address that is destination/source        0x10
                                 for DMA or untranslated input address for ATSRequest.

 DMA Length Register             Configure DMA transaction length.                       0x18

 DMA Status Register             Status of last DMA transaction.                         0x1c

 PASID Value Register            Store the pasid value passed for initiating DMA/ATS     0x20
                                 transaction.

 ATS Control Register            Configure and trigger ATS transaction(ATS_CTL_REG)      0x24

 ATS Reply Translated Address    Hold the translated address returned from an            0x28
 Register                        ATSRequest.

 ATS Translated Range Size       Size of the ATS translated address region in bytes.     0x30
 Register

 ATS Reply Permission            Permissions returned for translated memory address      0x38
 Register                        region as a result of ATSRequest.

 Illegal Requestor ID            This register is DANGEROUS, as it will cause illegal    0x3c
 Control Register                transactions to be generated by exerciser.
                                 Use case is when SW wants exerciser to issue a
                                 transaction with custom/illegal requestor id,
                                 rather than the one determined through PCI enumeration.

 Transaction Trace Database      Holds information about the config and memory           0x40
 Register                        transactions. Each read will return one 32 bit entry
                                 previously recorded.

 Transaction Trace Control       Control bit to start/stop transaction monitoring.       0x44
 Register


Execution model and flow control

DMA Transaction

 DMA Control Register  Bit     Description                                   r/w     Value at reset
 --------------------  ---     -----------                                   ---     --------------

 dmatxntrig            3:0     Trigger DMA for given configuration           RW           0h
                               Set to 1h to trigger DMA. This is reset
                               to 0h after transaction completes.
                               values 2h – Fh are reserved.

 Dmatxndir             4       DMA transaction type 0b : Read, 1b: Write     RW           0b

 dmatxnsnoop           5       Snoop attribute for DMA transaction           RW           0b
                               0: Snoop
                               1: No-snoop

 dmapasiden            6       PASID Enable bit                              RW           0b
                               0: PASID disabled
                               1: PASID enabled

 dmaIsPrivileged       7       Is Privileged access mode bit                 RW           0b
                               0: User access mode
                               1: Privileged access mode
                               (dmapasiden bit must be 1)

 dmaIsInstruction      8       Is Instruction access bit                     RW           0b
                               0: Data access
                               1: Instruction access (dmapasiden bit
                               must be 1)

 dmaUseATCforTranslation  9    Use ATC for input address translation         RW           0b
                               0: do no translation mapping.
                               1: If true then, if DMA input address is
                               in ATC input address range, then DMA address
                               will be mapped to output translated address
                               as per translation response of last ATSRequest

 dmaAddressType      11:10     DMA bus address type                          RW           0h
                               0: Default (treated as Untranslated)
                               1: Untranslated
                               2: Translated
                               3: Reserved
                               Note: Any transaction with Reserved address
                               type will fault by logging error in UR bit in
                               device status register in upstream rootport.
                               DMA status register will also reflect internal
                               error.

 dmactlres          31:12      Reserved                                      RO           0h


 DMA Status Register  Bit      Description                                  r/w       Value at reset
 -------------------  -- -     -----------                                  ---       --------------

 dmastatus            1:0      DMA error codes                               RO           0h
                               0h: No error. DMA successful
                               1h: DMA range out of bounds
                               2h: DMA internal error
                               3h: Reserved

 dmaStatusClear       2        Write 1 to clear DMA status                   WO           0b

 dmastatres          31:3      Reserved                                      RO           0h


 Pasid Value Register    Bit       Description                              r/w       Value at reset
 --------------------    ---       -----------                              ---       --------------

 PASIDValue              19:0      PASID value for DMA/ATSRequest            RW           0h

 PASID_Reserved          31:20     Reserved                                  RO           0h


 DMA Requestor ID Register  Bit     Description                              r/w      Value at reset
 ------------------------   ---     -----------                              ---      --------------

 RequestorID               15:0     Custom Requestor ID                      RW           0h
                                    for DMA(ReqIDValid must be 1)

 ReqIDReserved             30:16    Reserved                                 RO           0h

 ReqIDValid                31       Requestor ID Valid bit                   RW           0b
                                    0: Requestor ID is not valid
                                    1: Requestor ID passed is valid

HOW TO USE DMA

  • Before triggering DMA all the required DMA attribute fields like DMA bus address, dmatxnsnoop etc., fields should be correctly set in the following registers:

    • DMA Control Register
    • Bus Address Register
    • DMA Length Register
    • DMA Requestor ID Register
    • DMA Transaction Offset Register
    • PASID Value Register
  • If user wants to do a Pure Translated transaction for a virtual address which is in ATC's input address range then dmaUseATCforTranslation bit must be set to true, so that input virtual address will be mapped to translated address as per ATC fields and translated address will go on the bus for DMA.

  • User can set custom requestor ID for an DMA transaction by using Illegal Requestor ID Control Register. Note: This is illegal as per PCIe specification as requestor ID should be assigned during PCIe enumeration.

  • If passed address for DMA transaction is already translated one, then dmaAddressType must be set to translated. Note: This bypasses the ATC and so is also illegal as per PCIe specification as only a transaction translated by the ATC is allowed to go out on the bus as a Translated Transaction. Also, if dmaAddressType is set to translated and dmaUseATCforTranslation bit is set, then exerciser will fault by logging an error message while initiating DMA, as translation mapping of an already translated address might go wrong.

  • DMAPasidEn bit must be set to 1 when a valid PASID is passed for DMA transaction. If dmaIsPrivileged or dmaIsInstruction is 1, then DMAPasidEn bit must be set to 1 for correct DMA response.

  • Status of the last DMA request can be known from DMA Status Register.

MSI Generation and Service

 MSI Control Register   Bit           Description                       r/w      Value at reset
 --------------------   ---           -----------                       ---      --------------

 MSICTLID               10:0          Index of MSI to be triggered       RW         0h
                                      (0 to 2047)

 MSICTLRES              30:11         Reserved                           RO         0h

 MSICTLTRG              31            Set to 1 to trigger MSI.           RW         0b
                                      This bit is reset after MSI
                                      is sent.

HOW TO TRIGGER MSI:

  • Poll until (MSI Control Register).MSICTLTRG == 0.
  • Write MSI Control Register with MSICTLTRG == 1 and MSICTLID == desired MSI Index to initiate MSI from exerciser device.

Legacy Interrupt Generation and Service

 Legacy Interrupt Control Register  Bit      Description                    r/w    Value at reset
 ---------------------------------  ----     -----------                    ---    --------------

 intxctltrg                         0        Trigger legacy interrupt       RW      0b
                                             Set to 1 to trigger
                                             interrupt Reset this bit
                                             to clear interrupt

 intxctlres                        31:1      Reserved                       RO      0h

HOW TO TRIGGER LEGACY/PIN INTERRUPT:

  • Write 1 into intxctltrg to trigger legacy pin interrupt from exerciser device.

ATS Transaction

 ATS_CTL_REG                 Bit          Description                      r/w    Value at reset
 -----------                 ---          -----------                      ---    --------------

 ATSRequestTrigger           0            Trigger ATS Request for given    WO        0b
                                          configuration Set to 1h to
                                          trigger DMA. This gets reset to
                                          0h after transaction completes.

 ATSRequestIsPrivileged      1            Access mode for ATS Request is   RW        0b
                                          privileged or not.
                                          0: user access
                                          1: Privileged access
                                          (ATSPasidenabled bit must be 1)

 ATSRequestNoWriteRequested  2            Read permission requested for    RW        0b
                                          ATS translated address region
                                          0: Read & Write permission
                                             requested
                                          1: Only Read permission requested

 ATSPasidEnabled             3            PASID Enable bit for ATSRequest  RW        0b
                                          0: PASID disabled
                                          1: PASID enabled

 ATSExcutePermissionRequested  4          Is Execute permission Requested  RW        0b
                                          or not:
                                          0: execute permission not
                                             requested
                                          1: execute permission Requested
                                            (ATSPasidenabled bit must be 1)

 ATSInvalidate/ClearATC      5            Write 1 to clear ATC and         W1C       0b
                                          last ATS translated Results.

 ATSIsInFlight               6            ATS Request is in flight or not  RO        0b
                                          0: ATS Request is not in flight
                                          1: ATS Request is in flight

 ATSTranslationStatus        7            ATS Translation status           RO        0b
                                          0: Translation failed
                                          1: Translation Successful

 ATSTranslationCacheable     8            ATS Translation is cacheable     RO        0b
                                          or not.
                                          0: Not Cacheable, means
                                             RW permission is 0
                                          1: Cacheable, RW!=2’b00.

 ATCInvalidated              9            Whether ATC was invalidated      RO        0b
                                          or not.
                                          0: ATC not invalidated
                                          1: ATC was invalidated

 ATSReserved              31:10           Reserved                         RO        0h



 ATSPermissionReg         Bit            Description                      r/w    Value at reset
 ----------------         ---            -----------                      ---    --------------

 ExecutePermission         0             Whether execute permission is    RO         0b
                                         given for translated address or
                                         not
                                         0: Execute permission not given
                                         1: Execute permission given

 WritePermission           1             Whether write permission is      RO         0b
                                         given for translated address
                                         or not.
                                         0: Write permission not given
                                         1: Write permission given

 ReadPermission            2             Whether Read permission is       RO         0b
                                         given for translated address
                                         or not
                                         0: Read permission not given
                                         1: Read permission given

 ExcutePrivileged          3             If 1 Execute permission was      RO         0b
                                         given when ATSRequest was
                                         privileged.

 WritePrivileged           4             If 1 Write permission was        RO         0b
                                         given when ATSRequest was
                                         privileged.

 ReadPrivileged            5             If 1 Read permission was given   RO         0b
                                         when ATSRequest was privileged.

 ReplyPermissionReserved  31:6           Reserved                         RO         0h

HOW TO USE ATS:

  • Before starting an ATS request, untranslated input address for ATSRequest must be written onto Bus Address Register, with all related bit (is_read, etc.) set in ATS_CTL_REG. SW should enable ATS and PASID capability for exerciser (which will set exerciser_ats_enabled and exerciser_ats_pasid_enabled, where exerciser_ats_enabled will enable client side ATS Transaction capability for exerciser and exerciser_ats_pasid_enabled will enable PASID based ATSRequest for exerciser). Then ATS request can be triggered. Registers to be used for ATS:

    • ATS Control Register
    • PASID Value Register
    • ATS Reply Translated Address Register
    • ATS Reply Permission Register
    • ATS Translated Range Size Register
  • If there is any fault occurred during the translation, then error log will get reflected in command line as MSG_INFO and status of ATSRequest remains as 0 (failed - by default). User can detect whether error has occurred during the translation when following condition is true:

    • ATSRequest is not in flight (ATS_CTL_REG[6] == 0) and ATS transaction status bit (ATS_CTL_REG[7]) is 0.
  • If ATSRequest response result is cacheable (RW!= 2'b00 : Read/Write permission returned as ATSRequest result), then ATS response cacheable bit (ATS_CTL_REG[8]) is set to true.

  • If a matching ATC invalidation comes while ATS request is in flight, translation will be retried, discarding the current translation result.

  • If there is no error and translation response is cacheable, then translation result will be stored in ATC along with permissions returned.

  • Translation result will also get reflected in ATS Reply Translated Address Register, ATS Reply Permission Register, ATS Translated Range Size Register

  • DMA transaction's range must be fully covered by the ATC entry or it will fail.

  • Whenever any DMA transaction want to use ATC based translated address, it must set dmaUseATCforTranslation flag (DMA_CTL_REG[11]) to true, so that translated address will go into the bus after being remapped to translated output address if it has all the necessary read/write permissions.

  • Currently exerciser can store maximum 1 ATS request result, mappings are stored in ATC till invalidated by SMMU or user writes 1 to clear ATC bit (ATS_CTL_REG[5]) which will clear/invalidate all ATC mappings and fields.

  • ATC invalidation happens in following ways when an ATCInvalidate Request received by the exerciser:

    • If ATC is empty, then there is nothing to be invalidated, ATC invalidation call gets returned immediately.

    • Some form of invalidation will be done when invalidation range contains ATC's untranslated input address and either of the following is true:

      • Invalidation range contains address for which any ongoing ATSRequest is in progress.
      • SubstreamID is valid for both atc_invalidate request and atc_entry and either ATC invalidate's substreamID matches with atc_entry's substreamID or G-Global PASID bit is set.
    • If there is any ongoing ATS translation request with the address that is in atc_invalidation_address_range, then current translation result will be discarded, ATC will be invalidated and then current ATS translation request will be retried again.

    • If any DMA PCIe Translated transaction is using any address which is in atc_invalidation_address_range, then we wait for the DMA transaction to complete and then ATC invalidation is done.

    • If no PCIe Translated transaction is using any address which is in atc_invalidation_address_range, then we invalidate the ATC immediately if invalidation_range intersects/matches with atc_entry's untranslated address.

  • ATC can be invalidated by writing 1 to clear ATC bit (ATS_CTL_REG[5]), which will eventually invalidate ATC and discard all the mappings for last ATSRequest.

  • Note: the following ATS behaviours are illegal under the PCIe specification, but implemented in the model for validation purposes only:

    • reading the ATC entry
    • issuing a translated transaction without using the ATC

Transaction Monitoring

Transaction monitoring capabilities in the exerciser provides the ability to record the incoming transactions, for both config and memory transactions. This includes,

  • config read and write transactions serviced in PCIe endpoints.

  • memory transactions serviced in PCIe endpoint BARs.

  • memory transactions which are forwarded from PCIe endpoint to device(like exerciser).

Each recorded transaction has the following information:

  • a 32 bit entry indicating transaction attributes. It has bitfields for request type, R/W transaction, cfg/mem transaction and transaction byte size(bitfields given below)

  • a 32 bit entry indicating lower 32 bits of config or memory address

  • a 32 bit entry indicating upper 32 bits of config or memory address

  • a 32 bit entry indicating lower 32 bits of transaction data read or written to

  • a 32 bit entry indicating upper 32 bits of transaction data read or written to

Note: Each beat in a burst transaction is recorded as a single separate transaction.

               ________________________________
              |         TX ATTRIBUTES          |
              |      CFG/MEM ADDRESS_LO        |
              |      CFG/MEM ADDRESS_HI        |
              |           DATA_LO              |
              |           DATA_HI              |
              |________________________________|

                Recorded Information Format

Registers specific to transaction monitoring

 TransactionTraceReg         Bit          Description                      R/W    Value at reset
 -------------------         ---          -----------                      ---    --------------
  Transaction Info.          31:0         Returns one 32 bit entry
                                          mentioned above on each          RO      0xFFFFffff
                                          read.

  For transaction attributes (TX_ATTRIBUTES), lower 16 bits indicates the type, R/W and cfg/mem information.
  Upper 16 bits of transaction attributes indicates transaction byte size. The bit indicating the
  transaction byte size is calculated as log2(bytes). For example, if size is 8, bit3 of the upper 16 bits
  will be set.
  ------------------------------------------------------------------------------------------------
           31:16            15   14   13   12   11   10   9   8   7   6   5   4   3   2   1   0
  ------------------------------------------------------------------------------------------------
  | Transaction byte size  |                Reserved                                 C/M  R/W  Type

  Bit0: Request type(defaults to zero).
  Bit1: 1 for Read and 0 for Write.
  Bit2: 1 for CFG transaction and 0 for MEM transaction.
  Bits 31-16: Log2 of transaction size, in bytes.


 TransactionTraceControlReg  Bit          Description                      R/W    Value at reset
 --------------------------  ---          -----------                      ---    --------------
 Start/stop monitoring        0           Control bit to start or stop
                                          transaction monitoring.          RW          0x0

HOW TO USE TRANSACTION MONITORING:

  • Number of transactions recorded depends on the exerciser parameter 'max_transaction_trace_entries'. The maximum entries supported is 32, defaults to 16, minimum 1.
  • To start transaction monitoring, write 0x1 to transaction trace control register(offset 0x44).
  • To stop transaction monitoring, write 0x0 to transaction trace control register(offset 0x44).
  • Read the transaction trace database register(offset 0x40). Each read returns one 32 bit entry mentioned above, starting from first transaction. A value of 0xFFFFffff indicates invalid entry.

Copyright (c) 2023, Arm Limited and Contributors. All rights reserved.