Add Timebeat TimeCard Mini PCIe Card

[geerlingguy]

Timebeat's TimeCard mini PCIe card sits in a server's PCIe slot, exposing two serial interfaces (one to an optional Raspberry Pi CM4, one to an optional GPS unit), and the card also holds a TXCO (Temperature Controlled Crystal Oscillator) or OXCO (Oven-Controlled Crystal Oscillator) to hold time very accurately if the signal is lost.

[geerlingguy]

I have a couple versions of the card now, and am planning on a project using them at LTX23...

[geerlingguy]

For bringup, a few notes:

• It seems that power is only 12v through the PCIe pins right now, not sure if I can power it through USB-C on the back, or GPIO header. I tried powering with the 3A 5V power supply into the two ports on the back, and it looks like there was a momentary blip when power was sent through (my USB device blinked), but then I saw 0.0W of power used, so it's like something was protecting backfeeding of power.
  • For now, I'm plugging the card into a CM4 IO board and powering it that way :D
• The instructions for USB2/3 setup here currently seem to only include steps for eMMC CM4 users. The steps would need altering for users of Lite CM4. Going to ask about that.
• There's a set of dipswitches on the PCIe card itself (1-8) which control eMMC boot behavior (pin 1), and some routing for things like serial console access from the host system. See this document for more details
• I downloaded the official Raspberry Pi CM4 OS image from Timebeat, which seems to be based on the February 2023 Pi OS base image. I tried pi/raspberry for the login, but that is apparently not it. I'm not sure if this image is an officially supported way to manage things in production, or just a 'here's a helpful image with stuff preinstalled, but build your own for production' type image.

[geerlingguy]

More notes:

• The default user/pass is admin / 1234
• Out of the box, the date I got is Thu 8 Jun 14:24:28 BST 2023, so SSL certs for many things were failing.

admin@raspberrypi:~ $ timedatectl status
               Local time: Thu 2023-06-08 13:20:35 BST
           Universal time: Thu 2023-06-08 12:20:35 UTC
                 RTC time: n/a
                Time zone: Europe/London (BST, +0100)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

To quickly grab NTP time just to move forward, I ran:

sudo timedatectl set-timezone America/Chicago
sudo timedatectl set-ntp true  # then set to `false` again to turn it back off

Then I moved along to test out some interfaces.

• To control a Blinkstick Nano plugged into the USB type-A port, I ran sudo pip3 install blinkstick, but then ran into Error in command line tool and linux issues arvydas/blinkstick-python#73, so I cloned from github instead, and installed pyusb (sudo pip3 install pyusb)
• I could not see the blinkstick; lsusb only lists the root hub, and the script did not let me control the blinkstick.
• I then plugged in a mouse, and could not see that either. I then noticed this Enabling USB2&3 controller documentation, and am trying that now.
• To enable USB via the VL805 chip, I updated the CM4 eeprom and added VL805=1 to the boot.conf file at the appropriate time.

[geerlingguy]

After enabling USB 2/3, I can now see all the devices:

admin@raspberrypi:~ $ lsusb
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 093a:2510 Pixart Imaging, Inc. Optical Mouse
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And plugging in the BlinkStick works:

Bus 001 Device 004: ID 20a0:41e5 Clay Logic BlinkStick

And Blinkstick works too:

admin@raspberrypi:~/blinkstick-python/bin $ sudo ./blinkstick --info
Found device:
    Manufacturer:  Agile Innovative Ltd
    Description:   BlinkStick Nano
    Variant:       BlinkStick Nano
    Serial:        BS018916-3.0
    Current Color: #000000
    Mode:          3
    Info Block 1:  
    Info Block 2:

[geerlingguy]

I'm also going to try out GPIO, using an Adafruit 240x240 TFT display.

First, install CircuitPython on the Pi:

sudo pip3 install --upgrade adafruit-python-shell
wget https://raw.githubusercontent.com/adafruit/Raspberry-Pi-Installer-Scripts/master/raspi-blinka.py
sudo python3 raspi-blinka.py

Check that I2C and SPI interfaces are present after reboot:

admin@raspberrypi:~ $ ls /dev/i2c* /dev/spi*
/dev/i2c-1  /dev/i2c-20  /dev/i2c-21  /dev/spidev0.0  /dev/spidev0.1

Install the Python libraries required to drive the display:

pip3 install adafruit-circuitpython-rgb-display
pip3 install --upgrade --force-reinstall spidev

Install some font / rendering libraries useful for display:

sudo apt install fonts-dejavu python3-pil python3-numpy

Then I tested a few of Adafruit's examples, like their CPU/system stats example, and their Pillow Animated Gif player. All seemed to work okay.

[geerlingguy]

I also noticed there was a timebeat.service running on boot, and the logs were full of the following messages, every second:

Jul 24 11:00:46 raspberrypi timebeat[539]: ts,1969-12-31 18:26:54.000000003 -0600 CST,pps:eth0/0/0,primary,-3ns,0s,false
Jul 24 11:00:46 raspberrypi timebeat[539]: adj,eth0,-3ns,-3ns,3042

I'm guessing this is a PPS output that is running, I haven't yet measured anything off the outputs on the timebeat board, and I also haven't tried getting a GPS signal yet. Still just getting familiar with the hardware interfaces.

[geerlingguy]

Some checking on the U-blox GPS hardware (it should offer a serial UART on serial0 I think):

admin@raspberrypi:~ $ ls -l /dev/serial*
lrwxrwxrwx 1 root root 5 Jul 24 13:56 /dev/serial0 -> ttyS0
lrwxrwxrwx 1 root root 7 Jul 24 13:56 /dev/serial1 -> ttyAMA0

Observing the output of the module directly:

admin@raspberrypi:~ $ sudo cat /dev/serial0
$GNTXT,01,01,01,NMEA unknown msg*46
$GNTXT,01,01,01,NMEA unknown msg*46
$GNTXT,01,01,01,NMEA unknown msg*46
$GNTXT,01,01,01,NMEA unknown msg*46
$GNTXT,01,01,01,NMEA unknown msg*46
$GNRMC,204657.00,V,,,,,,,240723,,,N*61

This indicates that at least in my basement, I'm not getting any position data (heh, underground, inside... of course not). The NMEA Analyser is quite handy here.

I might also be able to use the Python gps library (sudo apt install python-gps) to poll the data and see if a satellite is acquired? See this example.

The NMEA Reference Manual has been extremely helpful learning the ins and outs of everything...

[geerlingguy]

Next tasks:

• Make sure GPS time sets the system clock when it is acquired (I can see it in the $GNRMC output).
• Make sure OXCO is being used for holdover (it's a SiTime X00711037)

It looks like right now the OXCO is not being used:

admin@raspberrypi:~ $ dmesg | grep clocksource
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[    0.025350] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.150995] clocksource: Switched to clocksource arch_sys_counter

admin@raspberrypi:~ $ cat /sys/devices/system/clocksource/clocksource*/available_clocksource
arch_sys_counter

[geerlingguy]

Out of the box, the CM4 image from Timebeat has the primary time source configured as PTP over eth0 inside /etc/timebeat/timebeat.yml (here's a reference for that file):

timebeat:
  clock_sync:
    adjust_clock: true
    primary_clocks:
      - protocol:                 ptp
        domain:                   10
        serve_multicast:         true
        server_only:             true
        announce_interval:        1
        sync_interval:            0
        delayrequest_interval:    0
        interface:               eth0
      - protocol:     pps
        interface:    eth0
        pin:          0
        index:        0
        utc_offset:   37
    secondary_clocks:
      - protocol:     timecard-mini
        device:       '/dev/serial0'
        baud:         9600

That seems to have been causing the 115% CPU load while it was running (and times were always showing as 1969 since it wasn't getting any time data there).

I decided to move the timecard-mini config up from secondary_clocks to primary, and restarted the timebeat service, and now it seems like times are coming through and the system clock is accurate:

Jul 25 13:14:44 raspberrypi timebeat[1717]: ts,2023-07-25 13:14:44.996408177 -0500 CDT,timecard-mini:/dev/serial0,secondary,3.591823ms,0s,false
Jul 25 13:14:44 raspberrypi timebeat[1717]: adj,system,3.591823ms,1.204µs,61382
Jul 25 13:14:46 raspberrypi timebeat[1717]: ts,2023-07-25 13:14:46.00457462 -0500 CDT,timecard-mini:/dev/serial0,secondary,-4.57462ms,0s,false
Jul 25 13:14:46 raspberrypi timebeat[1717]: adj,system,-4.57462ms,-2.829405ms,-301090

I also tried setting the output to false so it does not sit there writing a log line every few seconds forever (not great for microSD cards), but when I did that, the service had trouble starting. So I set it back to true.

[geerlingguy]

I was also wondering how to get the SiTime OXCO working with the U-blox GPS module, and right now at least using Timebeat's software, that configuration is not working—you could probably program the U-blox to use it manually, but we're getting into the arcane territory where I start remembering AT commands and how crazy fun cellular modem control is...

So for now I'll work with the GPS module's 100 ppb accurate internal oscillator, and let the little 5 ppb SiTime module bake its heart out, not to be used for the time being.

[geerlingguy]

I've put all the code I'm using for this card (along with detailed setup instructions) in this repo for LTX 2023: https://github.com/geerlingguy/ltx2023

[geerlingguy]

It's up! https://pipci.jeffgeerling.com/boards_cm/timebeat-timecard-mini-pcie.html