Reporting compile errors in GitHub Actions

[jgfoster]

I have some code that compiles and runs on my workstation but not in GitHub. In order to verify the output problem, I created a simple failing test with foo; as the only command in a function. Following is my local output with a nice, useful description of the error:

undle exec arduino_ci.rb --testfile-select filesystem.cpp
                           .                  __  ___
                   _, ,_  _| , . * ._   _    /  `  | 
                  (_| [ `(_] (_| | [ ) (_)   \__. _|_   v1.3.0

Host OS...                                                                   osx
Located arduino-cli binary...                         /Users/jfoster/arduino-cli
Found libraries directory...          /Users/jfoster/Documents/Arduino/libraries
Environment variable CUSTOM_INIT_SCRIPT...                                    ''
Environment variable USE_SUBDIR...                                            ''
Installing library under test...                                               ✓
Library installed at... /Users/jfoster/Documents/Arduino/libraries/TestSomething

================================================================================
Beginning the next phase of testing...                              Unit testing
The set of compilers (1) isn't empty...                                        ✓
Checking g++ version... 
    Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
    Apple clang version 12.0.0 (clang-1200.0.32.29)
    Target: x86_64-apple-darwin19.6.0
    Thread model: posix
    InstalledDir: /Library/Developer/CommandLineTools/usr/bin
...Checking g++ version                                                        ✓
libasan availability for g++...                                             true
Library conforms to Arduino library specification...                         1.5
Requested unittest platform 'uno' is defined in 'platforms' YML...             ✓
Configured platforms that match architectures in library.properties... 
    uno
...Configured platforms that match architectures in library.properties          

Unit testing filesystem.cpp with g++ for uno... 

Last command:  $ g++ -std=c++0x -o /Users/jfoster/Documents/Arduino/arduino_ci/SampleProjects/TestSomething/unittest_filesystem.cpp.bin -DARDUINO=100 -g -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize=address -D__AVR__ -D__AVR_ATmega328P__ -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -I/Users/jfoster/Documents/Arduino/arduino_ci/cpp/arduino -I/Users/jfoster/Documents/Arduino/arduino_ci/cpp/unittest -I/Users/jfoster/Documents/Arduino/libraries/TestSomething/src /Users/jfoster/Documents/Arduino/arduino_ci/cpp/arduino/Arduino.cpp /Users/jfoster/Documents/Arduino/arduino_ci/cpp/arduino/Godmode.cpp /Users/jfoster/Documents/Arduino/arduino_ci/cpp/arduino/stdlib.cpp /Users/jfoster/Documents/Arduino/arduino_ci/cpp/unittest/ArduinoUnitTests.cpp /Users/jfoster/Documents/Arduino/libraries/TestSomething/src/test-something.cpp /Users/jfoster/Documents/Arduino/libraries/TestSomething/test/filesystem.cpp

/Users/jfoster/Documents/Arduino/libraries/TestSomething/test/filesystem.cpp:9:3: error: use of undeclared identifier 'foo'; did you mean 'for'?
  foo;
  ^~~
  for
/Users/jfoster/Documents/Arduino/libraries/TestSomething/test/filesystem.cpp:9:6: error: expected '(' after 'for'
  foo;
     ^
2 errors generated.
...Unit testing filesystem.cpp with g++ for uno                                ✗

Following is the output in the GitHub Action:

    file filesystem.cpp (using g++)
Last command: 
========== Stdout:

========== Stderr:

      filesystem.cpp builds successfully (FAILED - 1)
      filesystem.cpp passes tests (PENDING: Can't run the test program because it failed to build)

This is distinctly unhelpful!

[ianfixes]

I'm seeing 2 things here. The GItHub action is running compilation in the context of an rspec unit test(bundle exec rspec --backtrace), and by its design it doesn't print any output because it's expected to pass. That could certainly be changed -- we would want to see the helpful error message at that point as well.

In this case, since the unit tests fail, the full test runner script (cd SampleProjects/TestSomething && bundle exec arduino_ci.rb) is not executed -- which I believe would produce a more helpful error message.

The second thing I'm seeing is a possible mismatch between the behavior on your machine (OSX) and the build agent (Ubuntu). Can you paste the diff of your purposefully-mangled TestSomething/test/filesystem.cpp?

[jgfoster]

A trivial example of an offending filesystem.cpp is as follows:

#include <ArduinoUnitTests.h>

unittest(foo)
{
  foo;
}

unittest_main()

[jgfoster]

While I don't understand the rspec unit test framework, I did find this that suggested the change I made in #289.