Simulated time

[hsaturn]

Hello bxparks

My project (TinyMqtt) unit tests needs to simulate the time.
This helps to

• Debuggin: millis() does not increase making debug session based on time impossible
• Simplify unit testing
• Allow very precise time management for code that needs to deal with delays
• Allow to write unit tests that run quickly for unit tests that make use of delay()

The last point is important, because AUnit fails if unit tests take too long !

I've modified Arduino.* in order to achieve this.
All is working very well for TinyMqtt / EspMock with these modifications.
By the way, thank you for mentioning EspMock in your README, this mock now support network simulation !

Here are the modifications:

diff --git a/cores/epoxy/Arduino.cpp b/cores/epoxy/Arduino.cpp
index 1f196da..387dbda 100644
--- a/cores/epoxy/Arduino.cpp
+++ b/cores/epoxy/Arduino.cpp
@@ -17,6 +17,9 @@
 #include <time.h> // clock_gettime()
 #include "Arduino.h"
 
+unsigned long epoxy_micros;
+bool epoxy_real_time = true;
+
 // -----------------------------------------------------------------------
 // Arduino methods emulated in Unix
 // -----------------------------------------------------------------------
@@ -36,25 +39,94 @@ int analogRead(uint8_t /*pin*/) { return 0; }
 void analogWrite(uint8_t /*pin*/, int /*val*/) {}
 
 unsigned long millis() {
-  struct timespec spec;
-  clock_gettime(CLOCK_MONOTONIC, &spec);
-  unsigned long ms = spec.tv_sec * 1000U + spec.tv_nsec / 1000000UL;
-  return ms;
+  if (epoxy_real_time)
+  {
+    struct timespec spec;
+    clock_gettime(CLOCK_MONOTONIC, &spec);
+    unsigned long ms = spec.tv_sec * 1000U + spec.tv_nsec / 1000000UL;
+    return ms;
+  }
+  else
+  {
+    return epoxy_micros / 1000;
+  }
+}
+
+void set_real_time()
+{
+  epoxy_real_time = true;
+}
+
+void set_seconds(unsigned long s)
+{
+  epoxy_micros = s * 1000 * 1000;
+  epoxy_real_time = false;
+}
+
+void add_seconds(unsigned long s)
+{
+  epoxy_micros += s * 1000 * 1000;
+  epoxy_real_time = false;
+}
+
+void set_millis(unsigned long ms)
+{
+  epoxy_micros = 1000 * ms;
+  epoxy_real_time = false;
+}
+
+void add_millis(unsigned long delta_ms)
+{
+  epoxy_micros += 1000 * delta_ms;
+  epoxy_real_time = false;
+}
+
+void set_micros(unsigned long us)
+{
+  epoxy_micros = us;
+  epoxy_real_time = false;
+}
+
+void add_micros(unsigned long delta_us)
+{
+  epoxy_micros += delta_us;
+  epoxy_real_time = false;
 }
 
 unsigned long micros() {
-  struct timespec spec;
-  clock_gettime(CLOCK_MONOTONIC, &spec);
-  unsigned long us = spec.tv_sec * 1000000UL + spec.tv_nsec / 1000U;
-  return us;
+  if (epoxy_real_time)
+  {
+    struct timespec spec;
+    clock_gettime(CLOCK_MONOTONIC, &spec);
+    unsigned long us = spec.tv_sec * 1000000UL + spec.tv_nsec / 1000U;
+    return us;
+  }
+  else
+  {
+    return 1000*epoxy_micros;
+  }
 }
 
 void delay(unsigned long ms) {
-  usleep(ms * 1000);
+  if (epoxy_real_time)
+  {
+    usleep(ms * 1000);
+  }
+  else
+  {
+    epoxy_micros += 1000 * ms;
+  }
 }
 
 void delayMicroseconds(unsigned int us) {
-  usleep(us);
+  if (epoxy_real_time)
+  {
+    usleep(us);
+  }
+  else
+  {
+    delay(1000*us);
+  }
 }
 
 unsigned long pulseIn(
diff --git a/cores/epoxy/Arduino.h b/cores/epoxy/Arduino.h
index 6efd1e7..ce2b402 100644
--- a/cores/epoxy/Arduino.h
+++ b/cores/epoxy/Arduino.h
@@ -223,8 +223,18 @@ void analogWrite(uint8_t pin, int val);
   void analogWriteRange(uint32_t range);
 #endif
 
+// Simulated time
+void set_millis(unsigned long);
+void add_millis(unsigned long);
+void set_micros(unsigned long);
+void add_micros(unsigned long);
+void add_seconds(unsigned long);
+void set_seconds(unsigned long);
+void set_real_time();
+
 unsigned long millis();
 unsigned long micros();
+
 void delay(unsigned long ms);
 void delayMicroseconds(unsigned int us);

[bxparks]

I understand the convenience of being able to mock the low-level functions of the Arduino API. But EpoxyDuino was never intended to be a mocking library. It was supposed to emulate the Arduino API as closely as possible on a Unix/Posix system. The thinking was that if some code needs to be unit tested, then it should be designed such that it can be tested within the microcontroller itself, instead of having to rely on mocking functions that exist only in EpoxyDuino. I accepted the digitalReadValue() and digitalWriteValue() functions because they seems innocuous enough, but I'm not sure how I feel about adding a lot more of these mocking functions. I feel like if EpoxyDuino does support them, they should be part of a unified framework, instead of adding each one on an ad hoc basis, one by one. Let me think about this a bit more.

[hsaturn]

Hi Brian

I understand this point of view. I do not really consider time simulation as a mock, but exactly in the 'unit test' spirit that EpoxyDuino was designed for. But yes, this is no more under the microcontroller itself.

Since I use EpoxyDuino, I find this library to be very powerfull and incredibly valuable for debugging some tricky code on the PC with a decent debugger. In fact, I even never unit testing on the microcontroller (maybe I should...).
Simulated time is not in my point of view a mock, but a real feature allowing to control more precisely the time and thus increase the power of the emulation wanted by Epoxy.

In my point of view or less: in my use of your libs

• AUnit allows me to unit test on github, which is a incredible feature
• EpoxyDuino helps me to do that by implementing most functions.
• EspMock helps to mock the WiFi network and allows to run the test within githib

I've forked EpoxyDuino just to add the simulated time and be able to continue to use AUnit on github with the help of EpoxyDuino and EspMock. Of course it's up to you to decide, but honestly, I don't feel comfortable by forking just for that. (I"d prefer my fork to be temporary). I'd prefer to stick to your repo and delete mine.

Of course, I do agree that simulated time won't run on a microcontroller and thus, breaks the rule that unit tests could be ran on the microcontroller also.

I do not have a solution too. Best would be to have a overview on how EpoxyDuino is used. Maybe very few uses Epoxy to run unit tests on the microcontroller.

My guess is that because EpoxyDuino make it possible to run tests on the PC, it is so more comfortable that noone would think to run it on the microcontroller.

My typical use case : I've added many features to TinyMqtt + released without even testing them on the microcontroller. (which maybe not a so good habit :-D)

---

That said, I can agree with you. The choice is hard to make and it's up to you.

Maybe a solution would be replace functions by 'hooks'.

EpoxyDuino sticks to the law : emulate api as close at the Arduino's one.
ExtDuino 'based' on EpoxyDuino uses hooks to change some behaviours.

Something like this (very bad code, just look at the idea behind):
(also bad prototypes, we don't care...)

// EpoxyDuino changes:
long epoxy_millis() {...}     // Default implementation

auto millis = epoxy_millis;   // Definition to millis function

millis is now a ptr on any function that can be overriden in ExtDuino

// ExtDuino -> overrides some implementations

long simulated_millis();    // Able to work as millis or in simulated time
long init_extduino()
{
  millis = simulated_mills;
}

Not a good design (I'd prefer to group functions by theme such as Time, under a class such as EpoxyTime which would be polymorphic, and would allow to override functions.

This way, we would keep EpoxyDuino in its orientation: provides an emulation of the API that is compatible with the microcontroller. And that allows to write unit tests designed for both uController and PC.

And ExtDuino (or EpoxyTimeDuino or whatever) : provides a way to change some implementations. And thus breaks the rule of 'runs on microcontroller'.

Another solution that may enforce to group functions together could be

#include <memory>

long now();
namespace EpoxyDuino {

  class TimeInterface
  {
    public:
      virtual long millis() = 0;
      virtual long micros() = 0;
      virtual void delay(long) = 0;
      virtual void delayMicroseconds(long) = 0;
  };

  class TimeImpl : public TimeInterface
  {
    public:
      long millis() override { return now(); }
      long micros() override { return now(); }
      void delay(long) override {}
      void delayMicroseconds(long) override {}
  };

  std::unique_ptr<EpoxyDuino::TimeInterface> time(new EpoxyDuino::TimeImpl);

}

long millis() { return EpoxyDuino::time->millis(); }
long micros() { return EpoxyDuino::time->micros(); }
void delay(long t) { EpoxyDuino::time->delay(t); }
void delayMicroseconds(long t) { EpoxyDuino::time->delay(t); }

// EXT_DUINO : Add simulated time feature inside EpoxyDuino, (breaking ability to run tests inside the uController)
namespace EpoxyDuinoSimulatedTime {
  class TimeImpl : public EpoxyDuino::TimeInterface
  {
    public:
      long millis() override { return time; }
      long micros() override { return time; }
      void delay(long) override {}
      void delayMicroseconds(long) override {}

    private:
      long time;
  };


  void init_ext_duino()
  {
    EpoxyDuino::time.reset(new EpoxyDuinoSimulatedTime::TimeImpl);
  }
}

To be frankly, I like the first solution because it is very simple to implement at a low cost, and I like the second solution because functions are grouped by theme. But ... I don't really like for some unknown reasons boths.

As you I have to think more about that too.

[hsaturn]

I've just thought about this, but millis() is intended to return the same value while it is called within interrupts. Only simulated time or thing near this could match this. Don't know what to do with this but I had to say.

[bxparks]

My preference is to limit the scope of EpoxyDuino to the Arduino API proper (which I admit is ill defined because each platform implements a slightly different variation of the API). Since the Arduino API does not support injectable functions, we are forced to do it at the application layer. You have already read bxparks/AUnit#83, so you can refer to the techniques listed there:

1)) If your application is OO based, you can use a virtual method on a class to inject the millis(), micro() methods. My AceButton project uses this technique for unit testing. The disadvantage is that each call to millis() is just a bit slower due to the virtual dispatch. My guess is that this overhead does not matter for the vast majority of applications. For those where the performance is important, we can use the next technique.

2)) If your application is C++ templated-based, you can use compile-time polymorphism to inject different versions of millis() and micros(). My AceSegment project uses this technique. The compiler should optimize away all of the template overhead and basically inline everything into direct calls to millis() and micros().

3)) The next option is to "Use a custom version of analogRead() and digitalRead()" as described in bxparks/AUnit#83. There is a slight variation of this that occurs to me right now. You could define a testableMillis() function, and use this everywhere in your app instead of millis(). Instead of using a runtime flag that selects different versions, you could use a compile-time flag to select the implementation, like this:

#if defined(USE_TESTABLE_FUNCTIONS)
unsigned long millisValue;
void testableMillis(unsigned long value) {
  millisValue = value;
}
#endif 

unsigned long testableMillis() {
  #if defined(USE_TESTABLE_FUNCTIONS)
    return millisValue;
  #else
    return ::millis();
  #endif
}

Then in your EpoxyDuino Makefile for your unit test, you add:

EXTRA_CXXFLAGS := -D USE_TESTABLE_FUNCTIONS

---

You could argue that this layer of testable functions should be provided by EpoxyDuino itself. The problem is that a given application may need to inject only 3-4 of these Arduino functions, but EpoxyDuino would need to implement this across the entire Arduino API surface. That's a lot of extra work that I am not motivated to spend. For me, unit testing is a mechanism that allows me to implement interesting apps. This feature falls too much on the side of "doing lots of work to pray to the gods of unit testing".

[hsaturn]

You could argue that this layer of testable functions should be provided by EpoxyDuino itself.
=> Of course I will :-D

This feature falls too much on the side of "doing lots of work to pray to the gods of unit testing".

Using ptr on functions is not such a important work, and little intrusive too.

But I easylly can ear that you do not want to spend this time. But I'm ready to spend this time if you want.

Writing unit tested libraries on Arduino and Esp seems to me to be the greatest value of EpoxyDuino, so I'll keep using my forked repo and will maintain it with your repo.

Thanks a lot for this job. And for your time.

[bxparks]

Good luck! If you can segregate your extensions into separate files, it might be easier to merge them into EpoxyDuino at a later time if we decide to do that. I'm willing consider that option if the separation is clean enough. I can then say to you, ok you're in charge of that part of the code base. :-)

[hsaturn]

Yes I was thinking about extensions segregation into separate files. Not so easy.
Easy for new functions of course (set_millis etc.), but not for others ... millis() has a legacy EpoxyDuino implementation and a different one for the fork.
I'll think about this later, when more needs will arrive. Yet, I have to deal with bugs on TinyMqtt that is used by others.

The first next thing I'll probably do is to distinguish between core functions and extra-core ones.
For simulated set_millis(...) beside millis() seems to belong to core functions while this is absolutely not the case !

I guess I'll put them on a dedicated namespace (CoreInjection, CoreTest, EpoxyTest ... whatever I' m not strong in naming...)

I really want set_millis() not to be confused with millis()
Also I'll wabt to add a compilation flag that explicitely allows those test functions in order to separate legacy role of EpoxyDuino from Testing role...

[bxparks]

I'm not sure I understand your design completely, but my first impression is that it seems to be unexpectedly complicated. Are you proposing to add a new layer indirection API (various C++ classes, in different namespaces), which the application layer is expected to use, instead of the normal Arduino API? I suppose that works for new applications, but that means that existing applications cannot take advantage of it. I am not sure that it is reasonable to ask users to use a completely different API to access normal Arduino functions.

I think it would more useful to use the same Arduino API for all normal Arduino functions and classes. Then expose 2 different implementation of the Arduino functions depending on a macro flag (something like -D EPOXY_USE_MOCKS, or -D EPOXY_TEST_MODE). By default, the Arduino functions are emulated on the Linux/Posix environment. But if the EPOXY_TEST_MODE is given, the Arduino functions become mocks, and the macro also exposes the various setter methods to set and read the values of those mocked functions, e.g. pinModeValue(), digitalWriteValue(), digitalReadValue(), analogReadValue(), analogReferenceValue(), analogWriteValue(), millisValue(), microsValue(), shiftOutValue(), shiftInValue(), pulseInValue(), pulseOutValue(), attachInterruptValue(), etc, etc.

Then there is the issue of the various C++ class: Print, Stream, Client, Udp, SPIClass, TwoWire, etc. Some of those classes were well-designed to allow injection of testable versions of these classes. But I think others use static mutable variables which are a pain-in-the-rear to mock out for testing. For example, I know for a fact that the TwoWire class is terribly designed and cannot be used polymorphically, even though it inherits from Stream and Print. It is unclear how fine-grained you want the mocking feature to be implemented in these various classes.

After you do all that, you then have to document all the additional layers of testing/mocking APIs, explain how they are used, and how they interact with each other.

I think my design is simpler than your design. But this is still far more work than I want to do personally.

[hsaturn]

Hello Brian

I do not have any design now.

My mind is to keep it as simple as possible. This may be a misunderstanding but as I'm not english, I may have used wrong words...

In few words, if the compilation flag is not set, then my EpoxyDuino sould compile exactly the same as your, and will expose exactly the same API (the existing one). But if the flag is set, then some new functions appear, allowing to override some functions.

In the case of my Simulated Time functions, you won't have access to them until the compilation flag is set. Just because the new functions does not belong to the Arduino API. They only appear once the flag is given.

That said, those new functions should not belong to the global namespace for the same reason.
Thus, set_millis(), set_microseconds() which are new and dedicated to test should be declared in a specific namespace because they are not part of the Arduino API.

Another way to explain what I mean is

• No compilation flag : no extension, standard Arduino API
• Compilation flag : a new namespace (e.g. epoxy_test) will contain function that allows extensions.

Yet one can write with my current fork

set_millis(100);
...
a = millis();

In this code, it is not clear that set_millis(100) does not belong to the standard but in this one:

EpoxyTest::set_millis(100);
...
a = millis();

This is a lot more explicit.

---

Now for TwoWire or other that are badly designed, it seems to me easy to bypass the original classes with -DEPOXY_TEST_MODE and wompile another version wich can be easy to extend.

But as I said, I will only extend what is needed for my needs (yet only set_time is usefull).
I also have an idea of queues<delay_ms_or_timepoint, value> for injection of values for functions such as digitalRead() analogRead() etc.. and even why not support some texts scenario such as:

group main
  time 0  -> set_millis(0)
  port A0 123
  wait 10ms
  port A0 127
  wait 10ms
  port A1 231
  port A1 129
endgroup

group A2
  port A2 100
  sync 1s   -> wait until time is one second (or more)
  loop 100
    port A2 100
    loop 10
      port A2++
      wait 1ms
    endloop
   loop 5
     port A2--
     wait 1ms
   endloop
  endloop
endgroup

group A3
  port A3 read port_a3.dat
engroup

group waves
  port A4 square 0-250 10% 1kHz
  port A5 sinus 10-80 100Hz
endgroup

group serial
  wait_serial 0 0x02
  serial 0 0xCAFE0102
  echo "com started"
  wait_serial 0 0x03
  serial 0 0x04
  echo "com end"
endgroup
  
...

[hsaturn]

This commit hsaturn@33bde3e is a first attempt to do this.
Test functions are moved in a namespace EpoxyTest and can be accessed only when -DEPOXY_TEST is set.

[hsaturn]

This commit hsaturn@f221664 is a step toward the ability to write scenarios.
It allows to post some events such as pin value change, Serial change etc...

[bxparks]

The -DEPOXY_TEST injection code at hsaturn@33bde3e seems reasonable.

But your multi-threaded integration testing framework at hsaturn@f221664, with its own custom DSL, is way too much overhead for me. I would not want to maintain something like that. It creates too many layers of abstraction between the code that you are testing and the testing framework. When something goes wrong, you would have difficulty figuring out which layer the problem occurs, and potentially spending more time debugging the testing framework rather than the actual application code.

I clearly don't have the full context of what you are trying to solve, but I don't understand why you can't create a Clock interface for your application, then inject a FakeClock when you want to write your integration tests.

[hsaturn]

Fair enough :-)
I don't expect anymore to merge my Epoxy with yours. More, I think I even could be against the idea. Your design is to mock in order to compile when my goal is to have a complete test framework with external stimulus injection. Forks are dirverging to much and one couldn't understand what is the goal of EpoxyDuino (yours). Kiss principle of your repo give it its strengh.

Simulated time was a real need for testing TinyMqtt. And it was light enough to be integrated to your Epoxy.
My Epoxy now takes another direction. And its fully evident that you do not want to deal with my code.
So thanks for this great repo (with AUnit also) that are your's and that are allowing since once year now to make TinyMqtt converge toward stability and non regression.

Also, Epoxy allowed me to run some Esp code on the PC, more than that, AUnit + Epoxy allowed me to find some very tricky bugs that I couldn't find of fix so strongly. And this is why EspMock was started and allowed me to simulate (not just mock) the wifi network, making once again a lot more debugging power resource.

And this is probably where I'm going now : not just mock everything, but bring more and more emulation. In fact, I guess the goal of my fork could be : total emulation of all Arduino's API, making it possible to take a sketch and run it on a PC. But I already know I won't reach such a goal.

I don't see clearly the idea of Clock vs FakeClock, except that it'd allow to have a cleaner interface. The idea behind my first approach was to keep it as simple as possible and very little intrusive. I may probably redesign this a day, but the goal is reached: have a precise control of time aspect for my unit tests.

Best regards, and thanks once again for the work you've done.

[hsaturn]

Script are available !!!
These commit hsaturn@72774a8 and hsaturn@cef6546

Such a script makes pins of the Arduino change:

pin 1 0     // digitalWrite(1,0)
sync 100ms  // Wait until millis() == 100
pin 1 1     // digitalWrite(1,1)