[WIP] Test framework #539

DigitalSmile · 2025-10-27T10:46:21Z

Pi4J testing framework for the win

Overview

Pi4J lacks proper tests within it's usage. Not only unit in core tests (which are good to test some basic logic), but provider unit, integration and performance tests. Moreover, new FFM API relies on the kernel, that is run within Linux OS. That's essentially means we have to keep an eye on what is changing in kernel driver/syscall development. To keep all this in place, I propose the following framework.

Unit tests

We need to use mock approach to test raw methods, that they works as expected (e.g. Mockito in FFM https://github.com/Pi4J/pi4j/tree/develop/plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/ffm/mocks).
Several layers of unit testing approach:

core unit tests, located at https://github.com/Pi4J/pi4j/tree/develop/pi4j-core/src/test/java/com/pi4j
FFM and any other provider (e.g. https://github.com/Pi4J/pi4j/tree/develop/plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/ffm/unit)

In this scope, we need to remove pi4j-test module and move corresponding unit and integration tests to core.

Todo

Move unit tests in pi4j-test module to pi4j-core, make them runnable
Make mocks in pi4j-core
Add tests coverage metric and write more unit tests

Model tests

Model tests are needed to make sure any interop structure are not broken.

FFM model tests (https://github.com/Pi4J/pi4j/tree/develop/plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/ffm/model)

Same should be applied to any other plugins we offer.

Todo

Add model tests to pi4j-core (if necessary)

Integration tests

This tests should check if interop and syscalls are not broken. Main idea is to write simple mock (or echo) drivers for each provider and test syscalls. I have created some drivers FFM provider within this PR:

This drivers requires a simple build with make and current loaded kernel headers (I have provided comments with source files, though there are really very simple). We can think of expanding this tests to all recent stable linux kernels in future.
Each driver is built and loaded with the test in java and removed upon test completion. This works in linux machines only, so a special annotation is used not to run them on other systems.

Todo

Create SPI driver and fix tests
Create PWM driver and fix tests
Create I2C driver and fix tests
Create GPIO driver and fix tests
Create Serial driver and fix tests

Performance tests

Since we are planning to expand the support from RaspberryPi only to every linux driven SBC, It is very possible, that pi4j will be used in high concurrent and/or highload environment. So we have to control and measure periodically how our changes affects the performance of interop and overall software responsiveness. I suggest to use the same approach as in integration testing - use simple echo/mock drivers to measure how fast the communication go. JMH tests for FFM provider are located here -> https://github.com/Pi4J/pi4j/tree/test-framework/plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/jmh
They include:

GPIO performance tests
PWM performance tests
I2C performance tests
SPI performance tests
Serial performance tests

We have to think on comparing ourselves to other / similar libraries.

Todo

Create FFM API tests for GPIO (regression and performance degradation)
Create FFM API tests for PWM (regression and performance degradation)
Create FFM API tests for SPI (regression and performance degradation)
Create FFM API tests for I2C (regression and performance degradation)
Create FFM API tests for Serial (regression and performance degradation)
Create comparing performance tests
Run tests periodically and publish results (https://github.com/kitlangton/jmh-benchmark-action)

Hardware Tests

TBD

.github/workflows/ci-test.yml

FDelporte · 2025-10-29T07:36:05Z

.github/workflows/ci-test.yml

+      - name: Install JDK25
+        uses: sfesenko/setup-sdkman@v1
+        with:
+          deps: java:25-amzn


25-zulu please :-)

.github/workflows/ci-test.yml

eitch · 2025-11-03T12:03:58Z

.github/workflows/ci-test.yml

+    name: Build Snapshot
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install dependencies
+        # run: sudo apt-get install --yes build-essential linux-headers-$(uname -r) sudo libi2c-dev gcc-arm-linux-gnueabihf  gcc-aarch64-linux-gnu libtool tree
+        run: |
+          sudo apt-get remove --purge man-db
+          sudo apt-get install --yes build-essential linux-headers-$(uname -r) sudo libi2c-dev
+      - name: Install JDK25
+        uses: sfesenko/setup-sdkman@v1
+        with:
+          deps: java:25-zulu
+      - name: Create and add current user to suitable groups
+        run: |
+          sudo addgroup spi
+          sudo adduser $USER spi
+          sudo adduser $USER i2c
+          sudo adduser $USER dialout
+          sudo -u $USER bash -c 'groups'
+      - name: Add proper udev rules and trigger udevadm
+        run: |
+          sudo cp udev.rules /etc/udev/rules.d/99-pi4j-io.rules
+          sudo udevadm control --reload-rules && sudo udevadm trigger
+      - name: Build entire Pi4J Project
+        run: |
+          sudo -u $USER bash -c 'source ~/.sdkman/bin/sdkman-init.sh && ./mvnw clean install -Pnative,cross-compile --batch-mode'


To fix this issue, add an explicit permissions block to the workflow. The best way is to set it at the workflow root level (immediately after the name: line, before on: or jobs:), so that it applies to all jobs unless jobs set their own block. For most CI workflows, the minimal permission necessary is contents: read, which enables jobs to checkout code but prevents writes to repository contents, avoiding privilege escalation. Insert the block immediately after the workflow name: entry, in .github/workflows/ci-test.yml at line 2. No imports or external dependencies are needed.

What does Github want here?

.github/workflows/ci-test.yml

DigitalSmile · 2025-11-02T16:14:35Z

Some other fixes to build process I have done:

removed docker from the workflow, it works fine even with native build (https://github.com/Pi4J/pi4j/blob/test-framework/.github/workflows/ci-test.yml). We can consider this workflow in future updates.
used maven cache to make build faster
created environment to run integration tests, which are calling compiled mock kernel driver (e.g. https://github.com/Pi4J/pi4j/actions/runs/19014192083/job/54299613423?pr=539)

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.pi4j.plugin.ffm.integration.SPITest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.773 s -- in com.pi4j.plugin.ffm.integration.SPITest
[INFO] Running com.pi4j.plugin.ffm.integration.I2CFileTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.624 s -- in com.pi4j.plugin.ffm.integration.I2CFileTest
[INFO] Running com.pi4j.plugin.ffm.integration.PwmTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.236 s -- in com.pi4j.plugin.ffm.integration.PwmTest
[INFO] Running com.pi4j.plugin.ffm.integration.SerialTest
Warning:  Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.001 s -- in com.pi4j.plugin.ffm.integration.SerialTest
[INFO] Running com.pi4j.plugin.ffm.integration.I2CDirectTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.326 s -- in com.pi4j.plugin.ffm.integration.I2CDirectTest
[INFO] Running com.pi4j.plugin.ffm.integration.GPIOTest
Warning:  Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 0.003 s -- in com.pi4j.plugin.ffm.integration.GPIOTest
[INFO] Running com.pi4j.plugin.ffm.integration.I2CSMBusTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.020 s -- in com.pi4j.plugin.ffm.integration.I2CSMBusTest
[INFO] Running com.pi4j.plugin.ffm.model.SPITest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.pi4j.plugin.ffm.model.SPITest
[INFO] Running com.pi4j.plugin.ffm.model.I2CTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.322 s -- in com.pi4j.plugin.ffm.model.I2CTest
[INFO] Running com.pi4j.plugin.ffm.model.PollTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.pi4j.plugin.ffm.model.PollTest
[INFO] Running com.pi4j.plugin.ffm.model.SerialTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.pi4j.plugin.ffm.model.SerialTest
[INFO] Running com.pi4j.plugin.ffm.model.GPIOTest
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 s -- in com.pi4j.plugin.ffm.model.GPIOTest
[INFO] Running com.pi4j.plugin.ffm.unit.SPITest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.225 s -- in com.pi4j.plugin.ffm.unit.SPITest
[INFO] Running com.pi4j.plugin.ffm.unit.PWMTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 s -- in com.pi4j.plugin.ffm.unit.PWMTest
[INFO] Running com.pi4j.plugin.ffm.unit.I2CTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.141 s -- in com.pi4j.plugin.ffm.unit.I2CTest
[INFO] Running com.pi4j.plugin.ffm.unit.GPIOTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.143 s -- in com.pi4j.plugin.ffm.unit.GPIOTest
[INFO] 
[INFO] Results:
[INFO] 
Warning:  Tests run: 70, Failures: 0, Errors: 0, Skipped: 10

…CI config

…tAllocator in I2CMessage and RDWRData

eitch

There are a few points

eitch · 2025-11-03T12:03:58Z

.github/workflows/ci-test.yml

+    name: Build Snapshot
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install dependencies
+        # run: sudo apt-get install --yes build-essential linux-headers-$(uname -r) sudo libi2c-dev gcc-arm-linux-gnueabihf  gcc-aarch64-linux-gnu libtool tree
+        run: |
+          sudo apt-get remove --purge man-db
+          sudo apt-get install --yes build-essential linux-headers-$(uname -r) sudo libi2c-dev
+      - name: Install JDK25
+        uses: sfesenko/setup-sdkman@v1
+        with:
+          deps: java:25-zulu
+      - name: Create and add current user to suitable groups
+        run: |
+          sudo addgroup spi
+          sudo adduser $USER spi
+          sudo adduser $USER i2c
+          sudo adduser $USER dialout
+          sudo -u $USER bash -c 'groups'
+      - name: Add proper udev rules and trigger udevadm
+        run: |
+          sudo cp udev.rules /etc/udev/rules.d/99-pi4j-io.rules
+          sudo udevadm control --reload-rules && sudo udevadm trigger
+      - name: Build entire Pi4J Project
+        run: |
+          sudo -u $USER bash -c 'source ~/.sdkman/bin/sdkman-init.sh && ./mvnw clean install -Pnative,cross-compile --batch-mode'


What does Github want here?

eitch · 2025-11-03T12:05:20Z

plugins/pi4j-plugin-ffm/src/main/java/com/pi4j/plugin/ffm/providers/i2c/impl/I2CFile.java

    @Override
    public int writeRegister(int register, byte b) {
-        return i2CBus.execute(this, (i2cFileDescriptor) -> FILE.write(i2cFileDescriptor, new byte[]{(byte) register, b}));
+        return i2CBus.execute(this, (i2cFileDescriptor) -> FILE.write(i2cFileDescriptor, new byte[]{(byte) register, b})) - 1;


put the -1 in a new statement, so it does not get ignored.

eitch · 2025-11-03T12:05:25Z

plugins/pi4j-plugin-ffm/src/main/java/com/pi4j/plugin/ffm/providers/i2c/impl/I2CFile.java

        buffer[0] = (byte) register;
        System.arraycopy(data, 0, buffer, 1, data.length);
-        return i2CBus.execute(this, (i2cFileDescriptor) -> FILE.write(i2cFileDescriptor, buffer));
+        return i2CBus.execute(this, (i2cFileDescriptor) -> FILE.write(i2cFileDescriptor, buffer)) - 1;


eitch · 2025-11-03T12:05:33Z

plugins/pi4j-plugin-ffm/src/main/java/com/pi4j/plugin/ffm/providers/i2c/impl/I2CFile.java

        System.arraycopy(register, 0, buffer, 0, register.length);
        System.arraycopy(data, 0, buffer, register.length, data.length);
-        return i2CBus.execute(this, (i2cFileDescriptor) -> FILE.write(i2cFileDescriptor, buffer));
+        return i2CBus.execute(this, (i2cFileDescriptor) -> FILE.write(i2cFileDescriptor, buffer)) - register.length;


eitch · 2025-11-03T12:09:28Z

plugins/pi4j-plugin-ffm/src/main/java/com/pi4j/plugin/ffm/providers/i2c/I2CFFMProviderImpl.java

In these checks, isn't it counter intuitive, that the user requested SMBUS, but we don't use it if it isn't supported? Shouldn't we throw an exception?

Thus if the user requests a specific impl, then we should use that and throw exceptions if something isn't as expected.

eitch · 2025-11-03T12:14:45Z

plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/jmh/SPIPerformanceTest.java

-        spi.read(buffer);
-        pi4j.shutdown();
-    }
+//    @Benchmark


eitch · 2025-11-03T12:16:19Z

plugins/pi4j-plugin-ffm/src/test/native/i2c/build.sh

@@ -0,0 +1,7 @@
+#!/bin/sh


I was hoping we could remove building with make from pi4j down the line... is there no better solution?

I have answered you somewhere else - this is needed for testing only, there will be no runtime dependencies and cross-compile builds. It is running during test phase, only in linux with kernel headers as a single dependency. Very primitive and as simple as possible.

Unfortunately there is no "mock driver pack" for each interface, most of existing mock drivers are not the default part of kernel (and in CI we have prebuild custom azure kernel). But I still think it is important to have integrity kernel tests to check changes we make.

eitch · 2025-11-03T12:17:41Z

plugins/pi4j-plugin-ffm/pom.xml

            <scope>test</scope>
        </dependency>
-        <dependency>
+<!--        <dependency>


let's just remove the dependency block all together

eitch · 2025-11-03T12:18:01Z

plugins/pi4j-plugin/pom.xml

    <!-- PROJECT MODULES <Pi4J PLUGINS> -->
    <modules>
-        <module>../pi4j-plugin-mock</module>
+<!--        <module>../pi4j-plugin-mock</module>


I think all of these should be activated again?

eitch · 2025-11-03T12:18:09Z

pom.xml

    <modules>
        <!-- THE FOLLOWING ARE THE Pi4J MODULES -->
-        <module>libraries/pi4j-library</module>
+        <!--<module>libraries/pi4j-library</module>-->


DigitalSmile · 2025-11-04T07:03:22Z

@eitch this PR is work in progress, I am cleaning up the code on my way with todo list.
I am asking to review the approach, stated in first comment.

eitch · 2025-11-04T07:16:33Z

Well, yes, the PR is ok with me.

# Conflicts: # plugins/pi4j-plugin-ffm/src/main/java/com/pi4j/plugin/ffm/providers/pwm/PwmFFMHardware.java # plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/ffm/integration/SPITest.java # plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/ffm/unit/PWMTest.java # plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/jmh/GPIOPerformanceTest.java # plugins/pi4j-plugin-ffm/src/test/java/com/pi4j/plugin/jmh/SPIPerformanceTest.java

Created new kernel drivers for I2C, PWM and SPI, fix tests

01970c8

DigitalSmile self-assigned this Oct 27, 2025

github-advanced-security bot found potential problems Oct 27, 2025

View reviewed changes