Testing a ESP32 LoRa device with Python and Jenkins

This article describes setting up a test suite in Jenkins using pytest which will determine if an esp32 LoRa development board transmits data. The milestones of the test suite are ...

  • Load the latest MicroPython firmware onto a esp32
  • Load python code onto a esp32
  • Listen for LoRa chirp with an SDR

To achieve these milestones Jenkins has to ...

  • Use a git repository
  • Use Pipenv to install python packages into its own user space
  • Use a git submodule for external MicroPython package
  • Upload and then execute code on the esp32 device
  • Use pytest to write html and xml reports
  • Archive outputs from the embedded device
  • Tag a git repository with successful builds


The Tool Chain

  • Linux kernel GNU tools and Gentoo distribution
  • Jenkins deploys and automates any project
  • Python is a programming language that lets you work quickly
  • Git is distributed version control system
  • pytest framework makes it easy to write small tests
  • MicroPython is a lean and efficient implementation of the Python 3
  • Pipenv Python development workflow for humans
  • RTL-SDR Project for [ab]using DVB-T USB receivers based on RTL2132

The Hardware

  • Heltec ESP32 LoRa Node
  • Noolec Software Defined Radio
  • A Computer with the installed tool chain. Gentoo is optional

Setting up Jenkins

Jenkins requires a Git plug-in for us to do what we want to do. A screen shot of the one we have installed is can be seen here. We also require a Git repository for our tests and MicroPython code which can be found here on git hub. To use this repository at the command line we do the following ...

git clone
cd jenkins_esp32_lora_testing
Now we want to setup a pipenv environment so we don't go overwriting system files and also make our repository portable to be used by Jenkins.
pipenv --python 3.6 install
Now we can install some python packages ...
pipenv run pip install pytest rshell esptool
If everything went ok you'll see something like ...
Successfully installed esptool-2.6 pyserial-3.4 pytest-4.4.0 rshell-0.0.21 ...
To record a list packages that pipenv has installed we can run ...
pipenv run pip freeze > requirements.txt
We need the requirements file for Jenkins to pick up and install packages into its own user space when it comes to execute the job. The reason for this will become evident later. If the Jenkins set up went OK we will be able to see something similar to this image. Clicking the create jobs link seen in the image will take us to a screen asking for details of the job. In this case it will be using a Freestyle project as show in this image Once we've clicked Ok to create the project you will see something very similar to this image

Configuring a Jenkins Job

This image details everything about the configuration. The sections to pay attention to are "Source Code Management" and "Build" -> "Execute shell". The source code management is set up to detect changes from the repository we created with git and github. The Execute shell commands does the following

pip install --user pipenv
Install pipenv
Set the local path for pipenv application
pipenv --python 3.6 run pip install -r requirements.txt
Install the required packages
pipenv run pytest --junitxml=./report.xml -c tests/pytest.ini
Run the tests

Creating Tests with pytest

In the "projects/python/jenkins_esp32_lora_testing" directory the following directory and files have been created. Refer to the repo for these files.

pytest configuration
The test suite
What we have done here is created a test suite to download and flash the latest firmware onto a esp32. We can now run / build this test suite in Jenkins.

Running Tests in Jenkins

Now that we have some tests lets get Jenkins to execute these tests to see what results we get back. To do this we can either set up Jenkins to poll the SCM, I.E check git for any changes, or we can click the "Build Now" option which is what we will be doing. The "Build Now" option can be found in the project's default page which can be seen in this this image. Once we've clicked the build now option a build will appear in the build history which is covered in more detail in the next Examining the Test Results section. If you have copied or cloned tests from the repository at this stage and they break you may want to follow the instructions in the Bigger and Better Test Reporting section and return back to this point.

Examining the Test Results

Now that the tests have been run we will have some build history which is shown in this image. To see what actually happened in the test we can click on the console output. This will show us what Jenkins executed and the output of that execution. Because the output is lengthy it has been been split up into three images. The first image shows the python packages being installed into a virtual environment The second image shows the tests being executed and this is where it all goes a bit Pete Tong. At the bottom of the third image the amount of tests that failed and passed can be seen. The important bits of the image to look at are ...

raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 13] could not open port /dev/ttyUSB0: [Errno 13] Permission denied: '/dev/ttyUSB0'
		cmd =, stdout=subprocess.PIPE)
		if cmd.returncode == 0:
			flashed = True
>       assert(flashed is True)
E       assert False is True
tests/ AssertionError
- generated xml file: /var/lib/jenkins/home/workspace/ESP32_LoRa_Testing/report.xml -
====================== 1 failed, 2 passed in 1.40 seconds ======================
Build step 'Execute shell' marked build as failure
Recording test results
Finished: FAILURE

The first couple of lines show that the test failed on a rasied exception. In this case Jenkins did not have the correct permissions to access the USB device. We will fix this in a jiffy. The last few lines detail ...

  • Where the test failed assert(flashed is True)
  • The name of the xml report generated xml file
  • Tests that failed and passed 1 failed, 2 passed in 1.40 seconds

Now that we've seen the tests failing lets have a look at what Jenkins is showing us back at the project page which can be seen here . The project overview page show us the number of times the tests have been run and a graph showing the trend of those tests. Lets go and fix that broken test.

Fixing the Test

To correct this test failure all we have to do is grant Jenkins access to the ttyUSB devices by issuing the following command as root.

usermod -a -G dialout jenkins
We can now re-run the tests and admire our successes which you can be seen in this image . Before we continue to write more tests there are a few more features in Jenkins that make analysing tests a little easier. For example in this screen shot you can see that Jenkins has now reported the failing test as fixed. You can navigate to this information by first clicking the build number then the "Test Result" link.

Expanding the Firmware Tests

Now that the we think the firmware is being loaded onto the esp32 we want to write a test to make sure that it is. To do this we will using rshell to copy and execute files on the esp32. The following directories and files have been created in the github repository so we can do just that.

We also created our last firmware test in file ./tests/ named test_firmware_loaded and the result of which can be seen here. Note the extra test added to the results graph. The purpose of the archive directory is store information about the build which Jenkins can be configured to archive. To do this click the "Add post-build action" and then "Archive the artifacts". This screen shot shows what it should look like. After re-running the tests again we will now have an archive of the version info collected from the esp32 device. This image shows Jenkins archiving the firmware_text.txt which contains the following text

(sysname='esp32', nodename='esp32', release='1.10.0', version='v1.10-315-ge70c438c7 on 2019-05-02', machine='ESP32 module with ESP32')
At this point it is also a good idea in the Jenkins build configuration under the "Source Code Management" section to add the option to "Clean before checkout". This will ensure that no artifacts will be left over from the previous build.

Pipenv Hiccups

When running the test suite again the following problem was encountered ...

⠋ Creating virtual environment...
⠙ Creating virtual environment...
⠹ Creating virtual environment...Using base prefix '/usr/lib/python-exec/python3.6/../../..'
New python executable in /var/lib/jenkins/.local/share/virtualenvs/ESP32_LoRa_Testing-AEcau7YN/bin/python3
Running virtualenv with interpreter /usr/bin/python3

✘ Failed creating virtual environment 
[pipenv.exceptions.VirtualenvCreationException]:   File "/var/lib/jenkins/.local/lib64/python3.5/site-packages/pipenv/vendor/click/", line 17, in new_func
  File "/usr/lib64/python3.6/", line 121, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 40] Too many levels of symbolic links: '/var/lib/jenkins/.local/share/virtualenvs/ESP32_LoRa_Testing-AEcau7YN/bin/python3'
To fix this problem the following commands were issued ...
rm -rf /var/lib/jenkins/home/workspace/ESP32_LoRa_Testing
rm -rf /var/lib/jenkins/home/workspace/ESP32_LoRa_Testing@tmp
rm -rf /var/lib/jenkins/.local/share/virtualenvs/ESP32_LoRa_Testing-AEcau7Y
The reason these errors occurred was because a build was stopped half way through its execution. This sort of problem should not happen and requires further investigation.

First Milestone Achieved

We have reached our first milestone by loading the esp32 device with the latest firmware and tested that it was loaded. We can now move onto importing and writing some MicroPython code to get the LoRa enabled device to chirp. Before we go and do that lets setup some other tools for better reporting.

Bigger and Better Test Reporting

To fine tune the tests and reporting we are going to ...

To make the output of the tests in the build console a bit more friendly and make pytest output a html report we need to change the line in the build configuration "Execute shell" from and to ...

pipenv run pytest --junitxml=./report.xml -c tests/pytest.ini
pipenv run pytest -v --junitxml=./report.xml --html=./html/report.html -c tests/pytest.ini
This will produce a better console output which will look like this. We have a bit more configuration to do to produce a html report which is below.

To add html reports to Jenkins we need to install a Jenkins plugin and the one we used can be seen here. We now have to configure pytest and Jenkins to produce and read the html report. Firstly we need to add a couple of python packages to our pip environment and we do this at the command line in our repository directory and execute ...

pipenv run pip install pytest-html pytest-ordering
pipenv run pip freeze > requirements.txt
mkdir html && touch html/.keepme
Jenkins now needs to know where the html report will be saved and this is done in the build configuration under the Post-build actions. This image shows the html report plugin being configured. The test ordering will become evident in future sections when we write tests to make sure that the MicroPython code is functioning and the signal is being sent.

Adding git submodules

For us to use the LoRa functionality on the esp32 device we need to add the uPyLora repository to our own repository as a submodule and configure Jenkins to pull it in. To do this at the command line in our repository directory execute ...

git submodule add esp32/lora
and in Jenkins configure the build to add additional git sub-modules behaviour as shown in this. screen shot . Also note the "Clean before checkout option" just below the "Additional Behaviours" section. If we now go and execute the tests again the console output in the Jenkins build should show something similar to this image.

The Chirpy LoRa Code

We almost have everything needed to make the esp32 device chirp we just have to write some Python code and copy all the required files over to the esp32. The below two files combined with the uPyLora repository do just that ...

Before we venture off into the world of SDR's and rtl-fm we can manually make sure that the esp32 is chirping by using the gqrx tool to visualize what is happening on a specific frequency. This image shows the esp32 device chirping. It is worth noting here how much bandwidth the transmitted signal is taking up. This will need further investigation and explanation when we come to use the rtl-fm.

Testing the LoRa Code

We have manually tested that the esp32 LoRa node is chirping we can now write a new set of tests to check that the esp32 is outputting the expected file which we write out whilst chirping. We will also be checking that our test ordering and html reporting is working.

To see how the ordering works the ./tests/ contains a test named test_firmware_directory_exists and just above this test is a decorator named This tells pytest to execute this test first. The subsequent tests in this file have decorators pointing to the test above them. The first test in ./tests/ has a decorator pointing to the last test in the ./tests/ file. This makes pytest upload and check the firmware before loading and executing our main MicroPython code.

We also now have html reports. This report snippet details a failed build and this report snippet details a successful one. There is allot of information to be gained from these reports such as build ID, build number and git branch which will hopefully come to use later in this article.

Second Milestone Achieved

We have reached our second milestone by loading the esp32 device with our own and external MicroPython code and tested it. We have also improved the reporting and added a bit more structure to the tests with ordering. The final milestone is to use a rtl-fm with a SDR to independently verify that the esp32 device is chirping. This does bring an extra complication in the form of threading but we will cross that bridge when we get to it. Before we dive into threads lets recap on what Jenkins is telling us and what can we gain from this.

Our current test trend looks like this. The last three failures in the graph where related to retrieving an outputted file from the esp32 device as this html report details. This is where Jenkins comes into its own by storing everything we need to know about failed and successful builds. We obviously want to know what makes a build fail so we can correct that problem whether it be with the tests or with another part of the stack. Conversely we also want to know when a build is successful as we may want to promote or release this build to other builds or projects.

Tagging Successful Builds

Because Jenkins does not point to our github repository but a local file system it took some tinkering and building to make it commit tags. We achieved this by adjusting the file permissions on the repository and added the Jenkins user to the users group in our OS. This is not the best way to use Jenkins but it suits our needs for now. So what has Jenkins done? And why go to all that trouble? We went to all this trouble because git now contains tags of our successful builds ...

$ git tag
This is important because if we wanted to promote or release this build / product we now have a record of all the required python package versions and firmware. This image shows an overview of build 55 and this is the console output and this is firmware output and this is the esp32 device chirp output file. All this information can be gathered with a couple of mouse clicks or better yet be automatically gathered and bundled with a product.

Listening for the Chirps

Our last two tests named test_sdr_listen_for_silence and test_sdr_listen_for_lora_chirp can be found in the ./tests/ file. The first test, test_sdr_listen_for_silence, tests the outputted file written by rtl_fm contains zero bytes because no chirps are being sent. The second test tests that the outputted file written by rtl_fm contains more than 800000 bytes with chirps being sent. It is worth mentioning that we side stepped the threading issue by putting the rtl_fm into a Linux background process and then started the esp32 device chirping. This is not the best way to thread the rtl_fm process because the kill command that stops that process may kill more than just the process we are interested in. This image shows out final test trend with the last two tests added and the outputted rtl_fm files.

Going Forward

Here a list of things that can be done to improve and progress the tests ...

  • Investigate the pipenv crash in Jenkins
  • Fine tune the LoRa node transmission signal
  • Auto detection of devices to kick of builds
  • Use Python threading instead of Linux background process
  • Build and test a esp32 LoRa node that listens for the chirps and extracts the data
  • Split the tests suites into their own builds and have Jenkins detect successes and failures
  • Use a configuration file for test parameters such as usb device and board rates
  • Decode the data in the radio signal and possibly use numpy's Fast Fourier Transforms for chirp frequency detection
  • ...


Hopefully this article outlines how a embedded device can be continually tested with Jenkins and python and provides some insight into the steps required to do this.