Embedded Continuous Integration setup
I setup a full continuous integration setup within one week from scratch. Without previous knowledge about CI. In this blog post I explain what a Continuous Integration (CI) setup is. Next, I share my thought process in deciding between various test approaches. Finally, we’ll have a short look at the system I built. You can find a more in depth explanation regarding my setup in the next blog posts.
Why is continuous integration a good thing?
I’m the founder of 5upercircuits and also its only employee for now. I bootstrap my business. That means, I have to be as efficient and effective as I can be. In my pursuit of achieving maximum performance, I focus on:
- High quality tools
- Doing what I can and outsource the rest
For me, continuous integration is about automation and monitoring. It can automatically perform work that I would have to do myself at a much slower pace. It never tires and is available 24h. Monitoring is important to keep the overall software and hardware quality as high as possible. It simplifies debugging a lot if you find bugs right after they creep into code.
What are the difficulties of embedded CI?
Usually CI setups roughly consist of the following tasks.
It’s relatively straightforward to implement the steps above, when you’re using only one development platform (e.g. web development). But, in Embedded we have to deal with different processor architectures, operating systems and hardware than our CI system (e.g. x86) is using.
Fortunately, automated building won’t be much of an issue, if we’re using makefiles and cross compliers such as GNU Arm Embedded Toolchain. However, testing Embedded systems is already a lot more difficult. First, the tests will either run on our CI system or on the Device Under Test (DUT), probably not on both. Second, the hardware boundary between the CI setup and the Device Unter Test (DUT) further complicates things. We have to find suitable interfaces and protocols so the CI setup can communicate with the DUT.
How are embedded system companies doing it?
Continuous Integration is used for embedded development by many companies. Build, Unit Tests and tests running on the target hardware are common setups.
Unit tests are tests that will make sure each and every function works the way it was intended. Note the focus on “intended” and the passive voice in the sentence. Without going into too much detail, here are the benefits of unit tests.
- Discover bugs right after they sneaked into code
- Easier refactoring, scaling of code because tests ensure nothing accidentally breaks.
- Better communication/teamwork because tests highlight the purpose of code segments (e.g. functions)
Why don’t you do it the same?
While unittests are great for embedded system companies it’s not appropriate for freelancers, startups and hobbyists. When I talk about embedded system companies, then what I really mean is a team of SW engineers (e.g. 10-20 people) working on one huge code base.
There is a lot more to know about unittests and CI tests for embedded in general. I won’t go into further detail. Instead, I will reference an other blogger who wrote a great article about it 🙂
Well, what’s your CI testing approach then?
Not to do unittests. But, create a mix between unittests and smoke tests. I call those tests Command Line Interface (CLI) tests.
Why? What’s your issue with unittests?
Unittests would either have to run on the target system, or we’d have to port the code to an x86 platform and run it on the CI system.
If we need a test for almost every function in code, that means code size = test size. Which will result in quite much memory consumption on the target system.
Also, if we decide to port the code to an x86 platform to avoid the memory consumption problem. Then, we have to spend a lot of extra work. We’ll be busy with porting, faking drivers and tricking the code into thinking it runs on the target hardware.
Both scenarios are not acceptable for freelance and bootstrap startups.
So, what are CLI tests all about?
The goal of CLI tests is to provide an interface for the CI system (running on x86) to run tests on the target platform. For any serious embedded system development a CLI/shell should be a must anyway. So, running tests on the target platform via shell is very convenient.
The CLI tests should be as detailed as possible and have direct access to even the most hidden system functions. It’s not an integration test, it’s supposed to be the closest possible thing to unittests.
CLI tests don’t need to run on the CI setup, so no porting. Also, we won’t write a CLI test for each and every function. Finally, some CLI tests can be temporarily excluded by preprocessor directives. So, we can have different test sets for different modules and only include it in the build if they are needed. Thereby, lowering the memory footprint of CLI tests.
What does your CI setup look like?
I use Gitlab for revision control, backup, project planing and note taking. So, I decided to go with Gitlab CI, because it’s a very streamlined process.
Gitlab CI consists of a CI panel which runs on your normal Gitlab instance. Then there is the Gitlab Runner which should run on a separate VM. The Gitlab Runner is responsible to perform all continuous integration tasks (e.g building, testing etc.).
Next, for performing CLI tests, we will need to interface the Device Under Test (DUT) via UART. While I generally trust my abilities as a developer…I still hesitate to connect potentially buggy hardware to my servers. Therefore, I have the Gitlab Runner VM control a Raspberry PI which will in turn run tests on the DUT via UART.
I tested the CI setup during development of the Moment-Rec project. In the picture below, you can see a first rough setup. On the left is the device under test (DUT) and on the right the Raspberry Pi board. I put both device in a metal box, to mitigate the risk of fire if the electronics blows up during tests 😉
What’s next? I would like to know more!
I decided to split this blog post into several parts. It’s going to be a very long explanation and I want to give you a deep insight into how I built the CI setup. In the following posts, we will cover:
- Setup of Gitlab-Runner + docker images
- Raspberry Pi and using the Python unittest library to conduct CLI tests.