Writing a functional unit test
Ryan Torvik - June 20, 2023 - 5 minute read
One of the biggest advantages of getting a piece of firmware running in an emulated environment is the ability to automate functional testing. Doing that type of testing all in software means you’ll be able to take advantage of modern software development practices, like a CI/CD. In this blog, we are going to write some pytest code that tests the functionality of our firmware by running it in out full-system firmware emulator Emerson.
This blog is one in a series that takes a piece of firmware, gets it running in our full-system emulator with appropriate peripherals, and puts that emulated system into a CI/CD for automated testing.
Pytest
Pytest is a unit testing framework for python. You can learn more about it here. We start by making a directory for our tests.
$ mkdir tenkey_tests
Then we’ll put a test in there.
def test_tenth_fibonacci_number():
assert(10 == 10)
Then we’ll run the test!
$ pytest
===================================== test session starts =====================================
platform linux -- Python 3.9.2, pytest-7.2.2, pluggy-1.0.0
rootdir: /workspaces/tenkey_test
collected 1 item
test_tenkey.py . [100%]
====================================== 1 passed in 0.05s ======================================
Great, now let’s do some connectivity with Emerson!
Connecting to Emerson with python
As we discussed previously, Emerson runs inside a docker container. To keep things simple, we have a docker image for our python interface as well. We can mount a local directory into that docker container and run pytest on that directory in the container. Then, we can use docker compose to connect the emerson server and the python tester.
version: "3.7"
services:
server:
image: emerson.server
ports:
- "10314:10314"
logging:
driver: "none"
volumes:
- ./project:/tmp/emerson/projects/tenkey
python_tester:
image: emerson.python
depends_on:
- server
volumes:
- ./tenkey_test:/tenkey_test
- ./wait-for-it.sh:/wait-for-it.sh
working_dir: /tenkey_test
command: /wait-for-it.sh -t 60 -s server:10314/is_good -- python3 -m pytest
This loads up our emerson server docker image, listening on port 10314, and mounts our project files in the appropriate location. Then we mount in our pytest directory into the emerson.python container. When we run this docker compose, it will start up the emerson server, wait for it to be up and running, and run pytest in the emerson.python container.
A Python script to run our test
Here’s a python test script that does the same thing we did by hand in the last blog post. It sets a breakpoint, runs the algorithm, and checks the computed value. It uses the emerson python library to interact with the emerson server. Anything you can do in the gui or in the command window is available to the python library too.
import pytest, os, time
from emerson import Connection, Machine
@pytest.fixture
def machine():
# create a connection to an emerson server
connection = Connection()
connection.connect(f"http://server:10314")
# if there is already a project running (unlikely), stop it
if connection.is_project_running():
connection.stop_project()
# start up the ten key project
connection.run_project("tenkey")
# make sure we wait until the project is running to move on
while not connection.is_project_running():
time.sleep(1)
assert connection.is_project_running()
# this returns a machine that can be used in the tests
return connection.attach()
def test_tenth_fibonacci_number(machine: Machine):
'''
set a breakpoint, run the fibonacci generator, verify the value at the end is correct
'''
expected_value = 55
assert "Paused" == machine.state
# set a breakpoint on the address where we are finished running our fibonacci algorithm
debug_point = machine.break_on_address(0x1fc00094)
assert debug_point.id == 0
# run the machine
assert machine.go()
assert "Paused" == machine.state
assert machine.pc == 0x1fc00094
# since we're paused at the breakpoint, check the value in v0
assert machine.get_register("v0") == expected_value
@pytest.fixture(scope="session", autouse=True)
def cleanup(request):
''' stop the project once we are finished '''
def stop_project():
connection = Connection()
connection.connect(f"http://server:10314")
connection.stop_project()
connection.disconnect()
request.addfinalizer(stop_project)
Running our pytest script
Running our script is as easy as running anything in docker compose. Since we’ve named our file docker-compose.yaml and our docker images are tagged correctly, it just runs!
$ docker compose up
[+] Building 0.0s (0/0)
[+] Running 2/0
✔ Container tenkey-server-1 Created 0.0s
✔ Container tenkey-python_tester-1 Created 0.0s
Attaching to tenkey-python_tester-1, tenkey-server-1
tenkey-server-1 | [2023-06-19T14:45:26Z INFO actix_server::builder] starting 12 workers
tenkey-server-1 | [2023-06-19T14:45:26Z INFO actix_server::server] Actix runtime found; starting in Actix runtime
tenkey-python_tester-1 | wait-for-it.sh: waiting 60 seconds for server:10314
tenkey-server-1 | [2023-06-19T14:45:26Z INFO actix_web::middleware::logger] 172.19.0.3 "GET / HTTP/1.1" 200 621 "-" "curl/7.74.0" 0.000316
tenkey-python_tester-1 | wait-for-it.sh: server:10314 is available after 0 seconds
tenkey-python_tester-1 | ============================= test session starts ==============================
tenkey-python_tester-1 | platform linux -- Python 3.9.17, pytest-7.3.2, pluggy-1.0.0
tenkey-python_tester-1 | rootdir: /tenkey_test
tenkey-python_tester-1 | collected 1 item
tenkey-python_tester-1 |
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "GET /is_project_running HTTP/1.1" 200 5 "-" "-" 0.000044
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "GET /run_project/tenkey HTTP/1.1" 200 17 "-" "-" 0.394474
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "GET /is_project_running HTTP/1.1" 200 4 "-" "-" 0.000039
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "GET /is_project_running HTTP/1.1" 200 4 "-" "-" 0.000033
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "POST /run_command HTTP/1.1" 200 73 "-" "-" 0.000203
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "POST /run_command HTTP/1.1" 200 34 "-" "-" 0.000251
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "POST /run_command HTTP/1.1" 200 49 "-" "-" 0.000149
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "POST /run_command HTTP/1.1" 200 73 "-" "-" 0.000205
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "POST /run_command HTTP/1.1" 200 42 "-" "-" 0.000173
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "POST /run_command HTTP/1.1" 200 36 "-" "-" 0.000213
tenkey-server-1 | [2023-06-19T14:45:27Z INFO actix_web::middleware::logger] 172.19.0.3 "GET /stop HTTP/1.1" 200 9 "-" "-" 0.000081
tenkey-python_tester-1 | test_tenkey.py F [100%]
tenkey-python_tester-1 |
tenkey-python_tester-1 | =================================== FAILURES ===================================
tenkey-python_tester-1 | _______________________________ test_state_basic _______________________________
tenkey-python_tester-1 |
tenkey-python_tester-1 | machine = <builtins.Machine object at 0x7fe2599aa6c0>
tenkey-python_tester-1 |
tenkey-python_tester-1 | def test_tenth_fibonacci_number(machine: Machine):
tenkey-python_tester-1 | '''
tenkey-python_tester-1 | set a breakpoint, run the fibonacci generator, verify the value at the end is correct
tenkey-python_tester-1 | '''
tenkey-python_tester-1 |
tenkey-python_tester-1 | expected_value = 55
tenkey-python_tester-1 |
tenkey-python_tester-1 | assert "Paused" == machine.state
tenkey-python_tester-1 | # set a breakpoint on the address where we are finished running our fibonacci algorithm
tenkey-python_tester-1 | debug_point = machine.break_on_address(0x1fc00094)
tenkey-python_tester-1 | assert debug_point.id == 0
tenkey-python_tester-1 |
tenkey-python_tester-1 | # run the machine
tenkey-python_tester-1 | assert machine.go()
tenkey-python_tester-1 | assert "Paused" == machine.state
tenkey-python_tester-1 | assert machine.pc == 0x1fc00094
tenkey-python_tester-1 |
tenkey-python_tester-1 | # since we're paused at the breakpoint, check the value in v0
tenkey-python_tester-1 | > assert machine.get_register("v0") == expected_value
tenkey-python_tester-1 | E AssertionError: assert 256 == 55
tenkey-python_tester-1 | E + where 256 = <built-in method get_register of builtins.Machine object at 0x7fe2599aa6c0>('v0')
tenkey-python_tester-1 | E + where <built-in method get_register of builtins.Machine object at 0x7fe2599aa6c0> = <builtins.Machine object at 0x7fe2599aa6c0>.get_register
tenkey-python_tester-1 |
tenkey-python_tester-1 | test_tenkey.py:44: AssertionError
tenkey-python_tester-1 | =========================== short test summary info ============================
tenkey-python_tester-1 | FAILED test_tenkey.py::test_state_basic - AssertionError: assert 256 == 55
tenkey-python_tester-1 | ============================== 1 failed in 0.56s ===============================
tenkey-python_tester-1 exited with code 1
Gracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Stopping 2/2
✔ Container tenkey-python_tester-1 Stopped 0.0s
✔ Container tenkey-server-1 Stopped
All the output from the server is tagged with tenkey-server-1. You can see that we log all the API requests. The test script checks if a project is running, runs a project, and then runs a bunch of commands. These commands set the breakpoint, run the project, and when the breakpoint is hit, checks the computed value in the register v0 just like we did by hand!
And the test failed! But in the same way that it did when we were checking it by hand! Next time, we’ll fix our code and fix this test!
Conclusion
This is a another installment in our series about developing firmware in Tulip Tree Technology’s emulation platform Emerson. We’ve created a piece of firmware, viewed the disassembly of the compiled firmware, created a device in python for the firmware to interact with, and ran the entire system in a debugger. Now we’ve written a unit test that can be run automatically without a human in the loop. This technology is a significant step up for firmware manufacturers. Reach out to us to see how we can significantly reduce your development time, open up firmware development to more main stream software engineers, and improve the security of your devices.