Automated Testing with Full-System Emulation
Ryan Torvik - March 31, 2023 - 3 minute read
Tulip Tree Technology has created a couple different emulators to apply vulnerability research concepts and techniques to automated testing. Automated testing provides data as input to a running system and sees how the system reacts. Testing a piece of code or a single application is fairly simple, if you can run it. But what if you need to see how an entire operating system and set of applications handles external input? Here’s where full-system emulation comes in.
What is an emulator
An emulator is software that lets you run a binary (a target) in an environment different from the one it was created for. The target might naturally run on an MIPS based Wifi router, but an emulator will run it on your x86 Windows laptop. A video game might naturally run on a Nintendo Wii, but an emulator will run it on your ARM based iPad.
An emulator can run a single application, like the web server for that WiFi router. Or it can run the entire operating system and applications for the whole firmware image of that WiFi router. We’ll call these two types application only and full-system emulation.
Application-only vs Full-System Emulation
Application only emulation is pretty handy. If you can extract a single application from a firmware image, you can run it using user-mode QEMU, Unicorn, or Qiling. Each of these tools gives you some introspection into how external data is being used inside the application. Unfortunately, this method doesn’t let you see how an application handles data in it’s native environment, like how it receives data from the kernel.
Full-system emulation, on the other hand, runs the entire operating system. The emulator is a stand in for the real hardware. It handles decoding individual bytes into instructions like a real processor does. It handles memory paging just like a real MMU does. It has devices that raise interrupts on the processor when data is available, just like real devices do. Even if you are only interested in testing a single application, by running it in its operating system in a full-system emulator, you can observe how that application will behave on hardware.
Testing on hardware
You could do full-system automated testing on hardware, sure. But hardware has its drawbacks. With supply-chain issues, acquiring hardware has become problematic. So, if you want to run more than one test at a time or your hardware fails, just acquiring the hardware can add weeks to your schedule. You might even have to redesign your entire product because you can’t get the right hardware to test on. If your environment is a number of interconnected devices, you’ve got to make sure everything is configured just right whenever you need to run a test. Just doing standard debugging can require physical access to test hardware and an oscilloscope.
Advantages of full-system emulation
There are several big advantages to using full-system emulation.
- Accuracy - You get to see how the entire firmware will function on hardware.
- Scalability - You can scale to support any number of concurrent tests and only incur additional compute cost.
- Visibility - You can see what’s actually happening to the registers, memory, and devices as the system runs.
These advantages make full-system emulation the obvious choice for testing embedded device software.
How fast is full-system emulation
Unfortunately, no emulator is as fast as real hardware. The instruction translation, emulated device interactions, and execution loop are all written in software. These things all take more execution time. Some emulators have made it a large priority to try to get speeds up near real hardware. But, by doing so, they’ve limited their visibility. If you are testing the validity and reliability of something, do you care more about speed or accuracy?
Conclusion
Full-system emulation is an important technology to enable people to do effective automated testing on their embedded systems. Tulip Tree Technology has created an emulator specifically to make the lessons we’ve learned doing offensive cyber security more accessible to embedded system manufacturers. Reach out to us to get a demo and help us see how we can help you reduce time to market and make more secure products.