To the Stars, Digital Rockets & Custom Hardware

In the summer of 2020, Sailboat had an idea for a custom motion controller and videogame. The main goal was to create a three-dimensional input device for games. The motion of the controller itself is typically, but not always, a secondary control feature in gaming. The ergonomics of traditional controllers and keyboards don’t lend themselves well to the control of vehicles such as rockets.

The Sailboat team and I (Justin) wanted to further explore this idea, so that fall, the company sponsored my Engineering group at the University of Calgary as a part of a capstone course. Our group came up with some fantastic results using a bit of math, mad science, engineering, and creativity. We made the StarBoard controller and a game, Landy Rocket.

How did we do it?

The StarBoard

We named our controller the StarBoard. Like all great prototypes, it was made in a garage. This controller went through many prototypes, burnt components, and learnings. Our primary goal was to make an ergonomic controller that could be comfortably rotated in the average person’s hand. Our secondary goal was to make a game that could be easily played, to demonstrate the capabilities of the controller.

We started by thinking about what our controller needs to function in a game:

• A high precision orientation/motion sensor that could match 1:1 precision
• A Bluetooth controller that could send and receive data
• An MCU (microcontroller/CPU) that could handle the compute of sending and receiving data
• Buttons that could enable input beyond motion
• A power unit that would support wireless control and long battery life
• Credit card dimensions (width and length) with minimal thickness

Our process was to build, measure, learn, and repeat. We iterated rapidly and started with simple steps, and slowly added complexity until we achieved our requirements.

Prototyping

Our first prototype was made with basic off-the-shelf components including an Arduino Wi-Fi controller, a premade orientation sensor board, buttons, and hard wires. The manufacturing was simple, we used this controller to start developing firmware to send data to our game. We programmed the controller with Arduino’s version of C++.

This prototype confirmed our assumptions that the idea could work. We created a 3D model of the prototype in Unreal Engine and mimicked the orientation of the controller with the 3D model. While this worked, results were unpredictable due to the nature of Wi-Fi communication. We were missing several pieces including Bluetooth, wireless control, and ergonomics.

Prototyping Again

Our next iteration took us into the realm of the custom PCB (Printed Circuit Board). We found components on Digi-Key, including a 32-bit STM32 microcontroller, BLE (Bluetooth Low-Energy) module, oscillator, resistors, capacitors, and inductors. Once we determined which components we needed, we could start PCB design. After completing the design, we ordered components and sent off the PCB design for manufacturing. With the PCB and components, we hand-soldered each component and started testing.

This prototype would effectively replace the Arduino with our PCB. Although, we still used the Arduino for power. While the code we had written in the previous prototype was helpful, we effectively rewrote it in STM32 C++ (with the STM32CubeIDE) to support our new MCU and Bluetooth module.

The Bluetooth module presented a multitude of challenges. The device could not connect through a computer’s Bluetooth options and required extra configuration. These findings led us to adopt the use of the Bluetooth Classic protocol over BLE. The results were on par with the previous prototype. We learned to program a custom PCB and had a foundation for future design. We were still missing wireless power, an onboard sensor, ergonomics, and practicality.

Even More Prototyping

Our next step was to bring all of the pieces together on one PCB. With the success of the previous prototype, we were confident in our ability to bring everything together. Unfortunately, we faced more challenges than we imagined in design and manufacturing.

After completing the design and manufacturing of our first board, we noticed a design error with our new Bluetooth Classic module. The module was not receiving power properly, so we were back to the drawing board with several fried components.

With our second iteration, everything was working — until we tried the device wirelessly. The coin cell we used could not supply enough power. Instead of starting again, we improvised by removing the coin cell and replacing it with an AA battery pack and electrical tape. While this was not an elegant solution, it worked with a 48-hour battery life. Unfortunately, we did not account for the scenario of connecting external power with a battery that was switched on — we had to make another controller.

The firmware we developed for the controller gave us the results we wanted. The orientation of the controller would match 1:1 with the test object in our game. We could connect the controller through Bluetooth on macOS and Windows to play our game without additional setup. Yet, we had some flaws. The buttons were in a weird location, the MCU was overpowered, the sensor’s output was drifting, the battery solution was non-ideal, the ergonomics were still poor, the PCB was too big, and we had loose wires.

Production

Our final controller met our requirements and significantly improved on the last prototype. We compressed the size of the entire controller, centered the buttons and found smaller ones, used a smaller MCU, and added a built-in battery pack. Our learnings from previous prototypes improved our manufacturing and testing processes.

This controller could be manufactured in roughly one hour for about $50 in parts at low volume. We developed a 3D printed enclosure as well that would protect the board and the user’s hand from the electronics and sharp edges. We adapted the firmware slightly to transition from a 32-bit MCU to 16-bit. Unfortunately, the MCU had trouble communicating with the sensor, so we still need to fix the firmware or replace the MCU.

Another issue we faced was the speed of the Bluetooth module. It used the SPI interface (Serial Peripheral Interface), but most input devices use HID (Human Interface Device). Unfortunately, Bluetooth modules with the HID protocol were not commercially available at the time of making the controller. However, we could implement our own firmware on the existing module to support this protocol.

The StarBoard weighed in at 50 grams, with a 48+ hour battery life and less than 250 ms latency. The final dimensions with the enclosure were 88 x 55 x 24mm (or roughly that of a deck of cards).

Manufacturing

Our manufacturing process looked something like this:

It is impossible to hand-solder some small components, so we used a reflow oven to “bake” components into their place. Reflow is a soldering technique that uses soldering paste to hold components in position while a temperature curve melts the paste into soldered joints. We used a stencil to apply solder paste to the component contact pads on the PCB. We then placed components in their respective positions and melted the solder paste to create solder joints.

A microscope was necessary to align tiny components and avoid solder bridging that could cause shorts and fry the board. Creating the controllers was a manual and high-precision task that required patience and craftsmanship. We still fried multiple components and boards along the way.

A standard reflow oven costs hundreds, if not thousands of dollars. Instead, we improvised with a toaster oven and reflow controller. This solution worked perfectly at a fraction of the price.

Reflow requires a temperature profile that prevents shock or cracking of solder joints. The controller we used would sense the oven temperature with a thermocouple (temperature sensor) and regulate the power provided to follow the heat profile. The entire process would take about 5 minutes and result in perfectly cooked PCBs.

Some components on the board could not be reflowed as they would melt. We only reflowed the MCU, Bluetooth module, sensor, and other small components that were difficult to attach by hand. After cooling down the PCBs, we would hand-solder the remaining components.

After completing manufacturing, we would upload the firmware to the controller, assign a serial number, attach an enclosure, and start testing it. A serial number is assigned to uniquely identify a controller without requiring a hardware inspection.

Hardware Design

At a high level, our hardware design looked like this:

We needed to test our controller with a wired connection, so we had to support external and battery power. Beyond that, the controller needed tight integration between the buttons, sensor, MCU, and Bluetooth module. We will spare the reader the intricate details of the resistor, capacitor, and inductor placement, but datasheets from component manufacturers came in handy.

Firmware Design & Implementation

Our firmware design looked something like this:

Our firmware design was dependent on the electrical design. We had to configure pin connections, data lines, and communication protocols for the various components. STM32CubeIDE provides an interface for configuring pins on the MCU, which would indirectly program the sensor and Bluetooth transceiver. Each component would need to be configured to a specific mode to meet our requirements. Components would also communicate with the MCU through different protocols such as GPIO, I2C, and UART.

The most important part of the firmware was sending and receiving data. If the controller is connected, it will send a data package with 6 bytes at about 20 Hz. The data contains the following information:

• Compressed Quaternion (4 bytes) represented the orientation of the controller
• State Byte (1 byte) containing button and calibration state
• EOL Character (1 byte) indicating the end of a message

We were able to compress our information from 22 bytes to 6, which improved the stability and consistency of the controller input.

We separated our codebase into small, manageable chunks that would separate important logic for each electrical component. We also implemented a low-power mode when the controller is not connected to another device. We did our best to optimize compute and memory efficiency. Our MCU was only running at about 20% utilization, indicating that we could opt-in for an even smaller and less powerful MCU. Although, the extra compute space would be helpful if we were to implement other firmware features for the controller.

Emulation

Most standard gaming input devices (controllers, keyboards, mice) are not sufficient or ergonomic for motion input. However, we carry a device with similar hardware to the StarBoard in our pockets — cell phones. So, we created an emulation of the StarBoard with Android/iOS applications.

The controller benefits from ergonomics and has a lower risk of breaking when dropped, but accessibility is an important factor. Most modern phones should have a high-caliber sensor that exceeds that of the StarBoard. Although, we used Wi-Fi instead of Bluetooth for implementation simplicity. The caveat to this is an extra connection step that requires the user to input an IP address displayed to the user in the game.

Landy Rocket

We built Landy Rocket as a demonstration for the StarBoard. We developed the game concurrently with the controller and tested it with our prototypes. The idea was to use the orientation of the StarBoard to fly and land a rocket in 3D space through a small solar system. The concept was a perfect complement to the StarBoard as other gaming devices could not provide this experience.

We developed Landy Rocket with Unity Engine with C# and Blender. We chose a low-poly theme to keep the art of the game relatively simple.

Rotations

Most of our time developing Landy Rocket was spent fighting the rotational data coming from the StarBoard. We had to synchronize the real-world orientation of the player’s screen and controller with in-game rotations for rockets, cameras, and gravity. We had to reconcile the following rotations:

• Magnetic north of the Earth
• Orientation of the user’s screen
• Orientation of the user’s controller
• In-game orientation of the rocket
• In-game orientation of the camera
• In-game camera offset
• In-game direction of gravity

To add to the difficulty, we had to use an unintuitive rotational representation that uses imaginary numbers: the quaternion. Quaternions use 4 numbers to represent a 3D rotation and have the benefit of high-efficiency computations in games. They also avoid a host of issues related to XYZ representations of rotations.

Our goals in order of priority were:

1. Provide a simple calibration process to synchronize the real-world and in-game up, right, and forward directions
2. Mimic the orientation of the controller with the in-game rocket
3. Move the camera with the rocket
4. Maintain proper orientation of the rocket and camera when the direction of gravity is changed (Flying around a planet).
5. Rotate the camera around the rocket while maintaining intuitive control

We achieved all of these goals with some critical thinking and problem-solving.

One API to Rule Them All

We thought beyond Landy Rocket when creating the interface between the game and the controller. We built a Unity API that could allow anyone to develop any game with our controller. The API handles the complex logic of rotations, bit manipulation, and sending/receiving data. Our API also supports 3 communication protocols including UDP, TCP, and SPI.

The goal of the API is to help someone start developing a game with this new input method without developing the complex underlying logic. The API also provides a calibration process and examples.

Our calibration looks something like this:

System Design

Our high-level design for the entire system looks like this:

The controller API is the mediator for all interactions. It acts as the “glue” that sticks the physical and digital together. We designed a basic navigation architecture that would allow the user to flow through the user interface developed for the game:

We used events over polling to handle interactions. Rather than constantly asking for the state of the controller, we are notifying when there is a change.

World Building

We designed over 30 custom assets in Blender to create the world in Landy Rocket. We started with one model:

The Raven 1 was the first vehicle we made and acted as a test vehicle for the StarBoard. We developed 5 other vehicles including a car, sailboat, tractor, hot air balloon, and UFO. We kept with the design theme of the Raven 1 for vehicles, obstacles, and planets.

We developed four planets and a star for our miniature solar system. We created ice, water, lava, and farm planets (Glaces, Aquila, Labesa, and Villum respectively). Each planet has a set of unique buildings, foliage, structures, and landing pads with drivable vehicles scattered across them. The planets rotate about the star in circular orbits. Each planet has unique physical properties such as gravity and pressure that change the flight mechanics of vehicles.

User Interface

Given the unique nature of the input device, we had to create user interface elements to tell users how to use the controller. The orientation mimicking is already intuitive, but the function of each button was not clear. There was also the problem of helping the user visualize their position on a planet, so we made a top-down minimap. Finally, we added some basic user interface elements to provide context for the current planet and the user’s vehicle.

All Together

Clearly, the project had many moving pieces. But, step by step, we built a unique experience in the physical and digital realms. The most important takeaway is that building, measuring, learning, and repeating are essential when striving for innovation. We achieved all of our goals and are incredibly proud of the project.

Wrapping Up

Seeing is believing:

This project required a lot of creativity and problem-solving in software and electrical engineering. This project would not be where it is without the incredible team that brought the idea to life. The controls feel intuitive and remind us of flying toys in our childhood. If you have any comments, I would love to hear them.

Thanks for reading!

Justin

Special Thanks To:

• Adam (Electrical)
• Erik (Software)
• Ethan (Software)
• Jath (Software)
• Kevin (Electrical)
• Tyler (Electrical)
• Thomas (Teaching Assistant)
• Mohammad (Academic Advisor)