Wednesday, December 10, 2014

Self-Balancing Robot

In my last class for my Masters, I decided to build a self-balancing robot for the final project. What I liked about this project, is that it involved a few very relevant areas of embedded systems all in one project. I'll try to write this post the way I wrote the presentation.

The class was 'Mixed Signal Embedded Systems', and revolved around the Cypress chip called the PSoC4 (programmable-system-on-chip). The chip is interesting in that it has both an ARM M0 processor AND some (emphasis on 'some') programmable logic (PLDs). It also has 2 embedded amplifiers and some other interesting HW components (all configurable through the ARM via registers). Anyway, this post is not about this chip, but feel free to read up on it. It is another example of the direction in which embedded systems are heading.

Motivations for this project

  • Interface to inertial measurement sensors (Gyros/Accelerometers)
  • Employ light-duty sensor fusion for robot pose estimation
  • Implement an embedded PID controller to maintain robot balance

The problem: The Inverted Pendulum

The goal of the inverted pendulum is to keep the pendulum from tipping over. Without intervention, the pendulum is naturally unstable and will eventually tip over. Controls must be used to counter act this tendency by applying a force in the horizontal direction.

Inertial Measurement Sensors

The MPU9150 was used for inertial measurement sensing for this project. The MPU9150 is:

  • 3 axis accelerometer
  • 3 axis gyro
  • 3 axis compass
  • ...all integrated in a single IC

The benefits of having these components on a single chip, is that the error caused by misaligned axis is greatly reduced and controlled by the manufacturing process. This also makes the packaging both smaller and cheaper.

The sensor data is accessed through the I2C protocol, so a microcontroller is needed to perform this (and I wrote a library both in C and C++ which I will post to my github page and provide a link).


Accelerometers measure linear acceleration. The diagram above shows the 3-axis accelerometer
placement inside the MPU9150. When an accelerometer is stationary, it measures a net acceleration the gravitational force of 9.8m/s^2 (at least on earth). The orientation of this sensor with respect to the gravitational force (let's call it earth's z-axis) can be estimated using trigonometry:

This function is accurate only when there is no external forces being applied. When external forces are applied, such as the robot moving/tipping, the orientation cannot be accurately estimated.


Gyros measure the the rate of angular change. The above diagram shows the 3-axis gyro placement inside the MPU9150. Gyros are not susceptible to linear forces such as vibrations or movement. They are only sensitive to angular displacements. Gyros cannot be used to directly measure physical orientation, but their readings can be integrated over time.

dt is the control loop time.

Gyros suffer from a phenomenon known as drift, where they show a small rate of change even though no actual rate of change is occurring physically. When integrating, this error adds up over time, causing an inaccurate estimate of angle.

Also note from the 2 pictures of the axis alignment: the x-axis gyro is perpendicular to the y-z plane of the accelerometer. So the angle using the accelerometer's x-z plane (as shown in the equation) would be the same angle using the gyro's y-axis.

Estimating Robot Pose

In order to accurately estimate angle using gyros and accelerometers, one has to combine these sensor
readings together in a way that takes advantage of their strengths in order to make up for their
weaknesses. This technique is known as sensor fusion. There are a few different pose estimation methods that have their own strengths and weaknesses. For this project, one that is easily implemented on a low-power microcontroller, called complementary filtering is used. The equation below shows how this is implemented:

Alpha is used to weight the two angles estimates before they are combined. Since the accelerometer is more prone to adding noise to the system, a large alpha is chosen to give the gyro integration more
weight in the equation, effectively applying a low-pass filter on the accelerometer. For this project, an
alpha of 0.99 was used (along with a control loop time of 10 milliseconds).

PID Control System

Proportional-Integral-Derivative (PID) is a control loop feedback technique using the error between a
set-point and observed output of a system (often referred to as the plant). A PID controller must be tuned by varying the 3 gains associated with the control algorithm: Kp, Kd and Ki. Different choices for these constants manipulate the systems response in terms of responsetime, overshoot and oscillations.

Kp: The proportional term. Depends on the present error only.
Ki: Integral term. Depends on the accumulation of past errors.
Kd: Derivative term. Prediction of future errors.

The following is a pseudo-code example of implementing a PID algorithm in a controlled loop.

The output of the PID algorithm is a sum of the gain-terms multiplied by their function on the error. The error is a simple difference from the target (in this case an angle) and the observed state (the estimated angle by the sensor fusion algorithm). It is important to keep the control loop rate consistent (so use an interrupt, or don't do extra processing).


I built my robot using parts of one of my other robots, so save on cost. I built a cheap frame out of small wood squares, a dowel and hot glue (all bought at Michaels for less than $5):

Starting off with just a p controller (Kp >0, Ki = Kd = 0), the system was nowhere near useful. The robot would over-shoot and bang itself on the floor. I thought it would destroy itself before I would be able to stabilize it. Fortunately, after adding some derivative gain (Kd), the robot was able to keep itself up-right, with some oscillation. When I added some disturbance (a little push) the robot would travel horizontally until it eventually fell over. After adding some integral gain (Ki) the robot seemed to stabilize itself very well.  My robot at this point would re-stabilize fast enough from a slight push, with some oscillation and small travel in the horizontal direction.My first set (and my favorite) set of gains were: Kp = 8, Ki = 0.5, Kd = 10. These gains won't mean much to you, as they are fit for my robot/system, which is dependent on my control loop, motor response, motor torque, speed resolution, robot height...and it goes on...). You would have to experiment with your system for a suitable set of gains.

What I can do, is share my next set of gains as a comparison to show the change in performance. I thought it would be good to rid of the oscillation, so I increased the derivative gain from 15 to 60. In order to keep my robot stable I had to increase the Kp and Ki gains slightly. This was good, in that my robot did not oscillate so much while trying to keep its balance in the non-added disturbance case. And when I did add a slight push, the robot corrected its angle almost instantly. However, the robot was forced to travel in the horizontal direction for a much larger distance to maintain this response.

I learned that added disturbances (like pushes) is like adding energy to the system. In order to deal with the extra energy, the robot can either oscillate back and forth a few times while minimizing travel in the horizontal direction, or it had to travel in the horizontal direction quite more just to maintain the angle I had programmed.

Anywho, here is a video of the robot in action:

I have a github site, so I'll post the code there. I have C, and C++ library for the MPU9150. The C library however is dependent upon the PSOC4, but you can strip what you need rather easily from it. I'll also include an application I wrote in processing (see which the robot used to communicate to my PC so I could understand what was going on a little better:

Link to the code:


Tuesday, July 29, 2014

Autonomous Robot with Zynq

In an earlier post, I talked up the Xilinx Zynq (an IC with both FPGA and Microcontroller). In my Advanced Embedded System Design course, we had to build an autonomous robot that can navigate around with some form of intelligence and seek out certain objects to 'destroy'. Now the term 'destroy' was really left up to the students to define; for our robot a laser pointer was used to mark enemies. Identifying 'enemies' was a challenge, so we incorporated a camera so the robot could see and track on its own. Meet our robot (below):

Equipped with:

  •  Zybo
  • OV7670 Camera
  • Arduino
  • Sabertooth Motor Controller
  • IR Proximity Sensors
  • LiPo Batteries (12V + 7V)
  • Pan/Tilt Servo motors
  • Laser
Custom logic was designed onto the FPGA portion of the Zynq, in order to maneuver the robot, control the pan/tilt bracket, capture frame data from the camera, and lastly - 'fire the laser'. The ARM portion of the Zynq was used as the algorithm prototyping environment (in C) to make use of the custom FPGA interfaces. The Arduino was used to configure the OV7670 over the I2C-like communications. The following diagram shows how all components were interfaced.

The reason the OV7670 was chosen was its parallel data interface and low cost ($20), allowing us to obtain an image capture rate of 30fps. However, you get what you pay for in terms of ease of interfacing. I had to design custom logic in VHDL to perform frame captures and store them in block-ram on the FPGA. It's hard to debug what you can't see. None the less, after days of toying around, the Zybo was finally able to see. In order to view what the Zybo saw, I wrote a quick program to transfer images from the Zynq to my laptop over a serial connection (using Processing). Below is the process flow for debugging the images.

The custom FPGA logic which interfaced to the OV7670 was designed to either stream frames into the block-ram, or take a single snapshot and leave it in the block-ram. The FPGA had to interface/synchronize to the vsync, href, pixel-clock, and 8-bit data of the OV7670. It also needed to interface with the ARM processor and to FPGA-based block-RAM. Below is the block model of the custom camera control as shown in Vivado.

We couldn't have done it without camera register settings provided by this source: Hamsterworks - OV7670

Otherwise the images didn't turn out so well:

I'll post some videos of the robot in action soon...

Tuesday, April 29, 2014

Xilinx Zynq7000 and the Zybo

If you are into getting your hands on the latest embedded technology, the Zynq7000 is a great platform to get familiar with. Xilinx, a leader in FPGA design (in which I have no affiliation with), partnered up with ARM to create the very first microprocessor with on-chip FPGA (or is it an FPGA with an on-chip processor?).

This technology combines the power of using an FPGA to perform high-speed paralleled tasks with a dual-core ARM processor with standard Micorcontroller peripherals (CAN, I2C, SPI, UART etc.). I was lucky enough to be introduced to this device in my Advanced Embedded System Design course at Oakland University, and will continue working with it as long as I can continue to make sense out of partitioning my embedded design projects into both 'some hardware' and 'some software'. I purchased the Zybo development board made by Digilient, however there are others out there (Zedboard, microZed by AVNet)

To paint a better picture of how this is so useful, imagine using a microcontroller to both interface to a camera and perform image processing in order to make a decision (maybe you are trying to track an object). In terms of cameras you are able to interface to, you are left with those with slower serial interfaces (SPI, UART). If you really wanted to interface a microcontroller to a camera with a faster parallel interface, you would have to spend a little more money to get a fast enough controller. But this doesn't leave your controller much time to do anything else other than image-processing and frame-grabbing! On the flip side, if you were to use an FPGA it becomes harder to actually implement your decision making and takes longer to develop and is harder to debug.

Having an FPGA and Microcontroller on the same IC solves this exact problem. Now, you can develop your frame-grabbing for your higher-speed camera and even do some image processing on the FPGA. In this way, the FPGA can be treated as a custom co-processor for your application running on the Processor. And the FPGA to Processor interface is seen by your application is either 'just another memory mapped peripheral' or even an 'external interrupt'.

The example I used comes directly from my project, where I did use the FPGA to communicate with a faster camera (well faster than UART or SPI based) and allow the processor to dictate what kind of operations to perform on the image, or even stream the image to the processor's RAM for streaming to a remote PC via UART. This was for an Autonomous Robot project, which I will add posts on in the near future.

I recommend checking this technology out. It is great to see innovations such as this, because they have the power to take industries into new directions. If you are interested in getting yourself one, I suggest either the Zedboard or the Zybo.

Link to Digilent's Zybo

Friday, April 20, 2012

Audio Localization

Just a little introduction on my interest in this topic:
I've always been intrigued by the way our mind is configured to interpret sound signals. To those of you with two working ears: Ever notice when you hear a noise, you know which direction it came from?... I mean you just 'know'; you don't have to sit down, grab a pencil and notepad and plot waveforms to triangulate the angle in which the sound likely originated. These calculations are done in the background of our mind. That's right, you and I (and even our pet cats) are pre-programmed to utilize these functions without having to 'think' about it. This way we can use our main processor-time for more important tasks.

I wanted to experiment with methods that the brain uses for indicating the direction of sound. A little background on the two methods:
  • Interaural Level Difference: used to describe difference in amplitude between two or more sensors
  • Interaural TIme Difference: used to describe difference in arrival time of two signals
A few links on the topic:
For simplicity, I focused on the 1st method and implemented an 'object tracking' approach.

Components used:
  • Arduino Uno
  • 2 Phidget Sound Sensors
  • Continuous Servo Motor
To continue to maintain simplicity, I chose the Phidget Sound Sensors because they outputs a 0-5 Volt signal representing  measured volume (opposed to a raw signal from a microphone). This also allows for a slower processor (such as the Atmega328) to be quick enough for the task. Below is a pic of the system (made for a class project).

Here is a functional diagram of the system I drew up:

A high-level schematic

A picture of the project

And at last, a video! (May be loud). We used the "Air-Horn" Phone-App

Video with improved code

Tuesday, May 3, 2011

Return of the RC car!!!

I've been very busy since the last post, but I finaly got back into the groove for the RC car. I wanted this car to be completely modifiable so the inside of the electronics control box = breadboards + velcro (which works pretty darn good). Here is a photo of the wiring.

Included in this box:
1. Boarduino (Atmega 328) controller
2. 18v15 Pololu Motor Driver (rear motor drive)
3. Dual AxisCompass Module - HMC6352
4. Xbee Pro 60mW U.FL module: for wireless comunication

I had to replace the stock steering motor/potentiometer setup with a servo motor to put ease on the programming (I'm now able to eliminate the steering control loop code). I also broke the ICSP header outside the box for programming (I had to wire up a switch to disconnect both the external power and the Xbee's transmit line from the microcontroller in order to externally program).

I'm working on the code for wireless control/feedback. Eventually I intend on adding a camera, ultrasonic rangefinder and GPS module.

Anyway, here is a short video on my car's first run with both functional steering/driving. (given the small area, I had it just do circles in my kitchen).

Wednesday, August 4, 2010

RC Car

I'm currently putting together a new project where I'm attempting to automate the control of an old RC Car. I've ripped out the original guts except for the the drive motor and steer motor. After numerous attempts at a DIY motor controller for the drive motor (and popping through transistors like popcorn), I've come to the conclusion that this motor needed a driver who could handle the spike currents without melting under pressure. Here is a list of all the parts that will be going into this project, and later I will provide updates on my progress.


1. Original RC Chasis (including suspension and wheels/tires).
2. Original RC 7.2V Battery Packs (2)
3. Original rear-wheel drive motor (specs unknown except that it draws 3.5A current running, and 15A stall).
4. Orignal steering motor + original feedback potentiometer (specs also unkown)
5. Boarduino (Atmega 328) controller
6. 18v15 Pololu Motor Driver (to properly drive the rear motor)
7. Dual AxisCompass Module - HMC6352
8. Xbee Pro 60mW U.FL modules + antenna (2): for wireless comunication
9. A few 2n2907/2n2222 BJTs for driving the steering motor

Sunday, April 25, 2010

Light Follower - 2 Axis

Here it is - My rendition of a 2-axis light follower. I went ahead and clipped the IR Cam from the wiimote to cut down on weight. I put the camera along with the required oscillator in a project box (from radio shack). I also included an LED on the box; it lights up when light is seen by the camera. A serial port is used to connect the box to the microcontroller.
Here are some Pictures:

Here is a video:
Here is a link to the code:

Wednesday, March 17, 2010

Nintendo nunchucks as Orientation Sensors

My senior design project was to make a wirelessly controlled robotic arm, that mimics human arm movements. The closest we got was the movement of a shoulder joint and an elbow joint at a very high accuracy and low time delay.

I made a sensor system out of 2 Wii Nunchucks, an Arduino and some external circuitry to switch between nunchuck sensor reading. At a deadline of one of our presentations, a plastic gear from our arm had chipped some of its teeth so we weren't able to give motion demos. I whipped up a program using Processing ( to communicate with the Arduino and move a 3D simulation of an arm based on the sensor outputs.

I thought I'd share a few screenshots of the program. I'll have both the Arduino code, and processing code up soon.

Processing is a great program for doing graphical manipulations. It can also compile code to execuatable files.

Wiimote light follower with servo

Everybody is familiar with the infamous Wiimote. When I look at it, I think about all the useful sensors/gadgets that this little 40$ package (new) comes with. Recently I've been playing with the IR Camera (It's really just a light sensing camera with an IR Filter).This particular camera is a standalone module that outputs coordinates of the 4 brightest "images", all via I2C communication.

I've only seen hacks with the Wiimote cam where the camera is desoldered/removed from the Wiimote. However, at 40$ a pop that seemed like a waste of a perfectly good Wiimote. Instead of removing the cam, I only made 1 small modification, which was drilling a very tiny hole near the camera and soldering a connection to it's "Clock" pin (which needs to be a 24MHz sine wave to replace the internal oscillator). Once you have this done, all you need to do is plug a cord into the Wiimote peripheral port to use anything on its I2C bus.
Moving on, I attached the Wiimote on a homemade stand that was fixed onto a continuous-rotation servo motor (servo without feedback). Add a little duct tape, and that servo isn't moving for at least an hour.

Using an IR Camera library already created (thanks to Hobley –, I used an Arduino to receive points from the camera and follow the 1st object (light source) that's noticed.

The code can be found here:


Monday, March 15, 2010

Two Wii Nunchucks with one arduino

In the midst of a senior design project, it was decided that we wanted to use 2 Wii Nunchucks as accelerometers to measure orientation of a human arm ( 1 for the upper arm and 1 for the forearm). In understanding I2C communication, there is no way to use 2 Nunchucks on the same I2C bus without some sort of external circuitry. (All nunchucks have the same slave address, leaving nothing to distinguish between the two when attempting to receive data).

I drew up a simple and cheap solution to interface two (or more) Wii Nunchucks on the same I2C bus. This is useful for projects that require multiple accelerometers at a cheap price.

Here is all you will need:

2 npn switching transistors (I used 2N3904)
2 current limiting resistors (I used 200 Ohm)

Just connect all nunchuck Power (PWR), Clock (SCL), and Ground (GND) wires to the same corresponding spots on your microcontroller. The microcontroller's SDL can be connected to the outputs of both transistors.

Programming notes:

In order to perform a read, all you have to do is set the pin of the corresponding transistor to HIGH (5v in our case), write/read to the I2C bus, then set that pin to LOW (0v)to disconnect that nunchuck from the bus. Also, during start up you must initialize each nunchuck individually in order to operate both nunchucks correctly.
Good luck,
B Dwyer (aka: johnnyonthespot)