Infrared Camera on the PIC32

Hardware & Software Tradeoffs

One significant decision that we made was to de-synchronize the camera from the PIC32 after taking a frame. De-synchronizing gives us an extended time frame to process and print the data to the TFT before we obtain another frame of data. De-synchronization after every frame is considerably simpler and easier to integrate at the cost of real-time efficiency as explained below.

The camera continuously streams thermal data while synchronized. The solutions with periodic processing were complex and challenging. If we cause a time delay through yielding to other protothreads or performing other operations in the original protothreads, we lose synchronization between the PIC32 and the camera. Because we needed to process and print the data, putting the data transaction software in a protothread while maintaining synchronization is challenging.

One possible solution is to maintain synchronization by putting the data transaction software in an ISR. The problem is that the processing and printing algorithms take substantially longer than multiple data transactions and requires storage for the input data. The camera streams at 10 Mhz and quickly accumulates more sets of data than the PIC32 can store before even one set of data is processed and printed.

Another possible solution is to discard the new data until the processing and printing are finished for one frame. The problem with this solution is dealing re-gaining synchronization in the event of synchronization loss. We would need to regain synchronization through a time-out, account for an unknown number of discard packets and also keep track of how much of the frame has been transferred to the PIC. This software for re-gaining synchronization would also need to integrate with the ISR’s timing constraints and the timing constraints of the processing and printing algorithms.

Another decision that we are making is to transfer data via single SPI transactions rather than using Direct Memory Access (DMA). The main issue is the variable length of the input data. The variable data length causes to us to not know how much memory to allocate for the data transfer and where the discard packets stop. This makes DMA unusable because the only solution is overestimating memory space allocation when space is limited.

Another design decision that we are making is to have the user-interface for mode transitions be primarily software-based. We designed a pushbutton system that rotated between Raw14, AGC and RBG888 respectively. The camera cannot be in RBG888 mode without having AGC already enabled. Limiting the user to this order prevented the user from ever trying to enable RBG888 without having AGC enabled.

Relationship of our design to available IEEE, ISO, ANSI, DIN, and other standards.

We used the NXP and IEEE standards for I2C [1] and SPI [2] Communication Protocols.

Existing patents, copyrights, and trademarks which are relevant to our project.

Many patents exist for thermal imaging, but we did not examine any of them while we were creating this project.

Software Design

There are 4 protothreads in our system: pt_tick, pt_datatrans, pt_processprint, pt_button. The purpose of each protothread is outlined below in Figure 2.

SPI Data Transactions

We opened up the SPI channel using the function SpiChnOpen with 8-bit transactions in SPI mode 3 (CPOL=1, CPHA=1). We set the SPI Channel to operate at a frequency of 10 MHz by dividing the overall clock frequency (40 MHz) by 4. The thermal camera can operate between 2 MHz and 20MHz [3].

In protothread_datatrans, we begin by synchronizing the PIC32 and the camera. The first step is to stop the camera for at least 200 ms. The next step is to begin taking data through SPI transactions. The camera transmits discard packets until it is ready to begin transmitting data from the frame. In a discard packet, the first four LSB bits of the packet consist of the Hexadecimal F.

We initiate this data transmission with the following steps: setting the 8-bit mode, writing junk data to the camera via the MOSI line, waiting for the transaction to end and then reading the SPI packet via the MISO line. If we detect a discard packet by checking the first byte of the packet, then we do not store the rest of the incoming packet in an array. When the first non-discard packet comes from the camera, we store the packet and break out of the loop. We then transition into a for-loop where we initiate the data transactions for the rest of the frame. The transmitted data is stored in an array that another protothread will process and print. The final step is to let the other protothreads know that one frame has arrived in its entirety.

Image Processing Algorithm

Once a frame has been taken, we store the data in a double array named frame_data. frame_data contains a row for each packet in the frame and a column for each byte in the packet. Every entry in frame_data is a one byte chunk of thermal data that represents a specific pixel in the camera’s area of vision.

The TFT contains 240 pixels in the x-direction and 320 pixels in the y-direction, and the camera contains 60 pixels in the x-direction and 80 pixels in the y-direction. This causes a 4-by-4 pixel square on the TFT to represent one pixel on the camera. We created two separate arrays of x and y coordinates, LineCoords and PixCoords, in main. The index of an element in the frame_data array is a pixel coordinate on the camera, and the value of the element is the coordinate of the upper left corner of a 4-by-4 square on the TFT.

We then implemented this mapping by parsing through the frame_data array and printing a colored, 4-by-4 square to the TFT. The coordinates of the square’s upper left corner are corresponding to an element of PixelCoords and LineCoords. See Figure 6 for further explanation.

When we switched to 4-Scr mode, we printed 4 separate images onto 4 separate sections of the TFT. Each section consists of a 120-by-160 pixel section on the TFT which is one fourth of the screen. To map the data frame to this truncated pixel area, we printed 2-by-2 squares to the TFT for every pixel from the camera. To print the square, we modified the coordinates used in 1-Scr mode by halfing and offsetting the coordinates. We repeated this process for all four sections on the TFT.

The top left and top right images in 4-Scr mode are processed at an angle. This is due to the angle of the camera when it is tilted and rotated by the servos. The camera perceives vertical images at approximately a 35 degree tilt. The top left image is 35 degrees Counterclockwise (CCW) from vertical and the top right is 35 degrees Clockwise (CW) from vertical. To account for this, we rotated the top left image 26.6 degrees CW and the top right 26.6 degrees CCW. The resolution of the TFT limited the angle rotation of the image.

To rotate by 26.6 degrees, we shift blocks of the image. The following is an example of printing a 26.6 CW rotated image to the top left quarter of the TFT screen beginning with the leftmost column. The first two columns of pixels fill the first two columns and rows 1:120 on the TFT. The next two columns fill columns 3 and 4 and are shifted down one pixel such that they fill rows 2:121. The next two columns fill columns 5 and 6 and fill rows 3:122. This process repeats until column 160 is reached which fills rows 81:200.

Equation 1: Top Left Vertical Rotational Offset

The equation for the top left vertical rotational offset is above.

To rotate the top right image 26.6 degrees CCW, the same approach is taken except that we begin at column 161 and shift up instead of down.

Equation 2: Top Right Vertical Rotational Offset

The equation for the top left vertical rotational offset is above.

Changes in Mode

The thermal camera’s default mode is Raw14. The data in Raw14 is a reading of the intensity for each pixel. In Raw14 mode, the camera transmits two bytes of thermal data for every pixel in the camera. This causes the payload to be 160 bytes because there are 80 pixels. Given the MSB byte for the pixel, we shift the byte left. We receive the LSB byte next and concatenate it with the MSB byte. The concatenated value is the color that we will print on the TFT.

Thermal data in AGC is only one byte per pixel. Even though only 80 bytes are necessary, the payload is 160 bytes. There is no change in communication protocol from Raw14 to AGC. The color is the LSB byte of the data transferred for the a given pixel. The MSB byte is just junk data.

RGB888 is a separate mode where the data for each pixel contains 3 bytes - a Red byte, a Green byte and a Blue byte. The total payload is 240 bytes. The color input to the TFT function is only 16 bits. We transformed the RGB888 data to RGB565 by concatenating the 5 LSB bits of the Red data, the 6 LSB bits of the Green data, and the 5 LSB bits of the Blue data.

Transitioning between Modes

The protothread_button is used to detect downward pushes for multiple buttons. A button push will trigger a mode change for the system. Button B toggles the printing mode between 1-Scr and 4-Scr. When Button B is pressed, the 4-Scr flag is toggled in one direction and the 1-Scr flag is toggled in the other direction. In other words, if 4-Scr flag is True, then it will become False. If 1-Scr flag is False, then it will become True. The result is that the flags always opposite to each other.

Button A controls changes to the mode of the outputted thermal data. It causes the system to rotates between the three aforementioned data modes. When Button A is first pressed, the protothread transitions from Raw14. On a second push, the protothread transitions to AGC. On a third push, the protothread transitions to RGB888. Finally, on a third push, the protothread loops back to Raw14. In each instance, the button press triggers an I2C function that communicates with the camera’s CCI.

Button C enables and disables the PWM signals to the servos. This turns the servos on and off, and controls the camera’s field of view. When the third button is pushed, the protothread toggles the flag control PWM signal output.

See Figure 10 for pushbutton labeling.

Pulse-Width-Modulation (PWM)

The pan-tilt servo sub-system uses two SG92R Mini Micro Servos to pan left and right and tilt up and down. The servos require a new PWM signal every 20 ms [4]. In other words, the signals are issued at a frequency of 50 Hz. The pulse length can range from 1 ms to 2 ms.

A pulse of 1ms corresponds to a -90° servo rotation. A pulse of 1.5ms corresponds to a 0° servo rotation and a pulse of 2 ms corresponds to a 90° servo rotation. To generate this signal, two output compare modules (OC) were used with two timers. To set the PWM Output Signal Frequency to 50 Hz, Timer 2 and Timer 3 were both set with a pre-scalar of 256 and time period of 3125 ticks from the system. This results in the following equation:

To set an angle on a servo, the pulse time is set between 1ms to 2ms. The following equation determines what number to assign to the duty cycle variable:

OC3 controls the Tilt servo. OC4 controls the Pan servo. Both are configured as a PWM source. The servos rotated between 4 different pan-tilt servo coordinates that form a rough square. Once the servos finish rotating, the PIC32 processes one frame of data. In 4-Scr mode, 1 servo location out of the 4 represents one fourth of the TFT screen.

I2C Data Transactions

We are using I2C protocol to communicate with the CCI on the Lepton board. The I2C.h header file contains the functions which setup and transmit the data to and from the camera. The LeptonCommand function takes in the following inputs: a read or write bit, command, moduleID, and commandID. The read or write bit tells the camera whether to receive or send data. The command variable sets whether or not the camera sets, gets or runs something. The camera has multiple modules. In this program, we use the OEM and AGC modules for the moduleID. The AGC module controls the Automatic Gain Control settings. These settings affect image brightness and contrast. The OEM module contains commands for FLIR’s OEM customers. The commandID identifies which command the camera uses.

The following procedure is followed to communicate with the CCI:

1. Poll the status register until camera is ready for a new command.

2. If executing a command rather than solely reading, send the command to the camera using the camera’s received data registers.

3. Write the number of data words written to the received data length register.

4. Writing the desired command ID to the Command Register.

5. Polling the Status Register to determine when the command is completed.

Whenever a command is executed, the camera raises the busy bit for the duration of the command. When the busy bit is clear, we know that the CCI is ready for a new operation. The status register for this busy bit is 0x02.

The Camera’s data register address is 0x08. In our application, we only write 2 bytes of data to the camera. Thus, we write 0x02 to the data length register which is located at 0x06.

One other piece of information to note is that AGC is enabled before RGB888 because RGB888 cannot function without having AGC enabled.

Here is the ordered stream of data that was written to the CCI to enable AGC mode and then to enable RGB888 mode.

Results

The resulting images from our system are below. The specifications of the images are discussed in other sections.

Conclusions

Results vs. Expectations

We were pleased with the results of this project. Our system constructed images on the TFT that were sensitive to a few degrees in temperature. The sensitivity in degrees was demonstrated by a user who rubbed their hands together to create a small heat change on their hands. The change in heat was noted by the image on the TFT. See Figures 23 and 24 for further explanation.

The system also created images that accounted for more dramatic heat differences. We placed a cup of hot water and cup of cold water in front of the camera, and the image on the TFT showed a substantial color change between the two cups. See Figure 22 for further explanation. We also placed a soldering iron in front of the camera, and an even more dramatic color change appeared on the TFT screen. See Figures 16-20 for further explanation.

One element of the project that we could have improved was to do a thorough timing analysis of the entire system. We have multiple yield statements scattered throughout the software. A yield statement causes a protothread to allow other protothreads to take over the PIC32 for a specified number of seconds rather than the original protothread continuing to run without interruption. Some yield statements have a specific purpose, while others are general estimate of the time needed. The rate of imaging could potentially be increased by reducing the amount of time that the a protothread yields to another protothread. This idea is especially prevalent for the yield statements where the specified amount of yield time is an estimate.

While synchronization is maintained, the Lepton Camera sends a unique frame three times before it sends an new frame. In our system, we only receive one frame of data. We could have received two or three frames of the same set of data and averaged the values at each pixel.

Further work can be done to enhance the stitching algorithm. The stitching between the top and bottom images are continuous, but the stitching between the top left and top right needs work. Images are duplicated near this intersection and the rotation is not fully corrected. Furthermore, the rotation equations distort the images.

Applicable Standards

Our implementation of SPI communication protocol was in adherence to the IEEE standard. The FLIR Lepton Camera registers are 16-bits wide instead of 8-bits, so only 16-bit data transfers are allowed. This is a break from the NXP standard. Other than this difference, our implementation adheres to the NXP Standard.

Intellectual Property Considerations

We modeled our design after standard examples given throughout the class. We modeled our I2C code after the example code from the FLIR Lepton Github Open Source code [5].

We used a protothreads library created by Adam Dunkels, and a TFT library created by AdaFruit and Syed Tahmid Mahbub. We used Microchip-generated peripheral software libraries that were constructed for the PIC32 [6]. All three libraries were released into the public domain.

We are not reverse-engineering a design. Thermal imagers are commercially available, so there are no patent opportunities. There is a publishing opportunity for this project because the detailed use and integration of a Microcontroller, Servo System and a Thermal Camera is an educational opportunity.

Ethical Considerations

This project highlighted our desire to fulfill IEEE standard 5. Standard 5 states that we must, “to improve the understanding by individuals and society of the capabilities and societal implications of conventional and emerging technologies, including intelligent systems.” Thermal images are easy for the average person to understand because the vast majority of people understand heat. A thermal imaging system can also be applied to common problems like night vision and heat loss analysis in homes.

This project also incorporated our adherence to IEEE standard 6. Standard 6 admonishes us, “to maintain and improve our technical competence and to undertake technological tasks for others only if qualified by training or experience, or after full disclosure of pertinent limitations”. We choose a project that was within our technical grasp and allowed both of us to make a substantial technical contribution.

Furthermore, our imaging system did not discriminate against or injure anyone in accordance with Standards 8 and 9 respectively. Standard 8 states that we must “treat fairly all persons and to not engage in acts of discrimination based on race, religion, gender, disability, age, national origin, sexual orientation, gender identity, or gender expression.” Standard 9 states that we must “avoid injuring others, their property, reputation, or employment by false or malicious action.”

Source: Infrared Camera on the PIC32