Voice Activated Robo Car on Microcontroller with TinyML

Using voice recognition with the in-built microphone, Wio Terminal (Voice Activated Robo Car) will be able to recognize the go, stop and background noise by TinyML.
Voice Activated Robo Car on Microcontroller with TinyML

Story of Voice Activated Robo Car

The microphone on Wio Terminal will only able to detect the sound intensity (check the schematic below or download to view full schematic) and hence, it will be challenging for it to do speech recognition for different words. Hence, I thought of some creative ways to capture the sound input and then train them with the Edge Impulse engine to see if it is able to do simple speech recognition. We will see the result at the end later.

Wio Terminal Mic Scematic

Curious how it’s possible? Check out the detailed steps below to make one yourself!

In this detailed tutorial, we will cover the following:

  • What’s UART Serial Communication?
  • UART Serial Communication between Wio Terminal and uKit Explore
  • 1.0 Train an Embedded Machine Learning Model Using Codecraft
  • 2.0 Arduino Text Code Modification
  • 3.0 Program the Robo Car (uKit Explore – Arduino Mega 2560 based)
  • 4.0 Expected Result
  • Giveaways

What’s UART Serial Communication?

The UART, or universal asynchronous receiver-transmitter, is one of the most used device-to-device communication protocols. This article shows how to use a UART as a hardware communication protocol by following the standard procedure.

When properly configured, the UART can work with many different types of serial protocols that involve transmitting and receiving serial data. In serial communication, data is transferred bit by bit using a single line or wire.

In layman term, UART allows an embedded device such as Arduino to send data over to another Arduino via the TX (transmit) and RX (receive) line as shown below.

TX (transmit) and RX (receive) line

Source: https://learn.sparkfun.com/tutorials/serial-communication/all

I often use UART as the communication protocols between embedded devices. Let me give you an example case study. Well, Arduino UNO has no built in WiFi and hence it’s not possible to do IOT related projects. With the understanding of the basic UART serial communication, I was able to leverage on the ESP8266/ESP32 as the co-processor to the Arduino UNO so that the data collected from the sensors attached to Arduino UNO will be sent over to ESP8266/ESP32 to send over to the cloud platform such as web server, Blynk or¬†FAVORIOT.

With the ability to exchange data between two embedded devices (example Arduino UNO and ESP8266/ESP32), the possibility is unlimited!

Example as below where I am trying to establish a simple serial communication between TTGO T-Display (Serial2 line) & uKit Explore (Serial line).

simple serial communication between TTGO T-Display (Serial2 line) & uKit Explore

 

We will see that how this concept being applied in my voice activated Robo Car project.

UART Serial Communication between Wio Terminal and uKit Explore for Voice Activated Robo Car

As we check the Wiki page for Wio Terminal, there’s¬†TX/ RX pin available on the pin 8 and 10.

UART Serial Communication

Whereas for uKit Explore, as it has the headers like in Arduino UNO pinout, there’s also TX/RX pin available on the pin D0 and D1. You can check the full uKit Explore pinout here:¬†https://ubtechedu.gitbook.io/ukit-explore/v/english/ukit-explore-quick-start/pinout

uKit Explore pinout

1.0 Train an Embedded Machine Learning Model Using Codecraft

In this first section, our goal is to create an embedded machine learning model (voice recognition) using Codecraft platform

1.1.0Create and select models

Go to¬†https://ide.tinkergen.com/.¬†Select¬†“(for TinyML) Wio Terminal”.

Wio Terminal

1.1.1 Create the “Wake-Up Words Recognition (built-in microphone)” model

Click on¬†“Model Creation”¬†on the embedded machine learning box on the middle left. Then select the “Wake-Up Words Recognition (built-in microphone)” as shown below.

Enter the name for the model according to the requirements.

Wake-Up Words Recognition (built-in microphone)

Click¬†Ok¬†and the window will automatically switch to the “Data Acquisition” interface.

Data Acquisition

1.2.0 Data Acquisition (on-board)

1.2.1 Default Labels

There are 3 default labels (hi wio, background and other words) that are automatically created for you.

Default Labels

You can use it without any changes unless you want to have different names for your labels. For my case, I changed two of the default labels as below:

  • hi wio¬†changedto¬†go
  • other words¬†changed to¬†stop

1.2.2 Collecting data from modified labels and Data Acquisition Program Modification

Click on the “hi wio” label, you will be prompted to change the label name. I renamed the label to¬†go¬†and click “Ok”

change the label name

Do the same for the “other words” label. You will now see your labels as shown below:

"other words" label

IMPORTANT: Now, you have to remember to change the labels on the default data acquisition program to reflect the correct modified labels:

default data acquisition program

default data acquisition program

1.2.3 Connect Wio Terminal and Upload Data Acquisition Program

Connect Wio Terminal to laptop using the USB-C Cable. Click on the Upload button, and this action will upload the default data acquisition program. Typically, it takes around 10 seconds to upload. Once successfully uploaded, a pop-up window will appear to indicate ‚ÄúUpload successfully‚ÄĚ.

Click ‚ÄúRoger‚ÄĚ to close the window and return to the data acquisition interface.

Caution

Note:¬†You need to download ‚ÄúCodecraft Assistant‚ÄĚ to be able to Connect and upload code on Codecraft online IDE.

Caution: For the web version of Codecraft, if you don’t install or run the Device Assistant, you may get the message in the image below that you haven’t opened the Device Assistant yet. In this case you can check this page for further information:¬†Download, installation and “Device Assistance” Usage.

need to download ‚ÄúCodecraft Assistant‚ÄĚ

1.2.4 Data Acquisition

In the upper right hyperlink, you will find a step-by-step introduction to data acquisition.

Follow the instructions to collect data accordingly to your modified labels.

Pay attention on below:

  • Wio Terminal button location (A, B, C)
  • Animated gif has been accelerated; the actual action can slightly slow down.
  • Please notice the red tips.
  • Point the curser over Description Texts for more detailed content

After collecting the sample data for the 3 labels, now, the data acquisition step is completed. I collected 15 seconds for each of the labels.

1.2.4 Data Acquisition

1.3: Training and Deployment

Click on ‚ÄúTraining & Deployment‚ÄĚ, and you will be seeing the model training interface as shown below.

1.3: Training and Deployment

Waveforms for go, background and stop raw data are as shown below for reference. (can be viewed from the “Sample data” tab.

go

If you observe the waveform, for label “go“, my sound input is “go, go, go!” In 5 seconds window, I recorded twice with about 1 seconds rest.

waveform for label "go"
waveform for label “go”

background

For label “background”, I will just hold the Wio Terminal on my hand to capture the surrounding noise inputs. Hence, you could see the sound input level is not 0 as there’s some surrounding noise such as fan rotating sound level.

waveform for label "background"
waveform for label “background”

stop

For label “stop”, to differentiate it with “go”, I try on my sound input as “stoppppp………” for about 3 seconds.

waveform for label "stop"
waveform for label “stop”

1.3.1 Select neural network and parameters

Select the suitable neural network size: small, medium and large

Set parameters:

  • number of training cycles (positive integer),
  • learning rate (number from 0 to 1)
  • minimum confidence rating (number from 0 to 1)

The interface provides default parameter values of training cycles of 50, however, the accuracy was not very good. Hence I changed the training cycles to 100.

Select neural network and parameters

1.3.2 Start training the model

Click ‚ÄúStart training‚ÄĚ. When you click ‚ÄúStart training‚ÄĚ, the windows will display ‚ÄúLoading..‚ÄĚ! Wait for the training to be done!

The duration of ‚ÄúLoading..‚ÄĚ varies depending on the size of the selected neural network (small, medium and large) and the number of training cycles. The larger the network size is and the greater number of training cycles are, the longer it will take.

You can also infer the waiting time by observing the ‚ÄúLog‚ÄĚ. In the figure below, ‚ÄúEpoch: 68/500‚ÄĚ indicates the total number of training rounds is 68 out of total of 500 rounds.

Start training the model

After loading, you can see “TrainModel Job Completed” in the “Log”, and “Model Training Report” tab will be appeared on the interface.

TrainModel Job Completed

1.3.3 Observe the model performance to select the ideal model

In the ‚ÄúModel Training Report‚ÄĚ window, you can observe the training result including the¬†accuracy, loss and performance¬†of the model.

If the training result is not satisfactory, you can go back to the first step of training the model by selecting another size of the neural network or adjust the parameters and train it until you get a model with satisfactory results. If changing the configurations do not work, you may want to go back to collect the data again.

Model Training Report

For my case, I am able to achieve 100% accuracy with modification on the training cycles to 100.

1.3.4 Deploy the ideal model

In the ‚ÄúModel Training Report‚ÄĚ window, click on ‚ÄúModel Deployment‚ÄĚ

Model Deployment

Once the deployment is completed, click ‚ÄúOk‚ÄĚ to go the ‚ÄúProgramming‚ÄĚ windows which is the last step before we deploy the model to the Wio Terminal.

 deployment is completed

1.4. Programming & Model Usage

Alright, so now we done trained the model and the fun part of integrating Artificial Intelligence (Machine Learning in this case) with robotics (a Robo Car) using the UART communication protocol.

This is the sample program that created from the block programming interface:

Programming & Model Usage

Let me try to explain what I am trying to achieve with this code. I make it into bullet points for easy understanding:

  • As previously, we use the if-else conditional statements to evaluate the confidence of the labels.
  • If the confidence of “go” is greater than 0.8 (80%), I will print “1” on the serial terminal.
  • If the confidence of “stop” is greater than 0.8 (80%), I will print “2” on the serial terminal.
  • Otherwise, if the confidence of “background” is greater than 0.8 (80%), I will print “0” on the serial terminal.

Ok, so for now, just remember 3 different conditions:

“go” > 0.8, the command is ‘1’

“stop” > 0.8, the command is ‘2’

“background”. the command is ‘0’

After some research online for the pinout of TX/RX for Wio Terminal, here’s my findings:

  • We have the¬†TX/ RX pin available on the pin 8 and 10¬†that can be used to connect to another board (uKit Explore in this case).

TX by RX for Wio Terminal

  • The serial line you can access from the 40 pin header is¬†Serial1¬†instead of usual Serial which basically shows the output via Serial Terminal.

Alright, so take note on the 2 important findings as above as it will be the critical part for our project!

If we have a look at the text coding for the corresponding blocks code, you will notice that the Serial.print is NOT using the Serial1 line. Hence, this leads us to our second step which is to continue our coding on Arduino IDE for customization.

Serial.print

 Serial.print
Serial.print

2.0 Wio Terminal Arduino Text Code Modification

In this first section, our goal is to modify the Arduino text code to use Serial1.println instead of Serial.println so that the data can be transmitted via the TX/ RX pin available on the pin 8 and 10.

2.1 Toggle to the Text Code area & Copy The Text Code

Wio Terminal Arduino Text Code Modification

On the text code area, copy all the code by pressing on CTRL + A to select all the code.

Open up Arduino IDE, create a new file, paste the code into the empty sketch by pressing on CTRL+V. Proceed to save the sketch with the desired name.

paste the code into the empty sketch by pressing on CTRL+V

2.2 Copy the Edge Impulse TinyML Arduino Library

So, the question now is, how do I get all files containing the data for the trained model on Codecraft platform onto Arduino IDE?

Well, it’s easy. Follow the steps below:

  • Navigate to¬†C:\Users\<User_Name>\AppData\Local\Programs\cc-assistant\resources\compilers\Arduino\contents\libraries
  • Find the folder name that has the same number with the Edge Impulse header file on top of your Arduino text code (in my case it will be¬†47606)folder name that has the same number with the Edge Impulse header file on top of your Arduino text code
  • Copy¬†the entire ei-project_47606 folder and paste it onto¬†C:\Users\<User_Name>\Documents\Arduino\libraries\

Folder Libraries

2.3 Modification on the Serial.println function

Lastly,¬†let’s look back at the code, scroll down to the void loop section and¬†modify the¬†Serial.println¬†function to¬†Serial1.println¬†instead.

modify the Serial.println

2.4 Upload the Code

Finally, we are ready to upload the code.

Make sure you have installed the Wio Terminal board support package. If not, please refer to the “Get Started with Wio Terminal” guide on Seeed’s Wiki.

Make sure you selected the correct board and COM port before you upload the code to Wio Terminal.

Upload the Code

Alright, we have done the necessary modifications! Right now, the below condition still true for each cases and this time the command ‘1’, ‘2’, and ‘0’ will be able to send out from the¬†TX/ RX pin available on the pin 8 and 10.

“go” > 0.8, the command is ‘1’

“stop” > 0.8, the command is ‘2’

“background”. the command is ‘0’

Source: Voice Activated Robo Car on Microcontroller with TinyML

About The Author