App Controlled Robot using Raspberry Pi with AI Features

Overview

In this guide, you will learn how to build an App Controlled Robot using Raspberry Pi that has AI Features. For this, we will use a Robotic Car Kit called Picar-X Robot from Sunfounder.

The app is called Sunfounder Controller and is available for both Android & IOS. The APP integrates Button, Switch, Joystick, D-pad, Slider, and Throttle Slider widgets; Digital Display, Ultrasonic Radar, Grayscale Detection, and Speedometer input widgets. In addition, this application provides a live video streaming service.

The PiCar-X is an AI self-driving robot car that uses Raspberry Pi as its control center. The Robotic Kit is designed by SunFounder. The Kit comes with a Robot HAT for Raspberry Pi. The HAT integrates the motor driving, servo driving, and preset ADC, PWM, and digital pins to extend the functionality. The PiCar-X’s has a 2-axis camera module, ultrasonic module, and line tracking modules that can provide the functions of color/face/traffic signs detection, automatic obstacle avoidance, automatic line tracking, etc.

Bill of Materials

We need the following components along with the SunFounder Robot Car Kit to make a App Controlled Robot using Raspberry Pi. You can purchase all these components from Amazon links:

S.N.	Components	Quantity	Purchase Links
1	Raspberry Pi Ai Car Kit PiCar-X	1	Amazon \| SunFounder
2	Raspberry Pi 4	1	Amazon \| SunFounder
3	Samsung 18650 Battery	2	AliExpress
4	SD Card 16/32 GB	1	Amazon \| SunFounder
5	SD Card Adapter	1	Amazon \| AliExpress
6	18650 Battery Charger	1	Amazon \| AliExpress

Robot HAT cannot charge the battery, so you need to buy a battery charger at the same time.

Robot Assembly

The Sunfounder Picar-X Robot Kit comes with all the necessary components required for the Robot which include the following components. You need to buy a Raspberry Pi 4 Board and a pair of Samsung 18650 Batteries separately.

The most important part of this Kit is Robot HAT. With the Robot HAT board, the PiCar-X integrates left/right driving motors, servo motors for steering, and the camera’s pan/tilt functions, and pre-sets the Robot HAT’s ADC, PWM, and Digital I2C pins to allow for extensions to the standard functionality of the Raspberry Pi.

Both a speaker and a Bluetooth chip have been engineered into the Robot HAT for remote control of Text-to-Speech, sound effects, or even background music functionality.

To assemble the components together and make a perfect Robotic Car you can follow the Assembly Guide. The PDF Document has all the images and a pictorial view for easy help with assembly.

The Robotic Kit Part details, Assembly guide, HAT parts & functionality, calibration process, Robot Testing and Setting Up Rasbian OS has been explained in PiCar-X Getting Started guide.

Read the tutorial thoroughly to understand the Hardware & Software Setup Part.

Installing Python Modules & Libraries

The OS installation part and setup part can be followed on Raspberry Pi OS Setup. But for this robotic project part, it is recommended to install Raspberry Pi OS(Legacy).

You can either use the HDMI Screen or enable SSH and connect the Raspberry Pi to VNC Viewer using the local IP Address, username, and password. It is best to connect the Robot Wirelessly via a Remote Desktop as we need a remote control for it.

The Picar-X Robot should be turned ON and Raspberry Pi should be connected to the network. Now let’s install all the Python Modules and libraries.

Let’s first update the system using the following commands.

sudo apt update
sudo apt upgrade

1 2	sudo apt update sudo apt upgrade

Then install the Python3-related packages.

sudo apt install git python3-pip python3-setuptools python3-smbus

1	sudo apt install git python3-pip python3-setuptools python3-smbus

Download & install the libraries for Robot-Hat.

cd /home/pi/
git clone https://github.com/sunfounder/robot-hat.git
cd robot-hat
sudo python3 setup.py install

cd /home/pi/

git clone https://github.com/sunfounder/robot-hat.git

cd robot-hat

sudo python3 setup.py install

Running ‘setup.py‘ will download some necessary components. Your download may have failed due to network issues. You may need to download it again at this point. See the following interface, type ‘Y‘, and press Enter.

Now, download and install the ;vilib‘ module.

cd /home/pi/
git clone https://github.com/sunfounder/vilib.git
cd vilib
sudo python3 install.py

cd /home/pi/

git clone https://github.com/sunfounder/vilib.git

cd vilib

sudo python3 install.py

Similarly download and install the ‘picar-x‘ module.

cd /home/pi/
git clone -b v2.0 https://github.com/sunfounder/picar-x.git
cd picar-x
sudo python3 setup.py install

cd /home/pi/

git clone -b v2.0 https://github.com/sunfounder/picar-x.git

cd picar-x

sudo python3 setup.py install

This will download a large number of files, therefore it will take time.

Once the download is complete, run the ‘i2samp.sh‘ script to install the components required by the i2s amplifier, otherwise the picar-x will have no sound.

cd /home/pi/picar-x
sudo bash i2samp.sh

1 2	cd /home/pi/picar-x sudo bash i2samp.sh

Type ‘y‘ and press enter to continue running the script.

Type ‘y‘ and press enter to run ‘/dev/zero‘ in the background.

Type y and press enter to restart the Picar-X.

After restarting, the Raspberry Pi will connect to the Network and will play a sound.

If there is no sound after restarting, you may need to run the ‘i2samp.sh‘ script several times.

Now we need to install SunFounder Controller dependencies & libraries. Install the SunFounder-controller module using the following commands.

cd ~
git clone https://github.com/sunfounder/sunfounder-controller.git
cd ~/sunfounder-controller
sudo python3 setup.py install

cd ~

git clone https://github.com/sunfounder/sunfounder-controller.git

cd ~/sunfounder-controller

sudo python3 setup.py install

All the installation and setup part is complete here.

SunFounder Controller App

The SunFounder controller is used to control Raspberry Pi-based robots.

The APP integrates Button, Switch, Joystick, D-pad, Slider, and Throttle Slider widgets; Digital Display, Ultrasonic Radar, Grayscale Detection and Speedometer input widgets. There are 17 areas A-Q, where you can place different widgets to customize your own controller. In addition, this application provides a live video streaming service.

Install SunFounder Controller from APP Store(iOS) or Google Play(Android).
Open and create a new controller. Create a new controller by clicking on the + sign in the SunFounder Controller APP.

Give it a name and select the Controller type. There are preset controllers for some products in the Preset section, which you can use as needed. You can also customize your own controller by following the steps below.

Add different widgets to this controller. You can add different types and shapes of widgets to the A-Q 17 small areas inside this controller.

In the A area, add a **Speedometer ** widget to display the car’s speed.

Note: You can delete the selected widget by clicking on it, swiping left to find the Delete button, and clicking on it.

Set the name, maximum and minimum values, and units by clicking the Settings icon in the upper right corner.

Set your current environment Line_Ref and Cliff_Ref for the Grayscale Detection widget in the D area.

Lastly, add the remaining widgets and click the top right button to save.

Connect to PiCar-x. When you click the Connect button, it will automatically search for robots nearby. Its name is defined picarx_control.py and it must be running at all times.

Once you click on the product name, the message “Connected Successfully” will appear and the product name will appear in the upper right corner.

Note: You need to make sure that your mobile device is connected to the same LAN as PiCar-X. If it doesn’t search automatically, you can also manually enter the IP to connect.

Python Code for App Controlled Robot using Raspberry Pi

Here is a complete Python Code for App Controlled Robot using Raspberry Pi .

You can directly run the code from the terminal using the following command.

cd ~/sunfounder-controller/examples
sudo python3 picarx_control.py

1 2	cd ~/sunfounder-controller/examples sudo python3 picarx_control.py

The code is also located in the source code path ‘sunfounder-controller/examples/picarx_control.py‘. You can directly open the code from this location. You can Modify/Reset/Copy/Run/Stop the code below.

from sunfounder_controller import SunFounderController
from picarx import Picarx
from robot_hat import utils, Music
from vilib import Vilib
import os
from time import sleep

utils.reset_mcu()
sleep(0.5)


sc = SunFounderController()
sc.set_name('Picarx-001')
sc.set_type('Picarx')
sc.start()

px = Picarx()
speed = 50
line_following_speed = 20
line_following_angle_offset = 20
avoid_obstacles_speed = 30

music = Music()


# get IP address
def getIP():
    wlan0 = os.popen("ifconfig wlan0 |awk '/inet/'|awk 'NR==1 {print $2}'").readline().strip('\n')
    eth0 = os.popen("ifconfig eth0 |awk '/inet/'|awk 'NR==1 {print $2}'").readline().strip('\n')

    if wlan0 == '':
        wlan0 = None
    if eth0 == '':
        eth0 = None

    return wlan0,eth0


def map(x, in_min, in_max, out_min, out_max):
    return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min


def horn(): 
    status, result = utils.run_command('sudo killall pulseaudio')
    music.sound_effect_threading('./sounds/car-double-horn.wav')
    

def avoid_obstacles():
    px.forward(avoid_obstacles_speed)
    distance = px.get_distance()
    if distance > 0 and distance < 300:
        if distance < 25:
            px.set_dir_servo_angle(-35)
        else:
            px.set_dir_servo_angle(0)   


def line_following():
    gm_val_list = px.get_grayscale_data()
    gm_status = px.get_line_status(gm_val_list)
    if gm_status == 'forward':
        px.forward(line_following_speed) 
    elif gm_status == 'left':
        px.set_dir_servo_angle(line_following_angle_offset)
        px.forward(line_following_speed) 
    elif gm_status == 'right':
        px.set_dir_servo_angle(-line_following_angle_offset)
        px.forward(line_following_speed) 
    else:
        px.set_dir_servo_angle(0)
        px.stop()



def main():
    global speed

    wlan0,eth0 = getIP()
    if wlan0 != None:
        ip = wlan0
    else:
        ip = eth0
    print('ip : %s'%ip)

    Vilib.camera_start(vflip=False,hflip=False)
    Vilib.display(local=False, web=True)
    speak = None
    while True:
        # sleep(0.2)

        # send data 
        sc.set('video','http://'+ip+':9000/mjpg')
        sc.set("A", speed)

        grayscale_data = px.get_grayscale_data()
        # print(px.get_grayscale_data())
        sc.set("D", grayscale_data )
        

        distance = px.get_distance()
        # sc.set("L", [0,distance])
        sc.set("F", distance)

        # if sc.get('J') == True:
        #     horn()

        # print(sc.get('J'), type(sc.get('J')), speak)
        if sc.get('J') != None:
            speak=sc.get('J')
        if speak == "forward":
            px.forward(speed)
        elif speak == "backward":
            px.backward(speed)
        elif speak == "left":
            px.left(speed)
        elif speak == "right":
            px.right(speed)
        else:
            px.stop()
            
        Joystick_K_Val = sc.get('K')
        if Joystick_K_Val != None:
            dir_angle = map(Joystick_K_Val[0], -100, 100, -45, 45)
            speed = Joystick_K_Val[1]
            px.set_dir_servo_angle(dir_angle)
            if speed > 0:
                px.forward(speed)
            elif speed < 0:
                speed = -speed
                px.backward(speed)
            else:
                px.stop()

        if sc.get('I') == True:
            line_following()
        elif sc.get('E') == True:
            avoid_obstacles()


        if sc.get('N') == True:
            Vilib.color_detect("red")
        else:
            Vilib.color_detect_switch(False)

        if sc.get('O') == True:
            Vilib.human_detect_switch(True)  
        else:
            Vilib.human_detect_switch(False)  

        if sc.get('P') == True:
            Vilib.object_detect_switch(True) 
        else:
            Vilib.object_detect_switch(False)

        
        Joystick_Q_Val = sc.get('Q')
        if Joystick_Q_Val != None:
            # pan = map(Joystick_Q_Val[0], -100, 100, -90, 90)
            # tilt = map(Joystick_Q_Val[1], -100, 100, -35, 75)
            pan = min(90,max(-90,Joystick_Q_Val[0]))
            tilt = min(75, max(-35, Joystick_Q_Val[1]))
            px.set_camera_servo1_angle(tilt)
            px.set_camera_servo2_angle(pan)


def servos_test():
    px = Picarx()
    px.set_camera_servo1_angle(0)
    px.set_camera_servo2_angle(0)
    sleep(0.5)

    while True:
        for angle in range(0,75):
            px.set_camera_servo1_angle(angle)
            sleep(0.01)
        for angle in range(75,-35,-1):
            px.set_camera_servo1_angle(angle)
            sleep(0.01)        
        for angle in range(-35,0):
            px.set_camera_servo1_angle(angle)
            sleep(0.01)


if __name__ == "__main__":
    try:
        main()
    finally:
        px.stop()
    # servos_test()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

from sunfounder_controller import SunFounderController

from picarx import Picarx

from robot_hat import utils, Music

from vilib import Vilib

import os

from time import sleep

utils.reset_mcu()

sleep(0.5)

sc = SunFounderController()

sc.set_name('Picarx-001')

sc.set_type('Picarx')

sc.start()

px = Picarx()

speed = 50

line_following_speed = 20

line_following_angle_offset = 20

avoid_obstacles_speed = 30

music = Music()

# get IP address

def getIP():

wlan0 = os.popen("ifconfig wlan0 |awk '/inet/'|awk 'NR==1 {print $2}'").readline().strip('\n')

eth0 = os.popen("ifconfig eth0 |awk '/inet/'|awk 'NR==1 {print $2}'").readline().strip('\n')

if wlan0 == '':

wlan0 = None

if eth0 == '':

eth0 = None

return wlan0,eth0

def map(x, in_min, in_max, out_min, out_max):

return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min

def horn():

status, result = utils.run_command('sudo killall pulseaudio')

music.sound_effect_threading('./sounds/car-double-horn.wav')

def avoid_obstacles():

px.forward(avoid_obstacles_speed)

distance = px.get_distance()

if distance > 0 and distance < 300:

if distance < 25:

px.set_dir_servo_angle(-35)

else:

px.set_dir_servo_angle(0)

def line_following():

gm_val_list = px.get_grayscale_data()

gm_status = px.get_line_status(gm_val_list)

if gm_status == 'forward':

px.forward(line_following_speed)

elif gm_status == 'left':

px.set_dir_servo_angle(line_following_angle_offset)

px.forward(line_following_speed)

elif gm_status == 'right':

px.set_dir_servo_angle(-line_following_angle_offset)

px.forward(line_following_speed)

else:

px.set_dir_servo_angle(0)

px.stop()

def main():

global speed

wlan0,eth0 = getIP()

if wlan0 != None:

ip = wlan0

else:

ip = eth0

print('ip : %s'%ip)

Vilib.camera_start(vflip=False,hflip=False)

Vilib.display(local=False, web=True)

speak = None

while True:

# sleep(0.2)

# send data

sc.set('video','http://'+ip+':9000/mjpg')

sc.set("A", speed)

grayscale_data = px.get_grayscale_data()

# print(px.get_grayscale_data())

sc.set("D", grayscale_data )

distance = px.get_distance()

# sc.set("L", [0,distance])

sc.set("F", distance)

# if sc.get('J') == True:

# horn()

# print(sc.get('J'), type(sc.get('J')), speak)

if sc.get('J') != None:

speak=sc.get('J')

if speak == "forward":

px.forward(speed)

elif speak == "backward":

px.backward(speed)

elif speak == "left":

px.left(speed)

elif speak == "right":

px.right(speed)

else:

px.stop()

Joystick_K_Val = sc.get('K')

if Joystick_K_Val != None:

dir_angle = map(Joystick_K_Val[0], -100, 100, -45, 45)

speed = Joystick_K_Val[1]

px.set_dir_servo_angle(dir_angle)

if speed > 0:

px.forward(speed)

elif speed < 0:

speed = -speed

px.backward(speed)

else:

px.stop()

if sc.get('I') == True:

line_following()

elif sc.get('E') == True:

avoid_obstacles()

if sc.get('N') == True:

Vilib.color_detect("red")

else:

Vilib.color_detect_switch(False)

if sc.get('O') == True:

Vilib.human_detect_switch(True)

else:

Vilib.human_detect_switch(False)

if sc.get('P') == True:

Vilib.object_detect_switch(True)

else:

Vilib.object_detect_switch(False)

Joystick_Q_Val = sc.get('Q')

if Joystick_Q_Val != None:

# pan = map(Joystick_Q_Val[0], -100, 100, -90, 90)

# tilt = map(Joystick_Q_Val[1], -100, 100, -35, 75)

pan = min(90,max(-90,Joystick_Q_Val[0]))

tilt = min(75, max(-35, Joystick_Q_Val[1]))

px.set_camera_servo1_angle(tilt)

px.set_camera_servo2_angle(pan)

def servos_test():

px = Picarx()

px.set_camera_servo1_angle(0)

px.set_camera_servo2_angle(0)

sleep(0.5)

while True:

for angle in range(0,75):

px.set_camera_servo1_angle(angle)

sleep(0.01)

for angle in range(75,-35,-1):

px.set_camera_servo1_angle(angle)

sleep(0.01)

for angle in range(-35,0):

px.set_camera_servo1_angle(angle)

sleep(0.01)

if __name__ == "__main__":

try:

main()

finally:

px.stop()

# servos_test()

Testing App Controlled AI Robot using Raspberry Pi

Click the Run button to start the controller, you will see the footage of the car shooting, and now you can control your PiCar-X with these widgets.

Here are the functions of the widgets.

A: Show the current speed of the car.
D: Show the data of the three sensors on the grayscale module, which have three states: black block: black line detected; white: white detected; exclamation point: cliff detected.
E: turn on the obstacle avoidance function.
I: turn on the line following function.
J: voice recognition, press and hold this widget to start speaking, and it will show the recognized voice when you release it. We have set forward, backard, left and right 4 commands in the code to control the car.
K: Control forward, backward, left, and right motions of the car.
Q: turn the head(Camera) up, down, left and right.
N: Turn on the color recognition function.
O: Turn on the face recognition function.
P: Turn on the object recognition function, it can recognize nearly 90 kinds of objects, for the list of models, please refer to: https://github.com/sunfounder/vilib/blob/master/workspace/coco_labels.txt.

This is how you can make an App Controlled Robot using Raspberry Pi with multipurpose AI Applications.

Video Tutorial & Guide

PiCar-X Self Driving AI Robot with Raspberry Pi || Obstacle, Line, Signal, Cliff Detection & TTS

Watch this video on YouTube.

View 1 Comment

1 Comment

Hansraj on January 2, 2023 7:52 PM

So grate that people like you make projects and upload it on internet step by step that others can make it too for free. I would kindly like to ask you for making a humanoid robot using raspberry pi 4b and 18 SG90 servo and smarthphone app for it