Introduction: The provided code implements a simple LED game controlled by capacitive touch sensors. The game utilizes an 8x8 LED matrix to display the game state and four capacitive sensors as input devices. The objective of the game is to press the correct capacitive sensor corresponding to the LED tile at the bottom row.

Components:

• LED Matrix: An 8x8 LED matrix is used to represent the game board. Each LED corresponds to a pixel in the game grid.
• Capacitive Sensors: Four capacitive sensors are employed as input devices. These sensors detect touch inputs and trigger corresponding actions in the game.
• Arduino Board: The code is designed to run on an Arduino board, which provides the necessary hardware interface for controlling the LED matrix and reading sensor inputs.

Initialization: The code begins with the initialization of necessary variables and pin configurations. Two arrays, row and col, store the pin numbers for rows and columns of the LED matrix, respectively. An array of CapacitiveSensor objects, buttonPins, is created to handle capacitive input from the sensors. Other variables like pixels, currentMillis, prevMillis, sensorMillis, lives, and score are initialized.

Main Loop: The loop() function contains the main game logic and is executed continuously. It updates the game state, handles input from capacitive sensors, and refreshes the LED matrix display. The game checks for button presses using capacitive sensors and updates the game state accordingly. Tiles are moved down one row, and a new tile is generated at the top row periodically.

Game Logic: The moveTilesDown() function moves existing tiles down one row, while the generateNewTile() function randomly generates a new tile at the top row. Button presses are detected using capacitive sensors. If the correct button is pressed, the score is incremented; otherwise, lives are decremented.

Screen Refresh: The refreshScreen() function updates the LED matrix display by iterating over rows and columns. It turns on LEDs where the row is HIGH and the column is LOW, representing the intersection of rows and columns. Pixels are turned off by setting the column pin to HIGH when the pixel state is LOW.

Introduction:

The provided code implements an interactive LCD controlled by photoresistors and capacitive sensors. The photoresistor detects if there is sunlight at least 4 hours a day and the capacitive sensor detects if it’s watered at least once per day. The state is updated at the end of each day.

Prototyping:

Our first idea was to create a variation of our previous project, the Piano Tiles game. We wanted a uniqur input device, and one idea was to use a conductive, squishy ball. How hard you squeezed would determine which piano tile you pressed. For example, a light squeeze would indicate the first key is pressed while a hard squeeze indicates the last key is pressed.

We eventually decided against the idea for three reason:
1. Difficulty to find effective candidates for conductive material that is squeezable.
2. The complex and extensive debounce development.
3. Idea seemed more frustrating than actually function.

We then decided to explore a new input material and we decided on dough. Our idea was to use four pieces of dough as the inputs for the Piano Tiles game.

Again, we abandoned the idea for three reasons:
1. Dough was a lot harder to make then anticipated.
2. The game was just too similar to our previous project.
3. Having to use the same dough for days on end with multiple people touching it just seemed nasty.

We ultimately decided against doing the Piano Tiles game. We couldn't find a way to improve it as it already was a really creative project with unique input. That's when we began brainstorming all over again. We looked back at some of the mini projects and recalled the photoresistor. We thought light itself as an input is so intriguing. Therefore, we came up with the idea of an interactive 8x8 screen. We could make a threshold where the screen would display a sad face if the photoresistor wasn't receiving sufficient lighting, a happy face if it receives a lot of lighting, and a content face for everything in between.

The big issue with this program was that it was too simple. It was creative, but not impressive. Also, the 8x8 LED matrix was just becoming boring as we have all programmed it multiple times. Again, we were stuck brainstorming until Akash made a remark, "Don't you guys think the photoresistor is basically like a plant, where it's happy to receive sunlight." That's when we came up with our project idea, a plant that could be treated like a virtual pet. We thought that we could use an LCD screen to determine whether the plant was happy, sad, or content based on how it's being treated, like how much sun or water it has received.

Components:

• 3D Printed Pot
• ESP-WROOM-32
• TFT LCD Display Module GC9A01 Driver
• Analog Soil Moisture Sensor
• Photoresistor
• Wires
• Breadboard

Usage:

The device monitors two crucial aspects for the plant's well-being: soil moisture level and light exposure.

• Soil Moisture Level: The analog soil moisture sensor measures the moisture level in the soil. When the soil becomes too dry, the device prompts the user to water the plant.
• Light Exposure: The photoresistor measures the intensity of light. Plants require a certain amount of sunlight to thrive. The device keeps track of the duration of sunlight exposure. If the plant receives insufficient sunlight, its happiness level decreases.

The device simulates a day-night cycle, with each simulated day lasting approximately 24 hours. Throughout the day, the plant's happiness level is updated based on its care and the environmental conditions.

Functionality:

• Plant Happiness: The device displays a digital representation of the plant's mood (either happy or sad) based on its care and environmental conditions.
• User Interaction:Users can interact with the device by watering the plant when prompted and ensuring it receives adequate sunlight.

Video Demonstration:

Ideation/Brainstorming:

When we first saw the project prompt, the idea of a chicken's head immediately popped up. It is so intriguing to see videos of how chickens automatically stabilize their heads while their body is moved aroun. We immediately thought of a camera stabilizer. With the use of an IMU and a few motors, the idea of a camera stabilizer was feasible. However, we wanted to make this device easily accessible and not everyone is carrying a digital camera. Therefore, we believed that the stabilizers should be designed to carry phones.

Protoype 1:

We wanted to practice our parallel prototyping skills by coming up with different designs of different ideas. The first design we thought of was a headband with a phone carrier, inspired by the head of chicken we brought up earlier.

We eventually passed out on this idea because we could not figure out how we would be able to support the weight of a phone on a headband. Also, it looked really ridiculous.

The second design we came up with for our first prototype was a hand-held phone stabilizer.

We knew this idea was feasible and useful. We eventually passed on this idea as we thought it was too basic. There are already so many handheld phone stabilizers out there that we could easily purchase. So this would not really be our design, just more of a recreation of an already well-established design.

Protoype 2:

For our second prototype, we thought about how IMU design could facilitate life in general, specifically for the disabled. There are many activities and hobbies people with disabilities cannot partake in due to their conditions. We came up with a contoller that uses an IMU as an input. We wanted to make a glove that had an IMU attached to it, and the user could give input on devices by tilting their hand forward, backward, left, or right. We could use the Arduino Nano's IMU and WiFi communication capabilities to acheive this.

Eventually, we decided on this prototype and design. We were fascinated by the idea of creating something for the benefit of others. We were also intrigued in using the WiFi communication capabilites of a microcontroller, something neither of us has done before.

Beginning Work:

We have not thought about what application we will use for our demonstration. The idea of a drone did get brought up. The glove could be used to not only direct the movement of the drone, but also at what speed. So for our very first prototype, we came up with a design that included six LEDS. Three LEDS indicating forward movement and three indicating backward movement. Each LED would light up based on how far forward or backward the Nano would move. The following is a video of our prototype.

The following is the code we used to program our prototype.

                        
                            #include <Arduino_LSM6DS3.h>

                            float x, y, z;
                            int forward = 0;
                            int backward = 0;
                            int left_movement = 0;
                            int right_movement = 0;

                            const int LED_PIN[] = [2, 3, 4, 5, 6, 7]; 

                            void lightLed(int, int);

                            void setup() {
                                Serial.begin(115200);
                                while (!Serial);
                                Serial.println("Started");

                                if (!IMU.begin()) {
                                    Serial.println("Failed to initialize IMU!");
                                    while (1);
                                }

                                for(int i = 0; i < 4; i++) {
                                    pinMode(LED_PIN[i], OUTPUT);
                                }

                                Serial.print("Accelerometer sample rate = ");
                                Serial.print(IMU.accelerationSampleRate());
                                Serial.println("Hz");
                            }

                            void loop() {

                                if (IMU.accelerationAvailable()) {
                                    IMU.readAcceleration(x, y, z);

                                }

                                if (x > 0.1) {
                                    x = 100 * x;
                                    backward = map(x, 0, 97, 0, 4);
                                    Serial.print("Moving backwards at speed setting ");
                                    Serial.print(backward);
                                    Serial.print("\n");
                                }
                                if (x < -0.1) {
                                    x = 100 * x;
                                    forward = map(x, 0, -100, 0, 4);
                                    Serial.print("Moving forward at speed setting ");
                                    Serial.print(forward);
                                    Serial.print("\n");
                                }
                                if (y > 0.1) {
                                    y = 100 * y;
                                    left_movement = map(y, 0, 97, 0, 4);
                                    Serial.print("Moving left at speed setting ");
                                    Serial.print(left_movement);
                                    Serial.print("\n");
                                }
                                if (y < -0.1) {
                                    y = 100 * y;
                                    right_movement = map(y, 0, -100, 0, 4);
                                    Serial.print("Moving right at speed setting ");
                                    Serial.print(right_movement);
                                    Serial.print("\n");
                                }

                                lightLed(forward, backward);

                                delay(50);
                            }

                            void lightLed(int f, int b) {
                                if(f == 0) {
                                    for(int i = 3; i < 6; i++) {
                                    digitalWrite(LED_PIN[i], 0);
                                    }
                                }
                                if (b == 0) {
                                    for(int i = 0; i < 3; i++) {
                                    digitalWrite(LED_PIN[i], 0);
                                    }
                                }
                                if(f > b) {
                                    for(int i = 0; i < 3; i++) {
                                    digitalWrite(LED_PIN[i], 0);
                                    }
                                    switch(f) {
                                    case 1:
                                        for(int i = 4; i < 6; i++) {
                                        digitalWrite(LED_PIN[i], 0);
                                        }
                                        digitalWrite(LED_PIN[3], 1);
                                        break;
                                    case 2:
                                        digitalWrite(LED_PIN[3], 1);
                                        digitalWrite(LED_PIN[4], 1);
                                        digitalWrite(LED_PIN[5], 0);
                                        break;
                                    case 3:
                                        for(int i = 3; i < 6; i++) {
                                        digitalWrite(LED_PIN[i], 1);
                                        }
                                        break;
                                    default:
                                        break;
                                    }
                                }else if (b > f) {
                                    for(int i = 3; i < 6; i++) {
                                    digitalWrite(LED_PIN[i], 0);
                                    }
                                    switch(b) {
                                    case 1:
                                        for(int i = 0; i < 2; i++) {
                                        digitalWrite(LED_PIN[i], 0);
                                        }
                                        digitalWrite(LED_PIN[2], 1);
                                        break;
                                    case 2:
                                        digitalWrite(LED_PIN[2], 1);
                                        digitalWrite(LED_PIN[1], 1);
                                        digitalWrite(LED_PIN[0], 0);
                                        break;
                                    case 3:
                                        for(int i = 0; i < 3; i++) {
                                        digitalWrite(LED_PIN[i], 1);
                                        }
                                        break;
                                    default:
                                        break;
                                    }
                                }
                            }
                        
                    

Ideation/Brainstorming:

When brainstorming ideas for MGP4, we pondered the exciting possibilities of utilizing Inertial Measurement Units (IMUs). To see more of our brainstorming process, you can view it under "MGP3: Intermediate Prototype" section of the blog. Basically, our research led us into real-world applications used by IMUs. For example, we found out that drones employ IMUs to relay contextual information about the drone's relative position and movement back to its remote controller. This technology facilitates precise and responsive control, enhancing the user's experience. Since this is a Human-Computer Interaction (HCI) course, we were also keen on approaching this project from a unique angle that would redefine the way we interact with technology.

While exploring YouTube, we came across a video featuring the Xbox team at Microsoft discussing the development of the adaptive controller. This innovative controller was specifically designed to accommodate individuals with disabilities, with the goal of providing a more inclusive gaming experience.

One particularly insightful statement from a team member responsible for creating the adaptive controller caught my attention. They asserted that "a disability is defined as a mismatch in human interaction." This perspective challenges the conventional understanding of disabilities, suggesting that many are not inherent flaws or limitations of the individual but rather arise from a lack of support and accommodation in their environment.

In essence, this viewpoint shifts the focus from the individual's impairment to the societal and environmental barriers that prevent them from fully participating in various activities, including gaming. By embracing this perspective, the Xbox team emphasized the importance of creating inclusive products and environments that empower individuals with disabilities to engage more fully in the activities they enjoy.

This approach to understanding and addressing disabilities highlights the need for more inclusive design practices across various industries, not just gaming. It showcases the potential for technology to break down barriers and create opportunities for people of all abilities to participate equally in society. Our team wanted to implement this ideology in our project. Of course, we had to think about the practicality of designing such a device.

A compelling question arises: why is it important to make such accommodations in something as seemingly trivial as video games? Beyond the simple premise that everyone deserves the chance to experience these enjoyable activities, there is a deeper significance. The genesis of the Adaptive Controller can be traced back to veterans who utilized video games as a form of prescribed therapy. Veterans account for a disproportionate number of suicides, and a psychological study demonstrated that veterans prescribed video games exhibited better signs of recovery from severe depression.

We wanted this device to not just be used for video games, but for any device that utilizes user inputs. People with disabilities deserve to have as much flexibility in their hobbies as everyone else. That is how we decided on a glove, which uses the tilt of one's hand as input.

Finalized Idea #1:

So we decided, on making a glove, with the Arduino Nano attached and using its internal IMU. We made a program that determines how far up, down, left, and right the user's glove tilts. We also added LEDs on all four sides of the Nano to indicate where the user is tilting. The following video is a demonstration of the glove we made:

Our idea was to use another microcontroller, an ESP32, to wirelessly communicate with the Arduino Nano. Since the ESP32 has two cores on its processor, we planned to use one of its core to communicate with the Arduino and receive input while the other core controlled the device we were wirelessly sending inputs to. The ESP32 would receive four inputs of the four directions the glove would tilt. The following is a video showcasing the inputs the ESP32 would receive:

We successfully sent the inputs to the ESP32. The following is a video of the ESP32 outputting the inputs it read:

The following is the code we used for this project.

                        
							#include <Arduino_LSM6DS3.h>
							#include <Wire.h>

							float x, y, z;
							int degreesX_f = 0;
							int degreesX_b = 0;
							int degreesY_r = 0;
							int degreesY_l = 0;
							int threshold = 23;

							const int LED_PIN[] = {17, 2, 9, 4}; 

							void setup() {

								for(int i = 0; i < 4; i++){
									pinMode(LED_PIN[i], OUTPUT);
								}

								Wire.begin();

								Serial.begin(9600);
								while (!Serial);
								Serial.println("Started");

								if (!IMU.begin()) {
									Serial.println("Failed to initialize IMU!");
									while (1);
								}

								Serial.print("Accelerometer sample rate = ");
								Serial.print(IMU.accelerationSampleRate());
								Serial.println("Hz");
							}

							void loop() {

								if (IMU.accelerationAvailable()) {
									IMU.readAcceleration(x, y, z);
								}

								if (x > 0.1) {
									digitalWrite(4, 0);
									x = 100 * x;
									degreesX_f = map(x, 0, 97, 0, 90);
									Serial.print("Tilting forward ");
									Serial.print(degreesX_f);
									Serial.println("  degrees");

									if (degreesX_f > threshold) digitalWrite(2, 1);
									else digitalWrite(2, 0);
								} else digitalWrite(2, 0);

								delay(50);

								if (x < -0.1) {
									digitalWrite(2, 0);
									x = 100 * x;
									degreesX_b = map(x, 0, -100, 0, 90);
									Serial.print("Tilting backward ");
									Serial.print(degreesX_b);
									Serial.println("  degrees");

									if (degreesX_b > threshold) digitalWrite(4, 1);
									else digitalWrite(4, 0);
								} else digitalWrite(4, 0);

								delay(50);

								if (y > 0.1) {
									digitalWrite(17, 0);
									y = 100 * y;
									degreesY_l = map(y, 0, 97, 0, 90);
									Serial.print("Tilting left ");
									Serial.print(degreesY_l);
									Serial.println("  degrees");
									if (degreesY_l > threshold) digitalWrite(9, 1);
									else digitalWrite(9, 0);
								} else digitalWrite(9,0);

								delay(50);

								if (y < -0.1) {
									digitalWrite(9, 0);
									y = 100 * y;
									degreesY_r = map(y, 0, -100, 0, 90);
									Serial.print("Tilting right ");
									Serial.print(degreesY_r);
									Serial.println("  degrees");
									if (degreesY_r > threshold) digitalWrite(17, 1);
									else digitalWrite(17, 0);
								} else digitalWrite(17,0);

								byte sendData[8];
								sendData[0] = (degreesX_f >> 8) & 0xFF;  // High byte of degreesX_f
								sendData[1] = degreesX_f & 0xFF;         // Low byte of degreesX_f
								sendData[2] = (degreesX_b >> 8) & 0xFF;  // High byte of degreesX_b
								sendData[3] = degreesX_b & 0xFF;         // Low byte of degreesX_b
								sendData[4] = (degreesY_r >> 8) & 0xFF;  // High byte of degreesY_r
								sendData[5] = degreesY_r & 0xFF;         // Low byte of degreesY_r
								sendData[6] = (degreesY_l >> 8) & 0xFF;  // High byte of degreesY_l
								sendData[7] = degreesY_l & 0xFF;         // Low byte of degreesY_l

								// Send data over I2C
								Wire.beginTransmission(8);    // Address of the ESP32
								Wire.write(sendData, 8);      // Send the data array
								Wire.endTransmission();       // Stop transmitting
								
								delay(1000);
								
							}
						
                    

Code Breakdown:

If you look at the code, we use the Arduino's bultin library to read in it's IMU values. We then map those readings to display the tilt as a measure between 0 and 90 degrees. Afterwards, we display each of those values if they are actively being tilted more than the threshold we set, which is 23 degrees. Then, the respective led is lit up. If an LED is lit up, then it's opposite LED will not be. This way, there wouldn't be any confusion on the tilt of the hands. However, its perpendicular LED could still be lit up, allowing the user to see if it's tilted above the threshold in both directions, the X and Y axis values.

We then use the Wire.h library to communicate with the ESP32 microcontroller. We send an array of 8 bytes, where all four inputs are divided into two bytes of data. The ESP32 reads this data and then converts it into an array of four ints. A for loop is used to combine two bytes of data into one, then inputs the recovered data into the array. Afterwards, each element of the array is assigned to it's respective integers. That is how the ESP32 receives the data, which can be used for many applications.

We thought about different cases we could use this for. We could use it as an interactive video game controller, but that idea has already been done, which was also the main inspiration for this project. We wanted to make something new, not just recreate something already made. That is when we decided to use the glove as a controller for a drone.

Finalized Idea #2:

So we're going to make an IMU controller for an fpv drone. Why an fpv drone?

The first challenge we anticipated was formalizing the IMU data to be read as valid input. It's difficult to determine the exact input format required, but what if we built a drone ourselves? Drones are expensive, and FPV drones are even more so. How about we build our own drone?

Building our own drone presents several advantages. We were inclined to use the ESP32, as it features a dual-core processor and can create a WiFi access point. By utilizing the ESP32, we can use it to act as the drone's controller and provide a point of connection for the Arduino with the built-in IMU to connect wirelessly.

Additionally, the extra layer of complexity beyond just getting imu readings also made this a more interesting project.

How to build a drone:

To build the drone we have to address a few important aspects.

• - What electrical parts do we need
• - We also need to implement some signal based communication channel between the remote controller and the drones onboard controller.
• - We need some sort of housing for the drones electronic components.

Addressing Electronics Parts

So drones have 7 important parts, the remote controller, the drone's onboard controller, the motor, the propellor, an Electronic Speed Controller, a power source, and a frame. We have two parts covered, the two controllers. We find a project from another individual who is working towards building an esp32 drone and he lists a few parts for us an 8.5x20mm Motor 3-5V brushed motor and propellor combo, a CAD file for the frame, and a P-channel MOSFET transistor for making his own ESC.

Addressing communication channel

As I said earlier we wanted to use Wifi. The esp32, has a way to creat a wifi access point, this lets up basically create this private network where connected devices can communicate with each other. Since the Arduino and ESP32 can both use the WiFi.h library this implementation is fairly simple.

Addressing Housing

While the online resource provided a CAD file, the built in housing meant to fit the motor didn't fit, so we had to design our own frame.

What went wrong?

In Short, the SMD Transistor was not an apt alternative for an actual Electronic Speed Controller. There is also a reason the motor was cheap. The transistor ended up being too small to actually try and write an electronic speed controller with, even though the GPIO pins on the esp32 actually have a way of sending varying voltages, which would've allowed us to manipulate the current to the motor. Because the transistor was too small, it was just almost impossible for unexperienced to work with using just tweezers and a soldering iron.

What did we learn?

Our journey in building an IMU drone controller provided us with valuable learning experiences and insights. Firstly, we gained a deeper understanding of the intricate components and technologies involved in drone construction, including the role of IMUs in flight stabilization and control.
Additionally, we learned the importance of thorough planning and research in project development. Identifying suitable electronic parts, establishing communication channels between controllers, and designing a suitable housing demanded meticulous attention to detail and problem-solving skills.
Furthermore, our project underscored the significance of experimentation and iteration in the prototyping process. Despite encountering challenges such as incompatible components and design limitations, each setback served as an opportunity for learning and refinement.
Importantly, our experience highlighted the value of interdisciplinary collaboration and knowledge-sharing. By drawing upon resources and insights from diverse sources, including online communities and existing projects, we were able to navigate challenges more effectively and enrich our understanding.
Ultimately, our venture into building an IMU drone controller not only expanded our technical competencies but also cultivated essential skills in teamwork, problem-solving, and adaptability. These lessons will undoubtedly inform and enhance our future endeavors in the realm of technology and innovation.