Regarding how to label images and what tools we should use, we have two choices, which are machine learning and image processing. They both require powerful calculation capability but the result of image processing is more likely to be influenced by shadows, so we finally chose the deep leaning technique.

Originally we applied STM32 microcontroller as our processor, but it cannot run the deep learning algorithm so we changed it to Raspberry Pi in the spring semester.

To ensure that users can share data at any time and at any place, will build this mesh to allow users to check road conditions even without the Internet.

Since our dataset is small, we have to retrain an existing neural network

It is difficult to determine what sampling frequency the GPS should be, then we did a large number of tests using 1Hz, 5Hz, and 10Hz, but we found that the default 1Hz is the best because the GPS is less possible to get errors under that frequency, and another reason is that the running speed of the bike is slow so the system can sample enough data in unit time.

Generally speaking, the key step in process of applying a machine learning classification of TensorFlow in the mobile system is transmitting the existing TF module which has been demonstrated accurate enough to the TensorFlow lite format. After doing that, we could write python program in the mobile system to order the model to classify the input data.

Unfortunately, as what I mentioned before, we met difficulties in this step. The difficulties are related to the model saving step in the model training program. The original model is not available for the official transmission tool. Thus, we need to redevelop a new model to fit the requirement of the official transmission tool.

Now, we are trying our best to solve this problem and tried to develop the new model and transmitted it into TensorFlow lite format in less than one week.

Like what I mentioned before, the app developed by the last cohort is not stable for the daily using. Thus, we need to optimize the application and also add some new features to it to make it useful for the dynamic threshold application. It is challenging for us because none of us had the experience of developing Android app before, which means we need to learn it from zero.

Now, we have a specific team member to learn the Android application development. He will focus on this part so that he could have higher efficiency to solve this problem. Actually, this problem has been solved halfway. The problem of crashing has been solved and the following task is to add new features.

Firstly, our system has been developed by the last cohort, but some functions didn’t work well. For the wireless communication module, due to the long storage time, several devices cannot work correctly. Thus, we need to test all of the components and try to find one whole system which could work. That takes us some time.

Secondly, due to the instability of some components, the basic logic of the system was also blocked. For example, at the very beginning, we found that the raw data was never collected and stored in the system of the Raspberry Pi. After we analyzed the program and the status of all components, we found that the GPS module was not working. Because the GPS cannot work correctly, the system’s program was still waiting for the data collected by the GPS to complete collection the first record and it didn’t get the data. That’s why the system couldn’t collect the data correctly.

Thirdly, the Android application was also not stable. The Android application we used was based on the last cohort’s achievement. When we were collection vibration data, it crashed totally. Thus, we need to take some other time to fix it.

At the very beginning, due to the process of fixing all components were not finished, the first constraint we met was actually that the Android application couldn’t work properly. To start collecting data as early as we could, we changed the mechanism of the inner program from a listener to a timer. By doing this, we could use the system individually without receiving the commands from the Android application

For the data collection, the second constraint we met is the timing and weather conditions. Because we need to make sure the system could work well. Our schedule was a little bit late than what we scheduled. However, due to this little change, Flagstaff started being warmer and warmer. That made us really struggle to collect the Snowy data and Rainy data.

Thirdly, in order to develop the machine learning model, we need to collect a large data set of images which contains more than 2000 images for model training. Each image was taken manually and it also met the constraints of the weather conditions. If we didn’t finish that, we could not start working on the machine learning model.

The data processing module was aimed at generating a thresholds table for accurate detection of bumps and holes. That application requires that we need to collect accurate vibration data and use them to generate thresholds. However, due to the long storage time, the sensor has drift. Thus, we need to develop a method to decrease the effect caused by the shifting.

Secondly, when we were developing the machine learning model, we were stacked on one tiny step, which is transferring the model developed in the TensorFlow platform to the TensorFlow Lite platform. Generally speaking, this step is not difficult, but when we followed the instructions of google, it didn’t work at all. After we made some research and asked for help through the internet, we used another method to transfer it.

Thirdly, due to the spread of the covid-19, although we have finished developing the machine learning model, we cannot test it on the real application scene. Thus, the accuracy we offered was just the accuracy in the test data set, which is not so accurate but also representative to evaluate its functionality.