In the digital era, technology is a universal key that allows each person to access knowledge and opportunities. Thus, conquering and mastering technology has always been the dream of many scientists. With the topic: “Hand gesture recognition using multimodal sensors based on intelligent edge computing, applied in human-machine interaction”, sponsored by Air Force Office of Scientific Research Research fund (AFOSR), Assoc. Prof. Tran Thi Thanh Hai, lecturer at the School of Electrical and Electronics Engineering, together with her research team has gained new progress in searching and manufacturing wrist sensors with multimodal and gesture recognition, applied in human-machine interaction in a smart environment. 

“Our hands make it all” 

Assoc. Prof. Tran Thi Thanh Hai is currently a senior lecturer at the Faculty of Communication Engineering. She also is the Director of the “Digital Communications and Multimedia Engineering” training program (ET-E16). Mrs. Hai received in-depth training in Computer Science, Image Processing Vision, and Robotics in particular. She then participated in teaching and researching at the School of Electrical and Electronics Engineering, Assoc. Prof. Tran Thi Thanh Hai realized that “Hanoi University of Science and Technology has given me many precious opportunities to learn and to improve my profession”. For many projects, with the role of leader or participant, Mrs. Hai has paid a lot of enthusiasm to AI applications to modernize human-machine interaction. 

The current and future trend of human-machine interaction is the application of sensor technology and, to create natural interaction, approaching human-human interaction. Among various interaction channels, hand gestures and voice or sound have been attracting the attention of many scientists to create smarter interaction. In this trend, the research team of Mrs. Hai conducted the project: “Recognizing hand gesture using multimodal sensor based on intelligent edge in human-machine interaction”. This is an interdisciplinary project based on knowledge of multimodal sensors (image, accelerometer, gyroscope), digital signals processing, AI, and computer vision. 

Control systems based on hand gestures using sensors installed in the environment (cameras) are often limited by observation space, and high installation costs when duplicated in different places. This project overcomes these inherent limitations by designing a multimodal sensor device similar to a smartwatch, which is easy to carry and move along with control objects everywhere in the environment. Signals of gesture and environment will be collected, processed, and detected on the accompanying edge device, thereby converting basic commands to control electronic devices such as light bulbs, TV, AC,… in a domestic environment. It enables users to control devices no matter where they are with only a few simple hand movements, without the help of a switch or remote control.  
 

During the research and implementation process of recognition methods of the wearable smart sensor, many new challenges arose. “Sensors installed in the environment often can observe the whole human body and hand’s movements, therefore, it is easier to distinguish the gestures. However, the wrist-worn sensor has a limited view angle so it is difficult to observe the hand, especially the trạjectory of hand’s movements, as our hands move along with the sensor", Mrs. Hai explained. Currently, the device is being tested in the laboratory to recognize 12 hand gestures. Hand waving up to “Turn on", Hand waving down to “Turn off”, Hand swiping to “Turn page”, Hand drawing along the circle to adjust volume,… 

Further assessments will be carried out, together with the continuous development of signal recognition, that will allow practical application to enhance the interaction ability between humans and devices in the environment, thereby making life more modern and easier. Unlike sound sensors such as light bulbs turning on and off according to hand claps, or “Hey Siri” on Apple’s electronic devices, the product of this project can expand to solve further problems, for example: recording daily activities or reminder assistance for elderly health care. 

Contributing dedicated talent to Vietnam 

Assoc. Prof. Tran Thi Thanh Hai is an alumnus of the 41st cohort at Hanoi University of Science and Technology. After graduating as an engineer with an Information Technology major, she received a Master's and Doctoral scholarship in French. Mrs. Hai then continued to carry out postdoctoral research at the Research Institute of Computer Science and Random Systems (IRISA) and the French Alternative Energies and Atomic Energy Commission (CEA). After finishing her research, Mrs. Hai returned to Vietnam and worked as a lecturer at HUST: “Working in Vietnam has both challenges and opportunities. The chance to live with family, to work with excellent colleagues and students, and to participate in a practical project in my beloved country”, Mrs. Hai sincerely shared.  

Besides basic scientific research, Assoc. Prof. Hai and her team also aim to bring the research’s results into practical applications associated with Vietnam, for agriculture (counting tea buds automatically through images, classifying rice seeds automatically), for education (analyzing students’ behavior in class automatically), for health care (detecting injuries through gastrointestinal endoscopic ovarian ultrasound images, predicting traumatic brain injuries...). Mrs. Hai is very happy when her contributions can have practical benefits for the development of Vietnam's technology.  
 

Regarding the topic of hand gesture recognition sponsored by the US Air Force Office of Scientific Research, the research team includes members of many universities in Hanoi and Toulon University, France. According to Mrs. Hai, connecting scientists from different universities will create a group of researchers with the same expertise and passion. “Our goal is not only to produce sample products, or scientific articles but also to train officials, increase knowledge and create domestic and international connections”. 

This is also one of the orientations of HUST: Promoting traditions and strength in collaboration and focusing on promoting in-depth collaboration activities with academic partners to create practical benefits for training, researching, and improving the workforce. 

“I am very glad to work at HUST”, Assoc. Prof. Tran Thi Thanh Hai expressed proudly. To Mrs. Hai, HUST is a good environment for teaching and researching. With the pride of being an alumnus of HUST, she has witnessed a whole new transformation in the work of operating and promoting lecturers to carry out scientific research. Mrs. Hai believes that lecturers of HUST are given many freedoms such as academic freedom, self-developing freedom to learn and improve knowledge. 

Research project of Assoc. Prof. Tran Thi Thanh Hai is sponsored by Air Force of Scientific Research (AFOSR), with the investment capital of 60.000 USD for 2 years, from 2020 to 2022. After finishing, the project has gained many achievements, worth-mentioning is the 12 publications in International conferences and journals. 

On the other hand, the project has some scientific contributions such as proposing information integration methods of camera and acceleration, hand gesture to increase recognition efficiency, contributing a database for researcher community. Assoc. Prof. Thanh Hai emphasized: “Data plays a very important role for people who work in the field of computer science or AI. Without the shared data of the research community, the work of collecting and labeling will be very challenging. Besides visible products, the ones with long-term values like database is also a significant contribution.”