Author: Elizabeth Gillooly

Building a good team is often cited as the key to success in a variety of spheres — from business to the arts, from sports to science. For Mason Mechanical Engineering professor Jeffrey Moran, however, the ability to hand-pick a team for the second phase of the 4-VA Collaborative Research project “Toward T-Shaped Graduates: A Joint Capstone Program at the Nexus of Mechanical Engineering and Science and Technology Policy”, was completely out of his hands. Fortunately, though, the eight students who signed up for the Capstone project last summer — indicating an interest in building on the continuing project — turned out to be Moran’s Dream Team.

As luck would have it for Moran, when surveys went out to rising mechanical engineering seniors outlining opportunities for capstone projects last summer, the ‘just right’ eight students chose the option of working on the vehicle alert system.

The project, which tackles barriers to operating a vehicle for the hard of hearing and deaf communities by building assistive technologies to alert drivers and passengers to sounds around the vehicle, was launched by Moran’s capstone students in collaboration with a counterpart team at James Madison University, in the 2020-21 academic year. Yet, in part because of pandemic-related restrictions that kept the teams from working together, the tools and technologies that were developed on the project needed some advancements to improve the final product. Refinements and enhancements were necessary.

The reasons the eight students selected the project were as diverse as their backgrounds. A sampling:

• Javeria Jawad, who acted as one of the two team leads, was drawn to the hands-on aspect of the task, explained, “I liked the practical implications of the project, it wasn’t just theory, it actually built something.”
• Wadeed Fakhoury felt that the project fulfilled two of his interests, “I like to work on cars, and I love to help people. This opportunity to benefit the hard of hearing operating a vehicle was just what I wanted.”
• Michael Mullins enjoys developing electronic games and has a computer science background. He saw the project as an opportunity to add to his coding skills. In fact, he led the coding efforts on the project, making crucial improvements to the machine learning-based sound detection platform developed by last year’s team.
• Faiza Al-Bahrani had perhaps the most compelling interest in joining the mechanical engineering team with its sights set on helping hard of hearing communities. Al-Bahrani has a cochlear implant, without which she is clinically deaf. Removing her implant during work sessions, Al-Bahrani served as the team’s resident Chief of Quality Assurance.

Moving Forward.

This second-year team met Moran’s hopes and expectations for the phase two of the project, combining their skills and interests to build on the existing project.

That initial effort began in 2019 when Moran reached out to 4-VA@Mason with an interest in creating an interdisciplinary capstone program in collaboration with colleagues in the School of Integrated Sciences at James Madison University. The faculty members’ backgrounds ranged from mechanical engineering to political science, Moran wanted to tackle problems that do not fall neatly into one disciplinary category, including the development of renewable energy technologies, autonomous vehicles (often called self-driving cars), the use of robotics in medicine, and more. Moran sought to task students with the goal of addressing public needs; undertaking problems that straddle boundaries between disciplines. “The overarching goal is to create T-shaped graduates who have a general level of knowledge about a broad span of subjects, forming the horizontal part of the T, while cultivating deep knowledge in their own specific area, representing the vertical part,” Moran says.

The first cohort of students, working in the pandemic-affected 2020-21 capstone year, sought to focus on making automobiles easier to use by deaf and hard of hearing populations, with an eye toward self-driving cars, which are expected to number in the hundreds of millions by 2030 and will be used by individuals with varying needs. Armed with data indicating the challenges that deaf and hard of hearing populations face when driving, the students set about outfitting a golf cart with a microphone to detect noises near the vehicle, using machine learning to identify the sound, and creating a seat cushion outfitted with a haptic sensor which vibrates to let the driver know that a hazard is nearby. The driver is then prompted to read a tablet screen mounted on the dashboard that identifies the noise.

Although the 2020 students got a good start, Moran knew there could be more to the project. Much of the in-person work was not possible because of the COVID-19 pandemic, which limited lab work. However, with funding still left in the budget, Moran opted to create a phase two of the effort and asked the next group of students to take up the project.

Take it up, they did. As Hoa “Andy” Huynh, another member of the team leads exclaimed, “We improved on it in every way!”

When the two team leads, Huynh and Jawad, got the project last September, they discussed a plan of activity and started preparing schedules. Jawad explains that the teams did theoretical work over Zoom meetings in the fall. Jawad’s team was tasked with input — working on microphone processing sounds to the laptop, while Huynh’s team focused on output — from the battery to the haptic feedback system. Huynh notes that the two teams initially worked separately on their efforts from September to December of 2021, and then they worked as a group every Tuesday and Thursday in a lab on the Sci Tech campus from January through May 2022. Huynh adds, “Professor Moran attended the Tuesday meetings to check in on our progress, give us feedback, and help us with questions.”

The teams did all the work in-house except for the original code which was written at James Madison University. Mullen accessed that code and built on it. He then went on to convert Spectron graph images to recognize sounds – a dog bark, gun shot, car horn, and siren — through machine learning.

Indeed, the new teams examined every element of the system and made improvements. Huynh says, “First, their system wasn’t integrated, there was only one microphone, the existing Raspberry Pi was not strong enough, we upgraded to a Windows laptop which is much more powerful.” Huynh adds that while the original cart had only one microphone, the new team installed four. “Based on the feedback from the deaf community surveys, we understood that it was important to indicate which direction the sound was coming from,” he says. “Michael was able to develop a system that identifies the sounds in 2.5 seconds and then it appears on the laptop monitor and also indicates the direction from which the sound is emanating.” Jawad adds that the seat cushion was also expanded to include four vibrating haptics which reveals to the driver the direction of the sound.

With the newly enhanced golf cart, the 2021-22 group was ready to share the results on the lawn in front of the Nguyen Engineering Building on the Mason Fairfax campus on May 5 for Capstone Day. Nathan M. Kathir, Associate Professor & Director of Senior Projects, says it was an opportunity to, “See their creativity in-person.”

Cars of the Future.

Moran reflects on the progress made by the second phase team and indicates that, with funding still left in the budget, he’d like to return to another collaboration with Integrated Science and Technology group at JMU next year. “Tremendous possibilities remain with this project; we can take this to the next level by making the sound detection system even faster, training it to recognize a wider array of sounds, or filtering the input noise to pick out the hazard from a noisy background,” says Moran. “We’re also interested in making the intensity of the haptic feedback depend on the distance between the sound source and the vehicle.”

Because it is expected that the coding requirements for the project will be expanded, Moran anticipates adding at least one Computer Science major in the team to take ownership of the increasing demands. Moran also hopes to continue to address the policy-related issues associated with the use of conventional and autonomous vehicles by deaf and hard of hearing communities, explaining, “In keeping with the original vision of this project, I would also like to see next year’s students look at the policy implications of this work, particularly the updates that are needed to the Americans with Disabilities Act of 1990 to enable individuals with various needs to use autonomous vehicles more effectively, since AVs are only going to become more numerous on our roads.”

“After the Capstone Day event, we had several people say to us, ‘I want this for my car’ – even people who have full use of both ears!” Moran adds. “We’re grateful for 4-VA’s continued support and flexibility as we’ve steered this project through two — and soon to be three — academic years, not to mention a global pandemic. We’re excited to see where it goes next.”

Team Vibe Vision: Hoa “Andy” Huynh (team lead), Michael Mullins, Wadeed Fakhoury, Faiza Al-Bahrani
Team Spider Sense: Javeria Jawad (team lead), Jimmy Torrico, Hamzeh Amin, Kyung Min

The Dream Team (Featured photo) Left to right: Wadeed Fakhoury, Kyung Min, Jimmy Torrico, Professor Moran (in cart), Hamzeh Amin, Faiza Al-Bahrani, Javeria Jawad, Hoa “Andy” Huynh, Michael Mullins (kneeling).

4-VA@Mason Team Leads Consumer Privacy Analysis of Personal Assistant Devices  

Today, smart home devices are ubiquitous, and increasingly, are playing a more prominent role in the lives of millions of Americans.  The question is, however, how big of a role?  And is it one that most of us are comfortable with?

These were the topics attracting the attention of Vivian Motti, Assistant Professor in Mason’s College of Engineering and Computing, Department of Information Sciences and Technology, along with two other Virginia professors — Ahmad Salman at James Madison University and Carol Fung (previously at Virginia Commonwealth University), now at Concordia University, Montreal.  Although Motti was aware of the similar paths of work being conducted by Salman and Fung, she saw an opportunity to intersect with them and combine their work via a grant from 4-VA.

Now — two years, dozens of interviews, hundreds of reviews, thousands of hours of analysis, and one pandemic later — “Human-Centric Privacy-Preserving Controls for Smart Home Devices” has delivered a concrete set of privacy controls for smart home devices that are effective and easy for consumers to adopt, and relevant and useful for practitioners to incorporate in the implementation of next-generation smart home devices.  In fact, the research team has already provided these controls for future implementation by community members from academia, industry, and standardization bodies including the National Institute of Standards and Technology (NIST).

Motti began her journey by considering how smart home devices were being used – by both tech savvy and not so tech savvy — consumers.  “What we learned is that the people who are very tech savvy were able to keep their data more private because they know how to configure their network. Consequently, the devices they use only have access to information inside the house and it does not get out,” says Motti.  The trouble began, however, with consumers that were not able to get control of their Amazon Alexa, Echo or Dot, or Google Home products.  “These consumers didn’t know how to access the log, or how to delete it, were in danger of losing their personal information.”  Further, Motti explains, the first versions of the device did not even provide access to these logs to the users.  Consequently, says Motti, consumers started to complain.

But just as Motti’s team’s pencils were sharpened, the pandemic hit.  Labs were shuttered and students were sent home. The plan to conduct individual consumer interviews needed to be scuttled.  The group continued, undaunted. “We could not meet with participants in person, so we modified and amended the protocol of the user studies,” says Motti. “Specifically, we relied more on the analysis of online reviews. Then, we conducted user studies using Zoom and Miro (for the co-design sessions). Lastly, we collected data through Amazon MTurk, reaching a larger number of users and analyzed publicly available online reviews.”

With the data (finally) in hand, the team began parsing out the work.  Salman handled the experimental design and data analysis while Fung evaluated physical prototypes to test controls, collaborating with data collection and analysis from user studies.  Motti’s students got involved remotely, with Chola Chhetri, a Mason Graduate Research Assistant leading the way with experimental design and data collection and analysis.  Chhetri also assisted with papers preparation, submission, and presentations. Graduate students Huining Feng and Haoran Lee helped with the analysis of online reviews, experimental design, and data collection while undergrads Jacob Cox and Joseph Aversa looked at graphic user interfaces for privacy controls.

The next hurdle was to aggregate the information and data collected and present it to stakeholders who could impact how the information is implemented within the industry.  “Chola led the meeting with advisory board members from academia, industry, and NIST, sharing the major findings as well as the recommendations and suggestions that we developed to improve current devices,” says Motti.  “It was great because he received very positive feedback about the validity of the work, and what the industry must first recognize to better understand the needs of consumers and end users, and second, to recommend what should be implemented and deployed in the next generation devices.”

While the pace of both compliance and legislation has been slow and reactive in the personal assistant environment, Motti says that a pathway forward is now in the hands of a breath of consumers, industry, and regulators thanks to the 4-VA grant.  In fact, their findings have been widely distributed at the National Cyber Summit, Human Factors in Cybersecurity, Human Aspects of Information Security & Assurance, and the International Conference on Information. Additionally, the study will appear in the Association for Computing Machinery Conference on Computer Supported Cooperative Work, and at the Human Factors and Ergonomics Society Annual Meeting.

But Motti sees a longer road ahead, “This grant allowed me to start with an exploratory approach — we looked at the online polls, looked at the literature, interviewed and surveyed participants. But it also sparked new research questions, new areas we would like to test and to go into more depth. Once we saw the results, we know that there is still more work to be done. So, we plan to apply for larger grants from the Commonwealth Cyber Initiative and the National Science Foundation to have more validation for future work related to the project.”

Alexa will be listening…