PeerSupport

PeerSupport is an anonymous peer support chat application that allows users to message each other without the worry of their identity being known by their peers. Users can flag risky messages (e.g. Suicidal Ideation, Potential Violence, Harassment), which only monitor admins (e.g. school counselors) can view and decide the best course of action to ensure a safe environment. The flagging of messages can also be done by a machine learning algorithm.

To view images of the functionality in action, CLICK HERE!

The video above was our winning submission to the ESHIP.2 Solutions Challenge (a.k.a. Change The World Challenge)

HackRPI

President, Director of Logistics, and Master of Ceremonies

In the Fall of 2018, I discussed the creation and rekindling of RPI's annual hackathon with Chuck Stewart, RPI's Computer Science Department head. The last HackRPI event was in the Fall of 2017, where the final organizers left the organization. My team and I built HackRPI from the ground up to host HackRPI in the Spring of 2019 where we offered more than 80K worth of prizes. Some of the companies we managed to get a sponsor from where Bloomberg, TripAdvisor, Tensorflow, Palantir, and many others! By our next event in the Fall of 2019, we partnered with IBM and had 400 attendees.

In order to keep HackRPI consistently hosting annual hackathons at RPI, I established HackRPI as an official organization through the Computer Science Department, wrote a Constitution & Bylaws, and constructed an effective organization hierarchy with a new member recruiter position.

SAFEWheel (Smart Alerts for Everyone Wheel)

SAFEWheel is a system made up of a steering wheel cover and a photo capturing/analyzing device, that sounds an alarm when the driver’s hands are off the wheel or their eyes are closed. The data is stored on a web application for data visualization and analysis on driving performance. The system uses Google App Engine, Cloud Storage, AutoML Vision, a Raspberry Pi w/ Camera Module, and pressure sensors. The work appeared on Google’s internal newsletter as it showcased the usage of their Google Cloud Platform.

Suicidal Ideation Text Classification

The team members and I who were working on the PeerSupport project, created our own suicidal ideation test set by self annotating almost 2K pieces of text from Twitter and Reddit (r/SuicideWatch). I performed a grid search on multiple Support Vector Machine hyperparameters. The final accuracy based on the test set came out to be 84%. The most important feature used was TF-IDF (term frequency-inverse document frequency). Each step of the process can be followed through my Google Colab project

Multiparty Interactions with a Robot Using Large Language Models

Speaker diarization is a key component of systems that support multiparty interactions of co-located users, such as meeting facilitation robots. The goal is to identify who spoke what, often to provide feedback, moderate participation, and personalize responses by the robot. Current systems use a combination of acoustic (e.g. pitch differences) and visual features (e.g. gaze) to perform diarization, but involve the use of additional sensors or require overhead signal processing efforts. Alternatively, automatic speech recognition (ASR) is a necessary step in the diarization pipeline, and utilizing the transcribed text to directly identify speaker labels in the conversation can eliminate such challenges. With that motivation, we leverage large language models (LLMs) to identify speaker labels from transcribed text and observe an exact match of 77% and a word level accuracy of 90%. We discuss our findings and the potential use of LLMs as a diarization tool for future systems.

Virtual Agents for Alcohol Use Counseling: Exploring LLM-Powered Motivational Interviewing

We introduce a novel application of LLMs in developing a virtual counselor capable of conducting motivational interviewing (MI) for alcohol use counseling. Access to effective counseling remains limited, particularly for substance abuse, and virtual agents offer a promising solution by leveraging LLM capabilities to simulate nuanced communication techniques inherent in MI. Our approach combines prompt engineering and integration into a user-friendly virtual platform to facilitate realistic, empathetic interactions. We evaluate the effectiveness of our virtual agent through a series of studies focusing on replicating MI techniques and human counselor dialog. Initial findings suggest that our LLM-powered virtual agent matches human counselors' empathetic and adaptive conversational skills, presenting a significant step forward in virtual health counseling and providing insights into the design and implementation of LLM-based therapeutic interactions.

Engaging and Entertaining Adolescents in Health Education Using LLM-Generated Fantasy Narrative Games and Virtual Agents

Games have been successfully used to provide engaging health interventions for adolescents. However, translating health education goals into a playable game has historically taken many person-months of effort, involving game designers, scriptwriters, and artists. This work presents an exploratory study into rapidly developing physician-validated health education games for adolescents using virtual agents and LLMs. We evaluated this approach in an intervention to promote Human Papillomavirus (HPV) vaccination among adolescents, as lack of knowledge and vaccine hesitancy contribute to suboptimal HPV vaccination rates. We conducted a between-subjects randomized study comparing a fantasy narrative game to a non-gamified pedagogical virtual agent, with both interventions conveying the same HPV information. Among our study’s 9-12-year-old adolescent participants, our findings demonstrate large pre-to-post improvements in HPV knowledge for both conditions. The gamified intervention showed higher engagement and entertainment than the pedagogical agent based on participant interviews, demonstrating that gamification enriched the educational experience for adolescents.

Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents

We introduce the concept of “empathic grounding” in conversational agents as an extension of Clark’s conceptualization of grounding in conversation in which the grounding criterion includes listener empathy for the speaker’s affective state. Empathic grounding is generally required whenever the speaker’s emotions are foregrounded and can make the grounding process more efficient and reliable by communicating both propositional and affective understanding. Both speaker expressions of affect and listener empathic grounding can be multimodal, including facial expressions and other nonverbal displays. Thus, models of empathic grounding for embodied agents should be multimodal to facilitate natural and efficient communication. We describe a multimodal model that takes as input user speech and facial expression to generate multimodal grounding moves for a listening agent using a large language model. We also describe a testbed to evaluate approaches to empathic grounding, in which a humanoid robot interviews a user about a past episode of pain and then has the user rate their perception of the robot’s empathy. We compare our proposed model to one that only generates non-affective grounding cues in a between-subjects experiment. Findings demonstrate that empathic grounding increases user perceptions of empathy, understanding, emotional intelligence, and trust. Our work highlights the role of emotion awareness and multimodality in generating appropriate grounding moves for conversational agents.

SimPatient: Emotionally Realistic Simulated Patients for Counselor Empathy Training

This research explores using LLMs to create emotionally realistic simulated patients for counselor empathy training. We introduce SimPatient, a system embodied by the Furhat robot, designed to help novice counselors practice their skills. Two versions were tested: a control with basic dialogue and an intervention with appraisal-based emotion modeling and nonverbal behaviors (NVBs). A study with 6 participants compared perceived realism and impact on counselor empathy. While the emotion model enhanced perceived realism, particularly regarding NVBs, it did not increase empathy and, surprisingly, significantly decreased it in the intervention group. This suggests that adding emotional responses alone may not improve empathy and highlights the complexity of simulating realistic patient interactions, especially with resistant patients. Future research with larger samples and refined implementations is needed.

RLAIF with User Simulations for Task-Specific Dialogue Agents

This project is a framework for fine-tuning LLMs using a dynamic Reinforcement Learning from AI Feedback (RLAIF) loop. Instead of relying on static datasets, it uses an AI User Simulator to generate live conversations and an AI Reward Model to provide real-time performance feedback, enabling the dialogue agent to learn and adapt continuously. The initial use case demonstrates training a therapeutic chatbot by interacting with simulated patient personas, comparing both Direct Preference Optimization and REINFORCE algorithms. The AI User Simulator incorporates a dynamic cognitive-affective model to both portray a clinically realistic patient, as well as provide nuanced reward signals for fine-tuning (e.g., risk, safety, therapeutic alliance metrics).

Generative AI Art Therapy Video Game

This project is a proof-of-concept interactive 3D environment using Unreal Engine 5 designed to assist in alcohol use counseling. It features a virtual art exhibit where users can engage in a conversation with an empathetic NPC guide. The guide is powered by a Large Language Model (LLM) and utilizes Motivational Interviewing (MI) techniques to facilitate a self-discovery process for the user. A key feature is the dynamic generation of personalized art based on the user's conversation, creating a unique and reflective therapeutic experience.

AI-Generated Social Media Content

As a hobby, I create AI-generated ASMR content using Google's Veo 3. Across TikTok, YouTube, and Instagram, I've built a community of over 100k subscribers and 150 million views.