CS294S is a research project course to explore how consumers will interact with software in the future. While Alexa, Siri, and Google Now can only handle simple commands today, the virtual assistant of the future will handle sentences such as “open the blinds when the alarm goes off”, “find a tennis partner in the afternoon”, and “remind me to call mom when I finish playing this VR game”. Ultimately, the goal is to develop technology that lets end-users “program” using natural language.
Speaker Recognition (Quinlan Jung, Michael Xing, Prasad Kawthekar, Junwon Park)
Understanding who is speaking is a crucial part of a multi-user virtual assistant. This project implements a multi-user speech interface that recognizes the speaker and maps the command to his virtual assistant. Applications include customization, parental control, as well as meeting transcriptions and the ability for a VA to sit passively in a conversation and react when called.
Data Visualization for Virtual Assistants (Michael Fischer)
Voice, the primary interaction modality of commercial VAs such as Echo and Google Home, is a great form of input, but not the best output modality for all answers. This project explores how a virtual assistant can produce semantically rich data visualizations on a large screen.
SNER: Specific (Snorkel) Named Entity Recognition for Almond (Brian Higgins)
Named Entity Recognition is a well-studied problem in the NLP literature, but the commonly used 5 classes (Person, Organization, Location, Misc, Other) are not sufficient for virtual assistants. This project explores the quick creation of a dataset to train a state-of-the-art recognizer for a new class of entities.
Mango: Location-Aware Assistant (Derin Dutz)
Location is a central concept in a mobile virtual assistant. As people move about, the virtual assistant moves with them, knows where they are, and reacts accordingly. This project explores the infrastructure and APIs to provide continuous tracking and support numerous use cases.
M-Almond: Automating Multi-Party Tasks with End-User Programmable Virtual Assistants (Rakesh Ramesh, Giovanni Campagna, Silei Xu, Michael Fischer)
How does a doctor get access to a patient’s IoT information? How can one monitor his parents security camera while they are away? M-Almond lets users specify these tasks in natural language, and automatically asks for permission, all without divulging personal information to a 3rd party.
ABM (Almond Bike Market): A Marketplace Virtual Assistant Augmented by Collaborative Information Sharing (Luke Chen, Silei Xu)
Every Stanford student has a checklist of tasks to do on his first days, and high on that checklist is buying a bike. ABM improves the experience of buying and selling bikes, with a marketplace that can be queried in natural language, and the ability to connect buyers and sellers through a virtual assistant.
Annie: a Social Virtual Assistant for Shooting (Alex Leishman, Megan Wilson)
Can a virtual assistant help with the sport of shooting? This project explores a virtual assistant that uses computer vision to let people track their shooting performance at ranges, with regulatory targets, and share it with friends.
VIRA: A Virtual Assistant for Gamers (Ikechi Akujobi, Matthew Chen, Chris Salguero)
Gaming, and specifically gaming in Virtual Reality, is all about immersion. Can we use a voice virtual assistant to help us with tasks, such as searching for walkthroughs online, that would otherwise interrupt the immersion?
NutriCoach: Tracking Nutrition with A Virtual Assistant (Katy Shi, Trina Sarkar, Ana Caro Mexia, Shawn Fenerin)
This project helps us manage our diet and find foods that satisfy our dietary restrictions. It provides a natural language interface to log meals, and it automatically learns recommendation for next meals based on the right calorie and protein budget, as well as user preferences.
Milo: Assistant for Anxiety (Clay Jones, Kevin Rakestraw, Andrew Declerck)
Many people, including students, are affected by chronic anxiety and anxiety attacks. This project helps them track and visualize their behavior over time, recorded from a wearable device (FitBit) and by answering questions. We perform an exploratory study, where one user is followed for 45 days, and report on the behavioral correlations we observe.
Virtual Assistants for Patients and Doctors (Jason Liu, Stephanie Palocz, Albert Chu)
More than 40% of Americans suffers from a chronic disease, and about 1 in 12 suffers from asthma. Managing these diseases requires constant monitoring and consistent compliance with medication. This project explores the use of a virtual assistant based on M-Almond to let the doctor and the patient share information automatically, both collected from IoT devices and manual input from the patient.
CS294S is a project course designed to help students with their first research experience. The development of future intelligent systems requires expertise in diverse areas, such as AI, HCI, programming languages, distributed systems, networking, and security. Students interested in any of the above topics are invited to attend. This class is intended for undergraduate and graduate students who have taken at least two Computer Science courses.
Students, in groups of 1 to 3, can propose their projects or choose among suggested topics. Available to the students is the open-source Almond virtual assistant infrastructure, which can be used to prototype a natural language interface in just one day.
Students are required to come to campus for this course, as in-class participation and after-class project meetings are required. Students can take this course multiple times for credit. CS 294S can be taken to fulfill the CS 194 senior project requirement. Students can sign up for CS 294W if they wish to fulfill their writing requirement as well.
10 minutes + 5 minutes for questions. Your proposal should include:
15 minutes + 3 minutes for questions. Your presentation should include:
There is no page limit (minimum or maximum), but CS294W students must have a substantial writing. The final report is due June 13 electronically to email@example.com. Your report should include:
What would the conversation be like with a programmable virtual assistant? For example, “please make sure the heat is on by the time I come back from work every day”, or “have the TV play every Warrior game whenever I am home”.
What if our virtual assistant is a home robot, a drone, or a bike? Can it go and get the delivery from Amazon when it arrives? Can it pick up laundry or pizza? Can it help make sure a kid is safe going to school?
A patient’s virtual assistant can help manage chronic diseases such as asthma. For example, an asthma patient should not run when a certain kind of pollen count is high or the doctor should be informed if the reading on the patient's flow meter falls below a threshold.
Uber enables drivers and riders find each other with the help of GPS. There is a long tail of sharing-economy markets, such as buying and selling bicycles on campus or finding partners to play tennis. Can we create a flexible system that enables virtual assistants to help connect people across a diverse set of markets?
With no access to the keyboard or the mouse, speech is a natural way for VR users to interact with their virtual surroundings. What makes a good speech-based virtual assistant for VR? (Two HTC Vives are available for students to work with in this class).
Almond has a repository to support semantic parsing, mapping natural language snippets into code. Can we scrape the web automatically or use crowdsourcing to populate this knowledge base?
How can we crowdsource data to enable machine learning? How can we improve the machine learning algorithm to translate natural language into code.
Graphical interfaces are useful when natural language processing fails and when there is a lot of information to display. How do we create multi-modal interfaces that allow users to easily move between speech and more traditional UIs or to use them together at the same time?
The course meets Tuesday and Thursday, from 10:30 AM to 11:50 AM in Gates 100.
This schedule is tentative and subject to change. Please pay attention to emails sent to the student list.
|Tue April 4||Overview of the course. Overview of the Almond virtual assistant.|
|Thu April 6||Intro + Brainstorming|
|Tue April 11||Project discussions|
|Thu April 13||More ideas + in-class discussions. Homework 1 released.|
|Tue April 18||Mini Hackathon|
|Thu April 20||Project Proposal. Homework 1 due|
|Tue April 25||Project Proposal|
|Thu April 27||Project Proposal|
|Tue May 2||Student-led discussion: Getting Started with NLP|
|Thu May 4||Student-led discussion: NLP Semantic Parsing|
|Tue May 9||Student-led discussion: NLP Everything Else|
|Thu May 11||Student-led discussion: Speech|
|Tue May 16||Mini hackathon / work session|
|Thu May 18||Student-led discussion: Displays (big screens, VR)|
|Tue May 23||Student-led discussion: IoT|
|Thu May 25||Student-led discussion: Health|
|Tue May 30||Final project discussion|
|Thu June 1||Final project discussion|
|Tue June 6||Final project discussion|
|Mon June 12: (12:15-3:15 pm)||Final Project Demo and Poster Session (during the scheduled Final Exam period)|
|Tue June 13||Final Report Due|
Office hours: by appointment
Office hours: Monday and Wednesday, 3 PM to 4 PM (starting Wed April 12th)