Democratizing Virtual Assistants:
A Social-Good Research Project Course

NN Icon
Almond Icon
idea Icon

Course Description

CS294S/CS294W is designed to offer students a remote research experience, in contrast to online lecture-style courses.

This course focuses on building a state-of-the-art social-good virtual assistant, a project supported by the Alfred P. Sloan Foundation to protect privacy and open access of knowledge. Opportunities to continue the research after the Fall Quarter are available.

  • State of the art. We can now automatically generate neural-based dialogue agents from a knowledge base, without writing dialogue trees and collecting annotated utterances painstakingly. This opens up many research opportunities in NLP, HCI, and systems.
  • Social good. We aim to give consumers an open-source, privacy-preserving assistant whose skills are all openly available in Thingpedia, a nonproprietary crowdsourced repository. This project provides a much-needed alternative to the emerging oligopoly of proprietary assistants and skill repositories.

This course is open to all undergraduates, masters, and PhD students, who have taken at least two courses in Computer Science. Students will learn:

  • The latest voice-assistant technology.
  • Interdisciplinary research in NLP, systems, HCI.
  • Experience with a large-scale social-good project.
  • Technology to disrupt surveillance capitalism.

This course consists of a few lectures on the latest technology, hands-on tutorials, small-group mentorships, interactive class discussions, and group presentations. Groups of 2 or 3 will choose or propose projects in subjects such as:

  • AI: machine learning, natural language processing, and knowledge extraction.
  • HCI: crowdsourced research, virtual assistant design, multimodal interface design, and user studies.
  • System: Formal knowledge representation and semantics, programming in natural language, and internet of things.

You can take this course multiple times for credit. CS294S can be taken to fulfill the CS194 requirement. Please sign up for CS294W if you wish to fulfill your writing requirement as well. (CS294W requires students to meet with an advisor from the Technical Communication Program.)

Grading

Attendance is mandatory, please let us know if you can’t make it to class.
  • Class Participation: 15%
  • Homework: 15%
  • Final Project: 70%

Course Links

Research Areas

Multidisciplinary

  • End-to-end projects. We plan to release the top 10 virtual assistant skills by June 2021. These skills are listening to music, radio, news, sports, podcasts, answering questions, checking the weather, setting timers and alarms, controlling IoTs, and popular third-party skills. Can we match and improve upon commercial assistants' performance? For example, can we answer questions about artists, or help users monitor the scores of their favorite teams.
  • Social-good projects. Examples include assistants that provide safe voting or legal advice.
  • Privacy-focused projects. Examples include a personal investment assistant that monitors the market, aggregates information, and trades algorithmically, such as "buy 3 shares of Apple stock when it drops to $120."
  • New assistant capabilities. Design dialogues and dashboards so assistants can perform "cron" jobs (lower the heat at 1am), monitor information (air quality, news story), or perform conditional actions (turn off the sprinkler if rain is forecast).

    • Natural Language Processing (NLP)

  • Accuracy: better entity disambiguation
  • Usability: improve naturalness in agent responses
  • Scalability: meta-learning new domains
  • Error recovery:effective dialogues to recover and learn from mistakes
  • Localization:use machine translation to create virtual assistants in all languages

    • Human-Computer Interaction (HCI)

  • Crowdsourcing contributions. How to build a community to crowdsource contributions from user feedback, new skills, new ontologies, and data for international languages.
  • Multimodal assistants. Explore using multimodality to let end-users program long-tail skills and better handle custom support calls.

    • Systems

  • Knowledge representation. Our virtual assistants map natural language into a formal execution language, called ThingTalk. How do we represent the semantics of well-known data types, such as time and space?
  • Dialogue design. Extend conversational models from simple transactions to include FAQs, on-topic discussions, etc.

    • Schedule

    The course meets Tuesday and Thursday, from 10:30am to 11:50am Pacific Time via Zoom. Please see Canvas for Zoom links.

    This schedule is tentative and subject to change. Please pay attention to emails sent to the student list. Schedule for previous school year (spring 2020) can be found here.


    Date Description Course Materials Events Deadlines
    Tue September 15 Course Introduction
    [slides]
    		Suggested Readings:
    1. Almond: The Architecture of an Open, Crowdsourced,Privacy-Preserving, Programmable Virtual Assistant
    2. Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commands
    		 Student Profiles out
    [link] (requires Stanford login)
    Signup spreadsheet out
    [link] (requires Stanford login)
    Thu September 17 Lecture: Schema → Q&A
    [slides]
    		Suggested Readings:
    1. Schema2QA: High-Quality Low-Cost Q&A Agents for the Structured Web
    2. AutoQA: From Databases To Q&A Semantic Parsers With Only Synthetic Training Data
    		 HW1 (See Canvas for more details) out
    [link]     		Student Profiles due
    Tue September 22 Project Discussions Project Pitches
    Thu September 24 Lecture: Schema → Dialogues
    [slides]
    		Suggested Readings:
    1. State-Machine-Based Dialogue Agents with Few-Shot Contextual Semantic Parsers
    2. Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking
    		 Homework due
    (11:59PM PT)
    Tue September 29 Project Discussions Project Pitches
    Thu October 1 Project Discussion
    Tue October 6 Project Proposals
    		Project Proposals
    Thu October 8 Project Proposals Project Proposals
    Tue October 13 Project Proposals Project Proposals
    Thu October 15 Lecture: An Overview of NLP
    [slides]
    		Suggested Readings:
    1. CS224n's Slides for the topic you are interested in
    Tue October 20 Weekly Group Meetings
    		Weekly Updates due
    Thu October 22 Students' Mini-Lecture Mini-lectures
    Tue October 27 Weekly Group Meetings Weekly Updates due
    Thu October 29 Students' Mini-Lecture Mini-lectures
    Tue November 3 Office Hour
    Thu November 5 Students' Mini-Lecture Mini-lectures Weekly Updates due
    Tue November 10 Weekly Group Meetings Weekly Updates due
    Thu November 12 Students' Mini-Lecture Mini-lectures
    Tue November 17 Final Project Presentations Final Project Presentations
    Thu November 19 Final Project Presentations Final Project Presentations
    Fri November 20 Project Report due at 11:59PM PT

    Resources

    Almond Virtual Assistant

    1. Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant
      Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam.
      In Proceedings of the 26th World Wide Web Conference - WWW 2017.

    2. Controlling Fine-Grain Sharing in Natural Language with a Virtual Assistant
      Giovanni Campagna, Silei Xu, Rakesh Ramesh, Michael Fischer, and Monica S. Lam.
      In Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing - Ubicomp 2018.

    3. Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commands
      Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam.
      In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation - PLDI 2019.

    Dialogue State Tracking

    1. State-Machine-Based Dialogue Agents with Few-Shot Contextual Semantic Parsers
      Giovanni Campagna, Sina J. Semnani, Ryan Kearns, Lucas Jun Koba Sato, Monica S. Lam
      arXiv preprint - 2020.

    2. Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking
      Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, and Monica S. Lam
      In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) - ACL 2020.

    3. MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines
      Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyang Gao, Dilek Hakkani-Tur
      In arXiv preprint - 2019.

    Semantic Parsing

    1. Seq2SQL: Generateing Structured Queries from Natural Language using Reinforcement Learning
      Victor Zhong, Caiming Xiong, and Richard Socher
      In arXiv preprint - 2017

    2. Neural Semantic Parsing with Type Constraints for Semi-Structured Tables
      Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner.
      In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing - EMNLP 2017

    3. Data Recombination for Neural Semantic Parsing
      Robin Jia and Percy Liang.
      In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

    4. Language to Logical Form with Neural Attention
      Li Dong and Mirella Lapata.
      In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

    5. SQLNet: Generating Structured Queries From Natural Language without Reinforcement Learning. [Github]
      Xiaojun Xu, Chang Liu, and Dawn Song.
      In arXiv preprint - 2017

    6. Learning a Neural Semantic Parser from User Feedback. [Github]
      Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, Luke Zettlemoyer.
      In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

    7. The Alexa Meaning Representation Language
      Thomas Kollar, Danielle Berry, Lauren Stuart, Karolina Owczarzak, Tagyoung Chung, Lambert Mathias, Michael Kayser, Bradford Snow, Spyros Matsoukas
      In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers) - NAACL - 2018

    Question Answering Over Knowledge Base

    1. Schema2QA: High-Quality and Low-Cost Q\&A Agents for the Structured Web
      Silei Xu, Giovanni Campagna, Jian Li, Monica S. Lam
      To appear in Proceedings of the 29th ACM International Conference on Information and Knowledge Management - CIKM - 2020

    2. AutoQA: From Databases To Q&A Semantic Parsers With Only Synthetic Training Data
      Silei Xu, Sina J. Semnani, Giovanni Campagna, Monica S. Lam
      To appear in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing - EMNLP - 2020.

    3. Localizing Q&A Semantic Parsers for Any Language In a Day
      Mehrad Moradshahi, Giovanni Campagna, Sina J. Semnani, Silei Xu, Monica S. Lam
      To appear in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing - EMNLP - 2020.

    4. Learning a Natural Language Interface with Neural Programmer
      Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum, and Dario Amodei
      In Proceedings of the 5th International Conference on Learning Representations - ICLR 2017

    5. Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
      Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, Ni Lao.
      In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

    6. SEMPRE: Semantic Parsing with Execution
      Jonathan Berant, Percy Liang at al. 2013 - 2017

    Question Answering Over Free Text

    1. Reading Wikipedia to Answer Open-Domain Questions. [Github]
      Danqi Chen, Adam Fisch, Jason Weston and Antoine Bordes.
      In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

    2. SQuAD: 100,000+ Questions for Machine Comprehension of Text
      Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
      In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing - EMNLP 2016
      [SQuAD leaderboard]

    Brassau

    1. Brassau: Automatically Generating Graphical User Interfaces for Virtual Assistants
      Michael Fischer, Giovanni Campagna, Silei Xu, and Monica S. Lam.
      In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI 2018

    Data Programming

    1. Snorkel: Fast Training Set Generation for Information Extraction
      Alexander J. Ratner, Stephen H. Bach, Henry R. Ehrenberg, and Chris Ré.
      In Proceedings of the 2017 ACM International Conference on Management of Data - SIGMOD 2017

    2. Fonduer: Knowledge Base Construction from Richly Formatted Data
      Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Chris Ré
      In Proceedings of the 2018 ACM International Conference on Management of Data - SIGMOD 2018

    Others

    1. World of Bits: An Open-Domain Platform for Web-Based Agents
      Tim (Tianlin) Shi, Andrej Karpathy, Jim Fan, Jonathan Hernedez, Percy Liang
      In Proceedings of the 34th Internationl Conference on Machine Learning - ICML 2017

    2. Projects from previous iterations (2018)
    3. Projects from previous iterations (2017)

    Teaching Staff

    Instructor

    Monica Lam

    Professor


    Office hours by appointment via Zoom

    Teaching Assistant

    Silei Xu

    silei@stanford.edu
    Office hours via Zoom