CS294S, CS294W: A Project Course on Human-Centered Software: AI meets Systems.

CS294S is a research project course to explore how machine learning can revolutionize consumers' software experience. Students are invited to do quarter-long projects to answer questions such as those listed below or propose their own topics

NN Icon
Almond Icon
idea Icon

Course Design

A Project Based Course.

CS294S is a cross-disciplinary research project course. Students interested in Systems (programming, distributed, blockchain), AI (NLP, ML, data engineering), and HCI (design and crowdsourcing) are invited to attend the class. Students must have taken at least two computer science courses; no prior research experience is needed. This course emphasizes individual mentorship and learning through projects, rather than lecture-style learning. Groups of 2 or 3 students define their own quarter-long projects, either in developing technology or applications that cross the Systems-AI-HCI boundaries. Some examples of projects are shown below.

Because of the cross-disciplinary nature of these projects, many faculty members have been involved and will continue to provide advice: Adjunct Prof. Ed Chang (President of Healthcare, HTC), Prof. James Landay (HCI), Prof. David Mazieres (Blockchain, distributed systems), Prof. Ge Wang (Music), Prof. Lei Xing (Radiology Oncology, Medicine), and Prof. Alan Yeung (Cardiology, Medicine).

Students are required to come to campus for this course, where in-class participation and after-class project meetings are required. You can take this course multiple times for credit. CS 294S can be taken to fulfill the CS 194 requirement. Please sign up for CS 294W if you wish to fulfill your writing requirement as well.

Logistics

Grading

  • Class Participation: 15%
  • Homework: 5%
  • Project: 80%

Class Spreadsheet: link

Research Areas & Project Topics

Here are some suggested project topics:

Machine Learning

We speculate that in about 5 years, there will be a deep neural network that can understand all the digital tasks consumers wish to automate. Examples of research include using dialogs to define a program incrementally and creating a linguistic database of digital interfaces.

HCI

Today’s commercial virtual assistants have poor discoverability, how do we create a much better experience? How do we combine the best of linguistic and graphical user interfaces in an application? Can we create a specification that generates either or both of these interfaces?

Systems

Can we build MedXChange: a federated medical image data exchange that lets users control the sharing of data between institutions. The key is to create smart contracts that are written in natural language and enforceable with a scalable block chain technology. Can we create an assistant-agnostic skill service that developers need to write once and have the skill be available to all assistants, hence preventing lock-ins to proprietary assistants.

Applications

Can we build special-purpose virtual assistants that

  • help a cardiologist monitor his patient’s blood pressure?
  • help a consumer create photo-realistic images using natural language or compose music?
  • help a consumer access and share all their data in natural language, taking advantage of the EU General Data Protection Regulation (GDPR)?
  • help a student with his studies, research, or social life?

Schedule

The course meets Tuesday and Thursday, from 10:30 AM to 11:50 AM in Gates 100.

This schedule is tentative and subject to change. Please pay attention to emails sent to the student list.

Date Topic
Tue April 2 Overview of the course (slides)
Thu April 4 Intro + brainstorming + demo. (Homework 1 released)
Tue April 9 Project discussions
Thu April 11. More ideas + in-class discussions. (Homework 1 due)
Tue April 16 Mini hackathon (slides)
Thu April 18 Project proposals
  1. I did not mean that! (Giovanni Campagna)
  2. A Framework to Build Domain-Specific Virtual Assistants (Silei Xu)
Tue April 23 Project proposals
Thu April 25 Project proposals
Tue April 30 Tutorial: NLP
Thu May 2 Tutorial: Recommender Systems
Tue May 7 Guest Speaker: Medical assistants (Dr. Edward Chang)
Thu May 9 Student-led discussion: Virtual Assistant Interventions: Positive & Negative Examples
Tue May 14 Student-led discussion: Virtual Assistants: Current & Future
Thu May 16 Mini hackathon / work session
Tue May 21 Student-led discussion: Privacy: Regulations & Technology
Thu May 23 Student-led discussion: Privacy: Distributed Systems
Tue May 28 Final project discussion
Thu May 30 Final project discussion
Tue June 4 Final project discussion
Mon June 10: (1:30-3:15 pm) Final project demo (3 minute video) and poster session (during the scheduled final exam period)
Tue June 11 Final report due

Guidelines

Project Proposal Guidelines

10 minutes + 10 minutes for questions and discussion. Your proposal should include:

  • What is the motivation of the project?
  • What is the state of the art?
  • What will you do?
  • What do you think you will learn at the end?
  • What will be your demo?
  • What will each of you do at each step? (Detailed schedule)

Final Discussion Guidelines

15 minutes + 3 minutes for questions. Your presentation should include:

  • What was the project at the beginning? (Quick refresher for the class)
  • What did you do?
  • What are your results?
  • What did you learn?
  • What would you do if you had more time? (Future work)

Final Project Demo Guidelines

A 3-minute video to show off your project!

Final Report Guidelines

There is no page limit (minimum or maximum), but CS294W students must have substantial writing. Your report should include:

  • Introduction and motivation
  • Design and implementation
  • Results
  • What did you learn
  • Related work
  • Future work
  • Conclusion

Resources

Almond virtual assistant

  1. Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant
    Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam.
    In Proceedings of the 26th World Wide Web Conference,
    Perth, Australia, April 2017.

  2. Controlling Fine-Grain Sharing in Natural Language with a Virtual Assistant
    Giovanni Campagna, Silei Xu, Rakesh Ramesh, Michael Fischer, and Monica S. Lam.
    In Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing - Ubicomp 2018.

  3. Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commands
    Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam.
    Conditionally accepted to Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, Phoenix, AZ, June 2019.

Semantic parsing

  1. Seq2SQL: Generateing Structured Queries from Natural Language using Reinforcement Learning
    Victor Zhong, Caiming Xiong, and Richard Socher
    In arXiv preprint - 2017

  2. Neural Semantic Parsing with Type Constraints for Semi-Structured Tables
    Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner.
    In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing - EMNLP 2017

  3. Data Recombination for Neural Semantic Parsing
    Robin Jia and Percy Liang.
    In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

  4. Language to Logical Form with Neural Attention
    Li Dong and Mirella Lapata.
    In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

  5. SQLNet: Generating Structured Queries From Natural Language without Reinforcement Learning. [Github]
    Xiaojun Xu, Chang Liu, and Dawn Song.
    In arXiv preprint - 2017

  6. Learning a Neural Semantic Parser from User Feedback. [Github]
    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, Luke Zettlemoyer.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

Question answering over knowledge base

  1. Learning a Natural Language Interface with Neural Programmer
    Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum, and Dario Amodei
    In Proceedings of the 5th International Conference on Learning Representations - ICLR 2017

  2. Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
    Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, Ni Lao.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

  3. SEMPRE: Semantic Parsing with Execution
    Jonathan Berant, Percy Liang at al. 2013 - 2017

Question answering over free text

  1. Reading Wikipedia to Answer Open-Domain Questions. [Github]
    Danqi Chen, Adam Fisch, Jason Weston and Antoine Bordes.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

  2. SQuAD: 100,000+ Questions for Machine Comprehension of Text
    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
    In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing - EMNLP 2016
    [SQuAD leaderboard]

Brassau and style transfer

  1. Brassau: Automatically Generating Graphical User Interfaces for Virtual Assistants
    Michael Fischer, Giovanni Campagna, Silei Xu, and Monica S. Lam.
    In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI 2018

Data programming

  1. Snorkel: Fast Training Set Generation for Information Extraction
    Alexander J. Ratner, Stephen H. Bach, Henry R. Ehrenberg, and Chris Ré.
    In Proceedings of the 2017 ACM International Conference on Management of Data - SIGMOD 2017

  2. Fonduer: Knowledge Base Construction from Richly Formatted Data
    Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Chris Ré
    In Proceedings of the 2018 ACM International Conference on Management of Data - SIGMOD 2018

Others

  1. World of Bits: An Open-Domain Platform for Web-Based Agents
    Tim (Tianlin) Shi, Andrej Karpathy, Jim Fan, Jonathan Hernedez, Percy Liang
    In Proceedings of the 34th Internationl Conference on Machine Learning - ICML 2017

  2. Projects from CS294S last year

Teaching Staff

Monica Lam

Professor


Office hours: by appointment

Sam Premutico

Teaching Assistant


Office hours: Wednesday 3PM to 4PM, Gates B26