CS294S, CS294W: Human-Centered Software: AI Meets Systems.

CS294S is a research project course to explore how machine learning can revolutionize consumers' software experience. Students are invited to do quarter-long projects to answer questions such as those listed below or propose their own topics

NN Icon
Almond Icon
idea Icon

Course Design

A Project Based Course.

CS294S is a cross-disciplinary research project course, designed to help students with their first research experience, emphasizing individual mentorship and learning through projects, rather than lecture-style learning. Students interested in AI, HCI, or programming systems, who have taken at least two computer science courses are welcome to attend.

Students, in groups of 1 to 3, can propose their projects or choose among suggested topics. Available to the students is the open-source Almond virtual assistant infrastructure, which can be used to prototype a natural language interface in just one day.

Students are required to come to campus for this course, where in-class participation and after-class project meetings are required. You can take this course multiple times for credit. CS 294S can be taken to fulfill the CS 194 requirement. Please sign up for CS 294W if you wish to fulfill your writing requirement as well.

Logistics

Grading

  • Class Participation: 15%
  • Homework: 5%
  • Project: 80%

Class Spreadsheet: link

Research Areas & Project Topics

Here are some suggested project topics:

End-user programming in natural language.

We wish to explore if end-user programming is possible. Can software be tailored, or even created, using natural language according to individuals' needs and interests?

Personalized natural language understanding

Virtual assistants are the first of such systems, giving users a uniform personalized natural-language based interface. Can we improve their accuracy with personalization? Can we enable non-technical individuals to automate mundane tasks by scripting their virtual assistant using natural language, possibly with the help of examples? We imagine in the future, knowledge workers like loan officers or stock traders will train their own professional virtual assistants.

Personal data sharing

Today, consumers have little choice other than Facebook to share information easily with each other. Sharing is technically difficult because of access control, which is notoriously user-unfriendly. Can machine learning revolutionize sharing by giving users a natural interface?

Complementing natural language with graphical interfaces

In the future, we will see a combination of language and graphical user interfaces in every app. How do we make the best of both worlds, easily? Can ideas like style transfer be applied to make our apps dramatically more aesthetically pleasing?

Neural network for the digital world

We imagine in the future, there will be a ThingNet, a neural network that understands how all of the digital world operates. How do we get there?

Schedule

The course meets Tuesday and Thursday, from 10:30 AM to 11:50 AM in Gates 100.

This schedule is tentative and subject to change. Please pay attention to emails sent to the student list.

Date Topic
Tue April 3 Overview of the course (slides).
Thu April 5 Intro + brainstorming + demo (slides). Homework 1 released.
Tue April 10 Project discussions
Thu April 12 More ideas + in-class discussions. Project proposal meeting signup.
Tue April 17 Mini hackathon. Homework 1 due.
Thu April 19 Project proposal.
  1. Motif web automation (Allan Jiang, Albert Feng, Jeyla Aranjo)
  2. Construction manager (Haiying Wang)
  3. WeSchedule (Tuan Caraballo)
Tue April 24 Project proposal.
  1. Transforming Almond (Giovanni Campagna, Jestin Ma, Sawyer Birmbaum)
  2. #RFB (Silei Xu, Matt Millican, Michael Araya)
  3. Exhibit: Data-driven UI Generation from NL Commands (Richard Yang, Yuguan Xing)
Thu April 26 Project proposal
Tue May 1 Tutorial: semantic parsing
Thu May 3 Guest lecture: data programming
Tue May 8 Student-led discussion
Thu May 10 Student-led discussion
Tue May 15 Mini hackathon / work session
Thu May 17 Student-led discussion
Tue May 22 Student-led discussion
Thu May 24 Student-led discussion
Tue May 29 Final project discussion
Thu May 31 Final project discussion
Tue June 5 Final project discussion
Mon June 11: (1:30-3:15 pm) Final project demo and poster session (during the scheduled final exam period)
Tue June 12 Final report due

Guidelines

Project Proposal Guidelines

10 minutes + 10 minutes for questions and discussion. Your proposal should include:

  • What is the motivation of the project?
  • What is the state of the art?
  • What will you do?
  • What do you think you will learn at the end?
  • What will be your demo?
  • What will each of you do at each step? (Detailed schedule)

Final Discussion Guidelines

15 minutes + 3 minutes for questions. Your presentation should include:

  • What was the project at the beginning? (Quick refresher for the class)
  • What did you do?
  • What are your results?
  • What did you learn?
  • What would you do if you had more time? (Future work)

Final Report Guidelines

There is no page limit (minimum or maximum), but CS294W students must have substantial writing. Your report should include:

  • Introduction and motivation
  • Design and implementation
  • Results
  • What did you learn
  • Related work
  • Future work
  • Conclusion

Resources

Almond virtual assistant

  1. Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant
    Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam.
    In Proceedings of the 26th World Wide Web Conference,
    Perth, Australia, April 2017.

  2. User-programmable Access Control via Communicating Virtual Assistant
    Giovanni Campagna, Silei Xu, Rakesh Ramesh, Michael Fischer, and Monica S. Lam.
    Under review for Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing - Ubicomp 2018.

Semantic parsing

  1. Compositional Neural Semantic Parsing for Compound Virtual Assistant Commands

  2. Seq2SQL: Generateing Structured Queries from Natural Language using Reinforcement Learning
    Victor Zhong, Caiming Xiong, and Richard Socher
    In arXiv preprint - 2017

  3. Neural Semantic Parsing with Type Constraints for Semi-Structured Tables
    Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner.
    In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing - EMNLP 2017

  4. Data Recombination for Neural Semantic Parsing
    Robin Jia and Percy Liang.
    In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

  5. Language to Logical Form with Neural Attention
    Li Dong and Mirella Lapata.
    In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

  6. SQLNet: Generating Structured Queries From Natural Language without Reinforcement Learning
    Xiaojun Xu, Chang Liu, and Dawn Song.
    In arXiv preprint - 2017

  7. Learning a Neural Semantic Parser from User Feedback
    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, Luke Zettlemoyer.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

Question answering over knowledge base

  1. Learning a Natural Language Interface with Neural Programmer
    Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum, and Dario Amodei
    In Proceedings of the 5th International Conference on Learning Representations - ICLR 2017

  2. Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
    Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, Ni Lao.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

  3. SEMPRE: Semantic Parsing with Execution
    Jonathan Berant, Percy Liang at al. 2013 - 2017

Question answering over free text

  1. Reading Wikipedia to Answer Open-Domain Questions.
    Danqi Chen, Adam Fisch, Jason Weston and Antoine Bordes.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

  2. SQuAD: 100,000+ Questions for Machine Comprehension of Text
    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
    In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing - EMNLP 2016
    [SQuAD leaderboard]

Brassau and style transfer

  1. Brassau: Automatically Generating Graphical User Interfaces for Virtual Assistants
    Michael Fischer, Giovanni Campagna, Silei Xu, and Monica S. Lam.
    Under review for 20th International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI 2018

  2. ImagineNet: Style Transfer from Fine Art to Graphical User Interfaces
    Michael Fischer, Richard R. Yang, and Monica S. Lam.
    Under review for Proceedings of the 31st User Interface Software and Technology Symposium - UIST 2018

Data programming

  1. Snorkel: Fast Training Set Generation for Information Extraction
    Alexander J. Ratner, Stephen H. Bach, Henry R. Ehrenberg, and Chris Ré.
    In Proceedings of the 2017 ACM International Conference on Management of Data - SIGMOD 2017

  2. Fonduer: Knowledge Base Construction from Richly Formatted Data
    Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Chris Ré
    In Proceedings of the 2018 ACM International Conference on Management of Data - SIGMOD 2018

Others

  1. World of Bits: An Open-Domain Platform for Web-Based Agents
    Tim (Tianlin) Shi, Andrej Karpathy, Jim Fan, Jonathan Hernedez, Percy Liang
    In Proceedings of the 34th Internationl Conference on Machine Learning - ICML 2017

  2. Projects from CS294S last year

Teaching Staff

Monica Lam

Professor


Office hours: by appointment

Silei Xu

Teaching Assistant


Office hours: Wednesday and Friday, 2 PM to 4 PM (Gates 407)