CS294S, CS294W: Human-Centered Software: AI Meets Systems.

CS294S is a research project course to explore how machine learning can revolutionize consumers' software experience. Students are invited to do quarter-long projects to answer questions such as those listed below or propose their own topics

NN Icon
Almond Icon
idea Icon

Course Design

A Project Based Course.

CS294S is a cross-disciplinary research project course, designed to help students with their first research experience, emphasizing individual mentorship and learning through projects, rather than lecture-style learning. Students interested in AI, HCI, or programming systems, who have taken at least two computer science courses are welcome to attend.

Students, in groups of 1 to 3, can propose their projects or choose among suggested topics. Available to the students is the open-source Almond virtual assistant infrastructure, which can be used to prototype a natural language interface in just one day.

Students are required to come to campus for this course, where in-class participation and after-class project meetings are required. You can take this course multiple times for credit. CS 294S can be taken to fulfill the CS 194 requirement. Please sign up for CS 294W if you wish to fulfill your writing requirement as well.

Logistics

Grading

  • Class Participation: 15%
  • Homework: 5%
  • Project: 80%

Class Spreadsheet: link

Photo by Rakesh Ramesh

Projects

Sharpei: Your Go-To Shopper by Jeyla Aranjo

An automated service for finding dresses at Macys.com, given keywords and price points.


Dora by Eni Asebiomo, Kaylee Bunner, Meghana Rao

Don't know what to do when visiting a city? Dora can help you. Dora is a travel virtual assistant that plans your day in a new location given a price parameter!


WeSchedule by Tuan Caraballo

WeSchedule is a platform to optimize and automate administrative operations at outpatient clinics. It leverages AI to compute optimal and realistic medical staff schedules and fair distributions of tasks across all team members. We aim to replace schedulers in the healthcare industry as well reduce the work of clinic managers by at least 50%.


Motif: Simple, Codeless, Demonstration-Based Programming for Creating Customized APIs for the Web by Allan Jiang, Albert Feng

Motif Automation enables an end-to-end, codeless system to create APIs for any set of web pages, which can later be used from a variety of interfaces, such as a voice-controlled personal assistant. The system has three stages: 1) recording a demonstration, 2) annotating the demonstration, and 3) executing the workflow as an API. Motif converts a typically arduous and manual task into a single API call, which can be created in a couple minutes via demonstration, rather than a couple hours of scripting and coding.


Almondify - Integrating Spotify into Almond by Hemanth Kini, Gabby Wright

Our project provides best-in-class support for natural language control of Spotify. We enable advanced music playback control, playlist modification, and information retrieval features, and demonstrate that Almond can provide superior control of a third-party service over commercial voice assistants.


Cow: Transforming Almond by Jestin Ma, Sawyer Birmbaum

We investigate the application of Google's "Transformer" model to semantic parsing problems, comparing it with the traditional RNN-based model used to parse Almond commands. Our initial results are promising; the transformer model performs at and, in some case, even slightly above the existing Almond parser while significantly reducing training times.


Friendhub by Matt Millican, Michael Araya, Silei Xu

Do we really need to rely on Facebook or Twitter to stay connected with our local social groups? We think not. We introduce FriendHub, a concept for a series of user-programmable social hubs curated by users to reinforce the real-world communities they belong to.


Locus by Sam Premutico, Matthew Stewart, Oluremi Oso

Locus ingests, analyzes, and reports your Google location history. With Locus, you can interface with your Google location data via natural language to learn about your location history, trends, and habits.


Hiya: Managing Construction Project Workflows by Haiyin Wang

Hiya is a tool enabling construction project managers to handle task workflows between multiple parties and dependency relationships. Its main features include a natural language interface for establishing task dependency based notifications, access control for tasks, and a web application interface to relate tasks to 3D model components of a construction project.


Exhibit: Data-driven Approach for Generating Automatic GUI Layouts by Richard Yang, Yuguan Xing

In the pursuit of automatically generating GUIs for virtual assistants, an important component of this task is to identify the locations of where GUI elements should be. Our project attempts to leverage a labeled dataset of existing GUIs and use machine learning models to learn how to predict the locations of elements.

Schedule

The course meets Tuesday and Thursday, from 10:30 AM to 11:50 AM in Gates 100.

This schedule is tentative and subject to change. Please pay attention to emails sent to the student list.

Date Topic
Tue April 3 Overview of the course (slides).
Thu April 5 Intro + brainstorming + demo (slides). Homework 1 released.
Tue April 10 Project discussions
Thu April 12 More ideas + in-class discussions. Project proposal meeting signup.
Tue April 17 Mini hackathon. Homework 1 due.
Thu April 19 Project proposal. [Discussions]
  1. Motif Web Automation (Allan Jiang, Albert Feng, Jeyla Aranjo)
  2. Construction Manager (Haiying Wang)
  3. WeSchedule (Tuan Caraballo)
Tue April 24 Project proposal. [Discussions]
  1. Transforming Almond (Giovanni Campagna, Jestin Ma, Sawyer Birmbaum)
  2. #RFB (Silei Xu, Matt Millican, Michael Araya)
  3. Exhibit: Data-driven UI Generation from NL Commands (Richard Yang, Yuguan Xing)
Thu April 26 Project proposal. [Discussions]
  1. Almonidify: A personal Assistant for Music (Hemanth Kini, Gabby Wright)
  2. Dora: Exploration Planning with a Virtual Assistant (Eni Asebiomo, Kaylee bunner, Meghana Rao)
  3. Locus (Sam Premutico, Matthew Stewart, Oluremi Oso)
Tue May 1 Guest lecture: Data programming (Alex Ratner)
Thu May 3 Tutorial: Semantic parsing (Giovanni Campagna)
Tue May 8 Student-led discussion: Multimodal user interface
Thu May 10 Student-led discussion: System concepts (crowdsourcing and software architecture)
Tue May 15 Mini hackathon / work session
Thu May 17 Student-led discussion: NLP (semantic parsing and question answering)
Tue May 22 Student-led discussion: Virtual assistant
Thu May 24 Student-led discussion: Virtual reality and augmented reality
Tue May 29 Final project discussion
Thu May 31 Final project discussion
Tue June 5 Final project discussion
Mon June 11: (1:30-3:15 pm) Final project demo (3 minute video) and poster session (during the scheduled final exam period)
Tue June 12 Final report due

Guidelines

Project Proposal Guidelines

10 minutes + 10 minutes for questions and discussion. Your proposal should include:

  • What is the motivation of the project?
  • What is the state of the art?
  • What will you do?
  • What do you think you will learn at the end?
  • What will be your demo?
  • What will each of you do at each step? (Detailed schedule)

Final Discussion Guidelines

15 minutes + 3 minutes for questions. Your presentation should include:

  • What was the project at the beginning? (Quick refresher for the class)
  • What did you do?
  • What are your results?
  • What did you learn?
  • What would you do if you had more time? (Future work)

Final Project Demo Guidelines

A 3-minute video to show off your project!

Final Report Guidelines

There is no page limit (minimum or maximum), but CS294W students must have substantial writing. Your report should include:

  • Introduction and motivation
  • Design and implementation
  • Results
  • What did you learn
  • Related work
  • Future work
  • Conclusion

Resources

Almond virtual assistant

  1. Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant
    Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam.
    In Proceedings of the 26th World Wide Web Conference,
    Perth, Australia, April 2017.

  2. User-programmable Access Control via Communicating Virtual Assistant

Semantic parsing

  1. Compositional Neural Semantic Parsing for Compound Virtual Assistant Commands

  2. Seq2SQL: Generateing Structured Queries from Natural Language using Reinforcement Learning
    Victor Zhong, Caiming Xiong, and Richard Socher
    In arXiv preprint - 2017

  3. Neural Semantic Parsing with Type Constraints for Semi-Structured Tables
    Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner.
    In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing - EMNLP 2017

  4. Data Recombination for Neural Semantic Parsing
    Robin Jia and Percy Liang.
    In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

  5. Language to Logical Form with Neural Attention
    Li Dong and Mirella Lapata.
    In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics - ACL 2016

  6. SQLNet: Generating Structured Queries From Natural Language without Reinforcement Learning. [Github]
    Xiaojun Xu, Chang Liu, and Dawn Song.
    In arXiv preprint - 2017

  7. Learning a Neural Semantic Parser from User Feedback. [Github]
    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, Luke Zettlemoyer.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

Question answering over knowledge base

  1. Learning a Natural Language Interface with Neural Programmer
    Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum, and Dario Amodei
    In Proceedings of the 5th International Conference on Learning Representations - ICLR 2017

  2. Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
    Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, Ni Lao.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

  3. SEMPRE: Semantic Parsing with Execution
    Jonathan Berant, Percy Liang at al. 2013 - 2017

Question answering over free text

  1. Reading Wikipedia to Answer Open-Domain Questions. [Github]
    Danqi Chen, Adam Fisch, Jason Weston and Antoine Bordes.
    In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics - ACL 2017

  2. SQuAD: 100,000+ Questions for Machine Comprehension of Text
    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
    In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing - EMNLP 2016
    [SQuAD leaderboard]

Brassau and style transfer

  1. Brassau: Automatically Generating Graphical User Interfaces for Virtual Assistants

  2. ImagineNet: Style Transfer from Fine Art to Graphical User Interfaces

Data programming

  1. Snorkel: Fast Training Set Generation for Information Extraction
    Alexander J. Ratner, Stephen H. Bach, Henry R. Ehrenberg, and Chris Ré.
    In Proceedings of the 2017 ACM International Conference on Management of Data - SIGMOD 2017

  2. Fonduer: Knowledge Base Construction from Richly Formatted Data
    Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Chris Ré
    In Proceedings of the 2018 ACM International Conference on Management of Data - SIGMOD 2018

Others

  1. World of Bits: An Open-Domain Platform for Web-Based Agents
    Tim (Tianlin) Shi, Andrej Karpathy, Jim Fan, Jonathan Hernedez, Percy Liang
    In Proceedings of the 34th Internationl Conference on Machine Learning - ICML 2017

  2. Projects from CS294S last year

Teaching Staff

Monica Lam

Professor


Office hours: by appointment

Silei Xu

Teaching Assistant


Office hours: Wednesday and Friday, 2 PM to 4 PM (Gates 407)