Project

Overview

In this course, you will do research on a project within a particular field of computer science. At a high level, successful research this quarter should propose a research question, then plan and execute a methodology for answering that question.

For information on how the project will be evaluated, see the individual rubrics under each assignment and the logistics page.

Click here for the full list of projects.

Timeline

Executing this project will be your team's focus for the rest of the course. Here is a rough sense of the timeline from here on out:

  • Week 3: Related Work due
  • Week 4: Introduction due; start executing the project
  • Weeks 5-8: Work on the project, with weekly check-in assignments
  • Week 8: Evaluation plan due in addition to the weekly check-in assignment
  • Week 9: Draft paper due
  • Week 10: Final presentation in class, Final paper due

Set up Overleaf

Many CS research folks use Overleaf (shared editing à la Google Docs for LaTeX) because it is a useful platform for writing papers collaboratively. Stanford has an institutional subscription to Overleaf. Here's a tutorial to the LaTeX typesetting language, and here's a tutorial to using BibTeX with LaTeX to handle format citations via the \cite{} command. We strongly suggest that you use Overleaf to write up all written assignments, as it will be way easier to copy over your work into your final paper (which will require Overleaf).

We provide an assignment template for each assignment that you complete on Overleaf.

  • To import templates into a new Overleaf project, download the template zip from the CS197 webpage (do not open/extract these files). Then, on the Overleaf website (register with your stanford.edu account), go to "New Project" -> "Upload Project" and upload your downloaded zip file. Overleaf will then automatically extract this zip and create a new Overleaf project.
  • To export a project from Overleaf, click the Download icon next to the Recompile button while editing a project. This generates and downloads a PDF that you can view on your local computer and upload to Canvas.
Assignment 2. Due Wednesday, October 6 at 2:00pm.

Doing research means making a novel contribution. Novelty, as we discussed in class, is a litmus test for a research idea: a novel idea is new to the academic literature. To be able to argue for novelty, you need to know what ideas exist in the space. Then, you can articulate a bit flip relative to that prior work to motivate your own project.

This assignment takes you paper-reading skills from Assignment 1 and puts them to good use. You'll be doing targeted reading of a number of papers in order to synthesize the state of the literature and articulate your bit flip. The result will be a Related Work section that you can eventually adapt for your final paper.

Proposed Timeline

This is your first group assignment. You will work on it with your project team, focused around the project you've selected. However, reading papers takes time and energy. Early on, we suggest you get logistics out of the way: complete Overleaf and set up your affinity mapping board from Part 3, such that each group member knows which topic they should explore in more depth. Then, individually complete Part 1 (reading nearest neighbor paper) and Part 2 (explore your topic, put the citations into the Overleaf document). Finally, come together for Part 3's affinity mapping and finish up the Related Work section.

Download Overleaf template: Assignment 2: Related Work

Part 1: Read your nearest neighbor paper

Read the nearest neighbor paper (in-depth). Your TA has provided your team with a nearest neighbor paper for your project. We refer to it as a nearest neighbor because it's the most adjacent piece of literature to the idea that you are pursuing. It's important that you understand it well. Read the nearest neighbor paper, and be able to state in your own words how your project is expanding on, or differing from, the nearest neighbor paper — and why that difference is an important one for researchers to pursue. It might be helpful to do a brief outline for the paper like you did in Assignment 1.

  • AI project list here.
  • HCI project list here.
  • CompBio project list here.

Skim the nearest neighbor's related work. Next, pull out the five most relevant pieces of related work cited in that nearest neighbor paper. This can be a mix of foundational work that is less adjacent to your project but sets the stage, and incremental work that is highly adjacent to the work you want to do. Skim each of these papers. You’re not necessarily reading them for every detail; you’re trying to understand how they’re situated relative to the work you want to do. Based on the outline structure for reading papers as in Assignment 1, if you can understand the Problem, the Assumption, and the Insight, you probably have it covered.

From here, follow citations in the papers you found, or in the nearest neighbor paper, to build out the local neighborhood.
Your TA has also provided 1-2 additional papers that provide context or insight into how you may start or understand your project.

Use strategies that we discussed in class:

  • Tracking the influential citations in the papers that you're reading
  • Using Google Scholar's "Cited By" to find more recent papers that cited the one you're looking at
  • Using Google Scholar's "Related articles" to find papers on similar topics, or searching for key keywords and phrases you've uncovered from the papers
  • Using Semantic Scholar's "Highly Influenced" classification to find papers that draw heavily on the one you're looking at
  • Skimming to understand each paper's high-level takeaways from the abstract, introduction, and related work sections.

To make your later writing easier, you may want to collect the BibTeX metadata for these papers into a shared Mendeley/Zotero library, or into a shared .bib file in Overleaf. You can typically find the BibTeX through a couple clicks on Google Scholar and on other sites that host papers.

You should get to the point that adding an additional paper is only telling you incrementally more information than you’ve already roughly assimilated. Most likely, you'll start hitting that asymptote after about fifteen papers: often, after this, the ideas will either be more of the same, or will start becoming more distant from your project.

You have a team, so you can split up some of the work here. You may want to split up papers by topic, rather than randomly, so each person is responsible for an area of related work. However, for the next phase, it is critical that you all have shared what you learned with each other, so that you share a collective understanding of the literature and how your project builds on it. You may want to meet up with your team and report back summaries of each paper you read.

At this point, you have all the raw materials that you need to write a Related Work section of a paper. In addition to being a useful exercise at this point in the project, you will be able to keep what you write for this assignment and adapt it for use in your final paper.

Perform an Affinity Mapping

As a first step, perform an affinity mapping of the papers you've collected.

  • With your group, grab some large post-it notes, write each paper on a post-it note, then start placing the post-it notes on a wall or whiteboard so that related papers are near each other.
  • Once everything is up on the wall, take a second pass to re-organize, combine and split groups, looking for shared assumptions or approaches, until you have made sense of the related work.
  • Note that you can also use Google Jamboard to create digital post-its.

Next, use this affinity mapping to organize your Related Work section.

  • Each cluster, typically covering 5 ± 2 papers, should become one paragraph.
  • In each paragraph, your goal is to summarize what the literature knows about the topic.
  • Order the paragraphs so that they build on each other, typically starting with the high-level motivation of the area, then exploring different approaches or assumptions that have been taken with one paragraph each. You may use one set of subheadings if it's helpful.

Finally, at the end of the Related Work section (or at the end of each paragraph), summarize your bit flip. This means stepping back and asking: what assumption did all of the literature in this section make, and how is our research going to flip the bit flip behind that assumption? You should lay out the bit that's being flipped as a summary of the work in this section, then articulate the flip itself.

For example, consider the below summary paragraph at the end of a Related Work section (sentence numbers in parenthesis):

(1) In summary, researchers have pushed for formal validation as a solution to runtime errors in foobars. (2) This error checking typically involves a formal specification of expected behavior, and the use of a computational logic prover to establish the validity of the asserted behavior. (3) Instead of formal error checking proofs, which are often intractable for non-trivial programs, this research demonstrates that running multiple independent foobars in parallel, and then comparing their outputs, allows a system to catch errors with very high probability and far less programmer effort. (4) While formal proofs are still desirable for mission-critical systems, we argue that this approach scales to far more complex software where formal proofs are not currently feasible.

Sentences (1) and (2) set up the bit being flipped: error checking via formal proof. Then sentences (3) and (4) flip the bit: instead of using formal proof, we run it multiple times and look for empirical disagreement.

If you'd like another example, check out the Related Work section in this paper. It spans pages 2 and 3. Notice that the first few paragraphs of each subsection survey the related work in a narrative way, and the last paragraph articulates the bit flip. Other genres of bit flip argument include: "everyone else's problem setup was slightly different, and our problem setup requires changing the solution entirely", and "similar solutions have been applied in other areas, but not to ours, and require an adaptation to solve our problem".

Writing Heuristics

Apply the following writing heuristics when writing your Related Work section:

  • The citations should be annotations, not proper nouns, in the sentence. In other words, you should be able to remove the citation from the sentence and it should still be readable. Avoid "[5] demonstrated that foobars can be optimized"; instead, "Foobars can be optimized [5]." In addition, it's often counternorm in computer science to name authors, except in CS theory papers. Instead of "Bernstein et al. demonstrated that foobars can be optimized", again focus on reporting the main claim and substantiate it with the citation instead: "Foobars can be optimized [5]." This is not the case for CompBio -- mentioning authors by name in the sentence is typical.
  • Avoid "various" phrases like "There are various approaches to optimizing foobars [1,2,3,4]." This is an empty calorie sentence that doesn't tell the reader anything. Instead, be slightly more detailed in your summary: "Optimizing foobars requires eating large volumes of tasty instant ramen [1, 3] or pulling all nighters [2, 4]."
  • Your goal is not to bash the related work. It's to summarize (and even celebrate) what's been done, and then articulate how your work extends it. Keep in mind that, when you submit a paper for peer review, the people you're citing are likely going to be reviewing your paper! So keep the conversation productive.

Your group's Related Work section should be at least 600 words, targeting 600-700 words and at least 15 citations.

Submission and Grading

Export a PDF of your group's Overleaf Related Work project and submit via Canvas. This is a group assignment; create a group for your team, and one member should submit on behalf of the group. Your related work will be graded on the following rubric:

  • Completeness: does the related work cover all the most relevant ideas in the literature?
  • Synthesis: does the related work combine the work and describe it in a narrative, rather than a paragraph-long list of each paper?
  • Bit flip: does the bit flip successfully articulate the assumption in the related work, and this project's bit flip from that assumption?
  • Clarity: is the writing overall clear and easy to follow for a technical expert in the field?

Introduction

Assignment 3. Due Wednesday, October 13 at 2:00pm.

Now that you understand the literature and the bit flip that you're making, it's time to write up a project proposal in the form of an Introduction section for a paper. As researchers, we often write up a paper introduction like this as we embark on the project as a check that we can explain a clear vision of the project. If we can't explain the project clearly in writing, then we don't understand the project and its contribution well enough to move forward!

The result of this assignment will be an Introduction section that you can eventually adapt for your final project. Think of this Introduction as also serving the role of a project plan for this quarter and next.

Download Overleaf template: Assignment 3: Introduction

Part 1: Outline your introduction

While there are many approaches to write an introduction, in Computer Science most of these approaches need to explain two things: What is the problem, and How are you solving it?

Michael Bernstein, an HCI faculty member in our department, tends to follow the following paragraph-by-paragraph outline for an Introduction (many of Lisa's papers follow a similar format):

  • Problem: What's the problem you're solving? Cite other work that helps motivate the problem. Make the reader feel the urgency of the problem, so that by the end of the paragraph they care a lot that someone finds a solution. Example topic sentence:

    We all want to build IoT applications that are responsive to our behaviors — for example, coffee machines that make coffee for us after we wake up, or phones that know not to ring while we're in meetings — but we lack the data to endow our algorithms with this information about our patterns and behaviors.

  • Set up the bit: This paragraph usually answers the question, "Why isn't this problem solved yet?" by setting up the bit that you're going to flip. The topic sentence typically summarizes existing approaches to solving the problem by articulating the bit that they share, and points out why those approaches and the bit more broadly hasn't been able to solve the problem. So, it's a brief summary of the most important related work — but it's a summary, not a survey, meaning that its goal is to set up the bit, not to cover everything. Example topic sentence:

    Existing solutions to this problem require manually authored rules (e.g., coffee is a morning drink, we drink morning drinks after we wake up), but the required manual effort results in incomplete data about human situations and behaviors.

  • Flip the bit: Here you introduce your insight by flipping the bit. The topic sentence of this paragraph is the thesis statement for the entire paper. It should lay out the big idea that you'll be pursuing. Spend this paragraph explaining your insight clearly, why it flips the bit that you set up, and what the implications of flipping that bit will be for the problem at hand. For the purposes of this assignment, it's fine to assume that your bit flip will work out — we all know that research is iterative and projects will evolve. Example topic sentence:

    In this paper, we suggest that data mining a large dataset of modern fiction offers a promising alternative to manual annotation.

  • Instantiate that bit flip in a solution: At this point, the reader understands the idea that you're proposing, but it's still very high level. In this paragraph, map that idea onto a concrete instantiation. This is where you introduce the system you've built, the model you're creating, the study design, or whatever is the main artifact of your research, and how it instantiates that insight. Example topic sentence:

    We introduce Augur, a knowledge base that relates human activities and the objects around them by mining 1.8 billion words of modern fiction from the online writing community Wattpad into a vector space model for predicting human activities.

  • Evaluate the solution: We haven't discussed evaluation yet in the class, but a good introduction summarizes the evaluation that you performed to demonstrate that flipping the bit has the effects that you suspect. For the purposes of this assignment, ask yourself: how would you prove to a critical reader that flipping the bit has the effect you promised in solving the problem? We expect you to articulate what you will be manipulating and measuring, but we don't require any actual results right now and we won't hold you to this evaluation strategy — especially since sometimes your evaluation strategy evolves as your project evolves.

    Example topic sentence: To evaluate Augur's ability to predict human behaviors, we first compared Augur's predictions to human estimates of behaviors given the same inputs, and second performed a field deployment to test Augur's precision and recall in a realistic application.

  • Implications: If you're right about this bit flip being the right way for the field to go, what implications will there be for the field? Don't overplay your hand here by stating that all of computing will shift, but do be clear and forceful in pushing the case as hard as you feel that the case warrants.

    Example topic sentence: This work demonstrates that fiction can provide deep insight into the minutiae of human behavior, and that modern NLP algorithms can structure those minutiae into useful hooks for IoT applications.

This outline will depend a bit on the kind of paper you're writing. For example, start by answering: are you an old problem / new solution paper, or a new problem / old solution paper? Old problem / new solution papers already have a warrant for the problem, so that part of the Introduction is easy, but they need to argue for the solution very carefully since there's no warrant there. Similarly, new problem / old solution papers need to be extra careful to argue the problem persuasively, but they can draw on existing warrants for the solution and only argue for how they're adapting it to the problem.

Start by creating the outline for your Introduction, using only the topic sentences that you will eventually expand into full paragraphs. This is harder than it sounds! It means that you only have those six sentences to tell the entire story of your paper. So, you can't add any unnecessary detail or extra concepts, or those six sentences won't be possible for a reader to understand. Just like a pro swimmer makes no unnecessary motions as they focus all energy on executing a stroke, and just like a pro chess player makes no unnecessary moves as they focus their actions on enacting their strategy, your topic-sentence outline should introduce no unnecessary concepts or ideas as it focuses on explaining the project.

You may want to get feedback from your TA on your outline before moving forward. We recommend office hours. If you find that you need to stray from the outline above for your project, definitely reach out to the TA first with your proposed deviation.

Part 2: From Outline to Introduction

Now, to write the Introduction section itself! Each topic sentence becomes the anchor of a paragraph in your Introduction, and the rest of each paragraph should set out to prove the point made by that topic sentence. If you need to split a paragraph into multiple paragraphs to be clearer, do so — but keep the rough proportions of those original six paragraphs. (In other words, don't spend three paragraphs on the problem, then rush through everything else.)

Don't forget to cite any work you're referencing that motivates your problem, sets up the bit, or helps explain why the flip is a good idea.

Here are example introduction sections for each section, selected by that section's TA. Bear in mind that these papers will stray from the outline format above—which is fine, experienced writers play with the format—but we still feel that they represent strong examples of Introduction sections.

Your Introduction should be 700-900 words (for a 3-4 person group) or 600-700 words (for a 2-person group). Include a References section with citations at the bottom; references are not included in the word count.

Submission and Grading

Submit a PDF with (1) your outline, and (2) your Introduction on Canvas. This is a group assignment; create a group for your team, and one member should submit on behalf of the group. Your introduction will be graded on the following rubric:

  • Outline: does your outline read as a clear and complete explanation of the project?
  • Problem and bit: does the set up of the project make sense and prepare the reader for the bit that will later be flipped?
  • Bit flip and description: does the bit flip follow through on the bit that was set up, and does the system description make clear how that bit flip gets instantiated in a system?
  • Clarity: is the writing overall clear and easy to follow for a technical expert in the field?

Progress Reports

Due Wednesday at 2:00PM for Assignments 4 (October 20), 5 (October 27), 6 (November 3), and part of 7 (November 10). Assignment 7 also includes Evaluation section.

At this point, your goal shifts to making progress on your project. Each week for the next four weeks, we will ask you to share an update on your team's progress during the week, and plans for the next week. Your TA will read these reports and give you feedback during section.

Download Overleaf template: Assignments 4-7: Progress Report

Part 1: Vectoring

Start the week by meeting with your team to perform vectoring. What is currently the riskiest unanswered question in your project? What is core to answering that question, and what is periphery?

Part 2: Plan

Your answer to the vectoring question should make clear what you need to focus on. Identify a goal that your team thinks is achievable in one week to reduce the risk on the dimension of your selected vector.

Execute that plan this week.

Part 3: Report

Meet with your group to perform vectoring for the next week. How far did you get, benchmarked against the goal you set? What is your new vector (in other words, what is now the riskiest unanswered question in your project)?

Submit a PDF of your progress report on Canvas. It should have the following sections, with a short paragraph for each section, totaling about one page:

  • This week's vector: what is the vector, and why is that the riskiest unanswered question right now? (In future weeks, it will be fine to copy-paste this section from the "next week's vector" section of last week's progress report.)
  • This week's plan: What is core and periphery, given that vector? What was your concrete goal to achieve by the end of the week to reduce uncertainty in that vector? (In future weeks, it will be fine to copy-paste this section from the "next week's plan" section of last week's progress report.)
  • This week's result: What happened? Did this reduce uncertainty in the vector you selected?
  • Next week's vector: what is the vector, and why is that the riskiest unanswered question given what you learned this week?
  • Next week's plan: What is core and periphery, given that vector? What is your concrete goal to achieve by the end of the week to reduce uncertainty in that vector?
  • One sentence per person: what that individual did this week

At the beginning of your section each week, you will give a very short 2-3 minute verbal update on your project to your CA and the other teams. This "stand-up" is not graded and is not intended to compare your progress against other teams; rather, it's a chance for you to practice speaking about your project succinctly and to hear what other groups are doing. You should briefly address (think 5 sentences total):

  • This week: Vector (what was the riskiest unanswered question), Plan (concrete goal to answer that vector), Result (what happened)
  • Next week: Vector (what is the riskiest unanswered question right now), Plan (concrete goal to answer that vector)

Submission and Grading

Submit a PDF with your weekly report. This is a group assignment; create a group for your team, and one member should submit on behalf of the group. Note that for Assignment 7, you should also submit a a PDF for Evaluation (more below).

Your progress report will be graded on the following rubric:

  • Vector: Did you select a vector that captures the main source of uncertainty in your project right now?
  • Plan: Did you create a plan that should reduce uncertainty in that vector? Is that plan achievable in a week?
  • Velocity: Did you make reasonable progress on your plan? We assume that plans may need to evolve as the week goes and you may need to re-vector midweek. We're not grading on whether you stuck exactly to the plan, but on whether you maintained high velocity on your project.

Evaluation Plan

Part of Assignment 7. Due Wednesday, November 10 at 2:00pm.

This assignment will center around what your plan is for the rest of the quarter. You should submit it in addition to a Weekly Progress Report.

To convince people that your idea is correct, you'll need some way to convince an expert that you have evaluated it fairly and correctly. In this assignment, you will develop an evaluation plan for your research project, and write it up.

Proposed Timeline

Some projects, which are more study- or measurement-oriented, need more lead time to complete their evaluation. If you are in this set, turn this assignment in early so that you can proceed with data collection. Other projects, which are more engineering- or design- oriented, need more lead time to design and construct their approach. If you are in this set, you can still turn the assignment in early, but it's not as strongly required.

We recommend getting Parts 1, 2, and 3 out of the way as early as possible. Once you are settled on your thesis, claim, and evaluation design, you can split up Part 4's writing work.

Download Overleaf template: Assignment 7: Evaluation Plan.

This template also includes space to copy in your Introduction and Related Work from prior assignments. Please copy them in so that your CA can give detailed feedback in preparation for your Draft Proposal assignment next week. While you are not required to incorporate feedback from past assignments at this point, you are strongly encouraged to so your TA can give you feedback on the most recent state of your paper. Include the correct references in the Bibliography.

Invite your CA as a collaborator (can edit) on this Overleaf project.

Part 1: Articulate Your Thesis

As we discussed in class, the first step in planning an evaluation is to articulate the main thesis of your work. (Remember from Assignment 3, Project Introduction, that the main thesis of the work is likely embedded in the topic sentence of your bit flip paragraph.) Go back and reflect on that statement — tweak it if necessary based on what you've learned from your project so far.

Write out your thesis at the top of your submission.

Part 2: Derive Your Claim

We discussed in lecture how theses imply a claim. For example, "x > y"-type ("X is better than Y") theses imply a claim that x is in fact better/faster/more performant/more enjoyable than y, and "∃ x"-type ("there exists an X") theses imply a claim that whereas x could not exist before, that it does with your system. Discuss with your team the claim implied by your thesis.

Write down your claim.

Part 3: Design Your Evaluation

Now, you need to work from your claim to design a specific evaluation plan. How do you prove what you have claimed? This evaluation plan typically specifies:

  • Dependent Variable (DV): what is your dependent variable? (This is the variable you measure as the outcome, such as accuracy on a test set.)
  • Independent Variable (IV): what is your independent variable? (This is the variable you manipulate for comparison to create conditions, such as the algorithm or the interface used.)
  • Task: what is the specific task that is being performed in order to measure the DV? (This might include executing a benchmark, a known ML classification task, or a specific sequence of behaviors that a user must perform.)
  • Threats: what are the factors that might influence your outcome? For example, in what situations might your result hold or not hold? What biases might creep in that you need to make sure to account for?

You don't need to re-invent the wheel here. Often your nearest neighbor paper or other papers in your related work establish an evaluation paradigm that you can import to your paper. In fact, this is often preferred, since then you don't need to convince a reader that your approach is valid, since it's already in the literature. So, go review the evaluations used in your prior work and use those to develop a few possible models. Then, share those models with your team and work together to develop a variant that works well for your project.

Based on your project's setup, your model might look slightly different than what is laid out above. If you believe this to be the case, talk to your TA about it.

Next, run the following unit test on your proposed design: does it directly test the thesis you articulated above? Imagine a few possible outcomes from your evaluation. Depending on how it comes out, does it directly prove or disprove your thesis, or only obliquely shed light on whether your thesis is correct?

Write out the DV, IV, Task and Threats for your evaluation design. Summarize your explanation of why that design directly tests your thesis.

Part 4: Write Your Evaluation

Your goal is now to write up your evaluation plan like it would appear in a published paper. Ideally, you will be able to reuse and update this for your final paper submission. Having a clear sense of what the evaluation will look like helps make sure that you are targeting your vectoring toward the goals that you need to.

Different areas structure their evaluation writeups differently. For example, in HCI, it is common to have a Method section separate from a Results section; in Systems, less so. Use your related work to develop a model of how evaluations should be reported in your area. Each section has also provided a sample strong evaluation section for you to use as a rough template below:

You obviously won't have final results ready at this point, so either make up the results you might reasonably expect to see or use any current pilot data that you do have. Include any graphs that you are going to want to include in your final writeup. You will, of course, be able to update this for your final submission as your project progresses, including changing your evaluation design (with clearance from your TA) and updating your results.

In a full paper, typically the Evaluation section is 2000-3000 words. However, that length includes a detailed analysis of the results, and at this point, you will only have pilot data at best. So, for this assignment, the requirement is 1000-1500 words (for a 3-4 person group) or 800-1200 words (for a 2-person group), mostly on methods and a bit of results scaffolding. We expect that your final Evaluation section (for the final paper) will be twice as long.

Submission and Grading

Submit two PDFs:

  • An evaluation PDF with (1) your thesis, (2) your claim, (3) your evaluation design, and (4) your evaluation writeup (also include copies of your Introduction and Related Work section), and
  • A progress report PDF.
  • You should also invite your CA as a collaborator (can edit) on your Evaluation Plan Overleaf document which includes your Introduction, Related Work, Evaluation Plan, and Evaluation sections.

Your progress report will be graded according to the usual rubric. Your evaluation will be graded on the following rubric:

  • Claim: is the thesis a correct articulation of the project, and does your claim derive from the thesis?
  • Evaluation design: does the design of the evaluation correctly evaluate the thesis and follow through on the claim?
  • Description: does the writeup clearly and correctly describe the design to an expert in the area?

Draft Paper

Assignment 8. Due Wednesday, November 17 at 2:00pm.

It's time to start putting everything together into the paper. This is an exciting moment — this is when all the work you've been doing starts to take shape into a paper that you can submit for publication. This won't be a complete draft, but it will put together the bones of the paper and incorporate text for a substantial proportion of the paper. You can continue to iterate on the final results. However, a common failure point for researchers is not to save enough time to write a convincing paper before the deadline, which is why we're starting now.

In section, your draft paper will be passed to other students for a mock peer review, and you'll do a mock peer review on their paper.

Download Overleaf template: Budget plenty of time for getting your paper to "work" in the right academic LaTeX template, which differs by field. Here's the template for each section:

Part 1: Find a Model Paper

Following the pathway laid out in lecture, first identify a strong model paper to use as you are writing. Keep in mind that a model paper should be making the same genre of argument as your paper; it doesn't necessarily need to be tightly related to the actual topic of the research contribution. There likely exists a good model paper somewhere in your related work.

You will be using this model paper as a template when you outline your own paper, so choose carefully. We strongly recommend checking with your TA to make sure that it's a good fit.

Part 2: Make Your Paper Outline

Now, outline a template for your own paper by considering what parts of the model paper to adapt and what parts to change out. In a bulleted list, outline your paper and make specific notes about where you may deviate from the model paper on a meta-level. Here is a list to consider: e + [Sections and subsections] How does your paper structure itself? How long is each section, in words or pages? What is the main expository goal of each section and sub-section? What are the headings and sub-theses for each section and subsection?

  • [Sections and subsections] Briefly describe any adaptations or changes: What sections/subsections will be longer in your paper than in the model paper? What sections will be shorter? What sections/subsections were added/removed in your paper compared to the model paper? Why did you make these decisions?

  • [Figures, tables, graphs] For each figure, table, and graph you will have in your paper, what role does it play? Why is it included in that section/sub-section?If there exists one, what will be the most important figure in your paper and why?

  • [Figures, tables, and graphs] Briefly describe any adaptations or changes: Which figures did you leave out of your paper that were in the model paper, and which have you included? Why did you make these decisions?

  • You are welcome to drop some elements of the model paper, add some elements that weren't present in their paper but you feel should be present in yours, or tweak lengths in your paper. The model paper is a template, not a straightjacket. However, if you're straying too far from the model paper template (more than 20% or so), it suggests that perhaps the model paper is not a good fit for your research, and you need to find a different one. Consult your TA if this is the case.

  • Overall, you should analyze the model paper at a meta-level: what does the paper do (e.g., paper structure, figure layout, exposition organization) that you like, and what changes will you have to make to write your own?

You should have about 1-2 pages for this part, but don't go overboard. It is more important that you spend your time this week writing a good draft paper, but we'd still like you to explicitly do this meta-analysis so that you have something to reference when writing.

Here's an example of a paper outline that outlines each component of the Mechanical Novel paper on crowdsourcing short story writing. Our goal with the below is to show you to what level of detail you should describe your paper's structure. You should also include any changes that you made from the model paper (we did not do this part).

  • Introduction (1 page): Argues for the bit flip: prior work does a one-way decomposition of work into constituent tasks, but this doesn't work for interrelated goals such as story writing, so they introduce a reflection-revision crowdsourcing technique that more closely mimics how writers actually write.
  • Figure 1 (in Introduction). Provides a high-level overview of the main workflow being proposed by the paper.
  • Related Work [1pg]: covers other papers on crowdsourcing complex work, and on how authors write stories. Two parts: Collaborating through context-free tasks, and crowdsourcing with global goals.
  • Mechanical Novel ("Section 3") [1.5pg]: describes the high-level (1st subsection) and low-level (next 3 subsections) iterative loop that makes the system work:
    • Designing workflows based on expert practice: summarizes the expert practice that they are drawing inspiration from in creating their system
    • Initialization — creating a first draft: describes the general set up of the system
    • Reflect — choosing a high-level goal: describes the first major part of the system, how the contributors identify a high level change they want to make
    • Revise — translate goals into actionable tasks: describes the second major part of the system, how the contributors split that high level change into concrete low level contributions to execute
    • Figures 2, 3, and 4 show screenshots of the reflection, decomposition, and voting interfaces respectively.
  • Evaluation [4.5pg]: tests whether the approach succeeds at writing.
    • Two figures show an example story (Figure 5) and visualize the control system vs the Mechanical Novel system (Figure 6).
    • Benchmark study subsection: Tested how well the process fixed stories with known problems in them. Subsubsections capture specific avenues of investigation, for example the kinds of changes that each system made.
      • Table 1. Summarizes the inputs to the evaluation
      • Table 2. Results: how often did the system identify the right part to fix?
      • Table 3. Results: how often did the system accurately make the right fix?
    • Story writing study: Tested how well the process could write a story from scratch, including quoted examples.
      • Table 4. Summarizes the inputs to the second evaluation
      • Table 5. Raw results of the readers' evaluation of the stories written by the crowd
      • Table 6. Report of the kinds of changes that contributors made as they revised the stories
  • Discussion [1pg]: articulates why the approach worked in practice, and how to expand it
    • Enabling flexibility and encouraging diversity: unpacked the reasons why Mechanical Novel did better than the control — what about the idea was right, and what mattered less in practice?
    • Going beyond short stories: a future work discussion of what would need to be changed about the system in order to write longer stories
    • Designing collaboration around reflection and revision: what are the possible generalizations of the reflection-revision technique introduced in this paper?
  • Conclusion [.25pg]: summarizes the main contributions again, as well as the results of the study. Closes with a "zoom out" to the broader vision.

Part 3: Write your draft paper

Now, it's time to flesh out your outline into a full draft paper. You are welcome to use content that you generated in previous assignments, e.g., the introduction, related work sections, and evaluation method. You may need to tweak your previous writing as your project has evolved.

At this stage, your draft paper can leave the Discussion and Conclusion sections in outline form, and Evaluation/Results can be incomplete or just pilot data. Most likely, you'll be focusing most of your effort on writing out the details of your approach, and reporting any results you have so far. Include a References section with citations at the bottom. There is no specific word count for this assignment, but keep in mind the final paper should be about 5-6 pages in length.

Your draft paper will be traded with another group for peer review in section after the deadline. We, the staff, will also send feedback on the draft.

Submission and Grading

Submit a PDF with (1) your draft paper, (2) your model paper title and link, and (3) your outline on Canvas by Wednesday, November 17 for peer review.

This is a group assignment; one member should submit on behalf of the group.

Your submission will be used in section for peer review and as the basis for your Completeness grade.

Your draft paper will be graded on the following rubric:

  • Model paper and template: did you identify an effective model paper, and analyze its template?
  • Your paper outline: did you effectively translate the model paper template into an outline for your paper?
  • Completeness: does your draft paper have reasonable content coverage for each of the sections of the outline?
  • Clarity: is the writing overall clear and easy to follow for a technical expert in the field?
  • Peer review: is your peer review on your peers' draft (in section after the deadline) thoughtful and helpful?

Final Presentation

Assignment 9. Due Wednesday, December 1 at 2:45pm (in-class).

Tell us what you've achieved this quarter! Talks and presentations are one of the main mechanisms through which researchers communicate their ideas. As part of your final project, you'll be giving a talk on your research.

Presentation Format

You have 5 minutes (+ about 1 minute for questions) to present your research to the rest of the class, the staff, and the research mentors. To ensure a smooth flow between projects, we will be running everything off a shared Google Slides deck. So, create your slides using Google Slides, and insert them after the title card for your project team. We will run the presentation in presenter mode, so you'll be able to see any presenter notes you leave.

The timing means you won't be able to cover every detail of your project. Think about what you really want to communicate. It likely draws heavily from your introduction and bit flip:

  • What's the problem, briefly, and why does it matter?
  • Why hasn't prior work been able to address the problem? Set up the bit.
  • What's your big idea? Flip the bit.
  • Explain enough technical detail for the listener to understand what you did, at a high level.
  • Explain enough about your results for the listener to understand what you observed, at a high level.
  • Conclude with a restatement of your thesis.

Assume that the audience for your talk is a researcher in your broad area (e.g., HCI, CompBio, or AI). So, you don't need to re-introduce the whole field of HCI, for example. A sentence or two to situate your work in the field is good, but spend the rest of the time telling us what you did.

This presentation is worth a substantial amount of your grade, so it's worth practicing it. At 5 minutes long, you and your team can practice it between 5 and 10 times in an hour! It's also short enough that you might consider writing out what you want to say long-hand, make sure you convey information efficiently and effectively. Practice it enough times so that you have it basically memorized, but not so memorized that you get flustered if you skip a word or someone asks a question.

Submission and Grading

Submit your slides to the Google Slides deck before class starts on December 1. In addition, submit a PDF static copy of your slides on Canvas by Wednesday, December 1. This is a group assignment; one member should submit on behalf of the group.

Your presentation will be graded on the following rubric:

  • Structure: did the presentation organize its question, thesis, and evidence effectively?
  • Clarity: did the presentation convey the details in a way that a general expert in the field could follow?
  • Timeliness: did the presentation organize its time well, and stick to the 5-minute limit?

Final Paper

Assignment 10. Due Tuesday, December 7 at 11:59pm PT.

This is it! Finish and submit the research paper that you've been working on this quarter. You have all the tools; now it's about finishing what you started.

As a reminder, when you're done, you can work with your TA to submit your research to a non-archival track in a top conference in your field, and get a conference grant from VPUE to support up to $1,500 to travel to the conference and present it if it gets accepted.

Download Overleaf template: Build on your Draft Paper project.

Part 1: Integrate new results and feedback

You've likely made a lot of progress since your draft paper. Go ahead and flesh out your Results, Discussion, and Conclusion sections. Use the outline for these sections that you drafted off your model paper, or revise that outline as needed. As a reminder:

  • Results: conveys the outcome of your evaluation, and interprets the results so the reader can understand what they should take away from the outcome.
  • Discussion: generally asks the "why?" questions. Explain: why did the results come out the way they did? What ended up mattering, and what didn't? What unforeseen outcomes arose, and why? In addition, lay out limitations: what can't your approach solve, and what questions can't your study answer? What does all this mean for future work in this space?
  • Conclusion: reiterate the problem and your idea, but focus more on the implications. Tie it up with a bow, and ride off into the sunset.

You've also received peer review feedback as well as feedback from your TA. Integrate these into your final submission. This feedback will likely involve minor or major revisions to your introduction, related work, approach, or other sections.

Include a brief changelog (no more than a few paragraphs) that details the major changes you made between your draft submission and final submission.

"What if it fails?" Sometimes, the results don't come out the way that you'd hope. That's an expected — and even important — part of science. Report the negative results, and unpack for us why you think it didn't turn out the way you hoped. A quote to remember: "If we knew for sure that it would work, it wouldn't be research." In the case of negative results, we will evaluate your final paper based on your effectiveness at unpacking the reasons why the idea didn't play out in the way that your thesis expected it to. You are welcome to do follow-up work if you have bandwidth.

Submit your final paper in the format specified by your section. Total word length should be 4000-5000 words (for a 3-4 person group) or 3000-5000 words (for a 2-person group), not including references.

Part 2: Complete the Team Dynamics Assessment Form

Generally, teams work constructively together to complete projects. It's important for us to know whether this is the case in your project. In the case of substantially imbalanced contribution, we will adjust participation grades to compensate.

So we can understand this, each member of the team needs to fill out the team feedback form by the paper deadline. We will not return grades to your team until every team member has submitted the form.

Fill out the form here.

Submission and Grading

Submit a PDF to Canvas containing (1) your final paper, (2) your changelog, and (3) a link to your Overleaf document, and (4) any code you wrote for this paper. This code could be a link to the colab, a link to a github repository, analysis scripts, etc. This is a group assignment; create a group for your team, and one member should submit on behalf of the group.

In addition, all team members must individually submit the team dynamics assessment form.

Your final paper will be graded on the following rubric:

  • Thesis and bit flip: is there a clear research question and idea that makes a significant addition to your research area?
  • Execution: has the thesis been explored thoroughly through the approach being pursued?
  • Evaluation: has the thesis been tested convincingly?
  • Clarity: is the writing overall clear and easy to follow for a technical expert in the field?
  • Related work: does the paper cover major points of related work, and explain how this research extends them?

If the paper is over or under the word limit, the project will lose 10% credit. Being appropriately pithy is a skill!

We have an adjusted late policy for the final paper. All papers submitted by the original deadline, Tuesday 11:59pm PT will be graded as usual. We will also pre-grant a 24-hr extension, where all papers submitted by Wednesday 11:59pm will be capped at (not scaled to) 90%. For example, a 92% paper grade would be capped at 90%. Groups should message their CA if they plan to use the extension so that staff can grade accordingly.