In this course, you will do research and write a report documenting your work and proposing your future work. At a high level, successful research this quarter should propose a research question and plan a methodology for answering that question. The exact progress you make on the execution of your research will depend on your prior knowledge and experience with the project, but the goal of this project is to help you write and think about the work that you are doing (or will do).
For information on how the project will be evaluated, see the individual rubrics under each assignment and the logistics page.
Here is a rough sense of the timeline from here on out (every week you will update your weekly log as you make progress on the below written assignments):
Many CS research folks use Overleaf (shared editing à la Google Docs for LaTeX) because it is a useful platform for writing papers collaboratively.
Stanford has an institutional subscription to Overleaf.
Here's a tutorial to the LaTeX typesetting language, and here's a tutorial to using BibTeX with LaTeX to handle format citations via the \cite{} command. Finally, here's an official Stanford library resource for using Overleaf.
We strongly suggest that you use Overleaf to write up all written assignments, as it will be way easier to copy over your work into your final paper (which will require Overleaf).
We provide an assignment template for each assignment that you complete on Overleaf.
In order to help you focus your time doing research over the quarter, you and your mentor will decide on a set of research goals to complete by the end of the quarter. The milestone should be doable in under 30 hours of work total (following the 3-unit courseload guidelines) and could be:
Through the week 2 section, you will discuss with Michelle potential ideas for a doable set of deliverables, then write them up here. You and your research mentor will then discuss the deliverables in your week 3 one-on-one and by the end of week 3 it should be clear to both you and your mentor what you are expected to finish by the end of the quarter as a satisfactory amount of progress. For this proposal, the format is flexible but in general should contain a bullet point for deliverable/s that:
This proposal will be a soft contract between you, your research mentor, and the CS197C teaching staff. It is expected that you complete your stated deliverables, but we also understand that research can and does morph throughout the course of a project. We will allow revisions to this plan, on the condition that your mentor signs off on the revisions. As a final note, there is a good chance your time estimates for the deliverables will be off, and that’s okay! Estimating time to completion for research deliverables is a skill, and one that you hone best through practice. If the time it takes to complete your deliverables changes (i.e.: you thought deliverable A would take 10 hours but it ends up taking you 20, leaving you less time to complete deliverable B which will take an estimated 20 hours as well), then as mentioned before revisions can be made with appropriate sign-off by your research mentors.
Please submit both a pdf containing your research milestone proposal and a URL to the web-hosted document containing your proposal. You can use whatever word processor you’d like, but we’d recommend using either overleaf or google docs, depending on yours and your mentor’s preferences.
This milestone proposal will be graded based on completeness: description of deliverables, why they are important for your project, and a time estimate for completing the deliverables.
Doing research means making a novel contribution. Novelty, as we discussed in class, is a litmus test for a research idea: a novel idea is new to the academic literature. To be able to argue for novelty, you need to know what ideas exist in the space. Then, you can articulate a bit flip relative to that prior work to motivate your own project. The point of a Related Works section is not to summarize past work, but to show readers how your work is novel in comparison.
This assignment takes your paper-reading skills from Assignment 1 and puts them to good use. You'll be doing targeted reading of a number of papers in order to synthesize the state of the literature and articulate your bit flip. The result will be a Related Work section that you can eventually adapt for your final paper.
Download Overleaf template: Assignment 2: Related Work. Make sure to upload the .zip file to Overleaf directly and not unzip it beforehand!
You have 2 weeks to do this assignment, but a literature search takes time–both to read papers and to connect big ideas. In the first week, we suggest you set up Overleaf and read/skim/understand about 10 pieces of related work (i.e., Part 1 and part of Part 2 below). Put references into the Overleaf document (.bib file). Then, read/skim another 5 related pieces of work and make your affinity mapping.
In the second week, complete Part 4 to write your Related Work section.
Read the nearest neighbor paper (in-depth). Your research mentor has provided you with several papers for your project. Pick the nearest neighbor paper, which is the most adjacent piece of literature to the idea that you are pursuing. It's important that you understand it well. This might be the same paper you read for assignment 1, or it might be another related paper.
Read the nearest neighbor paper, and be able to state in your own words how your project is expanding on, or differing from, the nearest neighbor paper — and why that difference is an important one for researchers to pursue. It might be helpful to do a brief outline for the paper like you did in Assignment 1.
Skim the nearest neighbor's related work. Next, pull out the five most relevant pieces of related work cited in that nearest neighbor paper. This can be a mix of foundational work that is less adjacent to your project but sets the stage, and incremental work that is highly adjacent to the work you want to do. Skim each of these papers. You’re not necessarily reading them for every detail; you’re trying to understand how they’re situated relative to the work you want to do. Based on the outline structure for reading papers as in Assignment 1, if you can understand the Problem, the Assumption, and the Insight, you probably have it covered. A good way to skim papers is to read the abstract and introduction closesly, as well as to look at the figures.
From here, follow citations in the papers you found, or in the nearest neighbor paper, to build out the local neighborhood.
You can also look through additional papers listed in your project.
Use strategies that we discussed in class:
To make your later writing easier, you may want to collect the BibTeX metadata for these papers into a shared Mendeley/Zotero library, or into a shared .bib file in Overleaf. You can typically find the BibTeX through a couple clicks on Google Scholar and on other sites that host papers.
You should get to the point that adding an additional paper is only telling you incrementally more information than you’ve already roughly assimilated. Most likely, you'll start hitting that asymptote after about fifteen papers: often, after this, the ideas will either be more of the same, or will start becoming more distant from your project. You may want to split up papers and tackle each a set of papers by topic.
At this point, you have all the raw materials that you need to write a Related Work section of a paper. In addition to being a useful exercise at this point in the project, you will be able to keep what you write for this assignment and adapt it for use in your final paper.
As a first step, perform an affinity mapping of the papers you've collected.
Submit a screenshot or photo of your affinity mapping on Canvas. Your affinity mapping is graded on completion; however, your grade for the next part will depend on the quality of your affinity mapping.
Next, use this affinity mapping to organize your Related Work section.
\subsection{} LaTeX command) if it's helpful.If you're unsure exactly what the goal of your current project is (e.g., you're doing mainly exploratory work right now), ask your mentor what they suggest as a novel direction for the purposes of this assignment, and mention it to course staff in your small group meetings.
As a starting point, some genres of bit flip argument include "people have done this style of work with an existing method X before, but method X has drawbacks; our new method Y is better because Z", "everyone else's problem setup was slightly different, and our problem setup requires changing the solution entirely", and "similar solutions have been applied in other areas, but not to ours, and require an adaptation to solve our problem".
Finally, instead of a sentence or two at the end of each paragraph, another format for the bit flip is an entire paragraph at the end of the Related Works section, such as in this paper (pages 2 & 3). The first few paragraphs of each subsection survey the related work in a narrative way, and the last paragraph articulates the bit flip. You are welcome to do either format for your assignment, but we recommend having a mini bit-flip at the end of each paragraph, as it's often easier to articulate how your project differs from a smaller subsect of other research that all share a common thread.
Apply the following writing heuristics when writing your Related Work section:
\cite{} command. Citations come before punctuation at the end of a sentence.\cite{} command (e.g., This is a sentence~\cite{}).Your Related Work section should target 600-700 words and have at least 15 citations (But citations doesn't means you have to discuss details about all of them).
Submit a PDF of your Overleaf Related Work project on Canvas.
Your related work will be graded on the following rubric:
Now that you understand the literature and the bit flip that you're making, it's time to write up a project proposal in the form of an Introduction section for a paper. As researchers, we often write up a paper introduction like this as we embark on the project as a check that we can explain a clear vision of the project. If we can't explain the project clearly in writing, then we don't understand the project and its contribution well enough to move forward!
The result of this assignment will be an Introduction section that you can eventually adapt for your final proposal. Think of this Introduction as also serving the role of a project plan for this quarter and the summer, if applicable.
Download Overleaf template: Assignment 3: Introduction
While there are many approaches to write an introduction, in Computer Science most of these approaches need to explain two things: What is the problem, and How are you solving it?
Michael Bernstein, an HCI faculty member in our department, tends to follow the following paragraph-by-paragraph outline for an Introduction:
Problem: What's the problem you're solving? Cite other work that helps motivate the problem. Make the reader feel the urgency of the problem, so that by the end of the paragraph they care a lot that someone finds a solution. Example topic sentence:
We all want to build IoT applications that are responsive to our behaviors — for example, coffee machines that make coffee for us after we wake up, or phones that know not to ring while we're in meetings — but we lack the data to endow our algorithms with this information about our patterns and behaviors.
Set up the bit: This paragraph usually answers the question, "Why isn't this problem solved yet?" by setting up the bit that you're going to flip. The topic sentence typically summarizes existing approaches to solving the problem by articulating the bit that they share, and points out why those approaches and the bit more broadly hasn't been able to solve the problem. So, it's a brief summary of the most important related work — but it's a summary, not a survey, meaning that its goal is to set up the bit, not to cover everything. Example topic sentence:
Existing solutions to this problem require manually authored rules (e.g., coffee is a morning drink, we drink morning drinks after we wake up), but the required manual effort results in incomplete data about human situations and behaviors.
Flip the bit: Here you introduce your insight by flipping the bit. The topic sentence of this paragraph is the thesis statement for the entire paper. It should lay out the big idea that you'll be pursuing. Spend this paragraph explaining your insight clearly, why it flips the bit that you set up, and what the implications of flipping that bit will be for the problem at hand. For the purposes of this assignment, it's fine to assume that your bit flip will work out — we all know that research is iterative and projects will evolve. Example topic sentence:
In this paper, we suggest that data mining a large dataset of modern fiction offers a promising alternative to manual annotation.
Instantiate that bit flip in a solution: At this point, the reader understands the idea that you're proposing, but it's still very high level. In this paragraph, map that idea onto a concrete instantiation. This is where you introduce the system you've built, the model you're creating, the study design, or whatever is the main artifact of your research, and how it instantiates that insight. Example topic sentence:
We introduce Augur, a knowledge base that relates human activities and the objects around them by mining 1.8 billion words of modern fiction from the online writing community Wattpad into a vector space model for predicting human activities.
Evaluate the solution: We haven't discussed evaluation yet in the class, but a good introduction summarizes the evaluation that you performed to demonstrate that flipping the bit has the effects that you suspect. For the purposes of this assignment, ask yourself: how would you prove to a critical reader that flipping the bit has the effect you promised in solving the problem? We expect you to articulate what you will be manipulating and measuring, but we don't require any actual results right now and we won't hold you to this evaluation strategy — especially since sometimes your evaluation strategy evolves as your project evolves.
Example topic sentence: To evaluate Augur's ability to predict human behaviors, we first compared Augur's predictions to human estimates of behaviors given the same inputs, and second performed a field deployment to test Augur's precision and recall in a realistic application.
Implications: If you're right about this bit flip being the right way for the field to go, what implications will there be for the field? Don't overplay your hand here by stating that all of computing will shift, but do be clear and forceful in pushing the case as hard as you feel that the case warrants.
Example topic sentence: This work demonstrates that fiction can provide deep insight into the minutiae of human behavior, and that modern NLP algorithms can structure those minutiae into useful hooks for IoT applications.
This outline will depend a bit on the kind of paper you're writing. For example, start by answering: are you an old problem / new solution paper, or a new problem / old solution paper? Old problem / new solution papers already have a warrant for the problem, so that part of the Introduction is easy, but they need to argue for the solution very carefully since there's no warrant there. Similarly, new problem / old solution papers need to be extra careful to argue the problem persuasively, but they can draw on existing warrants for the solution and only argue for how they're adapting it to the problem.
Start by creating the outline for your Introduction, using only the topic sentences that you will eventually expand into paragraphs. This is harder than it sounds! It means that you only have those six sentences to tell the entire story of your paper. So, you can't add any unnecessary detail or extra concepts, or those six sentences won't be possible for a reader to understand. Just like a pro swimmer makes no unnecessary motions as they focus all energy on executing a stroke, and just like a pro chess player makes no unnecessary moves as they focus their actions on enacting their strategy, your topic-sentence outline should introduce no unnecessary concepts or ideas as it focuses on explaining the project.
We highly suggest getting feedback from your TA on your outline before moving forward.
Now, to write the Introduction section itself! Each topic sentence becomes the anchor of a paragraph in your Introduction, and the rest of each paragraph should set out to prove the point made by that topic sentence. If you need to split a paragraph into multiple paragraphs to be clearer, do so — but keep the rough proportions of those original six paragraphs. (In other words, don't spend three paragraphs on the problem, then rush through everything else.)
Don't forget to cite any work you're referencing that motivates your problem, sets up the bit, or helps explain why the flip is a good idea.
Here are example introduction sections for different research fields, selected by previous CS197 CAs. Bear in mind that these papers will stray from the outline format above—which is fine, experienced writers play with the format—but we still feel that they represent strong examples of Introduction sections.
Your Introduction should be 700-900 words. Include a References section with citations at the bottom; references are not included in the word count.
Submit a PDF with (1) your outline, and (2) your Introduction on Canvas. Your introduction will be graded on the following rubric:
This assignment will center around articulating the details of your project. There are two parts: a proposed solution section and an evaluation plan.
Download Overleaf template: Assignment 4: Proposed Solution and Evaluation Plan
You have 1 week to do this assignment; compared to the previous assignments, we anticipate that Part 1 (which involves multi-paragraph writing) will take longer than Part 2. In the first part of this week, we suggest you write as many ideas down as possible for both sections; in the second part of this week, take more time to make your Part 1 cohesive and complete, finish up Part 2, and then return to Part 1 to address any missing details.
At this point, you have written about your project idea in 1-2 paragraphs in your Introduction, but you have likely omitted many of the finer details. Many papers include this as a "Section 3" to their work (after Introduction and Related Work, usually a "Methods" section), and you will eventually do the same for your final proposal. However, all of you are at different points in your projects: some of you are already evaluating solutions, while others of you have just found your idea for the summer. The goal of you writing this section is therefore to describe the high-level and low-level details that a reader would need to understand your Evaluation Plan (Part 2 of this assignment). In particular, it is an opportunity to give further information about the thesis you write at the top of your Evaluation Plan. Think of it as a mini approach section so that a future researcher (or a future you) could understand and begin implementing/evaluating what you described. We expect that you will have about 400-600 words for this part, though feel free to write more as is necessary.
The exact content and structure of this section will vary based on the type of contribution you are making, as well as your project field. Note that the similarly scoped papers linked below go into much more detail than what is expected of you at this point, but you can refer to them to get a sense of what you need to describe.
It is okay if you find yourself largely summarizing prior work, but you should use your own words. Be clear about what is existing work (prior to you joining the project) and what you are adding/plan to add.
If you are unclear on particular specifics of your proposed solution, ask your mentor to clarify where possible. We will also try our best to give feedback early in time for your next assignment.
We encourage you to include your own figures or figures from previous work (with citation) as needed to explain your proposed solution! Figures are a great way to visually communicate a proposed system or idea. Use the \includegraphics{} command in LaTex to add one.
To convince people that your idea is correct, you'll need some way to convince an expert that you have evaluated it fairly and correctly. In this part of the assignment, you will develop an evaluation plan for your research project. Unlike Part 1, you will not be writing a multi-paragraph section. Instead, complete the three subparts and include as much detail as you are aware of so that you or another researcher could carry out your evaluation plan.
As we discussed in class, the first step in planning an evaluation is to articulate the main thesis of your work. (Remember from Assignment 3, Project Introduction, that the main thesis of the work is likely embedded in the topic sentence of your bit flip paragraph.) Go back and reflect on that statement — tweak it if necessary based on what you've learned from your project so far.
Write out your thesis at the top of your submission.
We discussed in lecture how theses imply a claim. For example, "x > y"-type ("X is better than Y") theses imply a claim that x is in fact better/faster/more performant/more enjoyable than y, and "∃ x"-type ("there exists an X") theses imply a claim that whereas x could not exist before, that it does with your system. Discuss with your team the claim implied by your thesis.
Write down your claim.
Now, you need to work from your claim to design a specific evaluation plan. How do you prove what you have claimed? This evaluation plan typically specifies:
You don't need to re-invent the wheel here. Often your nearest neighbor paper or other papers in your related work establish an evaluation paradigm that you can import to your paper. In fact, this is often preferred, since then you don't need to convince a reader that your approach is valid, since it's already in the literature. So, go review the evaluations used in your prior work and use those to develop a few possible models. Then, share those models with your team and work together to develop a variant that works well for your project.
Based on your project's setup, your model might look slightly different than what is laid out above. If you believe this to be the case, talk to us about it.
Next, run the following unit test on your proposed design: does it directly test the thesis you articulated above? Imagine a few possible outcomes from your evaluation. Depending on how it comes out, does it directly prove or disprove your thesis, or only obliquely shed light on whether your thesis is correct?
Write out the DV, IV, Task and Threats for your evaluation design. Summarize your explanation of why that design directly tests your thesis.
Submit a single PDF with (1) your Proposed Solution section (500-700 words or longer), and (2) the three sub-parts of your Evaluation Plan. Your work will be graded on the following rubric:
This assignment compiles together the parts you completed from previous weeks. You will also include a timeline for the rest of the quarter and for the summer, if applicable. Finally, you will have 1 week (until Tues, June 6 at 4:00pm) to peer review and give feedback on one of your classmates' submissions.
It is okay if your project has changed in scope since the beginning of the quarter, meaning that your older introduction/related work is inconsistent with your current project. The goal of this proposal really is to help you understand the scope of your project and your contributions so that you are ready for the summer.
If your project has changed significantly, we are expecting the following:
Download Overleaf template: Assignment 5: Draft Proposal
You will receive peer review feedback on your small group partner from your small group partner. Note that this draft is due in Week 9, whereas both the final in-class presentation and the final proposal are due finals week. Therefore, aim to have this draft be close to your final proposal in terms of structure and content. Part 1 of this assignment is writing-heavy and we will focus our feedback there; Part 2 is mainly for your own project planning or reflection purposes.
Change the title of the document to your actual proposal title (instead of "Assignment 5: Draft Proposal").
Your Proposal will have several sections, in order:
The first four sections are edited versions of what you submitted for previous assignments. Revise these four sections and compile them into the same Overleaf document. We will release feedback on your Introduction and RW by Saturday 5/25 EOD -- be sure to incorporate the feedback that the CS197 staff have specifically highlighted for these sections. Note that the first three sections should be cohesive paragraphs of content; the Evaluation Plan outline follows the guidelines of the previous assignment.
Note that if your project has changed significantly, we are expecting that you rewrite parts of some sections (see this assignment's overview for specifics) to fit your new project. You can rewrite more if you feel it will be useful for you to refer back to during the summer, but the above is what we'll be looking for.
If you will be doing research over the summer:
Make a timeline (in as much detail as possible) for completing the work through the 10 weeks of the summer quarter. Unlike the Evaluation Plan in the previous assignment, this part is primarily for your own project planning purposes. Put another way, imagine that a reader who wants to do your research (i.e., you) will read through the previous sections of your proposal to understand the technical overview of the project. The reader, who has 10 weeks to work on this project, will then read this section to orient themselves to the current speed and direction of the project.
As mentioned in our vectoring and velocity lectures, it is very likely that you will deviate from this timeline and list of tasks! Nevertheless, it is worthwhile mapping out both the creative vectors and engineering vectors that you currently believe will need to happen. Also include estimates of how long it will take you to flesh out different aspects of your project. That way, you can return to this timeline throughout the summer and revise what aspects are important/not so important so that you can get where you want to be at the end of CURIS.
A bulleted list or table is preferred for this section, though feel free to flesh out details in paragraphs as needed.
If you did research this quarter and will not be continuing over the summer:
Write a brief (200-300 word) reflection of how your time on research has gone so far this quarter. Some things to consider:
Change the title of the document to your actual proposal title (instead of "Assignment 5: Draft Proposal").
Submit a PDF with all five sections of your proposal. Your draft proposal will be traded with another individual for peer review after the deadline; this will most often be your small group partner, though it may not be. We, the staff, will also send feedback on this draft.
After the deadline, please submit your draft proposal to your research mentor for feedback -- we recommend going over the draft with your mentor during your weekly meeting and taking detailed notes on their feedback. Please submit your research mentor's feedback on Canvas by Tues, June 6th. You are expected to incorporate this feedback in the final proposal, due Sunday June 9, 11:00pm.
Your draft proposal will be graded on the following rubric:
You will give a practice presentation (5 min) of your proposal in your 30-min staff meeting. You need to draft and prepare for your final presentation. Please see below for presentation tips and format.
As the first part of the assignment, please read through these slides on "Tips for Giving Clear Talks" by CS faculty Kayvon Fatahalian. Keep the suggestions in mind while designing your own talk; you will be graded on if you implement the tips. (E.g., one simple suggestion that everyone should follow, unless you have a strong reason not to, is having section headings.)
The timing means you won't be able to cover every detail of your project. The goal of your talk is to tell your audience what you have achieved this quarter. Think about what you really want to communicate. It likely draws heavily from your introduction and bit flip:
Assume that the audience for your talk is a researcher in your broad area (e.g., HCI, systems, AI, security, etc.). So, you don't need to re-introduce the whole field of HCI. A sentence or two to situate your work in the field is good, but spend the rest of the time telling us what you did.
Submit your draft presentation slides (a link and a static PDF copy) on Canvas by 4pm on Thursday, June 6. In addition, please submit your research mentor feedback on your draft proposal as a PDF on Cavnas.
Your practice presentation will be graded on the following rubric:
Please go over your draft proposal with your research mentor at your weekly meeting and obtain their feedback. Please submit their feedback on each section of your draft proposal:
If your research mentor thinks a section looks good (no changes needed), you can write "LGTM" for that section.
Note: You are expected to incorporate this feedback in the final proposal (due Sunday June 9, 11:00pm).
Submit a PDF of feedback from your research mentor on Canvas. Please list the feedback you obtained for each of the sections above. This assignment will be graded on completion.
Our final presentations will be on Monday June 9th 7:00-10:00pm, at Lathrop 299. You are highly expected to be in attendance the whole time. If you cannot make it with a hard conflict, you may altenatively submit a video record. Each student will present for 5 minutes followed by 2 minutes of Q&A.
Talks and presentations are one of the main mechanisms through which researchers communicate their ideas. As part of your final project, you'll be giving a talk on your research.
You have 5 minutes to present your research to the rest of the class, the staff, and the research mentors. To ensure a smooth flow between projects, we will be running everything off a shared Google Slides deck. So, create your slides using Google Slides, and insert them after the title card of your name. We will run the presentation in presenter mode, so you'll be able to see any presenter notes you leave.
This presentation is worth a substantial amount of your grade, so it's worth practicing it. At 5 minutes long, you can practice it a half dozen times in an hour. It's also short enough that you might consider writing out what you want to say long-hand, make sure you convey information efficiently and effectively. Practice it enough times so that you have it basically memorized, but not so memorized that you get flustered if you skip a word or someone asks a question, or that it sounds robotic when you deliver it.
Finally, to encourage supporting your classmates, we will be giving participation points for asking questions. Please ask at least one question to another student during the final.
Submit your slides to the Google Slides deck before Sunday June 9, 11:00pm. Make sure your media sharing permissions are correct, especially if you include video. In addition, submit a PDF static copy of your slides on Canvas by Sunday June 9, 11:00pm.
Your presentation will be graded on the following rubric:
This is it! Finish and submit the research proposal that you've been working on this quarter. You have all the tools; now it's about finishing what you started. As a reminder, when you're done, this is a document for you. Feel free to share it with your mentors or refer to it when you are working on your CURIS project.
No Overleaf template: Build on your Draft Proposal project.
You probably don't have too many new discoveries since your draft proposal, but you now have feedback from your peers and from your CS197 mentor. Integrate these into your final submission. This feedback will likely involve minor or major revisions to your introduction, related work, approach, or other sections.
Include a brief changelog (no more than a few paragraphs) that details the major changes you made between your draft submission and final submission.
Submit your final proposal in the format specified by your section. Total word length should be at least 1900 words and no longer than 4000 words, not including references.
Submit a PDF to Canvas containing (1) your final proposal and (2) a link to your Overleaf document (either as a separate text file or at the beginning/end of your proposal).
Your final proposal will be graded on the following rubric (it may be useful to compare these to the draft proposal):
Congrats on finishing your milestone! Please submit: 1) the deliverables in your research milestone proposal on Canvas (e.g., code, document, images), and 2) a PDF document (README) of what parts of the milestone you completed vs. not completed. The milestone will be graded based on completion.