Assignment 1: Reading a Paper

Summary

Research gets transmitted in many forms — demos, code, talks, and more — but its formal report most often occurs through a written paper. The paper explains the problem, the approach, and an evaluation. When ready, the author submits the paper for review by other academics; once it passes this peer review process, the paper appears at a venue such as conference or journal. Since most research is disseminated in the form of papers, it's critical to be able to read research papers and make sense of them.

Your first assignment will be to read a paper in the area of your project interest, and to synthesize its main argument into a detailed outline. In addition, you'll also schedule two weekly meetings (one with your mentor, and one with your CS197 small group) and set up your weekly log.

Part 0: Ask Your Mentor for a Paper to Read (ASAP)

Complete this step ASAP so you can start Part 2 (below).

  • Ask your mentor for a paper to read that is related to your project.
  • If your mentor has not responded to you by Monday, email the course staff, and we can assign you a reading.
  • Register your paper here (column I).

Part 1: Schedule Weekly Meetings

Schedule meetings with your research mentor and CS197 team.

  1. Research mentor meeting: Your research mentor will be the main point of contact for project scope, direction, and technical details. Establishing communication early and often with your mentor will be key to making progress.

    • You should schedule a 30-minute meeting time with your research mentor that you both will attend weekly starting (latest) in Week 2.
    • If you can, try to schedule this weekly meeting before the CS197 small group meeting.

  2. Research lab group meeting: If your research group has a regular group meeting, ask if you can join! Your mentors have been prompted to invite you to these weekly or biweekly research meetings, but reach out to them if they haven't informed you of the meeting time.

You will submit these meeting times as part of a spreadsheet (see the Submission section).

Part 2: Read a Paper

Read a paper and outline its argument and structure.

Your research mentor has given you a paper that is related to your research. Find yourself a quiet place and work your way through it. Don't worry if you can't understand every detail; focus on understanding the paper's big ideas and how they are argued. It can take time to read a paper — don't feel discouraged if it takes you a long time.

Outlining a Paper

Your outline should recompress the paper back into an outline format that explains the paper's argument. We suggest the following structure of one paragraph per bullet, with an example outline following the description:

  • Title: What paper did you read?
  • Problem: What problem is it solving? Why does this problem matter?
  • Assumption in prior work: What was the assumption that prior research made when solving this problem? Why was that assumption inadequate?
  • Insight: What is the novel idea that this paper introduces, breaking from that prior assumption?
  • Technical overview: How did the paper implement that insight? I.e., What did they build, or what did they prove, and how?
  • Proof: How did the paper evaluate or prove that its insight is correct, and is better than holding on to the old assumption?
  • Impact: What are the implications of this paper? How will it change how we think about the problem?

You may adapt this structure if needed for the paper you are outlining; check with us first if your proposed structure deviates by more than one bullet.

Example, for the paper Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations:

  • Title: Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations
  • Problem: Crowdsourcing has been used successfully for many goals that can be decomposed into small, modular microtasks, but it has struggled to achieve more complex goals such as design and engineering. For example, tasks such as image labeling work because it's modularizable, but design is interdependent and requires adapting as you go, so crowdsourcing has succeeded at image labeling but failed at design. If crowdsourcing is limited to modular tasks, then it will never be able to achieve goals of meaningful complexity, which will limit its impact on the world.
  • Assumption in prior work: Prior work all takes an algorithmic model of crowdsourcing: the programmer specifies who does what, and when, in a kind of big algorithmic recipe to follow. This assumption shows up in goals ranging from Wikipedia ("edit this page"), to interactive crowd-powered interfaces ("find errors" --> "fix errors" --> "verify fixes"), to open source software ("create a module with this fixed API").
  • Insight: This paper proposed that instead of coordinating crowds as we do algorithms, that we should be coordinating crowds as we do with organizations. They propose a series of computationally-enhanced versions of the structures that organizations use — roles, tasks, hierarchy, and so on — and introduce the idea of a flash organization, which is a rapidly assembled collective of online collaborators who use these computational organizational structures to coordinate.
  • Technical overview: The authors created a system called Foundry that implements these ideas. Foundry is a web interface that connects to the Upwork online labor marketplace to draw on-demand expertise. It uses a combination of first-come-first-served hiring queues and Slack integration to bring workers onboard and keep them updated. It introduces an adaptation model drawn from the metaphor of code branching and merging to enable the organization to adapt.
  • Proof: Three non-crowdsourcing experts used the system to convene and lead flash organizations to achieve proof-of-concept complex goals. These experts created (1) a tablet system for EMTs to report medical trauma cases enroute to the hospital, (2) a card game for storytelling, which was playtested and iterated upon, and (3) an enterprise-grade event planning system that had to meet branding and security standards.
  • Impact: Flash organizations offer a broad new view of crowdsourcing — one that's not rooted in Tayloristic algorithms, but instead in an organizational metaphor. This approach can achieve far more complex outcomes, enabling crowdsourcing to apply to a broad new class of problems. It has implications for the future of work (how do we protect labor rights?), for organizations (what will organizations look like in the future if flash organizations are widely deployed?), and for collaboration (will we all work remotely?).

Paper options

It is okay if you have read the paper before (particularly if you started your research work prior to this week), but you should pick a paper for which it would be useful for you to do this exercise.

Note: If you encounter a "Get Access" button on any paper links (e.g., for IEEE or ACM), make sure you are using Stanford's EZproxy.

Part 3: Start a Research Log

Create and share a weekly research log document with CS197 staff.

A central skill to your success in research projects and beyond will be the ability to plan, track and account for your efforts and achievements. In this course you will do this planning and tracking using an online research log. This is essentially an electronic research notebook that is viewable to the rest of your research team (or in this case, the CS197 team and your research mentor).

Research logs serve a variety of purposes, including (but not limited to!):

  • To help you plan and remember your goals for the week
  • To help you measure progress toward these goals, and get better at goal setting and time management
  • To communicate your activities and questions with me and with the other members of your group
  • To keep track of findings, ideas and results related to your research

Getting Familiar with a Research Log

Before you begin setting up your own research log, take a moment to get a feel for what a research log is all about by looking at the logs of some of past research students. These aren't exactly in the format that we'd like you to keep your logs, but as you get more advanced with research, you'll be able to develop your own style and these logs show some of the fundamentals as well as some of the acceptable freedom of keeping a research log.

Creating Your Own Research Log

You should create a new Google Doc with the name "[Your Name] Log" and share it with both Yanay and Michelle (i.e., the page should be viewable to mguo95@stanford.edu and yanay@stanford.edu, or publicly viewable to anyone with the link). Also share your log with your research mentor. This doc will be your log for the quarter; read the instructions below for what content to put in it.

Required entries and formatting

Every week you will be adding to your research log following the below guidelines (if you have any questions or want to add something, just let us know):

  • Every week you will add a header for the week (e.g. "Week 1" or "Week 7") with the dates. Our "Weeks" run Thurs to Wed. We'll be reading the logs before your small group meetings.
  • Entries are listed in reverse chronological order, with the most recent week and entry always at the top of the page.
  • Under that header, you will list the goals for the week.
  • Each day you work, you will enter the date of your work, and record your accomplishments. You may make additional notes as necessary.
  • Each day when you are done working, you should record the amount of time you spend working at the top of that day's entry.
  • You may also be prompted to put or link to reflections in your logs (this varies week-to-week).

Week 1 template

Here is a template/example for your entries (that you can copy and use much of for this first assignment).

Week 1 (4/3-4/7)

Goals:

  • (Maybe) Attend research group meeting
  • Set up research log
  • Record CS197 preliminary thoughts
  • Reflect on research logs

Friday, April 7 (1.5 hours):

  • Set up log.
  • Wrote log reflection
  • This is what I am most excited about in research:
  • ...
  • This is what I am most nervous about in research:
  • ... [etc.]

Initial thoughts

In your log (either directly, or on a separate Google doc that you link to--make sure that you make the page viewable to mguo95@stanford.edu, or publicly viewable to anyone with the link), write answers to the following questions:

  • What are you most excited about in research, and why?
  • What are you most nervous about in research, and why?

The answers to these questions should be in complete sentences or short paragraphs. Really take some time to reflect on what you hope to get out of the program and be honest about what you think you might struggle with.

You will submit a link to your weekly log as part of a spreadsheet (see the Submission section). Each week, we'll give you qualitative comments and address questions in your log as well.

Submission

  • You are expected to submit one PDF for this assignment (the paper outline) through Canvas.
  • Fill out the following spreadsheet which asks for the following:
    • a URL of your Weekly Log, which should be accessible to Michelle and your research mentor, and populated by class time on Tues of Week 2.

Grading

Your paper outline will be graded on the following rubric:

  • Accuracy: does the description correctly describe the paper?
  • Completeness: does the description capture all the important ideas in the paper? (Not all the ideas! All the important ideas.)
  • Clarity: does the description convey the ideas understandably to the reader?

Check out the logistics page for more details on how we grade each research log.