Proposing a Research Project

Day 4 activity — pitch an idea, read the feedback, revise

---

The task

The final project is the main work of the second half of the course and 40 percent of the grade. It starts today. By the end of class you should have pitched at least two ideas through the form on this page, read the feedback, and submitted a revised version of one of them. By Oct 22 you submit a proposal. Everything pitched, and the feedback it received, is visible to the whole class on this page, so you can also read what your classmates are proposing and what the first reader said about it.

The due date is two weeks out for a reason. A proposal that has been through two rounds of criticism is a different document from one written the night before, and the weeks after Oct 22 are for doing the work, not planning it.

Three kinds of project

A CS398 project is acceptable if it is one of three things. It does not need to be more than one.

Good science	A learning experience people love	Goes live in PAI
A precise question, data that can answer it, a baseline, and one measurement. The answer should be interesting whichever way it comes out. The output is a finding you can state in one sentence with a number in it, and a short paper or website that defends it.	Something a named kind of learner does, built well enough that real people use it before Dec 3, and evidence that they loved it: they came back, they stayed longer than they had to, they told you so without being asked. The output is the working thing, the people who used it, and what you measured.	A change to Probability for AI, a course running now with thousands of learners and volunteer section leaders. It is built to work at that scale, tested in stages, and measured with PAI's own data. The output is the component, the stage it reached, and the evidence for taking it further.
Example: do a chess puzzle's tactical theme tags carry difficulty information beyond its rating? Baseline: a logistic model on the rating gap. Measurement: held-out log-loss.	Example: a two-minute daily probability question delivered over WhatsApp to 30 CUHK students, with a streak. Measurement: how many are still answering on day 21, and what they say.	Example: a weekly note for each section leader naming the learners whose lesson answers suggest they are stuck. Tested first against past answers and who stopped attending, then read by a few section leaders for two weeks.

Projects that go live in PAI

Changing a course that thousands of people are taking is a high bar, so PAI projects are rolled out in stages. The more a project changes what a learner or a section leader sees, the more stages it goes through before it reaches everyone. The teaching team decides when a project moves to the next stage.

Stage	Who sees it	What it needs first
1. Offline	Nobody. It runs on PAI data that already exists.	A data request, and a way to check the output against what learners actually did
2. Staff	The teaching team, and a few volunteer section leaders who agree to try it	Evidence from stage 1, and a version safe to put in front of people
3. Pilot	A few sections, one lesson, or learners who opt in	Staff feedback, and a plan for what is measured and how it is switched off
4. Everyone	All PAI learners	A successful pilot, and the PAI team's agreement to maintain it

Most projects this quarter will finish at stage 1, 2 or 3, and that is a complete project. A component that only reads data, such as a model that flags learners for a section leader, can be a full result at stage 1. A change to a lesson or to the agent project needs staff and pilot stages before anyone else sees it. The proposal names the stage you are aiming for by Dec 3 and the result there that would justify the next one. PAI data is covered by the same conditions as the other datasets below.

Earlier offerings of this course described three options: recreate an advanced idea, produce a novel research idea, or make a new tool. Each fits one of the kinds above. A recreation is science if it tests whether a published result holds on new data. A tool is a learning experience if someone loves using it, and a PAI project if it is built for that course. Design work from a Hong Kong context, which most of the field has never done, is welcome in any kind.

What makes a good project

	Good science	Learning experience	Goes live in PAI
The centre	One precise question	One named learner, and the thing they do	One part of PAI, and the learners or section leaders who meet it
Either way	The result matters whichever way it comes out	A reason they would love it that is not "it uses AI"	A reason PAI is better with it, at thousands of learners and with no per-learner staff time
Evidence	Data that can answer the question, in hand within two weeks of the proposal	Real people who will use it before Dec 3, whom you can name now	The stage you aim for by Dec 3, and the result there that justifies the next one
Comparison	A baseline	What the learner does today instead	What PAI does on that surface today
Measurement	One named metric, and the sentence you hope to write	Return rate, time on task, what they say unprompted, plus one small check that they learned something	From PAI's own data at that stage: agreement with what learners did, use, what section leaders say
The hard part	Stated in the proposal, and scheduled first

Projects usually fail in one of these ways:

• There is no question. "Explore", "analyze" and "look into" describe a tour of a dataset, not a project.
• The answer is obvious either way, so nobody would change what they do because of it.
• It is too big for seven weeks, and no smallest version has been named.
• There is no baseline, so the number at the end cannot be judged.
• It is a platform with no particular learner, and no reason anyone would love it.
• The hard part is unstated, or is the part the team hopes to skip.
• Evaluation is "we will see how it goes".
• The data does not exist, or cannot be shared.
• The model is the project. The model should be the instrument.
• A PAI project only works if it reaches everyone at once, or needs staff time for every learner.

Scope

Seven weeks, teams of one to three, six to ten hours per person per week. Plan the smallest version first: the version that would still count as a result if everything else slipped. Name it in the proposal, then name the version you hope for. A smallest version done carefully is a complete project. The hoped-for version is what you do with the time left over.

Data you can request

Three datasets are available to this class, de-identified, by request. Each has already appeared in a class activity, so you have seen a sample of what the records look like.

Your own data. A learning-experience project usually needs people rather than a dataset: classmates, a CUHK course, a club, a school you have a connection to. That is encouraged, and it is also the part teams underestimate. Name the people in the proposal and how you will recruit them, and get their consent before collecting anything. Science projects can also bring their own data, with the same conditions.

How to request data. Name the dataset and the fields you need in the approach box of your pitch; the feedback will tell you if the request is vague. Once proposals are in, the teaching team prepares an extract sized to the question. Allow days, not hours. Conditions: the data stays de-identified and you make no attempt to re-identify anyone; it does not go into a public repository or a public demo; and you delete it at the end of the quarter unless we agree otherwise.

Writing the proposal

The proposal is the five boxes of the pitch form written carefully, plus a team and a plan with dates. Submit it through the form with the proposal box ticked. Keep each box under about 250 words. The feedback you receive is a first read; the teaching team reads every proposal and makes the call.

The five boxes follow Jennifer Widom's guidance on writing the introduction of a technical paper: what is the problem, why is it interesting and important, why is it hard, why has it not been solved before, and what are the key components of your approach and result. Her page is about papers. The same five questions are what a proposal has to answer, and the paragraphs you write now are the ones your final write-up will open with. Read the whole page; the advice on the body of a paper ("every section should tell a story") and on defining terms before using them applies to a proposal too.

Two additions for a proposal:

• Draw the figure now. Sketch the table or plot you expect to present on Dec 3, with made-up numbers. For a learning experience, sketch the retention curve or the quote you hope to have. If you cannot draw it, the question is not yet precise.
• A plan with dates. Three or four milestones between Oct 22 and Dec 3, each a thing that will exist, and what you will cut first if you fall behind. The hard part goes first.

For a learning-experience project the five boxes read differently: what the learner does; who it is for and why they would love it; why it is hard and what exists already; what you will build and with what; and how you will know. For a PAI project they are: what changes in PAI and for whom; why PAI is better with it; why it is hard and what PAI does today; what you will build and how it fits into PAI; and the stage you aim for by Dec 3 and how you will know. The form switches the labels when you pick the kind.

Get feedback on an idea

The form sends your pitch to a language model that has been given the criteria on this page, and returns a first read: how it understood the idea, what is strong, where it fails, questions to answer this week, the smallest version, one next step, and search terms for prior work. It takes about fifteen seconds. Then revise: submitting again continues the same idea as a new version, and the model sees the earlier versions and says what improved.

Expect it to be hard on the first version of anything. It is tuned to find the scenario where an idea fails, which is what you want before spending seven weeks on it. It is not the examiner. Everything you submit here, with your name but not your email, appears in the history below for the whole class. Reading feedback on other people's ideas is most of the value of this page, and this class is not graded on a curve.

Your name

Your email

Team (optional)

Kind of project

Good science

A learning experience people love

Goes live in PAI

Start a fresh idea instead

Title

Data No data yet Chess.com puzzles Code in Place Probability for AI Own data or own participants

Submit as my proposal (due Oct 22)

Plan with dates

Get feedback Start a fresh idea

At least three of the five boxes. Your name and the pitch are public to the class; your email is not. A pitch ticks Project Exploration on the My Progress checklist for the email you enter.

Ideas pitched and feedback given

Newest first. Each card is one idea; open Earlier versions to see how it changed and what the feedback said each time. Use Start a new pitch from this to borrow the structure of someone else's idea for your own.