Term projects for this class will take one of the following forms:

1. Take an existing off the shelf system and run it over some new data set with demographic information included. Do a demographically-informed error analysis. Present the error analysis and a discussion of its ethical implications, both in terms of what will happen if the system is used unmodified and how the system could/should be improved.
   • For example: running an off-the-shelf English ASR system over the data collected (and transcribed) by the Voices from the Rwanda Tribunal Project
2. Take an existing NLP task and analyze it in terms of Value Sensitive Design. Assume that systems approaching the task will be deployed by companies and/or government agencies and/or available for use by anyone as open source projects. Who are the stakeholders that should be consulted? What questions should the stakeholders be asked? If possible, find representative stakeholders from multiple categories and interview them. On the basis of these interviews (or your own analysis, if interviews aren’t possible), write recommendations for improvements in the training data, task design, or task deployment. If you are working on an MS thesis, and you know what NLP task it will involve, it is strongly recommended that you take this option with your thesis task.
3. Some other project that you propose.

Paper

Whichver project option you choose, the final paper will consist of a 6–8 page conference-like paper in ACL format, with an additional 1-2 pages focused on science communication.

Main Content

The main body of your paper will follow ACL conference format (LaTeX, BibTeX) (MS Word), and have the following sections:

• Abstract:
  • State the task/software/question discussed and your main conclusions.
• Introduction:
  • Briefly overview the questions you are approaching, summarize the main conclusions, and give an overview of the paper.
• Background/related work:
  • Present, with citations, the framework you are using for your analysis
  • Describe in siffucient detail the NLP task you are considering
  • Discuss any related work looking at ethical considerations pertaining to the same or similar tasks and how this work informs or differs from your approach.
• Methodology:
  • Describe what you did (e.g. running of software, interviewing people); describe any data sets you used
• Results:
  • Present what you found (error analysis, interview responses)
• Discussion:
  • Apply your analysis framework to your results, describe your recommendations
• Conclusion:
  • Summarize the main points and provide look ahead. What should happen next?
• Acknowledgments:
  • Acknowledge any help you received (e.g. thoughtful feedback from classmates, help from developers to run their software, your interviewees’ time)
• Bibliography:
  • References for all papers cited in the text. Does not count towards/against the page requirements. It is expected that your bibliography will include assigned readings from the class as well as additional readings specific to your term project.

SciComm Section

In addition to the conference-style paper above, your final paper should include a 1-2 page addendum that contains the following:

• Target Audience:
  • Describe in a couple sentences what community this lay summary is intended for.
  • This audience should be expected to be non-experts, but what is their relationship to the system?
    • Might data about them be used in the system?
    • Are they potentially in a position to influence the use of such systems?
• Summary:
  • Summarize the NLP task for your lay audience.
    • What does the system do?
    • How does it accomplish it?
    • What data is requied to train the system, if a machine learning approach. What decision making procedures are written into it if not ML-based?
• Risk/Benefit Analysis
  • What potential risks should your audience be aware of with the system?
  • What benefits do systems like this offer?

Milestones

To help you with this term paper, there will be several milestones along the way (dates TBD for the moment):

1. Project proposal: If taking options (1) or (2), this can be a paragraph explaining the system and data set (1) or task (2) you are approaching. If taking option (3), I’m looking for a page explaining the project.
2. Paper outline: Fill in the sections above (or modify, if somethign there doesn’t fit) with bullet points specific to your paper.
3. Paper draft
4. Comments on partner’s paper draft
5. Final paper