Here at Carleton we’re getting ready to do our next ILSW reading. It’s my first time coordinating the project (the amazing Heather Tompkins who coordinated in the past has moved on to another institution), and I’m working with our fantastic Institutional Research folks to select a sample of papers, but I’m also curious to know any ideas or advice you, gentle reader, might have about the sample selection process.
We select our papers from the campus-wide Sophomore Writing Portfolio, where every sophomore has to submit 3-5 papers for rating by the Writing Program and faculty and staff volunteers (which is actually going on as I type). We typically select 100 portfolios (sometimes a little more), creating a representative stratified sample… and of course the first big question is “representative of what.” In the past we’ve used gender and portfolio reader score for this first selection, making sure that we read portfolios from about the same proportion of men and women as we have on our campus, and reading proportional numbers of portfolios that were given a Pass, Needs Work, and Exemplary score. This year we’re also balancing proportional U.S. White, U.S. Student of Color, and International Student portfolios.
Once the portfolios are gathered, we then run at least one more pass to select individual papers from those portfolios (we read one paper per student). We have the option of balancing out papers based on:
- What the Writing Program calls “Gen Ed:” Science, Reasoning, Social Inquiry, Arts Practice, Humanistic Inquiry, Lit/Art Analysis
- Characteristics the students identify about each their submitted papers, choosing from: reporting an observation, analyzing complex information, interpretation (of data, text, art, etc), documenting sources, articulating and supporting a thesis driven argument. (Papers can fall into more than one category but must fall into at least one category).
- Department housing the course for which the paper was written (History, Biology, etc)
- “Overlay” (A Carleton thing where faculty have the option of saying that their courses are Writing Rich, enhance Quantitative Reasoning, Intercultural Domestic Studies, and International Studies)
The big questions, of course, are how far we want to control the sample given our relatively modest sample size, and exactly what controls matter to how we’re thinking about our reading and our results. If you were faced with these choices, what might you use to select that single paper from the full portfolio? And are there other demographics that you would use to select the portfolios themselves?