Peer Ranking with WebAssign

Peer ranking is now possible using WebAssign.  The following instructions give an overview of creating a ranking assignment.  It is assumed that the user is somewhat familiar with WebAssign - (you need to know how to create an essay question and how to create an assignment).

Types of Essays

It is important to realize that not every type of essay is appropriate for use as a ranking exercise. If the essay has multiple parts to it, the students will not be sure on which criteria to rank it. For the most part, the essay needs to be a simple question with one answer and an explanation. This does not mean it can not be a long essay or an essay that requires a long answer - just that it is straightforward with only one element. The following are two examples, the first is an essay that would not work well, and the second is one that would work well.

• When a punter kicks a football, is he doing any work on the ball while his toe is in contact with it? Is he doing any work on the ball after it loses contact with his toe? Are any forces doing work on the ball while it is in flight?   - Notice that the student must answer and explain three questions in this essay? How would you expect a student to rank another essay that answered two parts of this correctly with correct explanations compared to another essay that answered all three parts incorrectly?
• You are at the state fair and want to play a game where the object is to knock over a heavy board by throwing something at it. The vendor offers you a choice of things to throw, a tennis ball, or a ball of clay – both have the same mass. Which object would you choose to throw and why? - This is a much better essay question in terms of ranking. There is only one question to answer.

Type of Ranking

There are currently two "methods" of using peer ranking. The first method is method = "rank". This is the default method of peer ranking and it is described below. The other method is method = "review_essay". In this mode, the students will see the answer they had for a previous question. (see below for details)

Method="rank" - default

Step 1 - create essay assignment:

The current ranking process is actually 2 assignments.  The first assignment is a normal essay question.  The second assignment is the ranking assignment.  Thus the first step is to create an assignment with an essay question and make it available to students.  You will need to know two things about this - the essay question ID number and the assignment ID number.  It is helpful to let the students know that their essay will be viewed (anonymously) by other students and that this is the first part of a two part assignment. *Don't forget to weight your essay question.
 

Step 2 - create ranking question:

The best way to see how to create a ranking question is to first see an example.  Below is a ranking question that has been used. (Question ID = 134202 on webassign.net)

<eqn use Peer; ''>
<eqn $number_of_bins=3; $a=85670; $q=130245; ''>
<eqn $method='rank'; ($essays,$bins)=Peer::GET_ESSAYS($a,$q,$number_of_bins); $MODULE='Peer'; $GRADER='SPEARMANR';''>
<eqn @bounds = (22,15);''>
<eqn $MIN = 6;''>

 Below are 3 essays that you are to rank from 1 (best) to 4 (worst). No two
 essays are allowed to have the same rank, so even if two essays seem to be
 equally good, you must assign a unique rank number to each of them.(No ties
 allowed!) You may find it helpful to first read each essay and determine a
 numerical grade for each one, which can
 then be used to help decide the rank order.<P>
<EQN $$essays[0]><br><_><hr>
<EQN $$essays[1]><br><_><hr>
<EQN $$essays[2]><br><_>

the following needs to be in the answer box

 <EQN $$bins[0]>
 <EQN $$bins[1]>
 <EQN $$bins[2]>

The four red phrases are the ones that will most likely need to be changed.

• number_of_bins is a variable that basically represents how many essays each student will get
• MIN is the minimum score a student will receive on the ranking assignment (not the essay assignment).  So if MIN=5 and the ranking is worth 10 points, then all students will receive a score between 5 and 10.  This is used to make sure that students get some credit for doing the assignment.
• $a= and $q= are the assignment ID and question ID of the essay question that is to be ranked.  Basically this tells WebAssign where to look for the student essays.
• bounds = (8.9,6.9,4.9) are the essay score limits for the bins.  In this case, there are four bins and the essays were worth 10 points so the bin distribution would be as follows:

  Bin number	Essay score range
  1	10 - 9	Best essays
  2	7 - 8.9	Good essays
  3	5 - 6.9	Not so good essays
  4	0 - 4.9	Bad essays

• Note that for a four bin ranking question, there needs to be 3 bounds.
• Note: there must be an appropriate number of  <EQN $$essays[0]><br><_><hr> for the number of bins used as well as the appropriate number of <EQN $$bins[0]>
• method='rank', if you want to do normal ranking, then this is what you need. There are other methods, such as review_essay, or ALL. review essay only returns that particular student's own essay. ALL gives all ALL essays for that class.
• $GRADER='SPEARMANR' tells webassign to use the Spearman R correlation coefficient to determine the student score. The other option is to have GRADER='POLL' in which the correlation is not graded at all.

Step 3 - essay grading:

The essays must be graded in the normal fashion for WebAssign, but there are a couple of things to note.  It is not necessary for essays to be ranked, scored is fine and ties are fine as well.  In the example shown above, grading was based on what bin it should go into.  It is important to make sure that there is at least one essay in each bin.

Exemplar Essay

In some instances, you may wish to use an instructor created exemplar essay that all students see.  If this is the case, a few changes need to be made.

• <EQN exemplar_author = "userid"> needs to be added to the ranking question.  This tells WebAssign that there will be an exemplar essay and it will be written by whoever userid is.  To write the exemplar essay, make yourself a student and do the essay assignment.
• The exemplar will be in the first bin - thus the bin bounds need to be changed.  In the case of four bins, you would only need two bounds if you are using an exemplar.

An Example of Ranking Instructions:

Ranking Assignment Part I - essay submission

About this assignment

This is the first part of a two part peer ranking assignment. All students will answer this essay question. In part two of
this assignment, you will have the task of ranking 5 anonymous essays from other students in this class.

1. [58527] Two students are discussing the motion of a ball that is thrown straight up into the air. One student says that the acceleration at the highest point
is zero, the other student says that it is not zero. Which student is correct and why? (Be sure to give a full explanation as other students will be anonymously ranking your essay.)
 

Ranking Assignment Part II - essay ranking

About this assignment

Ball essay ranking exercise

This is part two of the two-part assignment that started with your submission of your essay question. In this part, you
are given 4 random essays to rank from other students.

1. [58529] Below are 4 essays that you are to rank from 1 (best) to 4 (worst). No two essays are allowed to have the same rank, so even if two essays seem
to be equally good, you must assign a unique rank number to each of them.(No ties allowed!) You may find it helpful to first read each essay and determine a numerical grade for each one, which can then be used to help decide the rank order.

Method="review_essay"

The "review_essay" method does not actually require the students to rank essays, but it does use some of the features of peer ranking. Review_essay allows a question to be created that shows the student one of their answers to a previous question. The way that this has been used is in having the student revise an essay question, but it could be used to have them expand on an essay question, or comment on one of their old responses. (An example of the review_essay method can be see in question  59158)

Step 1:

There must be a previous essay question that you want the students to see again. The assignment number and question number of this essay need to be known.

Step 2:

Now you must write a review_essay question. Below is a sample question (the important parts are highlighted in red):

<eqn use Peer; ''>
<EQN $method = 'review_essay'; ''>
<EQN $a = 13707; $q = 59153; ''>
<EQN $essay = &GET_ESSAYS($a,$q,$username); ''>
You answered the following question:

<p>You are at the state fair and want to play a game where the object is to knock over a heavy board by throwing something at it. The vendor offers you a choice of things to throw, a tennis ball, or a ball of clay - both have the same mass. Which object would you choose to throw and why?</p>

<p>Your previous answer was:<br>
<var>"<EQN $$essay[0]>"</var></P>


Give your previous answer a score from 0 - 10 and explain why it should be rated as you did (justify the score you gave yourself).
<_>

• require "peer.pl" - this is needed to tell webassign that you will be using the peer functions
• method='review_essay' - this is the type of peer ranking to be used, as of now, the only other option is 'rank' and this is used by default
• a=13707; $q=59153 - this is the assignment and question ID of the question you want the students to see their answer for.
• essay=&GET_ESSAYS($a,$q,$username) - is the function that gets the needed answer
• <EQN $$essay[0]> - this where the returned answer will go. Notice that this can be formated as though it were normal HTML text. In the above example, the answer is returned in quotes and in italics.
• Also note that the end of the above question includes a normal essay question. This second essay question could be ranked again (as though it were a normal essay question), or it could just be graded by the instructor.

rjallain@unity.ncsu.edu