Don’t panic. Remember, a strong correlation is a product of both the reliability of the test and the reliability of the outcome measure. Variables associated with either one can affect your correlation. There are several things you can do that may increase your validity coefficient given that there is, in fact, a relationship between your test and performance in your reading courses.
Tip 1:
Test students during the first two weeks of class rather than using the scores from the placement test given over a summer or long time period. The longer the time between the test and the outcome measure, the more intervening variables there are to mess up the results.
Tip 2:
If you test students in your reading classes, try excluding continuing students from your sample and just use new students. New students are the group that you will eventually be testing anyway and continuing students may learn at different rates than new students based on previous college experience. This can do bad things to correlations.
Tip 3:
If your reading classes are Credit/No Credit, see if you can get your instructors to convert them to letter grades that you can translate into numbers like A = 5, B = 4, C = 3, etc.
Tip 4:
Try excluding all grades that in your reading program are not true indicators of performance in the class. F and W grades are often given for reasons that are a mixture of performance and personal variables and tend to dilute a correlation that is looking to relate a test to performance in your reading class.
Tip 5:
If excluding W’s and F’s reduces your N too much or somehow doesn’t seem quite right to you (feelings are important in research too), try to get your instructors to issue midterm performance grades. This will reduce the number of W grades that you will have, and will allow an F grade to become a much clearer performance grade. It will also expand the range of scores you will have for your outcome measures when the F’s are included. This approach has a good chance of increasing your correlation.
Tip 6:
If your instructors are hopelessly different in the way they grade, you may, in desperation, want to have the department chairperson select several faculty that grade consistently and in accordance with the course outline of record and run your correlations with those instructors only. If you get a higher correlation doing this, you will at least have established that the placement test can relate to performance measures in reading and then lobby with the department to adopt more uniform grading practices so that down the road you can establish your correlation with more instructors involved. At least you will know that you won’t solve your correlation problem by just using a different test.
Tip 7:
If you are placing students across a set of reading courses arranged in a sequence and the students are already placed in those courses by some assessment process, you can correct for restriction of range. You basically need the correlation of the placement test with the outcome measure for each course level, the standard deviation of the placement test scores for each course, and the standard deviation for all the placement test scores. If you put them in the following formula for each course, you will get an adjusted correlation for each course. Don’t expect miracles, however. If your correlation is already in the high 20’s or low 30’s, you may get closer to .35.
Tip 8:
Nothing works. Try again another semester. It is amazing how much difference there is between one semester and another. Or consider the possibility that the test does not work for your population.
Tip 9:
Watch for minutes from the Chancellor’s Office Assessment Group. There may be some movement away from the .35 and a greater emphasis on content comparisons, and placement satisfaction by students and faculty.
Tip 10:
One reminder. If you do 20 statistical tests, one is apt to be significant by chance at the .05 level.
