It’s that time of year again where many parents of junior and senior high schoolers are concerned about their child’s test scores- the dreaded and scary SAT/ACT.
I’d like to offer some perspective. As most of you have heard, many, many schools have moved to some version of test optional. And this shift has caused some confusion and uncertainty as any change often does. But I think you can relax. This is a good thing.
Here’s my perspective including a brief lesson on psychometric theory.
First some background on my personal skills in the area of writing. (You’ll understand in a bit) A large portion of my job requirements, my day to day, requires writing. First, there are the emails, countless emails, to a variety of audiences including colleagues, supervisors, parents, students, and other professionals. Next there are the reports where the results of my evaluations are explained, interpreted, and summarized. These reports are to be read by a wide range of audiences with a wide range of reading abilities. It can be a particularly tricky task. The reports require descriptive writing in portions and analysis and interpretation in others.
In addition to all the writing I do for work, I also write for fun. I like to write about my observations about two subjects I know a thing or two about: life and education. So one might say I do a lot of writing, and I think I’m pretty good at it. I use varied sentence beginnings and structures, every word is carefully chosen, I know when to use a semicolon, I love my Oxford comma, and I know the difference between effect and affect. Yes, I’d say I’m a pretty good writer based on all measures.
One day, I snatched a practice copy of the English portion of the ACT and decided to give it go. Here’s what happened.
There are 75 questions on the English subtest that must be completed within 45 minutes. (That is 36 seconds per question).
The results? Going in cold? I scored a 27 out of possible 36. Now no need for condolences and the “that’s a great score for a start.” Remember, I’m not taking this test to try to improve my score for next time. It was an experiment on validity. I don’t want to know the “tricks” to doing better. My daughter learned those tricks and indeed got a better score than I did. Here’s the question that needs to be asked. Is my 17 year old daughter a better writer than I am? No offense to her but….. C’mon.
Let’s dig deeper. There’s a whole field of study out there called psychometrics.
Psychometrics is the study of psychological measurements including theory and technique. Concepts like standardization, reliability and validity are some of the terms psychometricians study. This is important because here’s the deal; we can make any test any day. It will measure SOMETHING. But if it doesn’t measure what we need it to measure, isn’t it useless?
Back to the SAT/ACT. The first question that needs to be asked is “What construct are you trying to measure?” This is called content validity. For example, some tests measure reading comprehension. Other tests may measure concepts of algebra 2. The English section of the ACT is intended to measure “writing.” (more like revising and editing skills from what I can tell) But does it? The ACT, a test that holds so much weight, rated me, an experienced writer of many decades, as an “ok writer, better than most, but definitely NOT a great writer, and absolutely not an expert writer." According to the test scores, my 17 year old daughter was actually a better writer than me!?!
This is the first flaw of the test. Is it really a good measure of the construct? Are the best writers getting the best scores? Do the medium writers get medium scores? Do the worst writers get the worst scores? If one can apply “test tricks”, is this still a measure of writing ability or does it really measure the application of test tricks?
The ACT/SAT tests should also be asked a second question (so much weight is placed on this answer.) Can a person’s performance on this measure predict their performance on a measure in the future? In this case, the measure is first year college GPA. Can a single test performance accurately predict a first year college GPA? This one is crazy! (It’s called predictive validity) And it’s what most laypeople think is the whole point of the SAT/ACT to begin with. Which is that the student’s performance on this test is an indication of the type of college they should attend. Another words, there’s an assumption that the highest performances on the test show a student will succeed at the most competitive of schools, medium scores are great for medium level schools and so on. A strong predictive validity says high performance on this measure means high performance on that measure: a perfect correlation of 1 (a psychometrician’s dream)
But does it? On its best day the ACT/SAT can predict to a .60 a student’s GPA their first year of college, and that number changes when demographics are broken down. Meaning it can be a pretty lousy predictor for subgroups. Meaning it’s a lousy predictor for women, a worse predictor for people of color, and a terrible predictor for underprivileged kids.
Additionally consider this.
If a student can get $1000 or more worth of private ACT/SAT tutoring and their score goes up dramatically, isn’t that a measure of who can afford tutoring? (I don’t want to take away any of the hard work that students put in on their own. That matters of course- but now we are clearly measuring multiple constructs like motivation, time, executive functioning, and of course, good ol’ access and money)
Let’s go back to my performance on the English portion of the ACT. Remember I had only 36 seconds per question. I have never in my professional or academic life had to revise a paragraph with a clock ticking down. Never. It’s not that I’m slow; it’s that I don’t work that way. Also, there are lots of ways to write something. There is no fixed answer. The task demands were unrealistic and the answer options were “tricky”. This is not real life, real work, or real academia.
ACT/SAT are certainly measures. But are they the best measures? If not the ACT/SAT, then what? Both tests claim that when the scores are merged with GPA, the predictions of first year performance are improved and good (good enough). But maybe, just maybe, the information is already built-in. Even with the enormous amount of subjectivity, high school grades are just as good if not a better predictor for college success. There have been multiple studies conducted to support this theory. Perhaps it’s because other valuable constructs are built into grades. Not only do grades measure performance over time (something a standardized test cannot do), but grades also measure other constructs such as motivation, creativity, leadership, problem-solving, and interpersonal skills. And maybe, just maybe, those constructs matter too, maybe more.
The pandemic has freed colleges big and small, private and public, elite and non-elite, to try something new. I’m thrilled to hear so many colleges are no longer requiring scores (for admittance or scholarship).
Let’s keep it up.
https://www.forbes.com/sites/nickmorrison/2020/01/29/its-gpas-not-standardized-tests-that-predict-college-success/
**There is research to suggest the SAT is the single best predictor of first year GPA, better than grades. This research is done by the College Board. You know, the folks that make the SAT!
“Not everything important is measurable and not everything measurable is important.” (Elliot Eisner)
