PDA

View Full Version : Tying teacher pay to test performance...



Doodler
2006-Jan-13, 12:48 PM
http://www.cnn.com/2006/EDUCATION/01/12/houston.teacher.pay.ap/index.html

If ever there were an insidious side of No Child Left Behind, here it is. Teachers reduced from educators to mercenary memorization coaches. Bad day for the kids, it is.

Moose
2006-Jan-13, 12:58 PM
Yup. And of course you've got the kids who will realize this and intentionally tank their tests (like they care) to lower the average on strict teachers.

[Edit: It should be noted that this thread brushes very close to the prohibition against partisan politics, but I'm not sure it's quite crossed it. Extra care should be taken when posting.]

gethen
2006-Jan-13, 02:10 PM
My neighbor, who teaches 2nd graders, says she's already spending way too much time preparing kids for tests instead of teaching them how to figure out difficult words while reading and other skills that they must have, but are not measured on standardized tests. She's livid that more tests are in the future.
Unfortunately, in most school systems, there is no efficient mechanism for removing a bad teacher who is protected by union membership. My own opinion is that if teachers don't want to deal with tests of their teaching skills, they would be well advised to be much more aggressive about weeding out the bad teachers in their ranks--and I think a bad teacher is probably recognized pretty quickly by his/her peers.

Jim
2006-Jan-13, 02:28 PM
HISD teachers are not happy with this plan for the reason gethen mentions in the first paragraph. They will not be rewarded for how well they teach, but how well they prepare kids to take a test. The two are not even close to the same.

Some years back, when I was teaching in an operator apprentice program at a community college, we had an educator speak to us about preparing meaningful tests. He had each of us take out a sheet of paper and number it 1-20. Then he had us go back and place T or F next to each number. Then he called out the "right" answers.

The average score was something like 80... and there were no questions!

Taking any multiple choice test comes down to a simple method.
1. If you know the right answer, choose it.
2. If you don't, but you spot any wrong answers, drop them.
3. From the remaining answers, guess.
There is no teaching involved, but following this method will get you a passing grade almost every time.

The problem with identifying "bad teachers" is that too many people equate that with how many kids pass the course. (And give or take additional points based on how their kid does.) Standardized tests are supposed to get around this, but when they become the only or primary yardstick for measuring "success"... Well, teachers ain't stoopid. "Teach to the Test."

mickal555
2006-Jan-13, 02:57 PM
My geography teacher was probobly the worst teacher ever.

All we did was watch video's... and then when the test came she'd give us 40 questions and answers tell us to memorise them, and 30 will be on the actual test so if we learn all of this so-called revision we'd get a good mark... and most people did.

I refused, and completly didn't do 1 (major) assignment, and bombed another because the task was so poorly writen... I still managed a S(Sound- C) average, I used to get VHA's(very high acheavment- A) in this subject(the year before different teacher).

Ridickulas.

SeanF
2006-Jan-13, 03:19 PM
Some years back, when I was teaching in an operator apprentice program at a community college, we had an educator speak to us about preparing meaningful tests. He had each of us take out a sheet of paper and number it 1-20. Then he had us go back and place T or F next to each number. Then he called out the "right" answers.

The average score was something like 80... and there were no questions!
You don't suppose students were changing their answers while the teacher called out the "right" answers, do you?

Unless your class were composed of legitimate psychics, the average score would be around 50.

Jim
2006-Jan-13, 04:20 PM
You don't suppose students were changing their answers while the teacher called out the "right" answers, do you?

Unless your class were composed of legitimate psychics, the average score would be around 50.

No, these weren't students trying for a grade; these were instructors learning about making good tests. They had more reason for the results to be valid than to get a good score.

And, yeah, you'd expect 50... but that's not how it worked out. According to the presenter, the results were/are replicatable. (He probably gave an explanation, but that was years... and years ago.)

Demigrog
2006-Jan-13, 04:28 PM
Taking any multiple choice test comes down to a simple method.
1. If you know the right answer, choose it.
2. If you don't, but you spot any wrong answers, drop them.
3. From the remaining answers, guess.
There is no teaching involved, but following this method will get you a passing grade almost every time.

A properly graded multiple choice test takes guessing into account--the SAT, for example, deducts fractional points for wrong answers. Statistically, students score roughly equally by not answering than they do by guessing. Of course, not many teachers do this on their own tests.



The problem with identifying "bad teachers" is that too many people equate that with how many kids pass the course. (And give or take additional points based on how their kid does.) Standardized tests are supposed to get around this, but when they become the only or primary yardstick for measuring "success"... Well, teachers ain't stoopid. "Teach to the Test."
Better than no yardstick, unfortunately. It is not like colleges and interviewers rely solely on standardized test scores, but the scores are about the only way to quantitatively compare students between schools (or even within a school, oftentimes).

Does anyone really dispute that a school's standardized test results are valid for comparing the actual skill level of its students? The real issue here is whether the test scores accurately reflect the ability of the school and teachers to teach. Clearly some schools have seriously different circumstances, everything from problems in home life to disruptive students to outright violence in school. A 1:1 comparison of test scores between schools (and even individual students) is clearly an absurd measure of teaching quality.

Standardized test results have legitimate uses, IMO, coupled with intimate knowledge of the school. A principal should be able to identify factors that are keeping their students from reaching average scores, and alter the local curriculum, programs, and policies to correct them. Needless to say, local and state school boards need to give the principles and teachers the authority and resources they need to make these changes; many do not. An individual teacher or principle would then be evaluated on their ability to identify student needs and their ability to fulfill them, as measured by improvements in the student’s test scores over a single year. The evaluation would be complicated, of course, and require actual people and not a rigid state “standard”.

There are obviously many exceptional teachers that overcome local problems and produce good results where other teachers in the same situation or school do not. We should not have to count on exceptional individuals to produce good results; that is a sign that the system itself is flawed. I could go on for pages about what I think the flaws are, but I won’t; nobody reads giant posts like that, and everyone knows the problems (just not the solutions) anyway. :)

tofu
2006-Jan-13, 05:25 PM
Here's something to think about. There is a country where everyone is concerned about proper diet and fighting obesity. So the government sets up free cafeterias all over the country, funded of course by tax dollars. Most people get all their food from these government cafeterias. You can "opt out" if you want, but since it's free few people do that. The result is that there aren't enough customers for ordinary restaurants, so most of those restaurants close. No more Olive Garden. No more Arby's. Some restaurants stay open by charging a lot of money to make up for the low number of customers. Since they have to charge a lot, few people can afford to go to them.

The problem with the government cafeterias though, is that there is a huge government bureaucracy behind them. They end up spending like $200 per person per meal, and the food is so bad that people don't want to eat it anyway. There are actually people walking around in this country who look like holocaust survivors! People regularly collapse in the street from their hunger. But what else can they do? They can't afford to go to the few remaining restaurants.

There is much discussion in this country about this problem. Many people argue that, if we'd only pay the cooks more money, this wouldn't happen! Other people say that we should just make the cafeterias do more paperwork to show that they are feeding everyone!

And of course, some people argue that you should do away with the government cafeterias altogether. But those people are obviously crazy. I mean, think about it. How could any country survive without free government cafeterias? It isn't possible to feed people for less than $200 per meal. And besides, if the private sector could open their own restaurants, you'd have restaurants like McDonalds and people would get fat! Omg! That's unhealthy! That's way worse that the current situation where people starve!

SeanF
2006-Jan-13, 05:52 PM
No, these weren't students trying for a grade; these were instructors learning about making good tests. They had more reason for the results to be valid than to get a good score.

And, yeah, you'd expect 50... but that's not how it worked out. According to the presenter, the results were/are replicatable. (He probably gave an explanation, but that was years... and years ago.)
The only possible way I can see it being replicatable is if people are, for whatever psychological reason, more likely to put a "T" next to "1" and a "F" next to "2" or something like that, and the teacher took that into account when determining the "right" answers. But that's not going to have anything to do with tests, because students taking actual tests look at a lot more than just the number of the question.

Also, most standardized tests are multiple choice, not simply T-F, which would make it even harder to get a passing grade by guessing.

I guess I'm just not convinced the experiment your class conducted was really indicative of anything meaningful. :neutral:

skepticfrog
2006-Jan-13, 07:34 PM
Oh, so many things to comment on. I think Demigrog is right -- standardized tests do have a place. Anyone who has ever had to do anything with admissions at either an undergraduate or graduate level has to bring scores in at some point. The problem I have with NCLB and the way it is implemented in many states is that the only thing considered are the tests. Many of the testing companies themselves do not believe that any test is sufficient to give you all of the performance information you need.

The second issue is teacher pay tied to performance on tests. Outside of encouraging teaching to the test, there are additional problems with any such program. First, the best prediction of how students do on any one test is how they did on previous tests (if you did bad on the 8th grade test, chances are you will do bad on the 10th grade test). Teachers with high percentages of low performing students will do worse on the test. One way some districts handle this is to evaluate "improvement." This, however, is usually done as improvement of a teacher's average score year to year. Rarely is improvement tied to the student.

<warning, pure anecdotal evidence ahead> A friend of mine received a substandard performance review after many of his students did poorly on a state standardized test while other teachers showed improvements over past years. This individual did a little research on his students and found that, on average, his students had done much worse on previous test than the students of other history teachers. In fact, his students had shown more improvement, on average, than students of other teachers.

Finally, on unions. Not all states have strong union protection for teachers. In the two states where my wife has taught, Virginia and Texas, union protection is not very good. Particularly in Texas, in some districts it is quite easy to punish teachers. So easy, in fact, (another anecdote) that one of our friends almost lost his job over a disagreement with an incompetent principal. Only a former student of his who was a lawyer helped save his job; even still, he lost some of his previous perks. In some states, unions probably are too powerful and protect bad teachers. In others, unions are not powerful enough and good teachers receive unfair treatment.

aurora
2006-Jan-13, 10:26 PM
Students that have learning disabilities, or have problems with written materials... do schools try to get rid of them so that they don't bring down their test scores?

snarkophilus
2006-Jan-13, 10:30 PM
You don't suppose students were changing their answers while the teacher called out the "right" answers, do you?

Unless your class were composed of legitimate psychics, the average score would be around 50.

Nope, because when you're composing or taking a T/F test, you expect that there will be a roughly equal number of true and false answers. You also expect that there won't be long streaks of the same answer. No one is going to guess "true" every time.

It makes some sense that the number would be higher than 50, but 80 seems really high. We'll have to try it out!

Gillianren
2006-Jan-14, 12:52 AM
There is a country where everyone is concerned about proper diet and fighting obesity.

Wouldn't that logically be malnutrition? I mean, if the solution was government cafeterias. (I think your analogy is wrong in a lot of other ways--the private school I could've attended wouldn't have met my needs as well as the publics schools I did attend--but that's a little jarring.)

Jim
2006-Jan-14, 01:53 AM
Students that have learning disabilities, or have problems with written materials... do schools try to get rid of them so that they don't bring down their test scores?

Yes. It has happened several times in Texas. "Certain" students were told to stay home the day of the test. Of course, this is against official policy and folks have lost their jobs for doing it... but they did it.

The more you tie someone's livelihood to one and only one meassure, the greater the incentive to cheat.


A friend of mine received a substandard performance review after many of his students did poorly on a state standardized test while other teachers showed improvements over past years. This individual did a little research on his students and found that, on average, his students had done much worse on previous test than the students of other history teachers. In fact, his students had shown more improvement, on average, than students of other teachers.

This is a very real possibility if the only measure is a test score. Consider also that some people are just not good at taking tests; as a result, a test - just about any objective test - is not a fair measure of what they have learned.

jrkeller
2006-Jan-14, 03:03 AM
Students that have learning disabilities, or have problems with written materials... do schools try to get rid of them so that they don't bring down their test scores?

Not at all, they can be exempted from taking the test.

SeanF
2006-Jan-14, 03:52 AM
Nope, because when you're composing or taking a T/F test, you expect that there will be a roughly equal number of true and false answers. You also expect that there won't be long streaks of the same answer. No one is going to guess "true" every time.

It makes some sense that the number would be higher than 50, but 80 seems really high. We'll have to try it out!
Meaningless. Flipping a coin a hundred times will give you about half heads and half tails, and you won't get long streaks of the same answer. Regardless, if you try and guess what's coming next, you're only going to be right about half the time.

HenrikOlsen
2006-Jan-14, 04:17 AM
You will get long streaks, in 100 you can expect about 1.5 runs of 7 equal in a row.

montebianco
2006-Jan-14, 04:54 AM
Meaningless. Flipping a coin a hundred times will give you about half heads and half tails, and you won't get long streaks of the same answer. Regardless, if you try and guess what's coming next, you're only going to be right about half the time.

If each question is truly an independent trial, then the probability of one person getting a score of 80% or better is approximately 0.0059. Making it a two-sided test, the probability of getting 80% or better or 20% or worse is 0.0118. With multiple people, it gets worse. With 10 people, an average score of 80% is an outcome that is about 8.5 standard deviations above what is expected; with 25 people, it is 13.4 standard deviations above the expected value. Both of those are extremely improbable events. In the case with 10 people, both MS Excel and Mathematica simply tell me the probability of an 8.5 standard deviation event (assuming a normal distribution, which is an approximation, but a very accurate one) is zero; they won't calculate enough decimal places to report something different than zero. By other means, I calculated the exact probability in the 10 person case, and it is about 0.00000000000000000169 (double it for two-sided, i.e., >=80% or <=20%). In the 25 person case, it is about 0.0000000000000000000000000000000000000000000830 (probability of >=80%; double to include <=20% as well). Not real likely events.

So, the choices are:

(1) Somewhere enroute from the event to our learning of it, the story has been embellished.

(2) The questions were not equal probability random trials, i.e., either they weren't chosen randomly, or the students had some hint or other way of telling what the "correct" answers were.

(3) Something far less likely than winning the lottery or being struck by lightning happened.

Each reader can decide, which of these is the most likely explanation...

If a teacher can teach someone to predict an intrinsically non-predictable equal probability event with 0.8 probability, I'd say that's quite a remarkable feat...

Edit - I just realized that even if the "correct" answers are truly random, the students' answers may be correlated with each other - Swift and I in the other thread have perfect correlation between our answers. This wouldn't change the expected value, but would change the probabilities of deviations from the expected value. I see no way around the 0.0059 probability of >=80% correct if the answers are truly random with equal probability, are independent of each other, and if the students get no "help"...

Gillianren
2006-Jan-14, 05:13 AM
Not at all, they can be exempted from taking the test.

Only a certain number per school. If your school just happens, as mine was, to be the one with a big Special Ed class . . . .

Enzp
2006-Jan-14, 05:28 AM
Not only special ed classes, but the community it lives in as well. We have an area school that perennially is criticized for poor performance. Politicians bluster, but no one wants to acknowledge that this school also sits in an area where a large portion of the kids live in homes of poverty and have single parents. Furthermore this is an area where drugs are prevalent. When a 9 year old goes home to find his mom drunk, high on crack, or gone altogether, he is bound to do less well than his friends in the upscale suburbs.

It is not the teacher's fault that junior doesn't get fed regularly, has drugged out people stumbling through is home, and so on. The authorities threaten to take over this school, as if that would change things.

There is more to this question than kids taking tests.

Klausnh
2006-Jan-14, 06:31 AM
My wife's a 5th grade teacher. She bonds well with the slower students and they are more open to learn from her. Because of her skills, every year she gets more of the slower and needy students. Another teacher does well with the smarter students, so she gets more of them. Obviously, my wife's students are going to do poorer on tests, even though she is a good teacher. How are tests a fair way to determine her teaching skills.
As any teacher knows, each year's classes are different. Some years most of the students are open to learning and make teaching a joy. Other years, the classes are made up of a lot of emotionally needy students. So if you use tests, does that mean one year the teachers are doing a good job and the next year they're not?
Each person has their own learning style. If a teacher's teaching style matches a student's learning style, the student will excel. Is the teacher incompetent if the styles don't match and the student does poorly in her class?
Without evidence that incompetent teachers are the reason for a poor education system, I don't see the need for testing teachers. If public education needs to be improved, use science to determine where the problem is, and fix the problem and don't just assume it's the teachers.


Unfortunately, in most school systems, there is no efficient mechanism for removing a bad teacher who is protected by union membership.
Not true in NH. An incompetent teacher can be fired or not have her contract renewed.

galacsi
2006-Jan-14, 10:08 AM
Ha ha When i was young i did not have a single test in all my school years.

But now tests are catching up and in my enterprise we have several each year.
About security , company policy , commercial practices , FDA reglementation and so on. It is all a big joke , nobody care to really learn the hundred of pages of data delivered with the tests. If you are not to dumb , you just find the fast lane in the course ,skip all the boring steps and finnaly rush to the test itself. Then with a combinaison of common sense, guessing , eliminating and sheer luck it s a shame if you cannot get the minimum notation !

Enzp
2006-Jan-14, 02:08 PM
It is also a shame when someone doesn't take his job any more seriously than that, and then boasting of it.

galacsi
2006-Jan-14, 03:55 PM
It is also a shame when someone doesn't take his job any more seriously than that, and then boasting of it.

Thank you for the lesson ! But it is not my job it is pure bureaucratic stupidity ! This is the common opinion not mine.

jrkeller
2006-Jan-14, 05:32 PM
Thank you for the lesson ! But it is not my job it is pure bureaucratic stupidity ! This is the common opinion not mine.

We have the same bureaucratic nonsense here too. I would say that each year, I must take a dozen classes related to ISO9000, diversity, safety, sexual harassment, etc, and they are almost the same year after year after year after year ..... and then have a test on the subject. Basically, the class (sometimes online) covers the test. All required by the federal government, because I work for a government contractor. I would say that once every four years would be good enough for me.

jrkeller
2006-Jan-14, 05:38 PM
Only a certain number per school. If your school just happens, as mine was, to be the one with a big Special Ed class . . . .

That's one thing that has changed recently. In my kids school district, each school has a Special Ed class and the kids aren't housed all in one or two schools.

I also think it is a good idea that these kids are tested, so that the school district can now where they are failing these kids as well. In fact, I've seen a really explosion of the number of Special Ed children attending regular Ed classes, instead of the usual segregated classrooms and the only time both classes get together might be a lunch.

Gillianren
2006-Jan-14, 07:17 PM
That's one thing that has changed recently. In my kids school district, each school has a Special Ed class and the kids aren't housed all in one or two schools.

I also think it is a good idea that these kids are tested, so that the school district can now where they are failing these kids as well. In fact, I've seen a really explosion of the number of Special Ed children attending regular Ed classes, instead of the usual segregated classrooms and the only time both classes get together might be a lunch.

It's fine for the kids to get tested, but the simple fact is, a lot of the Special Ed kids from my school were not going to do well on those tests. A lot of them had severe congenital retardation, and weren't like my sister, who's merely severely dyslexic. (She did some Special Ed classes as well, but largely regular classes, and no Special Ed classes at all until high school.) The higher-functional kids took the odd class with the rest of us (there were two Special Ed kids in my California History class; neither could talk and both were in wheelchairs), but a lot of those kids were doing the best they could to get up to a second-grade education.

Under NCLB, the testing scores of our school must've tanked, given that my entire graduating class was 120, and probably half-a-dozen of those were Special Ed--at least. Their teachers were great people who were doing the best they could, but I'd be surprised if some of those kids were even capable of high school-level testing. Ever. No matter how much teaching they had.

sarongsong
2006-Jan-15, 02:15 AM
Too bad the students themselves---who may be in the best position to judge a teacher's merit---don't have any input in rewarding or removing teachers.
Jan 13, 2006 (http://www.waff.com/Global/story.asp?S=4362064&nav=0hBE)
"...Alabama tested a pay for performance system in the 1980's. "Principals evaluated teachers, and it included testing and student achievement,"...After one year, the legislature abolished the merit pay plan. They said it was unfair and simply didn't work..."

montebianco
2006-Jan-15, 02:20 AM
Too bad the students themselves---who may be in the best position to judge a teacher's merit---don't have any input in rewarding or removing teachers.

They may very well be in the best position, but they may also not have the best incentives. Which is often where these sorts of problems arise, where the people with the best knowledge/information are not the same people who have the best incentives...