Can Students ‘Game the System’ in EdReady?
August 22, 2014 at 10:47 am #675
Can Students “game the System” with EdReady?
I am the Developmental Math Coach at Camden County College in New Jersey. I starting using EdReady in June to help our incoming students improve their math skills with the hope that they could than raise their Accuplacer scores and begin college at a higher math level. We have had many students work in EdReady over the summer, both in our lab and at home. I began to notice that some students who reached their Target did not test any better on the Accuplacer. I wondered why this would happen.
I looked at the detailed information for these students and saw many instances of them continually retaking both topic and unit assessments. I also looked to see the ‘Time Spent Studying’ vs ‘Time using EdReady’. Often these students who did not improve their placement scores had very little time spent Studying. The testing info showed that they took some assessments multiple times with a short time frame. This looks to me like they are ‘Gaming the System’.
I think that because there are:
1) limited questions in the test bank
2) Some topic assessments have only 1-4 questions
3) Students can retake assessments unlimited times
4) Nothing re-directs students to the resources after they are unsuccessful on an assessment
Students can just retest and until they happen to select the right answers and then can hit their target without improving their math skills.
To combat this I have started to warn students about testing without studying. I would like to suggest an option in EdReady to put the student into the Resources area right after any unsuccessful assessment. This would put the student where they need to be and certainly remind them that they need to do more work on the given topic.
I welcome comments from anyone on this. I would be especially interested to know if other users have seen similar behaviors from their students.
August 23, 2014 at 9:13 am #679
Randi, thanks for your observations and potential fixes. I know that NROC has prioritized developing and deploying more questions, and this is part of the solution, but I think your additional suggestions would address the issue in a different way.
I recently encountered a different way to “game the system,” but it didn’t involve students just clicking through. It was due to my misunderstanding the process by which students could increase their score. We gave a student an exit exam that was an “expedited scope assessment” in EdReady. He was below the Target Score. He raised his score exactly to the target by (correctly, as per my instructions) following the study path sequentially, from the easiest topics to the hardest. As a result, he raised his score to the target (which we had set equivalent to 80% mastery) without ever encountering the hardest material: He mastered enough of the pre-algebra and basic algebra topics without having to master any of the intermediate algebra units that actually are most essential to his success in College Algebra. I think there are ways to design our EdReady assessment that prevent this without the other fixes that you suggest. Our experience with that one student singles out a different, albeit inadvertent, way to “game the system.” Thoughts?
August 27, 2014 at 7:37 am #691
Thanks for your reply. I have also seen the same issue as you did with students hitting target score and being able to skip the last few important units. I also set my target to 80%. Since I noticed this I have looked at the student’s individual progress and suggested they complete additional units, even when they reach the target score. Soon I hope to solve this my creating more focused assessments with a higher target score. I will probably set the target at 90 or 95%. I seemed to me like a target of 80% would work, but this but allows students to miss a lot of important topics and units.
By the way, my students are community college students trying to test into college level math. What is your student population?
Keep in touch.
October 20, 2014 at 11:41 am #788
Randi, I teach developmental algebra to mostly freshmen at a 4-year university – Jacksonville State U in Alabama. Although I LUV EdReady and the intent with which it was created (to offer under-prepared students an inexpensive way to fill gaps in their understanding and shorten their path to a degree), our graduate assistants and I provide a human touch in the classroom that computer-based instruction cannot. We can also help students learn to study who come to us without knowing how to truly study – to wrestle intellectually (and emotionally) with a topic and gradually, through fits and starts and what may feel like chaotic progress – to build understanding that is always partial, always “under construction.” This is an entirely different skill than learning algebra, although learning algebra provides an excellent occasion to fall down many times – intellectually – and pick yourself up one additional time. Although we may be able to “strongly direct” students to use the resources before retaking an assessment, if a student hasn’t experienced what it “feels like” to “understand,” they will be dependent on the assessment to tell them if they “understand,” instead of learning to be the judge of their own understanding. I don’t see how EdReady can ever teach that skill: we must provide it in person.
August 28, 2014 at 11:18 am #703
Great question Randi, and thanks for the conversation all.
You are absolutely correct that there are some ways that students can “game the system”, as you say. I think that Randi did a good job capturing the specific elements to EdReady’s current design that contribute to this issue.
To briefly confirm what others have said, we are definitely expanding the question pool. The questions for this first expansion (coming soon, I hope!) will not be substantially different from those in the existing pool, but there will be many more of them. We are laying plans to also expand the diversity of questions (including questions similar in form and expectations to the Common Core) and ensure that there are more questions that are just about impossible to guess correctly.
It helps to remember that EdReady itself is not designed to actually replace a high-stakes test, as opposed to preparing for such tests (or other expectations). So students who “game the system” are just gaming themselves. Getting to the target without actually filling their knowledge gaps is counter-productive to EdReady’s purpose. Randi says that she has started warning students that simply raising their scores without gaining improved understanding is not the point, which is a good approach.
Of course, it’s hard for students who already struggle with this stuff to self-regulate their behavior. So we agree that we need to enhance a number of aspects to the handling of the questions themselves, and the subsequent referrals to resources for study. One option for sure is to force students who do not master a topic-specific assessment into the resources frame. Our challenge is that there are always a variety of unintended consequences to these types of changes that might make EdReady less effective for other students.
Please continue sharing your excellent suggestions in this community. As plans for specific enhancements firm up, we will ask for additional feedback here as well. And look ahead for us to roll out certain enhancements along these lines as we can.
October 20, 2014 at 11:52 am #789
Why would students <span style=”text-decoration: underline;”>want</span> to game the system? Unless we answer that question, we may not only design interventions that have unintended consequences, they may also be ineffective. Students are growing up in schools, and a country, that reifies passing “the test” as the Holy Grail of learning. I doubt that powerful conditioning can be overcome by changing EdReady, although I am continually awed by your team’s willingness to take on daunting challenges and emerge on top. How do we inspire students to reach for “the thrill of victory” and risk “the agony of defeat” (to quote an old, very old, commercial for the Olympic Games) without them seeing us – in person – take on the same risks and fall down 100 times only to get back up 101? And how can we be there to reach out a hand to help them up off the floor? This may be more than even EdReady can accomplish, although I will, as always, love watching you try!
October 16, 2014 at 8:21 am #783
Dealing with younger students, we also have concerns about students “gaming the system” especially since we are using EdReady to prepare students for their next level of courses, not for a test. Since improvement in EdReady scores is our only metric for seeing if students are increasing their mastery of topics, some way to force students to complete resources before testing on topics or units would be very helpful. Perhaps this could be an option when setting up a goal?
You must be logged in to reply to this topic.