[98299] in tlhIngan-Hol
Re: [Tlhingan-hol] Certification Test Woes
daemon@ATHENA.MIT.EDU (d'Armond Speers, Ph.D.)
Sun Mar 30 18:48:11 2014
In-Reply-To: <5337E85B.6030008@gmx.de>
Date: Sun, 30 Mar 2014 16:47:43 -0600
From: "d'Armond Speers, Ph.D." <speersd@georgetown.edu>
To: tlhIngan-Hol List <tlhingan-hol@kli.org>
Errors-To: tlhingan-hol-bounces@kli.org
--===============5631875566496730286==
Content-Type: multipart/alternative; boundary=001a11c1bdee14bb6104f5dab9e8
--001a11c1bdee14bb6104f5dab9e8
Content-Type: text/plain; charset=ISO-8859-1
I have tried to be consistent, fair and permissive in grading of KLCP
tests. The goal is not to make someone discouraged by grading
hyper-critically, and to allow room for creativity (language is not a 1:1
thing).
For each question in the test bank there is an explanation of the purpose
of the question, what is being tested by the question. These are printed
in the answer keys with the expected answer(s), which the proctor uses when
grading. When I'm grading tests, I am specifically looking for mistakes
that pertain to the purpose of the question. If someone answers the
question correctly but is not strictly adhering to the vocabulary or
grammar of their test level, I do not count that as an error. (I would not
count off if someone used {-'e'} in an answer in the level 1 test.) If
there are vocabulary or grammatical errors that do not pertain directly to
the purpose of the question I count off a single point rather than marking
the whole question wrong. If they completely miss the point of the
question, or the number of errors exceeds the available points per question
(5 points per question for levels 1 & 2), then I just count the question
wrong. (If someone is making that many errors per question, then they
really aren't at that level yet.)
Whenever I distribute tests to someone else to administer and grade, I
provide these instructions. With multiple test administrators over a
period of many years, it's easy to see how these guidelines may not be
followed consistently. I have not personally attended a qep'a' or qepHom
in several years, or administered tests, though I do still regularly
generate new tests from the test banks for qep'a'mey and qepHommey. It
sounds like we may need to get some written guidelines for test
administrators and strive for greater consistency in grading.
Creation of the KLCP was a group effort. I designed the structure with
Lawrence (3 levels with 3 pins, because of the cultural importance of the
numeral 3). There are 100 questions for each level in the test bank (plus
the reading comprehension questions for Level 3), and generating tests
means selecting questions from the test bank at random. Level 1 is a
sub-set of TKD; level 2 is all of TKD; and level 3 is open-season on all
available materials (mainly including additional materials from KGT). We
produced written guidelines for each level, describing the vocabulary and
grammar that was in scope. Questions were written to be distributed evenly
across the topics for each level. Each question in the test bank was
reviewed by multiple highly-skilled speakers when we were creating the
program, following the written guidelines for the level. I have all of
this in a MS Access database, which allows me to manage the randomization
and generation of tests and keys quickly and easily. We put a ton of
thought and effort into it, to make it as fair, correct and relevant as we
could. And ultimately, it is intended to encourage people to achieve
certification; it is supposed to be a positive experience.
All of that being said, we now have the benefit of many years of
experience, which we didn't have when we were creating the program. And if
it's not achieving its goals of encouraging learning and rewarding
progress, then we should be open to that feedback and willing to make
adjustments. The challenge is that the more significant the adjustments
(severely limiting the vocabulary for level 1, for example), the more
effort will be involved in re-developing the test bank. If we make Level 1
too simple, then the gap between Level 1 and Level 2 becomes huge. And to
be honest, I would not be able to do this re-development work, so someone
else would have to take it over completely. Taking out the questions about
the suffix number would be easier, and I agree that it's probably too
pedantic for the level 1 test. These questions would have to be replaced
with something else, maintaining balance across the topics for the level.
I'm not sure how many questions there are like this in Level 1, but if
anyone has interest in writing some new replacement questions, I'd be happy
to work with them on improving the test bank.
Another thing to consider is that if we simplify level 1, then people who
worked hard to accomplish that achievement might feel slighted because we
lowered the bar. Not sure if that's a real thing, it just occurred to me.
Sorry for the wall of text, just lots ideas bouncing around.
-- Holtej
--001a11c1bdee14bb6104f5dab9e8
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_extra"><div><div style=3D"font-family:=
arial,sans-serif;font-size:13px">I have tried to be consistent, fair and pe=
rmissive in grading of KLCP tests. =A0The goal is not to make someone disco=
uraged by grading hyper-critically, and to allow room for creativity (langu=
age is not a 1:1 thing).</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><span =
style=3D"font-family:arial,sans-serif;font-size:13px">For each question in =
the test bank there is an explanation of the purpose of the question, what =
is being tested by the question. =A0These are printed in the answer keys wi=
th the expected answer(s), which the proctor uses when grading. =A0When I&#=
39;m grading tests, I am specifically looking for mistakes that pertain to =
the purpose of the question. =A0If someone answers the question correctly b=
ut is not strictly adhering to the vocabulary or grammar of their test leve=
l, I do not count that as an error. =A0(I would not count off if someone us=
ed {-'e'} in an answer in the level 1 test.) =A0If there are vocabu=
lary or grammatical errors that do not pertain directly to the purpose of t=
he question I count off a single point rather than marking the whole questi=
on wrong. =A0If they completely miss the point of the question, or the numb=
er of errors exceeds the available points per question (5 points per questi=
on for levels 1 & 2), then I just count the question wrong. =A0(If some=
one is making that many errors per question, then they really aren't at=
that level yet.)</span><div style=3D"font-family:arial,sans-serif;font-siz=
e:13px">
<br></div><div style=3D"font-family:arial,sans-serif;font-size:13px">Whenev=
er I distribute tests to someone else to administer and grade, I provide th=
ese instructions. =A0With multiple test administrators over a period of man=
y years, it's easy to see how these guidelines may not be followed cons=
istently. =A0I have not personally attended a qep'a' or qepHom in s=
everal years, or administered tests, though I do still regularly generate n=
ew tests from the test banks for qep'a'mey and qepHommey. =A0It sou=
nds like we may need to get some written guidelines for test administrators=
and strive for greater consistency in grading.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">Creation of the KLCP w=
as a group effort. =A0I designed the structure with Lawrence (3 levels with=
3 pins, because of the cultural importance of the numeral 3). =A0There are=
100 questions for each level in the test bank (plus the reading comprehens=
ion questions for Level 3), and generating tests means selecting questions =
from the test bank at random. =A0Level 1 is a sub-set of TKD; level 2 is al=
l of TKD; and level 3 is open-season on all available materials (mainly inc=
luding additional materials from KGT). =A0We produced written guidelines fo=
r each level, describing the vocabulary and grammar that was in scope. =A0Q=
uestions were written to be distributed evenly across the topics for each l=
evel. =A0Each question in the test bank was reviewed by multiple highly-ski=
lled speakers when we were creating the program, following the written guid=
elines for the level. =A0I have all of this in a MS Access database, which =
allows me to manage the randomization and generation of tests and keys quic=
kly and easily. =A0We put a ton of thought and effort into it, to make it a=
s fair, correct and relevant as we could. =A0And ultimately, it is intended=
to encourage people to achieve certification; it is supposed to be a posit=
ive experience.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">All of that being said=
, we now have the benefit of many years of experience, which we didn't =
have when we were creating the program. =A0And if it's not achieving it=
s goals of encouraging learning and rewarding progress, then we should be o=
pen to that feedback and willing to make adjustments. =A0The challenge is t=
hat the more significant the adjustments (severely limiting the vocabulary =
for level 1, for example), the more effort will be involved in re-developin=
g the test bank. =A0If we make Level 1 too simple, then the gap between Lev=
el 1 and Level 2 becomes huge. =A0And to be honest, I would not be able to =
do this re-development work, so someone else would have to take it over com=
pletely. =A0Taking out the questions about the suffix number would be easie=
r, and I agree that it's probably too pedantic for the level 1 test. =
=A0These questions would have to be replaced with something else, maintaini=
ng balance across the topics for the level. =A0I'm not sure how many qu=
estions there are like this in Level 1, but if anyone has interest in writi=
ng some new replacement questions, I'd be happy to work with them on im=
proving the test bank.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">Another thing to consi=
der is that if we simplify level 1, then people who worked hard to accompli=
sh that achievement might feel slighted because we lowered the bar. =A0Not =
sure if that's a real thing, it just occurred to me.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">Sorry for the wall of =
text, just lots ideas bouncing around.</div></div><div style=3D"font-family=
:arial,sans-serif;font-size:13px">
<br></div><div style=3D"font-family:arial,sans-serif;font-size:13px">-- Hol=
tej</div><div style=3D"font-family:arial,sans-serif;font-size:13px"><br></d=
iv></div></div>
--001a11c1bdee14bb6104f5dab9e8--
--===============5631875566496730286==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Tlhingan-hol mailing list
Tlhingan-hol@kli.org
http://mail.kli.org/mailman/listinfo/tlhingan-hol
--===============5631875566496730286==--
