[98300] in tlhIngan-Hol
Re: [Tlhingan-hol] Certification Test Woes
daemon@ATHENA.MIT.EDU (Elizabeth Lawrence)
Sun Mar 30 19:43:58 2014
In-Reply-To: <CALN+49ahVhe0HV0Zw6F_fcWctnQ7XxdfqEFb0VnqnDV89FEZOw@mail.gmail.com>
Date: Sun, 30 Mar 2014 19:43:34 -0400
From: Elizabeth Lawrence <elizabeth.lawrence08@gmail.com>
Cc: tlhIngan-Hol List <tlhingan-hol@kli.org>
Errors-To: tlhingan-hol-bounces@kli.org
--===============7490895127268184842==
Content-Type: multipart/alternative; boundary=f46d044286c6cf687604f5db800f
--f46d044286c6cf687604f5db800f
Content-Type: text/plain; charset=ISO-8859-1
I don't think anyone believes we should simplify the level one test.
However, I do think that many people would like a .5 level test, so that
there is a benchmark at which people can say "hey, I know something" before
being able to clear the level one bar.
This need not be as large an undertaking as the extant KLCP tests, though
it would be nice to get it to a similar level. I would say that it should
use perhaps 100 words max, and that they should be specifically chosen to
be words a newbie would encounter or try to use. Certainly, when I first
started actually studying for level 1, there were a lot of words I knew,
and a lot more I had never heard. (SuvwI', vut, and bIQ should be on the
list, for example. 'Ip likely should not.)
We don't have to change the numbering of the tests, simply calling it a
beginner level test would do.
I also don't think that this thread was meant to call for a reorganization
of the testing system. As far as I can tell, Qov merely wanted input on
what caused the most difficulty on the beginner test so she could better
instruct people who want to pass the test as it is.
As far as the suffix questions go, I think the answer to that problem is to
alter the questions to ask for more information, not less. If the question
asked for the definition and classification type as well as classification
number, not knowing the number would only lose people a point or two on the
question, and that would be a truly minor revision to the test.
be''etlh
On Sun, Mar 30, 2014 at 6:47 PM, d'Armond Speers, Ph.D. <
speersd@georgetown.edu> wrote:
> I have tried to be consistent, fair and permissive in grading of KLCP
> tests. The goal is not to make someone discouraged by grading
> hyper-critically, and to allow room for creativity (language is not a 1:1
> thing).
>
> For each question in the test bank there is an explanation of the purpose
> of the question, what is being tested by the question. These are printed
> in the answer keys with the expected answer(s), which the proctor uses when
> grading. When I'm grading tests, I am specifically looking for mistakes
> that pertain to the purpose of the question. If someone answers the
> question correctly but is not strictly adhering to the vocabulary or
> grammar of their test level, I do not count that as an error. (I would not
> count off if someone used {-'e'} in an answer in the level 1 test.) If
> there are vocabulary or grammatical errors that do not pertain directly to
> the purpose of the question I count off a single point rather than marking
> the whole question wrong. If they completely miss the point of the
> question, or the number of errors exceeds the available points per question
> (5 points per question for levels 1 & 2), then I just count the question
> wrong. (If someone is making that many errors per question, then they
> really aren't at that level yet.)
>
> Whenever I distribute tests to someone else to administer and grade, I
> provide these instructions. With multiple test administrators over a
> period of many years, it's easy to see how these guidelines may not be
> followed consistently. I have not personally attended a qep'a' or qepHom
> in several years, or administered tests, though I do still regularly
> generate new tests from the test banks for qep'a'mey and qepHommey. It
> sounds like we may need to get some written guidelines for test
> administrators and strive for greater consistency in grading.
>
> Creation of the KLCP was a group effort. I designed the structure with
> Lawrence (3 levels with 3 pins, because of the cultural importance of the
> numeral 3). There are 100 questions for each level in the test bank (plus
> the reading comprehension questions for Level 3), and generating tests
> means selecting questions from the test bank at random. Level 1 is a
> sub-set of TKD; level 2 is all of TKD; and level 3 is open-season on all
> available materials (mainly including additional materials from KGT). We
> produced written guidelines for each level, describing the vocabulary and
> grammar that was in scope. Questions were written to be distributed evenly
> across the topics for each level. Each question in the test bank was
> reviewed by multiple highly-skilled speakers when we were creating the
> program, following the written guidelines for the level. I have all of
> this in a MS Access database, which allows me to manage the randomization
> and generation of tests and keys quickly and easily. We put a ton of
> thought and effort into it, to make it as fair, correct and relevant as we
> could. And ultimately, it is intended to encourage people to achieve
> certification; it is supposed to be a positive experience.
>
> All of that being said, we now have the benefit of many years of
> experience, which we didn't have when we were creating the program. And if
> it's not achieving its goals of encouraging learning and rewarding
> progress, then we should be open to that feedback and willing to make
> adjustments. The challenge is that the more significant the adjustments
> (severely limiting the vocabulary for level 1, for example), the more
> effort will be involved in re-developing the test bank. If we make Level 1
> too simple, then the gap between Level 1 and Level 2 becomes huge. And to
> be honest, I would not be able to do this re-development work, so someone
> else would have to take it over completely. Taking out the questions about
> the suffix number would be easier, and I agree that it's probably too
> pedantic for the level 1 test. These questions would have to be replaced
> with something else, maintaining balance across the topics for the level.
> I'm not sure how many questions there are like this in Level 1, but if
> anyone has interest in writing some new replacement questions, I'd be happy
> to work with them on improving the test bank.
>
> Another thing to consider is that if we simplify level 1, then people who
> worked hard to accomplish that achievement might feel slighted because we
> lowered the bar. Not sure if that's a real thing, it just occurred to me.
>
> Sorry for the wall of text, just lots ideas bouncing around.
>
> -- Holtej
>
>
> _______________________________________________
> Tlhingan-hol mailing list
> Tlhingan-hol@kli.org
> http://mail.kli.org/mailman/listinfo/tlhingan-hol
>
>
--f46d044286c6cf687604f5db800f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">I don't think anyone believes we should simplify the l=
evel one test. =A0However, I do think that many people would like a .5 leve=
l test, so that there is a benchmark at which people can say "hey, I k=
now something" before being able to clear the level one bar.=A0<div>
<br></div><div>This need not be as large an undertaking as the extant KLCP =
tests, though it would be nice to get it to a similar level. =A0I would say=
that it should use perhaps 100 words max, and that they should be specific=
ally chosen to be words a newbie would encounter or try to use. =A0Certainl=
y, when I first started actually studying for level 1, there were a lot of =
words I knew, and a lot more I had never heard. =A0(SuvwI', vut, and bI=
Q should be on the list, for example. =A0'Ip likely should not.)</div>
<div><br></div><div>We don't have to change the numbering of the tests,=
simply calling it a beginner level test would do.</div><div><br></div><div=
>I also don't think that this thread was meant to call for a reorganiza=
tion of the testing system. =A0As far as I can tell, Qov merely wanted inpu=
t on what caused the most difficulty on the beginner test so she could bett=
er instruct people who want to pass the test as it is.</div>
<div><br></div><div>As far as the suffix questions go, I think the answer t=
o that problem is to alter the questions to ask for more information, not l=
ess. =A0If the question asked for the definition and classification type as=
well as classification number, not knowing the number would only lose peop=
le a point or two on the question, and that would be a truly minor revision=
to the test.</div>
<div><br></div><div>be''etlh</div><div><br></div></div><div class=
=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Sun, Mar 30, 2014 at=
6:47 PM, d'Armond Speers, Ph.D. <span dir=3D"ltr"><<a href=3D"mailt=
o:speersd@georgetown.edu" target=3D"_blank">speersd@georgetown.edu</a>><=
/span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_extra">=
<div><div style=3D"font-family:arial,sans-serif;font-size:13px">I have trie=
d to be consistent, fair and permissive in grading of KLCP tests. =A0The go=
al is not to make someone discouraged by grading hyper-critically, and to a=
llow room for creativity (language is not a 1:1 thing).</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><span =
style=3D"font-family:arial,sans-serif;font-size:13px">For each question in =
the test bank there is an explanation of the purpose of the question, what =
is being tested by the question. =A0These are printed in the answer keys wi=
th the expected answer(s), which the proctor uses when grading. =A0When I&#=
39;m grading tests, I am specifically looking for mistakes that pertain to =
the purpose of the question. =A0If someone answers the question correctly b=
ut is not strictly adhering to the vocabulary or grammar of their test leve=
l, I do not count that as an error. =A0(I would not count off if someone us=
ed {-'e'} in an answer in the level 1 test.) =A0If there are vocabu=
lary or grammatical errors that do not pertain directly to the purpose of t=
he question I count off a single point rather than marking the whole questi=
on wrong. =A0If they completely miss the point of the question, or the numb=
er of errors exceeds the available points per question (5 points per questi=
on for levels 1 & 2), then I just count the question wrong. =A0(If some=
one is making that many errors per question, then they really aren't at=
that level yet.)</span><div style=3D"font-family:arial,sans-serif;font-siz=
e:13px">
<br></div><div style=3D"font-family:arial,sans-serif;font-size:13px">Whenev=
er I distribute tests to someone else to administer and grade, I provide th=
ese instructions. =A0With multiple test administrators over a period of man=
y years, it's easy to see how these guidelines may not be followed cons=
istently. =A0I have not personally attended a qep'a' or qepHom in s=
everal years, or administered tests, though I do still regularly generate n=
ew tests from the test banks for qep'a'mey and qepHommey. =A0It sou=
nds like we may need to get some written guidelines for test administrators=
and strive for greater consistency in grading.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">Creation of the KLCP w=
as a group effort. =A0I designed the structure with Lawrence (3 levels with=
3 pins, because of the cultural importance of the numeral 3). =A0There are=
100 questions for each level in the test bank (plus the reading comprehens=
ion questions for Level 3), and generating tests means selecting questions =
from the test bank at random. =A0Level 1 is a sub-set of TKD; level 2 is al=
l of TKD; and level 3 is open-season on all available materials (mainly inc=
luding additional materials from KGT). =A0We produced written guidelines fo=
r each level, describing the vocabulary and grammar that was in scope. =A0Q=
uestions were written to be distributed evenly across the topics for each l=
evel. =A0Each question in the test bank was reviewed by multiple highly-ski=
lled speakers when we were creating the program, following the written guid=
elines for the level. =A0I have all of this in a MS Access database, which =
allows me to manage the randomization and generation of tests and keys quic=
kly and easily. =A0We put a ton of thought and effort into it, to make it a=
s fair, correct and relevant as we could. =A0And ultimately, it is intended=
to encourage people to achieve certification; it is supposed to be a posit=
ive experience.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">All of that being said=
, we now have the benefit of many years of experience, which we didn't =
have when we were creating the program. =A0And if it's not achieving it=
s goals of encouraging learning and rewarding progress, then we should be o=
pen to that feedback and willing to make adjustments. =A0The challenge is t=
hat the more significant the adjustments (severely limiting the vocabulary =
for level 1, for example), the more effort will be involved in re-developin=
g the test bank. =A0If we make Level 1 too simple, then the gap between Lev=
el 1 and Level 2 becomes huge. =A0And to be honest, I would not be able to =
do this re-development work, so someone else would have to take it over com=
pletely. =A0Taking out the questions about the suffix number would be easie=
r, and I agree that it's probably too pedantic for the level 1 test. =
=A0These questions would have to be replaced with something else, maintaini=
ng balance across the topics for the level. =A0I'm not sure how many qu=
estions there are like this in Level 1, but if anyone has interest in writi=
ng some new replacement questions, I'd be happy to work with them on im=
proving the test bank.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">Another thing to consi=
der is that if we simplify level 1, then people who worked hard to accompli=
sh that achievement might feel slighted because we lowered the bar. =A0Not =
sure if that's a real thing, it just occurred to me.</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px">Sorry for the wall of =
text, just lots ideas bouncing around.</div></div><div style=3D"font-family=
:arial,sans-serif;font-size:13px">
<br></div><div style=3D"font-family:arial,sans-serif;font-size:13px">-- Hol=
tej</div><div style=3D"font-family:arial,sans-serif;font-size:13px"><br></d=
iv></div></div>
<br>_______________________________________________<br>
Tlhingan-hol mailing list<br>
<a href=3D"mailto:Tlhingan-hol@kli.org">Tlhingan-hol@kli.org</a><br>
<a href=3D"http://mail.kli.org/mailman/listinfo/tlhingan-hol" target=3D"_bl=
ank">http://mail.kli.org/mailman/listinfo/tlhingan-hol</a><br>
<br></blockquote></div><br></div>
--f46d044286c6cf687604f5db800f--
--===============7490895127268184842==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Tlhingan-hol mailing list
Tlhingan-hol@kli.org
http://mail.kli.org/mailman/listinfo/tlhingan-hol
--===============7490895127268184842==--
