[98311] in tlhIngan-Hol
Re: [Tlhingan-hol] Certification Test Woes
daemon@ATHENA.MIT.EDU (d'Armond Speers, Ph.D.)
Mon Mar 31 11:44:33 2014
In-Reply-To: <53391C9F.9090404@gmx.de>
Date: Mon, 31 Mar 2014 09:44:05 -0600
From: "d'Armond Speers, Ph.D." <speersd@georgetown.edu>
To: tlhIngan-Hol List <tlhingan-hol@kli.org>
Errors-To: tlhingan-hol-bounces@kli.org
--===============4281065757100016089==
Content-Type: multipart/alternative; boundary=001a11c124b0e8b30a04f5e8eb53
--001a11c124b0e8b30a04f5e8eb53
Content-Type: text/plain; charset=ISO-8859-1
On Mon, Mar 31, 2014 at 1:43 AM, Lieven <levinius@gmx.de> wrote:
>
>
> classification number, not knowing the number would only lose people a
>> point or two on the question, and that would be a truly minor revision
>> to the test.
>>
>
> That's what makes it hard for grading. Does the half answer deserve a half
> point? Then why does an entirely translated sentence only deserve one
> point? But thats another topic :-)
This does expose what might be a weakness in the test design. For Level 1
and 2 there are 20 questions, each question worth 5 points. The answers to
some questions can be rather long (multiple words, multiple affixes)
compared to other questions (a single word or affix). If the point of the
question is about affixes and you got the right affixes in the right
positions, but missed the root word, then you get partial credit. But if
the answer is only a single word ("What is the suffix for augmentation" or
whatever; not sure if that's a real question), then you either get it or
you don't. In effect, mistakes are not worth the same amount, the amount
is dependent upon the length of response and objective of the question, and
the student doesn't know what the test grader might actually be looking
for, beyond the specific wording of the question.
This is addressed by (a) encouraging the test taker to provide as much
information in their response as possible. Another way of saying this is
that if you don't know the answer, there's no penalty for guessing. You'll
lose all of the question points for a blank, but you might get some partial
credit if you guess (or you might get lucky and get full credit if you get
it right). And secondly (b) by having a large test bank that populates
questions at random for any given test. Over a range of tests the
proportion of low-value vs. high-value questions should be consistent, and
therefore fair.
As I recall we did discuss this at the time we were developing the KLCP.
The alternative is to balance the questions/answers so they all have the
same (or nearly the same) amount of content, and thus the same value.
Sounds nice on paper, but in practice this proved to be unworkable (how do
you measure "content" of a question; how do you limit all questions to a
similar range of content, etc.), so we went with the mitigations described
above. I can't decide if the mitigation is valid or a rationalization, but
that's why we get to revisit these issues with the benefit of hindsight.
(As an aside: I'm assuming it's clear, but let me state it explicitly: I am
not opposed to discussing the weaknesses of the tests or finding ways to
improve them; I do not intend any of my statements to come off as
defensive, and I apologize if they do; and I think this type of review is
healthy for us as a community to better support and encourage new speakers
of the language, which is in line with the objectives of the KLCP overall.
I am grateful to Qov for starting this discussion, which I find very
interesting. If we decide it would be in our interests to make
improvements, I'm happy to see the community pulling together in a positive
way. In a way, the fact that we're having this discussion is evidence of
our success; when the tests were being developed the number of speakers who
could participate in discussions about the test was very small. So yay!)
--Holtej
--001a11c124b0e8b30a04f5e8eb53
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">=
On Mon, Mar 31, 2014 at 1:43 AM, Lieven <span dir=3D"ltr"><<a href=3D"ma=
ilto:levinius@gmx.de" target=3D"_blank">levinius@gmx.de</a>></span> wrot=
e:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l=
eft:1px #ccc solid;padding-left:1ex">
<div class=3D""><br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
classification number, not knowing the number would only lose people a<br>
point or two on the question, and that would be a truly minor revision<br>
to the test.<br>
</blockquote>
<br></div>
That's what makes it hard for grading. Does the half answer deserve a h=
alf point? Then why does an entirely translated sentence only deserve one p=
oint? But thats another topic :-)</blockquote><div><br></div><div>=A0</div>
<div>This does expose what might be a weakness in the test design. =A0For L=
evel 1 and 2 there are 20 questions, each question worth 5 points. =A0The a=
nswers to some questions can be rather long (multiple words, multiple affix=
es) compared to other questions (a single word or affix). =A0If the point o=
f the question is about affixes and you got the right affixes in the right =
positions, but missed the root word, then you get partial credit. =A0But if=
the answer is only a single word ("What is the suffix for augmentatio=
n" or whatever; not sure if that's a real question), then you eith=
er get it or you don't. =A0In effect, mistakes are not worth the same a=
mount, the amount is dependent upon the length of response and objective of=
the question, and the student doesn't know what the test grader might =
actually be looking for, beyond the specific wording of the question. =A0</=
div>
<div><br></div><div>This is addressed by (a) encouraging the test taker to =
provide as much information in their response as possible. =A0Another way o=
f saying this is that if you don't know the answer, there's no pena=
lty for guessing. =A0You'll lose all of the question points for a blank=
, but you might get some partial credit if you guess (or you might get luck=
y and get full credit if you get it right). =A0And secondly (b) by having a=
large test bank that populates questions at random for any given test. =A0=
Over a range of tests the proportion of low-value vs. high-value questions =
should be consistent, and therefore fair.</div>
<div><br></div><div>As I recall we did discuss this at the time we were dev=
eloping the KLCP. =A0The alternative is to balance the questions/answers so=
they all have the same (or nearly the same) amount of content, and thus th=
e same value. =A0Sounds nice on paper, but in practice this proved to be un=
workable (how do you measure "content" of a question; how do you =
limit all questions to a similar range of content, etc.), so we went with t=
he mitigations described above. =A0I can't decide if the mitigation is =
valid or a rationalization, but that's why we get to revisit these issu=
es with the benefit of hindsight.</div>
<div><br></div><div>(As an aside: I'm assuming it's clear, but let =
me state it explicitly: I am not opposed to discussing the weaknesses of th=
e tests or finding ways to improve them; I do not intend any of my statemen=
ts to come off as defensive, and I apologize if they do; and I think this t=
ype of review is healthy for us as a community to better support and encour=
age new speakers of the language, which is in line with the objectives of t=
he KLCP overall. =A0I am grateful to Qov for starting this discussion, whic=
h I find very interesting. =A0If we decide it would be in our interests to =
make improvements, I'm happy to see the community pulling together in a=
positive way. =A0In a way, the fact that we're having this discussion =
is evidence of our success; when the tests were being developed the number =
of speakers who could participate in discussions about the test was very sm=
all. =A0So yay!)</div>
<div><br></div><div>--Holtej</div></div></div></div>
--001a11c124b0e8b30a04f5e8eb53--
--===============4281065757100016089==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Tlhingan-hol mailing list
Tlhingan-hol@kli.org
http://mail.kli.org/mailman/listinfo/tlhingan-hol
--===============4281065757100016089==--
