[235] in bugtraq
Re: [8lgm]-Advisory-14.UNIX.SCO-prwarn.12-Nov-1994
daemon@ATHENA.MIT.EDU (Gene Spafford)
Mon Nov 28 18:56:03 1994
To: Paul Howell <grue@engin.umich.edu>
Cc: "Jonathan M. Bresler" <jmb@kryten.Atinc.COM>,
Dave Brookshire <david@irc.umbc.edu>,
"[8LGM] Security Team" <8lgm@bagpuss.demon.co.uk>, bugtraq@fc.net
In-Reply-To: Message from Paul Howell <grue@engin.umich.edu> of
"Mon, 28 Nov 1994 13:32:52 -0500"
<199411281832.NAA16036@cyclorama.engin.umich.edu>
Date: Mon, 28 Nov 1994 15:01:53 -0500
From: spaf@cs.purdue.edu (Gene Spafford)
> Stating the obvious here, but we seem to be in the experiment now.
Hmm, not exactly. Experiments require controls and statistical bases,
not recollection of previous events.
If one wanted to do a controlled set of trials (once is not sufficient
for meaningful comparison; staff absence, illness, holidays, etc could
be confounding effects), one would need to do something like:
1) pick N bugs of roughly similar impact, severity, and type.
2) randomly, over time, release N/2 as full disclosure, and the
other N/2 as private communications to the vendor(s).
3) time and evaluate the responsiveness of the vendors to these
events.
4) don't let the vendors know they are being tested.
Let's look at a parallel to medicine. Suppose I remember that all my
previous patients with cancer died. Now, I have another one (or two)
come in to my office with similar symptoms, and I treat them by having
them eat their weight in cranberries every day. They both recover.
Does this mean I have found a general cure for cancer? In fact, have
I proven anything?
People will argue that we can't possible do a controlled study of this
problem. Maybe so, although I think we can get some good data
eventually.
My key concern is that people on the net, and on these lists in
particular, spout opinion as proven fact. This perpetuates folklore,
just as knocking on wood or avoiding black cats. We have no general
evidence to prove in any real way that full disclosre helps/hurts more
people than it hurts/helps. We have no evidence that full disclosure
hastens/delays release of a fix. And we have no evidence that the
majority of "black hats" know and use all of these flaws before they
are publicly announced (although there is some partial evidence to the
countrary).
If we are going to improve the way we handle security, we have to
start by examining what we really know and not what we have
experienced locally.
I'm open to anything that shows that full disclosure helps more than
partial or no disclosure. My personal hunch is that it doesn't, but I
won't claim that as fact. I'm simply trying to point out that we all
need to understand this difference between opinion and fact.
> With 8lgm in the past, going with full disclosure. One needs
> to recall how quickly sun/ibm came up with patches for published
> holes.
Were they similar in complexity? Scope? Systems impacted?
> Start the clock, then compare and contrast with how quickly the
> latest flaws are fixed.
It's a good start.
--spaf
