[19304] in s-news-athena
[S] Yet Another Vectorization Question Answered!
daemon@ATHENA.MIT.EDU (Kim Elmore)
Thu Sep 23 16:39:42 1999
Date: Thu, 23 Sep 1999 15:31:07 -0500 (CDT)
From: Kim Elmore <elmore@nssl.noaa.gov>
To: S-News mail list <s-news@wubios.wustl.edu>
Message-Id: <Pine.GSO.4.10.9909231519300.9281-100000@ev.nssl.noaa.gov>
Mime-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
As usual, the list is wonderfully helpful and informatve. I've
been inundated with replies. Again, as always, I'm mystified by the
number of ways people come up with solutions. It's grand!
My original query was:
> I have a very simple operation that I'd like to vectorize.
> However, I haven't figured out how to do it. Being an unrepentant Fortran
> 77 hacker, this vectorization stuff sometimes escapes me. Here's what I'm
> after: within a data frame, I want to know what columns contain negative
> values and I want the results returned as a logical vector.
>
> This works, but uses looping:
>
> test <- logical(0)
> for (i in 1:100)
> {
> test[i] <- all(df[,i] < 0)
> }
>
> Surely there's a way to vectorize this. Please, somebody rescue
> me from my ignorance!
So, without further ado, here are the various answers from all of
you:
From byers@research.att.com Thu Sep 23 15:18:08 1999
test <- apply(df,2,function(x){all(x<0)})
By the way, test is a bad name for any variable or function.
SDB
************************************************************
From: Rod Tjoelker 865-3197 <rod.tjoelker@boeing.com>
Kim,
Try the "apply" functions: "apply","sapply","lapply","tapply"
test <- apply(df,1,function(x) any(x < 0))
Hope this helps,
Rod
************************************************************
From SmithSJ@mar.dfo-mpo.gc.ca Thu Sep 23 15:18:23 1999
How about:
my.fun<-function(data){all(data<0)}
test<-apply(eg.df,2,my.fun)
#careful df is an splus function
>eg.df<-matrix(c(-1,1,-1,1),4,4,byrow=T)
> eg.df
[,1] [,2] [,3] [,4]
[1,] -1 1 -1 1
[2,] -1 1 -1 1
[3,] -1 1 -1 1
[4,] -1 1 -1 1
> my.fun<-function(data){all(data<0)}
> test<-apply(eg.df,2,my.fun)
> test
[1] T F T F
-------------------------------------------------------------
Stephen J. Smith
************************************************************
From Manuela.Huso@orst.edu Thu Sep 23 15:18:30 1999
I'm not sure what you want to do with it but here's an example:
v<-c(1,2,3,-1,3,-2,4,0,-5,9,0,1,-2)
v
[1] 1 2 3 -1 3 -2 4 0 -5 9 0 1 -2
v<0
[1] F F F T F T F F T F F F T
v[v<0]
[1] -1 -2 -5 -2
v[v<0]<-1
v
[1] 1 2 3 1 3 1 4 0 1 9 0 1 1
Manuela Huso
************************************************************
From peter@caliban.ucsd.edu Thu Sep 23 15:18:36 1999
> test[i] <- all(df[,i] < 0)
which is to say, you want to check which columns have ALL negative
values, right?
you could do something like this:
rep(1,nrow(df))%*%ifelse(df<0,1,0) == nrow(df)
hope this helps.
- peter
************************************************************
From dougj@statsci.com Thu Sep 23 15:18:41 1999
It's a good idea to avoid naming an S-PLUS data object 'df', since that
will mask the built-in S-PLUS 'df()' function, which returns the density
of the F distribution.
Here is a trio of examples using subscripting. The first selects from
a vector; the second selects from the second column of a data frame.
The third example returns a logical vector whose elements are T wherever
the corresponding element of the column is less than zero, and F elsewhere.
vector
``````
The following commands create and display a repeatable vector named 'daj':
> set.seed(1)
> daj <- round(rnorm(35), 2)
> daj
[1] -0.79 0.79 -0.89 0.11 1.37 1.42 1.17 -0.53 0.92 -0.58
[11] -0.09 -0.96 -0.27 -1.34 0.76 -0.56 0.19 -0.93 -0.18 -1.58
[21] 0.88 -0.22 0.06 -0.80 -0.49 -0.38 -2.17 -0.27 -0.66 1.43
[31] -0.59 0.18 -0.99 1.85 1.34
The following commands create and display those elements of 'daj'
whose values are less than zero:
> test <- daj[daj<0]
> test
[1] -0.79 -0.89 -0.53 -0.58 -0.09 -0.96 -0.27 -1.34 -0.56 -0.93
[11] -0.18 -1.58 -0.22 -0.80 -0.49 -0.38 -2.17 -0.27 -0.66 -0.59
[21] -0.99
data frame:
```````````
The following commands create and display a data frame named 'daj':
> daj <- data.frame( x=round(rnorm(15), 2),
y=round(rnorm(15), 2))
> daj
x y
1 0.06 -0.80
2 1.01 0.08
3 -1.19 1.42
4 1.93 -0.97
5 0.87 1.16
6 -1.30 -0.31
7 0.03 0.51
8 0.69 0.15
9 -0.32 -0.93
10 1.39 -2.37
11 -0.31 1.49
12 -1.12 -1.45
13 1.05 -0.68
14 -1.38 -0.85
15 0.79 -0.29
The following commands create and display a vector whose elements
are the negative-valued elements from the second column of the
data frame 'daj':
> test <- daj[ daj[,2] < 0, 2 ]
> test
[1] -0.80 -0.97 -0.31 -0.93 -2.37 -1.45 -0.68 -0.85 -0.29
logical test
````````````
The following command initializes the test vector:
> test <- as.logical(rep(0, length(daj[,2])))
The following command displays the initialized test vector:
> test
[1] F F F F F F F F F F F F F F F
The following command updates the test vector to be true wherever
the second column of 'daj' is less than zero:
> test[ daj[,2] < 0 ] <- T
The following command displays the resulting logical test vector:
> test
[1] T F F T F T F F T T F T T T T
You can find more information on the S-PLUS operators and functions
mentioned in this discussion in the on-line help files that are displayed
when you type
?df
?set.seed
?round
?rnorm
?"<-"
?"("
?"["
?"<"
?data.frame
?as.logical
?rep
?length
?masked
and
?find
at the S-PLUS prompt, in the Commands window.
I hope this helps,
-Doug Johnson
MathSoft Seattle
************************************************************
From: Renaud Lancelot <lancelot@telecomplus.sn>
Hi Kim,
See apply():
> M <- matrix(-10:9, ncol = 5)
> M
[,1] [,2] [,3] [,4] [,5]
[1,] -10 -6 -2 2 6
[2,] -9 -5 -1 3 7
[3,] -8 -4 0 4 8
[4,] -7 -3 1 5 9
> apply(M, 2, FUN = function(x) any(x < 0))
[1] T T T F F
Careful with any() or all()...
Hope this helps,
Renaud
************************************************************
From bill@statsci.com Thu Sep 23 15:19:02 1999
If your data is really a data frame (not a matrix) you can speed
this up a bit with
test <- unlist(lapply(df, function(dfi)all(dfi<0)))
However, the major speedup in this case comes from lapply
treating df as a list and using the equivalent of [[i]] instead
of [,i] on it. (The subscripting methods for data frames are
rather slow, those for lists are fast.) Thus you can squeeze
out some of the time by avoiding some of the overhead of lapply
and doing just
f3 <- function(df) {
test <- logical(length(df))
# names(test) <- names(df) # not needed, but handy
df <- as.list(df)
for(i in 1:length(df)) {
test[i] <- all(df[[i]] < 0)
}
test
}
Lets call your original method f0:
f0 <- function(df) {
test <- logical(ncol(df))
for (i in 1:ncol(df))
{
test[i] <- all(df[,i] < 0)
}
test
}
and the lapply method f4:
f4 <- function(df) unlist(lapply(df, function(dfi) all(dfi < 0)))
For a 10 row by 100 column data I get:
> unix.time(f0(df))
[1] 1.279999 0.000000 2.000000 0.000000 0.000000
> unix.time(f3(df))
[1] 0.1300011 0.0000000 0.0000000 0.0000000 0.0000000
> unix.time(f4(df))
[1] 0.2700005 0.0000000 1.0000000 0.0000000 0.0000000
If we uncomment the line that assigns the names to the output
in f3 the time goes up to that of f4.
The relative timings will change, depending on the dimensions
of df.
----------------------------------------------------------------------------
Bill Dunlap 22461 Mt Vernon-Big Lake Rd
Data Analysis Products Div. of MathSoft, Inc. Mount Vernon, WA 98274
bill@statsci.com 360-428-8146
************************************************************
From bates@stat.wisc.edu Thu Sep 23 15:19:08 1999
It is not exactly vectorization but you do this operation with the
"apply" function which applies a function to subsections of an array.
If I recall correctly it would look like
apply( as.matrix(df), 2, function(x) any(x < 0) ) # 2 indicates apply to columns
> df <- as.data.frame( matrix( rnorm(10000, mean = 3), nrow = 1000 ) )
> apply( as.matrix( df ), 2, function(x) any( x < 0 ) )
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10
TRUE TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE TRUE
If you want to apply to rows of a matrix you use 1 as the second
argument.
BTW, I think from your description you wanted "any", not "all" in the
function.
From mconklin@customresearch.com Thu Sep 23 15:19:14 1999
Date: Thu, 23 Sep 1999 14:28:32 -0500
From: Michael Conklin <mconklin@customresearch.com>
To: elmore@nssl.noaa.gov
Subject: Re: [S] Yet Another Vectorization Question
Here is my quick solution - I'm sure you will get better ones.
logvec<-is.element(1:100,unique(col(df)[df<0]))
good luck.
Michael Conklin
************************************************************
From: "CHASALOW, SCOTT [AG/2165]" <SCOTT.CHASALOW@cereon.com>
Kim,
Here's one way to add to the many I'm sure you've received:
colSums(dfr < 0) == nrow(dfr)
This assumes, as I guess from your example, you want columns
containing *only* negative values.
Cheers,
Scott
Cereon Genomics
Scott.Chasalow@cereon.com
************************************************************
From: Don MacQueen <macq@llnl.gov>
I'm going to assume all columns in your dataframe are numeric, and that
you're using Splus 5.1 on unix or Splus 2000 in Windows. More accurately,
using a version that has the rowSums() function.
Here's an example:
> foo <- data.frame(x=.5-runif(10),y=.2-runif(10),z=1:10)
> foom <- as.matrix(foo) <0
> colSums(foom) > 0
x y z
T T F
Or, as a one-liner
test <- rowSums( as.matrix(df) < 0) > 0
Or, if you don't have a version with colSums()
> apply(foom,2,FUN=function(x) sum(x)>0)
x y z
T T F
In which case, the oneliner is
test <- apply(as.matrix(df) < 0, 2, FUN=function(x) sum(x)>0)
Note that the assumption that the data frame is entirely numeric is crucial.
Yet another version is
sapply(foo,FUN=function(x) sum(x<0)>0)
This version seems to work even with columns that have character data,
since the following do not result in error messages:
> 'a' < 0
[1] F
> 'a' > 0
[1] T
Note that apply(), sapply(), and colSums() are all functions that have the
looping built in. Saving us the trouble of writing the loop explicitly.
-Don
************************************************************
Whew! Thanks again to every, single one of you. What a great
group!
Kim Elmore, [N5OP, PP ASMEL/Glider 2232456]
"All of Meteorology is divided into three parts: Yes, No and Maybe. The
greatest of these is Maybe." -- The original Latin appears to be garbled.
-----------------------------------------------------------------------
This message was distributed by s-news@wubios.wustl.edu. To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message: unsubscribe s-news
