[4337] in java-interest

home help back first fref pref prev next nref lref last post

Bug in StreamTokenizer!

daemon@ATHENA.MIT.EDU (Glenn Maughan)
Tue Dec 19 07:57:45 1995

Date: Tue, 19 Dec 1995 17:04:14 +1100
From: glennm@optimation.com.au (Glenn Maughan)
To: java-interest@java.sun.com
Cc: grege@optimation.com.au

We have found strange behaviour with the StreamTokenizer class when using pipes or redirection.  
The following example reads a line with a <word> <number> <number> and prints out each value read.
When reading from stdin (ie, the keyboard) the routine works correctly.  However, when
reading from a pipe or a redirected file the parsing of numbers does not work correctly.  For example,
if the first line to read is:

	A 123 456

the parse will read "A" as the label correctly.  But then it will read "1" as the first number and 
"2" as the second number instead of "123" and "456" respectively.  This only happens when reading 
from a pipe or redirection:

	$ java TestTokenizer < numbers.txt

or

	$ cat numbers.txt | java TestTokenizer

Here is the code.  Can anyone see what the problem is?
Thanks,
Glenn.


-----------------------------------------------------------------------------------------------------
import java.io.*;

class TestTokenizer {

  public void run() {
    StreamTokenizer tokenizer = new StreamTokenizer(System.in);
    int newToken;		
    String newLabel;			
    double newDouble;		
    
    // set up the tokenizer
    tokenizer.parseNumbers();

    // prime the loop with the first token
    try {
      newToken = tokenizer.nextToken();
      while (newToken != tokenizer.TT_EOF) {
	// first token should be a label
	if (newToken == tokenizer.TT_WORD) {
	  newLabel = tokenizer.sval;
	  System.out.println("Label: " + newLabel);
	  // get the first value
	  newToken = tokenizer.nextToken();
	  if (newToken == tokenizer.TT_NUMBER) {
	    newDouble = tokenizer.nval; 
	    System.out.println("First value: " + newDouble);
	    // get the second value
	    newToken = tokenizer.nextToken();
	    if (newToken == tokenizer.TT_NUMBER) {
	      newDouble = tokenizer.nval; 
	      System.out.println("Second value: " + newDouble);
	    }
	  }
	}
	newToken = tokenizer.nextToken();
      }
    } catch (IOException e) {}
  }

  public static void main(String args[]) {
    TestTokenizer t = new TestTokenizer();

    t.run();
  }
  
}
-------------------------------------------------------------------------------------------
-
This message was sent to the java-interest mailing list
Info: send 'help' to java-interest-request@java.sun.com

home help back first fref pref prev next nref lref last post