[21926] in Perl-Users-Digest
Perl-Users Digest, Issue: 4130 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Nov 16 18:11:21 2002
Date: Sat, 16 Nov 2002 15:10:14 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sat, 16 Nov 2002 Volume: 10 Number: 4130
Today's topics:
Mixing DBD::CSV and hand-edited files [Was Re: CSV SQL] <jeff@vpservices.com>
Re: Mixing DBD::CSV and hand-edited files [Was Re: CSV <jeff@vpservices.com>
Re: parameter <bwalton@rochester.rr.com>
pearl ??'s (Thedevilboy)
Re: pearl ??'s <mbudash@sonic.net>
Re: pearl ??'s <tk@WINDOZEdigiserv.net>
Re: pearl ??'s <tassilo.parseval@post.rwth-aachen.de>
Re: pearl ??'s <nobody@noplace.com>
Re: Special Characters from HTML form to Oracle DB <dsolbach@web.de>
Re: Special Characters from HTML form to Oracle DB <flavell@mail.cern.ch>
Re: Special Characters from HTML form to Oracle DB <flavell@mail.cern.ch>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 16 Nov 2002 11:10:20 -0800
From: Jeff Zucker <jeff@vpservices.com>
Subject: Mixing DBD::CSV and hand-edited files [Was Re: CSV SQL]
Message-Id: <3DD6981C.9000208@vpservices.com>
[posted and mailed]
Lois wrote:
> I created a CSV SQL using the following command:
> $UserDB=DBI->connect("DBI:CSV:f_dir=.") ...
> after the userdb file is created, how can I manually add records to the file
> by using vi and yet recognised by the select statement to retrieve record.
DBD::CSV will not work with evil vi, it only works with wonderful emacs!
Just kidding :-)
There are several things to watch out for. The main thing is line
endings. DBD::CSV defaults to using windows-style line endings
"\015\012" so if your vi is set to produce unix style line endings
"\012", you should set the csv_eol (end of line) within your scripts
like this:
my $table_name = # name of the table
$dbh->{csv_tables}->{$table_name}->{eol} = "\012";
Or if all your tables will use the same line endings, like this,
immediately after the connect:
$dbh->{csv_eol}="\012";
Alternatively, you could set vi to produce windows style line endings,
but that seems cumbersome to me.
The other thing to watch out for is how you format the data. Remember
that if there are embedded quote marks in the data, they should be
escaped by putting in two of them:
foo,bar,"this has ""embedded quotes"" in it",baz
That is a four field record. The third field will look like this when
you retrieve it with a DBI query:
this has "embedded quotes" in it
You could set the quote escape char to be something else (e.g. a
backwhack) with csv_escape_char, if you want, just be consistent between
what you produce with vi and what your script expects.
Another thing is that NULLs should be entered as blanks, not spaces:
foo,bar,,baz
That is also a four field record with the third field NULL.
When DBD::CSV writes records, it puts quotes only around fields that
need them (strings with embedded commas, quotes, or newlines) but it
understands other fields with quotes so here's another valid four-field
record:
foo,"bar","embedded comma, included",baz
The quotes around "bar" are ok but not required. The quotes around the
third field are required since there is an embedded comma.
That's all of the gotchas I can think of at the moment, let me know if
you run into problems. Good luck!
--
Jeff (maintainer of DBD::CSV)
------------------------------
Date: Sat, 16 Nov 2002 11:28:19 -0800
From: Jeff Zucker <jeff@vpservices.com>
Subject: Re: Mixing DBD::CSV and hand-edited files [Was Re: CSV SQL]
Message-Id: <3DD69C53.9050303@vpservices.com>
Ooops, I forgot one other important gotcha:
FLOCK IS ONLY ADVISORY. In other words, if you are hand-editing the
database file with vi and someone else attempts to access it with a
script, the script will have no way of knowing that you are hand editing
the database. So if your script is available online, take it offline
while you're hand editing, or have it check for a "no edit" file that
you create prior to doing your hand editing.
--
Jeff
------------------------------
Date: Sat, 16 Nov 2002 16:44:18 GMT
From: Bob Walton <bwalton@rochester.rr.com>
Subject: Re: parameter
Message-Id: <3DD675A4.1080509@rochester.rr.com>
Rodney Hunter wrote:
> Perl newbie q - i have taken all the real code out to leave the test
> 'print' - how come the subroutine reverts the parameter back to the value
> from the earlier pass?
>
> thanks
> Rodney Hunter
>
> first pass job_no=7010 result B 7010 C 7010 D 7010
> secd pass job_no=7011 result B 7011 C 7010 D 7011
> third pass job_no=7012 result B 7012 C 7010 D 7012
>
>
> # get value passed from form
> my $thisjobno = $q->param('job_no');
> # print HTML header
> printHead('EA Journal');
> print <<ENDOFTEXT;
> B $thisjobno
> ENDOFTEXT
> Sect1 ($thisjobno);
> print <<ENDOFTEXT;
> D $thisjobno
> ENDOFTEXT
>
> sub Sect1{
> print <<ENDOFTEXT;
> C $thisjobno
> ENDOFTEXT
> }
>
>
>
>
>
I'm unable to duplicate the problem you are having. I rewrote your
example (shown below) to use CGI so it would run on my system without my
having to write subs you didn't include, but the basic code is the same.
The results are that it always prints the latest value of the jobno
parameter every time, including from the sub. That's because the
lexical $thisjobno is still in scope when the sub is called.
BTW, you passed $thisjobno to the sub as a parameter, and then didn't
use that parameter in the sub. Sub parameters are passed in @_ -- see
perldoc perlsub
for additional info on that.
Are you by any chance using mod_perl or something like it? That's the
only way your CGI script could possibly remember the value of a Perl
variable from one execution to the next.
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);
use warnings;
use strict;
# get value passed from form
my $thisjobno = param('jobno');
# print HTML header
print header,title('EA Journal');
print <<ENDOFTEXT;
B $thisjobno
ENDOFTEXT
Sect1 ($thisjobno);
print <<ENDOFTEXT;
D $thisjobno
ENDOFTEXT
print start_form,textfield(-name=>'jobno'),submit,end_form,end_html;
sub Sect1{
print <<ENDOFTEXT;
C $thisjobno
ENDOFTEXT
}
--
Bob Walton
------------------------------
Date: 16 Nov 2002 12:57:31 -0800
From: Thedevilboy86@msn.com (Thedevilboy)
Subject: pearl ??'s
Message-Id: <1d7126be.0211161257.328d718e@posting.google.com>
HELLO I AM HAVING PROBLAMS WITH MY pearl.
I SET THA PATH TO pearl AND IT WONT WORK WHAT DID DO I WRONG.
I THINK I NEED TO LEARN SOME MORE pearl.
I WOOD LIKE SUM HELP.
#!/usr/bin/pearl
print"content-type: PEARL/PEARL\n\n";
print\"!!AT AM I DOING WRONG!'"\;
#!/usr/bin/pearl!;
I WAS THINKING MY PROBLEM IS pearl IS A INTERNAT SCRIPTING LANGUAGE
AND I AM TRYING TO USE IT ON MY COMPUTER.
MY WEBTV WORKED pearl FINE I DONT SEE WHAT THE PROBLEM IS WITH MY
COMPUTER AND pearl.
I AM TRYING TO USE pearl ON WINDOWS LONGHORN.
I EVEN NAME THE FILE .pearl
THANKS YOU AHEAD OF TIME.
------------------------------
Date: Sat, 16 Nov 2002 21:02:34 GMT
From: Michael Budash <mbudash@sonic.net>
Subject: Re: pearl ??'s
Message-Id: <mbudash-8BC261.13023316112002@typhoon.sonic.net>
In article <1d7126be.0211161257.328d718e@posting.google.com>,
Thedevilboy86@msn.com (Thedevilboy) wrote:
> HELLO I AM HAVING PROBLAMS WITH MY pearl.
> I SET THA PATH TO pearl AND IT WONT WORK WHAT DID DO I WRONG.
> I THINK I NEED TO LEARN SOME MORE pearl.
> I WOOD LIKE SUM HELP.
>
>
> #!/usr/bin/pearl
> print"content-type: PEARL/PEARL\n\n";
> print\"!!AT AM I DOING WRONG!'"\;
>
> #!/usr/bin/pearl!;
>
> I WAS THINKING MY PROBLEM IS pearl IS A INTERNAT SCRIPTING LANGUAGE
> AND I AM TRYING TO USE IT ON MY COMPUTER.
> MY WEBTV WORKED pearl FINE I DONT SEE WHAT THE PROBLEM IS WITH MY
> COMPUTER AND pearl.
>
> I AM TRYING TO USE pearl ON WINDOWS LONGHORN.
> I EVEN NAME THE FILE .pearl
>
>
> THANKS YOU AHEAD OF TIME.
perl, not pearl
hth-
------------------------------
Date: Sat, 16 Nov 2002 21:08:24 GMT
From: tk <tk@WINDOZEdigiserv.net>
Subject: Re: pearl ??'s
Message-Id: <ascdtugllf162q0crp7ba6jhn60lf8q74m@4ax.com>
-----BEGIN xxx SIGNED MESSAGE-----
Hash: SHA1
In a fit of excitement on 16 Nov 2002 12:57:31 -0800,
Thedevilboy86@msn.com (Thedevilboy) managed to scribble:
| HELLO I AM HAVING PROBLAMS WITH MY pearl.
And you CAPS button!?
| I SET THA PATH TO pearl AND IT WONT WORK WHAT DID DO I WRONG.
| I THINK I NEED TO LEARN SOME MORE pearl.
| I WOOD LIKE SUM HELP.
ROFL! I'd suggest www.dictionary.com as a starter.
|
|
| #!/usr/bin/pearl
Digging for pearls?
| print"content-type: PEARL/PEARL\n\n";
| print\"!!AT AM I DOING WRONG!'"\;
ATm, you're doing many things wrong.
|
| I WAS THINKING MY PROBLEM IS pearl IS A INTERNAT SCRIPTING LANGUAGE
| AND I AM TRYING TO USE IT ON MY COMPUTER.
You thought wrong, again.
| MY WEBTV WORKED pearl FINE I DONT SEE WHAT THE PROBLEM IS WITH MY
| COMPUTER AND pearl.
|
| I AM TRYING TO USE pearl ON WINDOWS LONGHORN.
| I EVEN NAME THE FILE .pearl
|
|
| THANKS YOU AHEAD OF TIME.
I'd strongly suggest RTFM for the time being.
Regards,
tk
-----BEGIN xxx SIGNATURE-----
Version: PGP Personal Privacy 6.5.3
iQA/AwUBPda0DSjNZg8h4REKEQJxBACguXHFnRgpX1PXLjeILOq/Jwfgm74AoNZP
53neGBoeLkPpmhgkyW8mZVyD
=bwqs
-----END PGP SIGNATURE-----
--
+--------------------------+
| digiServ Network |
| Web solutions | Remove WINDOZE to reply.
| http://www.digiserv.net/ |
+--------------------------+
------------------------------
Date: 16 Nov 2002 21:15:17 GMT
From: "Tassilo v. Parseval" <tassilo.parseval@post.rwth-aachen.de>
Subject: Re: pearl ??'s
Message-Id: <ar6ch5$6cb$1@nets3.rz.RWTH-Aachen.DE>
This simply must be in the top-ten of the most comical postings!
Also sprach Thedevilboy:
> HELLO I AM HAVING PROBLAMS WITH MY pearl.
> I SET THA PATH TO pearl AND IT WONT WORK WHAT DID DO I WRONG.
> I THINK I NEED TO LEARN SOME MORE pearl.
> I WOOD LIKE SUM HELP.
>
>
> #!/usr/bin/pearl
It's 'perl' not 'pearl'. The thing you mentioned is a different language
which - I think - you don't actually mean.
> print"content-type: PEARL/PEARL\n\n";
You sure with the Content-type?
> print\"!!AT AM I DOING WRONG!'"\;
This line is a syntax error.
> #!/usr/bin/pearl!;
Computers are more responsive if you don't shout at them. Drop the
trailing exclamation mark.
> I WAS THINKING MY PROBLEM IS pearl IS A INTERNAT SCRIPTING LANGUAGE
> AND I AM TRYING TO USE IT ON MY COMPUTER.
> MY WEBTV WORKED pearl FINE I DONT SEE WHAT THE PROBLEM IS WITH MY
> COMPUTER AND pearl.
>
> I AM TRYING TO USE pearl ON WINDOWS LONGHORN.
LONGHORN?
> I EVEN NAME THE FILE .pearl
The extension you have to give to the file is determined by a) the way
you call your scripts ('perl script.any_extentsion' will always work),
b) your operating system (unices don't care about extensions, they
inspect the shebang line) and c) some other circumstances, for instance in
the context of web-browsers and how they are configured.
I think you need to workout a few things before getting started. First
try to get rid of some of your deep confusions. Good luck and all the
best with that.
As for your posting style: Work on that immediately. Don't use allcaps.
It might not be fair to rant about your spelling but keep in mind that
your posting will cover newsservers around the world in a very short
time and archived elsewhere. Under such conditions I certainly try not
to make a fool out of myself, be it willingly or not.
Tassilo
--
$_=q!",}])(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
pam{rekcahbus;})(rekcah{lrePbus;})(lreP{rehtonabus;})(rehtona{tsuJbus!;
$_=reverse;s/sub/(reverse"bus").chr(32)/xge;tr~\n~~d;eval;
------------------------------
Date: Sat, 16 Nov 2002 21:48:51 GMT
From: "Gregory Toomey" <nobody@noplace.com>
Subject: Re: pearl ??'s
Message-Id: <01c28dbb$70bc8900$4c498a90@gmtoomey>
> MY WEBTV WORKED pearl FINE
Strange. I thought you were an AOL user.
gtoomey
------------------------------
Date: Tue, 12 Nov 2002 09:41:32 +0100
From: "David Solbach" <dsolbach@web.de>
Subject: Re: Special Characters from HTML form to Oracle DB
Message-Id: <ar5tdb$f14$02$1@news.t-online.com>
> > Here's the problem:
> >
> > I read data from textfields
>
> What format is the data encoded in?
to be honest, i don't know, I didn't specify any charsets in the html, perl
or javascript source (if that would be possible)
the application runs on an apache webserver on a linux box, so pherhaps you
can tell me which format the data is encoded in? ;)
> > and write them into an oracle db using DBI/DBD
> > modules. unfortunately all special characters like german umlauts or
> > any french special chars get screwed up in the database.
>
> These german umlauts and french special chars, are they in utf, latin2,
> or some other format?
again, sorry, how can I find it out and tell you?
> > (they are converted to common ascii chars)
>
> Which ascii characters are they converted to?
atm I've no opportunity to test it again, but when I testet it i think
german umlauts öäü where converted to e,u and | (don't know if the order is
correct)
> > But the oracle settings seem to be corrrect, because if I try to write
> > literals to the DB it works fine.
>
> From what program, through DBI, or through Oracle's sql-shell?
both works fine, DBI and sql-shell
> > I think this should be a common problem, pherhaps you can tell me
> > where to find a solution, if you need extra information just tell me
> > what you need to know.
>
> The solution is to convert all of the data to a common encoding before
> putting it in, and then convert it to whatever encoding you want when
> taking it out.
>
> Usually utf8 encoding or utf16 encoding is preferred, though latin1,
> latin2, etc, ... might suite your data better, depending on the
> language. But whichever you choose, you *must* store all the strings in
> the same encoding, otherwise your data will be corrupted.
>
> Either Encode.pm or Text::Iconv should be used for converting from one
> encoding to another.
>
ok, thanks four your help, I think I should use the biggest charset, which
probably is utf16? because the application
shall support as many languages as possible, at least english, german and
french is a must.
greets,
David
------------------------------
Date: Sat, 16 Nov 2002 18:37:59 +0100
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: Special Characters from HTML form to Oracle DB
Message-Id: <Pine.LNX.4.40.0211161800510.22974-100000@lxplus071.cern.ch>
On Nov 9, David Solbach inscribed on the eternal scroll:
> I read data from textfields and write them into an oracle db using DBI/DBD
> modules. unfortunately all special characters like german umlauts or any
> french special chars get screwed up in the database. (they are converted to
> common ascii chars)
You have a potentially complex problem - more complex, I think, than
just the symptoms which you are reporting here.
You are in a situation where you must (IMHO, at least) analyze very
carefully the various layers of coding that are involved in your
processing chain.
If I've understood you right, you have an HTML page on which there is
a form with text fields (they could be input type=text or could be
textarea, those aren't significantly different for the purposes of
this discussion). Users will somehow (typing or copy/pasting) cause
characters to be entered into these fields and "submitted" to a
server. At the server they will be decoded in a server-side script (I
guess this is where your Perl relevance comes in) and then used for DB
access. Have I got the picture right?
I see from your subsequent f'up that there's even javascript
involved...
_If_ you are only concerned with one browser and only with one
character coding (charset) of submission form, then things may become
somewhat simpler, but they still are not without potential complexity.
If you're in a WWW context, with no control over which browsers are in
use and how they are configured, then things can get quite hairy.
One part of your processing chain is the browser's conversion of
submitted characters into an encoded representation for submitting to
the server. I have some notes about that part of the business here
http://ppewww.ph.gla.ac.uk/~flavell/charset/form-i18n.html
where you will also find some account of the curious things which
happen in various browsers.
> But the oracle settings seem to be corrrect, because if I try to write
> literals to the DB it works fine.
What character coding do your DB calls expect? (is it iso-8859-1, is
it unicode-based, or what?) Is it the same as the coding you are
using (charset= on the content-type) for the HTML page that the user
is getting?
Which forms submission encoding are you using (probably
x-www-form-urlencoding if you aren't also offering file upload, though
multipart/form-data is also an option, and indeed is mandatory if you
also offer file upload in the same form); and how are you decoding the
result in the server script before passing it to your DB calls?
I would strongly recommend at least instrumenting the various parts of
the chain, printing out the data in hexadecimal at various stages, and
making sure that you understand what you have, and how it's being
transformed at each step. It's about interworking interfaces, and
making sure that the data really does conform to its intended
specification at each such interface. My recommendation would be that
some extra time taken initially to get comfortable with each step
separately, can save masses of confusion - and consequent wasted
effort - about the overall operation of such a system.
> I think this should be a common problem,
Well, they have certainly found some kind of 'modus vivendi' at the
various search engine services[1], but they don't always exactly
follow the specifications. Sometimes there are heuristic kludges, to
work around the shortcomings of browsers. We've discussed this on
occasion in the past in comp.infosystems.www.authoring.* groups, but
there are indeed some Perl-related issues too, so it might be worth
researching some past discussions in both newsgroup hierarchies.
Which Perl version are you working with, by the way? (Note the
progressive introduction of unicode capabilities, which might turn
out to be relevant to your problem - at the moment I'm not sure).
I'll comment separately on some detailed points from your later f'up.
good luck (you may need it ;-)
[1] Look for usenet commentaries on this from A.Prilop, who seems to
take an active interest in the topic of i18n in the popular search
engines, though I don't think he maintains a web page about it as
such.
------------------------------
Date: Sat, 16 Nov 2002 19:22:59 +0100
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: Special Characters from HTML form to Oracle DB
Message-Id: <Pine.LNX.4.40.0211161839070.22974-100000@lxplus071.cern.ch>
On Nov 12, David Solbach inscribed on the eternal scroll:
> > What format is the data encoded in?
>
> to be honest, i don't know, I didn't specify any charsets in the html,
Two things wrong here.
If you don't specify the coding (charset=) of an HTML page, then you
can have no idea what coding the user had set when they submitted the
data. This can cause potentially serious misunderstandings.
But I _think_ the question you're answering at this point was really
asking which forms-submission encoding you are using, rather than
about your HTML character coding.
> or javascript source (if that would be possible)
> the application runs on an apache webserver on a linux box, so pherhaps you
> can tell me which format the data is encoded in? ;)
But the character coding which is relevant to the submission is the
one that the _client_ was using; the server platform is irrelevant to
that. You should (and according to CA-2000-02 there are significant
additional security risks if you fail to do so) specify the character
coding (charset=) of every HTML page which you send out.
When form submission is involved, the browser will also default to
using the character coding of the page in which the form is contained,
so it's important to maintain this correctly (e.g to distinguish
between the original iso-8859-1 and the increasingly popular
iso-8859-15 in the European locale).
> > These german umlauts and french special chars, are they in utf, latin2,
> > or some other format?
>
> again, sorry, how can I find it out and tell you?
For the part of the chain which involves the HTML page, the form, and
its submission to the server (and to the server-side script) I'd refer
you to my page as a starting-point
http://ppewww.ph.gla.ac.uk/~flavell/charset/form-i18n.html
What happens after that is surely evident from your script, no?
> > > But the oracle settings seem to be corrrect, because if I try to write
> > > literals to the DB it works fine.
> >
> > From what program, through DBI, or through Oracle's sql-shell?
>
> both works fine, DBI and sql-shell
What kind of "literals" were these - you mean umlauted letters
actually written into your Perl script source? I think we need to see
some actual Perl code, and also to hear which Perl version is
involved. (Maybe utf-8 encoding is happening to your script
code behind the scenes, but if you want data treated that way, you
gotta ask for it explicitly).
> > The solution is to convert all of the data to a common encoding before
> > putting it in, and then convert it to whatever encoding you want when
> > taking it out.
Agreed.
> > Usually utf8 encoding or utf16 encoding is preferred, though latin1,
> > latin2, etc, ... might suite your data better, depending on the
> > language. But whichever you choose, you *must* store all the strings in
> > the same encoding, otherwise your data will be corrupted.
Indeed.
> ok, thanks four your help, I think I should use the biggest charset, which
> probably is utf16?
If you're going to make use of Unicode then at least please learn the
difference between a character set and a character encoding. utf-16
isn't capable of representing more or less characters than utf-8: they
are both encodings of Unicode.
The Perl documentation pages for 5.8.0 contain some useful background
material to this, start at
http://www.perldoc.com/perl5.8.0/pod/perlunicode.html
> because the application
> shall support as many languages as possible,
Actually, 'language' and 'writing system' are different properties.
I would say that Greek is still Greek (language) even when transcribed
into Roman writing; and English is still Engrish even when written in
Japanese characters. (SCNR!)
> at least english, german and french is a must.
But these are trivially covered by iso-8859-15 or (if you don't mind
upsetting the French by the lack of the oe-ligature) by the
traditional iso-8859-1. There's no need to go beyond a single 8-bit
repertoire if you only want to cover these and other widely-used
Western European languages (leaving aside Greek and Welsh, anyhow ;-)
So nothing conclusive on that point.
best
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 4130
***************************************
