[25684] in Perl-Users-Digest
Perl-Users Digest, Issue: 7925 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Tue Mar 29 18:10:24 2005
Date: Tue, 29 Mar 2005 15:10:14 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Tue, 29 Mar 2005 Volume: 10 Number: 7925
Today's topics:
YARQ - Yet another regex question <salvador@progressivetrail.org>
Re: YARQ - Yet another regex question <1usa@llenroc.ude.invalid>
Re: YARQ - Yet another regex question <mritty@gmail.com>
Re: YARQ - Yet another regex question <postmaster@castleamber.com>
Re: YARQ - Yet another regex question <1usa@llenroc.ude.invalid>
Re: YARQ - Yet another regex question <mritty@gmail.com>
Re: YARQ - Yet another regex question <salvador@progressivetrail.org>
Re: YARQ - Yet another regex question <postmaster@castleamber.com>
Re: YARQ - Yet another regex question <postmaster@castleamber.com>
Re: YARQ - Yet another regex question <postmaster@castleamber.com>
Re: YARQ - Yet another regex question <salvador@progressivetrail.org>
Re: YARQ - Yet another regex question <1usa@llenroc.ude.invalid>
Re: YARQ - Yet another regex question <pilkowsk@informatik.uni-marburg.de>
Re: YARQ - Yet another regex question <1usa@llenroc.ude.invalid>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Tue, 29 Mar 2005 18:37:50 GMT
From: sjp <salvador@progressivetrail.org>
Subject: YARQ - Yet another regex question
Message-Id: <2gh2e.15366$Go4.14046@trnddc05>
Hi folks,
I'm parsing through a series of delimited records. Some of the records
use '\t' for the delimiter, and others use '=09' as the delimiter. My
program handles the tab-delimited records fine, but records that use '=09'
have erroneous line breaks after '=' signs, like so:
93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA AVE=09=09NEWBERG=09OR=099=
71320000=098.33=09I=09CA=09PARK RANGER=09=0966.64=09=09=09=09=09=09PARKS & =
RECREATION DIVISION=09
I'd like to remove the '=' and EOL, but 'Sline =~ s/\=\n//g;' fails with
an "Can't modify constant item in substitution (s///) at
/usr/local/bin/mailparse line 18, near "s/\=\n//g;" error
What is the proper way to do it?
Thanks,
SJP
------------------------------
Date: Tue, 29 Mar 2005 18:41:03 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns96288B34813ADasu1cornelledu@127.0.0.1>
sjp <salvador@progressivetrail.org> wrote in
news:2gh2e.15366$Go4.14046@trnddc05:
> I'd like to remove the '=' and EOL, but 'Sline =~ s/\=\n//g;' fails
Is that supposed to be $line?
> What is the proper way to do it?
One way would be to read the error message, then fix the error in the
given location, instead of asking hundreds of people to guess what your
script looks like.
Sinan.
------------------------------
Date: Tue, 29 Mar 2005 18:44:54 GMT
From: Paul Lalli <mritty@gmail.com>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Gmh2e.39627$FB6.10503@trndny09>
sjp wrote:
> Hi folks,
>
> I'm parsing through a series of delimited records. Some of the records
> use '\t' for the delimiter, and others use '=09' as the delimiter. My
> program handles the tab-delimited records fine, but records that use '=09'
> have erroneous line breaks after '=' signs, like so:
>
> 93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA AVE=09=09NEWBERG=09OR=099=
> 71320000=098.33=09I=09CA=09PARK RANGER=09=0966.64=09=09=09=09=09=09PARKS & =
> RECREATION DIVISION=09
>
> I'd like to remove the '=' and EOL, but 'Sline =~ s/\=\n//g;' fails with
> an "Can't modify constant item in substitution (s///) at
> /usr/local/bin/mailparse line 18, near "s/\=\n//g;" error
What the heck is 'Sline'? Are you sure you don't mean $line?
Conceivably, perl thinks that 'Sline' is some sort of constant item.
You are enabling strict and warnings, right?
Also, = is not special in a regexp. There's no reason to escape it.
Beyond that, I don't understand what your actual issue is. How does the
records being delimited by '=09' relate to the records having \n
characters after some = characters?
Paul Lalli
------------------------------
Date: 29 Mar 2005 19:10:40 GMT
From: John Bokma <postmaster@castleamber.com>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns9628860D818B6castleamber@130.133.1.4>
Paul Lalli wrote:
> sjp wrote:
>> Hi folks,
>>
>> I'm parsing through a series of delimited records. Some of the
>> records use '\t' for the delimiter, and others use '=09' as the
>> delimiter. My program handles the tab-delimited records fine, but
>> records that use '=09' have erroneous line breaks after '=' signs,
>> like so:
>>
>> 93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA
>> AVE=09=09NEWBERG=09OR=099= 71320000=098.33=09I=09CA=09PARK
>> RANGER=09=0966.64=09=09=09=09=09=09PARKS & = RECREATION DIVISION=09
>>
>> I'd like to remove the '=' and EOL, but 'Sline =~ s/\=\n//g;' fails
>> with
> > an "Can't modify constant item in substitution (s///) at
> > /usr/local/bin/mailparse line 18, near "s/\=\n//g;" error
>
> What the heck is 'Sline'? Are you sure you don't mean $line?
> Conceivably, perl thinks that 'Sline' is some sort of constant item.
>
> You are enabling strict and warnings, right?
>
> Also, = is not special in a regexp. There's no reason to escape it.
>
> Beyond that, I don't understand what your actual issue is. How does
> the records being delimited by '=09' relate to the records having \n
> characters after some = characters?
=
09
is not
=09
the =xx encoding is used in email (I forgot the name), I would *fix*
that first, and then do the parsing.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html
------------------------------
Date: Tue, 29 Mar 2005 19:25:34 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns962892BF6C491asu1cornelledu@127.0.0.1>
John Bokma <postmaster@castleamber.com> wrote in news:Xns9628860D818B6castleamber@130.133.1.4:
> Paul Lalli wrote:
>
>> sjp wrote:
>>> Hi folks,
>>>
>>> I'm parsing through a series of delimited records. Some of the
>>> records use '\t' for the delimiter, and others use '=09' as the
>>> delimiter. My program handles the tab-delimited records fine, but
>>> records that use '=09' have erroneous line breaks after '=' signs,
>>> like so:
>>>
>>> 93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA
>>> AVE=09=09NEWBERG=09OR=099= 71320000=098.33=09I=09CA=09PARK
>>> RANGER=09=0966.64=09=09=09=09=09=09PARKS & = RECREATION DIVISION=09
>>>
...
>> Beyond that, I don't understand what your actual issue is. How does
>> the records being delimited by '=09' relate to the records having \n
>> characters after some = characters?
>
> =
> 09
>
> is not
>
> =09
But there no such cases in the data the OP posted.
> the =xx encoding is used in email (I forgot the name), I would *fix*
> that first, and then do the parsing.
Base64. The CPAN module MIME::Base64 allows one to convert
Base64 encoded strings. On the other hand, I am not sure
if the data the OP posted really is Base64.
The following seems to satisfy the OP's requirements:
#! perl
use strict;
use warnings;
my $d = q{93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA AVE=09=09NEWBERG=09OR=099=
71320000=098.33=09I=09CA=09PARK RANGER=09=0966.64=09=09=09=09=09=09PARKS & =
RECREATION DIVISION=09};
$d =~ s/=09/\t/g;
$d =~ s/=\n//g;
print $d;
__END__
------------------------------
Date: Tue, 29 Mar 2005 19:26:08 GMT
From: Paul Lalli <mritty@gmail.com>
Subject: Re: YARQ - Yet another regex question
Message-Id: <kZh2e.31552$qN3.27329@trndny01>
John Bokma wrote:
>
> Paul Lalli wrote:
>>
>>sjp wrote:
>>>
>>>My program handles the tab-delimited records fine, but
>>> records that use '=09' have erroneous line breaks after '=' signs,
>>> like so:
>>>93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA
>>>AVE=09=09NEWBERG=09OR=099= 71320000=098.33=09I=09CA=09PARK
>>>RANGER=09=0966.64=09=09=09=09=09=09PARKS & = RECREATION DIVISION=09
>>
>>Beyond that, I don't understand what your actual issue is. How does
>>the records being delimited by '=09' relate to the records having \n
>>characters after some = characters?
>
> =
> 09
>
> is not
>
> =09
>
> the =xx encoding is used in email (I forgot the name), I would *fix*
> that first, and then do the parsing.
There is no instance of
=
09
anywhere in the OP's data. The way it sounds to me is that the OP is
concerned about \n's after *any* = character.
I admit, of course, that I could be quite wrong. But in fact, there is
no instance of any "=\n" anywhere in the OP's data, so I don't think we
can really know what the OP is talking about until the OP himself clarifies.
Paul Lalli
------------------------------
Date: Tue, 29 Mar 2005 19:43:54 GMT
From: sjp <salvador@progressivetrail.org>
Subject: Re: YARQ - Yet another regex question
Message-Id: <_di2e.22452$wL6.5777@trnddc03>
On Tue, 29 Mar 2005 19:10:40 +0000, John Bokma wrote:
> Paul Lalli wrote:
>
>> sjp wrote:
>>> Hi folks,
>>>
>>> I'm parsing through a series of delimited records. Some of the
>>> records use '\t' for the delimiter, and others use '=09' as the
>>> delimiter. My program handles the tab-delimited records fine, but
>>> records that use '=09' have erroneous line breaks after '=' signs,
>>> like so:
>>>
>>> 93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA
>>> AVE=09=09NEWBERG=09OR=099= 71320000=098.33=09I=09CA=09PARK
>>> RANGER=09=0966.64=09=09=09=09=09=09PARKS & = RECREATION DIVISION=09
>>>
>>> I'd like to remove the '=' and EOL, but 'Sline =~ s/\=\n//g;' fails
>>> with
>> > an "Can't modify constant item in substitution (s///) at
>> > /usr/local/bin/mailparse line 18, near "s/\=\n//g;" error
>>
>> What the heck is 'Sline'? Are you sure you don't mean $line?
>> Conceivably, perl thinks that 'Sline' is some sort of constant item.
>>
>> You are enabling strict and warnings, right?
>>
>> Also, = is not special in a regexp. There's no reason to escape it.
>>
>> Beyond that, I don't understand what your actual issue is. How does
>> the records being delimited by '=09' relate to the records having \n
>> characters after some = characters?
>
> =
> 09
>
> is not
>
> =09
>
> the =xx encoding is used in email (I forgot the name), I would *fix*
> that first, and then do the parsing.
You're right, John. I'm parsing a very large email archive file and an
indeterminate number of attachments in the file are encoded
"quoted-printable". So the real issue, I suppose is how to properly
decode an indeterminate number of quoted-printable records from a mail
archive before processing the records contained in that archive.
Thanks for helping me to frame the problem.
------------------------------
Date: 29 Mar 2005 19:45:58 GMT
From: John Bokma <postmaster@castleamber.com>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns96288C07E13B9castleamber@130.133.1.4>
A. Sinan Unur wrote:
> But there no such cases in the data the OP posted.
Yup, classical bad post / wrong example :-D
I think "we" see this every day here?
> The following seems to satisfy the OP's requirements:
>
> #! perl
>
> use strict;
> use warnings;
>
> my $d = q{93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA
> AVE=09=09NEWBERG=09OR=099= 71320000=098.33=09I=09CA=09PARK
> RANGER=09=0966.64=09=09=09=09=09=09PARKS & = RECREATION DIVISION=09};
>
> $d =~ s/=09/\t/g;
> $d =~ s/=\n//g;
If you swap those two, yes.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html
------------------------------
Date: 29 Mar 2005 19:47:53 GMT
From: John Bokma <postmaster@castleamber.com>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns96288C5B2B0DCcastleamber@130.133.1.4>
Paul Lalli wrote:
> There is no instance of
> =
> 09
>
> anywhere in the OP's data.
Of course not, because the OP posted a wrong example :-D.
Does that never happen here?
> The way it sounds to me is that the OP is
> concerned about \n's after *any* = character.
>
> I admit, of course, that I could be quite wrong. But in fact, there
> is no instance of any "=\n" anywhere in the OP's data, so I don't
> think we can really know what the OP is talking about until the OP
> himself clarifies.
My best guess:
=
xx
should become
=xx
and then if xx = 09 it should be replaced with \t
I would have the decoding be handled by a dedicated Perl module.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html
------------------------------
Date: 29 Mar 2005 19:50:56 GMT
From: John Bokma <postmaster@castleamber.com>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns96288CE091855castleamber@130.133.1.4>
sjp wrote:
> On Tue, 29 Mar 2005 19:10:40 +0000, John Bokma wrote:
[ snip ]
>> the =xx encoding is used in email (I forgot the name), I would *fix*
>> that first, and then do the parsing.
>
> You're right, John. I'm parsing a very large email archive file and an
> indeterminate number of attachments in the file are encoded
> "quoted-printable".
Yup, that's the one :-D
> So the real issue, I suppose is how to properly
> decode an indeterminate number of quoted-printable records from a mail
> archive before processing the records contained in that archive.
I am really sure that there are Perl modules that handle this.
<http://search.cpan.org/~gaas/MIME-Base64-Perl-
1.00/lib/MIME/QuotedPrint/Perl.pm>
> Thanks for helping me to frame the problem.
:-) You're welcome.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html
------------------------------
Date: Tue, 29 Mar 2005 20:10:36 GMT
From: sjp <salvador@progressivetrail.org>
Subject: Re: YARQ - Yet another regex question
Message-Id: <0Di2e.21887$Ax.20157@trnddc04>
On Tue, 29 Mar 2005 19:25:34 +0000, A. Sinan Unur wrote:
> John Bokma <postmaster@castleamber.com> wrote in news:Xns9628860D818B6castleamber@130.133.1.4:
>
>> Paul Lalli wrote:
>>
>>> sjp wrote:
>>>> Hi folks,
>>>>
>>>> I'm parsing through a series of delimited records. Some of the
>>>> records use '\t' for the delimiter, and others use '=09' as the
>>>> delimiter. My program handles the tab-delimited records fine, but
>>>> records that use '=09' have erroneous line breaks after '=' signs,
>>>> like so:
>>>>
>>>> 93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA
>>>> AVE=09=09NEWBERG=09OR=099= 71320000=098.33=09I=09CA=09PARK
>>>> RANGER=09=0966.64=09=09=09=09=09=09PARKS & = RECREATION DIVISION=09
>>>>
>
> ...
>
>>> Beyond that, I don't understand what your actual issue is. How does
>>> the records being delimited by '=09' relate to the records having \n
>>> characters after some = characters?
>>
>> =
>> 09
>>
>> is not
>>
>> =09
>
> But there no such cases in the data the OP posted.
>
>> the =xx encoding is used in email (I forgot the name), I would *fix*
>> that first, and then do the parsing.
>
> Base64. The CPAN module MIME::Base64 allows one to convert
> Base64 encoded strings. On the other hand, I am not sure
> if the data the OP posted really is Base64.
>
> The following seems to satisfy the OP's requirements:
>
> #! perl
>
> use strict;
> use warnings;
>
> my $d = q{93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA AVE=09=09NEWBERG=09OR=099=
> 71320000=098.33=09I=09CA=09PARK RANGER=09=0966.64=09=09=09=09=09=09PARKS & =
> RECREATION DIVISION=09};
>
> $d =~ s/=09/\t/g;
> $d =~ s/=\n//g;
Actually, no. With a caveat being a typo in the OP (a cardinal sin, to be
sure, but not uncommon after working on a problem for several hours), your
solution, "$d =~ s/=\n//g;" is essentially the same solution that I
originally posted which does not work.
I suspect that this is because quoted-printable records may use '\r\n' as
EOL rather than '\n'.
The rather inelegant "solution" that I will use, based on John's
suggestion, will be to use conditional when 'quoted-printable' records are
found that will use MIME::QuotedPrint to decode those records before
processing.
Thanks,
SJP.
------------------------------
Date: Tue, 29 Mar 2005 20:11:01 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns96289A774456Aasu1cornelledu@127.0.0.1>
sjp <salvador@progressivetrail.org> wrote in news:_di2e.22452$wL6.5777
@trnddc03:
> I'm parsing a very large email archive file and an
> indeterminate number of attachments in the file are encoded
> "quoted-printable". So the real issue, I suppose is how to properly
> decode an indeterminate number of quoted-printable records from a mail
> archive before processing the records contained in that archive.
CPAN is your friend. Do use it. It is far easier to solve your problems
that way than by posting syntax errors and asking people to figure out
what the *real* issue is.
Sinan.
--
A. Sinan Unur <usa1@llenroc.ude.invalid>
(reverse each component and remove .invalid for email address)
comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: Tue, 29 Mar 2005 22:12:59 +0200
From: Fabian Pilkowski <pilkowsk@informatik.uni-marburg.de>
Subject: Re: YARQ - Yet another regex question
Message-Id: <3atr66F6869s7U1@individual.net>
* sjp schrieb:
>
> I'm parsing through a series of delimited records. Some of the records
> use '\t' for the delimiter, and others use '=09' as the delimiter. My
> program handles the tab-delimited records fine, but records that use '=09'
> have erroneous line breaks after '=' signs, like so:
>
> 93=093=094/1/2004=09=09HARNEY=09JAMES=09808 SITKA AVE=09=09NEWBERG=09OR=099=
> 71320000=098.33=09I=09CA=09PARK RANGER=09=0966.64=09=09=09=09=09=09PARKS & =
> RECREATION DIVISION=09
For me this looks like something written in quoted-printable (as used in
emails). Therefore I recommend the module MIME::QuotedPrint (shipped in
core since Perl 5.7.x, or have a look at CPAN).
use MIME::QuotedPrint;
$line = decode_qp( $line );
regards,
fabian
------------------------------
Date: Tue, 29 Mar 2005 20:17:16 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: YARQ - Yet another regex question
Message-Id: <Xns96289B865BC44asu1cornelledu@127.0.0.1>
sjp <salvador@progressivetrail.org> wrote in
news:0Di2e.21887$Ax.20157@trnddc04:
> The rather inelegant "solution" that I will use, based on John's
> suggestion, will be to use conditional when 'quoted-printable' records
> are found that will use MIME::QuotedPrint to decode those records
> before processing.
Why is doing the right this 'inelegant'?
Sinan
--
A. Sinan Unur <usa1@llenroc.ude.invalid>
(reverse each component and remove .invalid for email address)
comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 7925
***************************************
