[30784] in Perl-Users-Digest
Perl-Users Digest, Issue: 2029 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Thu Dec 4 06:09:42 2008
Date: Thu, 4 Dec 2008 03:09:09 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Thu, 4 Dec 2008 Volume: 11 Number: 2029
Today's topics:
Re: FAQ 8.44 How do I tell the difference between error <whynot@pozharski.name>
how detect english subject and predicate in a sentence <jack_posemsky@yahoo.com>
Re: how detect english subject and predicate in a sente <cwilbur@chromatico.net>
Re: how detect english subject and predicate in a sente <jack_posemsky@yahoo.com>
Re: how detect english subject and predicate in a sente <jurgenex@hotmail.com>
Re: how detect english subject and predicate in a sente <theo@polderland.nl>
Re: Linux disk errors - any early indications <james.harris.1@googlemail.com>
Re: Linux disk errors - any early indications <a@b.c>
Re: Mathematica 7 compares to other languages <jurgenex@hotmail.com>
Re: Mathematica 7 compares to other languages <geekmail@usenot.de>
new CPAN modules on Thu Dec 4 2008 (Randal Schwartz)
Re: perl segfault - how to troubleshoot <a@b.c>
Re: perl segfault - how to troubleshoot <james.harris.1@googlemail.com>
Re: Web programming: issues with large amounts og data <tim@burlyhost.com>
Re: Web programming: issues with large amounts og data <r.ted.byers@gmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Thu, 04 Dec 2008 12:20:04 +0200
From: Eric Pozharski <whynot@pozharski.name>
Subject: Re: FAQ 8.44 How do I tell the difference between errors from the shell and perl?
Message-Id: <slrngjfboh.hiv.whynot@orphan.zombinet>
On 2008-12-03, brian d foy <brian.d.foy@gmail.com> wrote:
> In article <slrngjbh51.hm.whynot@orphan.zombinet>, Eric Pozharski
><whynot@pozharski.name> wrote:
>
>> > which is not what the FAQ says. So this does not apply to all shells.
>>
>> What even worst it doesn't apply to some environments:
>>
>> Warning: unknown mime-type for "Hello World\n" -- using
>> "application/octet-stream"
>> Error: no such file "Hello World\n"
>>
>> print (1) - execute programs via entries in the mailcap file
>
> Which shell is that? Notice that the answer is for a shell, not some
> non-shell environment.
B<bash>. Either C<bash ./foo.pl> or C(bash <./foo.pl) produces that
output. The last line (C<print (1) ...>) is of B<apropos>. Sometimes
Debian renames distributed files strange ways (sometimes it removes them
at all). If you (or anyone else) accidentally have B<mime-support>
(that one can be named some other way though) that no-way means that you
have the shell command C<print>.
Checked for consistency: C<bash -c ./foo.pl> falls back to
C<bad-interpreter>.
--
Torvalds' goal for Linux is very simple: World Domination
------------------------------
Date: Wed, 3 Dec 2008 21:49:14 -0800 (PST)
From: Jack <jack_posemsky@yahoo.com>
Subject: how detect english subject and predicate in a sentence
Message-Id: <fef6225f-28dc-4370-9343-0e11c8b42fa3@p2g2000prn.googlegroups.com>
Hi there, just checking to see if anyone knows how to perform this
basic grammar function ? I searched CPAN and found nothing that was
able to in Perl..
Thank you,
Jack
------------------------------
Date: Thu, 04 Dec 2008 00:55:14 -0500
From: Charlton Wilbur <cwilbur@chromatico.net>
Subject: Re: how detect english subject and predicate in a sentence
Message-Id: <8663m08oxp.fsf@mithril.chromatico.net>
>>>>> "J" == Jack <jack_posemsky@yahoo.com> writes:
J> Hi there, just checking to see if anyone knows how to perform
J> this basic grammar function ? I searched CPAN and found nothing
J> that was able to in Perl..
That's because it's not a trivial problem.
Consider: "Time flies like an arrow. Fruit flies like a banana."
In the second sentence, what is the subject?
Charlton
--
Charlton Wilbur
cwilbur@chromatico.net
------------------------------
Date: Wed, 3 Dec 2008 22:07:00 -0800 (PST)
From: Jack <jack_posemsky@yahoo.com>
Subject: Re: how detect english subject and predicate in a sentence
Message-Id: <ea119eb8-b9c2-4a32-abff-e06da7630e6b@d36g2000prf.googlegroups.com>
On Dec 3, 9:55=A0pm, Charlton Wilbur <cwil...@chromatico.net> wrote:
> >>>>> "J" =3D=3D Jack =A0<jack_posem...@yahoo.com> writes:
>
> =A0 =A0 J> Hi there, just checking to see if anyone knows how to perform
> =A0 =A0 J> this basic grammar function ? =A0I searched CPAN and found not=
hing
> =A0 =A0 J> that was able to in Perl..
>
> That's because it's not a trivial problem.
>
> Consider: "Time flies like an arrow. =A0Fruit flies like a banana."
>
> In the second sentence, what is the subject?
>
> Charlton
>
> --
> Charlton Wilbur
> cwil...@chromatico.net
Thanks.. So my goal would be to capture what is the subject of the
sentence, Time, and Fruit, respectively. However you cant always take
the first noun that shows up as the subject for instance: "Despite
accusations, lawyers find a way to evade difficult situations"
Lawyers would be the subject of the sentence.. does that make sense ?
Essentially, identify the TOPIC/subject is what I want to do, for a
given sentence..
------------------------------
Date: Wed, 03 Dec 2008 23:07:59 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: how detect english subject and predicate in a sentence
Message-Id: <aa0fj4d27ogiluimge0a2fnauqiir8lslv@4ax.com>
Jack <jack_posemsky@yahoo.com> wrote:
>> That's because it's not a trivial problem.
>>
>> Consider: "Time flies like an arrow. Fruit flies like a banana."
>>
>> In the second sentence, what is the subject?
>
>Thanks.. So my goal would be to capture what is the subject of the
>sentence, Time, and Fruit, respectively. However you cant always take
You just proved Charlton's point:
In the second sentence the subject is not "fruit" but "fruit flies"
Natural language parsing is a _very_ difficult subject.
jue
------------------------------
Date: Thu, 4 Dec 2008 09:26:46 +0100
From: "Theo van den Heuvel" <theo@polderland.nl>
Subject: Re: how detect english subject and predicate in a sentence
Message-Id: <1228379209.221006@klei.intern.polderland.nl>
"Jack" <jack_posemsky@yahoo.com> wrote in message
news:fef6225f-28dc-4370-9343-0e11c8b42fa3@p2g2000prn.googlegroups.com...
> Hi there, just checking to see if anyone knows how to perform this
> basic grammar function ? I searched CPAN and found nothing that was
> able to in Perl..
>
> Thank you,
>
> Jack
Hi Jack,
This problem is hard. Don't try this at home.
If you have the time and the need is high and you have to do it in Perl , I
would propose that you look at Parser::Recdescent. It allows you to express
you knowledge of the language in terms of a contextfree grammar. You also
need a dictionary of English with part-of-speech information. This allows
you to at least approximate some solution. Note that natural language is not
context free.
Because of the nature of language (subclauses, modifiers), you have to
describe a lot of its rules even if you only want to extract small bits of
information. Words like "there", "and", "one", and "too" defy any attempt at
a quick and dirty solution. There are many more of those.
good luck,
Theo van den Heuvel
does not abide by the rules of context
------------------------------
Date: Thu, 4 Dec 2008 01:16:46 -0800 (PST)
From: James Harris <james.harris.1@googlemail.com>
Subject: Re: Linux disk errors - any early indications
Message-Id: <268b40d3-d1f8-42d2-a6f1-be48cf063c30@v4g2000yqa.googlegroups.com>
On 4 Dec, 08:35, The Natural Philosopher <a...@b.c> wrote:
...
> > 1. Is there a way to tell what file systems are corrupt while the
> > machine is running normally? - I.e. was Linux (Ubuntu) telling me of
> > the faults somewhere?
> > 2. If it was where does it report this?
> > 3. If it wasn't why not??? Fsck knew of faults on some of the file
> > systems on bootup without having to scan the disks for them. If it
> > knows there why not report it sooner?
...
> When this sort f thing started happeing on my Mac OSX, there was so much
> corruption that I ended up reinstalling the OS.
The really odd thing is that there seems to be no corruption - at
least none that I've found so far. Perl modules that failed prior to
fixing the file systems now work. I would have expected (e2)fsck to
fix the structure of the partitions. I'm surprised it has apparently
fixed or recovered the data too. Maybe that's something to do with
using the journalling ext3...? I don't know - but I'm glad it is OK!
James
------------------------------
Date: Thu, 04 Dec 2008 10:39:07 +0000
From: The Natural Philosopher <a@b.c>
Subject: Re: Linux disk errors - any early indications
Message-Id: <1228387147.4213.0@proxy00.news.clara.net>
James Harris wrote:
> On 4 Dec, 08:35, The Natural Philosopher <a...@b.c> wrote:
>
> ...
>
>>> 1. Is there a way to tell what file systems are corrupt while the
>>> machine is running normally? - I.e. was Linux (Ubuntu) telling me of
>>> the faults somewhere?
>>> 2. If it was where does it report this?
>>> 3. If it wasn't why not??? Fsck knew of faults on some of the file
>>> systems on bootup without having to scan the disks for them. If it
>>> knows there why not report it sooner?
>
> ...
>
>> When this sort f thing started happeing on my Mac OSX, there was so much
>> corruption that I ended up reinstalling the OS.
>
> The really odd thing is that there seems to be no corruption - at
> least none that I've found so far. Perl modules that failed prior to
> fixing the file systems now work. I would have expected (e2)fsck to
> fix the structure of the partitions. I'm surprised it has apparently
> fixed or recovered the data too. Maybe that's something to do with
> using the journalling ext3...? I don't know - but I'm glad it is OK!
>
That means that you didn't get MUCH disk corruption. Not beyond fsck's
ability to fix,and the disk image is probably OK now.
The question of what hardware caused the issue still remains though.
> James
------------------------------
Date: Wed, 03 Dec 2008 21:24:13 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: Mathematica 7 compares to other languages
Message-Id: <h9qej49380o45bgbpfmjfqttdc4eiv4cgv@4ax.com>
Kaz Kylheku <kkylheku@gmail.com> wrote:
>On 2008-12-04, Jürgen Exner <jurgenex@hotmail.com> wrote:
>> toby <toby@telegraphics.com.au> wrote:
>>>On Dec 3, 4:15 pm, Xah Lee <xah...@gmail.com> wrote:
>>>> On Dec 3, 8:24 am, Jon Harrop <j...@ffconsultancy.com> wrote:
>>>>
>>>> > My example demonstrates several of Mathematica's fundamental limitations.
>>>>
>>>> enough babble Jon.
>>>>
>>>> Come flying $5 to my paypal account, and i'll give you real code,
>>>
>>>I'll give you $5 to go away
>>
>> if you add "and never come back" then count me in, too.
>
>Really? I will trade you one Xah Lee for three Jon Harrops and I will even
>throw in a free William James.
Well, I've never seen those names on CL.perl.M, so I don't know them.
jue
------------------------------
Date: Thu, 4 Dec 2008 11:11:15 +0100
From: Andreas Waldenburger <geekmail@usenot.de>
Subject: Re: Mathematica 7 compares to other languages
Message-Id: <20081204111115.7cee2ecc@usenot.de>
On Wed, 03 Dec 2008 20:38:44 -0500 Lew <noone@lewscanon.com> wrote:
> Xah Lee wrote:
> > enough babble ...
>
> Good point. Plonk. Guun dun!
>
I vaguely remember you plonking the guy before. Did you unplonk him in
the meantime? Or was that just a figure of speech?
teasingly yours,
/W
--
My real email address is constructed by swapping the domain with the
recipient (local part).
------------------------------
Date: Thu, 4 Dec 2008 05:42:25 GMT
From: merlyn@stonehenge.com (Randal Schwartz)
Subject: new CPAN modules on Thu Dec 4 2008
Message-Id: <KBC7up.1Gw7@zorch.sf-bay.org>
The following modules have recently been added to or updated in the
Comprehensive Perl Archive Network (CPAN). You can install them using the
instructions in the 'perlmodinstall' page included with your Perl
distribution.
Algorithm-PageRank-XS-0.01
http://search.cpan.org/~axiak/Algorithm-PageRank-XS-0.01/
Fast PageRank implementation
----
Algorithm-PageRank-XS-0.02
http://search.cpan.org/~axiak/Algorithm-PageRank-XS-0.02/
Fast PageRank implementation
----
Algorithm-PageRank-XS-0.03
http://search.cpan.org/~axiak/Algorithm-PageRank-XS-0.03/
A Fast PageRank implementation
----
Apache2-WURFLFilter-0.5
http://search.cpan.org/~ifuschini/Apache2-WURFLFilter-0.5/
is a Apache Mobile Filter that manage content (text & image) to the correct mobile device
----
App-Nopaste-0.06
http://search.cpan.org/~sartak/App-Nopaste-0.06/
easy access to any pastebin
----
Bio-Das-ProServer-2.9
http://search.cpan.org/~rpettett/Bio-Das-ProServer-2.9/
----
CORBA-JAVA-2.66
http://search.cpan.org/~perrad/CORBA-JAVA-2.66/
----
Catalyst-Controller-HTML-FormFu-0.03006
http://search.cpan.org/~cfranks/Catalyst-Controller-HTML-FormFu-0.03006/
----
CatalystX-CRUD-Controller-RHTMLO-0.19
http://search.cpan.org/~karman/CatalystX-CRUD-Controller-RHTMLO-0.19/
Rose::HTML::Objects CRUD controller
----
DBIx-Compare-1.4b
http://search.cpan.org/~cjones/DBIx-Compare-1.4b/
Compare database content
----
Data-ParseBinary-0.10
http://search.cpan.org/~semuelf/Data-ParseBinary-0.10/
Yet Another parser for binary structures
----
Data-Rx-Type-PCRE-0.002
http://search.cpan.org/~rjbs/Data-Rx-Type-PCRE-0.002/
PCRE string checking for Rx (experimental)
----
Data-Util-0.31
http://search.cpan.org/~gfuji/Data-Util-0.31/
A selection of utilities for data and data types
----
File-Util-3.26
http://search.cpan.org/~tommy/File-Util-3.26/
Easy, versatile, portable file handling
----
HTML-FormFu-0.03006
http://search.cpan.org/~cfranks/HTML-FormFu-0.03006/
HTML Form Creation, Rendering and Validation Framework
----
HTML-FormFu-Imager-0.03006
http://search.cpan.org/~cfranks/HTML-FormFu-Imager-0.03006/
Imager.pm helpers for HTML::FormFu file uploads
----
HTML-FormFu-Model-DBIC-0.03006
http://search.cpan.org/~cfranks/HTML-FormFu-Model-DBIC-0.03006/
Integrate HTML::FormFu with DBIx::Class
----
HTML-TurboForm-0.28
http://search.cpan.org/~camelcase/HTML-TurboForm-0.28/
----
Kamaitachi-0.01
http://search.cpan.org/~typester/Kamaitachi-0.01/
perl flash media server
----
LWP-Curl-0.01
http://search.cpan.org/~lorn/LWP-Curl-0.01/
LWP methods implementation with Curl engine
----
LWP-Curl-0.02
http://search.cpan.org/~lorn/LWP-Curl-0.02/
LWP methods implementation with Curl engine
----
MultiThread-0.9
http://search.cpan.org/~dspadea/MultiThread-0.9/
----
MySQL-SlowLog-Filter-0.03
http://search.cpan.org/~fayland/MySQL-SlowLog-Filter-0.03/
MySQL Slow Query Log Filter
----
MySQL-SlowLog-Filter-0.04
http://search.cpan.org/~fayland/MySQL-SlowLog-Filter-0.04/
MySQL Slow Query Log Filter
----
MySQL-SlowLog-Filter-0.05
http://search.cpan.org/~fayland/MySQL-SlowLog-Filter-0.05/
MySQL Slow Query Log Filter
----
Net-OpenSSH-0.06
http://search.cpan.org/~salva/Net-OpenSSH-0.06/
Perl SSH client package implemented on top of OpenSSH
----
Net-sFlow-0.08
http://search.cpan.org/~elisa/Net-sFlow-0.08/
decode sFlow datagrams
----
POE-Component-Client-DNSBL-0.10
http://search.cpan.org/~bingos/POE-Component-Client-DNSBL-0.10/
A component that provides non-blocking DNSBL lookups
----
Parse-Gnaw-0.34
http://search.cpan.org/~gslondon/Parse-Gnaw-0.34/
An extensible parser. Define grammars using subroutine calls. Define your own grammar extensions by defining new subroutines. Parse text in memory or from/to files or other streams.
----
Path-Class-URI-0.04
http://search.cpan.org/~miyagawa/Path-Class-URI-0.04/
Serializes and deserializes Path::Class objects as file:// URI
----
SOAP-WSDL-2.00.06
http://search.cpan.org/~mkutter/SOAP-WSDL-2.00.06/
SOAP with WSDL support
----
SVN-Hooks-0.13.9
http://search.cpan.org/~gnustavo/SVN-Hooks-0.13.9/
A framework for implementing Subversion hooks.
----
SWISH-API-Object-0.12
http://search.cpan.org/~karman/SWISH-API-Object-0.12/
return SWISH::API results as objects
----
Subtitles-1.02
http://search.cpan.org/~karasik/Subtitles-1.02/
handle video subtitles in various text formats
----
Text-EmacsColor-0.01
http://search.cpan.org/~jrockway/Text-EmacsColor-0.01/
syntax-highlight code snippets with Emacs
----
Tie-Array-Boolean-0.0.1
http://search.cpan.org/~kyle/Tie-Array-Boolean-0.0.1/
A memory efficient array of boolean values.
----
Util-Any-0.01
http://search.cpan.org/~ktat/Util-Any-0.01/
Export any utilities and To create your own Util::Any
----
Util-Any-0.02
http://search.cpan.org/~ktat/Util-Any-0.02/
Export any utilities and To create your own Util::Any
----
autodie-1.996
http://search.cpan.org/~pjf/autodie-1.996/
Replace functions with ones that succeed or die with lexical scope
If you're an author of one of these modules, please submit a detailed
announcement to comp.lang.perl.announce, and we'll pass it along.
This message was generated by a Perl program described in my Linux
Magazine column, which can be found on-line (along with more than
200 other freely available past column articles) at
http://www.stonehenge.com/merlyn/LinuxMag/col82.html
print "Just another Perl hacker," # the original
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
------------------------------
Date: Thu, 04 Dec 2008 08:35:58 +0000
From: The Natural Philosopher <a@b.c>
Subject: Re: perl segfault - how to troubleshoot
Message-Id: <1228379759.9945.3@proxy02.news.clara.net>
James Harris wrote:
> On 3 Dec, 17:06, xhos...@gmail.com wrote:
> ...
>>> Does this mean that the segfault occurred as a result of the close(8)
>>> call - or at least that close() was the last system call prior to the
>>> fault?
>> No to the first--close(8) completed successfully before the segfault.
>> Yes to the second, close(8) was the last system call prior to the fault.
>> So the fault probably occurred in "user space".
>
> To follow up on this, I shut the machine down and ran memtest86+ to
> check the ram. That checked out OK for all tests.
>
> On restart, however, problems with at disk partitions were found. The
> problems reported included
>
> * Block bitmap differences
> * Free inode counts wrong
> * (Most alarming) Buffer I/O errors from which one can only a) ignore
> and b) force rewrite
> * (Most relevant, perhaps, as it relates to Perl's Socket.so though
> not directly):
>
> /usr/lib/perl/5.8.8/auto/Socket/Socket.so.dpkg-tmp mod time Nov 27,
> 2007
> has 2 multiply-claimed blocks shared with 0 files
>
> I spent a while running through the reported problems and then let (e2)
> fsck do the rest. It took some time but on subsequent reboot the Perl
> problem had gone away. I expected to have to reinstall Socket.so at
> least but so far it seems to be OK now. As scripts run I'll keep an
> eye on them. Hopefully they will all work now.
>
> I've added a Linux group because this has led to other queries:
>
> 1. Is there a way to tell what file systems are corrupt while the
> machine is running normally? - I.e. was Linux (Ubuntu) telling me of
> the faults somewhere?
> 2. If it was where does it report this?
> 3. If it wasn't why not??? Fsck knew of faults on some of the file
> systems on bootup without having to scan the disks for them. If it
> knows there why not report it sooner?
>
> Thanks to all in the Perl group for the education in debugging tools.
> I'll find other uses for them.
>
> James
When this sort f thing started happeing on my Mac OSX, there was so much
corruption that I ended up reinstalling the OS.
I had a bad RAM stick. And possibly a bad disk,. Or maybe the RAM
corrupted the disk. It was small and old, so it got replaced anyway.
------------------------------
Date: Thu, 4 Dec 2008 01:40:54 -0800 (PST)
From: James Harris <james.harris.1@googlemail.com>
Subject: Re: perl segfault - how to troubleshoot
Message-Id: <6eb4de8a-f9dc-4df7-ad25-ba49ecd47397@l42g2000yqe.googlegroups.com>
On 4 Dec, 05:05, Tim Greer <t...@burlyhost.com> wrote:
> James Harris wrote:
...
> > 1. Is there a way to tell what file systems are corrupt while the
> > machine is running normally? - I.e. was Linux (Ubuntu) telling me of
> > the faults somewhere?
>
> fsck, badblocks, smartctl, and various tools for your drive (raid
> specific tools/checks for some raid drives and their health, if you use
> raid, too).
I was thinking more of indications while Linux is running and the
disks are mounted, not of taking them down to scan them. I didn't
explain clearly enough but when restarting Linux the system knew
immediately that some file systems had errors. It didn't have to scan
the volumes to know there were errors. It simply said that file system
X has errors and will be scanned and checked.
If it knew that file system X had errors without scanning it there
must be a data value somewhere - probably in the affected partition -
that indicates error. If it wrote this value when closing the system
down it did so also without scanning the disks. Therefore Linux must
have known _prior to_ shutdown that file system X had errors. If it
did so I was wondering if this information is available to the system
admin prior to shutdown.
Does that make more sense now?
> > 2. If it was where does it report this?
>
> To the output or a log, depending on the tool and option used or
> direction of the output, or if you mean to see any warnings/errors as
> they happen, check dmesg as it happens and the messages log. Other
> logs if you use other tools to check and log automatically.
Checked both of those but cannot see a notification of disk or file
system errors - at least not prior to reboot and running fsck.
>
> > 3. If it wasn't why not??? Fsck knew of faults on some of the file
> > systems on bootup without having to scan the disks for them. If it
> > knows there why not report it sooner?
>
> Ensure your kernel has the proper error logging/debugging enabled and
> you're running the checks manually or automatically with the
> aforementioned tools.
Logging sounds good. I would prefer to avoid debugging as IMHO the
kernel or drivers should report the problem in all cases. It is fairly
important to know if there are file system corruptions, after all.
WRT logging syslog has *.* in syslog.conf. Just auth and authpriv are
set to none. I think that means syslog should contain any disk or file
system error messages but there are none I can see. Perhaps I need to
look for something specific...?
James
------------------------------
Date: Wed, 03 Dec 2008 21:09:11 -0800
From: Tim Greer <tim@burlyhost.com>
Subject: Re: Web programming: issues with large amounts og data
Message-Id: <XBJZk.16784$1s7.16721@newsfe02.iad>
Ted Byers wrote:
> The problem manifests itself as the download terminating and the
> script appearing to freeze.
There can be a number of reasons for this. Please post the relevant
portions of the (Perl) code. This might not be a perl code issue.
--
Tim Greer, CEO/Founder/CTO, BurlyHost.com, Inc.
Shared Hosting, Reseller Hosting, Dedicated & Semi-Dedicated servers
and Custom Hosting. 24/7 support, 30 day guarantee, secure servers.
Industry's most experienced staff! -- Web Hosting With Muscle!
------------------------------
Date: Wed, 3 Dec 2008 23:33:43 -0800 (PST)
From: Ted Byers <r.ted.byers@gmail.com>
Subject: Re: Web programming: issues with large amounts og data
Message-Id: <c87495e9-9d33-4dd9-a5ba-087b1565713b@g38g2000yqd.googlegroups.com>
On Dec 4, 12:09=A0am, Tim Greer <t...@burlyhost.com> wrote:
> Ted Byers wrote:
> > The problem manifests itself as the download terminating and the
> > script appearing to freeze.
>
> There can be a number of reasons for this. =A0Please post the relevant
> portions of the (Perl) code. =A0This might not be a perl code issue.
> --
> Tim Greer, CEO/Founder/CTO, BurlyHost.com, Inc.
> Shared Hosting, Reseller Hosting, Dedicated & Semi-Dedicated servers
> and Custom Hosting. =A024/7 support, 30 day guarantee, secure servers.
> Industry's most experienced staff! -- Web Hosting With Muscle!
Hi Tim,
Thanks.
Here is a simple program that shows this problem:
use strict;
use XML::Twig;
use DBI;
use IO::File;
use POSIX qw(strftime);
use Text::ParseWords;
use LWP::RobotUA;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Request::Common;
use HTTP::Response;
use Date::Manip;
Date_Init("TZ=3DEST5EDT");
my $affiliate;
my $merchant;
my $uid;
my $pwd;
my $opfile;
my $db=3D'test';
my $hostname =3D 'localhost';
my $port =3D '3306';
my $user =3D 'xxxxxxx';
my $dbpwd =3D 'yyyyyyyyyyy';
my $dbh =3D DBI->connect("DBI:mysql:database=3D$db;host=3D$hostname",
$user, $dbpwd, {RaiseError =3D> 1});
my $uri =3D "https://our.datafeed.site/query.php";
my $req_url;
#my $ua =3D LWP::UserAgent->new;
my $now_string =3D strftime "%a %b %d %Y Generic", localtime;
mkdir "$now_string";
chdir "$now_string";
my $start_date =3D $ARGV[0];
my $end_date =3D $ARGV[1];
my $query =3D "SELECT merchant_name, uid, pwd FROM merchants WHERE
is_active =3D 1";
my $sth =3D $dbh->prepare($query) or die $dbh->errstr;
$sth->execute();
while (my $ref =3D $sth->fetchrow_hashref()) {
$merchant =3D $ref->{'merchant_name'};
$uid =3D $ref->{'uid'};
$pwd =3D $ref->{'pwd'};
$req_url =3D "$uri?username=3D$uid&password=3D$pwd&start_date=3D
$start_date&end_date=3D$end_date";
STDOUT->print($req_url);STDOUT->print("\n\n");
$opfile =3D "$uid.xml";
STDOUT->print($opfile);STDOUT->print("\n");
system("..\\generic_download_child.pl $uid \"$req_url\" 1>
$uid.stdout 2>$uid.stderr");
# open(MYOUTPUT,"> $opfile");
# my $response =3D $ua->request(POST "$uri", ['username' =3D> "$uid",
'password' =3D> "$pwd",'start_date' =3D> "$start_date",'end_date' =3D>
"$end_date"]);
# if ($response->is_success) {
# print MYOUTPUT $response->content;
# print $response->content;
# } else {
# print STDERR $response->status_line, "\n";
# }
}
chdir "..";
Note, the string works for short time periods and a handful of
merchants is the presently commented out code is uncommented and the
call to generic_download_child.pl is commented out. But the only way
I could make this work for one week's worth of data was to move the
code that is commented out here into generic_download_child.pl . The
script shown here, when used as a driver or master for
generic_download_child.pl works fine for one week's worth of data,
but some merchants have so much data that even
generic_download_child.pl freezes if I try to get two weeks' worth of
data at a time.
What would you recommend I do to prevent this script from freeezing
while still allowing retrieval of larger chuncks of data?
In general, what would you add to this sort of script to detect any
kind of failure and take appropriate remedial action should a problem
arise? Sometimes the company that is hosting our machine has problems
with their ISP that prevents our machine from connecting to anything,
and there are occassional power issues: thus we are wrestling with
finding ways of reliably automatically detecting problems and redoing
downloads that have been interrupted. What would you be recommending
as best practice? this is a bit new to me as my strengths in software
engineering are more in the realm of high performance number crunching
and mathematical/statistical analysis (I recently completed a robust
empirical model of operational risk applicable to the services we
provide to our clients and used to manage this risk so that no one
client's problems can seriously damage us).
Thanks
Ted
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 2029
***************************************
