[32298] in Perl-Users-Digest
Perl-Users Digest, Issue: 3565 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Mon Dec 12 16:09:18 2011
Date: Mon, 12 Dec 2011 13:09:03 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Mon, 12 Dec 2011 Volume: 11 Number: 3565
Today's topics:
POSIX vs. filename encoding <oneingray@gmail.com>
Re: ways to check for octets outside of the safe ASCII <rweikusat@mssgmbh.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Mon, 12 Dec 2011 11:35:09 +0700
From: Ivan Shmakov <oneingray@gmail.com>
Subject: POSIX vs. filename encoding
Message-Id: <86y5uie7b6.fsf_-_@gray.siamics.net>
>>>>> Jürgen Exner <jurgenex@hotmail.com> writes:
>>>>> Shmuel (Seymour J.) Metz <spamtrap@library.lspace.org.invalid> wrote:
[Somehow, I believe that this discussion is more appropriate for
news:comp.unix.programmer. Set Followup-To: there.]
[…]
>> BTW, does POSIX limit file names to ASCII, or are, e.g., ISO-8859-1
>> accented letters, allowed?
> AFAIK (and I may be wrong) POSIX supports any Unicode file name.
> Therefore the OPs approach to look at isolated octets is a sure way
> to ask for trouble.
AIUI, POSIX filenames are arbitrary octet strings. They can be
in any encoding (e. g., ISO-8859-1, UTF-8, koi8-r) as long as it
doesn't make use of the \000 octet (i. e., UCS-16, UCS-32,
etc. cannot be used; which is, roughly, the very reason behind
UTF-8.)
In particular, it's perfectly possible for different users of
the same multi-user system (and filesystem) to stick to
different encodings. The software they use will interpret the
filenames according to the locale settings in effect for that
particular user (or, actually, for that particular application.)
Which may, indeed, fail if one user will try to access different
users' files without tweaking his or her locale to match the
other user's preference.
(That's why my software has to be encoding-agnostic.)
--
FSF associate member #7257
------------------------------
Date: Sun, 11 Dec 2011 22:21:44 +0000
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: ways to check for octets outside of the safe ASCII range?
Message-Id: <87mxayoikn.fsf@sapphire.mobileactivedefense.com>
Ben Morrow <ben@morrow.me.uk> writes:
> Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
>> Ben Morrow <ben@morrow.me.uk> writes:
>> > Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
>> >
>> > These people are not 'laymen'. They are the people writing perl.
>>
>> Unfortunately, these are not two mutually exclusive attributes: Just
>> because someone happens to be in the position where his possibly weird
>> personal ideas flow into 'open source projects' doesn't necessarily
>> mean that this someone has a clue about anything except 'effective
>> social networking'.
>
> Insulting people whom you have never met but who have put a large amount
> of time and effort, for free, into a software project you find useful
> is, by my standards, extremely rude.
I didn't address anyone in particular. People who feel insulted by
the fact that 'raising to a dominant position in some kind of social
systems' requires 'working the system' probably know very well why
they ought to behave as if they had been insulted by that. Given that
the black arts of reading and writing where invented a while ago,
there's no need to meet people in order to form an opinion on their
opinions. Someone who spends his spare time on changing a software
product which is useful to me in ways which make it less useful to me
can't exactly expect me to be grateful for that. Lastly, publishing an
extremely condescending fictional dialogue between the very wise
master and the really obtuse disciple as technical documentation, even
claiming this wouldn't be an exercise in fiction writing but a list of
frequently asked questions with answers, like the so-called 'Perl
unicode FAQ' is very rude by my standards, especially considering that
the opinion trying to hide behind the dialogue is more than
questionable.
Two simple examples:
1. Assuming that some large amount of data is sent through a
multistage processing pipeline in order to transform the input data to
some kind of output data, the sensible way to deal with externally
imposed encodings is obviously 'decode and re-encode the input data from
the external format to an internal format common to all tools making
up the pipeline as step 1 and after n processing steps, transform it
back to the external format' *and* *not* having 'decode from external
to internal/ encode from internal to external' as part of each
processing step' since this needs 2n extra copies of the complete data
with n being the length of the pipeline.
2. The internal encoding used by Perl doesn't "happen" to be UTF8,
that's the result of a conscious design descision someone other than the
author of this mock FAQ made in the past. Likely, this was the same
person who also chose to expose this encoding the Perl code. Given
that it is necessarily exposed to any Perl extension written in some
other language, it is part of the public API of Perl in any case and
enabling code written in Perl seems like a feature which could be nice
to have.
>> I think you need to do that. Perl scalars can store binary
>> data. Binary data is stored in Perl scalars. This requires the ability
>> to treat the scalar as a sequence of otherwise lanuage uninterpreted
>> bytes.
>
> I am attempting to describe the current (and past, and likely future)
> behaviour of perl, in the real world. In the real world, bytes.pm is
> broken, has always been broken (regardless of what its documentation
> used to say), and isn't going to get fixed.
As far as I could determine, "bytes.pm is broken" is code language for
"it enables code written in Perl to access the raw octets making up a
character string and the person who made this design descision really
should have made a different descision", as usual, no reason for that
supplied.
[...]
>> But I'm really referring to binary data. And 'the thread' is
>> suspiciously silent on that except that 'bytes is bad because it
>> enforces byte semantics and someone might use that erroneously !!1'.
>
> *No*. The official position (insofar as such a thing exists) is
> 'bytes.pm is bad because it *does* *not* properly enforce byte
> semantics'. In particular, it is impossible in practice to prevent
> SvUTF8 scalars from leaking into a section of code under 'bytes', and at
> that point you get ill-defined and rather uninituitive behaviour.
It doesn't do what the person (or persons) who don't want Perl code to
have access the raw octets making up a character string would have
liked to have instead of it. Instead of detailing what that would be
and justifying why this is sensible despite the original design quite
obviously included this feature, they've chosen to call what they
don't want 'improper'.
[...]
>> including the quite useful feature that Perl
>> scalars can function as extensible arrays with automatic memory
>> management but without any other 'magic semantics' attached to that?
>
> Perl has always had such a facility. It's called 'an array'.
I was using the term 'array' in the sense it is used in C --- to refer
to an ordered set of objects of some type which are contiguous in
memory. This is necessary to deal with binary data moving 'through'
Perl programs.
>> Or is that another hobby horse of the former 2nd and 3rd line people
>> who never managed to force their weirdo politics onto language users
>> before Mr Larry Wall finally grew tired of them? [conjecture]
>
> What, exactly, are you referring to when you mention 'weirdo politics'?
> I fail to see what anything political has to do with any of this.
This whole hullabaloo about 'the character string abstraction' is
about enforcing a certain language usage policy and this somewhat in
contrast to one of the great features of Perl5 as it was conceived --
enable people to get stuff done but don't tell them what stuff they
are allowed (and disallowed) to do and what procedures for doing them
they are allowed (and disallowed) to employ.
The "there's one way to do it RIGHT and we'll force you to stick to
it" language used to be Python ...
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 3565
***************************************
