[4294] in BarnOwl Developers
[barnowl/barnowl] Make bidi, tag and zero-width space characters
daemon@ATHENA.MIT.EDU (Daniel JB Clark)
Mon Nov 2 18:12:25 2026
Date: Mon, 02 Nov 2026 15:12:18 -0800
From: Daniel JB Clark <notifications@github.com>
Reply-To: barnowl/barnowl <reply+ABCERSNNQRJBX5II2KH5JILLMJRNFEVBNHHQAAAAAFX6ZVPJ@reply.github.com>
To: barnowl/barnowl <barnowl@noreply.github.com>
Cc: Subscribed <subscribed@noreply.github.com>
----==_mimepart_6ae918d236ae1_11112d09422d
Content-Type: text/plain;
charset=UTF-8
Content-Transfer-Encoding: 7bit
humanize only made control characters visible, so characters that reorder text or draw nothing reached the terminal as they are. A sender could make a header look different from what it is: an instance with a right-to-left override that reads reversed, or a name with a zero-width space that looks like another name.
What changed: humanize (oneline and multiline) now shows these as `\x{...}`, highlighted like the other characters it shows, and humanize_short replaces them with `?`: bidi embeddings, overrides and isolates (U+202A-202E, U+2066-2069), invisible mathematical operators and deprecated format controls (U+2061-2064, U+206A-206F), tag characters (U+E0000-E007F), zero width space, word joiner and zero width no-break space. The flags of England, Scotland and Wales (emoji plus tag characters) are kept. humanize and humanize_short also treat a string of bytes as characters (Latin-1), so such a string no longer comes back as bytes, which the C code reads as invalid UTF-8.
Not changed: zero width (non-)joiner, left-to-right, right-to-left and Arabic letter marks, soft hyphen, variation selectors and emoji sequences, which Persian, Indic, Hebrew and Arabic text and emoji need. Still not covered: U+180E and U+FFF9-FFFB.
Cost you should know about: the message body goes through the same pass as the header, so a zero width space in a body is shown too. In Khmer (and some Thai) it separates words, and relay bots put it in nicks, so those texts show `\x{200b}` in the default and oneline styles. The header cannot be separated from the body without humanizing every header field on its own, so this PR does not.
Tested: t/BarnOwl_Style_Humanize.t (new) checks every shown character in both modes and humanize_short, Trojan Source style inputs, ASCII smuggled in tags, the three flags and near misses, about 40 ordinary-text cases (Persian, Hebrew, Devanagari, Bengali, emoji sequences) that must not change, spaces and newlines, byte strings, and end-to-end format_message for default and oneline. `make check` passes on Linux (GitHub Actions, Debian trixie, 856 tests): https://github.com/djbclark/barnowl/actions/runs/41543282908
You can view, comment on, or merge this pull request online at:
https://github.com/barnowl/barnowl/pull/195
-- Commit Summary --
* Make bidi, tag and zero-width space characters visible in humanize
-- File Changes --
M perl/lib/BarnOwl/Style/Default.pm (66)
A t/BarnOwl_Style_Humanize.t (244)
-- Patch Links --
https://github.com/barnowl/barnowl/pull/195.patch
https://github.com/barnowl/barnowl/pull/195.diff
--
Reply to this email directly or view it on GitHub:
https://github.com/barnowl/barnowl/pull/195
You are receiving this because you are subscribed to this thread.
Message ID: <barnowl/barnowl/pull/195@github.com>
----==_mimepart_6ae918d236ae1_11112d09422d
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<p dir=3D"auto">humanize only made control characters visible, so charact=
ers that reorder text or draw nothing reached the terminal as they are. A=
sender could make a header look different from what it is: an instance w=
ith a right-to-left override that reads reversed, or a name with a zero-w=
idth space that looks like another name.</p>
<p dir=3D"auto">What changed: humanize (oneline and multiline) now shows =
these as <code class=3D"notranslate">\x{...}</code>, highlighted like the=
other characters it shows, and humanize_short replaces them with <code c=
lass=3D"notranslate">?</code>: bidi embeddings, overrides and isolates (U=
+202A-202E, U+2066-2069), invisible mathematical operators and deprecated=
format controls (U+2061-2064, U+206A-206F), tag characters (U+E0000-E007=
F), zero width space, word joiner and zero width no-break space. The flag=
s of England, Scotland and Wales (emoji plus tag characters) are kept. hu=
manize and humanize_short also treat a string of bytes as characters (Lat=
in-1), so such a string no longer comes back as bytes, which the C code r=
eads as invalid UTF-8.</p>
<p dir=3D"auto">Not changed: zero width (non-)joiner, left-to-right, righ=
t-to-left and Arabic letter marks, soft hyphen, variation selectors and e=
moji sequences, which Persian, Indic, Hebrew and Arabic text and emoji ne=
ed. Still not covered: U+180E and U+FFF9-FFFB.</p>
<p dir=3D"auto">Cost you should know about: the message body goes through=
the same pass as the header, so a zero width space in a body is shown to=
o. In Khmer (and some Thai) it separates words, and relay bots put it in =
nicks, so those texts show <code class=3D"notranslate">\x{200b}</code> in=
the default and oneline styles. The header cannot be separated from the =
body without humanizing every header field on its own, so this PR does no=
t.</p>
<p dir=3D"auto">Tested: t/BarnOwl_Style_Humanize.t (new) checks every sho=
wn character in both modes and humanize_short, Trojan Source style inputs=
, ASCII smuggled in tags, the three flags and near misses, about 40 ordin=
ary-text cases (Persian, Hebrew, Devanagari, Bengali, emoji sequences) th=
at must not change, spaces and newlines, byte strings, and end-to-end for=
mat_message for default and oneline. <code class=3D"notranslate">make che=
ck</code> passes on Linux (GitHub Actions, Debian trixie, 856 tests): <a =
href=3D"https://github.com/djbclark/barnowl/actions/runs/41543282908">htt=
ps://github.com/djbclark/barnowl/actions/runs/41543282908</a></p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href=3D'https://github.com/barnowl/barnowl/pull/195'>ht=
tps://github.com/barnowl/barnowl/pull/195</a></p>
<h4>Commit Summary</h4>
<ul>
<li><a href=3D"https://github.com/barnowl/barnowl/pull/195/commits/b530=
262f2a3c600e718a676fbcec06a8f0cefafc" class=3D"commit-link">b530262</a> =
Make bidi, tag and zero-width space characters visible in humanize</li>
</ul>
<h4 style=3D"display: inline-block">File Changes </h4> <p style=3D"displa=
y: inline-block">(<a href=3D"https://github.com/barnowl/barnowl/pull/195/=
files">2 files</a>)</p>
<ul>
<li>
<strong>M</strong>
<a href=3D"https://github.com/barnowl/barnowl/pull/195/files#diff-53a=
aefd5b60dc926b6ea112e29812e602750c8528602516818c98c23ac979853">perl/lib/B=
arnOwl/Style/Default.pm</a>
(66)
</li>
<li>
<strong>A</strong>
<a href=3D"https://github.com/barnowl/barnowl/pull/195/files#diff-2e2=
b29dc07ecc4077bcbc95ff83920c780f5454ce2491fdf5a27f9cce9eacb11">t/BarnOwl_=
Style_Humanize.t</a>
(244)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href=3D'https://github.com/barnowl/barnowl/pull/195.patch'>https=
://github.com/barnowl/barnowl/pull/195.patch</a></li>
<li><a href=3D'https://github.com/barnowl/barnowl/pull/195.diff'>https:=
//github.com/barnowl/barnowl/pull/195.diff</a></li>
</ul>
<p style=3D"font-size:small;-webkit-text-size-adjust:none;color:#666;">&m=
dash;<br />Reply to this email directly, <a href=3D"https://github.com/ba=
rnowl/barnowl/pull/195?email_source=3Dnotifications&email_token=3DABC=
ERSJPEHH3HWTN2DXZT4D5XEIFFA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2W=
K43UF42TCMRYHEZDINRSHCTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVRTG633UMVZF6=
Y3MNFRWW">view it on GitHub</a>, or <a href=3D"https://github.com/notific=
ations/unsubscribe-auth/ABCERSJBQPSOJPMZ7SMS2ZL5XEIFFAVCNFSNUABCKJSXA33TN=
F2G64TZHM3TINZTGA4TWSLTON2WKOZWGE3TENZVHE2TEONBOYBA">unsubscribe</a>.<br =
/>Triage notifications, keep track of coding agent tasks and review pull =
requests on the go with GitHub Mobile for <a href=3D"https://github.com/n=
otifications/mobile/ios/ABCERSJCTPL34GHO6ZAQJWD5XEIFFA5CNFSNUABEM5UWIORPF=
5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42TCMRYHEZDINRSHCTHEZLBONXW5KTTOVRHGY3SNF=
RGKZFFMV3GK3TUVJTG633UMVZF62LPOM">iOS</a> and <a href=3D"https://github.c=
om/notifications/mobile/android/ABCERSM4DMVXQNBOOEBWMDD5XEIFFA5CNFSNUABEM=
5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42TCMRYHEZDINRSHCTHEZLBONXW5KTTOV=
RHGY3SNFRGKZFFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE">Android</a>. Download it =
today!
<br />You are receiving this because you are subscribed to this thread.<i=
mg src=3D"https://github.com/notifications/beacon/ABCERSN4V62S3L3ECOH5KYT=
5XEIFFBFCNFSM6AAAAADEC6VOISWGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3J=
MTHQAAAAAFX6ZVPJUZZGKYLTN5XKU43VMJZWG4TJMJSWI.gif" height=3D"1" width=3D"=
1" alt=3D"" /><span style=3D"color: transparent; font-size: 0; display: n=
one; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: =
0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><barn=
owl/barnowl/pull/195</span><span>@</span><span>github</span><span>.</span=
><span>com></span></span></p>
<script type=3D"application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/barnowl/barnowl/pull/195?email_source=3Dnot=
ifications\u0026email_token=3DABCERSLCCEID5N5WHISMGUT5XEIFFA5CNFSNUABEM5U=
WIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42TCMRYHEZDINRSHCTHEZLBONXW5KTTOVRH=
GY3SNFRGKZFFMV3GK3TUVNTW2YLJNRPWG3DJMNVQ",
"url": "https://github.com/barnowl/barnowl/pull/195?email_source=3Dnotifi=
cations\u0026email_token=3DABCERSLCCEID5N5WHISMGUT5XEIFFA5CNFSNUABEM5UWIO=
RPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42TCMRYHEZDINRSHCTHEZLBONXW5KTTOVRHGY3=
SNFRGKZFFMV3GK3TUVNTW2YLJNRPWG3DJMNVQ",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>
----==_mimepart_6ae918d236ae1_11112d09422d--
