REST2: fix Content-Length truncation for non-ASCII JSON responses

[marek-veber]

Problem

JSON::to_json without utf8 => 1 returns a Perl wide-character string
(Unicode code points). Web::Machine computes Content-Length by calling
length() on this string, which counts characters, not bytes. When
Plack::Handler::FCGI outputs the response it encodes the wide string to
UTF-8, producing more bytes than Content-Length declared.

HTTP clients reading exactly Content-Length bytes receive a truncated
JSON body. For example, a Czech-language RT installation returns
["Komentáře přidány"] (27 characters, 31 UTF-8 bytes) with
Content-Length: 27, causing clients to get an unterminated JSON string.

Plack::Handler::FCGI emits a deprecation warning for each such response:

Use of wide characters in FCGI::Stream::PRINT is deprecated and will
stop working in a future version of FCGI at .../Plack/Handler/FCGI.pm

Fix

Pass utf8 => 1 to JSON::to_json so it returns a UTF-8 byte string.
length() then counts bytes correctly, Content-Length matches the actual
body, and no wide-character warning is emitted.

Two callsites affected:

• Resource/Message.pm: success response for comment/correspond POST
• Resource/Record/Readable.pm: GET response for individual records

Other callers (Util.pm, Record/Writable.pm) already use encode_json,
which encodes to UTF-8 bytes by default — those are correct.

Testing

Reproduced with an RT instance with Czech locale (Lang: cs). After the
fix, POST /REST/2.0/ticket/:id/comment returns a complete JSON body and
the FCGI wide-character warning disappears.

[cbrandtbuffalo]

The issue is confirmed. We fixed it with a slightly different change. @marek-veber could you confirm this patch also fixes the issue for you? https://github.com/bestpractical/rt/commit/ecf2371323b83d21c350afc51435ad97a5fdd48d.patch

The full branch with a test is here: https://github.com/bestpractical/rt/tree/5.0/rest2-content-length-utf8

[sunnavy]

5.0/rest2-content-length-utf8 has been merged into 5.0-trunk, closing.

thanks for the report!