[Templates] UTF8 support and issues

[Andy Wardley]

Mark Proctor wrote:
> Is there something I need to do to tell template toolkit to use utf8?
> Will upgrading to the latest version fix this? We have:

The simple answer is yes, upgrading to Perl 5.8 does seem to solve the 
problem.  You should probably upgrade TT to 2.08 as well, but I don't
think that's part of the problem or solution.

There's no code in TT that I'm aware of that does something "wrong" to 
cause UTF8 support to break.  To the best of my knowledge it's something 
in Perl pre 5.8 which doesn't properly handle UTF8 that causes the problem.
Unfortunately, I don't know what particular Perl feature it is that we're
using that breaks UTF8 support.  

I haven't been able to reliably reproduce the problem.  For example, 
this test works fine for me under 5.6.1 with TT 2.08c.

  use strict;
  use Template;

  my $leon = 'Léon Brocard';
  print "$leon is my friend\n";

  my $tt2 = Template->new();
  $tt2->process(\*DATA, { leon => $leon }) || die $tt2->error();

  __DATA__
  [% leon %] is my friend
  [% INCLUDE orange person=leon -%]
  [% BLOCK orange -%]
  [% person %]'s favourite colour is orange
  [% END %]

Output:

  Léon Brocard is my friend
  Léon Brocard is my friend
  Léon Brocard's favourite colour is orange

(assuming those accented characters manage to survive the transition through 
email... but trust me, it looks fine when I run it here)

> I've been going through the source code trying to understand the flow of
> information, placing print statements everwhere. I have identified in
> the ttc file where you can see the same command, one with correct
> output, one mangled(as part of he output string) 
[...]
>     eval { BLOCK: {
>     print "text3", $stash->get(['validFields', 0, 'itemText', 0]),

Can you post the source template, or as short a fragment of it that
replicates the problem.  Also, the complete ttc file would be useful.

Cheers
A