[Templates] Templates inserting huge files

[Shannon, Bryan]

I was wondering if anyone has a recommendation for this task:

I have a template that inserts many other files (in my particular process it
takes the output files generated from other templates and sticks them
in-place into a new output file).

I like the way this works (when it works) because it is nice and clean.  If
I have a database extract that comes from three separate db tables for which
I want to have three separate templates, then this template would allow me
to nicely wrap up the output from those templates into a single collected
XML document (for instance).

However, the problem is that the output files that I'm INSERTing are getting
larger all the time (hundreds and hundreds of megs), and it's causing the
process to use way too much memory (and sometimes my Perl commits suicide!).

I've only been dabbling in the internals of TT2 for a while, but I realize
that parser basically compiles the template into a sub that builds up an
$output string.  The string is then passed to whatever your output
(string/file/object->print() etc.) is.   This is slurping up the big files
for the INSERTs into memory and then dumping them out.  Looking more at the
internals, I don't see any way around this, short of creating a new
directive just for my needs, but even then it will break all the other cool
stuff (like the way filtering is done, for instance).
I'm kinda stuck here.  If anyone has some advice, I'd be more than happy to
hear it!

-Bryan

My template looks something like this:

--------------------------------------------------

Here is a header for file one:

[% INSERT $someobject.resolves_to_name_of_a_huge_outputfile  %]

Here is another one of my output files:

[% INSERT $someotherobject.another_output_filename %]

Here is my passwd file, hack away  ;-)

[% INSERT '/etc/passwd' %]

--------------------------------------------------






Bryan Shannon
Senior Applications Developer
Tribune Media Services
bshannon@tribune.com