[Templates] Templates inserting huge files

[Andy Wardley]

Shannon, Bryan wrote:
> [...] the output files that I'm INSERTing are getting larger all the time 
> (hundreds and hundreds of megs)

As you correctly observe, there's no way around the fact that TT builds 
the output into a string without some deep hacking that will break things
all over the place.

The best I can offer is a way to take the problem outside TT...

Instead of inserting the contents of the file you could insert the file
name into the output, suitably wrapped up in some kind of marker which you
can identify for post-processing.  

So instead of:

   [% INSERT $some.big.file %]

you write something like:

   [% bigfile(some.big.file) %]

with a macro defined something like this:

  [% MACRO bigfile(file) GET "<bigfile:$file>" %]

Your output then gets littered with directives like:

  <bigfile:name/of/some/big/file>

Then you take the output from the Template process() method (or write
a Template subclass which provides a wrapper around the process() method)
and write it direct to a file.  While you're doing this, you look for any
'<bigfile:xxx>' directives in the text and insert the contents of the
relevant files, chunk-by-chunk to avoid loading the whole thing into 
memory.

I admit it's a little crazy to writing your own mini-template post-processor 
to work around a limitation in TT, but it shouldn't be too hard and would 
at least give you something that works.

In TT3 (here I go again), the output buffer will be a list instead
of a string.  Directives will be able to push objects and/or subs onto
the output buffer in addition to plain text items.  So you could write
a directive which pushes a "BigAssFile" object onto the output instead
of the file content iteself.  

The objects/subs get called right at the end of the process when TT 
builds the final output.  If you're writing the output to a file then 
your objects will get passed the file pointer and will be able to write 
the BigAssFile content line-by-line direct to disk.

A