[mythtvnz] Integrating epgsnoop and mhegepgsnoop

[Stephen Worthington]

On Wed, 20 Mar 2013 15:23:55 +1300, you wrote:

>Greetings all
>
>Last weeks dropout of nzepg prompted me to look at mhegepgsnoop again - it
>works great! I'm trying to merge its output with the DVB-S epgsnoop to
>cover all my listings.

You do realise that the epg.org.nz download has had extra processing
done to it?  It has had xmltv-proc-nz run on it, which fetches json
data from epg.org.nz to do various fixups, such as rename misspellings
that commonly happen in the epg data.  And I think it is also where
the overnight BBC epg for TV1 gets added, by downloading the data from
the BBC web site.  So you really should be doing that to your
mhegepgsnoop and Sky data also.  But if you do, then you will still
need the epg.org.nz site for it to work.  So why not just download the
data from there anyway?

That said, I use a script that can get data from mhegepgsnoop or
epgsnoop for DVB-T and uses epgsnoop for Sky.  It checks to see if the
downloaded data is newer or the snooped data, and uses the latest.  So
if epg.or.nz is producing a bad freeview.xml.gz file, I can manually
run mhegepgsnoop or epgsnoop and then get it processed.  But if
epg.org.nz actually completely down, I still have a problem - I should
work out how to keep a local backup copy of the json data, but that
would be a lot of work.

Also, epgsnoop gets json data from epg.org.nz for some of its
processing.  You can probably turn that off if necessary, but it
improves the quality of the data to have it on.

>First off here is a patch to get the unicode standard the same in both
>files...
>
>
>--- storage/packages/mhegepgsnoop-0.5.1.py      2012-07-11
>21:16:42.000000000 +1200
>+++ /usr/local/bin/mhegepgsnoop-0.5.1.py        2013-03-19
>23:44:49.000000000 +1300
>@@ -1393,7 +1393,7 @@
> def map_channels():
>        for line in chaninfo:
>                if len(line) == 2:
>-                       chan = unicode(line[0], "UTF-8")
>+                       chan = unicode(line[0], "utf-8")
>                        xmltvid = line[1]
>                        channels.append([xmltvid, chan])
>                        ch = unicodedata.normalize('NFKD',
>chan).encode('ascii', 'replace')
>@@ -1500,7 +1500,7 @@
>        # Write the xml doc to disk
>        outfile = open(OUT_FILE, "w")
>        # Add manual declaration and doctype headers because it's tedious
>to do any other way
>-       outfile.write('<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE tv
>SYSTEM "xmltv.dtd">')
>+       outfile.write('<?xml version="1.0" encoding="utf-8"?><!DOCTYPE tv
>SYSTEM "xmltv.dtd">')
>        ET.ElementTree(root_element).write(outfile, encoding="utf-8")
>        outfile.close()
>
>
>and here is my script for getting and merging the two output files. It all
>pretty much works as expected.
>
>
>#!/bin/bash
>
>export HOME="/home/mythtv"
>export PATH=$PATH:/usr/local/bin:/usr/bin:/bin
>
>LISTINGS_SKY="/tmp/listings-sky-$$.xml"
>LISTINGS_DVBT="/tmp/listings-dvbt-$$.xml"
>LISTINGS_ALL=$HOME/public_html/epg/listings-all.xml
>
>epgsnoop --quiet --adapter=1 --lnb=11300 --tune=12671 --tune-retries=7
>--config-dir=$HOME/.epgsnoop | iconv -c -f ISO_8859-1//TRANSLIT//IGNORE -t
>UTF-8 -o $LISTINGS_SKY
>mhegepgsnoop-0.5.1.py -d "/dev/dvb/adapter3/demux0" -p -f
>$HOME/.mhegsnoop/channel_map -o $LISTINGS_DVBT
>
>tv_cat --output $LISTINGS_ALL $LISTINGS_SKY $LISTINGS_DVBT
>gzip -f $LISTINGS_ALL
>
>rm -f $LISTINGS_SKY
>rm -f $LISTINGS_DVBT
>
>
>When I run tv_grab_nz-py I get the following error but no clue as to where
>in the file it occurs :(
>
>INFO - Downloading data from gilks.ath.cx...
>INFO - Done
>INFO - Parsing listings...
>Traceback (most recent call last):
>  File "/usr/local/bin/tv_grab_nz-py", line 278, in <module>
>    time.strptime(element.get('start')[:-6], '%Y%m%d%H%M%S')
>  File "/usr/lib64/python2.7/_strptime.py", line 454, in _strptime_time
>    return _strptime(data_string, format)[0]
>  File "/usr/lib64/python2.7/_strptime.py", line 325, in _strptime
>    (data_string, format))
>ValueError: time data '20130320' does not match format '%Y%m%d%H%M%S'
>
>Can someone assist in more debug to pin down where the error is?

Last time I had a problem like that I had to get the tv_grab_nz-py
script to dump the data it was working on at the time of the error.
That gave me enough to search the xmltv file to find the faulty data
and work out what the problem was with my gathering of the epg data.
So what are your Python skills like?  You could put a try / except and
catch the ValueError and dump data at that point, but what I did was
to just dump all the data as it was read by the script, so the last
data dumped was where the problem was.

>Passing the xmltv file into mfdb manually for each of the sources works OK
>and populates the listings, its just the top level grabber stuff is
>failing.
>
>Cheers

BTW Why do you need to combine the DVB-T and Sky data?  Don't you need
to have two different sources defined in MythTV and get the epg
separately for each?  That is what I found I had to do to make it work
properly.