[mythtvnz] Data in freeview.xml.gz goes to end of Aug 3rd

[Stephen Worthington]

On Thu, 2 Aug 2018 12:08:15 +1200, you wrote:

>On 02/08/18 09:17, Hadley Rich wrote:
>> I'm happy to keeping hosting the EPG and fixing things here and there 
>> but I have very little time to commit to it so I would suggest 
>> everyone considers it a less robust source than it used to be.
>>
>> That said, I've managed to sort things for right now. The current 
>> issue appears to be at the source i.e. nothing has changed at this end 
>> but I've managed to work around it for now.
>>
>Following Hads' comments I thought I would change to using the EIT data. 
>The setup seemed to go fine and the Channel editor process was fine but 
>now after running mythfilldatabase there is no data.
>
>Is this caused by the same problem experienced by Hads (and the TiVo EPG 
>providers)?
>
>Is there a "workaround" or should I go back to using Hads' data?
>
>Anyone else have the same situation?

Using the EIT data directly inside MythTV has its problems, even when
it is working.  All those "New:" and "All New" bits on the front of
titles are a big hazard.  You are better off using an external EPG
grabber.  For now, I would recommend going back to Hads' data, and
then considering your options and trying things before making such a
big change.

The available choices are epgsnoop to get the EIT EPG data, or
mhegepgsnoop.py to get the MHEG5 EPG data.  Once you have downloaded
the EPG data, then you can massage it using xmltv-proc-nz to remove
some of those nasty things, and load it into MythTV.  The EIT data and
MHEG5 data seem to be pretty comparable to each other, last time I
looked at both, but that was quite a while ago.  I am using the MHEG5
data as my source, as when I first started doing my own EPG, the MHEG5
data was 7 days and the EIT data just now/next (an hour or two).

I have my version of xmltv-proc-nz on my web server if anyone wants it
- it is adjusted to get settings from my web server on my MythTV box
as well as the external site.  It is available here:

http://www.jsw.gen.nz/mythtv/xmltv-proc-nz-jsw

(rename it to xmltv-proc-nz to use it).

And its config files are here:

http://www.jsw.gen.nz/mythtv/xmltv-proc-nz-json.tgz

You untar the -json file in your /var/www/html directory to create a
json subdirectory with the config files.

I also run an SQL script that does other fix-ups after
mythfilldatabase has been run to load the EPG data into MythTV.  It
does more complex fixes, such as making all the "LivePD" titles on
Duke be in the same format so I do not record multiple copies of the
same episode (update program set title='Live PD: Police Patrol' where
title like "live pd%police patrol%";).  After those fixups are done, I
run mythutill --resched to make the scheduler see the fixes.

The whole EPG processing is now a quite large and complicated script,
due to my having Sky to do as well as Freeview, and it has grown
rather than been designed properly, so I think that using it directly
by other people would not be particularly easy.  But I can post it all
if there are people who want to take a look at it.

My recommendation would be to use mhegepgsnoop.py to get the data, run
xmltv-proc-nz on that data, then run mythfilldatabase to load the
results.

I run a slightly modified version of mhegepgsnoop.py v0.6.2, which has
a couple of extra fixes in it.  It is available here:

http://www.jsw.gen.nz/mythtv/mhegsnoop-0.6.2-jsw.py

(rename it to mhegepgsnoop.py to use it).

And the script and supporting files I use to run it are here:

http://www.jsw.gen.nz/mythtv/do_mhegepgsnoop.sh
http://www.jsw.gen.nz/mythtv/mhegepgsnoop_channel_map.txt
http://www.jsw.gen.nz/mythtv/mhegepgsnoop_sed_replacements

I use it with a external channel map file, due to various problems I
have had with its internal automatic channel mapping mode (like the
Maori macron characters in the channel names).  I run it multiple
times if it crashes, because there is at times EPG data that makes it
crash with a buffer overflow problem, but often that bad data is
temporary and goes away if it is run again immediately.

Again, though, using my script directly is not going to work - you
will need to customise it.  My do_mhegepgsnoop.sh script just creates
downloadable EPG file on my MythTV web server so that all the devices
on my network have access to it.  That file gets downloaded in my main
EPG script and processed there before being loaded with
mythfilldatabase.  And it probably has fixes for problems that have
likely stopped happening for now, like the sed replacements it does.

I could try making a more user friendly and usable version of my
scripts, but at the moment all my free time is being used up getting
tv_grab_dvb_plus working for getting OpenTV EPG from Sky, since Sky
has stopped providing full EIT EPG.  If anyone does need the Sky EPG,
I have tv_grab_dvb_plus working well enough now to do that, so please
email me off list for a copy.