[mythtvnz] XMLTV ID and on air guide
Stephen Worthington
stephen_agent at jsw.gen.nz
Tue Aug 29 03:30:54 BST 2017
On Tue, 29 Aug 2017 12:13:33 +1200, you wrote:
>Hi All
>
>I have just performed a scan to correct the changes to Maori TV and have
>encountered a couple of hitches.
>
>First,I have forgotten how to enter activate the XMLTV ID field in the
>Mythbackend channel editor so that it will accept input.Is this the
>correct way to enter the XMLTV ID?
>
>In spite of not being able to enter the XMLTV ID for Maori TV ,I am now
>getting the programme guide populated with data for Maori TV.Before the
>scan I wasn't.
>
>I am using over the air guide data.Is the XMLTV ID not needed when this
>is the EPG data source?
>
>
>Cheers
>
>-Paul
The over-the-air guide data is EIT, and as far as I know does not need
an XMLTV ID specified, as the EIT data is tagged to say which channels
it applies to, and all that is handled internally by mythbackend. You
only need to use XMLTV IDs if you are using XMLTV guide data, normally
gathered by some external mechanism and loaded from a .xmltv file.
Originally, the EIT data on FreeviewHD (DVB-T) was only Now/Next, and
you had to use the MHEG5 data to get proper EPG. But a while ago the
full EIT data was also added as a separate stream alongside the MHEG5
and Now/Next EIT data, so you can choose which you use. I have never
compared the MHEG5 data to the EIT data, so I can not say if they are
identical or not. But they come from the same source, so are likely
to be similar. The problem with using the mythbackend internal
over-the-air option to gather EIT data is that you do not get to
process it before it is used. The downloaded data has some extra
processing done using the xmltv-proc-nz script to remove nasties such
as "All New " and "New: " on the front of titles. MythTV itself
automatically removes "New " from the front of titles.
To use the MHEG5 data, either use the downloaded EPG, or run
mhegepgsnoop and load its data using mythfilldatabase. You can run
xmltv-proc-nz on the data yourself if you do not want to use the
downloaded data. I do that, and also do some more processing for
other nasties. You can also use epgsnoop to download the EIT data,
then run xmltv-proc-nz and load that with mythfilldatabase.
This is what my last run on xmlt-proc-nz found to fix:
SearchReplaceTitle: Changed from "New: Modern Life is Goodish" to
"Modern Life is Goodish"
SearchReplaceTitle: Changed from "New: Behind Bars - Rookie Year" to
"Behind Bars - Rookie Year"
SearchReplaceTitle: Changed from "New: The Blacklist Redemption" to
"The Blacklist Redemption"
SearchReplaceTitle: Changed from "New: Duck Quacks Don't Echo" to
"Duck Quacks Don't Echo"
SearchReplaceTitle: Changed from "New: Taskmaster" to "Taskmaster"
SearchReplaceTitle: Changed from "New: Bar Rescue" to "Bar Rescue"
SearchReplaceTitle: Changed from "New: Duck Dynasty" to "Duck Dynasty"
SearchReplaceTitle: Changed from "New: Swamp People" to "Swamp People"
SearchReplaceTitle: Changed from "New: Hardliners" to "Hardliners"
SearchReplaceTitle: Changed from "All New 2 Broke Girls" to "2 Broke
Girls"
SearchReplaceTitle: Changed from "All New The Middle" to "The Middle"
SearchReplaceTitle: Changed from "All New The Simpsons" to "The
Simpsons"
SearchReplaceTitle: Changed from "All New 2 Broke Girls" to "2 Broke
Girls"
SearchReplaceTitle: Changed from "All New The Middle" to "The Middle"
SearchReplaceTitle: Changed from "All New The Simpsons" to "The
Simpsons"
It is also possible to run an SQL script to do fixups of the guide
data after it is already in the MythTV database. Doing that when
using the over-the-air EIT option will not work well though, as MythTV
will be downloading new EIT data all the time, so you will not know
when there is some new data that needs correcting and be able to run
the SQL script. So using an SQL script is best done when using an
external grabber and you can run the script as soon as you have run
mythfilldatabase. I have put my current SQL script on my web server
if you want to take a look at it:
http://www.jsw.gen.nz/mythtv/epg_fixes.sql
It includes fixes for my Sky channels as well as Freeview. Here is
the output of running the script yesterday:
--------------
DROP PROCEDURE IF EXISTS How_Its_Made_epg_fix
--------------
Query OK, 0 rows affected, 1 warning (0.00 sec)
Note (Code 1305): PROCEDURE mythconverg.How_Its_Made_epg_fix does not
exist
--------------
CREATE PROCEDURE How_Its_Made_epg_fix()
BEGIN
DECLARE p_finished INTEGER DEFAULT 0;
DECLARE v_chanid INT(10) UNSIGNED;
DECLARE v_starttime DATETIME;
DECLARE v_manualid INT(10) UNSIGNED;
DECLARE v_description VARCHAR(16000);
DECLARE v_pos INTEGER UNSIGNED;
DECLARE v_pos_right INTEGER UNSIGNED;
DECLARE v_subtitle VARCHAR(16000);
DECLARE v_right VARCHAR(16000);
DECLARE p CURSOR FOR
SELECT `chanid`,`starttime`,`manualid`,`description`
FROM program
WHERE TITLE LIKE "%how it's made%" AND
subtitle='' AND
(description LIKE '%:Find%' OR
description LIKE '%: Find%' OR description LIKE '%:See%' OR
description LIKE '%: See%')
order by starttime;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET p_finished = 1;
SELECT `chanid`,`starttime`,`manualid`,`description` FROM
program
WHERE title="How It's Made" AND subtitle='' AND
(description LIKE '%:Find%' OR description
LIKE '%: Find%' OR description LIKE '%:See%' OR description LIKE '%:
See%')
order by starttime;
OPEN p;
bad_subtitle_loop: LOOP
FETCH p INTO
v_chanid,v_starttime,v_manualid,v_description;
SET v_pos = INSTR(v_description, ':');
SET v_subtitle = LEFT(v_description, v_pos-1);
IF SUBSTR(v_description, v_pos+1, 1) = ' ' THEN
SET v_pos_right = v_pos+2;
ELSE
SET v_pos_right = v_pos+1;
END IF;
SET v_right=SUBSTR(v_description, v_pos_right);
UPDATE program SET subtitle=v_subtitle,
description=v_description WHERE chanid=v_chanid AND
starttime=v_starttime AND manualid=v_manualid;
IF p_finished != 0 THEN
LEAVE bad_subtitle_loop;
END IF;
END LOOP;
CLOSE p;
END
--------------
Query OK, 0 rows affected (0.00 sec)
--------------
select chanid,starttime,title,subtitle,programid from program where
title like 'All New %'
--------------
Empty set (0.00 sec)
--------------
update program set title=substr(title,9) where title like 'All New %'
--------------
Query OK, 0 rows affected (0.16 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
select chanid,starttime,title,subtitle,programid from program where
title like 'All New: %'
--------------
Empty set (0.00 sec)
--------------
update program set title=substr(title,10) where title like 'All New:
%'
--------------
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
select chanid,starttime,title,substr(title,6) as
title6,subtitle,programid from program where title like 'New: %'
--------------
Empty set (0.00 sec)
--------------
update program set title=substr(title,6) where title like 'New: %'
--------------
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
select chanid,starttime,title,subtitle,programid from program where
title like 'Cleverman %'
--------------
Empty set (0.00 sec)
--------------
update program set title='Cleverman' where title like 'Cleverman %'
--------------
Query OK, 0 rows affected (0.01 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
select
chanid,starttime,title,subtitle,left(description,20),season,episode,category,seriesid,programid
from program where title like '%trans-si%'
--------------
Empty set (0.15 sec)
--------------
update program set programid='' where title="Joanna Lumley's
Trans-Siberian"
--------------
Query OK, 0 rows affected (0.02 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
select
chanid,starttime,title,subtitle,left(description,20),season,episode,category,seriesid,programid
from program where title="Going Deep w' David Rees"
--------------
Empty set (0.00 sec)
--------------
update program set title='Going Deep with David Rees' where
title="Going Deep w' David Rees"
--------------
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
select
chanid,starttime,title,subtitle,left(description,20),season,episode,category,seriesid,programid
from program where title="Rebellion:" and subtitle='The Stuarts In
Exile'
--------------
Empty set (0.00 sec)
--------------
update program set title='Rebellion: The Stuarts In Exile',subtitle=''
where title="Rebellion:" and subtitle='The Stuarts In Exile'
--------------
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
select
chanid,starttime,title,subtitle,left(description,20),season,episode,category,seriesid,programid
from program where title="Grand Tours Of" and subtitle='The Scottish
Islands'
--------------
Empty set (0.00 sec)
--------------
update program set title='Grand Tours Of The Scottish
Islands',subtitle='' where title="Grand Tours Of" and subtitle='The
Scottish Islands'
--------------
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0 Changed: 0 Warnings: 0
--------------
CALL How_Its_Made_epg_fix()
--------------
+--------+---------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| chanid | starttime | manualid | description |
+--------+---------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2050 | 2017-08-29 02:10:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 4050 | 2017-08-29 02:10:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 10070 | 2017-08-29 02:10:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 2050 | 2017-08-29 19:30:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 4050 | 2017-08-29 19:30:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 10070 | 2017-08-29 19:30:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 4050 | 2017-09-01 02:10:00 | 0 | Surgical Instruments, Ketchup, Double Decker Buses, Walking Sticks: See how everything from hand-crafted to factory-made products are put together. |
| 2050 | 2017-09-01 02:10:00 | 0 | Surgical Instruments, Ketchup, Double Decker Buses, Walking Sticks: See how everything from hand-crafted to factory-made products are put together. |
| 10070 | 2017-09-01 02:10:00 | 0 | Surgical Instruments, Ketchup, Double Decker Buses, Walking Sticks: See how everything from hand-crafted to factory-made products are put together. |
| 2050 | 2017-09-02 19:30:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 4050 | 2017-09-02 19:30:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 10070 | 2017-09-02 19:30:00 | 0 | Leather Wallets, French Horns, Soy Sauce, Children's Ride-On Cars: Find out how Leather wallets; French horns; soy sauce; and children's ride-on cars are made. |
| 2050 | 2017-09-03 19:30:00 | 0 | Surgical Instruments, Ketchup, Double Decker Buses, Walking Sticks: See how everything from hand-crafted to factory-made products are put together. |
| 4050 | 2017-09-03 19:30:00 | 0 | Surgical Instruments, Ketchup, Double Decker Buses, Walking Sticks: See how everything from hand-crafted to factory-made products are put together. |
| 10070 | 2017-09-03 19:30:00 | 0 | Surgical Instruments, Ketchup, Double Decker Buses, Walking Sticks: See how everything from hand-crafted to factory-made products are put together. |
| 2050 | 2017-09-04 02:10:00 | 0 | Kitchen Accessories, Vacuums, Paper Mache, Hydraulic Cylinders: See how everything from hand-crafted to factory-made products are put together. |
| 4050 | 2017-09-04 02:10:00 | 0 | Kitchen Accessories, Vacuums, Paper Mache, Hydraulic Cylinders: See how everything from hand-crafted to factory-made products are put together. |
| 10070 | 2017-09-04 02:10:00 | 0 | Kitchen Accessories, Vacuums, Paper Mache, Hydraulic Cylinders: See how everything from hand-crafted to factory-made products are put together. |
| 2050 | 2017-09-04 19:30:00 | 0 | Kitchen Accessories, Vacuums, Paper Mache, Hydraulic Cylinders: See how everything from hand-crafted to factory-made products are put together. |
| 4050 | 2017-09-04 19:30:00 | 0 | Kitchen Accessories, Vacuums, Paper Mache, Hydraulic Cylinders: See how everything from hand-crafted to factory-made products are put together. |
| 10070 | 2017-09-04 19:30:00 | 0 | Kitchen Accessories, Vacuums, Paper Mache, Hydraulic Cylinders: See how everything from hand-crafted to factory-made products are put together. |
+--------+---------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
21 rows in set (0.01 sec)
Query OK, 0 rows affected (0.97 sec)
--------------
DROP PROCEDURE IF EXISTS How_Its_Made_epg_fix
--------------
Query OK, 0 rows affected (0.01 sec)
Bye
For DVB-S, the guide data was always a full EIT feed, although now if
you download from the Sky mux that broadcasts Prime, you can also get
MHEG5 and OpenTV EPG now. I checked the OpenTV data using EPG
Collector on Windows and it is a bit different from the EIT data.
More information about the mythtvnz
mailing list