[mythtvnz] Plummeting WAF with DVB-T artifacts

[Stephen Worthington]

On Fri, 29 Nov 2013 12:59:57 +1300, you wrote:

>On 29 November 2013 10:52, Greg Brackley
><lists-mythtv-nz at lucidsolutions.co.nz> wrote:
>> After running MythTV with DVB-S for many years I have upgraded to MythTV
>> 0.27 and cut over to a DVB-T tuner. We are seeing a lot of video corruption
>> in TV scenes with movement (on channels 1, 2 & 3). Typically a head or
>> moving object will pixelate or get fuzzy edges.  This will happen every few
>> minutes. The sound sometimes glitches as well.
>>
>> I'm getting signal from Sugarloaf in Christchurch. Are other people getting
>> a good picture?  I have spoken to a couple of people with different tuner
>> hardware and a different version of MythTV and they are reporting similar
>> issues.  Is it a MythTV thing?  I don't have a reliable non-mythtv tuner to
>> compare against. In theory the MythTV backend is just copying the transport
>> stream to disk, so I am trying to figure out where things are going wrong.
>>
>> I've tried to re-mediate and check off possible problems:
>>
>> I've changed the antenna to a UHF only antenna (02MM-MDU36) - this is higher
>> gain that required
>> The cabling is all properly crimped and sealed
>> I do have some trees that aren't feasible to trim in the line of sight
>> I am using a HDHomerun
>>
>> with the latest firmware (20130328)
>> reasonable signal strength, 100% SNR and symbol error quality
>>
>> my backend recordings are on a NFS drive that can write at greater than 40M
>> bytes/sec
>> the network switch isn't reporting any error frames
>>
>> GbE switch
>> Intel NICs
>>
>> on the frond end, rewinding over the glitch doesn't overcome the issue
>> copying the files to a windows machine and watching with VLC doesn't fix the
>> issue
>>
>> VLC seems to pause through the errors rather than showing pixellation
>>
>> I've tried leaving a ping running against the hdhomerun
>> the backend is:
>>
>> a CentOS v6.4 virtual
>> MythTV 0.27
>> NFS for the video store
>>
>> I don't have a backout plan as the old machine has ECC memory errors, very
>> old disks and the DVB-S tuner is a 5volt only PCI card
>>
>> Any ideas please?
>>
>> Greg.
>>
>> --
>>
>> # hdhomerun_config ffffffff get /sys/version
>> 20130328
>>
>> # hdhomerun_config ffffffff get /tuner0/debug
>> tun: ch=t8qam64:594000000 lock=t8qam64:594000000 ss=93 snq=100 seq=100
>> dbg=-417/-12288
>> dev: bps=26344064 resync=0 overflow=0
>> ts:  bps=26344064 ut=95 te=0 miss=0 crc=0
>> flt: bps=2210880
>> net: pps=210 err=0 stop=0
>>
>>
>> _______________________________________________
>> mythtvnz mailing list
>> mythtvnz at lists.linuxnut.co.nz
>> http://lists.ourshack.com/mailman/listinfo/mythtvnz
>> Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
>>
>
> Do you have another way of checking the signal i.e. a
>freeview-capable TV? If it's glitchy on a TV, the signal is probably
>not good enough for a PC-based solution.
>
>Can you check your setup under another OS e.g. Windows Media Centre?
>If it's not pixellating there, then at least you know the hardware /
>signal side *should* be usable in MythTV.
>
>I had similar dilemmas (although the pixellation and sound glitches
>were every 10-15 minutes) previously and tried everything - checked
>antenna, wires etc. Tried an attenuator. Reformmated hard drives to
>XFS. Pulled hair out. In the end the problem turned out to be that I
>didn't have the correct firmware installed for my DVB-T card. This
>*shouldn't* be a problem for the HD Homerun, but you never know.
>
>Lastly, do the logs tell you anything?

Yes, the logs in 0.27 can now tell you more.  At the end of each
recording, there is a RecordingQuality log line.  Here is an example
of a good recording:

Nov 29 10:34:00 mypvr mythbackend: mythbackend[28422]: I TVRecEvent
tv_rec.cpp:834 (FinishedRecording) TVRec[27]:
FinishedRecording(4049_2013-11-28T20:29:00Z) good
recq:<RecordingQuality overall_score="1"
key="4049_2013-11-28T20:29:00Z" countinuity_error_count="0"
packet_count="7597322" />

The overall_score values is 1 and there are no errors listed.  Here is
an example of a slightly bad recording:

Nov 26 17:31:00 mypvr mythbackend: mythbackend[4101]: I TVRecEvent
tv_rec.cpp:832 (FinishedRecording) TVRec[27]:
FinishedRecording(4095_2013-11-26T03:29:00Z) good
recq:<RecordingQuality overall_score="0.998333"
key="4095_2013-11-26T03:29:00Z" countinuity_error_count="0"
packet_count="9588056">#012    <Gap start="2013-11-26T03:41:20Z"
end="2013-11-26T03:41:21Z" duration="1" />#012    <Gap
start="2013-11-26T03:46:50Z" end="2013-11-26T03:46:51Z" duration="1"
/>#012    <Gap start="2013-11-26T04:06:30Z" end="2013-11-26T04:06:31Z"
duration="1" />#012    <Gap start="2013-11-26T04:08:20Z"
end="2013-11-26T04:08:21Z" duration="1" />#012    <Gap
start="2013-11-26T04:10:21Z" end="2013-11-26T04:10:22Z" duration="1"
/>#012    <Gap start="2013-11-26T04:15:51Z" end="2013-11-26T04:15:52Z"
duration="1" />#012</RecordingQuality>

In this case, the overall_score was not bad enough for MythTV to
declare the recording "damaged".  But there are a number of small one
second gaps in the recording.  And here is one that mythbackend
decided was "damaged":

Nov 15 15:54:00 mypvr mythbackend: mythbackend[13410]: I TVRecEvent
tv_rec.cpp:832 (FinishedRecording) TVRec[27]:
FinishedRecording(4088_2013-11-15T01:50:00Z) damaged
recq:<RecordingQuality overall_score="0.93625"
key="4088_2013-11-15T01:50:00Z" countinuity_error_count="0"
packet_count="7820955">#012    <Gap start="2013-11-15T02:11:50Z"
end="2013-11-15T02:12:21Z" duration="30" />#012    <Gap
start="2013-11-15T02:12:31Z" end="2013-11-15T02:12:41Z" duration="10"
/>#012    <Gap start="2013-11-15T02:12:51Z" end="2013-11-15T02:13:01Z"
duration="10" />#012    <Gap start="2013-11-15T02:13:11Z"
end="2013-11-15T02:13:21Z" duration="9" />#012    <Gap
start="2013-11-15T02:13:31Z" end="2013-11-15T02:13:41Z" duration="10"
/>#012    <Gap start="2013-11-15T02:14:01Z" end="2013-11-15T02:14:31Z"
duration="30" />#012    <Gap start="2013-11-15T02:14:40Z"
end="2013-11-15T02:14:51Z" duration="10" />#012    <Gap
start="2013-11-15T02:15:00Z" end="2013-11-15T02:15:41Z" duration="40"
/>#012</RecordingQuality>

You can see that mythbackend detected quite large gaps in the
recording.  For any "damaged" recording, mythbackend will try to
re-record the programme, if possible, and the recording will be marked
in the list of recordings as damage.  In Mythcenter-wide, it is yellow
coloured.  In the database, the recordedprogram.videprop flags field
has the DAMAGED flag set.

If you use Live TV, there is a problem where almost all the LiveTV
recordings (of short duration anyway - I do not use LiveTV for more
than checking my tuners) will be marked as "damaged".  It looks as
though LiveTV does not shut down the recording process the same way as
with a normal recording and that causes this problem.

However, despite this new RecordingQuality mechanism, it is still
possible to get recordings where there are very short dropouts or
corruptions in the recording that it does not seem to detect.  But it
is still a very useful thing to check for.

With DVB-T and DVB-S recordings, there are typically only a few ways
the recording gets damaged.  The recording process is pretty simple -
the tuner is tuned to the correct multiplex, and then the data for the
channel is selected out of the stream of packets that are received.
That can be done in hardware by the tuner, or in software by the CPU,
depending on what sort of tuner it is.  DVB recordings contain some
redundant data, to protect against transmission errors in packets, so
genre rally small errors will be corrected for without you knowing
they have happened.  When the errors get too big, you start to get
malformed data that can not be recognised as valid packets, and the
bad data is dropped, leaving a gap in the recording.  I think you can
also get recorded packets where the error correction is insufficient
to recover the data, but the packet was able to be recorded, but I
have never verified that.

There are two further places where damage or loss of packets can
occur, firstly in getting from the tuner to the CPU, and then from the
CPU to the hard disk.  With internal or USB tuners, there is usually
no way for packets to be lost or corrupted before they get to the CPU
unless the whole PC is sick (eg memory problems), but with a network
connected tuner, it is possible for the packets to get lost or
corrupted in transit between the tuner and PC.  HDHomeRun tuners send
their packets using UDP protocol, so there is no error correction
mechanism to retransmit missing or damaged packets.  It is therefore
very important that the ethernet connection the tuners are on never
gets swamped, even momentarily, by other traffic.  So it might be a
good idea to run WireShark to check on what traffic is happening
during your recordings.  If you suspect a problem like that, then you
probably need to add an ethernet interface used solely for the tuners.

Again, when storing recordings to local hard disks, there is little
chance of corruption in the process, unless you are using a very slow
hard disk, or the wrong sort of partition type.  The recommended
partition types for recordings are JFS and XFS, although EXT4 is also
reported to be OK.  But when you are storing to drives connected via
ethernet, it is possible to have some problems in transit.  The other
problem possible with recording to local drives is if your recording
partition shares a hard drive with the system or database.  MythTV has
heavy peak disk usage when the scheduler is using the database, and
that needs to be taken into account.  But there should be no problem
recording one programme at once on the same drive as the system and
database if the drive is a 7200 rpm one.

You have previously been recording DVB-S without problems, but you
need to recognise that DVB-T recordings, especially HD ones from TV
One, TV2 and TV3, have much higher bit rates.  An hour of HD recording
(with 1 minute pre-roll and 4 minutes post-roll) can be expected to
take 4-5 Gibytes, where a DVB-S SD recording might well be only 1-2
Gibytes.  So you need to check your bandwidth, and particularly the
possible peak bandwidth that can occur during fast movement on screen.

So the first thing that I think you should try is to record programs
to local hard disk on the mythbackend box.  The hard disk is
presumably small and will fill rapidly, but it should allow you to
isolate where the problem is happening.  You can set up a local
recording partition that is not used by default by going into
mythtv-setup and adding a new storage group, rather than adding the
local recording partition to the "Default" storage group.  Then go to
Manage Recordings > Upcoming Recordings and add an override rule to
tell a recording to use the new storage group instead of "Default".
Hit Enter, select "Add override rule" > "Storage options" and change
the "Store in" setting.  I would suggest setting up multiple HD
recordings at once, and try recording them to local hard disk, and
then to your NFS disks.  You can record from more and more channels at
once until problems occur.

MythTV does not worry about where its recordings are stored as long as
they are in a storage group somewhere, so if you find that you are
running out of NFS bandwidth, you could still store the recordings on
NFS mounts, but only record to local hard drives.  You would do that
by having the NFS partitions in storage groups other than "Default",
and having the only the local hard disks in "Default".  Then you set
up a script that is run automatically as a MythTV "job" at the end of
a recording.  The job script would copy the local hard disk recording
file to one of the NFS mounted drives.

It is also possible that you are having signal problems at the tuners.
You did not say what sort of HDHomeRun tuners you are using, but they
are often dual tuner devices, with an internal splitter.  If so, then
the splitting process means that each of the dual tuners will be
getting less than half the signal strength at the aerial plug into the
HDHomeRun box.  Presumably, your aerial will have a TV on it somewhere
as well as the tuners, and will therefore already have a splitter. So,
for example, let us say you have one TV and one dual tuner, with a two
way splitter.  The TV will then be getting a little under half the
signal from the aerial, but each of the dual tuners will only be
getting less than one quarter of the aerial signal.  If the signal
levels are high then that is not a problem, but if they are at all
marginal, halving the signal level at the tuners can cause problems.
However, your tuners are reporting good signal levels, so I think this
is probably not the cause of your problems.