[mythtvnz] System freezes thrashing HDD

[Stephen Worthington]

On Mon, 5 Apr 2010 11:51:07 +1200, you wrote:

>On Mon, 05 Apr 2010 10:25:22 Aaron Whitehouse wrote:
>> Hello all,
>> 
>> I've spent a long time trying to narrow this down, but I'm really 
>> struggling and could do with any help/suggestions that you can come up with.
>> 
>> I have run Mythbuntu for quite a long time. We used 9.10 without too 
>> much trouble. After some updates near the beginning of this year, 
>> Mythbuntu started stalling for 1-3 minutes at a time, with the hard disk 
>> light on solidly and nothing else working. The system would be virtually 
>> unresponsive, but would slowly crawl to a terminal etc. Then, after 
>> about 1-3 mins, it would be back to normal. I can't see anything that 
>> could be causing the problem in top, iotop or the system logs. There are 
>> some entries in the logs about different kernel tasks being killed 
>> because they didn't start properly in 120 seconds, but I think that is a 
>> symptom rather than a cause.
>> 
>> I have now changed the card to an nVidia and am using the proprietary 
>> drivers. I've removed one of the tuner cards. I upgraded the system to 
>> Lucid Beta and downloaded all the latest updates. The problem is still 
>> occurring, even when all I was doing was browsing folders in Thunar, so 
>> that suggests to me that it isn't MythTV causing this (mythbackend is 
>> still running, but I haven't loaded in the database or set anything up). 
>> To give an idea of the problem, running a 42 minute video in VLC took 53 
>> mins with the stalls.  The memory looks okay (I have 2GB in it), with 
>> free -m showing most "used", but "cached".
>> 
>> I haven't yet sorted out all of the VDPAU etc., but I'm guessing that 
>> those are unlikely to solve the problem when it stalls the system so 
>> much that even a terminal can't work.  Given that it isn't showing in 
>> top/iotop, I'm wondering if it is something really low-level. On the 
>> other hand, I would have thought it would need to be hardware-specific 
>> or everyone would be complaining.
>> 
>> If anyone can help me, it would make a huge difference -- even if only 
>> to suggest more diagnostics or logs to review.
>> 
>> Thanks in advance,r
>> 
>> Aaron

>I had problems that were kind of simlar to what you describe after I went to 
>9.10/0.22 and moved to DVB-T. Turned out to be mythcommflag when processing HD 
>files. I could never get it to work without bringing everything down so have 
>turned mythcommflag off (sadly).

I have had similar problems with the menus in mythfrontend becoming
unresponsive for long periods.  Once I am playing a program or video
in mythfrontend, it generally does not stall though, nor does mplayer
run from mythfrontend - maybe VLC is not running at high enough
priority and that is why it is stalling while playing.  I have found a
number of problems and have been able to make my system usable,
although the problem still happens to a much lesser extent.  In my
case, I have my system and data on one 1 Tbyte Samsung drive, with my
swap on another 500 Gbyte drive - if your system, swap and recording
storage is all on one drive, you will be even more badly affected if
you have the same problems as me.  BTW I do not have an external drive
activity LED to check, so I can not say if that symptom matches.  I
have 2 Gibytes of RAM like you, Nvidia 8400GS and Athlon X2.

These are the things I have found:

1) The number one culprit in my case was mythcommflag.  I had it set
to run two mythcommflag tasks at once (since each only uses one CPU
core and I have a dual core CPU).  But they were set to high priority
in the MythTV settings.  Changing them to low priority has been the
most important fix.  It seems that mythcommflag uses much more
resources when running on DVB-T recordings, especially HD recordings -
I used to have no problems with two high priority mythcommflag tasks
at the same time when there were only MPEG2 recordings from my
Hauppauge PVR-500 cards.  I think that running just one mythcommflag
task at once would probably fix the remaining stalls that I still
have, but I prefer to run two still as otherwise it would take too
long to process everything.

2) mythfrontend.re seems to have a memory leak.  If I let it run for
too long, it takes more and more memory and eventually everything
starts swapping.  So now I shut it down every day or so.

3) I am running KDE instead of XFCE as my desktop.  It seems that
KDE's plasma-desktop task has a massive memory leak.  I have seen it
using more than 1 Gibyte of RAM!  So I also shut it down regularly
now, usually daily.

4) Firefox also seems to have a memory leak.  If I leave it running
for a long time with lots of tabs open, it can also get to be using
more than 500 Mibytes of RAM.  It takes longer for this to happen
though.

BTW Turning off mythcommflag completely is also a reasonable option at
the moment.  It does not work on DVB-T recordings at all - the points
that it jumps to when skipping ads are *never* correct.  This may be
fixed in MythTV 0.23 (Mythbuntu 10.04), but I have yet to test that. I
have not done this as I do want to flag all my Sky recordings.

With these fixes, I can happily be recording four programs at once and
playing back another, with two mythcommflag tasks running.  The menus
are slow to respond, but not so slow as to be unusable.  I think it
would be much better again if I moved my system partition (and the
MythTV database) to a different drive, so I am looking at trying that
when (if) I find the time.  In the mean time, it is usable instead of
having to wait for minutes for a menu to respond.  A further fix would
be to add another drive for the recordings, and set mythbackend to
spread recordings across both drives.  Then things would most likely
be very responsive even with four programs recording at the same time.