[mythtvnz] cron.daily mythfilldatabase environment issue

[Stephen Worthington]

On Mon, 28 Sep 2015 10:28:08 +1300, you wrote:

>On Sun, Sep 27, 2015 at 10:08 AM, Paulgir <paulgir at gmail.com> wrote:
>
>> On Fri, 11 Sep 2015 01:13:48 +1200, Stephen Worthington <
>> stephen_agent at jsw.gen.nz> wrote:
>>
>> On Thu, 10 Sep 2015 20:47:58 +1200, you wrote:
>>>
>>> I have been trying to setup a daily cron job to udpate my listings
>>>> from http://epg.org.nz/freeview.xml.gz
>>>>
>>>> My script runs fine if I run in manually but it fails when run from
>>>> cron with these errors:
>>>>
>>>> 2015-09-10 07:55:52.485142 W  This application expects to be running a
>>>> locale that specifies a UTF-8 codeset, and many features may behave
>>>> improperly with your current language settings. Please set the LC_ALL
>>>> or LC_CTYPE, and LANG variable(s) in the environment in which this
>>>> program is executed to include a UTF-8 codeset (such as
>>>> 'en_US.UTF-8').
>>>> 2015-09-10 07:55:52.485154 W  Cannot locate your home directory.
>>>> Please set the environment variable HOME
>>>> 2015-09-10 07:55:52.485158 E  Failed to init MythContext, exiting.
>>>>
>>>> Here is my script:
>>>> #! /bin/bash
>>>> echo "$(date) : starting epg import " >> /tmp/epggrubber.log
>>>> PATH=/usr/sbin:/usr/bin:/sbin:/bin
>>>> HOME=/home/mythtv
>>>> LANGUAGE=en_NZ:en
>>>> LANG=en_NZ.UTF-8
>>>> wget -O /tmp/freeview.xml.gz http://epg.org.nz/freeview.xml.gz >>
>>>> /tmp/epggrubber.log 2>&1
>>>> gunzip /tmp/freeview.xml.gz >> /tmp/epggrubber.log 2>&1
>>>> mythfilldatabase --update --file --sourceid 2 --xmlfile
>>>> /tmp/freeview.xml --update >> /tmp/epggrubber.log 2>&1
>>>> rm /tmp/freeview.xml
>>>> echo "$(date) : finished epg import " >> /tmp/epggrubber.log
>>>> exit
>>>>
>>>> As you can see I am setting the environment variables for home and
>>>> language.
>>>>
>>>> Anyone know what I need to do to make it work?
>>>>
>>>> Cheers,
>>>> Daniel Hughes
>>>>
>>>
>>> Doing this properly is a tricky problem that took me days to get
>>> working to my satisfaction.  So I will set out what I did and why.
>>>
>>> mythfilldatabase needs to be run by a mythtv group user.  If you want
>>> to run it from a cron job, then you need to run it from a user cron
>>> job, not a system one.  The easiest user to do this from is normally
>>> the one you use for mythfrontend.  The mythtv user is often not fully
>>> set up properly, and its crontab may not work.  Any cron job needs to
>>> have it full environment set up as part of the script that is being
>>> run, as it will not inherit any environment from cron.
>>>
>>> So, login as the mythfrontend user from ssh or a terminal window.  Put
>>> your script in say /home/<mythfrontend user>/bin.  Make sure your
>>> EDITOR variable points to your favourite command line editor (eg vi,
>>> nano), then run the command "crontab -e" which will bring up the user
>>> crontab for that user using $EDITOR.  Edit the crontab to add your
>>> script, save the crontab file and exit.
>>>
>>> Note that manually trying to edit the user crontab file does not work
>>> properly - you need to use the crontab command to do it.
>>>
>>> I use two scripts for doing my EPG.  By setting things up with the two
>>> scripts, the first script can log all the output of the second script
>>> to a log file, which makes it easy to see exactly what has gone wrong
>>> when the EPG processing fails.  My first script (epg_temp) is run by
>>> cron, and sets up the environment for the second script, and also
>>> rotates its log file.  The second (epg_temp.sh) does my EPG processing
>>> and runs mythfilldatabase.  The cron script (epg_temp) uses a helper
>>> script logrotate.sh that has been added to sudoers so that it can
>>> rotate the logs.
>>>
>>> I have put epg_temp and logrotate.sh on my web server so you can see
>>> how they work:
>>>
>>>   http://www.jsw.gen.nz/mythtv/epg_temp
>>>   http://www.jsw.gen.nz/mythtv/logrotate.sh
>>>
>>> The logrotate.sh file lives in my /home/stephen/bin directory, but is
>>> chown root:root in order to be able to access the log files in
>>> /var/log/mythtv.  You also need a logrotate.conf file in
>>> /home/<mythfrontenduser>/.logrotate, which also needs to be chown
>>> root:root:
>>>
>>>     http://www.jsw.gen.nz/mythtv/logrotate.conf
>>>
>>> There needs to be a file using the name of the mythfilldatabase user
>>> in /etc/sudoers.d to allow logrotate.sh to work.  This is what I have
>>> in my /etc/sudoers.d/stephen file:
>>>
>>> stephen ALL=NOPASSWD:/home/stephen/bin/logrotate.sh
>>>
>>> The sudoers file needs to be chown root:root and readonly for user and
>>> group only, or it will not work.  It can not have any write
>>> permissions at all.
>>>
>>> I have not posted my epg_temp.sh file, as it is too specific to how I
>>> do EPG and not a good example to follow (it really needs to be
>>> re-written).
>>>
>>> _______________________________________________
>>> mythtvnz mailing list
>>> mythtvnz at lists.linuxnut.co.nz
>>> http://lists.ourshack.com/mailman/listinfo/mythtvnz
>>> Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
>>>
>>
>> I run the following:
>>
>> #! /bin/bash
>> wget http://epg.org.nz/freeview.xml.gz
>> gzip -d freeview.xml.gz
>> mythfilldatabase --update --file --sourceid 1 --xmlfile freeview.xml
>> rm freeview.xml
>> exit
>>
>> It's been working for several years
>>
>> -Paul
>>
>>
>Why when we have tv-grab_nz=py which works with mythbackend. No cron, so
>environment problems.

I have to do Sky EPG as well, and need to have mythfilldatabase run at
a known time, as it can cause errors in my Sky recordings due to high
database activity.  I also want to be able to run mhegepgsnoop if
there is a problem getting the downloaded EPG.  My scripts do all
that, and also allow me to use epgsnoop on the Freeview DVB-S channels
to get EPG if I should ever need it.  The scripts also put the fully
processed EPG data on my internal web server, from where it is
downloaded by my mother's MythTV box, my laptop MythTV and MediaPortal
on my Windows box.