Issue 46: "Log debugging information in home directory" thrashes Spotlight
Status:  Fixed
Owner:
Closed:  Dec 2008
Reported by snov...@gmail.com, Nov 29, 2008
When enabled, iTivo logs debugging messages to ~/iTivo.log, and it re-saves the file each time a 
new line is written, which can be many times a second during encoding.  Meanwhile, each time 
the file is saved, Spotlight tries to re-index it — several times a second.  As a result, if you turn 
on debug logging and then start a download, Spotlight's mds and mdworker processes go nuts 
and start eating up tons of processor usage — which, aside from being bad in its own right, also 
takes away from the processor cycles available for the encoder (although I've set up x264 
encoding options to only use a single thread, as my MBP isn't thrilled by the thrashing it gets 
even just from mencoder doing x264 encoding full-bore).

There are a couple ways to deal with this, though none ideal.  One would be to have iTivo keep 
track of debugging messages internally and only write them to file once a queue item is finished 
(or cancelled).  Another would be to tell Spotlight not to index the iTivo log file, except Spotlight 
doesn't allow you to exclude individual files, just directories.  Excluding a user's entire home 
directory from Spotlight indexing would be a bad idea, but perhaps save the file to its own 
directory.  Alternatively, I believe that Spotlight doesn't index certain hidden files, such as files 
whose names start with a dot, or you could change the extension to something unique that 
doesn't have a Spotlight indexer.  Or perphaps you might even save the debug file to something 
within the iTiVo package, on the assumption that anybody who should be tracking debug 
messages will also know how to view package contents.
Nov 29, 2008
Project Member #1 yoav.yer...@gmail.com
Gah.. ok looking into it (although in general you shouldn't have it run with
logging turned on as it spews a lot of crap to that file).   Unless you're debugging
something that is  :)

Status: Accepted
Nov 29, 2008
#2 snov...@gmail.com
I've been playing around with the internals of your code a bit (and Tivo Butler's as well – both appear to be 
converging in functionality, but for now they've got some different strengths and weaknesses).  Right now I'm 
looking to switch the encoder over from the custom build of mencoder to the latest version of HandBrake 
released last week (through its command-line interface), which is much better maintained and has some nice 
bells and whistles (like multithreaded deinterlacing).  I've also run into some odd quirks with mencoder, such 
as not being able to get an audio bitrate lower than 128 kbps when using -oac faac, although I can get it 
down to 64 kbps if i use -oac lavc and tell lavc to use acodec=libfaac.

The only disadvantage of HandBrake that I've found is that it doesn't currently support ComSkip's EDL files.  
But that can be mostly rectified by removing the commercials before encoding with "mencoder -of mpeg -oac 
copy -ovc copy" (with one minor catch: you lose -hr-edl-seek, so cutpoints might be a few frames less 
precise).  I've been looking through the HandBrake source this weekend, and it looks like it might not be 
overly difficult to add in EDL support -- just tell sync.c not to pass the frame to the encoder if its PTS falls in 
a range that the EDL says to exclude.  That would also make it possible to add a chapter marker at the start of 
each segment of a show, which would be handy for navigating through a show.
Nov 29, 2008
Project Member #3 yoav.yer...@gmail.com
Hey tivo butler looks awesome  :)

  Last I used it it was a pretty lame one-window thing, which was why I put the
effort into tivodecodemanager..  hmm .. maybe I'll just point people at butler..

Nov 29, 2008
Project Member #4 yoav.yer...@gmail.com
as for switching encoder.. how big is the handbrake one?  I wasn't even aware it was
possible to run it as an external encoder.  There's 'crappy' support for using a
different encoder (yes the code is a hack)..  So if you can point me at something
relevant to using the handbrake encoder, I can add support for it directly..

Nov 29, 2008
#5 snov...@gmail.com
And meanwhile, the TB developer recently told me he had no idea that tivodecodemanager was taken over under 
a new name by a new developer, and that iTivo looks awesome :-)  Like I said, both programs seem to have been 
converging in a similar direction.  I'd wonder if you might be able to merge the two into one joint effort, rather 
than having to spend a lot of time implementing things twice, but it looks like your codebases are very different 
(mostly Objective-C versus mostly Applescript).  I've actually made use of both -- often TB for the nice interface, 
but set to use the newer build of mencoder from iTivo and a variation on its x264 encoding settings; but at other 
times using iTivo when I want the built-in comskip or to download and encode in one step (especially for large 
movies; I'm a little low on disk space).
Nov 29, 2008
#6 snov...@gmail.com
For documentation of HandBrake's command-line interface, see http://trac.handbrake.fr/wiki/CLIGuide -- I 
think it's a relatively new feature (it's clear from the source that the GUI came first, and the CLI was tacked on 
later as a bit of a hack; the file with the CLI routines is still called test.c).  And HandBrake only added support for 
reading from movie files, rather than just from DVDs, with the new release on Sunday.  I haven't been able to 
play around with it yet -- I've been away from home and my Tivo during the holiday weekend here.  The 
argument names and formats are pretty different from mencoder's, so that'll take a bit of tweaking.
Nov 30, 2008
Project Member #7 yoav.yer...@gmail.com
Ok to resolve the main query, as of the current beta, log files are put in
  /tmp/iTiVo.log
  and /tmp is ignored by spotlight in most installs.

  As for handbrake.  I just looked at it, and it looks like I'm gonna need to spend a
while figuring it out.  I don't want to include the whole encoder in iTiVo, but may
want to allow people to use it if they have it installed...  It looks like they ALSO
use libx264 for h.264 encoding, so it might not give any improvements in encoding,
although their scaling/deinterlacing code might be faster...
Nov 30, 2008
#8 snov...@gmail.com
Interestingly, it looks like the HandBrake executable (all you need for the command line) is 11.2 MB, whereas 
the mencoder binary you're currently including is 17.1 MB.

I don't think HandBrake offers any encoding options that aren't already available through mencoder.  But the 
nice thing about HandBrake is that it's an extremely popular OS X app which is very well maintained and 
updated frequently.  mencoder, by contrast, doesn't offer any official binaries, and the latest OS X binaries are 
an old release that's even marked as "outdated" on the official website.  In fact, the publicly available builds of 
mencoder don't even support x264, so I take it you compiled a recent version yourself for inclusion in iTivo.  
(And even that seems buggy -- try setting the audio to encode with AAC at a bitrate less than 128 kbps; no 
matter what I try, even 32 kbps, it seems to give me 128 kbps audio in the output.)  Moreover, finding good 
documentation of how to use mencoder and all the many different encoder options is extremely difficult (just 
try!), whereas HandBrake has the best documentation of various video encoder settings I've seen anywhere.  
All told, HandBrake would be much easier for your users to use and tweak encoder settings for themselves, 
and to make sure they've got an up-to-date build without depending on you to continue to produce new 
custom builds of mencoder.
Nov 30, 2008
#9 snov...@gmail.com
You'll also notice that HandBrake regularly updates their libx264 with the latest enhancements and bugfixes (see, 
e.g., http://trac.handbrake.fr/changeset/1928) -- so you might find that HandBrake does offer better and faster 
H.264 encoding than you currently do
Dec 2, 2008
Project Member #10 yoav.yer...@gmail.com
Ok, so:
  I added handbrake (turns out it's around 18meg since I needed a universal binary)

  the mencoder I use I build (probably more frequently than handbrake does theirs)
with the latest libx264 and faac, so chances are it will be 'fresher'.  But I figure
why not give the option of running whichever encoder you like..

  So the beta now has handbrake and mencoder.  As you mentioned, right now you can't
use comskip with handbrake as the encoder, and handbrake won't let you inline
download/encoder (it tries to scan the whole file to figure out titles)..  So if you
want a fast download/encoder inline, you're stuck with mencoder.  Same if you want
comskip.

  But hopefully this will change as they make changes.

  Unfortunately, it's now 10.5 only, since handbrake is compiled only for 10.5

Dec 3, 2008
Project Member #11 yoav.yer...@gmail.com
Ok marking this 'fixed' since I've dealt with the thrashing and the handbrake issue.
  People are reporting issues with handbrake already (sigh), but at least it's
available as an option...
Status: Fixed
Owner: yoav.yerushalmi