FlacLikeABoss

From FarmShare

Revision as of 17:21, 4 May 2012 by Bishopj (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

Flac like a boss

Introduction

Flac is a lossless audio codec designed to compress a .wav file. As it is lossless, the only tradeoff when creating flac files is the eventual size of the content. A worthwhile experiment before us then is:

What is the best combination of options to create the smallest flac file for a given .wav?

Executive summary

In the experiment detailed below we determine that this flac command line generates the smallest flac file, in 15 minutes, using the barley cluster:

flac --lax -b 4096 -l 32 -p -e -r 0,16 foo.wav -o out0010.flac 

Running this sequentially (IE: on a corn) running flac directly would have taken some 1 day 14hrs to arrive at the same result.

NOTE: this example may not work exactly as written because we changed the max allowed number of jobs per user to less than the ~3000 needed for this example.

Methodology

Assessing cardinality

Looking at the flac command line options, --best as described in the man page is synonymous with these options: -l 12 -b 4096 -m -e -r 6

But are these options really the best for my .wav file?

Looking at the set of all possible options it is not obvious to me at all that these are the exact options which give the best results for my .wav file, or any wave file for that matter.

       -m, --mid-side
             Try mid-side coding for each frame (stereo input only)

       -M, --adaptive-mid-side
             Adaptive mid-side coding for all frames (stereo input only)

       -0..-8, --compression-level-0..--compression-level-8
             Fastest compression..highest compression (default is -5).  These
             are synonyms for other options:

              -0, --compression-level-0
                    Synonymous with -l 0 -b 1152 -r 3

              -1, --compression-level-1
                    Synonymous with -l 0 -b 1152 -M -r 3

              -2, --compression-level-2
                    Synonymous with -l 0 -b 1152 -m -r 3

              -3, --compression-level-3
                    Synonymous with -l 6 -b 4096 -r 4

              -4, --compression-level-4
                    Synonymous with -l 8 -b 4096 -M -r 4

              -5, --compression-level-5
                    Synonymous with -l 8 -b 4096 -m -r 5

              -6, --compression-level-6
                    Synonymous with -l 8 -b 4096 -m -r 6

              -7, --compression-level-7
                    Synonymous with -l 8 -b 4096 -m -e -r 6

              -8, --compression-level-8
                    Synonymous with -l 12 -b 4096 -m -e -r 6

       --fast Fastest compression.  Currently synonymous with -0.

       --best Highest compression.  Currently synonymous with -8.

       -e, --exhaustive-model-search
             Do exhaustive model search (expensive!)

       -l #, --max-lpc-order=#
             Specifies  the maximum LPC order. This number must be <= 32. For
             Subset streams, it must be <=12 if the sample rate  is  <=48kHz.
             If  0,  the  encoder will not attempt generic linear prediction,
             and use only fixed predictors. Using fixed predictors is  faster
             but usually results in files being 5-10% larger.

       -p, --qlp-coeff-precision-search
             Do  exhaustive  search  of  LP  coefficient quantization (expen-
             sive!).  Overrides -q; does nothing if using -l 0

       -q #, --qlp-coeff-precision=#
             Precision of the quantized linear-predictor coefficients,  0  =>
             let encoder decide (min is 5, default is 0)

       -r [#,]#, --rice-partition-order=[#,]#
             Set the [min,]max residual partition order (0..16). min defaults
             to 0 if unspecified.  Default is -r 5.

Let us see if the barley cluster can help us answer this question.

The first task before us is to write a script which will generate all possible invocations of flac. This will help us characterize the cardinality of the problem. We do this with the following python script:

#!/usr/bin/python

import os
import datetime 

startjob = datetime.datetime.now() 

#--lax 
#-b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384 
#-l &lt;= 32 
#-p 
#-e 
#-r 0..16,0..16 
#-m, -M 
#

numjobs = 0 

for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 8192, 16384 ]: 
    for l in [ 0, 1, 4, 8, 16, 32 ]:
       for p in [ '-p', '' ]:
           for e in [ '-e', '' ]:
               for r in [ '%d,%d' % (x, y) for x in xrange(0,16+1,2) for y in xrange(0,16+1,2) if x < y ]:
                   for m in [ '-m', '' ]:
                       for M in [ '-M', '' ]:
                           numjobs += 1
                           jobout = 'out%04d.flac' % numjobs
                           jobargs = '--lax -b %d -l %d %s %s -r%s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
                           print jobargs

print 'numjobs',numjobs 

endjob = datetime.datetime.now() 

print 'time to submit all jobs: ', endjob-startjob


Running the script says there are 41472 individual flac invocations to run through all possible of options. This is quite a few, and while possible, I see an easy way to reduce cardinality by performing the experiments with a constant block size of 4096.


#!/usr/bin/python

import os
import datetime 

startjob = datetime.datetime.now() 

#--lax 
#-b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384 
#-l &lt;= 32 
#-p 
#-e 
#-r 0..16,0..16 
#-m, -M 
#

numjobs = 0

for b in [ 4096 ]: 
    for l in [ 0, 1, 4, 8, 16, 32 ]:
       for p in [ '-p', '' ]:
           for e in [ '-e', '' ]:
               for r in [ '%d,%d' % (x, y) for x in xrange(0,16+1,2) for y in xrange(0,16+1,2) if x < y ]:
                   for m in [ '-m', '' ]:
                       for M in [ '-M', '' ]:
                           numjobs += 1
                           jobout = 'out%04d.flac' % numjobs
                           jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
                           os.system("qsub -N flac-%d ~/flac.submit %s" % (numjobs, jobargs))
                           print jobargs

print 'numjobs',numjobs 

endjob = datetime.datetime.now() 

print 'time to submit all jobs: ', endjob-startjob

flac.submit job script:

#
#$ -cwd
#$ -j y
#$ -S /bin/bash

echo "Got $NSLOTS slots"
cat $PE_HOSTFILE 
echo "running with" echo $* 

time flac -s $*

This reduces our cardinality to 3456. This is quite doable on the barley cluster. Lets run the job.

Running the job

Execute the python script to submit the jobs:

$ python ~/doflacsubmit.py
Your job 41540 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 -m -M foo.wav -o out0001.flac
Your job 41541 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 -m foo.wav -o out0002.flac
Your job 41542 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 -M foo.wav -o out0003.flac
Your job 41543 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 foo.wav -o out0004.flac
Your job 41544 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 -m -M foo.wav -o out0005.flac
Your job 41545 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 -m foo.wav -o out0006.flac
Your job 41546 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 -M foo.wav -o out0007.flac
Your job 41547 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 foo.wav -o out0008.flac
Your job 41548 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 -m -M foo.wav -o out0009.flac
Your job 41549 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 -m foo.wav -o out0010.flac
Your job 41550 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 -M foo.wav -o out0011.flac
Your job 41551 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 foo.wav -o out0012.flac
.
.
. 

Feel the Power

Checking qstat shows that in a little bit over 10 minutes (start @ Wed Dec 28 13:13:08 PST 2011, end @ Wed Dec 28 13:23:34 PST 2011) all the jobs are finished:

However, looking at the accumulated cpu time, this experiment consumed 1 day and 13hrs cpu time.

$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py
total 1 day, 13:17:16.980000 

This shows that even for tasks which do not scream HPC, using the barley cluster is still win. If you had executed flac in the inner loop of the python script instead of qsub, you would have waited for over a day and a half to get the results.

Looking at the .flac files produced, ls -lrS shows the smallest files are these:

$ ls -lrS *.flac | head
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2268.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2266.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2264.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2262.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2260.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2258.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2256.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2254.flac
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2252.flac
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2250.flac 

and the largest are these:

-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0447.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0445.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0223.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0221.flac 

This is over an 11% spread in size, even while keeping the block size (-b option) constant. While --best is decent, it did not give the best results.

$ flac --best foo.wav -o best.flac 

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details. 

foo.wav: wrote 26454456 bytes, ratio=0.578 

$ ls -l best*
-rw-r--r-- 1 bishopj root 26454456 Dec 21 12:58 best.flac 

So now lets take the options which performed the best and pop those into a new submit script which will run through all the possible block sizes.

$ for i in `ls -rS *.flac | head -10`; do grep $i *submit*; done
flac.submit.o43807:--lax -b 4096 -l 32 -p -e -r 0,14 foo.wav -o out2268.flac
flac.submit.o43805:--lax -b 4096 -l 32 -p -e -r 0,14 -m foo.wav -o out2266.flac
flac.submit.o43803:--lax -b 4096 -l 32 -p -e -r 0,12 foo.wav -o out2264.flac
flac.submit.o43801:--lax -b 4096 -l 32 -p -e -r 0,12 -m foo.wav -o out2262.flac
flac.submit.o43799:--lax -b 4096 -l 32 -p -e -r 0,10 foo.wav -o out2260.flac
flac.submit.o43797:--lax -b 4096 -l 32 -p -e -r 0,10 -m foo.wav -o out2258.flac
flac.submit.o43795:--lax -b 4096 -l 32 -p -e -r 0,8 foo.wav -o out2256.flac
flac.submit.o43793:--lax -b 4096 -l 32 -p -e -r 0,8 -m foo.wav -o out2254.flac
flac.submit.o43791:--lax -b 4096 -l 32 -p -e -r 0,6 foo.wav -o out2252.flac
flac.submit.o43789:--lax -b 4096 -l 32 -p -e -r 0,6 -m foo.wav -o out2250.flac 

Looks like --lax -b 4096 -l 32 -p -e -r 0,14 is what we want. (0,16 actually, as I had a booboo in my previous script which is fixed below).

#!/usr/bin/python

import os
import datetime 

startjob = datetime.datetime.now() 

numjobs = 0 

for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 4096+1024, 4096+2048, 8192, 16384 ]: 
    for l in [ 32 ]:
       for p in [ '-p' ]:
           for e in [ '-e' ]:
               for r in [ '%d,%d' % (x, y) for x in xrange(0,1) for y in xrange(16,16+1) if x < y ]:
                   for m in [ '' ]:
                       for M in [ '' ]:
                           numjobs += 1
                           jobout = 'out%04d.flac' % numjobs
                           jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
                           os.system("qsub -N flac-%d ~/flac.submit %s" % (numjobs, jobargs))
                           print jobargs

print 'numjobs',numjobs 

endjob = datetime.datetime.now() 

print 'time to submit all jobs: ', endjob-startjob


Lets run it:

$ python ~/doflacsubmit2.py
Your job 44258 ("flac.submit") has been submitted
--lax -b 192 -l 32 -p -e -r 0,16   foo.wav -o out0001.flac
Your job 44259 ("flac.submit") has been submitted
--lax -b 576 -l 32 -p -e -r 0,16   foo.wav -o out0002.flac
Your job 44260 ("flac.submit") has been submitted
--lax -b 1152 -l 32 -p -e -r 0,16   foo.wav -o out0003.flac
Your job 44261 ("flac.submit") has been submitted
--lax -b 2304 -l 32 -p -e -r 0,16   foo.wav -o out0004.flac
Your job 44262 ("flac.submit") has been submitted
--lax -b 4608 -l 32 -p -e -r 0,16   foo.wav -o out0005.flac
Your job 44263 ("flac.submit") has been submitted
--lax -b 256 -l 32 -p -e -r 0,16   foo.wav -o out0006.flac
Your job 44264 ("flac.submit") has been submitted
--lax -b 512 -l 32 -p -e -r 0,16   foo.wav -o out0007.flac
Your job 44265 ("flac.submit") has been submitted
--lax -b 1024 -l 32 -p -e -r 0,16   foo.wav -o out0008.flac
Your job 44266 ("flac.submit") has been submitted
--lax -b 2048 -l 32 -p -e -r 0,16   foo.wav -o out0009.flac
Your job 44267 ("flac.submit") has been submitted
--lax -b 4096 -l 32 -p -e -r 0,16   foo.wav -o out0010.flac
Your job 44268 ("flac.submit") has been submitted
--lax -b 5120 -l 32 -p -e -r 0,16   foo.wav -o out0011.flac
Your job 44269 ("flac.submit") has been submitted
--lax -b 6144 -l 32 -p -e -r 0,16   foo.wav -o out0012.flac
Your job 44270 ("flac.submit") has been submitted
--lax -b 8192 -l 32 -p -e -r 0,16   foo.wav -o out0013.flac
Your job 44271 ("flac.submit") has been submitted
--lax -b 16384 -l 32 -p -e -r 0,16   foo.wav -o out0014.flac
numjobs 14
time to submit all jobs:  0:00:00.511025
$ qstat

job-ID prior name user state submit/start at queue slots ja-task-ID -----------------------------------------------------------------------------------------------------------------

44258 0.25001 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley13.stanford.edu       1        
44259 0.25001 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley13.stanford.edu       1        
44260 0.25001 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley07.stanford.edu       1        
44261 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley01.stanford.edu       1        
44262 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley08.stanford.edu       1        
44263 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley14.stanford.edu       1        
44264 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu     1        
44265 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley15.stanford.edu       1        
44266 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley13.stanford.edu       1        
44267 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley07.stanford.edu       1        
44268 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley01.stanford.edu       1        
44269 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley08.stanford.edu       1        
44270 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley14.stanford.edu       1        
44271 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu     1        

Wait 5 minutes for the jobs to finish....

$ flac --best foo.wav -o best.flac 

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details. 

foo.wav: wrote 26454456 bytes, ratio=0.578 

$ ls -lrS *.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out0010.flac
-rw-r--r-- 1 bishopj root 26422887 Dec 21 12:58 out0005.flac
-rw-r--r-- 1 bishopj root 26433325 Dec 21 12:58 out0011.flac
-rw-r--r-- 1 bishopj root 26436989 Dec 21 12:58 out0004.flac
-rw-r--r-- 1 bishopj root 26451960 Dec 21 12:58 out0009.flac
-rw-r--r-- 1 bishopj root 26454456 Dec 21 12:58 best.flac
-rw-r--r-- 1 bishopj root 26454801 Dec 21 12:58 out0012.flac
-rw-r--r-- 1 bishopj root 26494744 Dec 21 12:58 out0013.flac
-rw-r--r-- 1 bishopj root 26575590 Dec 21 12:58 out0003.flac
-rw-r--r-- 1 bishopj root 26611006 Dec 21 12:58 out0008.flac
-rw-r--r-- 1 bishopj root 26612491 Dec 21 12:58 out0014.flac
-rw-r--r-- 1 bishopj root 26857694 Dec 21 12:58 out0002.flac
-rw-r--r-- 1 bishopj root 26923480 Dec 21 12:58 out0007.flac
-rw-r--r-- 1 bishopj root 27469566 Dec 21 12:58 out0006.flac
-rw-r--r-- 1 bishopj root 27796933 Dec 21 12:58 out0001.flac


This is our winning flac command line:

--lax -b 4096 -l 32 -p -e -r 0,16 foo.wav -o out0010.flac 

Incidentally, this last set of jobs still took significant resources (just under an hour)

$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py
total 0:54:31.850000


Full disclosure: the code for parseelapsedtime.py is:

from datetime import datetime, timedelta
import dateutil.parser
import time
import sys

runningtotal = timedelta()

for i in sys.stdin.readlines():
    foo = i.rstrip()
    tuple_datetime = datetime.strptime(foo, "%M:%S.%f")

    delta = timedelta(minutes=tuple_datetime.minute, seconds=tuple_datetime.second, microseconds=tuple_datetime.microsecond)
    runningtotal += delta

print 'total', runningtotal
Personal tools
Toolbox
LANGUAGES