Can not find fonts when export pdf using matplotlib Agg backend with usetex=True

While running the following code:

import matplotlib as mpl
import pylab

mpl.use('agg')
mpl.rcParams['text.usetex']=True

pylab.plot(range(5))
pylab.savefig("test.pdf")

produce an error:

Traceback (most recent call last):
  File "bug_test.py", line 12, in <module>
    pylab.savefig("test.pdf")
  File "/usr/lib/pymodules/python2.7/matplotlib/pyplot.py", line 561, in savefig
    return fig.savefig(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/figure.py", line 1421, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/backend_bases.py", line 2220, in print_figure
    **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/backend_bases.py", line 1952, in print_pdf
    return pdf.print_pdf(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_pdf.py", line 2358, in print_pdf
    file.close()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_pdf.py", line 507, in close
    self.writeFonts()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_pdf.py", line 604, in writeFonts
    realpath, stat_key = get_realpath_and_stat(filename)
  File "/usr/lib/pymodules/python2.7/matplotlib/cbook.py", line 982, in __call__
    stat = os.stat(realpath)
OSError: [Errno 2] No such file or directory: '<current directury>/cmr17'

The font cmr17 do exists in the standard latex distribution. (mf, afm and tfm) files. exporting to postscript or EPS works perfectly. But I want a PDF output to import as image to a XeTeX document. Removing usetex=True part runs OK. But I want my fonts to correspond with my LaTeX document.

What might be wrong?

(Debian unstable, matplotlib 1.3.1 or 1.4 from git master, python 2.7, texlive 2013.20140408-1)

Thanks,
Ronen