[matplotlib-devel] [ Curiosa ] Fun with unicode

Hello,

I've been playing with the unicode rendering code that got added in
matplotlib 0.74. All tests have been done on Linux with the 0.74-1
debian package, lazy me... They should work on any platform, but you
will have to find out the gory details (!).

[ Most of these are examples, but there are a few remarks for the ps backend
   maintainers mostly, enclosed in square brackets in the text. No patches yet,
   but if you find the ideas useful, I can give it a try later. ]

To try this out, you may also need the rather complete FreeFont unicode font:
1) download the ttf from
    http://savannah.nongnu.org/download/freefont/freefont-ttf.tar.gz
2) put the .ttf files in your prefered Truetype fonts directory
3) remove ~/.ttffont.cache
4) restart matplotlib

Part I: Guess what I do for a living

Get your favorite interactive backend and display pretty plots

 >>> plot([0.3,0.01,-0.01,-0.01,-0.1,-0.1,-0.3,-0.01,0.01,0.01,0.1,0.1,0.3],\
... [105,100,98,90,92,101,105,100,98,90,92,101,105],'kD-')
 >>> ylim(85,110)
 >>> xlabel(u'\u03bc\u2080H(T)', name='FreeSans')
 >>> ylabel(u'R(\u03a9)', name='FreeSans')
 >>> ^D

or weird formulas

 >>> figtext(0.5,0.5,u'\u0127\u03c9 \u226a k\u0432T',name='FreeSerif',\
... size=30, ha='center', va='center', color='r')
 >>> ^D

You can also save to svg, and even to postscript (or eps) provided you set the
ps.useafm preference to False for now.

Part II: All work and no play...

 >>> plot([0.3,0.01,-0.01,-0.01,-0.1,-0.1,-0.3,-0.01,0.01,0.01,0.1,0.1,0.3],\
... [105,100,107,90,92,101,105,100,98,90,92,101,105],'kD-')
 >>> ylim(85,110)
 >>> text(-0.01,107,u'   \u261c booh! the ugly artifact!',name='FreeSerif',\
... size=20, va='top', ha='left')
 >>> ^D

;-)

Part III: Ugly, dirty and mean

Now it's time to produce a PDF. Run ps2pdf on one of the plots above,
at look at the ugly Type 3 fonts in your prefered PDF viewer.
The only way to get decent PDF is to set ps.useafm to True again.

For this to work, we have to provide the AFM files for FreeFont:
1) download the source of the font from:
    http://savannah.nongnu.org/download/freefont/freefont-sfd.tar.gz
2) download fontforge from fontforge.sourceforge.net
3) open each .sfd file in freefont, and run File\Generate Fonts;
    choose type 'PS Type 0'; this should produce a corresponding
    .afm and .ps file; save the .ps file for later.
4) move the afm file to a directory which is searched by matplotlib.
    Any subdirectory of /usr/share/fonts/ will do, provided said
    subdirectory is not a symlink
    [ is this a bug ? the implementation is in lib/font_manager.py,
      function x11FontDirectory(); os.path.walk() ignores symlinks ]
5) remove ~/.afmfont.cache
6) restart matplotlib
7) when you save the first .ps figure, the cache is rebuilt

Now that we have a postscript, we need to convince ghostscript to display it.
The first step is to provide the Type 0 fonts, like this:
1) move the .ps files we previously saved into a directory in ghostscript's
    path (try gs -h). A subdirectory won't work this time. Don't ask me why.
2) rename the font file to the name of the font, without extension, like
    $ mv FreeSans.ps FreeSans

If we try to look at our figure now, ghostscript will complain about
'/rangecheck yada yada' and fail to display the figure. This is because
postscript doesn't understand utf-8 encoding.
Postscript does however understand unicode hexa codes. So we have to replace
(R(\316\251))    [ octal representation of utf-8 characters ]
with
<0052002803a90029>    [ each 4 hexa figures are one character ]
For now, we have to do that manually in our favorite text editor. To compute
the hexa code in python, we do:
 >>> unistr=u'R(\u03a9)'
 >>> print ('<'+'%04x'*len(unistr)+'>') % tuple([ord(c) for c in unistr])
 >>> ^D

[ It looks that the ps backend should do just that with unicode strings if
   ps.useafm is True, as utf-8 is useless anyway.
   Using unicode hexa may also allow a much simpler implementation of
   draw_unicode() (in lib/backends/backend_ps.py) in the Type 42 case, by
   avoiding to position the glyphs one by one ]

I successfully tested .eps files produced with this procedure on both
a recent ghostscript and acrobat distiller; distiller or ps2pdf will produce
PDFs with nice embedded Type 1 fonts.

Part IV: Publish or perish

Producing pretty PDFs is well and nice, but most publishers will ask for
.eps with all fonts embedded. So we have to embed the fonts into the .eps
file. I could find no program to do this. DO NOT use gs -sDEVICE=pswrite
for that. Not only will ghostscript mangle the fonts, but also the plots (!).

Luckily, the FreeSans.ps from above is already a postscript with embedded
fonts, so we are golden. Just cat the font files together with the .eps
and merge the headers and footers by hand.

[ It would be nice to have a ps.embedfonts preference. For Type 0, this is
easy, as above; I don't know for Type 1. Also, it would be good to embed
only the needed glyphs, but I haven't looked at how to do it ]

Well, that's all for tonight. In conclusion, unicode support works already
very well, and allows lots of fun things. Thank you guys for the good
work.

BC