EHN encoding parameter for to_latex #11914

nbonnotte · 2015-12-27T17:52:12Z

I'm working on solving issue #7061: currently, with Python 2 it is not possible to use .to_latex() when the dataframe contains utf-8 strings. This PR adds an encoding parameter to make it possible.

The work is not completely done though, and I'm making this PR to start a discussion here. In its current state, the PR quickly fixes the issue and provides tests, so it is kind of a minimal PR.

@jreback in the discussion on issue #7061, you made the following suggestion:

you may want to add a LatexFormatter (as a sub-class of DataFrameFormatter) as this will allow some re-factoring to be internally done later on.

I would be glad to work in that direction, but I'm a bit confused by the current organization of the code. This LatexFormatter would seem to me to be similar to HTMLFormatter, which inherits from TableFormatter (exactly as DataFrameFormatter). In that case, DataFrameFormatter would use LatexFormatter for .to_latex() exactly as it uses HTMLFormatter for .to_html(). Is that what you had in mind?

jreback · 2015-12-28T12:40:15Z

pandas/tests/test_format.py

+        df = DataFrame([[u'au\xdfgangen']])
+        with tm.ensure_clean('test.tex') as path:
+            df.to_latex(path, encoding='utf-8')
+            import codecs


put the import at the top of the module

jreback · 2015-12-28T12:46:26Z

@nbonnotte what I mean is that I think a LatexFormatter be created in format.py which is like HTMLFormatter (its a sub-class of TableFormatter). So you don't have all of the latex formatting code living in to_latex itself (which instead just instantiates a class and calls .write_result, similarly to to_html).

This will allow easier refactoring later.

nbonnotte · 2015-12-28T22:57:01Z

BTW, this does not solves the missing decimal parameter I mentioned in the discussion of issue #7061

jreback · 2015-12-29T19:51:04Z

@nbonnotte what was the decimal issue? (can't seem to find it in that issue)

nbonnotte · 2016-01-10T12:26:32Z

I just meant the decimal parameter is still lacking, I didn't add it. I should do another PR to do that.

jreback · 2016-01-11T13:41:31Z

doc/source/whatsnew/v0.18.0.txt

@@ -302,6 +302,9 @@ Other API Changes

 - ``.memory_usage`` now includes values in the index, as does memory_usage in ``.info`` (:issue:`11597`)

+- ``DataFrame.to_latex()`` now supports non-ascii encodings (eg utf-8) in Python 2 with the parameter ``encoding``


add the issue number (7061)

jreback · 2016-01-11T13:43:46Z

@nbonnotte create another issue for the decimal parameter (we are going to close #7061) with this PR

jreback · 2016-01-11T13:44:10Z

comments, ping when green.

nbonnotte · 2016-01-13T14:47:01Z

I've created another issue for decimal (#12031)

Working on the docstring, I realized there was something odd: I was initializing LatexFormatter with a DataFrameFormatter and a variable strcols containing the result of DataFrameFormatter._to_str_columns.

I think it is simpler if the parameter strcols is removed and the call to DataFrameFormatter._to_str_columns is performed in LatexFormatter. You can see in the 2nd commit the changes (dc425cb0e959ccab2941cab27f329ec41cb768d7).

There is still one thing that bothers me: now I'm calling a "private" method (_to_str_columns) from outside (that is, from LatexFormatter). Is that a problem?

jreback · 2016-01-13T15:01:58Z

@nbonnotte no that's ok

nbonnotte · 2016-01-14T08:46:32Z

I've squashed my commits then.

All green!

jreback · 2016-01-14T12:47:27Z

pandas/core/format.py

+    def __init__(self, formatter, column_format=None, longtable=False):
+        self.fmt = formatter
+        self.frame = self.fmt.frame
+        self.columns = self.fmt.tr_frame.columns


are you using these variables? (e.g. .columns); I'd rather you not define them, or use a cached-property if its more clear (e.g. for .frame)

jreback · 2016-01-14T12:49:05Z

pls git diff master | flake8 --diff

nbonnotte · 2016-01-15T12:36:58Z

@jreback should be good now ^^

jreback · 2016-01-15T16:24:35Z

merged via 3a832df

thanks!

jreback reviewed Dec 28, 2015
View reviewed changes

jreback added IO LaTeX to_latex Unicode Unicode strings labels Dec 28, 2015

jreback reviewed Jan 11, 2016
View reviewed changes

jreback added this to the 0.18.0 milestone Jan 11, 2016

jreback reviewed Jan 14, 2016
View reviewed changes

ENH in .to_latex() support for utf-8 encoding in Python 2

bd6876d

jreback closed this Jan 15, 2016

jreback mentioned this pull request Jan 15, 2016

Unicode handling in to_latex. Needs encoding? #7061

Closed

nbonnotte deleted the to_latex-encoding-7061 branch January 15, 2016 16:27

		@@ -302,6 +302,9 @@ Other API Changes

		- ``.memory_usage`` now includes values in the index, as does memory_usage in ``.info`` (:issue:`11597`)

		- ``DataFrame.to_latex()`` now supports non-ascii encodings (eg utf-8) in Python 2 with the parameter ``encoding``

Uh oh!

EHN encoding parameter for to_latex #11914

EHN encoding parameter for to_latex #11914

Uh oh!

Conversation

nbonnotte commented Dec 27, 2015

Uh oh!

jreback Dec 28, 2015

Choose a reason for hiding this comment

Uh oh!

jreback commented Dec 28, 2015

Uh oh!

nbonnotte commented Dec 28, 2015

Uh oh!

jreback commented Dec 29, 2015

Uh oh!

nbonnotte commented Jan 10, 2016

Uh oh!

jreback Jan 11, 2016

Choose a reason for hiding this comment

Uh oh!

jreback commented Jan 11, 2016

Uh oh!

jreback commented Jan 11, 2016

Uh oh!

nbonnotte commented Jan 13, 2016

Uh oh!

jreback commented Jan 13, 2016

Uh oh!

nbonnotte commented Jan 14, 2016

Uh oh!

jreback Jan 14, 2016

Choose a reason for hiding this comment

Uh oh!

jreback commented Jan 14, 2016

Uh oh!

nbonnotte commented Jan 15, 2016

Uh oh!

jreback commented Jan 15, 2016

Uh oh!

Uh oh!