BUG: nanops._maybe_null_out doesn't work with complex numbers

[toddrjen]

A number of nanops functions rely on the private internal function nanops._maybe_null_out. These include nansum, nanmin, nanmax, and nanprod. All of these, in principle, should work with arrays with a complex dtype. However, due to a bug in _maybe_null_out, they convert the array to a float under certain situations, resulting in erroneous results. The easiest way to trigger this is to have a complex array where all the imaginary values are nan, and make axis argument something other than None.

>>> from pandas.core import nanops
>>> import bottleneck
>>> import numpy as np
>>> nanops._USE_BOTTLENECK = False
>>> 
>>> val = np.tile(1, (11, 7)) + np.tile(np.nan*1j, (11, 7))
>>> 
>>> bottleneck.nansum(val, axis=0)
array([ nan+0.j,  nan+0.j,  nan+0.j,  nan+0.j,  nan+0.j,  nan+0.j,  nan+0.j])
>>> nanops.nansum(val, axis=0)
array([ nan,  nan,  nan,  nan,  nan,  nan,  nan])
>>> bottleneck.nansum(val, axis=0).dtype
dtype('complex128')
>>> nanops.nansum(val, axis=0).dtype
dtype('float64')

[jreback]

this is failing on numpy 1.9-dev: https://travis-ci.org/pydata/pandas/jobs/27401353

might be spurious or maybe the comparison needs to have less precision...

pls lmk

[toddrjen]

I can submit another pull request later or include a precision change in
the current one. The problem only appears to occur with complex numbers.
I encountered it before with another function and fixed it there, it seems
to depend on the random numbers chosen.

On Thu, Jun 12, 2014 at 3:38 PM, jreback [email protected] wrote:

this is failing on numpy 1.9-dev:
https://travis-ci.org/pydata/pandas/jobs/27401353

might be spurious or maybe the comparison needs to have less precision...

pls lmk

—
Reply to this email directly or view it on GitHub
#7353 (comment).

[jreback]

just modify the latest PR to fix;

you could set np.random.seed(123456) or whatever in the setUp for the nanops tests
and/or use less precision (maybe just for complex)