Strange behaviour with MultiIndex and more than one million records

[olevold]

Code Sample

r = list(range(999999))
df2 = pd.DataFrame({'a':r, 'b':r}, index=pd.MultiIndex.from_tuples([(x,x) for x in r]))
df2['a'].blabla()
# result as expected: AttributeError: 'Series' object has no attribute 'blabla'

r = list(range(1000000))
df2 = pd.DataFrame({'a':r, 'b':r}, index=pd.MultiIndex.from_tuples([(x,x) for x in r]))
df2['a'].blabla()
#strange result: TypeError: unorderable types: numpy.ndarray() < str()

from scipy.stats.mstats import winsorize
winsorize(df2['a'])
#result: TypeError: unorderable types: numpy.ndarray() < str()

Problem description

When there are exactly one million records or more with a MultiIndex, I get this TypeError when I call a non-existent method on a column. I get the same error when I try to call the winsorize function from scipy.stats on a column. The type error originates here: File "pandas/index.pyx", line 481, in pandas.index._bin_search

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-98-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: nb_NO.UTF-8
LOCALE: nb_NO.UTF-8

pandas: 0.21.0
pytest: 3.2.2
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.26.1
numpy: 1.13.1
scipy: 0.19.1
pyarrow: None
xarray: 0.9.6
IPython: 2.4.1
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: 2.4.4
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[jreback]

show versions as the issue template indicates

[olevold]

Done

[jreback]

so indexing with a million elements is different internally
so we are prob not catching and reraising the correct exception here
this hits indexing because you are accessing an attribute , which u then try to call
attribute access can be for columns as well so that’s why indexing is hit

if you would trace would be great - this will end up in cython in index.pyx

[olevold]

File "/usr/local/lib/python3.5/site-packages/scipy/stats/mstats_basic.py", line 1419, in winsorize
a = ma.array(a, copy=np.logical_not(inplace))
File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 6152, in array
ndmin=ndmin, shrink=shrink, order=order)
File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 2762, in new
_baseclass = getattr(data, '_baseclass', type(_data))
File "/usr/local/lib/python3.5/site-packages/pandas/core/generic.py", line 2742, in getattr
if name in self._info_axis:
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1042, in contains
self.get_loc(key)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1643, in get_loc
loc = self._get_level_indexer(key, level=0)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1905, in _get_level_indexer
loc = level_index.get_loc(key)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/numeric.py", line 341, in get_loc
tolerance=tolerance)
File "/usr/local/lib/python3.5/site-packages/pandas/indexes/base.py", line 2134, in get_loc
return self._engine.get_loc(key)
File "pandas/index.pyx", line 132, in pandas.index.IndexEngine.get_loc (pandas/index.c:4433)
File "pandas/index.pyx", line 140, in pandas.index.IndexEngine.get_loc (pandas/index.c:4075)
File "pandas/index.pyx", line 481, in pandas.index._bin_search (pandas/index.c:9823)
TypeError: unorderable types: numpy.ndarray() < str()

[olevold]

BTW, the winsorize function works with 999999 records

[jreback]

ok the dundee contains methods in indexes/multi

a) move way up in the file it should be much closer to the top near other finder methods. (i know get/set state are there as well but leave those)
b) catch TypeError as well as LookupError to return False
c) test all of the cases u gave
can u do a PR?

[olevold]

I'm afraid I don't have the time to look further into it right now, I'm in the process of working around it in my project.

[jreback]

@olevold ok ; things get fixed by folks contributing