Skip to content

REF/BUG/API: factorizing categorical data #19938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Mar 15, 2018
Merged

REF/BUG/API: factorizing categorical data #19938

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
18de376
REF/BUG/API: factorizing categorical data
TomAugspurger Feb 23, 2018
9ef5be2
Explicit dtype for expected
TomAugspurger Feb 28, 2018
5e52b6f
Clean : imports / remove sort
TomAugspurger Mar 6, 2018
e19ae86
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 6, 2018
121b682
Restore sort
TomAugspurger Mar 6, 2018
a6bc405
REF: remove sort from Categorical.factorize
TomAugspurger Mar 9, 2018
0bfbc47
Updated comment
TomAugspurger Mar 9, 2018
b25f383
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 12, 2018
2688c4f
Fixed new sort algo
TomAugspurger Mar 12, 2018
2395a99
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 12, 2018
ab4f01c
Implement interface
TomAugspurger Mar 13, 2018
65150f4
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 13, 2018
ad8173b
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 13, 2018
1e006d1
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
REF: remove sort from Categorical.factorize
  • Loading branch information
@TomAugspurger
TomAugspurger committed Mar 9, 2018
commit a6bc40594ac706198fcc44057bd99e7869a76f38
12 changes: 9 additions & 3 deletions pandas/core/algorithms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ def factorize(values, sort=False, order=None, na_sentinel=-1, size_hint=None):

if is_categorical_dtype(values):
values = getattr(values, '_values', values)
labels, uniques = values.factorize(sort=sort)
labels, uniques = values.factorize()
dtype = original.dtype
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my comment below, but you might simpy dispatch on categricals and just return, mixing the impl is really confusing here.

else:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this actually be a check on the values if they have a .factorize() method (or check is_extension_array)? instead of specifically checking for categorical? (of course categorical will pass these checks). as this will then make pd.factorize(an_extension_array) work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is just a bugfix for categorical. But the structure will be very similar (I'll just change is_categorical_dtype to is_extension_array_dtype.)

I'll implement EA.factorize today hopefully, but have to get things like unique and argsort working first.

values, dtype, _ = _ensure_data(values)
Expand All @@ -516,8 +516,14 @@ def factorize(values, sort=False, order=None, na_sentinel=-1, size_hint=None):
na_sentinel=na_sentinel,
size_hint=size_hint)

if sort and len(uniques) > 0:
from pandas.core.sorting import safe_sort
if sort and len(uniques) > 0:
from pandas.core.sorting import safe_sort
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could move to the except (but no big deal)

try:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed all the sorting from Categorical.factorize. All that logic is here.

I don't think we want to just call safe_sort for two reasons

  1. that function does a lot of unnescessary work when we know that uniques is an ndarray or EA.
  2. It coerces categoricals to object ndarrays.
  3. EAs (like Categorical) may have special sorting rules.

On some small bencharks (10,000 elements) this is about 25-40% faster. The only slow case, for which we still need safe_sort, is when the array is mixed. In that case things are about 10% slower.

order = uniques.argsort()
labels = take_1d(order, labels, fill_value=na_sentinel)
uniques = uniques.take(order)
except TypeError:
# Mixed types, where uniques.argsort fails.
uniques, labels = safe_sort(uniques, labels,
na_sentinel=na_sentinel,
assume_unique=True)
Expand Down
8 changes: 2 additions & 6 deletions pandas/core/arrays/categorical.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2069,7 +2069,7 @@ def unique(self):
take_codes = sorted(take_codes)
return cat.set_categories(cat.categories.take(take_codes))

def factorize(self, sort=False, na_sentinel=-1):
def factorize(self, na_sentinel=-1):
"""Encode the Categorical as an enumerated type.

Parameters
Expand Down Expand Up @@ -2110,7 +2110,7 @@ def factorize(self, sort=False, na_sentinel=-1):
[a, b]
Categories (2, object): [a, b]
"""
from pandas.core.algorithms import _factorize_array, take_1d
from pandas.core.algorithms import _factorize_array

codes = self.codes.astype('int64')
codes[codes == -1] = iNaT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interface we have to hashtable.get_labels() is very odd right now, IOW we have a check_null flag which then makes the caller know to substitute values to iNaT (for int64) and know which are the sentinels. This is breaking the abstrastion. Rather would either like to be able to pass in the actual sentinel (not the output sentinel, but that's another confusion). e.g . you would simply pass -1 here.

I think its worth re-factoring this (maybe before this PR), though I suppose could be after.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#20328

Yes, that'd be nicer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we actually want this to be public?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

factorize in general? I don’t see why not. It’s present on series and index.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#19938 (comment) was in reference to the API docs. We whitelist the methods on Categorical that are included in the API docs (just __array__ and from_codes for now).

Expand All @@ -2121,10 +2121,6 @@ def factorize(self, sort=False, na_sentinel=-1):
uniques = self._constructor(self.categories.take(uniques),
categories=self.categories,
ordered=self.ordered)
if sort:
order = uniques.argsort()
labels = take_1d(order, labels, fill_value=na_sentinel)
uniques = uniques.take(order)
return labels, uniques

def equals(self, other):
Expand Down