Skip to content

ESQL: CATEGORIZE as a BlockHash #114317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 85 commits into from
Nov 27, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
a4647fc
WIP
nik9000 Oct 4, 2024
9cb4425
Merge branch 'main' into esql_blockh2
nik9000 Oct 8, 2024
95f767b
Stay plz:
nik9000 Oct 8, 2024
e4b9e4b
Update docs/changelog/114317.yaml
nik9000 Oct 8, 2024
f2d1806
Make things compile
alex-spies Oct 17, 2024
a309133
Move new block hashes to typical location
alex-spies Oct 22, 2024
e50d5b9
Add almost passing test
alex-spies Oct 22, 2024
5674046
Close stuff (makes unit test pass)
jan-elastic Oct 24, 2024
31e9e20
undo making BytesRefBlockHash public
jan-elastic Oct 24, 2024
82cc74a
Move Categorize BlockHash tests to separate file
jan-elastic Oct 28, 2024
a234326
Unit test for CategorizedIntermediateBlockHash.
jan-elastic Oct 28, 2024
3f94143
Unit test with Driver
jan-elastic Oct 31, 2024
239d159
Add aggregator to unit test
jan-elastic Nov 4, 2024
260b572
Passing unit test with driver and aggregators
jan-elastic Nov 6, 2024
975d6ef
Improve code
jan-elastic Nov 6, 2024
f44774d
Fix ES compile errors
jan-elastic Nov 6, 2024
f7264f2
Update Categorize function/evaluator (works end2end in discover)
jan-elastic Nov 7, 2024
5b1cfc1
fix CategorizeTests
jan-elastic Nov 8, 2024
f05a3c2
Fix
jan-elastic Nov 8, 2024
fe3d536
Misc fixes to make CSV test work
jan-elastic Nov 8, 2024
1b7705b
Release blocks in CategorizedIntermediateBlockHash
alex-spies Nov 8, 2024
c3c0c68
multivalued fields
nik9000 Nov 12, 2024
f207321
ROW MV
nik9000 Nov 12, 2024
790770b
More cases
nik9000 Nov 12, 2024
3f65d30
unicode
nik9000 Nov 12, 2024
28d0012
More cases
nik9000 Nov 12, 2024
1af1c68
Drop all columns
nik9000 Nov 12, 2024
2b00cc2
Decode category
jan-elastic Nov 11, 2024
cc9f84a
Add test case with expression inside CATEGORIZE
alex-spies Nov 12, 2024
8c54acd
Disallow evaluating CATEGORIZE outside aggs
alex-spies Nov 12, 2024
6040f6b
WIP: wire in CATEGORIZE without new datatype
alex-spies Nov 12, 2024
8c96df9
Wire in CATEGORIZE without needing new datatype
alex-spies Nov 12, 2024
4485b0e
ON
nik9000 Nov 13, 2024
6bf3b31
Move BlockHash creation to Grouping function
ivancea Nov 18, 2024
64ca567
Remove CATEGORY DataType and ElementType
ivancea Nov 18, 2024
b5ad72a
Format all
ivancea Nov 18, 2024
46ac649
Removed unused code and added new test using CATEGORY() return value …
ivancea Nov 19, 2024
e311014
Remove ToBlockHash interface, and simplify logic for Categorize
ivancea Nov 19, 2024
5cd145d
Merge branch 'main' into esql_blockh2
ivancea Nov 19, 2024
3da877d
Remove missing CATEGORY enum constant from Csv tests
ivancea Nov 19, 2024
1bad621
Keep alias on ReplaceAggNested Categorize grouping
ivancea Nov 19, 2024
51dc274
Do not null/propagate fold Categorize
ivancea Nov 20, 2024
d421281
Fixed Categorize layout using source field, and added failing tests w…
ivancea Nov 20, 2024
bf466f7
Added failing alias CSV tests for Categorize
ivancea Nov 21, 2024
b5b387c
Make CombineProjections work with expressions in the grouping, like C…
ivancea Nov 21, 2024
3b75c65
Added Categorize tests for FoldNull and nested aggs rules
ivancea Nov 21, 2024
006768d
Format with spotlessApply
ivancea Nov 21, 2024
30a25b9
Fix benchmark compilation
ivancea Nov 21, 2024
147023d
Fix AttributeSet add() return value
ivancea Nov 21, 2024
8cb0e26
Fix/Remove NOCOMMITs
ivancea Nov 21, 2024
043f99d
Fix Javadoc HTML p tag not allowing nested blocks
ivancea Nov 22, 2024
9dd8973
Fixed tests compilation and added more csv tests for other commands
ivancea Nov 22, 2024
df39b40
Remove remaining NOCOMMITs
ivancea Nov 22, 2024
cbffe6d
Format
ivancea Nov 22, 2024
483cbfe
Updated Categorize tests to not fail on evaluator creation
ivancea Nov 22, 2024
0aeb50c
Merge branch 'main' into esql_blockh2
ivancea Nov 22, 2024
6ad0804
Fixed union_types CSV tests, failing after new dataset was addition
ivancea Nov 25, 2024
32fb924
Merge branch 'main' into esql_blockh2
ivancea Nov 25, 2024
5ad52de
Mute eval const failing tests until issue is fixed
ivancea Nov 26, 2024
27da936
Restored pre-DataType-change pieces of code (Extractors, BlockHash Ag…
ivancea Nov 26, 2024
0372701
Added javadocs and some tests with mixed cases (nulls, multivalue...)
ivancea Nov 26, 2024
c6cda24
Some docs and refactors on CategorizedIntermediateHashBlock
ivancea Nov 26, 2024
f833ba0
Make Categorize non-foldable
ivancea Nov 26, 2024
ddacdc3
Format
ivancea Nov 26, 2024
9ed6262
ReplaceNestedAgg ruie cleanup
ivancea Nov 26, 2024
ea13baf
Merge branch 'main' into esql_blockh2
ivancea Nov 26, 2024
dc8ecfd
Added failing tests for empty strings throwing NPE
ivancea Nov 26, 2024
c1c70fc
Rename sample_data_mv
ivancea Nov 26, 2024
aecacde
Ignore failing empty string tests
ivancea Nov 26, 2024
7c7b604
Undo union-types tests changes
ivancea Nov 26, 2024
9dc6ad1
Format
ivancea Nov 26, 2024
e0b84f0
Fixed failing test after indices change
ivancea Nov 26, 2024
9d1bfd3
Merge branch 'main' into esql_blockh2
ivancea Nov 26, 2024
6cf6ec1
Unmute Categorize tests
ivancea Nov 26, 2024
f677b47
Added a bunch of other tests from PR suggestions
ivancea Nov 26, 2024
f37d2bd
Removed semantic text index, which is failing in cross cluster
ivancea Nov 26, 2024
d10709b
Improved optimizer tests on categorize
ivancea Nov 27, 2024
b622e6c
Unmute last categorize test
ivancea Nov 27, 2024
30412ad
Rename capability to CATEGORIZE_V2
ivancea Nov 27, 2024
74aeae3
Ignore and fix null tests
ivancea Nov 27, 2024
3771532
Fixed multivalues on Categorize and added tests using other fields
ivancea Nov 27, 2024
599b02b
Use checked circuit breaker and fix CategorizeBlockHashTests
ivancea Nov 27, 2024
3763f74
Restore Aggregate computeReferences()
ivancea Nov 27, 2024
d2325aa
Remove AggregatorMode parameter from BlockHash.build(), and format all
ivancea Nov 27, 2024
43a335f
Merge branch 'main' into esql_blockh2
ivancea Nov 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove missing CATEGORY enum constant from Csv tests
  • Loading branch information
ivancea committed Nov 19, 2024
commit 3da877d8faac9c7e27bf8ef41288ed8f7d6f8848
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
import static org.elasticsearch.common.logging.LoggerMessageFormat.format;
import static org.elasticsearch.xpack.esql.CsvTestUtils.ExpectedResults;
import static org.elasticsearch.xpack.esql.CsvTestUtils.Type;
import static org.elasticsearch.xpack.esql.CsvTestUtils.Type.CATEGORY;
import static org.elasticsearch.xpack.esql.CsvTestUtils.Type.UNSIGNED_LONG;
import static org.elasticsearch.xpack.esql.CsvTestUtils.logMetaData;
import static org.elasticsearch.xpack.esql.core.util.DateUtils.UTC_DATE_TIME_FORMATTER;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -478,8 +478,7 @@ public enum Type {
GEO_POINT(x -> x == null ? null : GEO.wktToWkb(x), BytesRef.class),
CARTESIAN_POINT(x -> x == null ? null : CARTESIAN.wktToWkb(x), BytesRef.class),
GEO_SHAPE(x -> x == null ? null : GEO.wktToWkb(x), BytesRef.class),
CARTESIAN_SHAPE(x -> x == null ? null : CARTESIAN.wktToWkb(x), BytesRef.class),
CATEGORY(Object::toString, BytesRef.class);
CARTESIAN_SHAPE(x -> x == null ? null : CARTESIAN.wktToWkb(x), BytesRef.class);

private static final Map<String, Type> LOOKUP = new HashMap<>();

Expand Down