qCube: Efficient integration of range query operators over a high dimension data cube

qCube:
Efficient integration of range query operators over a high
dimension data cube
Rodrigo Rocha Silva
Doctorate Student
Prof. Dr. Celso Massaki Hirata
Advisor
Prof. Dr. Joubert de Castro Lima
Co-Advisor
ITA – INSTITUTO TECNOLÓGICO DE AERONÁUTICA

Electronic Engineering and Computer Science Division - EEC/I
Department of Computer Science
Brazil

qCube: Efficient integration of range query operators over a high dimension data cube

Goal
Present a new cube approach, designed for
high dimension range queries. Our cube
approach, named Query Cube (qCube),
implements Equal, Not Equal, Greater or
Less than, Some, Between and Similar
range query operators and Distinct, Subcube and Top-k Similar inquire query
operators

Wednesday, October 02, 2012

28º Simpósio Brasileiro de Banco de Dados - Rodrigo Rocha Silva

2


Topics
–
–
–
–
–
–
–

Motivation
Data Cube
Related Work
Query Cube (qCube)
Experiments
Results
Conclusions



3


Motivation
Users need to view data in a tangible way, such as reports,
cross tables and histograms



4


Motivation
• Suppose that at some decision-making
process it is necessary the following
information :
“What is the women journal research papers
variance impact, using months {1, 3, 5, 7,
11}, year 2012 and ages varying from 25-40
years? Return results for all countries”

“The average temperatures above 30 degrees
Celsius on the weekends of leap years in the last
200 years.”


5


Data Cube
A data cube, introduced by Gray et al., 1996, is
a generalization of the group-by operator over all
possible combinations of dimensions with
various granularity aggregates.



6


Data Cube

A data cube has exponential
complexity with respect to the
number of dimensions
For an input with size d the
output has size 2d



7


Data Cube
• Hierarchies
Year

Discipline

Day

Department

Year


Hour


8


Data Cube
A

C

COUNT

A

B

C

COUNT

*

*

*

11

*

b2

c1

1

a1

*

*

3

*

b2

c2

1

a2

*

*

5

*

b3

c2

3

a3

Base Relation R – 11 tuples

B

*

*

3

a1

b1

c1

1

A

B

C

COUNT

*

b1

*

6

a3

b3

c2

1

a1

b1

c1

1

*

b2

*

2

a2

b3

c2

1

a3

b3

c2

1

*

b3

*

3

a3

b1

c1

1

a2

b3

c2

1

*

*

c1

4

a2

b1

c1

1

a3

b1

c1

1

*

*

c2

7

a2

b2

c2

1

a2

b1

c1

1

a1

b1

*

2

a1

b1

c2

1

a2

b2

c2

1

a1

b3

*

1

a2

b2

c1

1

a1

b1

c2

1

a2

b1

*

2

a3

b1

c2

1

a2

b2

c1

1

a2

b2

*

2

a1

b3

c2

1

a3

b1

c2

1

a2

b3

*

1

a2

b1

c2

1

a1

b3

c2

1

a3

b1

*

2

a2

b1

c2

1

a3

b3

*

1

a1

*

c1

1

a1

*

c2

2

a2

*

c1

2

a2

*

c2

3

a3

*

c1

1

a3

*

c2

2

*

b1

c1

3

*

b1

c2

3


FULL 3D CUBE

+

38 tuples


9


Related Work – Frag-Cubing Approach
•

Partitions the data vertically

•

Reduces high-dimensional cube into a set of lower
dimensional cubes

•

Lossless reduction

•

Offers tradeoffs between the amount of pre-processing
and the speed of online computation

From book Han and Kamber: Data Mining Concepts and Techniques


10


Related Work – Frag-Cubing Example
• Let the cube aggregation function be count
tid

A

B

C

D

E

1

a1

b1

c1

d1

e1

2

a1

b2

c1

d2

e1

3

a1

b2

c1

d1

e2

4

a2

b1

c1

d1

e2

5

a2

b1

c1

d1

e3

• Divide the 5 dimensions into 2 shell fragments:
– (A, B, C) and (D, E)


11


Related Work – Frag-Cubing 1-D Inverted Indices
• Build traditional invert index or RID list
Attribute Value

TID List

List Size

a1

123

3

a2

45

2

b1

145

3

b2

23

2

c1

12345

5

d1

1345

4

d2

2

1

e1

12

2

e2

34

2

e3

5

1



12


• Generalize the 1-D inverted indices to multi-dimensional
ones in the data cube sense
• Compute all cuboids for data cubes ABC and DE while
retaining the inverted indices
• For example, shell
fragment cube ABC
contains 7 cuboids:
– A, B, C
– AB, AC, BC
– ABC
• This completes the offline
computation stage

Cell

Intersection

TID List List Size

a1 b1

1 2 3 ∩1 4 5

1

1

a1 b2

1 2 3 ∩2 3

23

2

a2 b1

4 5 ∩1 4 5

45

2

a2 b2

4 5 ∩2 3

⊗

0



13


Related Work – Frag-Cubing Measure Table
• If measures other than count are present, store in
ID_measure table separate from the shell fragments
tid

count

sum

1

5

70

2

3

10

3

8

20

4

5

40

5

2

30



14


Related Work – Frag-Cubing Query
•

Given the fragment cubes, process a query as follows

1.

Divide the query into fragment, same as the shell

2.

Fetch the corresponding TID list for each fragment
from the fragment cube

3.

Intersect the TID lists from each fragment to construct
instantiated base table

4.

Compute the data cube using the base table with any
cubing algorithm



15


A B C D E F G H I J K L M N …

Base Table

Online
Computation



16


qCube Approach
Implements a set of tuple identifiers per dimension attribute,
similar to Frag-Cubing;
Therefore, qCube can answer point queries using tuple
identifiers intersections and range queries using unions plus
intersections algorithms, regardless measure function types.
Frag-Cubing just implements point and some inquire queries.
There is no Frag-Cubing solution for queries like

“What is the women journal research papers variance impact,
using months {1, 3, 5, 7, 11}, year 2012 and ages varying
from 25-40 years? Return results for all countries”


17


qCube Approach
Implements the range query operators:
• Equal;
• Not Equal;
• Greater or Less than;
• Some;
• Between and Similar.
Also implements inquire query operators:
• Distinct;
• Sub-cube;
• Top-k Similar.
Over a high dimension data cube.


18


qCube Architecture



19


qCube Computation
TID
1
2
3
4
5
6

A
a1
a2
a1
a3
a1
a5

Function
tid
1
2
3
4
5
6

B
b1
b2
b1
b3
b1
b5

C
c1
c2
c1
c3
c4
c5

D
d1
d2
d1
d2
d1
d2

Variance
M1
2.56
3.14
2.45
6.7
9
1


E
e1
e2
e1
e2
e2
e2

Count
M2
1
1
1
1
1
1

Attribute Value TID List

Attribute Value TID List

a1
a2
a3
a5
b1
b2
b3
b5
c1

c2
c3
c4
c5
d1
d2
e1
e2

Average
M3
10
20
10
11
3
1

1, 3, 5
2
4
6
1, 3, 5
2
4
6
1, 3

Skewness
M4
1
0
1
1
1
1

2
4
5
6
1, 3, 5
2, 4, 6
1, 3
2, 4, 5, 6

Standard deviation
M5
877686769698
7986676867.99
-7878789.8777
-99974333.23
100045.655
1


20


qCube Update

The same qCube Computation algorithm



21


qCube Update
TID
1
2
3
4
5
6

A
a1
a2
a1
a3
a1
a5

B
b1
b2
b1
b3
b1
b5

C
c1
c2
c1
c3
c4
c5

D
d1
d2
d1
d2
d1
d2

E
e1
e2
e1
e2
e2
e2

Attribute Value
a1
a2
a3
a5
b1
b2
b3
b5
c1
c2
c3

tid
5
7
8
9

TID List
1, 3
2, 8
4, 5, 7
6, 9
1, 3, 5
2, 7
4, 8
6, 9
1, 3
2
4, 7

A
a3
a3
a2
a5

B
b1
b2
b3
b5

C
c4
c3
c4
c5

Attribute Value
c4
c5
d1
d2
d3
e1
e2
e3
f1
f2

D
d1
d3
d3
d1

E
e2
e3
e2
e1

F

f1
f2

TID List
5, 8
6, 9
1, 3, 5, 9
2, 4, 6
7, 8
1, 3, 9
2, 4, 5, 6, 8
7
8
9


22


qCube Query
pQ= a1:*:*:*:e1
Attribute Value
a1
a2
a3
a5
b1
b2
b3
b5
c1
c2
c3


TID List
1, 3
2, 8
4, 5
6, 9
1, 3, 5
2, 7
4, 8
6, 9
1, 3
2
4, 7

Attribute Value
c4
c5
d1
d2
d3
e1
e2
e3
f1
f2

TID List
5, 8
6, 9
1, 3, 5, 9
2, 4, 6
7, 8
1, 3, 9
2, 4, 5, 6, 8
7
8
9


23


qCube Range and Inquire Query
rOp= (greater than + less than + between + some + different +

similar x (fv1 … fvn))
iOp =(sub-cube + distinct + top-k similar x (fv1 … fvn))
qCube rearranges Q sub-queries in order to improve query
response times

a result of Q we have qR=(TID1, TID2 … TIDk), where TIDi is
the ith tuple identifier of relation R.


24


qCube Query - example



25


“What is the women journal research papers variance impact,
using months {1, 3, 5, 7, 11}, year 2012 and ages varying from 2540 years? Return results for all countries”
In Q, they are (sex = women, paperType=journal, year=2012).
The range queries (month = (1,3,5,7,11), age <>25-40) are also
sorted according to their cardinalities.
In Q, there is inquire query (country=distinct).



26





27


Experiments
•

We tested qCube Computation and Query algorithms against Frag-Cubing
algorithm used in [Li et al. 2004];

•

The qCube algorithms were coded in Java 64 bits;

•

Frag-Cubing is a free and open source C++
application(http://illimine.cs.uiuc.edu/);

•

The synthetic base relations were created using data generator provided by the
IlliMine project;

•

The IlliMine project is an open-source project to provide various approaches for
data mining and machine learning.

•

Frag-Cubing approach is part of IlliMine project.

•

We ran the algorithms in two Intel Xeon six-core processors with 2.4GHz each
core, 12MB cache and 128GB of RAM DDR3 1333MHz.

•

The system runs Windows Server 2008 64 bits, High Performance version.



28


Results - Performance Evaluation of Point Queries and Skewed
Relations

Response time per query over
100 trials: T=107; C=5000;

D=30, S=0

Response time per query over 100
trials: T=107; C=5000; D=30,

S=2.5



29


Results - Performance Evaluation of Range Query Operators
and Skewed Relations

Response time queries with one infrequent point
operator: T=107; C=5000; D=30, S=2.5


30


Results - Performance Evaluation of of Inquire Operators
and Skewed Relations

Response time queries with inquire operators: T = 107; C = 5000; D = 30, S = 2.5.



31


Results - Runtime and Memory Consumption



32


Conclusions
• qCube has linear runtime and memory consumption, similar to

Frag-Cubing;
• It implements Not Equal, Greater or Less than, Some, Between
and Similar range query operators and Distinct, Sub-cube and
Top-k Similar inquire query operators;
• When compared with Frag-Cubing, qCube is faster to answer
point and inquire queries with sub-cube operators.
• It introduces a different cube representation with less empty cells
than Frag-Cubing;
• Frag-Cubing cannot answer two sub-cube operators in a data
cube with 107 tuples, C=5000, D=30 and S=2.5.


33


Conclusions
Interesting research directions to further extend qCube:
First, we must experiment it with holistic measures. Update and computation
experiments with many holistic measures are a hard problem;
TIDs can become huge, thus memory consumption and intersection costs can
become impracticable, and therefore we must address an efficient solution to
partition TIDs with fast data retrieval.
Multicore and multicomputer versions of qCube must be implemented.

qCube must be improved to answer top-k queries combined with range, point
and inquire queries.
Experiments with high dimensional text cubes must be made to evaluate qCube ,
specially its text measures computing.



34


Acknowlegements



35

qCube: Efficient integration of range query operators over a high dimension data cube

Recommended

More Related Content

What's hot (9)

Similar to qCube: Efficient integration of range query operators over a high dimension data cube (20)

Recently uploaded (20)

qCube: Efficient integration of range query operators over a high dimension data cube