postgres-xc-developers Mailing List for Postgres-XC

Some scenarios, user would have to be careful about assuming that the
query would be run at the datanode. If a query has some non-shippable
expressions, the query might not get completely shipped. Or if the
query does not have tables, the user would use execute direct. I think
we need to keep both ways.

On 29 February 2012 11:50, David E. Wheeler <da...@ju...> wrote:
> On Feb 28, 2012, at 10:14 PM, Ashutosh Bapat wrote:
>
>> SELECT * FROM dn1.table;
>>
>> Instead of mixing with schema qualification syntax, we can also use SELECT * from table@dn1.
>
> +1, although I think @ is allowed to be used for custom operators.
>
>  http://www.postgresql.org/docs/current/static/sql-createoperator.html
>
> So that could cause problems. But maybe not in the context of a table, where such an operator perhaps could not be used?
>
> Best,
>
> David
>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

On Feb 28, 2012, at 10:14 PM, Ashutosh Bapat wrote:

> SELECT * FROM dn1.table;
> 
> Instead of mixing with schema qualification syntax, we can also use SELECT * from table@dn1.

+1, although I think @ is allowed to be used for custom operators.

  http://www.postgresql.org/docs/current/static/sql-createoperator.html

So that could cause problems. But maybe not in the context of a table, where such an operator perhaps could not be used?

Best,

David

On Wed, Feb 29, 2012 at 11:44 AM, Ashutosh Bapat
<ash...@en...> wrote:
>
>
> On Wed, Feb 29, 2012 at 11:17 AM, Pavan Deolasee
> <pav...@en...> wrote:
>>
>> On Wed, Feb 29, 2012 at 11:03 AM, Koichi Suzuki <koi...@gm...>
>> wrote:
>> > I agree that the suggestion looks better.
>> >
>> > With this modification, EXECUTE DIRECT will receive parsed tree, not a
>> > text which should go straight to the target node.   EXECTE DIRECT
>> > should rebuild the statement.   May the order of the predicate or
>> > nesting parenthesis be different from the original input?   I hope
>> > not.
>>
>> Just for the record, another mechanism that we had discussed in the
>> past is to have some kind of schema for every node in the cluster. So
>> user can schema qualify the objects to force run a query on a
>> particular node. For example,
>>
>> SELECT * FROM dn1.table;
>
>
> Instead of mixing with schema qualification syntax, we can also use SELECT *
> from table@dn1.
>

We can do that. But that gets a bit tricky with function invokation
etc. Also, its probably better to use existing syntax unless we
conclude that it can't be done for some reason.

>>
>>
>> This would return all the tuples from the partition on data node dn1.
>> So this would be equivalent to running:
>>
>> EXECUTE DIRECT ON dn1 'SELECT * FROM table';
>>
>> Once we add the capability of qualify the objects with nodes, user can
>> write many powerful queries for administrative as well as other
>> purposes.
>>
>
> Although, this means that such a query has to go through planner, which can
> be non-performant. With EXEC DIRECT kind of syntax, we know that the query
> does not need any planning at coordinator.
>

Yeah. But then these are mostly administrative or special purpose
queries. We don't expect users to use them a lot. So some additional
planning cost at the coordinator may be a non-issue.

Thanks,
Pavan

-- 
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

On Feb 28, 2012, at 9:47 PM, Pavan Deolasee wrote:

> Just for the record, another mechanism that we had discussed in the
> past is to have some kind of schema for every node in the cluster. So
> user can schema qualify the objects to force run a query on a
> particular node. For example,
> 
> SELECT * FROM dn1.table;
> 
> This would return all the tuples from the partition on data node dn1.
> So this would be equivalent to running:

That's awesome, but what if table is in a schema?

I think you'd have to add another level of path, e.g.,

    SELECT * FROM dn1.public.table;

Best,

David

On Wed, Feb 29, 2012 at 11:17 AM, Pavan Deolasee <
pav...@en...> wrote:

> On Wed, Feb 29, 2012 at 11:03 AM, Koichi Suzuki <koi...@gm...>
> wrote:
> > I agree that the suggestion looks better.
> >
> > With this modification, EXECUTE DIRECT will receive parsed tree, not a
> > text which should go straight to the target node.   EXECTE DIRECT
> > should rebuild the statement.   May the order of the predicate or
> > nesting parenthesis be different from the original input?   I hope
> > not.
>
> Just for the record, another mechanism that we had discussed in the
> past is to have some kind of schema for every node in the cluster. So
> user can schema qualify the objects to force run a query on a
> particular node. For example,
>
> SELECT * FROM dn1.table;
>

Instead of mixing with schema qualification syntax, we can also use SELECT
* from table@dn1.


>
> This would return all the tuples from the partition on data node dn1.
> So this would be equivalent to running:
>
> EXECUTE DIRECT ON dn1 'SELECT * FROM table';
>
> Once we add the capability of qualify the objects with nodes, user can
> write many powerful queries for administrative as well as other
> purposes.
>
>
Although, this means that such a query has to go through planner, which can
be non-performant. With EXEC DIRECT kind of syntax, we know that the query
does not need any planning at coordinator.


> Thanks,
> Pavan
>
> --
> Pavan Deolasee
> EnterpriseDB http://www.enterprisedb.com
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Wed, Feb 29, 2012 at 11:28 AM, David E. Wheeler <da...@ki...> wrote:
> On Feb 28, 2012, at 9:47 PM, Pavan Deolasee wrote:
>
>> Just for the record, another mechanism that we had discussed in the
>> past is to have some kind of schema for every node in the cluster. So
>> user can schema qualify the objects to force run a query on a
>> particular node. For example,
>>
>> SELECT * FROM dn1.table;
>>
>> This would return all the tuples from the partition on data node dn1.
>> So this would be equivalent to running:
>
> That's awesome, but what if table is in a schema?
>
> I think you'd have to add another level of path, e.g.,
>
>    SELECT * FROM dn1.public.table;
>

Right. The node qualification will be one level above schema. I think
we may need to also think about the order in which we match nodes vs
schema, especially if user creates schema with the same name as node.
For example,

- SELECT * FROM dn1.test => if there is no schema with name dn1, we
try to match it to a node name and if found, run the query on that
node. The default name resolution mechanism will kick in node dn1

- SELECT * FROM dn1.test => if there is a schema with name dn1 and
there is also a node with the same name, the schema take precedence.
So the query runs on all the nodes

- SELECT * FROM dn1.dn1.test => the first dn1 will qualify node name
while the next dn1 will qualify schema name. So the query runs on dn1
and table named "test" in schema dn1.

Thanks,
Pavan

-- 
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

On Wed, Feb 29, 2012 at 11:03 AM, Koichi Suzuki <koi...@gm...> wrote:
> I agree that the suggestion looks better.
>
> With this modification, EXECUTE DIRECT will receive parsed tree, not a
> text which should go straight to the target node.   EXECTE DIRECT
> should rebuild the statement.   May the order of the predicate or
> nesting parenthesis be different from the original input?   I hope
> not.

Just for the record, another mechanism that we had discussed in the
past is to have some kind of schema for every node in the cluster. So
user can schema qualify the objects to force run a query on a
particular node. For example,

SELECT * FROM dn1.table;

This would return all the tuples from the partition on data node dn1.
So this would be equivalent to running:

EXECUTE DIRECT ON dn1 'SELECT * FROM table';

Once we add the capability of qualify the objects with nodes, user can
write many powerful queries for administrative as well as other
purposes.

Thanks,
Pavan

-- 
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

On 29 February 2012 11:03, Koichi Suzuki <koi...@gm...> wrote:
> I agree that the suggestion looks better.
>
> With this modification, EXECUTE DIRECT will receive parsed tree, not a
> text which should go straight to the target node.   EXECTE DIRECT
> should rebuild the statement.

We don't have to rebuild the sql string. We can easily save the sql
string snippet in the grammar file.

> May the order of the predicate or
> nesting parenthesis be different from the original input?   I hope
> not.
>
> Regards;
> --------
> Koichi Suzuki
>
>
>
> 2012/2/29 Ashutosh Bapat <ash...@en...>:
>> If that's possible +1. It will also simplify Execute Direct handling, which
>> needs an extra parsing step.
>>
>>
>> On Wed, Feb 29, 2012 at 6:11 AM, David E. Wheeler <da...@ju...>
>> wrote:
>>>
>>> PGXCers,
>>>
>>> Michael Paquier pointed out EXECUTE DIRECT to me today, which is quite
>>> handy. I'm wondering, however, if it might not be too late to change the
>>> parser to allow a raw query to be passed to it, rather than a string. That
>>> is, instead of this:
>>>
>>>    execute direct on node dn1 'select * from nodes';
>>>
>>> I'd like to do this:
>>>
>>>    execute direct on node dn1 select * from nodes;
>>>
>>> Note the lack of single quotes. It seems to me that there is a precedent
>>> in EXPLAIN for this. If it needs to support more parameters, perhaps they
>>> could be included in parentheses, again following EXPLAIN:
>>>
>>>    execute direct (node nd1, coordinator coord2) SELECT * FROM nodes;
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>>
>>> David
>>>
>>> ------------------------------------------------------------------------------
>>> Virtualization & Cloud Management Using Capacity Planning
>>> Cloud computing makes use of virtualization - but cloud computing
>>> also focuses on allowing computing to be delivered as a service.
>>> http://www.accelacomm.com/jaw/sfnl/114/51521223/
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>> ------------------------------------------------------------------------------
>> Virtualization & Cloud Management Using Capacity Planning
>> Cloud computing makes use of virtualization - but cloud computing
>> also focuses on allowing computing to be delivered as a service.
>> http://www.accelacomm.com/jaw/sfnl/114/51521223/
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

On Feb 28, 2012, at 9:33 PM, Koichi Suzuki wrote:

> I agree that the suggestion looks better.
> 
> With this modification, EXECUTE DIRECT will receive parsed tree, not a
> text which should go straight to the target node.   EXECTE DIRECT
> should rebuild the statement.   May the order of the predicate or
> nesting parenthesis be different from the original input?   I hope
> not.

Is that what EXPLAIN does?

Best,

David

I agree that the suggestion looks better.

With this modification, EXECUTE DIRECT will receive parsed tree, not a
text which should go straight to the target node.   EXECTE DIRECT
should rebuild the statement.   May the order of the predicate or
nesting parenthesis be different from the original input?   I hope
not.

Regards;
--------
Koichi Suzuki



2012/2/29 Ashutosh Bapat <ash...@en...>:
> If that's possible +1. It will also simplify Execute Direct handling, which
> needs an extra parsing step.
>
>
> On Wed, Feb 29, 2012 at 6:11 AM, David E. Wheeler <da...@ju...>
> wrote:
>>
>> PGXCers,
>>
>> Michael Paquier pointed out EXECUTE DIRECT to me today, which is quite
>> handy. I'm wondering, however, if it might not be too late to change the
>> parser to allow a raw query to be passed to it, rather than a string. That
>> is, instead of this:
>>
>>    execute direct on node dn1 'select * from nodes';
>>
>> I'd like to do this:
>>
>>    execute direct on node dn1 select * from nodes;
>>
>> Note the lack of single quotes. It seems to me that there is a precedent
>> in EXPLAIN for this. If it needs to support more parameters, perhaps they
>> could be included in parentheses, again following EXPLAIN:
>>
>>    execute direct (node nd1, coordinator coord2) SELECT * FROM nodes;
>>
>> Thoughts?
>>
>> Thanks,
>>
>> David
>>
>> ------------------------------------------------------------------------------
>> Virtualization & Cloud Management Using Capacity Planning
>> Cloud computing makes use of virtualization - but cloud computing
>> also focuses on allowing computing to be delivered as a service.
>> http://www.accelacomm.com/jaw/sfnl/114/51521223/
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

If that's possible +1. It will also simplify Execute Direct handling, which
needs an extra parsing step.

On Wed, Feb 29, 2012 at 6:11 AM, David E. Wheeler <da...@ju...>wrote:

> PGXCers,
>
> Michael Paquier pointed out EXECUTE DIRECT to me today, which is quite
> handy. I'm wondering, however, if it might not be too late to change the
> parser to allow a raw query to be passed to it, rather than a string. That
> is, instead of this:
>
>    execute direct on node dn1 'select * from nodes';
>
> I'd like to do this:
>
>    execute direct on node dn1 select * from nodes;
>
> Note the lack of single quotes. It seems to me that there is a precedent
> in EXPLAIN for this. If it needs to support more parameters, perhaps they
> could be included in parentheses, again following EXPLAIN:
>
>    execute direct (node nd1, coordinator coord2) SELECT * FROM nodes;
>
> Thoughts?
>
> Thanks,
>
> David
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

> I could not reproduce this on my end. How many coordinators/data nodes
> you are running ? Also, does it happen every time for you or once in a
> while ?
>
OK, I have more precisions here.
It happens only after running regressions with 1 Coordinator and 2
Datanodes. On a fresh cluster it runs OK.
It looks that something got messed up. It may be related to the issues you
saw when you ran regressions several times in a row.
This is not reproducible on master, but I doubt it is directly related to
your patch.

Hmm.. this looks a bit weird. There should not have been more than 4
> connections because there are 5 Co. Seems like there is a bug in
> maintaining the connection list involved in the transaction. I may
> find difficult to reproduce this because I can't run so many
> coordinators and data nodes. But let me look at the code again and see
> if I can spot something.
>
I have run in the past such clusters even on small laptop. So it should be
OK if shared_buffers of each node is small.
However, I don't see this issue anymore today...
OK so let's do that. Once Sudou-san is back, I'll see with him for the
performance tests and if it works OK, let's commit this feature.
Regards,
-- 
Michael Paquier
http://michael.otacoo.com

PGXCers,

Michael Paquier pointed out EXECUTE DIRECT to me today, which is quite handy. I'm wondering, however, if it might not be too late to change the parser to allow a raw query to be passed to it, rather than a string. That is, instead of this:

    execute direct on node dn1 'select * from nodes';

I'd like to do this:

    execute direct on node dn1 select * from nodes;

Note the lack of single quotes. It seems to me that there is a precedent in EXPLAIN for this. If it needs to support more parameters, perhaps they could be included in parentheses, again following EXPLAIN:

    execute direct (node nd1, coordinator coord2) SELECT * FROM nodes;

Thoughts?

Thanks,

David

On Feb 27, 2012, at 5:49 PM, Koichi Suzuki wrote:

> As we got a report from David, we should consider to include *.c file
> generated from *.l and *.y file in release tar balls, as done in
> vanilla PostgreSQL.    XC has specific handling of sgml docs through
> *.sgmlin, to make merge easier from PostgreSQL docs.   Although *.sgml
> can be generated from *.sgmlin using makesgml as found in
> doc-xc/tools, it may help users if we have *.sgml (for XC, English)
> files in tarball.
> 
> Any thoughts?

I chatted with Andrew Dunstan about it yesterday. He said the distcheck target should handle it, and if not it can be patched to do so.

Best,

David