postgres-xc-developers Mailing List for Postgres-XC

Thanks for correcting it, Michael.

On Wed, Oct 31, 2012 at 7:59 AM, Michael Paquier
<mic...@gm...>wrote:

> Hi,
>
> This commit broke the buildfarm, explaining why the automatic test has not
> been able to complete last night and did not send an email. The problem was
> due to the files of test xc_limit missing. I already fixed the source code
> this morning, and Sudou-san will provide a fix to stop the regression tests
> in case a regression file is missing and send an email informing about the
> error type to be able to detect automatically such problems in the future.
>
> Regards,
>
>
> On Tue, Oct 30, 2012 at 9:45 PM, Ashutosh Bapat <
> ash...@us...> wrote:
>
>> Project "Postgres-XC".
>>
>> The branch, master has been updated
>>        via  8cb32febe56d62b986598cdad0ae66aeffeb1f63 (commit)
>>       from  16992e6b07d28d66c183adf45dd8e1cae2cdf25b (commit)
>>
>>
>> - Log -----------------------------------------------------------------
>>
>> http://postgres-xc.git.sourceforge.net/git/gitweb.cgi?p=postgres-xc/postgres-xc;a=commitdiff;h=8cb32febe56d62b986598cdad0ae66aeffeb1f63
>>
>> commit 8cb32febe56d62b986598cdad0ae66aeffeb1f63
>> Author: Ashutosh Bapat <ash...@en...>
>> Date:   Tue Oct 30 19:20:07 2012 +0530
>>
>>     Push LIMIT  clause to the datanode if the expressions in those
>>     clauses are shippable and the underlying plan tree allows that.
>>     In case there is only one datanode involved in the execution of the
>> RemoteQuery,
>>     we just push the clause to the datanode and no LIMIT and OFFSET
>> processing is required at
>>     the coordinator. But for distributed execution of RemoteQuery, we
>> need LIMIT and
>>     OFFSET processing at the coordinator.
>>     If OFFSET clause is present, we do not push it, but include it in the
>> LIMIT
>>     clause and apply OFFSET at the coordinator.
>>
>> M       src/backend/optimizer/path/costsize.c
>> M       src/backend/optimizer/plan/pgxcplan.c
>> M       src/backend/optimizer/plan/planner.c
>> M       src/backend/utils/misc/guc.c
>> M       src/include/optimizer/cost.h
>> M       src/include/optimizer/planmain.h
>> M       src/test/regress/expected/rangefuncs.out
>> M       src/test/regress/expected/rangefuncs_1.out
>> M       src/test/regress/expected/xc_FQS.out
>> M       src/test/regress/parallel_schedule
>> M       src/test/regress/serial_schedule
>>
>> -----------------------------------------------------------------------
>>
>> Summary of changes:
>>  src/backend/optimizer/path/costsize.c      |    1 +
>>  src/backend/optimizer/plan/pgxcplan.c      |  170
>> ++++++++++++++++++++++++++++
>>  src/backend/optimizer/plan/planner.c       |    5 +
>>  src/backend/utils/misc/guc.c               |    9 ++
>>  src/include/optimizer/cost.h               |    1 +
>>  src/include/optimizer/planmain.h           |    1 +
>>  src/test/regress/expected/rangefuncs.out   |    3 +-
>>  src/test/regress/expected/rangefuncs_1.out |    3 +-
>>  src/test/regress/expected/xc_FQS.out       |   24 ++--
>>  src/test/regress/parallel_schedule         |    2 +-
>>  src/test/regress/serial_schedule           |    1 +
>>  11 files changed, 205 insertions(+), 15 deletions(-)
>>
>>
>> hooks/post-receive
>> --
>> Postgres-XC
>>
>>
>> ------------------------------------------------------------------------------
>> Everyone hates slow websites. So do we.
>> Make your web apps faster with AppDynamics
>> Download AppDynamics Lite for free today:
>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>> _______________________________________________
>> Postgres-xc-committers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>>
>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi,

This commit broke the buildfarm, explaining why the automatic test has not
been able to complete last night and did not send an email. The problem was
due to the files of test xc_limit missing. I already fixed the source code
this morning, and Sudou-san will provide a fix to stop the regression tests
in case a regression file is missing and send an email informing about the
error type to be able to detect automatically such problems in the future.

Regards,

On Tue, Oct 30, 2012 at 9:45 PM, Ashutosh Bapat <
ash...@us...> wrote:

> Project "Postgres-XC".
>
> The branch, master has been updated
>        via  8cb32febe56d62b986598cdad0ae66aeffeb1f63 (commit)
>       from  16992e6b07d28d66c183adf45dd8e1cae2cdf25b (commit)
>
>
> - Log -----------------------------------------------------------------
>
> http://postgres-xc.git.sourceforge.net/git/gitweb.cgi?p=postgres-xc/postgres-xc;a=commitdiff;h=8cb32febe56d62b986598cdad0ae66aeffeb1f63
>
> commit 8cb32febe56d62b986598cdad0ae66aeffeb1f63
> Author: Ashutosh Bapat <ash...@en...>
> Date:   Tue Oct 30 19:20:07 2012 +0530
>
>     Push LIMIT  clause to the datanode if the expressions in those
>     clauses are shippable and the underlying plan tree allows that.
>     In case there is only one datanode involved in the execution of the
> RemoteQuery,
>     we just push the clause to the datanode and no LIMIT and OFFSET
> processing is required at
>     the coordinator. But for distributed execution of RemoteQuery, we need
> LIMIT and
>     OFFSET processing at the coordinator.
>     If OFFSET clause is present, we do not push it, but include it in the
> LIMIT
>     clause and apply OFFSET at the coordinator.
>
> M       src/backend/optimizer/path/costsize.c
> M       src/backend/optimizer/plan/pgxcplan.c
> M       src/backend/optimizer/plan/planner.c
> M       src/backend/utils/misc/guc.c
> M       src/include/optimizer/cost.h
> M       src/include/optimizer/planmain.h
> M       src/test/regress/expected/rangefuncs.out
> M       src/test/regress/expected/rangefuncs_1.out
> M       src/test/regress/expected/xc_FQS.out
> M       src/test/regress/parallel_schedule
> M       src/test/regress/serial_schedule
>
> -----------------------------------------------------------------------
>
> Summary of changes:
>  src/backend/optimizer/path/costsize.c      |    1 +
>  src/backend/optimizer/plan/pgxcplan.c      |  170
> ++++++++++++++++++++++++++++
>  src/backend/optimizer/plan/planner.c       |    5 +
>  src/backend/utils/misc/guc.c               |    9 ++
>  src/include/optimizer/cost.h               |    1 +
>  src/include/optimizer/planmain.h           |    1 +
>  src/test/regress/expected/rangefuncs.out   |    3 +-
>  src/test/regress/expected/rangefuncs_1.out |    3 +-
>  src/test/regress/expected/xc_FQS.out       |   24 ++--
>  src/test/regress/parallel_schedule         |    2 +-
>  src/test/regress/serial_schedule           |    1 +
>  11 files changed, 205 insertions(+), 15 deletions(-)
>
>
> hooks/post-receive
> --
> Postgres-XC
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Postgres-xc-committers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>



-- 
Michael Paquier
http://michael.otacoo.com

On Tue, Oct 30, 2012 at 8:05 AM, Koichi Suzuki <koi...@gm...>wrote:

> Thanks Ashutosh for the patch.   This is very useful in many cases.
>

Totally agree...this will help optimize many cases...


> ----------
> Koichi Suzuki
>
>
> 2012/10/29 Ashutosh Bapat <ash...@en...>:
> > Hi All,
> > PFA the patch for pushing down LIMIT clause to the datanodes.
> >
> > Pushing down LIMIT clause whenever possible can help reducing the
> bandwidth
> > in case the LIMIT + OFFSET is too less as compared to the qualifying rows
> > fetched from the datanodes.
> >
> > The patch has comments about working of the code.
> >
> > Regression does not show any failures. Expected outputs have been fixed
> for
> > two testcases, where plan changed, because of this optimization.
> >
> > --
> > Best Wishes,
> > Ashutosh Bapat
> > EntepriseDB Corporation
> > The Enterprise Postgres Company
> >
> >
> >
> ------------------------------------------------------------------------------
> > The Windows 8 Center - In partnership with Sourceforge
> > Your idea - your app - 30 days.
> > Get started!
> > http://windows8center.sourceforge.net/
> >
> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> > _______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>



-- 
Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

On Tue, Oct 30, 2012 at 9:05 PM, Ashutosh Bapat <
ash...@en...> wrote:

>
>
> On Tue, Oct 30, 2012 at 11:39 AM, Michael Paquier <
> mic...@gm...> wrote:
>
>> Hi Ashutosh,
>>
>> I think that this is a great addition to the set of clauses that can be
>> pushed down. Benchmarks using intensively LIMIT clauses like Dell (?)
>> benchmarks would have there performance greatly improved thanks to that.
>> Here is some global feedback:
>> - There are whitespaces detected by git apply:
>> /home/michael/download/xc_limit.patch:95: trailing whitespace.
>>
>> /home/michael/download/xc_limit.patch:101: trailing whitespace.
>>     if (IsA(local_plan, Limit))
>> /home/michael/download/xc_limit.patch:130: trailing whitespace.
>>     /*
>> /home/michael/download/xc_limit.patch:137: trailing whitespace.
>>
>> /home/michael/download/xc_limit.patch:142: trailing whitespace.
>>         /*
>> - no warnings
>> - no problems with regressions
>>
>> Feedback about the code:
>> 1) You should put create_remotelimit_plan just below
>> create_remotegrouping_plan, let's put the same functionalities together.
>
>  2) You should correct "coordinator" with an upper case as first letter =>
>> "Coordinator", same for "Datanode".
>>
>
> Capitalized names are used for the C structures. There is no structure
> named Coordinator or Datanode.
>
Please check the comments in your code.


>
>
>> 3) In create_remotelimit_plan, I think that the variable temp_plan is
>> useless, you could also do something like that:
>> /* Only a LIMIT plan can be evaluated in this function */
>> if (IsA(local_plan, Limit))
>>     local_limit = (Limit *)local_plan;
>>
>> /* Left-tree has to be a remote scan for pushdown evaluation */
>> if (IsA(local_plan->lefttree, RemoteQuery))
>>     remote_scan = (RemoteQuery *) local_plan->lefttree;
>>
>
> Right now this code handles Limit->RemoteQuery tree, but in future it can
> handle Limit->Sort->RemoteQuery or Limit->Group->RemoteQuery. Usage of
> temp_plan simplifies such trees. See create_remotegrouping_plan().
>
OK. I just think that it is worthless now but...



>
>
>> 4) Please respect the comment format. If a comment has more that one
>> line, its beginning and end marks have to be on separate lines, like that:
>> /*
>>  * comment line 1...
>>  * comment line 2...
>>  */
>> I see some places like this:
>> /* comment line 1
>>  * comment line 2
>>  */
>> 5) Please add the following header in xc_limit.sql and its related output:
>> --
>> -- XC_LIMIT
>> --
>> OK, this is not mandatory but all the other files have it, *normally*.
>>
> I randomly sampled few test files and found half of them used this
> convention and half of them did not. This doesn't look normal. There is no
> point in following a convention which does not serve any purpose.
>
The XC files do. Let's respect that at least.


>
>
>> 6) You should perhaps add some tests for roundrobin tables. Just to check
>> correctness of the feature for all the types of tables and avoid any side
>> effects.
>>
>
> For this functionality the difference of distributed and replicated is
> enough. Otherwise, we would have to repeat tests for every distribution
> kind, unnecessarily increasing size of testcase and increase maintenance
> burden.
>
>
>> 7) Please specify the distribution type when those tables are created in
>> xc_limit.sql:
>> create table xc_limit_tab1 (val int, val2 int);
>> create table xc_limit_tab2 (val int, val2 int);
>> They need to be completed with "distribute by hash(val)". For the time
>> being default distribution is hash on the first column available but if
>> someone modifies the code area to make the default as replicated this test
>> has no sense. We should protect at least such potential problems.
>> 8) This comment is not completely correct:
>> +
>> /*
>>
>> +        * While checking shippability for LIMIT and OFFSET clauses, we
>> don't
>> expect
>>
>> +        * any aggregates in
>> there
>>
>> +        */
>>
>
> aggregates are special case, because they can act as Vars in some cases.
> And the comment is correct from the perspective of call to
> pgxc_is_expr_shippable, which has specific argument for aggregate presence.
>
>
>> You should not only mention aggregates but globally unshippable
>> expressions. I believe that functions can also be used with limit, no?
>> 9) I am not honestly a fan of using a magic variable of the type "+" to
>> generate the addition operation for OFFSET + LIMIT, and I got no real idea
>> how to do that in a cleaner way...
>> Any ideas someone here?
>>
>>
> That's how it's everywhere in the code. If you can find a macro for '+',
> let me know. I will use that.
>
>
>> Except those comments, the logic for the calculation of the number of
>> rows returned from remote nodes based on OFFSET and LIMIT is OK for me.
>> This is really the important point of your patch.
>>
>> Also, something that I just noticed with your patch: there is no
>> documentation for the GUC parameters enable_remotejoin, enable_remotegroup
>> and enable_remotelimit which are things you implemented. Could you provide
>> a separate patch adding this documentation. The file that needs to be
>> completed is doc-xc/src/sgml/config.sgmlin. I can perform any compilation
>> checks on the docs if necessary with my environments here.
>>
>>
> enable_remotejoin is not mine. These GUCs are undocumented for good. They
> are only for testing purposes and not to be used in production or even user
> environment. At some point of time, we will remove these GUCs. I don't know
> when.
>
>
>> Regards,
>>
>>
>> On Mon, Oct 29, 2012 at 7:35 PM, Ashutosh Bapat <
>> ash...@en...> wrote:
>>
>>> Hi All,
>>> PFA the patch for pushing down LIMIT clause to the datanodes.
>>>
>>> Pushing down LIMIT clause whenever possible can help reducing the
>>> bandwidth in case the LIMIT + OFFSET is too less as compared to the
>>> qualifying rows fetched from the datanodes.
>>>
>>> The patch has comments about working of the code.
>>>
>>> Regression does not show any failures. Expected outputs have been fixed
>>> for two testcases, where plan changed, because of this optimization.
>>>
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>


-- 
Michael Paquier
http://michael.otacoo.com

On Tue, Oct 30, 2012 at 11:39 AM, Michael Paquier <mic...@gm...
> wrote:

> Hi Ashutosh,
>
> I think that this is a great addition to the set of clauses that can be
> pushed down. Benchmarks using intensively LIMIT clauses like Dell (?)
> benchmarks would have there performance greatly improved thanks to that.
> Here is some global feedback:
> - There are whitespaces detected by git apply:
> /home/michael/download/xc_limit.patch:95: trailing whitespace.
>
> /home/michael/download/xc_limit.patch:101: trailing whitespace.
>     if (IsA(local_plan, Limit))
> /home/michael/download/xc_limit.patch:130: trailing whitespace.
>     /*
> /home/michael/download/xc_limit.patch:137: trailing whitespace.
>
> /home/michael/download/xc_limit.patch:142: trailing whitespace.
>         /*
> - no warnings
> - no problems with regressions
>
> Feedback about the code:
> 1) You should put create_remotelimit_plan just below
> create_remotegrouping_plan, let's put the same functionalities together.

2) You should correct "coordinator" with an upper case as first letter =>
> "Coordinator", same for "Datanode".
>

Capitalized names are used for the C structures. There is no structure
named Coordinator or Datanode.


> 3) In create_remotelimit_plan, I think that the variable temp_plan is
> useless, you could also do something like that:
> /* Only a LIMIT plan can be evaluated in this function */
> if (IsA(local_plan, Limit))
>     local_limit = (Limit *)local_plan;
>
> /* Left-tree has to be a remote scan for pushdown evaluation */
> if (IsA(local_plan->lefttree, RemoteQuery))
>     remote_scan = (RemoteQuery *) local_plan->lefttree;
>

Right now this code handles Limit->RemoteQuery tree, but in future it can
handle Limit->Sort->RemoteQuery or Limit->Group->RemoteQuery. Usage of
temp_plan simplifies such trees. See create_remotegrouping_plan().


> 4) Please respect the comment format. If a comment has more that one line,
> its beginning and end marks have to be on separate lines, like that:
> /*
>  * comment line 1...
>  * comment line 2...
>  */
> I see some places like this:
> /* comment line 1
>  * comment line 2
>  */
> 5) Please add the following header in xc_limit.sql and its related output:
> --
> -- XC_LIMIT
> --
> OK, this is not mandatory but all the other files have it, *normally*.
>
I randomly sampled few test files and found half of them used this
convention and half of them did not. This doesn't look normal. There is no
point in following a convention which does not serve any purpose.


> 6) You should perhaps add some tests for roundrobin tables. Just to check
> correctness of the feature for all the types of tables and avoid any side
> effects.
>

For this functionality the difference of distributed and replicated is
enough. Otherwise, we would have to repeat tests for every distribution
kind, unnecessarily increasing size of testcase and increase maintenance
burden.


> 7) Please specify the distribution type when those tables are created in
> xc_limit.sql:
> create table xc_limit_tab1 (val int, val2 int);
> create table xc_limit_tab2 (val int, val2 int);
> They need to be completed with "distribute by hash(val)". For the time
> being default distribution is hash on the first column available but if
> someone modifies the code area to make the default as replicated this test
> has no sense. We should protect at least such potential problems.
> 8) This comment is not completely correct:
> +
> /*
>
> +        * While checking shippability for LIMIT and OFFSET clauses, we
> don't
> expect
>
> +        * any aggregates in
> there
>
> +        */
>

aggregates are special case, because they can act as Vars in some cases.
And the comment is correct from the perspective of call to
pgxc_is_expr_shippable, which has specific argument for aggregate presence.


> You should not only mention aggregates but globally unshippable
> expressions. I believe that functions can also be used with limit, no?
> 9) I am not honestly a fan of using a magic variable of the type "+" to
> generate the addition operation for OFFSET + LIMIT, and I got no real idea
> how to do that in a cleaner way...
> Any ideas someone here?
>
>
That's how it's everywhere in the code. If you can find a macro for '+',
let me know. I will use that.


> Except those comments, the logic for the calculation of the number of rows
> returned from remote nodes based on OFFSET and LIMIT is OK for me. This is
> really the important point of your patch.
>
> Also, something that I just noticed with your patch: there is no
> documentation for the GUC parameters enable_remotejoin, enable_remotegroup
> and enable_remotelimit which are things you implemented. Could you provide
> a separate patch adding this documentation. The file that needs to be
> completed is doc-xc/src/sgml/config.sgmlin. I can perform any compilation
> checks on the docs if necessary with my environments here.
>
>
enable_remotejoin is not mine. These GUCs are undocumented for good. They
are only for testing purposes and not to be used in production or even user
environment. At some point of time, we will remove these GUCs. I don't know
when.


> Regards,
>
>
> On Mon, Oct 29, 2012 at 7:35 PM, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Hi All,
>> PFA the patch for pushing down LIMIT clause to the datanodes.
>>
>> Pushing down LIMIT clause whenever possible can help reducing the
>> bandwidth in case the LIMIT + OFFSET is too less as compared to the
>> qualifying rows fetched from the datanodes.
>>
>> The patch has comments about working of the code.
>>
>> Regression does not show any failures. Expected outputs have been fixed
>> for two testcases, where plan changed, because of this optimization.
>>
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi Ashutosh,

I think that this is a great addition to the set of clauses that can be
pushed down. Benchmarks using intensively LIMIT clauses like Dell (?)
benchmarks would have there performance greatly improved thanks to that.
Here is some global feedback:
- There are whitespaces detected by git apply:
/home/michael/download/xc_limit.patch:95: trailing whitespace.

/home/michael/download/xc_limit.patch:101: trailing whitespace.
    if (IsA(local_plan, Limit))
/home/michael/download/xc_limit.patch:130: trailing whitespace.
    /*
/home/michael/download/xc_limit.patch:137: trailing whitespace.

/home/michael/download/xc_limit.patch:142: trailing whitespace.
        /*
- no warnings
- no problems with regressions

Feedback about the code:
1) You should put create_remotelimit_plan just below
create_remotegrouping_plan, let's put the same functionalities together.
2) You should correct "coordinator" with an upper case as first letter =>
"Coordinator", same for "Datanode".
3) In create_remotelimit_plan, I think that the variable temp_plan is
useless, you could also do something like that:
/* Only a LIMIT plan can be evaluated in this function */
if (IsA(local_plan, Limit))
    local_limit = (Limit *)local_plan;

/* Left-tree has to be a remote scan for pushdown evaluation */
if (IsA(local_plan->lefttree, RemoteQuery))
    remote_scan = (RemoteQuery *) local_plan->lefttree;
4) Please respect the comment format. If a comment has more that one line,
its beginning and end marks have to be on separate lines, like that:
/*
 * comment line 1...
 * comment line 2...
 */
I see some places like this:
/* comment line 1
 * comment line 2
 */
5) Please add the following header in xc_limit.sql and its related output:
--
-- XC_LIMIT
--
OK, this is not mandatory but all the other files have it, *normally*.
6) You should perhaps add some tests for roundrobin tables. Just to check
correctness of the feature for all the types of tables and avoid any side
effects.
7) Please specify the distribution type when those tables are created in
xc_limit.sql:
create table xc_limit_tab1 (val int, val2 int);
create table xc_limit_tab2 (val int, val2 int);
They need to be completed with "distribute by hash(val)". For the time
being default distribution is hash on the first column available but if
someone modifies the code area to make the default as replicated this test
has no sense. We should protect at least such potential problems.
8) This comment is not completely correct:
+
/*

+        * While checking shippability for LIMIT and OFFSET clauses, we
don't
expect

+        * any aggregates in
there

+        */
You should not only mention aggregates but globally unshippable
expressions. I believe that functions can also be used with limit, no?
9) I am not honestly a fan of using a magic variable of the type "+" to
generate the addition operation for OFFSET + LIMIT, and I got no real idea
how to do that in a cleaner way...
Any ideas someone here?

Except those comments, the logic for the calculation of the number of rows
returned from remote nodes based on OFFSET and LIMIT is OK for me. This is
really the important point of your patch.

Also, something that I just noticed with your patch: there is no
documentation for the GUC parameters enable_remotejoin, enable_remotegroup
and enable_remotelimit which are things you implemented. Could you provide
a separate patch adding this documentation. The file that needs to be
completed is doc-xc/src/sgml/config.sgmlin. I can perform any compilation
checks on the docs if necessary with my environments here.

Regards,

On Mon, Oct 29, 2012 at 7:35 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi All,
> PFA the patch for pushing down LIMIT clause to the datanodes.
>
> Pushing down LIMIT clause whenever possible can help reducing the
> bandwidth in case the LIMIT + OFFSET is too less as compared to the
> qualifying rows fetched from the datanodes.
>
> The patch has comments about working of the code.
>
> Regression does not show any failures. Expected outputs have been fixed
> for two testcases, where plan changed, because of this optimization.
>
-- 
Michael Paquier
http://michael.otacoo.com

GTM-Standby is the same as GTM, as PG-slave is the same as PG-Master.
 On the other hand, GTM-Proxy is just ilke pgboucer which groups and
proxies requirements from coordinator/datanode to GTM.   So, ovbiously
GTM and GMT-Proxy are DIFFERENT both in feature and code.
GTM-Standby is in fact GTM running in different mode which has to be
GTM when promoted.

Hope this helps.

Regards;
----------
Koichi Suzuki


2012/10/29 Nikhil Sontakke <ni...@st...>:
> I have found writing -Z GTM/GTM_STANDY/GTM_PROXY as part of the cli pretty
> painful.
>
> Infact there's not much difference between GTM and GTM_STANDBY at all in
> terms of processing from the front end. Also I am not sure why we have
> gtm_proxy.conf for the proxy and gtm.conf for GTM and the STANDBY.
>
> Like pg, where starting a replica has the same cli as starting the pg
> server, we should have had the same here for all the cases and should have
> allowed the role be picked up from the contents of the gtm.conf file.
>
> One area where would this pain is in HA. Typical HA configs expect same
> directory structures across the nodes and the same clis to start the
> resources etc. Now adding additional logic to include "-Z gtm/standby/proxy"
> introduces additional pain in this.
>
> My 2 cents.
>
> Regards,
> Nikhils
>
> On Mon, Oct 29, 2012 at 5:29 PM, Michael Paquier <mic...@gm...>
> wrote:
>>
>>
>>
>> On Mon, Oct 29, 2012 at 7:19 PM, Ashutosh Bapat
>> <ash...@en...> wrote:
>>>
>>> Thanks for taking this up.
>>>
>>> On Mon, Oct 29, 2012 at 3:08 PM, Michael Paquier
>>> <mic...@gm...> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I am currently working on cleaning up the dependencies and duplications
>>>> that GTM code has.
>>>> This is a pretty complicated task, as GTM has many code duplication with
>>>> libpq and also src/backend.
>>>>
>>>> By the way, I am trying to do this clean up with a maximum number of
>>>> discrete steps, making the progression clear. One of the things I noticed is
>>>> related to the GTM configuration system. It is a duplication of what can be
>>>> found in guc.c. However, guc.c is used exclusively by PG backend and all its
>>>> structures remain internal to GUC. In the case of GTM, those APIs are used
>>>> by both GTM and GTM-proxy as they both use a configuration file, so GTM
>>>> configuration APIs need to be externalized to avoid duplication between the
>>>> things used by GTM and by GTM-proxy. As far as I saw, there is no way to
>>>> resolve the PG-GUC/GTM conf duplication as we would need to externalize all
>>>> the structures in guc.c.
>>>> However, the GTM configuration APIs are located in src/gtm/common, which
>>>> generates an archive called libgtm.a, this archive being used by src/backend
>>>> for the construction of the binary file postgres. We need to remove this
>>>> dependency between the GTM configuration and PG backend for clarity, so
>>>> please find attached a patch that moves the GTM configuration APIs into a
>>>> new folder src/gtm/config and avoids a useless dependency of this code with
>>>> PG backend.
>>>
>>>
>>> On using guc.c, is there a separation of (I haven't looked at the file
>>> itself) separate the datastructures (the array where GUCs and there values
>>> are stored or fetched from etc.) and functions exposed. If so, we might copy
>>> the data structures but leave the functions common to PG and GTM. Are there
>>> any other tools in PG code which also require a configuration file and share
>>> the guc interfaces.
>>
>> Each configuration file in PG uses a different structure, with GUC being
>> the most complex.
>> For example recovery.conf uses readRecoveryCommandFile in xlog.c,
>> postgresql.conf uses guc.c. ident.conf and pg_hba.conf also use something
>> different (didn't have a look where but it is easy to find out). However,
>> regarding the GUC part, there is an interface available with guc-file.l. In
>> this file there are functions that GTM is duplicating and we should
>> definitely to use that (functions like ProcessConfigFile or ParseConfigFile
>> for example). Please note that they may need some extension as the existing
>> functions are designed only for src/backend as far as I saw. The ultimate
>> goal would be to use those APIs also with GTM for the configuration part,
>> and to keep the list of parameters in GTM. Before reaching that point I want
>> to reduce the dependencies between GTM modules to understand how to untie
>> one by one all the duplications that GTM has with things like memory, list,
>> configuration, StringInfo, Assert, etc.
>> Even before untying everything there are a couple of things missing for
>> this work:
>> - change the Makefile structure in GTM to avoid relying on static archives
>> and reach the same state as src/backend that use for example objfiles.txt. I
>> have basically a patch, just need to clean it up
>> - move gtm_ctl out of src/gtm to src/bin, also got a patch here, but it
>> needs a little bit more work to have clean makefiles.
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>>
>>
>> ------------------------------------------------------------------------------
>> The Windows 8 Center - In partnership with Sourceforge
>> Your idea - your app - 30 days.
>> Get started!
>> http://windows8center.sourceforge.net/
>> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Service
>
> ------------------------------------------------------------------------------
> The Windows 8 Center - In partnership with Sourceforge
> Your idea - your app - 30 days.
> Get started!
> http://windows8center.sourceforge.net/
> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Thanks Ashutosh for the patch.   This is very useful in many cases.
----------
Koichi Suzuki


2012/10/29 Ashutosh Bapat <ash...@en...>:
> Hi All,
> PFA the patch for pushing down LIMIT clause to the datanodes.
>
> Pushing down LIMIT clause whenever possible can help reducing the bandwidth
> in case the LIMIT + OFFSET is too less as compared to the qualifying rows
> fetched from the datanodes.
>
> The patch has comments about working of the code.
>
> Regression does not show any failures. Expected outputs have been fixed for
> two testcases, where plan changed, because of this optimization.
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
> ------------------------------------------------------------------------------
> The Windows 8 Center - In partnership with Sourceforge
> Your idea - your app - 30 days.
> Get started!
> http://windows8center.sourceforge.net/
> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

It's great if we use guc.c.   The background of the current  code is that guc.c is accompanied with flex code and it has to be included in option handler.    Option set is slightly different from gtm and gtm_proxy.    If we can solve all these, it will be nice ovbiously.   I also simplified the code to omit SIGINT handling to reload the option.   Enabling this is also welcome.

If we need to enhance the core (guc.c) to include these, I'm not sure if we should do this against the core code or make it separate.

Regards;
---
Koichi Suzuki

On Mon, 29 Oct 2012 15:49:44 +0530
Ashutosh Bapat <ash...@en...> wrote:

> Thanks for taking this up.
> 
> On Mon, Oct 29, 2012 at 3:08 PM, Michael Paquier
> <mic...@gm...>wrote:
> 
> > Hi all,
> >
> > I am currently working on cleaning up the dependencies and duplications
> > that GTM code has.
> > This is a pretty complicated task, as GTM has many code duplication with
> > libpq and also src/backend.
> >
> > By the way, I am trying to do this clean up with a maximum number of
> > discrete steps, making the progression clear. One of the things I noticed
> > is related to the GTM configuration system. It is a duplication of what can
> > be found in guc.c. However, guc.c is used exclusively by PG backend and all
> > its structures remain internal to GUC. In the case of GTM, those APIs are
> > used by both GTM and GTM-proxy as they both use a configuration file, so
> > GTM configuration APIs need to be externalized to avoid duplication between
> > the things used by GTM and by GTM-proxy. As far as I saw, there is no way
> > to resolve the PG-GUC/GTM conf duplication as we would need to externalize
> > all the structures in guc.c.
> > However, the GTM configuration APIs are located in src/gtm/common, which
> > generates an archive called libgtm.a, this archive being used by
> > src/backend for the construction of the binary file postgres. We need to
> > remove this dependency between the GTM configuration and PG backend for
> > clarity, so please find attached a patch that moves the GTM configuration
> > APIs into a new folder src/gtm/config and avoids a useless dependency of
> > this code with PG backend.
> >
> 
> On using guc.c, is there a separation of (I haven't looked at the file
> itself) separate the datastructures (the array where GUCs and there values
> are stored or fetched from etc.) and functions exposed. If so, we might
> copy the data structures but leave the functions common to PG and GTM. Are
> there any other tools in PG code which also require a configuration file
> and share the guc interfaces.
> 
> 
> >
> > Comments?
> > --
> > Michael Paquier
> > http://michael.otacoo.com
> >
> >
> > ------------------------------------------------------------------------------
> > The Windows 8 Center - In partnership with Sourceforge
> > Your idea - your app - 30 days.
> > Get started!
> > http://windows8center.sourceforge.net/
> > what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> > _______________________________________________
> > Postgres-xc-developers mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >
> >
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company

On 2012/10/29, at 21:14, Nikhil Sontakke <ni...@st...> wrote:

> I have found writing -Z GTM/GTM_STANDY/GTM_PROXY as part of the cli pretty painful. 
> 
> Infact there's not much difference between GTM and GTM_STANDBY at all in terms of processing from the front end. Also I am not sure why we have gtm_proxy.conf for the proxy and gtm.conf for GTM and the STANDBY. 
> 
> Like pg, where starting a replica has the same cli as starting the pg server, we should have had the same here for all the cases and should have allowed the role be picked up from the contents of the gtm.conf file. 
> 
> One area where would this pain is in HA. Typical HA configs expect same directory structures across the nodes and the same clis to start the resources etc. Now adding additional logic to include "-Z gtm/standby/proxy" introduces additional pain in this. 
Thanks for those arguments, but here my goal is to clean up the code and facilitate maintenance. I am not planning to change the current configuration specs.

> 
> My 2 cents.
> 
> Regards,
> Nikhils
> 
> On Mon, Oct 29, 2012 at 5:29 PM, Michael Paquier <mic...@gm...> wrote:
> 
> 
> On Mon, Oct 29, 2012 at 7:19 PM, Ashutosh Bapat <ash...@en...> wrote:
> Thanks for taking this up.
> 
> On Mon, Oct 29, 2012 at 3:08 PM, Michael Paquier <mic...@gm...> wrote:
> Hi all,
> 
> I am currently working on cleaning up the dependencies and duplications that GTM code has.
> This is a pretty complicated task, as GTM has many code duplication with libpq and also src/backend.
> 
> By the way, I am trying to do this clean up with a maximum number of discrete steps, making the progression clear. One of the things I noticed is related to the GTM configuration system. It is a duplication of what can be found in guc.c. However, guc.c is used exclusively by PG backend and all its structures remain internal to GUC. In the case of GTM, those APIs are used by both GTM and GTM-proxy as they both use a configuration file, so GTM configuration APIs need to be externalized to avoid duplication between the things used by GTM and by GTM-proxy. As far as I saw, there is no way to resolve the PG-GUC/GTM conf duplication as we would need to externalize all the structures in guc.c.
> However, the GTM configuration APIs are located in src/gtm/common, which generates an archive called libgtm.a, this archive being used by src/backend for the construction of the binary file postgres. We need to remove this dependency between the GTM configuration and PG backend for clarity, so please find attached a patch that moves the GTM configuration APIs into a new folder src/gtm/config and avoids a useless dependency of this code with PG backend.
> 
> On using guc.c, is there a separation of (I haven't looked at the file itself) separate the datastructures (the array where GUCs and there values are stored or fetched from etc.) and functions exposed. If so, we might copy the data structures but leave the functions common to PG and GTM. Are there any other tools in PG code which also require a configuration file and share the guc interfaces.
> Each configuration file in PG uses a different structure, with GUC being the most complex.
> For example recovery.conf uses readRecoveryCommandFile in xlog.c, postgresql.conf uses guc.c. ident.conf and pg_hba.conf also use something different (didn't have a look where but it is easy to find out). However, regarding the GUC part, there is an interface available with guc-file.l. In this file there are functions that GTM is duplicating and we should definitely to use that (functions like ProcessConfigFile or ParseConfigFile for example). Please note that they may need some extension as the existing functions are designed only for src/backend as far as I saw. The ultimate goal would be to use those APIs also with GTM for the configuration part, and to keep the list of parameters in GTM. Before reaching that point I want to reduce the dependencies between GTM modules to understand how to untie one by one all the duplications that GTM has with things like memory, list, configuration, StringInfo, Assert, etc.
> Even before untying everything there are a couple of things missing for this work:
> - change the Makefile structure in GTM to avoid relying on static archives and reach the same state as src/backend that use for example objfiles.txt. I have basically a patch, just need to clean it up
> - move gtm_ctl out of src/gtm to src/bin, also got a patch here, but it needs a little bit more work to have clean makefiles.
> -- 
> Michael Paquier
> http://michael.otacoo.com
> 
> ------------------------------------------------------------------------------
> The Windows 8 Center - In partnership with Sourceforge
> Your idea - your app - 30 days.
> Get started!
> http://windows8center.sourceforge.net/
> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> 
> 
> 
> -- 
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Service

I have found writing -Z GTM/GTM_STANDY/GTM_PROXY as part of the cli pretty
painful.

Infact there's not much difference between GTM and GTM_STANDBY at all in
terms of processing from the front end. Also I am not sure why we have
gtm_proxy.conf for the proxy and gtm.conf for GTM and the STANDBY.

Like pg, where starting a replica has the same cli as starting the pg
server, we should have had the same here for all the cases and should have
allowed the role be picked up from the contents of the gtm.conf file.

One area where would this pain is in HA. Typical HA configs expect same
directory structures across the nodes and the same clis to start the
resources etc. Now adding additional logic to include "-Z
gtm/standby/proxy" introduces additional pain in this.

My 2 cents.

Regards,
Nikhils

On Mon, Oct 29, 2012 at 5:29 PM, Michael Paquier
<mic...@gm...>wrote:

>
>
> On Mon, Oct 29, 2012 at 7:19 PM, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Thanks for taking this up.
>>
>> On Mon, Oct 29, 2012 at 3:08 PM, Michael Paquier <
>> mic...@gm...> wrote:
>>
>>> Hi all,
>>>
>>> I am currently working on cleaning up the dependencies and duplications
>>> that GTM code has.
>>> This is a pretty complicated task, as GTM has many code duplication with
>>> libpq and also src/backend.
>>>
>>> By the way, I am trying to do this clean up with a maximum number of
>>> discrete steps, making the progression clear. One of the things I noticed
>>> is related to the GTM configuration system. It is a duplication of what can
>>> be found in guc.c. However, guc.c is used exclusively by PG backend and all
>>> its structures remain internal to GUC. In the case of GTM, those APIs are
>>> used by both GTM and GTM-proxy as they both use a configuration file, so
>>> GTM configuration APIs need to be externalized to avoid duplication between
>>> the things used by GTM and by GTM-proxy. As far as I saw, there is no way
>>> to resolve the PG-GUC/GTM conf duplication as we would need to externalize
>>> all the structures in guc.c.
>>> However, the GTM configuration APIs are located in src/gtm/common, which
>>> generates an archive called libgtm.a, this archive being used by
>>> src/backend for the construction of the binary file postgres. We need to
>>> remove this dependency between the GTM configuration and PG backend for
>>> clarity, so please find attached a patch that moves the GTM configuration
>>> APIs into a new folder src/gtm/config and avoids a useless dependency of
>>> this code with PG backend.
>>>
>>
>> On using guc.c, is there a separation of (I haven't looked at the file
>> itself) separate the datastructures (the array where GUCs and there values
>> are stored or fetched from etc.) and functions exposed. If so, we might
>> copy the data structures but leave the functions common to PG and GTM. Are
>> there any other tools in PG code which also require a configuration file
>> and share the guc interfaces.
>>
> Each configuration file in PG uses a different structure, with GUC being
> the most complex.
> For example recovery.conf uses readRecoveryCommandFile in xlog.c,
> postgresql.conf uses guc.c. ident.conf and pg_hba.conf also use something
> different (didn't have a look where but it is easy to find out). However,
> regarding the GUC part, there is an interface available with guc-file.l. In
> this file there are functions that GTM is duplicating and we should
> definitely to use that (functions like ProcessConfigFile or ParseConfigFile
> for example). Please note that they may need some extension as the existing
> functions are designed only for src/backend as far as I saw. The ultimate
> goal would be to use those APIs also with GTM for the configuration part,
> and to keep the list of parameters in GTM. Before reaching that point I
> want to reduce the dependencies between GTM modules to understand how to
> untie one by one all the duplications that GTM has with things like memory,
> list, configuration, StringInfo, Assert, etc.
> Even before untying everything there are a couple of things missing for
> this work:
> - change the Makefile structure in GTM to avoid relying on static archives
> and reach the same state as src/backend that use for example objfiles.txt.
> I have basically a patch, just need to clean it up
> - move gtm_ctl out of src/gtm to src/bin, also got a patch here, but it
> needs a little bit more work to have clean makefiles.
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> The Windows 8 Center - In partnership with Sourceforge
> Your idea - your app - 30 days.
> Get started!
> http://windows8center.sourceforge.net/
> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Service

On Mon, Oct 29, 2012 at 7:19 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Thanks for taking this up.
>
> On Mon, Oct 29, 2012 at 3:08 PM, Michael Paquier <
> mic...@gm...> wrote:
>
>> Hi all,
>>
>> I am currently working on cleaning up the dependencies and duplications
>> that GTM code has.
>> This is a pretty complicated task, as GTM has many code duplication with
>> libpq and also src/backend.
>>
>> By the way, I am trying to do this clean up with a maximum number of
>> discrete steps, making the progression clear. One of the things I noticed
>> is related to the GTM configuration system. It is a duplication of what can
>> be found in guc.c. However, guc.c is used exclusively by PG backend and all
>> its structures remain internal to GUC. In the case of GTM, those APIs are
>> used by both GTM and GTM-proxy as they both use a configuration file, so
>> GTM configuration APIs need to be externalized to avoid duplication between
>> the things used by GTM and by GTM-proxy. As far as I saw, there is no way
>> to resolve the PG-GUC/GTM conf duplication as we would need to externalize
>> all the structures in guc.c.
>> However, the GTM configuration APIs are located in src/gtm/common, which
>> generates an archive called libgtm.a, this archive being used by
>> src/backend for the construction of the binary file postgres. We need to
>> remove this dependency between the GTM configuration and PG backend for
>> clarity, so please find attached a patch that moves the GTM configuration
>> APIs into a new folder src/gtm/config and avoids a useless dependency of
>> this code with PG backend.
>>
>
> On using guc.c, is there a separation of (I haven't looked at the file
> itself) separate the datastructures (the array where GUCs and there values
> are stored or fetched from etc.) and functions exposed. If so, we might
> copy the data structures but leave the functions common to PG and GTM. Are
> there any other tools in PG code which also require a configuration file
> and share the guc interfaces.
>
Each configuration file in PG uses a different structure, with GUC being
the most complex.
For example recovery.conf uses readRecoveryCommandFile in xlog.c,
postgresql.conf uses guc.c. ident.conf and pg_hba.conf also use something
different (didn't have a look where but it is easy to find out). However,
regarding the GUC part, there is an interface available with guc-file.l. In
this file there are functions that GTM is duplicating and we should
definitely to use that (functions like ProcessConfigFile or ParseConfigFile
for example). Please note that they may need some extension as the existing
functions are designed only for src/backend as far as I saw. The ultimate
goal would be to use those APIs also with GTM for the configuration part,
and to keep the list of parameters in GTM. Before reaching that point I
want to reduce the dependencies between GTM modules to understand how to
untie one by one all the duplications that GTM has with things like memory,
list, configuration, StringInfo, Assert, etc.
Even before untying everything there are a couple of things missing for
this work:
- change the Makefile structure in GTM to avoid relying on static archives
and reach the same state as src/backend that use for example objfiles.txt.
I have basically a patch, just need to clean it up
- move gtm_ctl out of src/gtm to src/bin, also got a patch here, but it
needs a little bit more work to have clean makefiles.
-- 
Michael Paquier
http://michael.otacoo.com

Thanks for taking this up.

On Mon, Oct 29, 2012 at 3:08 PM, Michael Paquier
<mic...@gm...>wrote:

> Hi all,
>
> I am currently working on cleaning up the dependencies and duplications
> that GTM code has.
> This is a pretty complicated task, as GTM has many code duplication with
> libpq and also src/backend.
>
> By the way, I am trying to do this clean up with a maximum number of
> discrete steps, making the progression clear. One of the things I noticed
> is related to the GTM configuration system. It is a duplication of what can
> be found in guc.c. However, guc.c is used exclusively by PG backend and all
> its structures remain internal to GUC. In the case of GTM, those APIs are
> used by both GTM and GTM-proxy as they both use a configuration file, so
> GTM configuration APIs need to be externalized to avoid duplication between
> the things used by GTM and by GTM-proxy. As far as I saw, there is no way
> to resolve the PG-GUC/GTM conf duplication as we would need to externalize
> all the structures in guc.c.
> However, the GTM configuration APIs are located in src/gtm/common, which
> generates an archive called libgtm.a, this archive being used by
> src/backend for the construction of the binary file postgres. We need to
> remove this dependency between the GTM configuration and PG backend for
> clarity, so please find attached a patch that moves the GTM configuration
> APIs into a new folder src/gtm/config and avoids a useless dependency of
> this code with PG backend.
>

On using guc.c, is there a separation of (I haven't looked at the file
itself) separate the datastructures (the array where GUCs and there values
are stored or fetched from etc.) and functions exposed. If so, we might
copy the data structures but leave the functions common to PG and GTM. Are
there any other tools in PG code which also require a configuration file
and share the guc interfaces.


>
> Comments?
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> The Windows 8 Center - In partnership with Sourceforge
> Your idea - your app - 30 days.
> Get started!
> http://windows8center.sourceforge.net/
> what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Thu, Oct 18, 2012 at 3:37 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Any change in allowing certain distribution scheme has direct impact on
> the planner code. The planner code decides pushability of certain things
> like aggregates, grouping, JOINs based on the distribution column. Hence,
> any change in the distribution code needs a corresponding change in planner.
>

> I think, we need to get the distribution strategy linked with the
> constraint exclusion somehow, so that constraint exclusion can be used by
> planner for optimizations and then work on elaborate distribution
> strategies. That would help in supporting range and list distribution as
> well.
>
We agree on this, and that is the role of the shippability part of the
planner (pgxcship.c) to check for a given Query if its constraints and
foreign keys evaluation need the Coordinator to be evaluated or not. That
is basically the work I did by providing transparent APIs. For the time
being those APIs are only used to block the creation of a constraint that
cannot be evaluated on remote nodes, but once XC supports global constraint
it will be necessary to plug them directly in the shippability planner.
That is basically the work I did last week.

My point here is that the index expression shippability is not checked
enough yet, and needs to be reinforced in the structure I added with my
recent commits, structure that is easily extensible for distribution lists
and range things.
-- 
Michael Paquier
http://michael.otacoo.com

Any change in allowing certain distribution scheme has direct impact on the
planner code. The planner code decides pushability of certain things like
aggregates, grouping, JOINs based on the distribution column. Hence, any
change in the distribution code needs a corresponding change in planner.

I think, we need to get the distribution strategy linked with the
constraint exclusion somehow, so that constraint exclusion can be used by
planner for optimizations and then work on elaborate distribution
strategies. That would help in supporting range and list distribution as
well.

On Thu, Oct 18, 2012 at 12:03 PM, Michael Paquier <mic...@gm...
> wrote:

>
>
> On Thu, Oct 18, 2012 at 3:24 PM, Koichi Suzuki <ko...@in...>wrote:
>
>> Year, this is the case we don't support yet and we need global constraint
>> to generally enforce this.
>>
>> On the other hand, I think we can extend distribution based on function
>> (abs(a) in this case) such as
>> create table tab (a int) distribute by hash(abs(a));   This may help some
>> applications.
>>
> Sure. In this case a unique constraint containing the expression abs(a) is
> even pushable.
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Thu, Oct 18, 2012 at 3:24 PM, Koichi Suzuki <ko...@in...>wrote:

> Year, this is the case we don't support yet and we need global constraint
> to generally enforce this.
>
> On the other hand, I think we can extend distribution based on function
> (abs(a) in this case) such as
> create table tab (a int) distribute by hash(abs(a));   This may help some
> applications.
>
Sure. In this case a unique constraint containing the expression abs(a) is
even pushable.
-- 
Michael Paquier
http://michael.otacoo.com

Year, this is the case we don't support yet and we need global constraint to generally enforce this.

On the other hand, I think we can extend distribution based on function (abs(a) in this case) such as
create table tab (a int) distribute by hash(abs(a));   This may help some applications.

Regards;
---
Koichi


On Thu, 18 Oct 2012 14:49:48 +0900
Michael Paquier <mic...@gm...> wrote:

> Hi all,
> 
> I have been spending some time thinking about the pushdown of constraint
> expressions in the case of hash-modulo tables, and for the time being the
> code is still unsafe. For example:
> postgres=# create table tab (a int) distribute by hash(a);
> CREATE TABLE
> postgres=# create unique index tab_unique on tab(abs(a));
> CREATE INDEX
> postgres=# insert into tab values (2); -- sent to datanode 1
> INSERT 0 1
> postgres=# insert into tab values (-2); -- sent to datanode 2
> INSERT 0 1
> I did that with a cluster of 2 Datanodes. The 2nd INSERT should not be
> authorized as it breaks the unique constraint. For a hash-distributed
> table, the unique constraint abs(a) can be created because with current
> master (9ce38d3) the expression is thought as safe for remote evaluation as
> it contains the distribution column. So for unique/primary key expressions
> on value-distributed tables, I was thinking about the following rules to
> determine if the expression can be safely evaluated on remote nodes:
> - parent relation data is on a single node (true whatever the constraint
> PKEY, UNIQUE, EXCLUDE type and distribution type btw)
> - unique index contains no expressions
> - unique index contains the distribution column
> This set of rules makes the unique index pushable if it uses as least the
> distribution column and no expressions, it is OK even with other columns.
> 
> For the time being, hash/modulo supports only single column, but when this
> support will be added the following rules are necessary:
> - unique constraint is pushable all the distribution expression, a
> distribution expression being a column or an expression based on several
> columns.
> 
> Please note that XC does not support global constraint but can only manage
> constraints that can be evaluated on remote nodes.
> 
> This has as consequences the patch attached.
> Comments?
> -- 
> Michael Paquier
> http://michael.otacoo.com

On Sun, Oct 14, 2012 at 6:39 PM, Nikhil Sontakke <ni...@st...>wrote:

>
>
>>  Could you add a test in regression xc_copy to cover that we do not
>> reintroduce such issues in the future?
>> The same test can be used with both 1.0 and head.
>> Thanks,
>>
>>
> Alright, here you are.
>
Thanks. Patches have been committed, I just applied some esthetic changes.
-- 
Michael Paquier
http://michael.otacoo.com

2012/10/14 Michael Paquier <mic...@gm...>

>
>
> On Sun, Oct 14, 2012 at 3:52 PM, Nikhil Sontakke <ni...@st...>wrote:
>
>> Getting back to this:
>>
>> Even the column names have issues with respect to quoting. Consider the
>> following:
>>
>> postgres=# create table foo (a int, "placing" varchar);
>> postgres=# \copy foo (a, "placing") from '/tmp/foo.copy';
>> ERROR:  syntax error at or near "placing"
>>
>> The RemoteCopy_QuoteIdentifier function does not handle keywords based
>> variables at all. Is there a reason why we did not take inspiration from
>> the existing quote_identifier function when we wrote this remote version?
>>
> Good question. Andrei is the former writer of this part of the code. I
> think it has just been forgotten and not thought about.
>
>
That was many years ago, I do not remember details. I might not even know
about the existing quote_identifier function.


>
> Regards,
>> Nikhils
>>
>> On Fri, Aug 17, 2012 at 12:33 PM, Nikhil Sontakke <ni...@st...>wrote:
>>
>>> >
>>> > Not really. Take a look at my latest patch; there the Query structure
>>> is
>>> > constructed and deparsed, for shipping JOINs.
>>> >
>>>
>>> Yup, as long as we can generate a proper Query structure, it can be
>>> deparsed properly. Infact if we follow this rule everywhere then one
>>> standard set of functions will get used and we will avoid such
>>> handcrafting mistakes that are sprinkled around.
>>>
>>> Regards,
>>> Nikhils
>>>
>>> >>
>>> >>
>>> >> > If the deparsing
>>> >> > logic is not in place, how difficult is it to construct it?
>>> >> >
>>> >>
>>> >> Most of the glue is present in utils/adt/ruleutils.c anyways, so that
>>> >> can be re-used most of the times IMO.
>>> >>
>>> > That's good.
>>> >
>>> >>
>>> >> > Will you be able to take that up, as part of this fix?
>>> >> >
>>> >> > Otherwise the patch looks good.
>>> >> >
>>> >>
>>> >> Am swamped currently, so unfortunately I cannot take this up for now.
>>> >> I think this patch is simple enough and affects important activities
>>> >> like dump, copy etc. So we should get this in for now.
>>> >>
>>> >
>>> > That's what we keep on doing and fix small things at a time. I am
>>> afraid
>>> > that, there are many such unnoticed bugs out there, because of
>>> hand-crafting
>>> > the queries.
>>> >
>>> >>
>>> >> Regards,
>>> >> Nikhils
>>> >>
>>> >> >
>>> >> > On Thu, Aug 16, 2012 at 8:22 PM, Nikhil Sontakke <
>>> ni...@st...>
>>> >> > wrote:
>>> >> >>
>>> >> >> > I would imagine that this issue exists also in 1.0 stable.
>>> >> >>
>>> >> >> Indeed it does. PFA, patch against 1.0 stable.
>>> >> >>
>>> >> >> Regards,
>>> >> >> Nikhils
>>> >> >>
>>> >> >> > Did you test it on master only?
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> Regards,
>>> >> >> >> Nikhils
>>> >> >> >>
>>> >> >> >> On Thu, Aug 16, 2012 at 6:57 PM, Nikhil Sontakke
>>> >> >> >> <ni...@st...>
>>> >> >> >> wrote:
>>> >> >> >> > Hi,
>>> >> >> >> >
>>> >> >> >> > I think there's an issue with the COPY command either due to
>>> an
>>> >> >> >> > improper merge or due to some XC specific code in the COPY
>>> code
>>> >> >> >> > path:
>>> >> >> >> >
>>> >> >> >> > postgres=# create table "user"(x int, y int);
>>> >> >> >> > postgres=# insert into "user" values (1,2);
>>> >> >> >> > postgres=# copy public."user"(x, y) to stdout;
>>> >> >> >> > ERROR:  syntax error at or near "user"
>>> >> >> >> >
>>> >> >> >> > I myself will be on the lookout for the issue, but just
>>> wanted to
>>> >> >> >> > post
>>> >> >> >> > it here to make everyone aware..
>>> >> >> >> >
>>> >> >> >> > Regards,
>>> >> >> >> > Nikhils
>>> >> >> >> > --
>>> >> >> >> > StormDB - http://www.stormdb.com
>>> >> >> >> > The Database Cloud
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> --
>>> >> >> >> StormDB - http://www.stormdb.com
>>> >> >> >> The Database Cloud
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> ------------------------------------------------------------------------------
>>> >> >> >> Live Security Virtual Conference
>>> >> >> >> Exclusive live event will cover all the ways today's security
>>> and
>>> >> >> >> threat landscape has changed and how IT managers can respond.
>>> >> >> >> Discussions
>>> >> >> >> will include endpoint security, mobile security and the latest
>>> in
>>> >> >> >> malware
>>> >> >> >> threats.
>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> >> >> >> _______________________________________________
>>> >> >> >> Postgres-xc-developers mailing list
>>> >> >> >> Pos...@li...
>>> >> >> >>
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Michael Paquier
>>> >> >> > http://michael.otacoo.com
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> StormDB - http://www.stormdb.com
>>> >> >> The Database Cloud
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> ------------------------------------------------------------------------------
>>> >> >> Live Security Virtual Conference
>>> >> >> Exclusive live event will cover all the ways today's security and
>>> >> >> threat landscape has changed and how IT managers can respond.
>>> >> >> Discussions
>>> >> >> will include endpoint security, mobile security and the latest in
>>> >> >> malware
>>> >> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> >> >> _______________________________________________
>>> >> >> Postgres-xc-developers mailing list
>>> >> >> Pos...@li...
>>> >> >>
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Best Wishes,
>>> >> > Ashutosh Bapat
>>> >> > EntepriseDB Corporation
>>> >> > The Enterprise Postgres Company
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> StormDB - http://www.stormdb.com
>>> >> The Database Cloud
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Best Wishes,
>>> > Ashutosh Bapat
>>> > EntepriseDB Corporation
>>> > The Enterprise Postgres Company
>>> >
>>>
>>>
>>>
>>> --
>>> StormDB - http://www.stormdb.com
>>> The Database Cloud
>>>
>>
>>
>>
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>> Postgres-XC Support and Service
>>
>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Andrei Martsinchyk

StormDB - http://www.stormdb.com
The Database Cloud

On Sun, Oct 14, 2012 at 6:02 PM, Nikhil Sontakke <ni...@st...>wrote:

>
>
>> postgres=# create table foo (a int, "placing" varchar);
>>> postgres=# \copy foo (a, "placing") from '/tmp/foo.copy';
>>> ERROR:  syntax error at or near "placing"
>>>
>>> The RemoteCopy_QuoteIdentifier function does not handle keywords based
>>> variables at all. Is there a reason why we did not take inspiration from
>>> the existing quote_identifier function when we wrote this remote version?
>>>
>>
>
>> Good question. Andrei is the former writer of this part of the code. I
>> think it has just been forgotten and not thought about.
>>
>>
>>
> Ok, I hope this was not done on purpose for some historical or other
> reasons I am not aware of. Here are patches for both head and 1.0. I think
> we should just stick to using the existing quote_identifier function. Seem
> to work ok with some bit of testing.
>
Could you add a test in regression xc_copy to cover that we do not
reintroduce such issues in the future?
The same test can be used with both 1.0 and head.
Thanks,
-- 
Michael Paquier
http://michael.otacoo.com

Getting back to this:

Even the column names have issues with respect to quoting. Consider the
following:

postgres=# create table foo (a int, "placing" varchar);
postgres=# \copy foo (a, "placing") from '/tmp/foo.copy';
ERROR:  syntax error at or near "placing"

The RemoteCopy_QuoteIdentifier function does not handle keywords based
variables at all. Is there a reason why we did not take inspiration from
the existing quote_identifier function when we wrote this remote version?

Regards,
Nikhils

On Fri, Aug 17, 2012 at 12:33 PM, Nikhil Sontakke <ni...@st...>wrote:

> >
> > Not really. Take a look at my latest patch; there the Query structure is
> > constructed and deparsed, for shipping JOINs.
> >
>
> Yup, as long as we can generate a proper Query structure, it can be
> deparsed properly. Infact if we follow this rule everywhere then one
> standard set of functions will get used and we will avoid such
> handcrafting mistakes that are sprinkled around.
>
> Regards,
> Nikhils
>
> >>
> >>
> >> > If the deparsing
> >> > logic is not in place, how difficult is it to construct it?
> >> >
> >>
> >> Most of the glue is present in utils/adt/ruleutils.c anyways, so that
> >> can be re-used most of the times IMO.
> >>
> > That's good.
> >
> >>
> >> > Will you be able to take that up, as part of this fix?
> >> >
> >> > Otherwise the patch looks good.
> >> >
> >>
> >> Am swamped currently, so unfortunately I cannot take this up for now.
> >> I think this patch is simple enough and affects important activities
> >> like dump, copy etc. So we should get this in for now.
> >>
> >
> > That's what we keep on doing and fix small things at a time. I am afraid
> > that, there are many such unnoticed bugs out there, because of
> hand-crafting
> > the queries.
> >
> >>
> >> Regards,
> >> Nikhils
> >>
> >> >
> >> > On Thu, Aug 16, 2012 at 8:22 PM, Nikhil Sontakke <ni...@st...
> >
> >> > wrote:
> >> >>
> >> >> > I would imagine that this issue exists also in 1.0 stable.
> >> >>
> >> >> Indeed it does. PFA, patch against 1.0 stable.
> >> >>
> >> >> Regards,
> >> >> Nikhils
> >> >>
> >> >> > Did you test it on master only?
> >> >> >>
> >> >> >>
> >> >> >> Regards,
> >> >> >> Nikhils
> >> >> >>
> >> >> >> On Thu, Aug 16, 2012 at 6:57 PM, Nikhil Sontakke
> >> >> >> <ni...@st...>
> >> >> >> wrote:
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > I think there's an issue with the COPY command either due to an
> >> >> >> > improper merge or due to some XC specific code in the COPY code
> >> >> >> > path:
> >> >> >> >
> >> >> >> > postgres=# create table "user"(x int, y int);
> >> >> >> > postgres=# insert into "user" values (1,2);
> >> >> >> > postgres=# copy public."user"(x, y) to stdout;
> >> >> >> > ERROR:  syntax error at or near "user"
> >> >> >> >
> >> >> >> > I myself will be on the lookout for the issue, but just wanted
> to
> >> >> >> > post
> >> >> >> > it here to make everyone aware..
> >> >> >> >
> >> >> >> > Regards,
> >> >> >> > Nikhils
> >> >> >> > --
> >> >> >> > StormDB - http://www.stormdb.com
> >> >> >> > The Database Cloud
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> StormDB - http://www.stormdb.com
> >> >> >> The Database Cloud
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> ------------------------------------------------------------------------------
> >> >> >> Live Security Virtual Conference
> >> >> >> Exclusive live event will cover all the ways today's security and
> >> >> >> threat landscape has changed and how IT managers can respond.
> >> >> >> Discussions
> >> >> >> will include endpoint security, mobile security and the latest in
> >> >> >> malware
> >> >> >> threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >> >> >> _______________________________________________
> >> >> >> Postgres-xc-developers mailing list
> >> >> >> Pos...@li...
> >> >> >>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Michael Paquier
> >> >> > http://michael.otacoo.com
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> StormDB - http://www.stormdb.com
> >> >> The Database Cloud
> >> >>
> >> >>
> >> >>
> >> >>
> ------------------------------------------------------------------------------
> >> >> Live Security Virtual Conference
> >> >> Exclusive live event will cover all the ways today's security and
> >> >> threat landscape has changed and how IT managers can respond.
> >> >> Discussions
> >> >> will include endpoint security, mobile security and the latest in
> >> >> malware
> >> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >> >> _______________________________________________
> >> >> Postgres-xc-developers mailing list
> >> >> Pos...@li...
> >> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best Wishes,
> >> > Ashutosh Bapat
> >> > EntepriseDB Corporation
> >> > The Enterprise Postgres Company
> >> >
> >>
> >>
> >>
> >> --
> >> StormDB - http://www.stormdb.com
> >> The Database Cloud
> >
> >
> >
> >
> > --
> > Best Wishes,
> > Ashutosh Bapat
> > EntepriseDB Corporation
> > The Enterprise Postgres Company
> >
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Service

On Sun, Oct 14, 2012 at 3:52 PM, Nikhil Sontakke <ni...@st...>wrote:

> Getting back to this:
>
> Even the column names have issues with respect to quoting. Consider the
> following:
>
> postgres=# create table foo (a int, "placing" varchar);
> postgres=# \copy foo (a, "placing") from '/tmp/foo.copy';
> ERROR:  syntax error at or near "placing"
>
> The RemoteCopy_QuoteIdentifier function does not handle keywords based
> variables at all. Is there a reason why we did not take inspiration from
> the existing quote_identifier function when we wrote this remote version?
>
Good question. Andrei is the former writer of this part of the code. I
think it has just been forgotten and not thought about.


Regards,
> Nikhils
>
> On Fri, Aug 17, 2012 at 12:33 PM, Nikhil Sontakke <ni...@st...>wrote:
>
>> >
>> > Not really. Take a look at my latest patch; there the Query structure is
>> > constructed and deparsed, for shipping JOINs.
>> >
>>
>> Yup, as long as we can generate a proper Query structure, it can be
>> deparsed properly. Infact if we follow this rule everywhere then one
>> standard set of functions will get used and we will avoid such
>> handcrafting mistakes that are sprinkled around.
>>
>> Regards,
>> Nikhils
>>
>> >>
>> >>
>> >> > If the deparsing
>> >> > logic is not in place, how difficult is it to construct it?
>> >> >
>> >>
>> >> Most of the glue is present in utils/adt/ruleutils.c anyways, so that
>> >> can be re-used most of the times IMO.
>> >>
>> > That's good.
>> >
>> >>
>> >> > Will you be able to take that up, as part of this fix?
>> >> >
>> >> > Otherwise the patch looks good.
>> >> >
>> >>
>> >> Am swamped currently, so unfortunately I cannot take this up for now.
>> >> I think this patch is simple enough and affects important activities
>> >> like dump, copy etc. So we should get this in for now.
>> >>
>> >
>> > That's what we keep on doing and fix small things at a time. I am afraid
>> > that, there are many such unnoticed bugs out there, because of
>> hand-crafting
>> > the queries.
>> >
>> >>
>> >> Regards,
>> >> Nikhils
>> >>
>> >> >
>> >> > On Thu, Aug 16, 2012 at 8:22 PM, Nikhil Sontakke <
>> ni...@st...>
>> >> > wrote:
>> >> >>
>> >> >> > I would imagine that this issue exists also in 1.0 stable.
>> >> >>
>> >> >> Indeed it does. PFA, patch against 1.0 stable.
>> >> >>
>> >> >> Regards,
>> >> >> Nikhils
>> >> >>
>> >> >> > Did you test it on master only?
>> >> >> >>
>> >> >> >>
>> >> >> >> Regards,
>> >> >> >> Nikhils
>> >> >> >>
>> >> >> >> On Thu, Aug 16, 2012 at 6:57 PM, Nikhil Sontakke
>> >> >> >> <ni...@st...>
>> >> >> >> wrote:
>> >> >> >> > Hi,
>> >> >> >> >
>> >> >> >> > I think there's an issue with the COPY command either due to an
>> >> >> >> > improper merge or due to some XC specific code in the COPY code
>> >> >> >> > path:
>> >> >> >> >
>> >> >> >> > postgres=# create table "user"(x int, y int);
>> >> >> >> > postgres=# insert into "user" values (1,2);
>> >> >> >> > postgres=# copy public."user"(x, y) to stdout;
>> >> >> >> > ERROR:  syntax error at or near "user"
>> >> >> >> >
>> >> >> >> > I myself will be on the lookout for the issue, but just wanted
>> to
>> >> >> >> > post
>> >> >> >> > it here to make everyone aware..
>> >> >> >> >
>> >> >> >> > Regards,
>> >> >> >> > Nikhils
>> >> >> >> > --
>> >> >> >> > StormDB - http://www.stormdb.com
>> >> >> >> > The Database Cloud
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> StormDB - http://www.stormdb.com
>> >> >> >> The Database Cloud
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> ------------------------------------------------------------------------------
>> >> >> >> Live Security Virtual Conference
>> >> >> >> Exclusive live event will cover all the ways today's security and
>> >> >> >> threat landscape has changed and how IT managers can respond.
>> >> >> >> Discussions
>> >> >> >> will include endpoint security, mobile security and the latest in
>> >> >> >> malware
>> >> >> >> threats.
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> >> >> >> _______________________________________________
>> >> >> >> Postgres-xc-developers mailing list
>> >> >> >> Pos...@li...
>> >> >> >>
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Michael Paquier
>> >> >> > http://michael.otacoo.com
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> StormDB - http://www.stormdb.com
>> >> >> The Database Cloud
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> ------------------------------------------------------------------------------
>> >> >> Live Security Virtual Conference
>> >> >> Exclusive live event will cover all the ways today's security and
>> >> >> threat landscape has changed and how IT managers can respond.
>> >> >> Discussions
>> >> >> will include endpoint security, mobile security and the latest in
>> >> >> malware
>> >> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> >> >> _______________________________________________
>> >> >> Postgres-xc-developers mailing list
>> >> >> Pos...@li...
>> >> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best Wishes,
>> >> > Ashutosh Bapat
>> >> > EntepriseDB Corporation
>> >> > The Enterprise Postgres Company
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> StormDB - http://www.stormdb.com
>> >> The Database Cloud
>> >
>> >
>> >
>> >
>> > --
>> > Best Wishes,
>> > Ashutosh Bapat
>> > EntepriseDB Corporation
>> > The Enterprise Postgres Company
>> >
>>
>>
>>
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Service
>



-- 
Michael Paquier
http://michael.otacoo.com

Hi Ashutosh,

Thanks for spending time on that. Even simply moving the code from
src/backend/pgxc/planner to src/backend/optimizer/plan makes it more
postgres-like. No problems with compilation and regressions.
I found the following things about this patch:
- presence of a space in tab at line 900 => (errmsg("Partition column can't
be updated in current version"))));
- src/include/optimizer/pgxcplan.h still refers to planner.h in its header.
You need also to update the path written in header.

Once all those things are fixed, it looks OK for commit.
Thanks,

On Tue, Oct 9, 2012 at 3:04 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi All,
> PFA the patch to move XC planner code to PG specific directories.
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Michael Paquier
http://michael.otacoo.com

Thanks. I took the patch, fixed it and committed it.
Fix has also been included in 1.0 stable.

On Wed, Oct 3, 2012 at 6:38 PM, Nikhil Sontakke <ni...@st...> wrote:

> Hi,
>
> There's an issue in the RemoteCopy_GetRelationLoc function. For certain
> tables like catalog tables or contrib configuration tables there is no
> relation location info available as of now.
>
> Trying a COPY on such a table causes a segmentation fault. A simple but a
> bit contrived way to reproduce this is:
>
> postgres=# copy pg_catalog.pgxc_node from stdin;
> The connection to the server was lost. Attempting reset: Failed.
>
> The attached patch fixes this.
>
> Regards,
> Nikhils
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
> Postgres-XC Support and Service
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Michael Paquier
http://michael.otacoo.com

On Thu, Oct 4, 2012 at 7:21 AM, Ashutosh Bapat
<ash...@en...> wrote:
>
>
> On Thu, Oct 4, 2012 at 4:49 PM, Michael Paquier <mic...@gm...>
> wrote:
>>
>> ¥d contains also views and sequences, which have no distribution data. So
>> to me it doesn't make sense to add it there. It should definitely be done by
>> another command. For example ¥dp shows privilege accesses, so why not doing
>> a new command like ¥dz or similar to list only the list of tables (no views,
>> no sequences) with their distribution data?
>
>
> That's a fair argument. But if tomorrow PG uses z for something else, we
> need to change breaking backward compatibility. We should find existing
> subcommand to \d which is only be used for tables.

I tend to agree with Michael here. While it is probably fairly safe to
modify the existing commands, I would prefer some new command option
and stay compatible as much as possible.

-- 
Mason Sharp

StormDB - http://www.stormdb.com
The Database Cloud
Postgres-XC Support and Services