postgres-xc-developers Mailing List for Postgres-XC

On Fri, Mar 30, 2012 at 8:05 PM, Andrei Martsinchyk <
and...@gm...> wrote:

> Hi Michael,
>
> Just found the reason why referential integrity check does not work on
> datanodes.
> Here is the code in RI_FKey_Check (ri_triggers.c):
>
> #ifdef PGXC
>  /*
>  * XXX Referential integrity is not yet supported. We just come out of the
>  * function without actually performing any integrity checks. This must be
>  * fixed when we support local/global referential integrities.
>  */
> return PointerGetDatum(NULL);
> #endif
>
> I guess it will work if these lines will be removed.
>
Hum, OK. It would be nice to make a try, but XC does not support yet local
and global referential integrity.
So perhaps in the particular case of the patch it will work, but I would
except side effects.
Could you update your patch with those lines deleted?
-- 
Michael Paquier
http://michael.otacoo.com

Hi Michael,

Just found the reason why referential integrity check does not work on
datanodes.
Here is the code in RI_FKey_Check (ri_triggers.c):

#ifdef PGXC
/*
 * XXX Referential integrity is not yet supported. We just come out of the
 * function without actually performing any integrity checks. This must be
 * fixed when we support local/global referential integrities.
 */
return PointerGetDatum(NULL);
#endif

I guess it will work if these lines will be removed.

23 марта 2012 г. 3:32 пользователь Michael Paquier <
mic...@gm...> написал:

>
>
> On Thu, Mar 22, 2012 at 5:50 PM, Andrei Martsinchyk <
> and...@gm...> wrote:
>
>> Hi Michael,
>>
>> So you find "A tuple cannot be fetched from Coordinator" error is better
>> then "could not read block 0 in file "base/12052/32771": read only 0 of
>> 8192 bytes" ?
>> I think both are equally misty for the end user.
>> I have reported the bug some time ago (ID: 3485997) and proposed a patch,
>> that fixes the problem.
>> The idea is to skip AFTER UPDATE/DELETE triggers if target relation is
>> remote. The triggers enforcing the foreign key are still fired on the
>> datanodes.
>> Is something wrong with my patch?
>>
> Your patch does not protect data consistency on foreign constraints, which
> is perhaps worse than the bug referenced.
> With this test case:
>
> create table x1 (id int, value timestamp);
> create table x2 (id int, value timestamp);
> alter table x1 add primary key (id);
> alter table x2 add foreign key (id) references x1(id);
> insert into x1 values (1, now());
> insert into x2 values (2, now());
>
> XC inserts the tuple.
>
> Postgres returns this error:
> ERROR:  insert or update on table "x2" violates foreign key constraint
> "x2_id_fkey"
> DETAIL:  Key (id)=(2) is not present in table "x1".
>
>
>> 22 марта 2012 г. 9:31 пользователь Michael Paquier <
>> mic...@gm...> написал:
>>
>> >
>> > Hi,
>> >
>> > Please find attached a patch that handles smoothly the following case:
>> > postgres=# create table x1 (id int, value timestamp);
>> > CREATE TABLE
>> > postgres=# create table x2 (id int, value timestamp);
>> > CREATE TABLE
>> > postgres=# alter table x1 add primary key (id);
>> > NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index
>> "x1_pkey" for table "x1"
>> > ALTER TABLE
>> > postgres=# alter table x2 add foreign key (id) references x1(id);
>> > ALTER TABLE
>> > postgres=# insert into x1 values (1, now());
>> > INSERT 0 1
>> > postgres=# insert into x2 values (1, now());
>> > INSERT 0 1
>> > postgres=# update x1 set value=now() where id=1;
>> > ERROR: could not read block 0 in file "base/12052/32768": read only 0
>> of 8192 bytes
>> > postgres=# update x2 set value=now() where id=1;
>> > ERROR: could not read block 0 in file "base/12052/32771": read only 0
>> of 8192 bytes
>> >
>> > What happens here is that the update query tries to fetch a tuple on
>> Coordinator to check foreign key consistency by using a trigger mechanism.As
>> > As we do not target that in 1.0, I propose simply to block such cases
>> with the patch attached.
>> > Opinions?
>> > --
>> > Michael Paquier
>> > http://michael.otacoo.com
>> >
>> >
>> ------------------------------------------------------------------------------
>> > This SF email is sponsosred by:
>> > Try Windows Azure free for 90 days Click Here
>> > http://p.sf.net/sfu/sfd2d-msazure
>> > _______________________________________________
>> > Postgres-xc-developers mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> >
>>
>>
>>
>> --
>> Best regards,
>> Andrei Martsinchyk                   mailto:and...@gm...
>>
>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
Best regards,
Andrei Martsinchyk                   mailto:and...@gm...

I wonder why ctid is involved here.   This might have been introduced by very old code and may need to look into it.  Not a simple work but this may be related to many other potential problems.

Regards;
---
Koichi Suzuki

On Fri, 30 Mar 2012 11:09:10 +0900
Michael Paquier <mic...@gm...> wrote:

> Hi Ashutosh,
> 
> I have been working around the area of remote DELETE and remote UPDATE
> planning. I am trying to have a general support to allow users to use
> freely rules and joins on those queries.
> A first step of my work was to have a look at what was happening for
> DELETE, and I found that the target list generated in rewriteHandler is not
> correct because it fetched some TLEs that are not always from this relation.
> It resulted in the commit I did yesterday which was a quick fix, and you
> were correct, it is possible to use the varno in the TLE's expressions, so
> setting origtbl is not necessary.
> 
> However, before doing anything else, I have been digging deeply in the code.
> Just by having a look at the code now, I can say the following things
> regarding remote UPDATE and DELETE:
> 1) The code in create_remotedelete_plan is correct. But...
> I saw some issues with the materialization target list at the bottom of
> remote DELETE node in planning.
> Let's take a simple example:
> create table aa (a int, b int);
> create table bb (a int, b int);
> 
> And here is a resulting plan:
> postgres=# explain verbose delete from aa using bb where bb.a = aa.a and
> bb.b = aa.b;
> 
> QUERY PLAN
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>  Delete on public.aa  (cost=0.00..0.00 rows=0 width=0)
>    Node/s: dn1, dn2
>    Remote query: DELETE FROM public.aa WHERE a = $1 AND b = $2 AND ctid =
> $3 AND $4 = $4
>    ->  Result  (cost=0.00..0.00 rows=0 width=0)
>          Output: "__REMOTE_JOIN_QUERY__".a_1_1_1,
> "__REMOTE_JOIN_QUERY__".b_1_2_1, "__REMOTE_JOIN_QUERY__".ctid_1_1_1,
> "__REMOTE_JOIN_QUERY__".ctid_2_1_1
>          ->  Materialize  (cost=0.00..0.00 rows=0 width=0)
>                Output: "__REMOTE_JOIN_QUERY__".a_1_1_1,
> "__REMOTE_JOIN_QUERY__".b_1_2_1, "__REMOTE_JOIN_QUERY__".ctid_1_1_1,
> "__REMOTE_JOIN_QUERY__".ctid_2_1_1
>                ->  Data Node Scan on "__REMOTE_JOIN_QUERY__"
> (cost=0.00..0.00 rows=1000 width=14)
>                      Output: "__REMOTE_JOIN_QUERY__".a_1_1_1,
> "__REMOTE_JOIN_QUERY__".b_1_2_1, "__REMOTE_JOIN_QUERY__".ctid_1_1_1,
> "__REMOTE_JOIN_QUERY__".ctid_2_1_1
>                      Node/s: dn1, dn2
>                      Remote query: SELECT in_1.a AS a_1_1_1 ...
> 
> The output node on top of __REMOTE_JOIN_QUERY__ is obviously incorrect.
> What we get here is "aa.a,aa.b, aa.ctid, bb.ctid", this is incorrect
> because the ctid of the second relation involved in the join should not be
> included in the final target list to generate a correct remote DELETE.
> I suppose there is an error at the top of create_remotejoin_plan, but it
> will be helpful if you could direct me where this could happen.
> 
> For the time being, I added in master branch a trick to bypass parameters
> that are not from the relation (here refer to the "$4 = $4"), but this is
> definitely incorrect.
> 
> 2) the code in create_remoteupdate_plan is incorrect.
> This code needs to be definitely reworked. Postgres has already some
> mechanisms to pre-generate a target list, but what we should do is
> something like that to complete it:
> 
> Based on the example above (This is based on current master with my patch
> but is incorrect):
> postgres=# explain verbose update bb set b = 2 where b = 5;
>                                      QUERY
> PLAN
> ------------------------------------------------------------------------------------
>  Update on public.bb  (cost=0.00..0.00 rows=1000 width=14)
>    Node/s: dn1, dn2
>    Node expr: bb.a
>    Remote query: UPDATE public.bb SET b = $2 WHERE b = $3 AND ctid = $4 AND
> a = $1
>    ->  Result  (cost=0.00..0.00 rows=1000 width=14)
>          Output: a, 2, b, ctid
>          ->  Data Node Scan on bb  (cost=0.00..0.00 rows=1000 width=14)
>                Output: a, b, ctid
>                Node/s: dn1, dn2
>                Remote query: SELECT a, b, ctid FROM ONLY bb WHERE (b = 5)
> In this case the target list that we define in rewriter is "b, ctid". I
> added in rewriteHandler a function to scan the vars from parse tree and to
> add in target list some new conditions are ctid is not always consistent in
> cluster.
> Thanks to Postgres, this target list is completed by the new values to set
> for the update query, here on plan it becomes "new.a, new.b, targetlist.b,
> targetlist.ctid".
> What I am planning to do is to generate the UPDATE query like this:
> UPDATE bb SET a = $1, b = $2 WHERE b = $3, ctid = $4;
> 
> You can see that for current planner the query generated is incorrect.
> I believe that UPDATE has the same problems as DELETE regarding the target
> list received from remote join plan, so my first focus will be to fix the
> final target list to insure correct outputs.
> Then the rest of the code will be simple to implement as I am already
> experienced in this area.
> 
> I am adding to this email a WIP patch I used for my investigations.
> Regards,
> -- 
> Michael Paquier
> http://michael.otacoo.com

There are a couple of things I want to discuss about in this area, and it
is related to general query planning.
I'll send an email soon.

On Thu, Mar 29, 2012 at 3:42 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi Michael,
> Was this patch discussed on the mailing list? I did not see any relevant
> discussion.
>
> There can a possibility that resorigtbl is set the result table but it's
> not added in rewriting. In this case, would there be erroneous column
> updation? Instead of resorigtbl can you use some attribute inside the Var
> node like varno?
>
> On Thu, Mar 29, 2012 at 10:01 AM, Michael Paquier <
> mic...@us...> wrote:
>
>> Project "Postgres-XC".
>>
>> The branch, master has been updated
>>       via  8427ea322eabca18fa21f286c2284aa4ffa105a8 (commit)
>>      from  0f11b36d79827bd8b2ed66a31e074e1a8a86fba3 (commit)
>>
>>
>> - Log -----------------------------------------------------------------
>>
>> http://postgres-xc.git.sourceforge.net/git/gitweb.cgi?p=postgres-xc/postgres-xc;a=commitdiff;h=8427ea322eabca18fa21f286c2284aa4ffa105a8
>>
>> commit 8427ea322eabca18fa21f286c2284aa4ffa105a8
>> Author: Michael P <mi...@ot...>
>> Date:   Thu Mar 29 13:32:53 2012 +0900
>>
>>    Correct remote DELETE planning for multiple relations
>>
>>    In case several relations were involved in a remote DELETE (rules,
>>    join, etc.), remote query was not correctly generated because it
>>    took into account not only its target entries, but also the ones
>>    from other relations.
>>    A reference of the original table is added in target entries and
>>    this is compared to the relation whose remote DELETE is generated
>>    to insure that all the operations are consistent.
>>
>>    This had consequences like bug 3512657, but also on the usage of
>>    rules on DELETE.
>>
>> M       src/backend/optimizer/plan/createplan.c
>> M       src/backend/rewrite/rewriteHandler.c
>>
>> -----------------------------------------------------------------------
>>
>> Summary of changes:
>>  src/backend/optimizer/plan/createplan.c |   13 ++++++++++---
>>  src/backend/rewrite/rewriteHandler.c    |    9 +++++++++
>>  2 files changed, 19 insertions(+), 3 deletions(-)
>>
>>
>> hooks/post-receive
>> --
>> Postgres-XC
>>
>>
>> ------------------------------------------------------------------------------
>> This SF email is sponsosred by:
>> Try Windows Azure free for 90 days Click Here
>> http://p.sf.net/sfu/sfd2d-msazure
>> _______________________________________________
>> Postgres-xc-committers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
>
> ------------------------------------------------------------------------------
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure
> _______________________________________________
> Postgres-xc-committers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-committers
>
>


-- 
Michael Paquier
http://michael.otacoo.com

On Thu, Mar 29, 2012 at 2:49 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Sorry,
> Here's the correct one,

Thanks. Sorry for my late reply.
It looks good to go, I don't get any failures.
-- 
Michael Paquier
http://michael.otacoo.com

On Tue, Mar 27, 2012 at 1:00 AM, Michael Paquier
<mic...@gm...> wrote:
> Hi all,
>
> Please find attached a finalized version of the patch, fully stabilized.
> There are no problem with regressions and sequence DDL management is
> consistent in the cluster.
> Regards,
>

Nice. Is there an existing regression test for this? If not, you may
want the patch to include adding a new test case.

Regards,

Mason


>
> On Tue, Mar 27, 2012 at 11:38 AM, Michael Paquier
> <mic...@gm...> wrote:
>>
>> Hi all,
>>
>> Please find attached a patch that implements a callback mechanism on GTM
>> to maintain gtm creation and drop on GTM consistent with transactions.
>> Here are some examples:
>>
>> 1) Rollback a drop transaction
>> postgres=# begin;
>> BEGIN
>> postgres=# create sequence foo;
>> CREATE SEQUENCE
>> postgres=# commit;
>> COMMIT
>> postgres=# select nextval('foo');
>>  nextval
>> ---------
>>        1
>> (1 row)
>> postgres=# select nextval('foo');
>>  nextval
>> ---------
>>        2
>> (1 row)
>> postgres=# begin;
>> BEGIN
>> postgres=# drop sequence foo;
>> DROP SEQUENCE
>> postgres=# rollback;
>> ROLLBACK
>> postgres=# select nextval('foo');
>>  nextval
>> ---------
>>        3
>> (1 row)
>>
>> 2) rollback a create transaction
>> postgres=# begin;
>> BEGIN
>> postgres=# create sequence hoge;
>> CREATE SEQUENCE
>> postgres=# rollback;
>> ROLLBACK
>> postgres=# select nextval('hoge');
>> ERROR:  relation "hoge" does not exist
>> LINE 1: select nextval('hoge');
>>                        ^
>> postgres=# create sequence hoge;
>> CREATE SEQUENCE
>> postgres=# select nextval('hoge');
>>  nextval
>> ---------
>>        1
>> (1 row)
>>
>> The patch is not completely stable and I am doing some refactoring, but
>> gives an idea of the mechanisms I am using.
>> Comments are welcome.
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>
>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>
> ------------------------------------------------------------------------------
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Hi,

This is the last patch for pgxc_clean.   This patch is pgxc_clean itself, based upon several preceding patches.

Regards;
---
Koichi Suzuki

On Thu, Mar 29, 2012 at 1:35 PM, Pavan Deolasee <
pav...@en...> wrote:

>
> On Tue, Mar 27, 2012 at 5:00 AM, Michael Paquier <
> mic...@gm...> wrote:
>
>> Hi all,
>>
>> Please find attached a finalized version of the patch, fully stabilized.
>> There are no problem with regressions and sequence DDL management is
>> consistent in the cluster.
>>
>
> I saw you committed the patch. It looked good to me and I was about to
> send a note regarding the same. The only thing I wondered was if its
> required to invent new event types and could we have used the existing
> commit/prepare/abort events defined in xact.h
>
Perhaps it was not necessary but it makes the definition more consistent
with xact and subxact.
It is just a question of viewpoint, but it made more sense to my mind to
have a definition of events separated for GTM to make them more clear and
independant.
-- 
Michael Paquier
http://michael.otacoo.com

On Tue, Mar 27, 2012 at 5:00 AM, Michael Paquier <mic...@gm...>
 wrote:

> Hi all,
>
> Please find attached a finalized version of the patch, fully stabilized.
> There are no problem with regressions and sequence DDL management is
> consistent in the cluster.
>

I saw you committed the patch. It looked good to me and I was about to send
a note regarding the same. The only thing I wondered was if its required to
invent new event types and could we have used the existing
commit/prepare/abort events defined in xact.h

Thanks,
Pavan

-- 
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

Hi all,

As defined by the initial plan, we will move forward to beta1 on the master
branch tomorrow afternoon (JP time).
This means that officially features are freezed and that we'll focus only
on stabilization.
Regards,
-- 
Michael Paquier
http://michael.otacoo.com

Hum, you uploaded here the wrong patch I believe...

On Wed, Mar 28, 2012 at 4:27 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi All,
> As per our discussion in thread "floating point values get truncated
> during I/O between datanode and coordinator", I am adding alternate
> output file for aggregates test. The aggregates are giving correct results
> barring the error introduced by rounding effect.
>
> Apart from that, the aggregates test prints plans for certain aggregates
> like min, max which can be optimized as index scan + LIMIT. In Postgres-XC,
> however, we do not apply this optimization on coordinator as coordinator
> does not have any index data. So, we will never see those optimized plans.
> Hence changed the plans in expected output and alternate expected output.
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>
>
> ------------------------------------------------------------------------------
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Michael Paquier
http://michael.otacoo.com

Thanks.   This looks much better.   I will include this change after
committing xc_maintenance_mode patch.

Regards;
----------
Koichi Suzuki



2012年3月27日20:52 Pavan Deolasee <pav...@en...>:
>
>
> On Tue, Mar 27, 2012 at 2:29 AM, Koichi Suzuki <ko...@in...>
> wrote:
>>
>> Hi,
>>
>> Enclosed is a patch to extend COMMIT/ROLLBACK PREPARED statements with
>> xc_maintenance_mode GUC.   When this GUC is on, these command will not take
>> care of remote node.   It just takes care of local node status and update
>> GTM status.
>>
>> Pavan:
>> Could you please take a look at this patch?   I tried to be careful where
>> to touch.   It is very helpful if you review this and point out if other
>> module should be modified.
>>
>
> The patch looks OK to me. It doesn't apply though because I think there are
> some issues with the doc related diffs. That should be easy to handle
> though. The xc_maintenance_mode.h file is still missing from the patch.
>
> I reworked the execRemote.c to minimize changes. I haven't tested the change
> though. I also removed couple of lines of commented code. Please see
> attached.
>
> Thanks,
> Pavan
>
>
>
> --
> Pavan Deolasee
> EnterpriseDB http://www.enterprisedb.com
>
> ------------------------------------------------------------------------------
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Hi All,
As per our discussion in thread "floating point values get truncated during
I/O between datanode and coordinator", I am adding alternate output file
for aggregates test. The aggregates are giving correct results barring the
error introduced by rounding effect.

Apart from that, the aggregates test prints plans for certain aggregates
like min, max which can be optimized as index scan + LIMIT. In Postgres-XC,
however, we do not apply this optimization on coordinator as coordinator
does not have any index data. So, we will never see those optimized plans.
Hence changed the plans in expected output and alternate expected output.

-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hi,
----------
Koichi Suzuki



2012年3月27日19:27 Pavan Deolasee <pav...@en...>:
>
>
> On Tue, Mar 27, 2012 at 8:27 AM, Koichi Suzuki <ko...@in...>
> wrote:
>>
>> Hi,
>>
>> Enclosed is a revised patch to check xid for implicit 2PC.   Now it is
>> checked and not allowed to use in explicit PREPARE TRANSACTION.
>>
>
> Couple of quick comments:
>
> 1. I think it should suffice to just check for GIDs starting with the
> pattern "_$XC$" to declare then invalid. Also, a hint regarding system
> reserved GIDs will be very useful to the user

Agreed.

>
> 2. Instead of hardcoding the length as 5, why not just use strlen() to do
> compute that ? I think compiler would be smart enough to optimize repeated
> evaluation if thats what worries you.

I was not sure if strlen() for the constant string can be computed by
the compiler.

I will replace this with strlen().

> Thanks,
> Pavan
>
> --
> Pavan Deolasee
> EnterpriseDB http://www.enterprisedb.com
>
> ------------------------------------------------------------------------------
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

On Tue, Mar 27, 2012 at 8:27 AM, Koichi Suzuki <ko...@in...>wrote:

> Hi,
>
> Enclosed is a revised patch to check xid for implicit 2PC.   Now it is
> checked and not allowed to use in explicit PREPARE TRANSACTION.
>
>
Couple of quick comments:

1. I think it should suffice to just check for GIDs starting with the
pattern "_$XC$" to declare then invalid. Also, a hint regarding system
reserved GIDs will be very useful to the user

2. Instead of hardcoding the length as 5, why not just use strlen() to do
compute that ? I think compiler would be smart enough to optimize repeated
evaluation if thats what worries you.

Thanks,
Pavan

-- 
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

Hmm, ok,
Here's the patch.

On Tue, Mar 27, 2012 at 2:01 PM, Michael Paquier
<mic...@gm...>wrote:

>
>
>
>
> On 2012/03/27, at 17:12, Ashutosh Bapat <ash...@en...>
> wrote:
>
>
>
> On Tue, Mar 27, 2012 at 1:34 PM, Michael Paquier <
> mic...@gm...> wrote:
>
>> Thanks.
>> It is always better to avoid using boolean and have enums in this case,
>> so it looks great.
>> Some headers should be added for pgxc_FQS_set_reason and
>> pgxc_FQS_test_reason.
>> Regressions also look good.
>>
>>
> The headers for these functions are added in the same file? Do you mean
> something else?
>
> The short explanation on top of those functions.
>
>
>
>>
>> On Tue, Mar 27, 2012 at 4:29 PM, Ashutosh Bapat <
>> ash...@en...> wrote:
>>
>>> Here's patch aligned to latest HEAD.
>>>
>>>
>>> On Tue, Mar 27, 2012 at 12:45 PM, Michael Paquier <
>>> mic...@gm...> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Mar 27, 2012 at 4:06 PM, Ashutosh Bapat <
>>>> ash...@en...> wrote:
>>>>
>>>>> Hi All,
>>>>> PFA patch which encorporates unshippability reasons in FQS code. Right
>>>>> now, we only have a single flag canShip to decide whether the query is
>>>>> shippable or not. But we need more granularity in this status, i.e why the
>>>>> query is not shippable. This helps in 1. debugging 2. do some more
>>>>> optimizations in FQS. An example of 2 is queries with ORDER BY on
>>>>> distributed table. Such queries can not be shipped right now as they are.
>>>>> What needs to be done is to add a Sort node with merging capability on top
>>>>> of RemoteQuery node. Similar optimizations will be possible for LIMIT,
>>>>> DISTINCT, aggregates etc, if we know the reason why the query is not
>>>>> shippable as it is.
>>>>>
>>>>> Tested regression, no new regression found.
>>>>>
>>>> This patch does not apply correctly on current head. You should realign
>>>> it.
>>>> --
>>>> Michael Paquier
>>>> http://michael.otacoo.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Enterprise Postgres Company
>>>
>>>
>>
>>
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On 2012/03/27, at 17:12, Ashutosh Bapat <ash...@en...> wrote:

> 
> 
> On Tue, Mar 27, 2012 at 1:34 PM, Michael Paquier <mic...@gm...> wrote:
> Thanks.
> It is always better to avoid using boolean and have enums in this case, so it looks great.
> Some headers should be added for pgxc_FQS_set_reason and pgxc_FQS_test_reason.
> Regressions also look good.
> 
> 
> The headers for these functions are added in the same file? Do you mean something else?
The short explanation on top of those functions.
>  
> 
> On Tue, Mar 27, 2012 at 4:29 PM, Ashutosh Bapat <ash...@en...> wrote:
> Here's patch aligned to latest HEAD.
> 
> 
> On Tue, Mar 27, 2012 at 12:45 PM, Michael Paquier <mic...@gm...> wrote:
> 
> 
> On Tue, Mar 27, 2012 at 4:06 PM, Ashutosh Bapat <ash...@en...> wrote:
> Hi All,
> PFA patch which encorporates unshippability reasons in FQS code. Right now, we only have a single flag canShip to decide whether the query is shippable or not. But we need more granularity in this status, i.e why the query is not shippable. This helps in 1. debugging 2. do some more optimizations in FQS. An example of 2 is queries with ORDER BY on distributed table. Such queries can not be shipped right now as they are. What needs to be done is to add a Sort node with merging capability on top of RemoteQuery node. Similar optimizations will be possible for LIMIT, DISTINCT, aggregates etc, if we know the reason why the query is not shippable as it is.
> 
> Tested regression, no new regression found.
> This patch does not apply correctly on current head. You should realign it.
> -- 
> Michael Paquier
> http://michael.otacoo.com
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
> 
> 
> 
> 
> -- 
> Michael Paquier
> http://michael.otacoo.com
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
> 

Hi,

Enclosed is a revised patch to check xid for implicit 2PC.   Now it is checked and not allowed to use in explicit PREPARE TRANSACTION.

Regards;
---
Koichi Suzuki

On Tue, Mar 27, 2012 at 1:34 PM, Michael Paquier
<mic...@gm...>wrote:

> Thanks.
> It is always better to avoid using boolean and have enums in this case, so
> it looks great.
> Some headers should be added for pgxc_FQS_set_reason and
> pgxc_FQS_test_reason.
> Regressions also look good.
>
>
The headers for these functions are added in the same file? Do you mean
something else?


>
> On Tue, Mar 27, 2012 at 4:29 PM, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Here's patch aligned to latest HEAD.
>>
>>
>> On Tue, Mar 27, 2012 at 12:45 PM, Michael Paquier <
>> mic...@gm...> wrote:
>>
>>>
>>>
>>> On Tue, Mar 27, 2012 at 4:06 PM, Ashutosh Bapat <
>>> ash...@en...> wrote:
>>>
>>>> Hi All,
>>>> PFA patch which encorporates unshippability reasons in FQS code. Right
>>>> now, we only have a single flag canShip to decide whether the query is
>>>> shippable or not. But we need more granularity in this status, i.e why the
>>>> query is not shippable. This helps in 1. debugging 2. do some more
>>>> optimizations in FQS. An example of 2 is queries with ORDER BY on
>>>> distributed table. Such queries can not be shipped right now as they are.
>>>> What needs to be done is to add a Sort node with merging capability on top
>>>> of RemoteQuery node. Similar optimizations will be possible for LIMIT,
>>>> DISTINCT, aggregates etc, if we know the reason why the query is not
>>>> shippable as it is.
>>>>
>>>> Tested regression, no new regression found.
>>>>
>>> This patch does not apply correctly on current head. You should realign
>>> it.
>>> --
>>> Michael Paquier
>>> http://michael.otacoo.com
>>>
>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Thanks.
It is always better to avoid using boolean and have enums in this case, so
it looks great.
Some headers should be added for pgxc_FQS_set_reason and
pgxc_FQS_test_reason.
Regressions also look good.

On Tue, Mar 27, 2012 at 4:29 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Here's patch aligned to latest HEAD.
>
>
> On Tue, Mar 27, 2012 at 12:45 PM, Michael Paquier <
> mic...@gm...> wrote:
>
>>
>>
>> On Tue, Mar 27, 2012 at 4:06 PM, Ashutosh Bapat <
>> ash...@en...> wrote:
>>
>>> Hi All,
>>> PFA patch which encorporates unshippability reasons in FQS code. Right
>>> now, we only have a single flag canShip to decide whether the query is
>>> shippable or not. But we need more granularity in this status, i.e why the
>>> query is not shippable. This helps in 1. debugging 2. do some more
>>> optimizations in FQS. An example of 2 is queries with ORDER BY on
>>> distributed table. Such queries can not be shipped right now as they are.
>>> What needs to be done is to add a Sort node with merging capability on top
>>> of RemoteQuery node. Similar optimizations will be possible for LIMIT,
>>> DISTINCT, aggregates etc, if we know the reason why the query is not
>>> shippable as it is.
>>>
>>> Tested regression, no new regression found.
>>>
>> This patch does not apply correctly on current head. You should realign
>> it.
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>


-- 
Michael Paquier
http://michael.otacoo.com

Here's patch aligned to latest HEAD.

On Tue, Mar 27, 2012 at 12:45 PM, Michael Paquier <mic...@gm...
> wrote:

>
>
> On Tue, Mar 27, 2012 at 4:06 PM, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Hi All,
>> PFA patch which encorporates unshippability reasons in FQS code. Right
>> now, we only have a single flag canShip to decide whether the query is
>> shippable or not. But we need more granularity in this status, i.e why the
>> query is not shippable. This helps in 1. debugging 2. do some more
>> optimizations in FQS. An example of 2 is queries with ORDER BY on
>> distributed table. Such queries can not be shipped right now as they are.
>> What needs to be done is to add a Sort node with merging capability on top
>> of RemoteQuery node. Similar optimizations will be possible for LIMIT,
>> DISTINCT, aggregates etc, if we know the reason why the query is not
>> shippable as it is.
>>
>> Tested regression, no new regression found.
>>
> This patch does not apply correctly on current head. You should realign it.
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On Tue, Mar 27, 2012 at 4:06 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi All,
> PFA patch which encorporates unshippability reasons in FQS code. Right
> now, we only have a single flag canShip to decide whether the query is
> shippable or not. But we need more granularity in this status, i.e why the
> query is not shippable. This helps in 1. debugging 2. do some more
> optimizations in FQS. An example of 2 is queries with ORDER BY on
> distributed table. Such queries can not be shipped right now as they are.
> What needs to be done is to add a Sort node with merging capability on top
> of RemoteQuery node. Similar optimizations will be possible for LIMIT,
> DISTINCT, aggregates etc, if we know the reason why the query is not
> shippable as it is.
>
> Tested regression, no new regression found.
>
This patch does not apply correctly on current head. You should realign it.
-- 
Michael Paquier
http://michael.otacoo.com

Hi All,
PFA patch which encorporates unshippability reasons in FQS code. Right now,
we only have a single flag canShip to decide whether the query is shippable
or not. But we need more granularity in this status, i.e why the query is
not shippable. This helps in 1. debugging 2. do some more optimizations in
FQS. An example of 2 is queries with ORDER BY on distributed table. Such
queries can not be shipped right now as they are. What needs to be done is
to add a Sort node with merging capability on top of RemoteQuery node.
Similar optimizations will be possible for LIMIT, DISTINCT, aggregates etc,
if we know the reason why the query is not shippable as it is.

Tested regression, no new regression found.


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On 27 March 2012 09:43, Michael Paquier <mic...@gm...> wrote:

> We also maintain Pgconn pointers inside pooler manager.
> So perhaps we could use some libpq APIs to set up connection parameters
> when connections are created.
>

True. The libpq API are anyway getting used by pooler manager to connect to
datanodes. We can insert parameters in the connection string.

This is just a suggestion though, not sure if it is feasable.
>
> On Tue, Mar 27, 2012 at 12:08 PM, Amit Khandekar <
> ami...@en...> wrote:
>
>>
>>
>> On 27 March 2012 05:14, Michael Paquier <mic...@gm...>wrote:
>>
>>> I am wondering about something.
>>> Those are environment variables. So on the same coordinator we are sure
>>> that all the connections will use the same parameters.
>>> So why extending SET commands to set up environment parameters to have a
>>> pooler agent-based mechanismwhen we know all the environment variables have
>>> the same values for all the session of the node? We should use a pool-based
>>> mechanism for this purpose.
>>> I believe that what we should do is first to register all the
>>> environment variables in the shape of a string in DatabasePool, and set up
>>> this list of parameters once a new connection to a node is made as those
>>> environment parameters will be the same for all the sessions of the node.
>>>
>>
>> Ok, there are two places one can set the default datestyle:
>>
>> 1. One is at the server side. There we can set in postgresql.conf, or
>> using env variable. By env var, I mean, when booting up the server, it
>> reads the PGDATESTYLE var that is set in the shell from which the server is
>> being booted, and if it is set, that is the one it chooses as its default.
>>
>> 2. A user sets PGDATESTYLE env variable in his own shell, and then starts
>> the psql session from this shell. This env var becomes the default
>> datestyle for the particular *backend session*, regardless of whatever is
>> the default date format in the sever-side. The database connection string
>> itself contains this new datestyle value.
>>
>> Our scenario is the scenario #2 above. The coordinator and datanodes are
>> brought up without datestyle set. Then a psql client is started with the
>> new PGDATESTYLE="Postgres, MDY" whereas the nodes datestyle is the default
>> i.e. "ISO, MDY".
>>
>> So, we cannot assume that all connections use the same parameters. Each
>> user client session using a particular coordinator can have different
>> PGDATESTYLE values. That means, even if the coordinator is same, the pooler
>> connections from that coordinator to datanodes can have different
>> datestyles.
>>
> OK didn't know about that. So it has to be agent-based.
> So, what you could do is extend the pooler agent with a env_param string
> with the same model as local_params and global_params.
> It might be also necessary to add a SET param type in poolmgr.h, like
> SET_ENV.
>

Yes, definitely we need to maintain a new set of parameters, which can be
called connection_params, and would be used in libpq connection APIs . This
set can contain the server's own env variable values, which will be
overwritten by those set by the client session.


>
>> It's true that the coordinator can have it's own default datestyle set
>> using the same env variable. (scenario#1) So, we should anyway pass the
>> coordinator default value in the coord->datanode connection string, so that
>> the timestamp values returned always match the coordinator datestyle. Over
>> and above that, if the client session has it's own value set, then that
>> will override the coordinator value, and also the pooler connection would
>> not be shared with other sessions because datestyle is session-specific.
>>
>
>
>>
>> Then what we need to do is to determine what are the parameters we can
>>> pass. Just by having a look at your list all the parameters related to
>>> dates are necessary. But there are also parameters related to ssl
>>> connections. We would need to pass down that too perhaps.
>>>
>>
>> Yes, we need to carefully go through these parameters to see if we should
>> propagate these, regardless of whether they are set using SET or usign env
>> vars. I had mentioned about the same work while  working on set_config()
>> function.
>>
>>
>>> There is one remaining problem. Does a SET command on an environment
>>> variable (I mean typed by the user or application, not decided by an
>>> environment variable on coordinator) gets propagated to the remote nodes?
>>> Smth like: SET PGDATESTYLE to "Postgres, MDY". I think yes but...
>>> If the answer is yes, the current SET mechanism is enough as it behaves
>>> like common SET LOCAL/GLOBAL commands.
>>> If the answer is no, we would need to set up a new mechanism for SET
>>> strings like the one you mention below.
>>>
>>
>> Yes, the SET DATESTYLE command gets correctly propagated. So we don't
>> want to change the SET mechanism. But the datestyle is not propagated if it
>> is set using env variable. So to propagate the env variable, we can either
>> propagate it by converting to SET param statement and the SET mechanism
>> will take care of it, or propagate it using coord-datanode connection
>> string.  The problem with converting to SET param is, when a user runs
>> RESET, it will reset back to the hardcoded one, when it should be reset to
>> the env var value, not the hard-coded default one.
>>
> So if you extend pooler agent with an environment parameter string, we
> will need to refire the environment parameters on pooler each time a reset
> is done.
> This gives an additional reason to have an additional parameter layer in
> pooler for session parameters.
> --
> Michael Paquier
> http://michael.otacoo.com
>

For this patch this is it.

The old FQS code, which is dead now, is scattered at quite a few places.
Somehow the dead code is defined in such a way, that it doesn't show as
warnings in unused code. So, I just clean it up whenever I see one.
Hopefully, there is not much left now.

On Tue, Mar 27, 2012 at 6:02 AM, Michael Paquier
<mic...@gm...>wrote:

>
>
> On Mon, Mar 26, 2012 at 7:09 PM, Ashutosh Bapat <
> ash...@en...> wrote:
>
>> Hi All,
>>
>> PFA patch with some more things cleaned up. I do not see any new failures
>> in the regression.
>>
> Looks good. Is it the last patch you are planning to do or will you add
> smth else?
>
>>
>> On Mon, Mar 12, 2012 at 12:18 PM, Ashutosh Bapat <
>> ash...@en...> wrote:
>>
>>> Hi All,
>>> PFA patch for more cleanup.
>>>
>>> With the new FQS code, we don't create child plans beneath RemoteQuery
>>> node. Now RemoteQuery node should be leaf node in plan tree. I also removed
>>> member tts_dataNodeIndex from TupleTableSlot.
>>>
>>> Regression ran without any new failures.
>>>
>>> On Fri, Mar 9, 2012 at 5:22 AM, Michael Paquier <
>>> mic...@gm...> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Mar 8, 2012 at 8:13 PM, Ashutosh Bapat <
>>>> ash...@en...> wrote:
>>>>
>>>>> Hi All,
>>>>> PFA patch for cleanup of dead-code. There are many functions and
>>>>> structure members, which are not used by the new FQS code. Removed them.
>>>>>
>>>>> Regression does not show any new error.
>>>>>
>>>> This patch looks good, but it is not realigned with the latest head.
>>>> Could you realign it?
>>>> Except that, the content looks OK.
>>>> --
>>>> Michael Paquier
>>>> http://michael.otacoo.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Enterprise Postgres Company
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Enterprise Postgres Company
>>
>>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company