postgres-xc-developers Mailing List for Postgres-XC

On Mon, Jul 30, 2012 at 1:31 PM, Joshua D. Drake <jd...@co...> wrote:
>
> Hello,
>
> I wonder if it makes sense to only allow long options after a short
> option. For example:
>
> -XC --coordinator
>

I like your original long options idea.  I think it is sufficiently
unlikely to conflict with PostgreSQL that it is enough.

> So the only way to use the XC options is to pass -XC and then a long
> option. That should keep us completely clear of PostgreSQL.
>
> Sincerely,
>
> Joshua D. Drake
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers



-- 
Mason Sharp

StormDB - http://www.stormdb.com
The Database Cloud
Also Offering Postgres-XC Support and Services

Hi,

Problem Description
===================

Consider the following use case

create table junk (a int, b int);
insert into junk values(1,1),(1,2),(1,3),(2,4),(2,5),(3,6),(3,7),(3,8);
create table junk2(a int, b int);
insert into junk2
values(0,11),(0,22),(0,33),(1,44),(1,55),(10,66),(10,77),(20,88);

select * from junk, junk2 where junk.a=junk2.a;

 a | b | a | b
---+---+---+----
 1 | 1 | 1 | 44
 1 | 1 | 1 | 55
 1 | 2 | 1 | 44
 1 | 2 | 1 | 55
 1 | 3 | 1 | 44
 1 | 3 | 1 | 55
(6 rows)

BEGIN
 declare c1 cursor for select * from junk, junk2 where junk.a=junk2.a for
update;
 fetch 1 from c1;

At this point try this query from another psql prompt
 update junk set b=200 where a=2;

should not block since we should have locked only the rows where a=1
In our current implementation the above query blocks since we lock complete
tables.


Suggested Solution
==================

1. Select for update/share if run inside a transaction and all cursor
queries
   will contain nodeid and ctid as additional target columns.

2. Support for a new SQL command used to lock individual rows will be added

    LOCK ROWS FROM TABLE name IN lockmode MODE WHERE condition

    where lockmode is one of:

        ACCESS SHARE | ACCESS EXCLUSIVE

3. The LockRows node would work differently on coordinator, whereas it
would work
   as it it on data nodes. On the coordinator It will create a list of
nodeid,ctid
   pairs one for each table whose rows are supposed to be locked.
   Consider the example where we have two data nodes

    test=# explain verbose select * from junk, junk2 where junk.a=junk2.a
for update;
                                           QUERY
PLAN

-------------------------------------------------------------------------------------------------
     LockRows  (cost=0.00..0.02 rows=1 width=16)
       Output: junk.a, junk.b, junk2.a, junk2.b
       ->  Nested Loop  (cost=0.00..0.01 rows=1 width=16)
         Output: junk.a, junk.b, junk2.a, junk2.b
         Join Filter: (junk.a = junk2.a)
         ->  Data Node Scan on junk "_REMOTE_TABLE_QUERY_"
(cost=0.00..0.00 rows=1000 width=8)
               Output: junk.a, junk.b
               Node/s: data_node_1, data_node_2
               Remote query: SELECT a, b FROM ONLY junk WHERE true FOR
UPDATE OF junk
         ->  Data Node Scan on junk2 "_REMOTE_TABLE_QUERY_"
(cost=0.00..0.00 rows=1000 width=8)
               Output: junk2.a, junk2.b
               Node/s: data_node_1, data_node_2
               Remote query: SELECT a, b FROM ONLY junk2 WHERE true FOR
UPDATE OF junk2
    (13 rows)


    In each run the LockRows node would add one nodeid,ctid pair for each
table in the list.
    When the coordinator executes ExecEndLockRows it will prepare LOCK ROWS
query
    one for each table and for each data node. In case you have two data
nodes and
    the list of nodeid,ctid pairs contain nodeids of both the datanodes then
    we will have a total of four queries two for one data node and two for
the other.
    Each query would look like

    LOCK ROWS FROM TABLE junk IN ACCESS EXCLUSIVE MODE
    WHERE ctid in (.....) containing ctids of first data node

    LOCK ROWS FROM TABLE junk IN ACCESS EXCLUSIVE MODE
    WHERE ctid in (.....) containing ctids of second data node

    LOCK ROWS FROM TABLE junk2 IN ACCESS EXCLUSIVE MODE
    WHERE ctid in (.....) containing ctids of first data node

    LOCK ROWS FROM TABLE junk2 IN ACCESS EXCLUSIVE MODE
    WHERE ctid in (.....) containing ctids of second data node


4. Now consider the example of a cursor query

   BEGIN;
     declare c1 cursor for select * from abc for update;
     fetch 1 from c1;
     -- do some stuff here
     fetch 1 from c1;
     -- do some stuff here
   END;

   In this case when the first fetch finishes its task in PortalRunSelect
   We will then issue the LOCK ROWS statement to the appropriate datanode
   with appropriate ctid. In case of a 'fetch all' the LOCK ROWS would
contain
   multiple ctids and may span over multiple datanodes too.

   It is also possible that while PortalRunSelect accumulates rows in
tuplestore
   we create LOCK ROWS statements in parallel and when PortalRunSelect
finishes
   we run those LOCK ROWS queries.

5. There will be no mechanism to unlock the rows, rows will be unlocked
implicitly when
   the transaction ends.

--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

On Mon, Jul 30, 2012 at 11:01 PM, Joshua D. Drake <jd...@co...>wrote:

>
> Hello,
>
> I wonder if it makes sense to only allow long options after a short
> option. For example:
>
> -XC --coordinator
>
> So the only way to use the XC options is to pass -XC and then a long
> option. That should keep us completely clear of PostgreSQL.
>
> Sincerely,
>
>
+0.5


> Joshua D. Drake
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

Hello,

I wonder if it makes sense to only allow long options after a short 
option. For example:

-XC --coordinator

So the only way to use the XC options is to pass -XC and then a long 
option. That should keep us completely clear of PostgreSQL.

Sincerely,

Joshua D. Drake

On Mon, Jul 30, 2012 at 7:17 PM, Ashutosh Bapat <
ash...@en...> wrote:

> We need to look at this in broader perspective. Using long options would
> fix the -C problem for now.

This option has been chosen long ago... This reminds me of the problem with
node list...
 I think that long options are enough safe.
Like for initdb with --nodename, I can pretty assure that options like
--coordinator and --datanode will never be used in Postgres.
So we should try to maximize the use of long options with keywords only
associated to XC.

But all the new options we would add will have this problem, at some or
> other point. Should we provide a solution which will take care of such
> problems for once and all? Changing user interfaces from release to release
> would alienate users from us.
>
Yes, and we also cannot forget that XC is directly a child of Postgres, so
in case of an option conflict, well postgres has to win and will win.
What we can do is choosing our new options enough wisely (launch options,
guc params, etc) to avoid any conflicts in future merges.
The past choice of -C and -X was indeed really bad as 9.2 merge is showing.


Hence, long options with really XC-specific keywords is a good deal. I am
open to other suggestions though.


> On Mon, Jul 30, 2012 at 7:01 AM, Michael Paquier <
> mic...@gm...> wrote:
>
>> Hi all,
>>
>> XC is now using -C to start up a node as a Coordinator, and -X to start
>> up a node as a Datanode.
>> In PostgreSQL 9.2, -C is now used for configuration parameters, so we
>> need to change the way we start nodes.
>>
>> In order to definitely take care of this problem that may happen in the
>> next release, I think we should change the start-up option
>> of XC nodes to long options, with --coordinator and --datanode. By using
>> that, we are sure that out options won't interact with postgres ones.
>>
>> Comments?
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Enterprise Postgres Company
>
>


-- 
Michael Paquier
http://michael.otacoo.com

We need to look at this in broader perspective. Using long options would
fix the -C problem for now. But all the new options we would add will have
this problem, at some or other point. Should we provide a solution which
will take care of such problems for once and all? Changing user interfaces
from release to release would alienate users from us.

On Mon, Jul 30, 2012 at 7:01 AM, Michael Paquier
<mic...@gm...>wrote:

> Hi all,
>
> XC is now using -C to start up a node as a Coordinator, and -X to start up
> a node as a Datanode.
> In PostgreSQL 9.2, -C is now used for configuration parameters, so we need
> to change the way we start nodes.
>
> In order to definitely take care of this problem that may happen in the
> next release, I think we should change the start-up option
> of XC nodes to long options, with --coordinator and --datanode. By using
> that, we are sure that out options won't interact with postgres ones.
>
> Comments?
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Enterprise Postgres Company

On 07/29/2012 06:31 PM, Michael Paquier wrote:
> Hi all,
>
> XC is now using -C to start up a node as a Coordinator, and -X to start
> up a node as a Datanode.
> In PostgreSQL 9.2, -C is now used for configuration parameters, so we
> need to change the way we start nodes.
>
> In order to definitely take care of this problem that may happen in the
> next release, I think we should change the start-up option
> of XC nodes to long options, with --coordinator and --datanode. By using
> that, we are sure that out options won't interact with postgres ones.

Only long options works for me. I prefer verbose options anyway. It 
allows us to understand exactly what is going on.

Sincerely,

Joshua D. Drake



-- 
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On Mon, Jul 30, 2012 at 10:48 AM, Koichi Suzuki <ko...@in...>wrote:

> On Mon, 30 Jul 2012 10:31:38 +0900
> Michael Paquier <mic...@gm...> wrote:
>
> > Hi all,
> >
> > XC is now using -C to start up a node as a Coordinator, and -X to start
> up
> > a node as a Datanode.
> > In PostgreSQL 9.2, -C is now used for configuration parameters, so we
> need
> > to change the way we start nodes.
> >
> > In order to definitely take care of this problem that may happen in the
> > next release, I think we should change the start-up option
> > of XC nodes to long options, with --coordinator and --datanode. By using
> > that, we are sure that out options won't interact with postgres ones.
> >
> > Comments?
>
> So we're going to have only long options for this?   I've no objection to
> it anyway.
>
Yes, looks like a better long-term option than anything else.
However I am open to other ideas.

> Also, we're maintaining -Z option, right?
>
There is no problem on this side, so we can keep it as is.
-- 
Michael Paquier
http://michael.otacoo.com

On Mon, 30 Jul 2012 10:31:38 +0900
Michael Paquier <mic...@gm...> wrote:

> Hi all,
> 
> XC is now using -C to start up a node as a Coordinator, and -X to start up
> a node as a Datanode.
> In PostgreSQL 9.2, -C is now used for configuration parameters, so we need
> to change the way we start nodes.
> 
> In order to definitely take care of this problem that may happen in the
> next release, I think we should change the start-up option
> of XC nodes to long options, with --coordinator and --datanode. By using
> that, we are sure that out options won't interact with postgres ones.
> 
> Comments?

So we're going to have only long options for this?   I've no objection to it anyway.
Also, we're maintaining -Z option, right?

---
Koichi

> -- 
> Michael Paquier
> http://michael.otacoo.com

Hi all,

XC is now using -C to start up a node as a Coordinator, and -X to start up
a node as a Datanode.
In PostgreSQL 9.2, -C is now used for configuration parameters, so we need
to change the way we start nodes.

In order to definitely take care of this problem that may happen in the
next release, I think we should change the start-up option
of XC nodes to long options, with --coordinator and --datanode. By using
that, we are sure that out options won't interact with postgres ones.

Comments?
-- 
Michael Paquier
http://michael.otacoo.com

I've heard another group is working together with Linux HA Japan to
provide XC RA's.

Sakata-san, could you provide related info if available?

Regards;
----------
Koichi Suzuki


2012/7/24 Nikhil Sontakke <ni...@st...>:
> Hi,
>
> So what's the latest status of these HA activities for PGXC?
>
> Like the coordinator/datanode agents being discussed here, do we have
> agents for GTM for example? Also is this happening in some open source
> group where we can participate and see the latest and greatest source
> changes?
>
> Regards,
> Nikhils
>
> On Mon, Jul 23, 2012 at 12:06 AM, Koichi Suzuki
> <ko...@in...> wrote:
>> Thanks.   I found the latter works for the master.
>> ---
>> Koichi
>>
>> On Mon, 23 Jul 2012 12:53:50 +0900
>> Michael Paquier <mic...@gm...> wrote:
>>
>>> > 2. select application_name,upper(state),upper(sync_state) from
>>> > pg_stat_replication;
>>> >
>>> > This does not work.  pg_stat_replication has zero row.
>>> >
>>> Because this is datanode slave, we don't have GTM connection or
>>> > snapshot.  This might be a cause of the problem 2, or we might have
>>> > blocked to update pg_stat_replication system catalog.
>>> >
>>> pg_stat_replication has by definition one row per wal sender process:
>>> http://www.postgresql.org/docs/9.1/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE
>>> AFAIK, slave has no wal sender processes running, so it is normal to see
>>> nothing on the slave side, all the data is on the master side. The data is
>>> only available on master, Am I missing smth?
>>> You might be able to see rows in pg_stat_replication on a slave if you use
>>> cascading replication however, but this feature is only available in 9.2.
>>> Also, in the case of XC, you can use EXECUTE DIRECT to get the information
>>> of pg_stat_replication from remote nodes, Coordinators and Datanodes
>>> included.
>>> --
>>> Michael Paquier
>>> http://michael.otacoo.com
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Hi Micheal,
Are we not suppose to just merge the changes from the 92 stable branch?

Thanks,
Ahsan

On Thu, Jul 26, 2012 at 10:10 AM, Michael Paquier <mic...@gm...
> wrote:

> Hi all,
>
> I am going to work on merging Postgres 9.2 code with XC.
> The goal is to plug in XC code up to the intersection of 9.2 stable branch
> and postgres master branch. By respecting this, we will then be able to
> plug in 9.2 stable branch for our future stable releases and postgresql
> master branch in the future.
>
> I am just beginning to have a look at that, so I still cannot really
> estimate the effort necessary to do that yet. However here is the commit I
> will merge up to:
> commit 80edfd76591fdb9beec061de3c05ef4e9d96ce56
> Author: Tom Lane <tg...@ss...>
> Date:   Wed Jun 13 19:43:35 2012 -0400
>
>     Revisit error message details for JSON input parsing.
>
>     Instead of identifying error locations only by line number (which could
>     be entirely unhelpful with long input lines), provide a fragment of the
>     input text too, placing this info in a new CONTEXT entry.  Make the
>     error detail messages conform more closely to style guidelines, fix
>     failure to expose some of them for translation, ensure compiler can
>     check formats against supplied parameters.
> This commit can be found with this command: git merge-base postgres/master
> postgres/REL9_2_STABLE.
>
> Once I got something compilable, I will commit a version of the merge in a
> new branch of my github repository of XC =>
> https://github.com/michaelpq/pgxc.
> And once I got some stabilization work done, I will simply merge this
> branch with the existing XC master and commit everything into XC master.
> I might begin the merge from commit
> d03ea805cef9375bee9b751e65d698c07c138bf5 on XC master branch.
> There will be for sure some other commits while I do the merge, but I will
> just include them once I got a faily stable version.
>
> Comments are welcome.
> --
> Michael Paquier
> http://michael.otacoo.com
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

Hi all,

I am going to work on merging Postgres 9.2 code with XC.
The goal is to plug in XC code up to the intersection of 9.2 stable branch
and postgres master branch. By respecting this, we will then be able to
plug in 9.2 stable branch for our future stable releases and postgresql
master branch in the future.

I am just beginning to have a look at that, so I still cannot really
estimate the effort necessary to do that yet. However here is the commit I
will merge up to:
commit 80edfd76591fdb9beec061de3c05ef4e9d96ce56
Author: Tom Lane <tg...@ss...>
Date:   Wed Jun 13 19:43:35 2012 -0400

    Revisit error message details for JSON input parsing.

    Instead of identifying error locations only by line number (which could
    be entirely unhelpful with long input lines), provide a fragment of the
    input text too, placing this info in a new CONTEXT entry.  Make the
    error detail messages conform more closely to style guidelines, fix
    failure to expose some of them for translation, ensure compiler can
    check formats against supplied parameters.
This commit can be found with this command: git merge-base postgres/master
postgres/REL9_2_STABLE.

Once I got something compilable, I will commit a version of the merge in a
new branch of my github repository of XC =>
https://github.com/michaelpq/pgxc.
And once I got some stabilization work done, I will simply merge this
branch with the existing XC master and commit everything into XC master.
I might begin the merge from commit
d03ea805cef9375bee9b751e65d698c07c138bf5 on XC master branch.
There will be for sure some other commits while I do the merge, but I will
just include them once I got a faily stable version.

Comments are welcome.
-- 
Michael Paquier
http://michael.otacoo.com

Hi,

So what's the latest status of these HA activities for PGXC?

Like the coordinator/datanode agents being discussed here, do we have
agents for GTM for example? Also is this happening in some open source
group where we can participate and see the latest and greatest source
changes?

Regards,
Nikhils

On Mon, Jul 23, 2012 at 12:06 AM, Koichi Suzuki
<ko...@in...> wrote:
> Thanks.   I found the latter works for the master.
> ---
> Koichi
>
> On Mon, 23 Jul 2012 12:53:50 +0900
> Michael Paquier <mic...@gm...> wrote:
>
>> > 2. select application_name,upper(state),upper(sync_state) from
>> > pg_stat_replication;
>> >
>> > This does not work.  pg_stat_replication has zero row.
>> >
>> Because this is datanode slave, we don't have GTM connection or
>> > snapshot.  This might be a cause of the problem 2, or we might have
>> > blocked to update pg_stat_replication system catalog.
>> >
>> pg_stat_replication has by definition one row per wal sender process:
>> http://www.postgresql.org/docs/9.1/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE
>> AFAIK, slave has no wal sender processes running, so it is normal to see
>> nothing on the slave side, all the data is on the master side. The data is
>> only available on master, Am I missing smth?
>> You might be able to see rows in pg_stat_replication on a slave if you use
>> cascading replication however, but this feature is only available in 9.2.
>> Also, in the case of XC, you can use EXECUTE DIRECT to get the information
>> of pg_stat_replication from remote nodes, Coordinators and Datanodes
>> included.
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers



-- 
StormDB - http://www.stormdb.com
The Database Cloud

On Mon, Jul 23, 2012 at 2:17 PM, Abbas Butt <abb...@en...>wrote:

> Agreed. Please go ahead and check in this fix.

Thanks, I fixed that and also another issue I found on the way.
-- 
Michael Paquier
http://michael.otacoo.com

Agreed. Please go ahead and check in this fix.

On Mon, Jul 23, 2012 at 10:14 AM, Michael Paquier <mic...@gm...
> wrote:

> Yeah OK. Btw, it should be better to set up the entire condition inside
> this loop, checking for parent/child INSERT SELECT condition on remote
> nodes doesn't make sense either.
> if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
> {
>     target_rte = rt_fetch(qry->resultRelation, pstate->p_rtable);
>     if (is_relation_child(target_rte, selectQuery->rtable))
>     {
>         qry->is_ins_child_sel_parent = true;
>         SetSendCommandId(true);
>
>     }
> }
>
> On Mon, Jul 23, 2012 at 2:05 PM, Abbas Butt <abb...@en...>wrote:
>
>> Thanks Michael, I just reviewed the patch and found one small thing.
>>
>> In the function transformInsertStmt where we set
>> SetSendCommandId(true);
>> I think it would be better to put this statement under an
>> if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>
>> Other than that I am OK with the changes.
>>
>> Regards
>>
>>
>> On Mon, Jul 23, 2012 at 7:08 AM, Michael Paquier <
>> mic...@gm...> wrote:
>>
>>> Hi,
>>>
>>> I spent some time this morning rewriting your patch.
>>> I reduced the number of functions and parameters to a minimum number and
>>> eliminated the useless message 'Y'.
>>> 'Y' definitely doesn't make sense as it has absolutely the same role as
>>> 'M'. In the former patch, the notion of notification is simply to send a
>>> message from Coordinator to Datanode, but a Datanode can also send a GXID,
>>> snapshot and timestamp using the same message types as the one used when a
>>> Coordinator sends data to a remote node. So we shouldn't lose this
>>> portability of the new messages. The only thing that changes for
>>> Coordinator->remoteNode and remoteCode->Coordinator communication protocol
>>> is the place where message is read. in the first case, message is received
>>> by remote node in postgres.c stuff. In the second case, message is received
>>> inside execRemote.c, so this distinction makes perfectly safe the fact of
>>> using the same message 'M', making 'Y' unnecessary.
>>> Also, it is really important to limit the number of used message types
>>> to reduce the footprint of XC code on PostgreSQL.
>>>
>>> There are no problems with regressions, and performance is not impacted.
>>> I also corrected some comment format and added some other comments at
>>> various places.
>>> In consequence, and to simplify the work of everybody, I am going to
>>> commit this largely simplified version of the patch.
>>>
>>> Thanks,
>>>
>>>
>>> On Sat, Jul 21, 2012 at 6:29 PM, Abbas Butt <abb...@en...
>>> > wrote:
>>>
>>>>
>>>>
>>>> On Fri, Jul 20, 2012 at 11:04 AM, Michael Paquier <
>>>> mic...@gm...> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry for being a little bit picky here:
>>>>> 1) In GetCurrentSubTransactionId of xact.c, you can replace:
>>>>> if (IS_PGXC_DATANODE && isCommandIdReceived)
>>>>> {
>>>>>    foo
>>>>> }
>>>>> else
>>>>> {
>>>>>    if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>>>       re_foo
>>>>> }
>>>>> by:
>>>>> if (IS_PGXC_DATANODE && isCommandIdReceived)
>>>>> {
>>>>>    foo
>>>>> }
>>>>> else if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>>> {
>>>>>     re_foo
>>>>> }
>>>>>
>>>>
>>>> Done
>>>>
>>>>
>>>>> 2) I am not so much a fan of this 'Y' message as for now it just has a
>>>>> single usage, and M is doing exactly the same thing... But seen as a
>>>>> message that a remote node sends a notification to its backend, ok it makes
>>>>> some sense. In this case you should avoid to use a magic number like '1'.
>>>>> Please use a define like:
>>>>> #define DATANODE_NOTIFICATION_COMMAND_ID '1'
>>>>> A place like pgxc.h would make sense to define that.
>>>>> Then use this variable at the places where you need it
>>>>> (HandleDatanodeNotification of execRemote.c and ReportCommandIdChange of
>>>>> xact.c). It will bring more visibility to your code.
>>>>>
>>>>
>>>> Done
>>>>
>>>>
>>>>> 3) SetCmdIdSending and SetCmdIdReporting should be combined, they
>>>>> serve the same purpose. Their only difference is the node type where they
>>>>> are used. Let's make that generic. As a general name, SetCommandIdSending
>>>>> would be enough.
>>>>>
>>>>
>>>> I am not getting how to do that, consider the example of
>>>> StartTransaction function where we want to SetCmdIdReporting(false).
>>>> Suppose we combine both functions and have the new function like this
>>>>
>>>> void SetCmdIdSending(bool enb)
>>>> {
>>>>   if(IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>>     sendCommandId = enb;
>>>>   else if(IS_PGXC_DATANODE)
>>>>     sendCommandId = enb;
>>>> }
>>>>
>>>> and that we call this function instead of SetCmdIdReporting. It will
>>>> work OK for data node but when the same code executes for coordinator we
>>>> will set sendCommandId to false which was not required. Perhaps I did not
>>>> get your point.
>>>>
>>>>
>>>>> 4) GetCmdIdOfDatanode and GetCmdIdOfCoordinator should also be
>>>>> combined. They do exactly the same thing, the only difference is that one
>>>>> is launched on Datanode and the other on Coordinator. A general name like
>>>>> GetCommandIdReceived would be nicer.
>>>>>
>>>>
>>>> See my comment for point (3)
>>>>
>>>> 5) In portalcmds.c, PerformCursorOpen. You need to replace
>>>>> if (IS_PGXC_COORDINATOR)
>>>>>                GetCurrentCommandId(true);
>>>>> by:
>>>>> if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>>>                GetCurrentCommandId(true);
>>>>>
>>>>
>>>> Done.
>>>>
>>>>
>>>>> 6) For the regressions, I am still getting the diffs. Based on your
>>>>> test, I get this ordering for PG and XC:
>>>>> pgxc# select * from test_text order by 1;
>>>>>                  a
>>>>> ------------------------------------
>>>>>  Bancroft                      Ave
>>>>>  Bancroft                      Ave
>>>>>  Birch                         St
>>>>>  Birch                         St
>>>>>  Blacow                        Road
>>>>>  Bridgepointe                  Dr
>>>>>  Broadmore                     Ave
>>>>>  Broadway
>>>>>  B                             St
>>>>> (9 rows)
>>>>> If we are getting the same diffs we should change the output, it looks
>>>>> to be a side-effect of your fix for INSERT SELECT on parent-child,
>>>>>
>>>>
>>>> Done
>>>>
>>>>
>>>>>
>>>>> On Fri, Jul 20, 2012 at 2:15 AM, Abbas Butt <
>>>>> abb...@en...> wrote:
>>>>>
>>>>>> Attached please find the updated patch.
>>>>>>
>>>>>> On Thu, Jul 19, 2012 at 11:01 AM, Abbas Butt <
>>>>>> abb...@en...> wrote:
>>>>>>
>>>>>>> Thanks Michael for the review, Thanks Ashutosh for the comments,
>>>>>>> Michael I will accommodate your valuable feedback and will send
>>>>>>> revised patch.
>>>>>>> Ashutosh I will send the summarized design in the email with the
>>>>>>> revised patch.
>>>>>>> BTW enb is short for enable!
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 19, 2012 at 10:37 AM, Michael Paquier <
>>>>>>> mic...@gm...> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Finally I took some time to look at this code, it is important
>>>>>>>> support.
>>>>>>>> 1) The code is not realigned with current master (xc_misc.sql) and
>>>>>>>> contains whitespaces (2 in xc_misc.out, perhaps other files as well). So I
>>>>>>>> realigned it myself to test the code.
>>>>>>>>
>>>>>>>
>>>>>> The attached patch is aligned with the current master and I have
>>>>>> removed white spaces.
>>>>>>
>>>>>>
>>>>>>>  2) In GetCurrentCommandId xact.c, you need to set
>>>>>>>> !IsConnFromCoord() with (IS_PGXC_COORDINATOR) to avoid remote coordinators
>>>>>>>> incrementing the command ID. Remote Coordinators do not receive any command
>>>>>>>> IDs from remote nodes. Also here using an if/else if structure won't hurt
>>>>>>>> and bring clarity in the operations.
>>>>>>>>
>>>>>>>
>>>>>> Added the check and used if/else
>>>>>>
>>>>>>
>>>>>>>  3) In BeginTransactionBlock of xact.c, you need to add
>>>>>>>> !IsConnFromCoord() as a secondary condition to avoid remote Coordinator
>>>>>>>> problems.
>>>>>>>>
>>>>>>>
>>>>>> Done
>>>>>>
>>>>>>
>>>>>>> 4) By looking at the patch It is possible to reduce the number of
>>>>>>>> variables used for the control of Command ID. In the current version of
>>>>>>>> your patch, you are using 5 static variables:
>>>>>>>> static CommandId    cmdIdFromCoord;        /* Datanode will use
>>>>>>>> command id received from coordinator */
>>>>>>>> static bool        cmdIdRcvdAtDatanode;    /* Has the coordinator
>>>>>>>> sent command id to the data node? */
>>>>>>>> static bool        enbCmdIdChgReporting;    /* Should datanode
>>>>>>>> report command id changes to coordinator? */
>>>>>>>> static bool        enbCoordCmdIdSending;    /* Should coordinator
>>>>>>>> send command id to the data node? */
>>>>>>>> static CommandId    cmdIdFromDatanode;    /* Coordinator will
>>>>>>>> receive updated command id from datanode */
>>>>>>>> However, cmdIdRcvdAtDatanode, cmdIdFromCoord and
>>>>>>>> enbCmdIdChgReporting are used only at Datanode.
>>>>>>>> cmdIdFromDatanode and enbCoordCmdIdSending are only used at
>>>>>>>> Coordinator level.
>>>>>>>> You should combine enbCoordCmdIdSending and enbCmdIdChgReporting
>>>>>>>> into a single variable with a generic name, like let's say sendCommandId or
>>>>>>>> reportCommandId. Specifying in comments the role of this unique variable
>>>>>>>> for local Coordinator and remote nodes will help. Those 2 variables play
>>>>>>>> the same role which is to control if a
>>>>>>>> In the case of Coordinator, this variable is flagged to true when
>>>>>>>> transforming INSERT statements (transformInsertStmt) and when a transaction
>>>>>>>> block is begun (BeginTransactionBlock).
>>>>>>>> In the case of a Datanode, this variable is flagged to true only
>>>>>>>> when a command ID has been received from Coordinator.
>>>>>>>>
>>>>>>>>
>>>>>> Done, I used functions to retain code readability and inside the
>>>>>> functions I used the same variables as u suggested.
>>>>>>
>>>>>>
>>>>>>> Then, you should combine cmdIdFromCoord and cmdIdFromDatanode into a
>>>>>>>> single variable, like receivedCommandId. In this case, being at Coordinator
>>>>>>>> or at Datanode means that this command ID has been received from a remode
>>>>>>>> node. If you have other ideas for names please feel free!
>>>>>>>>
>>>>>>>
>>>>>> Done, again I used functions to retain code readability. I however
>>>>>> had to add checks inside the functions to make sure that the functions
>>>>>> intended for data node work only for data nodes and the ones intended for
>>>>>> coordinators work only for coordinators.
>>>>>>
>>>>>>
>>>>>>> Also changing the name cmdIdRcvdAtDatanode to isCommandIdReceived
>>>>>>>> might help...
>>>>>>>>
>>>>>>>
>>>>>> Done
>>>>>>
>>>>>>
>>>>>>> Just a question, what means the preffix enb? (just by curiosity) :)
>>>>>>>>
>>>>>>>> 5) In xact.c, at the end of CommitTransaction, PrepareTransaction,
>>>>>>>> CleanupTransaction, you will need to change the cleanup of the variables if
>>>>>>>> you take into account comment 4).
>>>>>>>>
>>>>>>>
>>>>>> Done
>>>>>>
>>>>>>
>>>>>>> 6) In BeginTransactionBlock, PrepareTransactionBlock,
>>>>>>>> EndTransactionBlock and UserAbortTransactionBlock of xact.c, you need to
>>>>>>>> set an additional !IsConnFromCoord() at this place, this will avoid remote
>>>>>>>> Coordinators trying to send CommandIds to inexistent nodes.
>>>>>>>> if(IS_PGXC_COORDINATOR)
>>>>>>>> +        SetCmdIdSending(false);
>>>>>>>>
>>>>>>>
>>>>>> Done
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> 7) You should combine the messages 'Y' and 'M'. The content and the
>>>>>>>> goals are the same. You can also eliminate the 2nd byte of 'Y' without
>>>>>>>> consequences I think.
>>>>>>>>
>>>>>>>
>>>>>> I did not combine these two. The intention of having them separate is
>>>>>> because 'Y' should not be tied to 'M', In future we might need data nodes
>>>>>> to notify coordinators of some thing else. And for the same reason I have
>>>>>> added a second byte in 'Y' so that we can have the same command notify
>>>>>> coordinators of some thing else.
>>>>>>
>>>>>>
>>>>>>> 8) Why didn't you put the reception of message 'M' in PostgresMain
>>>>>>>> of postgres.c at the same place as the reception of GXID, snapshot,
>>>>>>>> timestamp, etc.
>>>>>>>> Putting the reception of message at the end of SocketBackend makes
>>>>>>>> the code kind of inconsistent. If you keep it inside SocketBackend, you
>>>>>>>> should at least manage the message inside the switch, however I do not see
>>>>>>>> any particular reason why you receive this message in SocketBackend and not
>>>>>>>> in PostgresMain. So anything I should know?
>>>>>>>> You should also remove the flag IS_PGXC_DATANODE when receiving the
>>>>>>>> command ID. Remote Coordinator might need to be able to receive command IDs
>>>>>>>> as well, even if there is no direct usage now, but for stuff like triggers??
>>>>>>>>
>>>>>>>
>>>>>> Done, I added handling of 'M' in PostgresMain.
>>>>>>
>>>>>>
>>>>>>>  9) I found some diffs with the test select_views. It might be worth
>>>>>>>> to look at what is happening. It is perhaps an environment-related issue.
>>>>>>>> Btw, my regression diffs are attached.
>>>>>>>>
>>>>>>>
>>>>>> I was also getting these diffs but it was difficult for me to think
>>>>>> that the changes I did could cause a change in oder of rows. Here is what I
>>>>>> did to dig the issue deeper.
>>>>>>
>>>>>> create table test_text(a text);
>>>>>>
>>>>>> insert into test_text values('Bancroft                      Ave');
>>>>>> insert into test_text values('Bancroft                      Ave');
>>>>>> insert into test_text values('Birch                         St');
>>>>>> insert into test_text values('Birch                         St');
>>>>>> insert into test_text values('Blacow                        Road');
>>>>>> insert into test_text values('Bridgepointe                  Dr');
>>>>>> insert into test_text values('Broadmore                     Ave');
>>>>>> insert into test_text values('Broadway                          ');
>>>>>> insert into test_text values('B                             St');
>>>>>>
>>>>>> select * from test_text order by 1
>>>>>> produces same results with plain PG and XC
>>>>>>
>>>>>> Also I changed the query in plain PG select_views test case to
>>>>>> SELECT * FROM street ORDER BY name,cname,thepath::text;
>>>>>> and obtained same results with plain PG and XC.
>>>>>>
>>>>>> I am not sure why the test case was passing earlier with the current
>>>>>> possible expected outputs.
>>>>>>
>>>>>> Summarized design notes are attached with the mail for reference.
>>>>>>
>>>>>>  10) The modifications you are bringing to combocid_1.out are
>>>>>>>> consistent with postgres, cool!
>>>>>>>>
>>>>>>>> No performance impact with this patch! Tested and approved. This is
>>>>>>>> good.
>>>>>>>> No regression impact with this patch.
>>>>>>>>
>>>>>>>> So, to conclude, I think it is a good piece of work, really cool
>>>>>>>> stuff that I believe will be helpful for triggers and even other things.
>>>>>>>> It just needs to be polished and simplified a bit. However basics
>>>>>>>> are here, and performance is not impacted. Really it was worth spending
>>>>>>>> time on that. The additional test cases also look to be sufficient, but I
>>>>>>>> might be missing smth so feel free to add new tests if necessary.
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> --
>>>>>>>> Michael Paquier
>>>>>>>> http://michael.otacoo.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --
>>>>>>> Abbas
>>>>>>> Architect
>>>>>>> EnterpriseDB Corporation
>>>>>>> The Enterprise PostgreSQL Company
>>>>>>>
>>>>>>> Phone: 92-334-5100153
>>>>>>>
>>>>>>> Website: www.enterprisedb.com
>>>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>>>>>
>>>>>>> This e-mail message (and any attachment) is intended for the use of
>>>>>>> the individual or entity to whom it is addressed. This message
>>>>>>> contains information from EnterpriseDB Corporation that may be
>>>>>>> privileged, confidential, or exempt from disclosure under applicable
>>>>>>> law. If you are not the intended recipient or authorized to receive
>>>>>>> this for the intended recipient, any use, dissemination,
>>>>>>> distribution,
>>>>>>> retention, archiving, or copying of this communication is strictly
>>>>>>> prohibited. If you have received this e-mail in error, please notify
>>>>>>> the sender immediately by reply e-mail and delete this message.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Abbas
>>>>>> Architect
>>>>>> EnterpriseDB Corporation
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>> Phone: 92-334-5100153
>>>>>>
>>>>>> Website: www.enterprisedb.com
>>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>>>>
>>>>>> This e-mail message (and any attachment) is intended for the use of
>>>>>> the individual or entity to whom it is addressed. This message
>>>>>> contains information from EnterpriseDB Corporation that may be
>>>>>> privileged, confidential, or exempt from disclosure under applicable
>>>>>> law. If you are not the intended recipient or authorized to receive
>>>>>> this for the intended recipient, any use, dissemination, distribution,
>>>>>> retention, archiving, or copying of this communication is strictly
>>>>>> prohibited. If you have received this e-mail in error, please notify
>>>>>> the sender immediately by reply e-mail and delete this message.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Michael Paquier
>>>>> http://michael.otacoo.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Abbas
>>>> Architect
>>>> EnterpriseDB Corporation
>>>> The Enterprise PostgreSQL Company
>>>>
>>>> Phone: 92-334-5100153
>>>>
>>>> Website: www.enterprisedb.com
>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>>
>>>> This e-mail message (and any attachment) is intended for the use of
>>>> the individual or entity to whom it is addressed. This message
>>>> contains information from EnterpriseDB Corporation that may be
>>>> privileged, confidential, or exempt from disclosure under applicable
>>>> law. If you are not the intended recipient or authorized to receive
>>>> this for the intended recipient, any use, dissemination, distribution,
>>>> retention, archiving, or copying of this communication is strictly
>>>> prohibited. If you have received this e-mail in error, please notify
>>>> the sender immediately by reply e-mail and delete this message.
>>>>
>>>
>>>
>>>
>>> --
>>> Michael Paquier
>>> http://michael.otacoo.com
>>>
>>
>>
>>
>> --
>> --
>> Abbas
>> Architect
>> EnterpriseDB Corporation
>> The Enterprise PostgreSQL Company
>>
>> Phone: 92-334-5100153
>>
>> Website: www.enterprisedb.com
>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>
>> This e-mail message (and any attachment) is intended for the use of
>> the individual or entity to whom it is addressed. This message
>> contains information from EnterpriseDB Corporation that may be
>> privileged, confidential, or exempt from disclosure under applicable
>> law. If you are not the intended recipient or authorized to receive
>> this for the intended recipient, any use, dissemination, distribution,
>> retention, archiving, or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify
>> the sender immediately by reply e-mail and delete this message.
>>
>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

Yeah OK. Btw, it should be better to set up the entire condition inside
this loop, checking for parent/child INSERT SELECT condition on remote
nodes doesn't make sense either.
if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
{
    target_rte = rt_fetch(qry->resultRelation, pstate->p_rtable);
    if (is_relation_child(target_rte, selectQuery->rtable))
    {
        qry->is_ins_child_sel_parent = true;
        SetSendCommandId(true);
    }
}

On Mon, Jul 23, 2012 at 2:05 PM, Abbas Butt <abb...@en...>wrote:

> Thanks Michael, I just reviewed the patch and found one small thing.
>
> In the function transformInsertStmt where we set
> SetSendCommandId(true);
> I think it would be better to put this statement under an
> if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>
> Other than that I am OK with the changes.
>
> Regards
>
>
> On Mon, Jul 23, 2012 at 7:08 AM, Michael Paquier <
> mic...@gm...> wrote:
>
>> Hi,
>>
>> I spent some time this morning rewriting your patch.
>> I reduced the number of functions and parameters to a minimum number and
>> eliminated the useless message 'Y'.
>> 'Y' definitely doesn't make sense as it has absolutely the same role as
>> 'M'. In the former patch, the notion of notification is simply to send a
>> message from Coordinator to Datanode, but a Datanode can also send a GXID,
>> snapshot and timestamp using the same message types as the one used when a
>> Coordinator sends data to a remote node. So we shouldn't lose this
>> portability of the new messages. The only thing that changes for
>> Coordinator->remoteNode and remoteCode->Coordinator communication protocol
>> is the place where message is read. in the first case, message is received
>> by remote node in postgres.c stuff. In the second case, message is received
>> inside execRemote.c, so this distinction makes perfectly safe the fact of
>> using the same message 'M', making 'Y' unnecessary.
>> Also, it is really important to limit the number of used message types to
>> reduce the footprint of XC code on PostgreSQL.
>>
>> There are no problems with regressions, and performance is not impacted.
>> I also corrected some comment format and added some other comments at
>> various places.
>> In consequence, and to simplify the work of everybody, I am going to
>> commit this largely simplified version of the patch.
>>
>> Thanks,
>>
>>
>> On Sat, Jul 21, 2012 at 6:29 PM, Abbas Butt <abb...@en...>wrote:
>>
>>>
>>>
>>> On Fri, Jul 20, 2012 at 11:04 AM, Michael Paquier <
>>> mic...@gm...> wrote:
>>>
>>>> Hi,
>>>>
>>>> Sorry for being a little bit picky here:
>>>> 1) In GetCurrentSubTransactionId of xact.c, you can replace:
>>>> if (IS_PGXC_DATANODE && isCommandIdReceived)
>>>> {
>>>>    foo
>>>> }
>>>> else
>>>> {
>>>>    if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>>       re_foo
>>>> }
>>>> by:
>>>> if (IS_PGXC_DATANODE && isCommandIdReceived)
>>>> {
>>>>    foo
>>>> }
>>>> else if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>> {
>>>>     re_foo
>>>> }
>>>>
>>>
>>> Done
>>>
>>>
>>>> 2) I am not so much a fan of this 'Y' message as for now it just has a
>>>> single usage, and M is doing exactly the same thing... But seen as a
>>>> message that a remote node sends a notification to its backend, ok it makes
>>>> some sense. In this case you should avoid to use a magic number like '1'.
>>>> Please use a define like:
>>>> #define DATANODE_NOTIFICATION_COMMAND_ID '1'
>>>> A place like pgxc.h would make sense to define that.
>>>> Then use this variable at the places where you need it
>>>> (HandleDatanodeNotification of execRemote.c and ReportCommandIdChange of
>>>> xact.c). It will bring more visibility to your code.
>>>>
>>>
>>> Done
>>>
>>>
>>>> 3) SetCmdIdSending and SetCmdIdReporting should be combined, they serve
>>>> the same purpose. Their only difference is the node type where they are
>>>> used. Let's make that generic. As a general name, SetCommandIdSending would
>>>> be enough.
>>>>
>>>
>>> I am not getting how to do that, consider the example of
>>> StartTransaction function where we want to SetCmdIdReporting(false).
>>> Suppose we combine both functions and have the new function like this
>>>
>>> void SetCmdIdSending(bool enb)
>>> {
>>>   if(IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>     sendCommandId = enb;
>>>   else if(IS_PGXC_DATANODE)
>>>     sendCommandId = enb;
>>> }
>>>
>>> and that we call this function instead of SetCmdIdReporting. It will
>>> work OK for data node but when the same code executes for coordinator we
>>> will set sendCommandId to false which was not required. Perhaps I did not
>>> get your point.
>>>
>>>
>>>> 4) GetCmdIdOfDatanode and GetCmdIdOfCoordinator should also be
>>>> combined. They do exactly the same thing, the only difference is that one
>>>> is launched on Datanode and the other on Coordinator. A general name like
>>>> GetCommandIdReceived would be nicer.
>>>>
>>>
>>> See my comment for point (3)
>>>
>>> 5) In portalcmds.c, PerformCursorOpen. You need to replace
>>>> if (IS_PGXC_COORDINATOR)
>>>>                GetCurrentCommandId(true);
>>>> by:
>>>> if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>>                GetCurrentCommandId(true);
>>>>
>>>
>>> Done.
>>>
>>>
>>>> 6) For the regressions, I am still getting the diffs. Based on your
>>>> test, I get this ordering for PG and XC:
>>>> pgxc# select * from test_text order by 1;
>>>>                  a
>>>> ------------------------------------
>>>>  Bancroft                      Ave
>>>>  Bancroft                      Ave
>>>>  Birch                         St
>>>>  Birch                         St
>>>>  Blacow                        Road
>>>>  Bridgepointe                  Dr
>>>>  Broadmore                     Ave
>>>>  Broadway
>>>>  B                             St
>>>> (9 rows)
>>>> If we are getting the same diffs we should change the output, it looks
>>>> to be a side-effect of your fix for INSERT SELECT on parent-child,
>>>>
>>>
>>> Done
>>>
>>>
>>>>
>>>> On Fri, Jul 20, 2012 at 2:15 AM, Abbas Butt <
>>>> abb...@en...> wrote:
>>>>
>>>>> Attached please find the updated patch.
>>>>>
>>>>> On Thu, Jul 19, 2012 at 11:01 AM, Abbas Butt <
>>>>> abb...@en...> wrote:
>>>>>
>>>>>> Thanks Michael for the review, Thanks Ashutosh for the comments,
>>>>>> Michael I will accommodate your valuable feedback and will send
>>>>>> revised patch.
>>>>>> Ashutosh I will send the summarized design in the email with the
>>>>>> revised patch.
>>>>>> BTW enb is short for enable!
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 19, 2012 at 10:37 AM, Michael Paquier <
>>>>>> mic...@gm...> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Finally I took some time to look at this code, it is important
>>>>>>> support.
>>>>>>> 1) The code is not realigned with current master (xc_misc.sql) and
>>>>>>> contains whitespaces (2 in xc_misc.out, perhaps other files as well). So I
>>>>>>> realigned it myself to test the code.
>>>>>>>
>>>>>>
>>>>> The attached patch is aligned with the current master and I have
>>>>> removed white spaces.
>>>>>
>>>>>
>>>>>>  2) In GetCurrentCommandId xact.c, you need to set !IsConnFromCoord()
>>>>>>> with (IS_PGXC_COORDINATOR) to avoid remote coordinators incrementing the
>>>>>>> command ID. Remote Coordinators do not receive any command IDs from remote
>>>>>>> nodes. Also here using an if/else if structure won't hurt and bring clarity
>>>>>>> in the operations.
>>>>>>>
>>>>>>
>>>>> Added the check and used if/else
>>>>>
>>>>>
>>>>>>  3) In BeginTransactionBlock of xact.c, you need to add
>>>>>>> !IsConnFromCoord() as a secondary condition to avoid remote Coordinator
>>>>>>> problems.
>>>>>>>
>>>>>>
>>>>> Done
>>>>>
>>>>>
>>>>>> 4) By looking at the patch It is possible to reduce the number of
>>>>>>> variables used for the control of Command ID. In the current version of
>>>>>>> your patch, you are using 5 static variables:
>>>>>>> static CommandId    cmdIdFromCoord;        /* Datanode will use
>>>>>>> command id received from coordinator */
>>>>>>> static bool        cmdIdRcvdAtDatanode;    /* Has the coordinator
>>>>>>> sent command id to the data node? */
>>>>>>> static bool        enbCmdIdChgReporting;    /* Should datanode
>>>>>>> report command id changes to coordinator? */
>>>>>>> static bool        enbCoordCmdIdSending;    /* Should coordinator
>>>>>>> send command id to the data node? */
>>>>>>> static CommandId    cmdIdFromDatanode;    /* Coordinator will
>>>>>>> receive updated command id from datanode */
>>>>>>> However, cmdIdRcvdAtDatanode, cmdIdFromCoord and
>>>>>>> enbCmdIdChgReporting are used only at Datanode.
>>>>>>> cmdIdFromDatanode and enbCoordCmdIdSending are only used at
>>>>>>> Coordinator level.
>>>>>>> You should combine enbCoordCmdIdSending and enbCmdIdChgReporting
>>>>>>> into a single variable with a generic name, like let's say sendCommandId or
>>>>>>> reportCommandId. Specifying in comments the role of this unique variable
>>>>>>> for local Coordinator and remote nodes will help. Those 2 variables play
>>>>>>> the same role which is to control if a
>>>>>>> In the case of Coordinator, this variable is flagged to true when
>>>>>>> transforming INSERT statements (transformInsertStmt) and when a transaction
>>>>>>> block is begun (BeginTransactionBlock).
>>>>>>> In the case of a Datanode, this variable is flagged to true only
>>>>>>> when a command ID has been received from Coordinator.
>>>>>>>
>>>>>>>
>>>>> Done, I used functions to retain code readability and inside the
>>>>> functions I used the same variables as u suggested.
>>>>>
>>>>>
>>>>>> Then, you should combine cmdIdFromCoord and cmdIdFromDatanode into a
>>>>>>> single variable, like receivedCommandId. In this case, being at Coordinator
>>>>>>> or at Datanode means that this command ID has been received from a remode
>>>>>>> node. If you have other ideas for names please feel free!
>>>>>>>
>>>>>>
>>>>> Done, again I used functions to retain code readability. I however had
>>>>> to add checks inside the functions to make sure that the functions intended
>>>>> for data node work only for data nodes and the ones intended for
>>>>> coordinators work only for coordinators.
>>>>>
>>>>>
>>>>>> Also changing the name cmdIdRcvdAtDatanode to isCommandIdReceived
>>>>>>> might help...
>>>>>>>
>>>>>>
>>>>> Done
>>>>>
>>>>>
>>>>>> Just a question, what means the preffix enb? (just by curiosity) :)
>>>>>>>
>>>>>>> 5) In xact.c, at the end of CommitTransaction, PrepareTransaction,
>>>>>>> CleanupTransaction, you will need to change the cleanup of the variables if
>>>>>>> you take into account comment 4).
>>>>>>>
>>>>>>
>>>>> Done
>>>>>
>>>>>
>>>>>> 6) In BeginTransactionBlock, PrepareTransactionBlock,
>>>>>>> EndTransactionBlock and UserAbortTransactionBlock of xact.c, you need to
>>>>>>> set an additional !IsConnFromCoord() at this place, this will avoid remote
>>>>>>> Coordinators trying to send CommandIds to inexistent nodes.
>>>>>>> if(IS_PGXC_COORDINATOR)
>>>>>>> +        SetCmdIdSending(false);
>>>>>>>
>>>>>>
>>>>> Done
>>>>>
>>>>>
>>>>>>
>>>>>>> 7) You should combine the messages 'Y' and 'M'. The content and the
>>>>>>> goals are the same. You can also eliminate the 2nd byte of 'Y' without
>>>>>>> consequences I think.
>>>>>>>
>>>>>>
>>>>> I did not combine these two. The intention of having them separate is
>>>>> because 'Y' should not be tied to 'M', In future we might need data nodes
>>>>> to notify coordinators of some thing else. And for the same reason I have
>>>>> added a second byte in 'Y' so that we can have the same command notify
>>>>> coordinators of some thing else.
>>>>>
>>>>>
>>>>>> 8) Why didn't you put the reception of message 'M' in PostgresMain of
>>>>>>> postgres.c at the same place as the reception of GXID, snapshot, timestamp,
>>>>>>> etc.
>>>>>>> Putting the reception of message at the end of SocketBackend makes
>>>>>>> the code kind of inconsistent. If you keep it inside SocketBackend, you
>>>>>>> should at least manage the message inside the switch, however I do not see
>>>>>>> any particular reason why you receive this message in SocketBackend and not
>>>>>>> in PostgresMain. So anything I should know?
>>>>>>> You should also remove the flag IS_PGXC_DATANODE when receiving the
>>>>>>> command ID. Remote Coordinator might need to be able to receive command IDs
>>>>>>> as well, even if there is no direct usage now, but for stuff like triggers??
>>>>>>>
>>>>>>
>>>>> Done, I added handling of 'M' in PostgresMain.
>>>>>
>>>>>
>>>>>>  9) I found some diffs with the test select_views. It might be worth
>>>>>>> to look at what is happening. It is perhaps an environment-related issue.
>>>>>>> Btw, my regression diffs are attached.
>>>>>>>
>>>>>>
>>>>> I was also getting these diffs but it was difficult for me to think
>>>>> that the changes I did could cause a change in oder of rows. Here is what I
>>>>> did to dig the issue deeper.
>>>>>
>>>>> create table test_text(a text);
>>>>>
>>>>> insert into test_text values('Bancroft                      Ave');
>>>>> insert into test_text values('Bancroft                      Ave');
>>>>> insert into test_text values('Birch                         St');
>>>>> insert into test_text values('Birch                         St');
>>>>> insert into test_text values('Blacow                        Road');
>>>>> insert into test_text values('Bridgepointe                  Dr');
>>>>> insert into test_text values('Broadmore                     Ave');
>>>>> insert into test_text values('Broadway                          ');
>>>>> insert into test_text values('B                             St');
>>>>>
>>>>> select * from test_text order by 1
>>>>> produces same results with plain PG and XC
>>>>>
>>>>> Also I changed the query in plain PG select_views test case to
>>>>> SELECT * FROM street ORDER BY name,cname,thepath::text;
>>>>> and obtained same results with plain PG and XC.
>>>>>
>>>>> I am not sure why the test case was passing earlier with the current
>>>>> possible expected outputs.
>>>>>
>>>>> Summarized design notes are attached with the mail for reference.
>>>>>
>>>>>  10) The modifications you are bringing to combocid_1.out are
>>>>>>> consistent with postgres, cool!
>>>>>>>
>>>>>>> No performance impact with this patch! Tested and approved. This is
>>>>>>> good.
>>>>>>> No regression impact with this patch.
>>>>>>>
>>>>>>> So, to conclude, I think it is a good piece of work, really cool
>>>>>>> stuff that I believe will be helpful for triggers and even other things.
>>>>>>> It just needs to be polished and simplified a bit. However basics
>>>>>>> are here, and performance is not impacted. Really it was worth spending
>>>>>>> time on that. The additional test cases also look to be sufficient, but I
>>>>>>> might be missing smth so feel free to add new tests if necessary.
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --
>>>>>>> Michael Paquier
>>>>>>> http://michael.otacoo.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Abbas
>>>>>> Architect
>>>>>> EnterpriseDB Corporation
>>>>>> The Enterprise PostgreSQL Company
>>>>>>
>>>>>> Phone: 92-334-5100153
>>>>>>
>>>>>> Website: www.enterprisedb.com
>>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>>>>
>>>>>> This e-mail message (and any attachment) is intended for the use of
>>>>>> the individual or entity to whom it is addressed. This message
>>>>>> contains information from EnterpriseDB Corporation that may be
>>>>>> privileged, confidential, or exempt from disclosure under applicable
>>>>>> law. If you are not the intended recipient or authorized to receive
>>>>>> this for the intended recipient, any use, dissemination, distribution,
>>>>>> retention, archiving, or copying of this communication is strictly
>>>>>> prohibited. If you have received this e-mail in error, please notify
>>>>>> the sender immediately by reply e-mail and delete this message.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Abbas
>>>>> Architect
>>>>> EnterpriseDB Corporation
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>> Phone: 92-334-5100153
>>>>>
>>>>> Website: www.enterprisedb.com
>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>>>
>>>>> This e-mail message (and any attachment) is intended for the use of
>>>>> the individual or entity to whom it is addressed. This message
>>>>> contains information from EnterpriseDB Corporation that may be
>>>>> privileged, confidential, or exempt from disclosure under applicable
>>>>> law. If you are not the intended recipient or authorized to receive
>>>>> this for the intended recipient, any use, dissemination, distribution,
>>>>> retention, archiving, or copying of this communication is strictly
>>>>> prohibited. If you have received this e-mail in error, please notify
>>>>> the sender immediately by reply e-mail and delete this message.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Michael Paquier
>>>> http://michael.otacoo.com
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Abbas
>>> Architect
>>> EnterpriseDB Corporation
>>> The Enterprise PostgreSQL Company
>>>
>>> Phone: 92-334-5100153
>>>
>>> Website: www.enterprisedb.com
>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>
>>> This e-mail message (and any attachment) is intended for the use of
>>> the individual or entity to whom it is addressed. This message
>>> contains information from EnterpriseDB Corporation that may be
>>> privileged, confidential, or exempt from disclosure under applicable
>>> law. If you are not the intended recipient or authorized to receive
>>> this for the intended recipient, any use, dissemination, distribution,
>>> retention, archiving, or copying of this communication is strictly
>>> prohibited. If you have received this e-mail in error, please notify
>>> the sender immediately by reply e-mail and delete this message.
>>>
>>
>>
>>
>> --
>> Michael Paquier
>> http://michael.otacoo.com
>>
>
>
>
> --
> --
> Abbas
> Architect
> EnterpriseDB Corporation
> The Enterprise PostgreSQL Company
>
> Phone: 92-334-5100153
>
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
>
> This e-mail message (and any attachment) is intended for the use of
> the individual or entity to whom it is addressed. This message
> contains information from EnterpriseDB Corporation that may be
> privileged, confidential, or exempt from disclosure under applicable
> law. If you are not the intended recipient or authorized to receive
> this for the intended recipient, any use, dissemination, distribution,
> retention, archiving, or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify
> the sender immediately by reply e-mail and delete this message.
>



-- 
Michael Paquier
http://michael.otacoo.com

Thanks Michael, I just reviewed the patch and found one small thing.

In the function transformInsertStmt where we set
SetSendCommandId(true);
I think it would be better to put this statement under an
if (IS_PGXC_COORDINATOR && !IsConnFromCoord())

Other than that I am OK with the changes.

Regards

On Mon, Jul 23, 2012 at 7:08 AM, Michael Paquier
<mic...@gm...>wrote:

> Hi,
>
> I spent some time this morning rewriting your patch.
> I reduced the number of functions and parameters to a minimum number and
> eliminated the useless message 'Y'.
> 'Y' definitely doesn't make sense as it has absolutely the same role as
> 'M'. In the former patch, the notion of notification is simply to send a
> message from Coordinator to Datanode, but a Datanode can also send a GXID,
> snapshot and timestamp using the same message types as the one used when a
> Coordinator sends data to a remote node. So we shouldn't lose this
> portability of the new messages. The only thing that changes for
> Coordinator->remoteNode and remoteCode->Coordinator communication protocol
> is the place where message is read. in the first case, message is received
> by remote node in postgres.c stuff. In the second case, message is received
> inside execRemote.c, so this distinction makes perfectly safe the fact of
> using the same message 'M', making 'Y' unnecessary.
> Also, it is really important to limit the number of used message types to
> reduce the footprint of XC code on PostgreSQL.
>
> There are no problems with regressions, and performance is not impacted. I
> also corrected some comment format and added some other comments at various
> places.
> In consequence, and to simplify the work of everybody, I am going to
> commit this largely simplified version of the patch.
>
> Thanks,
>
>
> On Sat, Jul 21, 2012 at 6:29 PM, Abbas Butt <abb...@en...>wrote:
>
>>
>>
>> On Fri, Jul 20, 2012 at 11:04 AM, Michael Paquier <
>> mic...@gm...> wrote:
>>
>>> Hi,
>>>
>>> Sorry for being a little bit picky here:
>>> 1) In GetCurrentSubTransactionId of xact.c, you can replace:
>>> if (IS_PGXC_DATANODE && isCommandIdReceived)
>>> {
>>>    foo
>>> }
>>> else
>>> {
>>>    if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>       re_foo
>>> }
>>> by:
>>> if (IS_PGXC_DATANODE && isCommandIdReceived)
>>> {
>>>    foo
>>> }
>>> else if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>> {
>>>     re_foo
>>> }
>>>
>>
>> Done
>>
>>
>>> 2) I am not so much a fan of this 'Y' message as for now it just has a
>>> single usage, and M is doing exactly the same thing... But seen as a
>>> message that a remote node sends a notification to its backend, ok it makes
>>> some sense. In this case you should avoid to use a magic number like '1'.
>>> Please use a define like:
>>> #define DATANODE_NOTIFICATION_COMMAND_ID '1'
>>> A place like pgxc.h would make sense to define that.
>>> Then use this variable at the places where you need it
>>> (HandleDatanodeNotification of execRemote.c and ReportCommandIdChange of
>>> xact.c). It will bring more visibility to your code.
>>>
>>
>> Done
>>
>>
>>> 3) SetCmdIdSending and SetCmdIdReporting should be combined, they serve
>>> the same purpose. Their only difference is the node type where they are
>>> used. Let's make that generic. As a general name, SetCommandIdSending would
>>> be enough.
>>>
>>
>> I am not getting how to do that, consider the example of StartTransaction
>> function where we want to SetCmdIdReporting(false). Suppose we combine both
>> functions and have the new function like this
>>
>> void SetCmdIdSending(bool enb)
>> {
>>   if(IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>     sendCommandId = enb;
>>   else if(IS_PGXC_DATANODE)
>>     sendCommandId = enb;
>> }
>>
>> and that we call this function instead of SetCmdIdReporting. It will work
>> OK for data node but when the same code executes for coordinator we will
>> set sendCommandId to false which was not required. Perhaps I did not get
>> your point.
>>
>>
>>> 4) GetCmdIdOfDatanode and GetCmdIdOfCoordinator should also be combined.
>>> They do exactly the same thing, the only difference is that one is launched
>>> on Datanode and the other on Coordinator. A general name like
>>> GetCommandIdReceived would be nicer.
>>>
>>
>> See my comment for point (3)
>>
>> 5) In portalcmds.c, PerformCursorOpen. You need to replace
>>> if (IS_PGXC_COORDINATOR)
>>>                GetCurrentCommandId(true);
>>> by:
>>> if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>>>                GetCurrentCommandId(true);
>>>
>>
>> Done.
>>
>>
>>> 6) For the regressions, I am still getting the diffs. Based on your
>>> test, I get this ordering for PG and XC:
>>> pgxc# select * from test_text order by 1;
>>>                  a
>>> ------------------------------------
>>>  Bancroft                      Ave
>>>  Bancroft                      Ave
>>>  Birch                         St
>>>  Birch                         St
>>>  Blacow                        Road
>>>  Bridgepointe                  Dr
>>>  Broadmore                     Ave
>>>  Broadway
>>>  B                             St
>>> (9 rows)
>>> If we are getting the same diffs we should change the output, it looks
>>> to be a side-effect of your fix for INSERT SELECT on parent-child,
>>>
>>
>> Done
>>
>>
>>>
>>> On Fri, Jul 20, 2012 at 2:15 AM, Abbas Butt <abb...@en...
>>> > wrote:
>>>
>>>> Attached please find the updated patch.
>>>>
>>>> On Thu, Jul 19, 2012 at 11:01 AM, Abbas Butt <
>>>> abb...@en...> wrote:
>>>>
>>>>> Thanks Michael for the review, Thanks Ashutosh for the comments,
>>>>> Michael I will accommodate your valuable feedback and will send
>>>>> revised patch.
>>>>> Ashutosh I will send the summarized design in the email with the
>>>>> revised patch.
>>>>> BTW enb is short for enable!
>>>>>
>>>>>
>>>>> On Thu, Jul 19, 2012 at 10:37 AM, Michael Paquier <
>>>>> mic...@gm...> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Finally I took some time to look at this code, it is important
>>>>>> support.
>>>>>> 1) The code is not realigned with current master (xc_misc.sql) and
>>>>>> contains whitespaces (2 in xc_misc.out, perhaps other files as well). So I
>>>>>> realigned it myself to test the code.
>>>>>>
>>>>>
>>>> The attached patch is aligned with the current master and I have
>>>> removed white spaces.
>>>>
>>>>
>>>>>  2) In GetCurrentCommandId xact.c, you need to set !IsConnFromCoord()
>>>>>> with (IS_PGXC_COORDINATOR) to avoid remote coordinators incrementing the
>>>>>> command ID. Remote Coordinators do not receive any command IDs from remote
>>>>>> nodes. Also here using an if/else if structure won't hurt and bring clarity
>>>>>> in the operations.
>>>>>>
>>>>>
>>>> Added the check and used if/else
>>>>
>>>>
>>>>>  3) In BeginTransactionBlock of xact.c, you need to add
>>>>>> !IsConnFromCoord() as a secondary condition to avoid remote Coordinator
>>>>>> problems.
>>>>>>
>>>>>
>>>> Done
>>>>
>>>>
>>>>> 4) By looking at the patch It is possible to reduce the number of
>>>>>> variables used for the control of Command ID. In the current version of
>>>>>> your patch, you are using 5 static variables:
>>>>>> static CommandId    cmdIdFromCoord;        /* Datanode will use
>>>>>> command id received from coordinator */
>>>>>> static bool        cmdIdRcvdAtDatanode;    /* Has the coordinator
>>>>>> sent command id to the data node? */
>>>>>> static bool        enbCmdIdChgReporting;    /* Should datanode report
>>>>>> command id changes to coordinator? */
>>>>>> static bool        enbCoordCmdIdSending;    /* Should coordinator
>>>>>> send command id to the data node? */
>>>>>> static CommandId    cmdIdFromDatanode;    /* Coordinator will receive
>>>>>> updated command id from datanode */
>>>>>> However, cmdIdRcvdAtDatanode, cmdIdFromCoord and enbCmdIdChgReporting
>>>>>> are used only at Datanode.
>>>>>> cmdIdFromDatanode and enbCoordCmdIdSending are only used at
>>>>>> Coordinator level.
>>>>>> You should combine enbCoordCmdIdSending and enbCmdIdChgReporting into
>>>>>> a single variable with a generic name, like let's say sendCommandId or
>>>>>> reportCommandId. Specifying in comments the role of this unique variable
>>>>>> for local Coordinator and remote nodes will help. Those 2 variables play
>>>>>> the same role which is to control if a
>>>>>> In the case of Coordinator, this variable is flagged to true when
>>>>>> transforming INSERT statements (transformInsertStmt) and when a transaction
>>>>>> block is begun (BeginTransactionBlock).
>>>>>> In the case of a Datanode, this variable is flagged to true only when
>>>>>> a command ID has been received from Coordinator.
>>>>>>
>>>>>>
>>>> Done, I used functions to retain code readability and inside the
>>>> functions I used the same variables as u suggested.
>>>>
>>>>
>>>>> Then, you should combine cmdIdFromCoord and cmdIdFromDatanode into a
>>>>>> single variable, like receivedCommandId. In this case, being at Coordinator
>>>>>> or at Datanode means that this command ID has been received from a remode
>>>>>> node. If you have other ideas for names please feel free!
>>>>>>
>>>>>
>>>> Done, again I used functions to retain code readability. I however had
>>>> to add checks inside the functions to make sure that the functions intended
>>>> for data node work only for data nodes and the ones intended for
>>>> coordinators work only for coordinators.
>>>>
>>>>
>>>>> Also changing the name cmdIdRcvdAtDatanode to isCommandIdReceived
>>>>>> might help...
>>>>>>
>>>>>
>>>> Done
>>>>
>>>>
>>>>> Just a question, what means the preffix enb? (just by curiosity) :)
>>>>>>
>>>>>> 5) In xact.c, at the end of CommitTransaction, PrepareTransaction,
>>>>>> CleanupTransaction, you will need to change the cleanup of the variables if
>>>>>> you take into account comment 4).
>>>>>>
>>>>>
>>>> Done
>>>>
>>>>
>>>>> 6) In BeginTransactionBlock, PrepareTransactionBlock,
>>>>>> EndTransactionBlock and UserAbortTransactionBlock of xact.c, you need to
>>>>>> set an additional !IsConnFromCoord() at this place, this will avoid remote
>>>>>> Coordinators trying to send CommandIds to inexistent nodes.
>>>>>> if(IS_PGXC_COORDINATOR)
>>>>>> +        SetCmdIdSending(false);
>>>>>>
>>>>>
>>>> Done
>>>>
>>>>
>>>>>
>>>>>> 7) You should combine the messages 'Y' and 'M'. The content and the
>>>>>> goals are the same. You can also eliminate the 2nd byte of 'Y' without
>>>>>> consequences I think.
>>>>>>
>>>>>
>>>> I did not combine these two. The intention of having them separate is
>>>> because 'Y' should not be tied to 'M', In future we might need data nodes
>>>> to notify coordinators of some thing else. And for the same reason I have
>>>> added a second byte in 'Y' so that we can have the same command notify
>>>> coordinators of some thing else.
>>>>
>>>>
>>>>> 8) Why didn't you put the reception of message 'M' in PostgresMain of
>>>>>> postgres.c at the same place as the reception of GXID, snapshot, timestamp,
>>>>>> etc.
>>>>>> Putting the reception of message at the end of SocketBackend makes
>>>>>> the code kind of inconsistent. If you keep it inside SocketBackend, you
>>>>>> should at least manage the message inside the switch, however I do not see
>>>>>> any particular reason why you receive this message in SocketBackend and not
>>>>>> in PostgresMain. So anything I should know?
>>>>>> You should also remove the flag IS_PGXC_DATANODE when receiving the
>>>>>> command ID. Remote Coordinator might need to be able to receive command IDs
>>>>>> as well, even if there is no direct usage now, but for stuff like triggers??
>>>>>>
>>>>>
>>>> Done, I added handling of 'M' in PostgresMain.
>>>>
>>>>
>>>>>  9) I found some diffs with the test select_views. It might be worth
>>>>>> to look at what is happening. It is perhaps an environment-related issue.
>>>>>> Btw, my regression diffs are attached.
>>>>>>
>>>>>
>>>> I was also getting these diffs but it was difficult for me to think
>>>> that the changes I did could cause a change in oder of rows. Here is what I
>>>> did to dig the issue deeper.
>>>>
>>>> create table test_text(a text);
>>>>
>>>> insert into test_text values('Bancroft                      Ave');
>>>> insert into test_text values('Bancroft                      Ave');
>>>> insert into test_text values('Birch                         St');
>>>> insert into test_text values('Birch                         St');
>>>> insert into test_text values('Blacow                        Road');
>>>> insert into test_text values('Bridgepointe                  Dr');
>>>> insert into test_text values('Broadmore                     Ave');
>>>> insert into test_text values('Broadway                          ');
>>>> insert into test_text values('B                             St');
>>>>
>>>> select * from test_text order by 1
>>>> produces same results with plain PG and XC
>>>>
>>>> Also I changed the query in plain PG select_views test case to
>>>> SELECT * FROM street ORDER BY name,cname,thepath::text;
>>>> and obtained same results with plain PG and XC.
>>>>
>>>> I am not sure why the test case was passing earlier with the current
>>>> possible expected outputs.
>>>>
>>>> Summarized design notes are attached with the mail for reference.
>>>>
>>>>  10) The modifications you are bringing to combocid_1.out are
>>>>>> consistent with postgres, cool!
>>>>>>
>>>>>> No performance impact with this patch! Tested and approved. This is
>>>>>> good.
>>>>>> No regression impact with this patch.
>>>>>>
>>>>>> So, to conclude, I think it is a good piece of work, really cool
>>>>>> stuff that I believe will be helpful for triggers and even other things.
>>>>>> It just needs to be polished and simplified a bit. However basics are
>>>>>> here, and performance is not impacted. Really it was worth spending time on
>>>>>> that. The additional test cases also look to be sufficient, but I might be
>>>>>> missing smth so feel free to add new tests if necessary.
>>>>>> Thanks!
>>>>>>
>>>>>> --
>>>>>> Michael Paquier
>>>>>> http://michael.otacoo.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Abbas
>>>>> Architect
>>>>> EnterpriseDB Corporation
>>>>> The Enterprise PostgreSQL Company
>>>>>
>>>>> Phone: 92-334-5100153
>>>>>
>>>>> Website: www.enterprisedb.com
>>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>>>
>>>>> This e-mail message (and any attachment) is intended for the use of
>>>>> the individual or entity to whom it is addressed. This message
>>>>> contains information from EnterpriseDB Corporation that may be
>>>>> privileged, confidential, or exempt from disclosure under applicable
>>>>> law. If you are not the intended recipient or authorized to receive
>>>>> this for the intended recipient, any use, dissemination, distribution,
>>>>> retention, archiving, or copying of this communication is strictly
>>>>> prohibited. If you have received this e-mail in error, please notify
>>>>> the sender immediately by reply e-mail and delete this message.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Abbas
>>>> Architect
>>>> EnterpriseDB Corporation
>>>> The Enterprise PostgreSQL Company
>>>>
>>>> Phone: 92-334-5100153
>>>>
>>>> Website: www.enterprisedb.com
>>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>>
>>>> This e-mail message (and any attachment) is intended for the use of
>>>> the individual or entity to whom it is addressed. This message
>>>> contains information from EnterpriseDB Corporation that may be
>>>> privileged, confidential, or exempt from disclosure under applicable
>>>> law. If you are not the intended recipient or authorized to receive
>>>> this for the intended recipient, any use, dissemination, distribution,
>>>> retention, archiving, or copying of this communication is strictly
>>>> prohibited. If you have received this e-mail in error, please notify
>>>> the sender immediately by reply e-mail and delete this message.
>>>>
>>>
>>>
>>>
>>> --
>>> Michael Paquier
>>> http://michael.otacoo.com
>>>
>>
>>
>>
>> --
>> --
>> Abbas
>> Architect
>> EnterpriseDB Corporation
>> The Enterprise PostgreSQL Company
>>
>> Phone: 92-334-5100153
>>
>> Website: www.enterprisedb.com
>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>
>> This e-mail message (and any attachment) is intended for the use of
>> the individual or entity to whom it is addressed. This message
>> contains information from EnterpriseDB Corporation that may be
>> privileged, confidential, or exempt from disclosure under applicable
>> law. If you are not the intended recipient or authorized to receive
>> this for the intended recipient, any use, dissemination, distribution,
>> retention, archiving, or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify
>> the sender immediately by reply e-mail and delete this message.
>>
>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

Thanks.   I found the latter works for the master.
---
Koichi

On Mon, 23 Jul 2012 12:53:50 +0900
Michael Paquier <mic...@gm...> wrote:

> > 2. select application_name,upper(state),upper(sync_state) from
> > pg_stat_replication;
> >
> > This does not work.  pg_stat_replication has zero row.
> >
> Because this is datanode slave, we don't have GTM connection or
> > snapshot.  This might be a cause of the problem 2, or we might have
> > blocked to update pg_stat_replication system catalog.
> >
> pg_stat_replication has by definition one row per wal sender process:
> http://www.postgresql.org/docs/9.1/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE
> AFAIK, slave has no wal sender processes running, so it is normal to see
> nothing on the slave side, all the data is on the master side. The data is
> only available on master, Am I missing smth?
> You might be able to see rows in pg_stat_replication on a slave if you use
> cascading replication however, but this feature is only available in 9.2.
> Also, in the case of XC, you can use EXECUTE DIRECT to get the information
> of pg_stat_replication from remote nodes, Coordinators and Datanodes
> included.
> -- 
> Michael Paquier
> http://michael.otacoo.com

> 2. select application_name,upper(state),upper(sync_state) from
> pg_stat_replication;
>
> This does not work.  pg_stat_replication has zero row.
>
Because this is datanode slave, we don't have GTM connection or
> snapshot.  This might be a cause of the problem 2, or we might have
> blocked to update pg_stat_replication system catalog.
>
pg_stat_replication has by definition one row per wal sender process:
http://www.postgresql.org/docs/9.1/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE
AFAIK, slave has no wal sender processes running, so it is normal to see
nothing on the slave side, all the data is on the master side. The data is
only available on master, Am I missing smth?
You might be able to see rows in pg_stat_replication on a slave if you use
cascading replication however, but this feature is only available in 9.2.
Also, in the case of XC, you can use EXECUTE DIRECT to get the information
of pg_stat_replication from remote nodes, Coordinators and Datanodes
included.
-- 
Michael Paquier
http://michael.otacoo.com

Hi,

I had a chance to meet people from Linux-HA Japan an discussed how to
provide Pacemaker RA (resource agent) for coordinator/datanode.   I
found present PostgreSQL RA (for 9.1) uses the following queries to
the hot standby node.

1. select pg_is_in_recovery();

works fine.

2. select application_name,upper(state),upper(sync_state) from
pg_stat_replication;

This does not work.  pg_stat_replication has zero row.

3. select pg_last_xlog_replay_location(),pg_last_xlog_receive_location();

works fine.

4. select now();

works fine.

Because this is datanode slave, we don't have GTM connection or
snapshot.   This might be a cause of the problem 2, or we might have
blocked to update pg_stat_replication system catalog.

Any suggestions?
----------
Koichi Suzuki

On Sat, Jul 21, 2012 at 8:28 AM, Noe Gutierrez <noe...@gm...> wrote:

> I have a problem.
>
> Could you help me?
>
>
> I have  three machines:
>
>
> machine 1:  I have GTM
>
> machine 2: I have PGXC  coordinator1 and node 1
>
> machine 3 : I have PGXC corrdinator 2 and node 2
>
>
> The Replication works successfull but when the machine 1 crashes  the
> replication does not work.
>
Unlike OracleRAC, XC has no feature to fallback to a non-failed node if you
try to send a query to a failed node for a replicated table.
However, you can define a preferred node for a read operation of replicated
table. This will allow to redirect all the read operation to the node that
did not fail and avoid this error.
Write operations will still fail as long as the node 1 is not back online
however.

How I can configure feature failover or something like it because machine 2
> does not execute the querys and shows mistakes about pooling failed.

You will need slave nodes for Datanodes to do that, like postgres.
-- 
Michael Paquier
http://michael.otacoo.com

On Fri, Jul 20, 2012 at 11:04 AM, Michael Paquier <mic...@gm...
> wrote:

> Hi,
>
> Sorry for being a little bit picky here:
> 1) In GetCurrentSubTransactionId of xact.c, you can replace:
> if (IS_PGXC_DATANODE && isCommandIdReceived)
> {
>    foo
> }
> else
> {
>    if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>       re_foo
> }
> by:
> if (IS_PGXC_DATANODE && isCommandIdReceived)
> {
>    foo
> }
> else if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
> {
>     re_foo
> }
>

Done


> 2) I am not so much a fan of this 'Y' message as for now it just has a
> single usage, and M is doing exactly the same thing... But seen as a
> message that a remote node sends a notification to its backend, ok it makes
> some sense. In this case you should avoid to use a magic number like '1'.
> Please use a define like:
> #define DATANODE_NOTIFICATION_COMMAND_ID '1'
> A place like pgxc.h would make sense to define that.
> Then use this variable at the places where you need it
> (HandleDatanodeNotification of execRemote.c and ReportCommandIdChange of
> xact.c). It will bring more visibility to your code.
>

Done


> 3) SetCmdIdSending and SetCmdIdReporting should be combined, they serve
> the same purpose. Their only difference is the node type where they are
> used. Let's make that generic. As a general name, SetCommandIdSending would
> be enough.
>

I am not getting how to do that, consider the example of StartTransaction
function where we want to SetCmdIdReporting(false). Suppose we combine both
functions and have the new function like this

void SetCmdIdSending(bool enb)
{
  if(IS_PGXC_COORDINATOR && !IsConnFromCoord())
    sendCommandId = enb;
  else if(IS_PGXC_DATANODE)
    sendCommandId = enb;
}

and that we call this function instead of SetCmdIdReporting. It will work
OK for data node but when the same code executes for coordinator we will
set sendCommandId to false which was not required. Perhaps I did not get
your point.


> 4) GetCmdIdOfDatanode and GetCmdIdOfCoordinator should also be combined.
> They do exactly the same thing, the only difference is that one is launched
> on Datanode and the other on Coordinator. A general name like
> GetCommandIdReceived would be nicer.
>

See my comment for point (3)

5) In portalcmds.c, PerformCursorOpen. You need to replace
> if (IS_PGXC_COORDINATOR)
>                GetCurrentCommandId(true);
> by:
> if (IS_PGXC_COORDINATOR && !IsConnFromCoord())
>                GetCurrentCommandId(true);
>

Done.


> 6) For the regressions, I am still getting the diffs. Based on your test,
> I get this ordering for PG and XC:
> pgxc# select * from test_text order by 1;
>                  a
> ------------------------------------
>  Bancroft                      Ave
>  Bancroft                      Ave
>  Birch                         St
>  Birch                         St
>  Blacow                        Road
>  Bridgepointe                  Dr
>  Broadmore                     Ave
>  Broadway
>  B                             St
> (9 rows)
> If we are getting the same diffs we should change the output, it looks to
> be a side-effect of your fix for INSERT SELECT on parent-child,
>

Done


>
> On Fri, Jul 20, 2012 at 2:15 AM, Abbas Butt <abb...@en...>wrote:
>
>> Attached please find the updated patch.
>>
>> On Thu, Jul 19, 2012 at 11:01 AM, Abbas Butt <abb...@en...
>> > wrote:
>>
>>> Thanks Michael for the review, Thanks Ashutosh for the comments,
>>> Michael I will accommodate your valuable feedback and will send revised
>>> patch.
>>> Ashutosh I will send the summarized design in the email with the revised
>>> patch.
>>> BTW enb is short for enable!
>>>
>>>
>>> On Thu, Jul 19, 2012 at 10:37 AM, Michael Paquier <
>>> mic...@gm...> wrote:
>>>
>>>> Hi,
>>>>
>>>> Finally I took some time to look at this code, it is important support.
>>>> 1) The code is not realigned with current master (xc_misc.sql) and
>>>> contains whitespaces (2 in xc_misc.out, perhaps other files as well). So I
>>>> realigned it myself to test the code.
>>>>
>>>
>> The attached patch is aligned with the current master and I have removed
>> white spaces.
>>
>>
>>>  2) In GetCurrentCommandId xact.c, you need to set !IsConnFromCoord()
>>>> with (IS_PGXC_COORDINATOR) to avoid remote coordinators incrementing the
>>>> command ID. Remote Coordinators do not receive any command IDs from remote
>>>> nodes. Also here using an if/else if structure won't hurt and bring clarity
>>>> in the operations.
>>>>
>>>
>> Added the check and used if/else
>>
>>
>>>  3) In BeginTransactionBlock of xact.c, you need to add
>>>> !IsConnFromCoord() as a secondary condition to avoid remote Coordinator
>>>> problems.
>>>>
>>>
>> Done
>>
>>
>>> 4) By looking at the patch It is possible to reduce the number of
>>>> variables used for the control of Command ID. In the current version of
>>>> your patch, you are using 5 static variables:
>>>> static CommandId    cmdIdFromCoord;        /* Datanode will use command
>>>> id received from coordinator */
>>>> static bool        cmdIdRcvdAtDatanode;    /* Has the coordinator sent
>>>> command id to the data node? */
>>>> static bool        enbCmdIdChgReporting;    /* Should datanode report
>>>> command id changes to coordinator? */
>>>> static bool        enbCoordCmdIdSending;    /* Should coordinator send
>>>> command id to the data node? */
>>>> static CommandId    cmdIdFromDatanode;    /* Coordinator will receive
>>>> updated command id from datanode */
>>>> However, cmdIdRcvdAtDatanode, cmdIdFromCoord and enbCmdIdChgReporting
>>>> are used only at Datanode.
>>>> cmdIdFromDatanode and enbCoordCmdIdSending are only used at Coordinator
>>>> level.
>>>> You should combine enbCoordCmdIdSending and enbCmdIdChgReporting into a
>>>> single variable with a generic name, like let's say sendCommandId or
>>>> reportCommandId. Specifying in comments the role of this unique variable
>>>> for local Coordinator and remote nodes will help. Those 2 variables play
>>>> the same role which is to control if a
>>>> In the case of Coordinator, this variable is flagged to true when
>>>> transforming INSERT statements (transformInsertStmt) and when a transaction
>>>> block is begun (BeginTransactionBlock).
>>>> In the case of a Datanode, this variable is flagged to true only when a
>>>> command ID has been received from Coordinator.
>>>>
>>>>
>> Done, I used functions to retain code readability and inside the
>> functions I used the same variables as u suggested.
>>
>>
>>> Then, you should combine cmdIdFromCoord and cmdIdFromDatanode into a
>>>> single variable, like receivedCommandId. In this case, being at Coordinator
>>>> or at Datanode means that this command ID has been received from a remode
>>>> node. If you have other ideas for names please feel free!
>>>>
>>>
>> Done, again I used functions to retain code readability. I however had to
>> add checks inside the functions to make sure that the functions intended
>> for data node work only for data nodes and the ones intended for
>> coordinators work only for coordinators.
>>
>>
>>> Also changing the name cmdIdRcvdAtDatanode to isCommandIdReceived might
>>>> help...
>>>>
>>>
>> Done
>>
>>
>>> Just a question, what means the preffix enb? (just by curiosity) :)
>>>>
>>>> 5) In xact.c, at the end of CommitTransaction, PrepareTransaction,
>>>> CleanupTransaction, you will need to change the cleanup of the variables if
>>>> you take into account comment 4).
>>>>
>>>
>> Done
>>
>>
>>> 6) In BeginTransactionBlock, PrepareTransactionBlock,
>>>> EndTransactionBlock and UserAbortTransactionBlock of xact.c, you need to
>>>> set an additional !IsConnFromCoord() at this place, this will avoid remote
>>>> Coordinators trying to send CommandIds to inexistent nodes.
>>>> if(IS_PGXC_COORDINATOR)
>>>> +        SetCmdIdSending(false);
>>>>
>>>
>> Done
>>
>>
>>>
>>>> 7) You should combine the messages 'Y' and 'M'. The content and the
>>>> goals are the same. You can also eliminate the 2nd byte of 'Y' without
>>>> consequences I think.
>>>>
>>>
>> I did not combine these two. The intention of having them separate is
>> because 'Y' should not be tied to 'M', In future we might need data nodes
>> to notify coordinators of some thing else. And for the same reason I have
>> added a second byte in 'Y' so that we can have the same command notify
>> coordinators of some thing else.
>>
>>
>>> 8) Why didn't you put the reception of message 'M' in PostgresMain of
>>>> postgres.c at the same place as the reception of GXID, snapshot, timestamp,
>>>> etc.
>>>> Putting the reception of message at the end of SocketBackend makes the
>>>> code kind of inconsistent. If you keep it inside SocketBackend, you should
>>>> at least manage the message inside the switch, however I do not see any
>>>> particular reason why you receive this message in SocketBackend and not in
>>>> PostgresMain. So anything I should know?
>>>> You should also remove the flag IS_PGXC_DATANODE when receiving the
>>>> command ID. Remote Coordinator might need to be able to receive command IDs
>>>> as well, even if there is no direct usage now, but for stuff like triggers??
>>>>
>>>
>> Done, I added handling of 'M' in PostgresMain.
>>
>>
>>>  9) I found some diffs with the test select_views. It might be worth to
>>>> look at what is happening. It is perhaps an environment-related issue. Btw,
>>>> my regression diffs are attached.
>>>>
>>>
>> I was also getting these diffs but it was difficult for me to think that
>> the changes I did could cause a change in oder of rows. Here is what I did
>> to dig the issue deeper.
>>
>> create table test_text(a text);
>>
>> insert into test_text values('Bancroft                      Ave');
>> insert into test_text values('Bancroft                      Ave');
>> insert into test_text values('Birch                         St');
>> insert into test_text values('Birch                         St');
>> insert into test_text values('Blacow                        Road');
>> insert into test_text values('Bridgepointe                  Dr');
>> insert into test_text values('Broadmore                     Ave');
>> insert into test_text values('Broadway                          ');
>> insert into test_text values('B                             St');
>>
>> select * from test_text order by 1
>> produces same results with plain PG and XC
>>
>> Also I changed the query in plain PG select_views test case to
>> SELECT * FROM street ORDER BY name,cname,thepath::text;
>> and obtained same results with plain PG and XC.
>>
>> I am not sure why the test case was passing earlier with the current
>> possible expected outputs.
>>
>> Summarized design notes are attached with the mail for reference.
>>
>>  10) The modifications you are bringing to combocid_1.out are consistent
>>>> with postgres, cool!
>>>>
>>>> No performance impact with this patch! Tested and approved. This is
>>>> good.
>>>> No regression impact with this patch.
>>>>
>>>> So, to conclude, I think it is a good piece of work, really cool stuff
>>>> that I believe will be helpful for triggers and even other things.
>>>> It just needs to be polished and simplified a bit. However basics are
>>>> here, and performance is not impacted. Really it was worth spending time on
>>>> that. The additional test cases also look to be sufficient, but I might be
>>>> missing smth so feel free to add new tests if necessary.
>>>> Thanks!
>>>>
>>>> --
>>>> Michael Paquier
>>>> http://michael.otacoo.com
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Abbas
>>> Architect
>>> EnterpriseDB Corporation
>>> The Enterprise PostgreSQL Company
>>>
>>> Phone: 92-334-5100153
>>>
>>> Website: www.enterprisedb.com
>>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>>
>>> This e-mail message (and any attachment) is intended for the use of
>>> the individual or entity to whom it is addressed. This message
>>> contains information from EnterpriseDB Corporation that may be
>>> privileged, confidential, or exempt from disclosure under applicable
>>> law. If you are not the intended recipient or authorized to receive
>>> this for the intended recipient, any use, dissemination, distribution,
>>> retention, archiving, or copying of this communication is strictly
>>> prohibited. If you have received this e-mail in error, please notify
>>> the sender immediately by reply e-mail and delete this message.
>>>
>>
>>
>>
>> --
>> --
>> Abbas
>> Architect
>> EnterpriseDB Corporation
>> The Enterprise PostgreSQL Company
>>
>> Phone: 92-334-5100153
>>
>> Website: www.enterprisedb.com
>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>
>> This e-mail message (and any attachment) is intended for the use of
>> the individual or entity to whom it is addressed. This message
>> contains information from EnterpriseDB Corporation that may be
>> privileged, confidential, or exempt from disclosure under applicable
>> law. If you are not the intended recipient or authorized to receive
>> this for the intended recipient, any use, dissemination, distribution,
>> retention, archiving, or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify
>> the sender immediately by reply e-mail and delete this message.
>>
>
>
>
> --
> Michael Paquier
> http://michael.otacoo.com
>



-- 
--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

GTM provides consistent transaction management to all the coordinators
and datanodes.   In this case, you should run another GTM on another
machine (could be machine 2 or machine 3) as standby.   GTM standby
backs up everything of the original GTM (master).    Also, please
configure machine 2 and machine 3 with gtm_proxy and connect all the
coordinator/datanode to their local gtm_proxy.

When GTM fails, you can promote the standby as the new master and
reconnect gtm_proxies to the new master.   They can be done by
gtm_ctl.

Good luck;
----------
Koichi Suzuki


2012/7/21 Noe Gutierrez <noe...@gm...>:
> I have a problem.
>
> Could you help me?
>
>
> I have  three machines:
>
>
> machine 1:  I have GTM
>
> machine 2: I have PGXC  coordinator1 and node 1
>
> machine 3 : I have PGXC corrdinator 2 and node 2
>
>
> The Replication works successfull but when the machine 1 crashes  the
> replication does not work.
>
> How I can configure feature failover or something like it because machine 2
> does not execute the querys and shows mistakes about pooling failed.
>
>
>
> Regards
>
>
>
>
>
> I.S.C. Noe Adahi Gutiérrez Romero
> Oracle® Certified Professional, Java® SE 6 Programmer
> IBM® InfoShere® Guardium® Technical Professional v1
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

I have a problem.

Could you help me?


I have  three machines:


machine 1:  I have GTM

machine 2: I have PGXC  coordinator1 and node 1

machine 3 : I have PGXC corrdinator 2 and node 2


The Replication works successfull but when the machine 1 crashes  the
replication does not work.

How I can configure feature failover or something like it because machine 2
does not execute the querys and shows mistakes about pooling failed.



Regards





*I.S.C. Noe Adahi Gutiérrez Romero*
***Oracle® Certified Professional, Java® SE 6 Programmer*
*IBM® InfoShere® Guardium® Technical Professional v1*