postgres-xc-developers Mailing List for Postgres-XC

On Sun, Oct 30, 2011 at 8:47 PM, Michael Paquier
<mic...@gm...> wrote:
>
>
> On Mon, Oct 31, 2011 at 9:25 AM, Mason <ma...@us...>
> wrote:
>>
>> On Sun, Oct 30, 2011 at 7:29 PM, Michael Paquier
>> <mic...@gm...> wrote:
>> >
>> >
>> > On Mon, Oct 31, 2011 at 6:56 AM, Mason <ma...@us...>
>> > wrote:
>> >>
>> >> I have some feedback and questions from looking at the code. Note, I
>> >> have not been able to get the latest code running yet.
>> >>
>> >> Instead of "TO" in CREATE TABLE ... TO NODE and TO GROUP, I think "ON"
>> >> sounds more correct.
>> >
>> > This was the first thought.
>> > However, using ON made bison complaining about conflicting keywords.
>>
>> Ah, the ON CONFIG clause.
>
> There are no such clauses existing but... This is tricky here.

Sorry I meant ON COMMIT,   Onconfig is an Informix term (which I recently used).

>
>>
>> That is what I meant asking if there were any dangers with this
>> scheme. I think you said though that these are only assigned at initdb
>> time, and it appears can never be changed, so perhaps it is not a
>> danger.  I was getting at if partitions should be named and node
>> instances also named. That way a given partition name will be on a
>> master data node as well as slaves, which may be named differently,
>> but still use the same partition name/id.  I think you said we don't
>> need to worry about slaves, but I was just confused by the new DDL for
>> slaves. I guess the slaves need to be renamed to be the same as the
>> master if one fails over to them to ensure sort order?
>
> Yes, this would be necessary to insure that OID sort order is not changed.
> However, if a slave becomes a master I think that having the slave node
> changing its name to the former master one is not a big matter.
> This is another discussion related to failover though.
>

I was not sure of the overall design, just trying to point out the
theoretical danger, and again, because DDL was added for slaves, I was
originally not sure if the intention was to allow them to retain their
names and yet be able to be promoted. That is why I mentioned naming
the partitions and using that order. Alternatively, perhaps one could
use an internal id/OID behind the scenes for the partions, sort by
that for determine hash & modulo buckets, and have a mapping of the
partitions and node instances. Each master and standby should know
what its partion id/oid is, perhaps returned at connection time when
the connection comes from a coordinator.  This might do away with the
node renaming issue.  Just something to mull over.  Or, maybe some
standby naming convention will help. We should just think ahead a
little bit about possible HA management scenarios for flexibility.

>> >> How would (outside directed) failover occur if the standby has a
>> >> different
>> >> name?
>> >>
>> >> Does GTM insist on the standby having the same name if the primary
>> >> disconnects and the standby connects?
>> >
>> > I am not sure about the internal mechanisms used by GTM-Standby when
>> > registering on GTM.
>> > Suzuki-san will be better placed than me to answer.
>>
>> I meant related to data node standbys. I was wondering if a standby
>> data node takes over for a data node master and the standby has a
>> different name, how GTM handles that.
>
> Just about that....
> The first meaning of registering nodes on GTM was to keep a track of all the
> node information in the cluster.
> But now that node names are the same and have to remain constant in the
> cluster, is this really necessary?
> Removing that will also allow to remove pgxc_node_name from guc params.
> Then identifying a node self for Coordinator/Datanode could be done by
> initdb with a SELF/NOT SELF keyword as an extension of CREATE DDL.

Maybe it is a good idea. How would a node rename itself later then?
ALTER NODE oldname RENAME newname? Then if it sees that its own name
matches oldname, change itself to newname? Otherwise just update
catalog info?


> --
> Michael Paquier
> http://michael.otacoo.com
>

On Mon, Oct 31, 2011 at 9:25 AM, Mason <ma...@us...>wrote:

> On Sun, Oct 30, 2011 at 7:29 PM, Michael Paquier
> <mic...@gm...> wrote:
> >
> >
> > On Mon, Oct 31, 2011 at 6:56 AM, Mason <ma...@us...>
> > wrote:
> >>
> >> I have some feedback and questions from looking at the code. Note, I
> >> have not been able to get the latest code running yet.
> >>
> >> Instead of "TO" in CREATE TABLE ... TO NODE and TO GROUP, I think "ON"
> >> sounds more correct.
> >
> > This was the first thought.
> > However, using ON made bison complaining about conflicting keywords.
>
> Ah, the ON CONFIG clause.
>
There are no such clauses existing but... This is tricky here.


> That is what I meant asking if there were any dangers with this
> scheme. I think you said though that these are only assigned at initdb
> time, and it appears can never be changed, so perhaps it is not a
> danger.  I was getting at if partitions should be named and node
> instances also named. That way a given partition name will be on a
> master data node as well as slaves, which may be named differently,
> but still use the same partition name/id.  I think you said we don't
> need to worry about slaves, but I was just confused by the new DDL for
> slaves. I guess the slaves need to be renamed to be the same as the
> master if one fails over to them to ensure sort order?
>
Yes, this would be necessary to insure that OID sort order is not changed.
However, if a slave becomes a master I think that having the slave node
changing its name to the former master one is not a big matter.
This is another discussion related to failover though.

 >> How would (outside directed) failover occur if the standby has a
> different
> >> name?
> >>
> >> Does GTM insist on the standby having the same name if the primary
> >> disconnects and the standby connects?
> >
> > I am not sure about the internal mechanisms used by GTM-Standby when
> > registering on GTM.
> > Suzuki-san will be better placed than me to answer.
>
> I meant related to data node standbys. I was wondering if a standby
> data node takes over for a data node master and the standby has a
> different name, how GTM handles that.
>
Just about that....
The first meaning of registering nodes on GTM was to keep a track of all
the node information in the cluster.
But now that node names are the same and have to remain constant in the
cluster, is this really necessary?
Removing that will also allow to remove pgxc_node_name from guc params.
Then identifying a node self for Coordinator/Datanode could be done by
initdb with a SELF/NOT SELF keyword as an extension of CREATE DDL.
-- 
Michael Paquier
http://michael.otacoo.com

On Sun, Oct 30, 2011 at 7:29 PM, Michael Paquier
<mic...@gm...> wrote:
>
>
> On Mon, Oct 31, 2011 at 6:56 AM, Mason <ma...@us...>
> wrote:
>>
>> I have some feedback and questions from looking at the code. Note, I
>> have not been able to get the latest code running yet.
>>
>> Instead of "TO" in CREATE TABLE ... TO NODE and TO GROUP, I think "ON"
>> sounds more correct.
>
> This was the first thought.
> However, using ON made bison complaining about conflicting keywords.

Ah, the ON CONFIG clause.

>>
>> We may want to consider just renaming HOSTIP to HOST.  I don't feel
>> that strongly about it though. "HOSTIP" makes it sound like an ip
>> address is required. PORT might be enough instead of HOSTPORT.
>>
>> RELATED (TO) is somewhat awkward.  How about defining slaves instead
>> as something like:
>>
>> CREATE NODE datanode1a WITH
>> (NODE SLAVE FOR datanode1
>>  HOSTIP = 192.168.1.2,
>>  NODEPORT = 15243
>> );
>>
>>
>> I like the possibility to load balance reads, hence the PREFERRED
>> option for which data node in a related group to use, however I think
>> coordinators each should still be able to specify where they would
>> like to go for load balancing reads for replicated tables, so that
>> they can choose to override the default one for a data node, allowing
>> them to pick a local instance, for example. Something like:
>>
>> CREATE NODE COORDINATOR coord1 WITH
>> (HOSTIP = localhost,
>>  NODEPORT = 15244,
>>  PREFERRED = datanode1)
>
> On each Coordinator is locally stored a version of pgxc_node catalog.
> As a next step, I was planning to add an option with host name, preferred
> and primary in ALTER NODE to be able to modify the data of a node only on a
> local Coordinator.
> It would have the shape of an SQL like this:
> CREATE/ALTER LOCAL NODE nodename WITH (...)

I think allowing local only changes may be dangerous. I suppose you
could do something like you say but heavily restrict it, like only
allow the PREFERRED option to be used.  I think what I suggested about
would work well and still allow the info to be global.

>>
>> I see some comments about sorting the nodes so that modulo evaluates
>> to right node consistently, etc. Is this still safe when say, a master
>> goes down, we promote a standby who has an OID (or name) that changes
>> the sorted order?
>
> Yes it is. When a session goes up, it reads the information located in
> pgxc_node and stores in cache the information related to nodes by sorting
> them into sort names.
> This works as long as the information cached in pooler is not changed.
> That's why an additional command able to change the data cached in pooler
> once the information of catalog has been modified is necessary.
> Launching this command also forces to refresh all the other session caches
> though and might not be launched if a session has an open connection to a
> node.

>>
>> Also, can you please describe more about how OIDs are used for nodes,
>> and how/if they are consistent across the cluster in such a case? (The
>> primary coordinator assigns these I assume?)
>
> Oid are used to get the host/port/preferred/primary data.
> Also, when a session begins, this session reads information from pgxc_node,
> classifies the node information by sorting Oids with node names, and then
> uses the sorting order to specify a node index.
> Have a look at the end of pgxnode.c with APIs PGXCNodeGetNodeOid and
> PGXCNodeGetNodeId.
> Those ones are the important folks.
>
>> We could think about it in terms of a logical partition/segment having
>> an id/oid associated with it, which is the same whether it is on the
>> master or one or of the slaves. I am not sure if OID in the current
>> code here refers to an actual node instance, instead of a logical
>> number.
>
> The node index is the same on all the nodes.
> However the node Oid is different.
> For example, let's say that you have 2 Coordinators with the following
> Datanode information:
> Coordinator 1:
> - dn1, OID = 100
> - dn3, OID = 101
> - dn2, OID = 102
> Coordinator 2:
> - dn3, OID = 100
> - dn1, OID = 101
> - dn2, OID = 102
> Each Coordinator will use dn1 with index 1, dn2 with index 2, and dn3 with
> index 3.
> When requesting connections from pooler, sessions use those indexes.

OK, so on all coordinators, we sort by name.

Because the new DDL has stuff in there for slaves, I was unclear what
you had in mind related to that. Say dn3's slave is named dn1a for
some crazy reason. If someone were to stop the cluster and change the
node info we would then sort and have

  dn1 with index 1, dn1a with index 2, dn2 with index 3

That is what I meant asking if there were any dangers with this
scheme. I think you said though that these are only assigned at initdb
time, and it appears can never be changed, so perhaps it is not a
danger.  I was getting at if partitions should be named and node
instances also named. That way a given partition name will be on a
master data node as well as slaves, which may be named differently,
but still use the same partition name/id.  I think you said we don't
need to worry about slaves, but I was just confused by the new DDL for
slaves. I guess the slaves need to be renamed to be the same as the
master if one fails over to them to ensure sort order?


>
>>
>> A logical id would allow for making high availability more flexible.
>> It would be nice if the connection is only obtained via an id for a
>> logical partition, instead of a physical node id that was used during
>> planning, in case failover occurred.
>
> It could be possible to change pooler to request connections not from a node
> index, but from a node name, or from the locally consistent node Oid.
> I am more a fan of the 2nd option, as node names are unique in the cluster.
>>
>> I see some stuff using nodestrings instead of arrays of nodeIds in
>> places. I understand wanting to give the DBA the ability to name these
>> nodes, but once things are happening internally, it might be good to
>> try and have these use some internal integer id instead for
>> efficiency. Admittedly, I don't see a lot of string comparisons going
>> on though.
>
> As explained below, the node names are used only to sort the node
> information in cache of pgxcnode.c.
>>
>> Does GTM need to worry about slaves?
>
> GTM do not need to care about slaves I think. This is another discussion
> though.

OK, just confused by the new DDL for slaves, was not sure.

>>
>> If not, why not just register a logical partition id instead of a node
>> name for data nodes?
>
> And why Postgres does not use a partition ID when identifying slaves for
> syncrep?
> Names look more flexible.

Has to do with the issue above, but I guess it is not an issue if node
names are fixed.

>
>> How would (outside directed) failover occur if the standby has a different
>> name?
>>
>> Does GTM insist on the standby having the same name if the primary
>> disconnects and the standby connects?
>
> I am not sure about the internal mechanisms used by GTM-Standby when
> registering on GTM.
> Suzuki-san will be better placed than me to answer.

I meant related to data node standbys. I was wondering if a standby
data node takes over for a data node master and the standby has a
different name, how GTM handles that.

Thanks,

Mason

>
> Regards,
> --
> Michael Paquier
> http://michael.otacoo.com
>