Re: [Postgres-xc-developers] Patch correcting implicit 2-phase commit

On 12/14/10 9:37 PM, Michael Paquier wrote:
>
>     Just took a brief look so far.  Seems better.
>
>     I understand that recovery and HA is in development and things are
>     being done to lay the groundwork and improve, and that with this
>     patch we are not trying to yet handle any and every situation.
>     What happens if the coordinator fails before it can update GTM though?
>
> In this case the information is not saved on GTM.
> For a Coordinator crash, I was thinking of an external utility 
> associated with the monitoring agent in charge of analyzing prepared 
> transactions of the crashed Coordinator.
> This utility would analyze in the cluster the prepared transaction of 
> the crashed Coordinator, and decide automatically which one to abort, 
> commit depending on the transaction situation.
>
> For this purpose, it is essential to extend the 2PC information sent 
> to Nodes (Datanodes of course, but Coordinators included in case of DDL).
> The patch extending 2PC information on nodes is also on this thread 
> (patch based on version 6 of implicit 2pc patch).
> In this case I believe it is not necessary to save any info on GTM as 
> the extended 2PC information only would be necessary to analyze the 
> 2PC transaction of the crashed Coordinator.
>
>
>     Also, I did a test and got this:
>
>
>     WARNING:  unexpected EOF on datanode connection
>     WARNING:  Connection to Datanode 1 has unexpected state 1 and will
>     be dropped
>
>     ERROR:  Could not commit prepared transaction implicitely
>     server closed the connection unexpectedly
>        This probably means the server terminated abnormally
>        before or while processing the request.
>     The connection to the server was lost. Attempting reset: Failed.
>
>     #0  0x907afe42 in kill$UNIX2003 ()
>     #1  0x9082223a in raise ()
>     #2  0x9082e679 in abort ()
>     #3  0x003917ce in ExceptionalCondition (conditionName=0x433f6c
>     "!(((proc->xid) != ((TransactionId) 0)))", errorType=0x3ecfd4
>     "FailedAssertion", fileName=0x433f50 "procarray.c",
>     lineNumber=283) at assert.c:57
>     #4  0x00280916 in ProcArrayEndTransaction (proc=0x41cca70,
>     latestXid=1018) at procarray.c:283
>     #5  0x0005905c in AbortTransaction () at xact.c:2525
>     #6  0x00059a6e in AbortCurrentTransaction () at xact.c:3001
>     #7  0x00059b10 in AbortCurrentTransactionOnce () at xact.c:3094
>     #8  0x0029c8d6 in PostgresMain (argc=4, argv=0x1002ff8,
>     username=0x1002fc8 "masonsharp") at postgres.c:3622
>     #9  0x0025851c in BackendRun (port=0x7016f0) at postmaster.c:3607
>     #10 0x00257883 in BackendStartup (port=0x7016f0) at postmaster.c:3216
>     #11 0x002542b5 in ServerLoop () at postmaster.c:1445
>     #12 0x002538c1 in PostmasterMain (argc=5, argv=0x7005a0) at
>     postmaster.c:1098
>     #13 0x001cf2f1 in main (argc=5, argv=0x7005a0) at main.c:188
>
> I suppose you enabled assertions when doing this test.
> The Coordinator was complaining that its transaction ID in PGProc was 
> not correct.
> It is indeed true as in the case tested the transaction has ever 
> committed on Coordinator.
I tried out the latest patch and it still crashes the coordinator.

#0  pgxc_node_implicit_commit_prepared (prepare_xid=924, commit_xid=925, 
pgxc_handles=0x1042c0c, gid=0xbfffef4f "T924", is_commit=1 '\001') at 
execRemote.c:1826
1826        int        co_conn_count = pgxc_handles->co_conn_count;
(gdb) bt
#0  pgxc_node_implicit_commit_prepared (prepare_xid=924, commit_xid=925, 
pgxc_handles=0x1042c0c, gid=0xbfffef4f "T924", is_commit=1 '\001') at 
execRemote.c:1826
#1  0x001c2b0d in PGXCNodeImplicitCommitPrepared (prepare_xid=924, 
commit_xid=925, gid=0xbfffef4f "T924", is_commit=1 '\001') at 
execRemote.c:1775
#2  0x0005845f in CommitTransaction () at xact.c:2013
#3  0x0005948f in CommitTransactionCommand () at xact.c:2746
#4  0x0029a6d7 in finish_xact_command () at postgres.c:2437
#5  0x002980d2 in exec_simple_query (query_string=0x103481c "commit;") 
at postgres.c:1070
#6  0x0029ccbb in PostgresMain (argc=4, argv=0x1002ff8, 
username=0x1002fc8 "masonsharp") at postgres.c:3766
#7  0x0025848c in BackendRun (port=0x7016f0) at postmaster.c:3607
#8  0x002577f3 in BackendStartup (port=0x7016f0) at postmaster.c:3216
#9  0x00254225 in ServerLoop () at postmaster.c:1445
#10 0x00253831 in PostmasterMain (argc=5, argv=0x7005a0) at 
postmaster.c:1098
#11 0x001cf261 in main (argc=5, argv=0x7005a0) at main.c:188

pgxc_handles looks ok though. It works ok in your environment?


>
>
>     I did the same test as before. I killed a data node after it
>     received a COMMIT PREPARED message.
>
>     I think we should be able to continue.
>
>     The good news is that I should not see partially committed data,
>     which I do not.
>
>     But if I try and manually commit it from a new connection to the
>     coordinator:
>
>     mds=# COMMIT PREPARED 'T1018';
>     ERROR:  Could not get GID data from GTM
>
>     Maybe GTM removed this info when the coordinator disconnected? (Or
>     maybe implicit transactions are only associated with a certain
>     connection?)
>
> Yes it has been removed when your Coordinator instance crashed.
>
>     I can see the transaction on one data node, but not the other.
>
>     Ideally we would come up with a scheme where if the coordinator
>     session does not notify GTM, we can somehow recover.  Maybe this
>     is my fault- I believe I advocated avoiding the extra work for
>     implicit 2PC in the name of performance. :-)
>
>     We can think about what to do in the short term, and how to handle
>     in the long term.
>
>     In the short term, your approach may be good enough once debugged,
>     since it is a relatively rare case.
>
>     Long term we could think about a thread that runs on GTM and wakes
>     up every 30 or 60 seconds or so (configurable), collects implicit
>     transactions from the nodes (extension to pg_prepared_xacts
>     required?) and if it sees that the XID does not have an associated
>     live connection, knows that something went awry.  It then sees if
>     it committed on any of the nodes. If not, rollback all, if it did
>     on at least one, commit on all. If one of the data nodes is down,
>     it won't do anything, perhaps log a warning. This would avoid user
>     intervention, and would be pretty cool. Some of this code you may
>     already have been working on for recovery and we could reuse here.
>
> This is a nice idea.
> It depends of course on one thing; if we decide to base the HA 
> features on a monitoring agent only or if XC should be able to run on 
> its own (or even allow both modes).

We can think about it... It could be separate from GTM, part of a 
monitoring process.

Mason

>
> -- 
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.