postgres-xc-developers Mailing List for Postgres-XC

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Amit/Andrei,
    Setting a primary node (the same node on all coordinators) seems to fix the problem. We have not had a hangs far. Appears to be humming smoothly now.

________________________________________

Venky Kandaswamy

Principal Engineer, Adchemy Inc.

925-200-7124

________________________________
From: Venky Kandaswamy
Sent: Thursday, April 11, 2013 8:09 AM
To: pos...@li...
Cc: pos...@li...
Subject: RE: [Postgres-xc-developers] PGXC hangs when run with concurrent inserts

Thanks Amit & Andrei.
There is only one update and 4 inserts - at total of 5 threads which are waiting on each other. Not sure if this is a typical deadlock situation.

These are all replicated tables and none of the datanodes are marked as primary. I will mark one as primary and see if the problem happens again.
Thanks for your insight.

________________________________________

Venky Kandaswamy

Principal Engineer, Adchemy Inc.

925-200-7124

________________________________
From: Andrei Martsinchyk [and...@gm...]
Sent: Thursday, April 11, 2013 2:01 AM
To: Amit Khandekar
Cc: Venky Kandaswamy; pos...@li...
Subject: Re: [Postgres-xc-developers] PGXC hangs when run with concurrent inserts

2013/4/11 Amit Khandekar <ami...@en...<mailto:ami...@en...>>

On 11 April 2013 13:35, Andrei Martsinchyk <and...@gm...<mailto:and...@gm...>> wrote:
I see the not granted tuple level locks on datanodes, they are requested by the "INSERT waiting" processes. Guess they are updating indexes.
It seems like these locks are not granted because  of exclusive lock held by "UPDATE waiting".

But it is not clear, what the update is faiting for?
I thnk there are multiple updates. That's why I think they might be waiting for each other, causing a deadlock possibly because there is no primary node.

I see only one update in the ps outputs.

2013/4/11 Amit Khandekar <ami...@en...<mailto:ami...@en...>>
Hi Venky,

Thanks for the details. Have you defined one of the data nodes as a primary node ? If no, we need to define one, because replicated table updates need that in order to avoid deadlocks. If you have already marked a node as a primary node, is the primary node one of the nodes on which the feature table is replicated on ? If no, you may have hit this bug :
http://sourceforge.net/tracker/index.php?func=detail&aid=3547808&group_id=311227&atid=1310232

Currently we hit this bug because the primary node is not table-specific, it should be implemented table-specific. For now you need to make sure one of the nodes on which the table is replicated is defined as a primary node.

On 11 April 2013 06:51, Venky Kandaswamy <ve...@ad...<mailto:ve...@ad...>> wrote:
We are processing inserts/updates using multiple threads. Here is the trace log of the actual statements that are hung. The scenario shows the statements on the coordinator and 2 datanodes. The scenario is similar across all the datanodes.
The same data updates did not cause Postgres 9.1.2 to hang. This could be related to an application problem, although we could not reproduce it on Postgres 9.1.2.

At a high level, there is an update on the 'feature' table that is holding an exclusive lock on the row. The inserts are inserting to another table that has a foreign key that references the row being locked by the update. Pid 7174 and 7179 are waiting to complete and they are also similar inserts. The only thing in common seems to be that the update is locking the feature row that is referenced in a foreign key in the other inserts. This should not cause a deadlock, I believe.

The question in my mind is whether pids 7181 and 7186 should have been granted exclusive access to a tuple while others were granted share access. This might cause a race condition.

This causes PGXC to hang. Obviously, the update is in turn waiting for something (which we cannot figure out from the logs) and therefore not committing the update.

[postgres@sv4-pgxc-db01 pgxc]$ ps -ef | grep adchemy1234
<COORDINATOR>
postgres  7169  7113  0 16:41 ?        00:00:02 postgres: adchemy adchemy1234 192.168.51.73(49186) INSERT
postgres  7170  7113  0 16:41 ?        00:00:02 postgres: adchemy adchemy1234 192.168.51.73(49187) INSERT
postgres  7171  7113  0 16:41 ?        00:00:02 postgres: adchemy adchemy1234 192.168.51.73(49188) UPDATE
postgres  7172  7113  0 16:41 ?        00:00:02 postgres: adchemy adchemy1234 192.168.51.73(49189) INSERT
postgres  7173  7113  0 16:41 ?        00:00:02 postgres: adchemy adchemy1234 192.168.51.73(49190) INSERT
<COORDINATOR>

<DATANODE1>
postgres  7174  7127  0 16:41 ?        00:00:01 postgres: adchemy adchemy1234 172.17.28.61(51909) idle in transaction
postgres  7175  7127  0 16:41 ?        00:00:01 postgres: adchemy adchemy1234 172.17.28.61(51910) INSERT waiting
postgres  7181  7127  0 16:41 ?        00:00:01 postgres: adchemy adchemy1234 172.17.28.61(51924) UPDATE waiting
postgres  7182  7127  0 16:41 ?        00:00:01 postgres: adchemy adchemy1234 172.17.28.61(51925) INSERT waiting
postgres  7183  7127  0 16:41 ?        00:00:01 postgres: adchemy adchemy1234 172.17.28.61(51926) INSERT waiting
<DATANODE1>

<DATANODE2>
postgres  7179  7140  0 16:41 ?        00:00:00 postgres: adchemy adchemy1234 172.17.28.61(48957) idle in transaction
postgres  7180  7140  0 16:41 ?        00:00:00 postgres: adchemy adchemy1234 172.17.28.61(48962) INSERT waiting
postgres  7184  7140  0 16:41 ?        00:00:00 postgres: adchemy adchemy1234 172.17.28.61(48970) INSERT waiting
postgres  7185  7140  0 16:41 ?        00:00:00 postgres: adchemy adchemy1234 172.17.28.61(48975) INSERT waiting
postgres  7186  7140  0 16:41 ?        00:00:00 postgres: adchemy adchemy1234 172.17.28.61(48980) UPDATE waiting
<DATANODE2>

-----LOGS-----  formatted %t %u %p

2013-04-10 16:42:16 PDT adchemy 7169 LOG:  execute S_1: BEGIN
2013-04-10 16:42:16 PDT adchemy 7169 LOG:  execute <unnamed>: select nextval ('hibernate_sequence')
2013-04-10 16:42:16 PDT adchemy 7169 LOG:  execute <unnamed>: insert into biods.product_feature (category_id, category_semid, created_ts, feature_id, feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id, prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
2013-04-10 16:42:16 PDT adchemy 7169 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.343-07', $4 = '42318', $5 = 'description', $6 = '46105', $7 = 'description,Give your riches the designer treatment with Mcms leather heritage wallet. The logo-stamped little number stores your essentials in luxe vintage style.', $8 = NULL, $9 = '46449', $10 = '7630015470685', $11 = '2013-04-10 15:02:42.343-07', $12 = '46455'

2013-04-10 16:42:16 PDT adchemy 7170 LOG:  execute S_1: BEGIN
2013-04-10 16:42:16 PDT adchemy 7170 LOG:  execute <unnamed>: select nextval ('hibernate_sequence')
2013-04-10 16:42:16 PDT adchemy 7170 LOG:  execute <unnamed>: insert into biods.product_feature (category_id, category_semid, created_ts, feature_id, feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id, prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
2013-04-10 16:42:16 PDT adchemy 7170 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:43.413-07', $4 = '42318', $5 = 'description', $6 = '46326', $7 = 'description,Rich leather is dressed up with a bold logo-stamped plaque in this utility chic wallet from Marc By Marc Jacobs.', $8 = NULL, $9 = '46438', $10 = '883936992041', $11 = '2013-04-10 15:02:43.413-07', $12 = '46445'

2013-04-10 16:42:15 PDT adchemy 7171 LOG:  execute S_1: BEGIN
2013-04-10 16:42:15 PDT adchemy 7171 LOG:  execute <unnamed>: select feature0_.feature_id as feature1_8_1_, feature0_.created_ts as created2_8_1_, feature0_.feature_name as feature3_8_1_, feature0_.feature_semid as feature4_8_1_, feature0_.modified_by as modified5_8_1_, feature0_.source_msg_ts as source6_8_1_, feature0_.updated_ts as updated7_8_1_, featureval1_.feature_id as feature9_8_3_, featureval1_.feature_value_id as feature1_14_3_, featureval1_.feature_value_id as feature1_14_0_, featureval1_.created_ts as created2_14_0_, featureval1_.feature_id as feature9_14_0_, featureval1_.feature_semid as feature3_14_0_, featureval1_.feature_value as feature4_14_0_, featureval1_.feature_value_semid as feature5_14_0_, featureval1_.modified_by as modified6_14_0_, featureval1_.source_msg_ts as source7_14_0_, featureval1_.updated_ts as updated8_14_0_ from biods.feature feature0_ left outer join biods.feature_value featureval1_ on feature0_.feature_id=featureval1_.feature_id where feature0_.feature_id=$1
2013-04-10 16:42:15 PDT adchemy 7171 DETAIL:  parameters: $1 = '42318'
2013-04-10 16:42:15 PDT adchemy 7171 LOG:  execute <unnamed>: update biods.feature set created_ts=$1, feature_name=$2, feature_semid=$3, modified_by=$4, source_msg_ts=$5, updated_ts=$6 where feature_id=$7
2013-04-10 16:42:15 PDT adchemy 7171 DETAIL:  parameters: $1 = '2013-04-10 15:02:34.706-07', $2 = 'description', $3 = 'description', $4 = NULL, $5 = '2013-04-10 15:02:43.576-07', $6 = '2013-04-10 15:02:43.573-07', $7 = '42318'

2013-04-10 16:42:17 PDT adchemy 7172 LOG:  execute S_1: BEGIN
2013-04-10 16:42:17 PDT adchemy 7172 LOG:  execute <unnamed>: select nextval ('hibernate_sequence')
2013-04-10 16:42:17 PDT adchemy 7172 LOG:  execute <unnamed>: insert into biods.product_feature (category_id, category_semid, created_ts, feature_id, feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id, prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
2013-04-10 16:42:17 PDT adchemy 7172 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.003-07', $4 = '42318', $5 = 'description', $6 = '44831', $7 = 'description,A chic logo-detailed cosmetic case for the contemporary girl from Tory Burch. Exclusive to Bloomingdales.', $8 = NULL, $9 = '46453', $10 = '885427179580', $11 = '2013-04-10 15:02:42.003-07', $12 = '46460'

2013-04-10 16:42:15 PDT adchemy 7173 LOG:  execute S_1: BEGIN
2013-04-10 16:42:15 PDT adchemy 7173 LOG:  execute <unnamed>: select nextval ('hibernate_sequence')
2013-04-10 16:42:15 PDT adchemy 7173 LOG:  execute <unnamed>: insert into biods.product_feature (category_id, category_semid, created_ts, feature_id, feature_semid, feature_value_id, feature_value_semid, modified_by, prd_id, prd_semid, updated_ts, prd_feature_id) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
2013-04-10 16:42:15 PDT adchemy 7173 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.674-07', $4 = '42318', $5 = 'description', $6 = '46154', $7 = 'description,Keep the essentials close with LeSportsacs crossbody bag in matte black nylon practical interior zip compartments make those daily errands a little bit easier.', $8 = NULL, $9 = '46425', $10 = '883681258669', $11 = '2013-04-10 15:02:42.674-07', $12 = '46435'

2013-04-10 16:42:15 PDT adchemy 7174 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:15 PDT adchemy 7174 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:15 PDT adchemy 7174 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.674-07', $4 = '42318', $5 = 'description', $6 = '46154', $7 = 'description,Keep the essentials close with LeSportsacs crossbody bag in matte black nylon practical interior zip compartments make those daily errands a little bit easier.', $8 = NULL, $9 = '46425', $10 = '883681258669', $11 = '2013-04-10 15:02:42.674-07', $12 = '46435'

2013-04-10 16:42:16 PDT adchemy 7175 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:16 PDT adchemy 7175 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:16 PDT adchemy 7175 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.343-07', $4 = '42318', $5 = 'description', $6 = '46105', $7 = 'description,Give your riches the designer treatment with Mcms leather heritage wallet. The logo-stamped little number stores your essentials in luxe vintage style.', $8 = NULL, $9 = '46449', $10 = '7630015470685', $11 = '2013-04-10 15:02:42.343-07', $12 = '46455'

2013-04-10 16:42:15 PDT adchemy 7179 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:15 PDT adchemy 7179 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:15 PDT adchemy 7179 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.674-07', $4 = '42318', $5 = 'description', $6 = '46154', $7 = 'description,Keep the essentials close with LeSportsacs crossbody bag in matte black nylon practical interior zip compartments make those daily errands a little bit easier.', $8 = NULL, $9 = '46425', $10 = '883681258669', $11 = '2013-04-10 15:02:42.674-07', $12 = '46435'

2013-04-10 16:42:16 PDT adchemy 7180 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:16 PDT adchemy 7180 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:16 PDT adchemy 7180 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.343-07', $4 = '42318', $5 = 'description', $6 = '46105', $7 = 'description,Give your riches the designer treatment with Mcms leather heritage wallet. The logo-stamped little number stores your essentials in luxe vintage style.', $8 = NULL, $9 = '46449', $10 = '7630015470685', $11 = '2013-04-10 15:02:42.343-07', $12 = '46455'

2013-04-10 16:42:15 PDT adchemy 7181 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:15 PDT adchemy 7181 LOG:  execute <unnamed>: UPDATE biods.feature SET feature_semid = $3, feature_name = $2, created_ts = $1, updated_ts = $6, source_msg_ts = $5, modified_by = $4 WHERE (feature_id = $7)
2013-04-10 16:42:15 PDT adchemy 7181 DETAIL:  parameters: $1 = '2013-04-10 15:02:34.706-07', $2 = 'description', $3 = 'description', $4 = NULL, $5 = '2013-04-10 15:02:43.576-07', $6 = '2013-04-10 15:02:43.573-07', $7 = '42318'

2013-04-10 16:42:17 PDT adchemy 7182 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:17 PDT adchemy 7182 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:17 PDT adchemy 7182 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.003-07', $4 = '42318', $5 = 'description', $6 = '44831', $7 = 'description,A chic logo-detailed cosmetic case for the contemporary girl from Tory Burch. Exclusive to Bloomingdales.', $8 = NULL, $9 = '46453', $10 = '885427179580', $11 = '2013-04-10 15:02:42.003-07', $12 = '46460'

2013-04-10 16:42:16 PDT adchemy 7183 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:16 PDT adchemy 7183 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:16 PDT adchemy 7183 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:43.413-07', $4 = '42318', $5 = 'description', $6 = '46326', $7 = 'description,Rich leather is dressed up with a bold logo-stamped plaque in this utility chic wallet from Marc By Marc Jacobs.', $8 = NULL, $9 = '46438', $10 = '883936992041', $11 = '2013-04-10 15:02:43.413-07', $12 = '46445'

2013-04-10 16:42:17 PDT adchemy 7184 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:17 PDT adchemy 7184 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:17 PDT adchemy 7184 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:42.003-07', $4 = '42318', $5 = 'description', $6 = '44831', $7 = 'description,A chic logo-detailed cosmetic case for the contemporary girl from Tory Burch. Exclusive to Bloomingdales.', $8 = NULL, $9 = '46453', $10 = '885427179580', $11 = '2013-04-10 15:02:42.003-07', $12 = '46460'

2013-04-10 16:42:16 PDT adchemy 7185 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:16 PDT adchemy 7185 LOG:  execute <unnamed>: INSERT INTO biods.product_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
2013-04-10 16:42:16 PDT adchemy 7185 DETAIL:  parameters: $1 = '42302', $2 = 'Handbags', $3 = '2013-04-10 15:02:43.413-07', $4 = '42318', $5 = 'description', $6 = '46326', $7 = 'description,Rich leather is dressed up with a bold logo-stamped plaque in this utility chic wallet from Marc By Marc Jacobs.', $8 = NULL, $9 = '46438', $10 = '883936992041', $11 = '2013-04-10 15:02:43.413-07', $12 = '46445'

2013-04-10 16:42:15 PDT adchemy 7186 LOG:  statement: START TRANSACTION ISOLATION LEVEL read committed READ WRITE
2013-04-10 16:42:15 PDT adchemy 7186 LOG:  execute <unnamed>: UPDATE biods.feature SET feature_semid = $3, feature_name = $2, created_ts = $1, updated_ts = $6, source_msg_ts = $5, modified_by = $4 WHERE (feature_id = $7)
2013-04-10 16:42:15 PDT adchemy 7186 DETAIL:  parameters: $1 = '2013-04-10 15:02:34.706-07', $2 = 'description', $3 = 'description', $4 = NULL, $5 = '2013-04-10 15:02:43.576-07', $6 = '2013-04-10 15:02:43.573-07', $7 = '42318'

LOCKS ON COORDINATOR:

[venky@sv4-pgxc-db01 ~]$ /usr/local/pgsql/bin/psql -p 5432 -U postgres -d adchemy1234 -c "SELECT pid, relname, locktype, mode, granted from pg_locks, pg_class where relation=oid and relname not like 'pg_%' order by mode;"
 pid  |      relname       | locktype |       mode       | granted
------+--------------------+----------+------------------+---------
 7169 | hibernate_sequence | relation | AccessShareLock  | t
 7173 | hibernate_sequence | relation | AccessShareLock  | t
 7172 | hibernate_sequence | relation | AccessShareLock  | t
 7171 | feature_value      | relation | AccessShareLock  | t
 7171 | feature            | relation | AccessShareLock  | t
 7170 | hibernate_sequence | relation | AccessShareLock  | t
 7171 | feature            | relation | RowExclusiveLock | t
 7172 | product_feature    | relation | RowExclusiveLock | t
 7170 | product_feature    | relation | RowExclusiveLock | t
 7173 | product_feature    | relation | RowExclusiveLock | t
 7169 | product_feature    | relation | RowExclusiveLock | t
(11 rows)

LOCKS ON DATANODE1:

[venky@sv4-pgxc-db01 ~]$ /usr/local/pgsql/bin/psql -p 5433 -U postgres -d adchemy1234 -c "SELECT pid, relname, locktype, mode, granted from pg_locks, pg_class where relation=oid and relname not like 'pg_%' order by mode;"
 pid  |        relname        | locktype |           mode           | granted
------+-----------------------+----------+--------------------------+---------
 7174 | prd_id                | relation | AccessShareLock          | t
 7182 | feature_id            | relation | AccessShareLock          | t
 7174 | feature_value_id      | relation | AccessShareLock          | t
 7183 | feature_id            | relation | AccessShareLock          | t
 7174 | feature_id            | relation | AccessShareLock          | t
 7175 | feature_id            | relation | AccessShareLock          | t
 7181 | feature               | tuple    | ExclusiveLock            | t
 7181 | feature_semid         | relation | RowExclusiveLock         | t
 7181 | feature_id            | relation | RowExclusiveLock         | t
 7181 | feature               | relation | RowExclusiveLock         | t
 7175 | cat_prd_feature_semid | relation | RowExclusiveLock         | t
 7183 | cat_prd_feature_semid | relation | RowExclusiveLock         | t
 7183 | prd_feature_id        | relation | RowExclusiveLock         | t
 7183 | product_feature       | relation | RowExclusiveLock         | t
 7182 | cat_prd_feature_semid | relation | RowExclusiveLock         | t
 7182 | prd_feature_id        | relation | RowExclusiveLock         | t
 7182 | product_feature       | relation | RowExclusiveLock         | t
 7175 | prd_feature_id        | relation | RowExclusiveLock         | t
 7175 | product_feature       | relation | RowExclusiveLock         | t
 7174 | product_feature       | relation | RowExclusiveLock         | t
 7206 | feature_semid         | relation | RowExclusiveLock         | t
 7206 | feature_id            | relation | RowExclusiveLock         | t
 7174 | product               | relation | RowShareLock             | t
 7182 | feature               | relation | RowShareLock             | t
 7174 | feature_value         | relation | RowShareLock             | t
 7183 | category              | relation | RowShareLock             | t
 7174 | feature               | relation | RowShareLock             | t
 7174 | category              | relation | RowShareLock             | t
 7175 | category              | relation | RowShareLock             | t
 7175 | feature               | relation | RowShareLock             | t
 7183 | feature               | relation | RowShareLock             | t
 7182 | category              | relation | RowShareLock             | t
 7182 | feature               | tuple    | ShareLock                | f
 7175 | feature               | tuple    | ShareLock                | f
 7183 | feature               | tuple    | ShareLock                | f
 7206 | feature               | relation | ShareUpdateExclusiveLock | t

LOCKS ON DATANODE2:

[venky@sv4-pgxc-db01 ~]$ /usr/local/pgsql/bin/psql -p 5434 -U postgres -d adchemy1234 -c "SELECT pid, relname, locktype, mode, granted from pg_locks, pg_class where relation=oid and relname not like 'pg_%' order by mode;"
 pid  |        relname        | locktype |           mode           | granted
------+-----------------------+----------+--------------------------+---------
 7185 | feature_id            | relation | AccessShareLock          | t
 7179 | feature_value_id      | relation | AccessShareLock          | t
 7179 | prd_id                | relation | AccessShareLock          | t
 7184 | feature_id            | relation | AccessShareLock          | t
 7180 | feature_id            | relation | AccessShareLock          | t
 7179 | feature_id            | relation | AccessShareLock          | t
 7186 | feature               | tuple    | ExclusiveLock            | t
 7184 | prd_feature_id        | relation | RowExclusiveLock         | t
 7184 | product_feature       | relation | RowExclusiveLock         | t
 7186 | feature_semid         | relation | RowExclusiveLock         | t
 7186 | feature_id            | relation | RowExclusiveLock         | t
 7186 | feature               | relation | RowExclusiveLock         | t
 7185 | cat_prd_feature_semid | relation | RowExclusiveLock         | t
 7185 | prd_feature_id        | relation | RowExclusiveLock         | t
 7185 | product_feature       | relation | RowExclusiveLock         | t
 7184 | cat_prd_feature_semid | relation | RowExclusiveLock         | t
 7180 | cat_prd_feature_semid | relation | RowExclusiveLock         | t
 7180 | prd_feature_id        | relation | RowExclusiveLock         | t
 7180 | product_feature       | relation | RowExclusiveLock         | t
 7179 | product_feature       | relation | RowExclusiveLock         | t
 7202 | feature_semid         | relation | RowExclusiveLock         | t
 7202 | feature_id            | relation | RowExclusiveLock         | t
 7179 | product               | relation | RowShareLock             | t
 7184 | feature               | relation | RowShareLock             | t
 7179 | feature_value         | relation | RowShareLock             | t
 7185 | category              | relation | RowShareLock             | t
 7179 | feature               | relation | RowShareLock             | t
 7179 | category              | relation | RowShareLock             | t
 7180 | feature               | relation | RowShareLock             | t
 7180 | category              | relation | RowShareLock             | t
 7185 | feature               | relation | RowShareLock             | t
 7184 | category              | relation | RowShareLock             | t
 7185 | feature               | tuple    | ShareLock                | f
 7180 | feature               | tuple    | ShareLock                | f
 7184 | feature               | tuple    | ShareLock                | f
 7202 | feature               | relation | ShareUpdateExclusiveLock | t
(36 rows)

________________________________________

Venky Kandaswamy

Principal Engineer, Adchemy Inc.

925-200-7124

________________________________
From: Koichi Suzuki [koi...@gm...<mailto:koi...@gm...>]
Sent: Monday, April 08, 2013 10:41 PM
To: Amit Khandekar
Cc: Venky Kandaswamy; pos...@li...<mailto:pos...@li...>
Subject: Re: [Postgres-xc-developers] PGXC hangs when run with concurrent inserts

Because insert is being done in parallel, I'm afraid there could be a possibility that we have internal lock conflicts, which should not happen.

Regards;
----------
Koichi Suzuki

2013/4/9 Amit Khandekar <ami...@en...<mailto:ami...@en...>>

On 9 April 2013 06:46, Venky Kandaswamy <ve...@ad...<mailto:ve...@ad...>> wrote:
All,
   We have been running into a hang issue on our app that appears to be related to PGXC. Our app processes messages from RabbitMQ and inserts/updates tables. We run 5 concurrent threads. The incoming queues are replicated, one feeding Postgres 9.1 and the other feeding PGXC (current git master). PGXC is hanging on inserts after processing a few transactions. It does not appear to be related to the actual data itself. IT looks like all the sessions are waiting for something. There is no  information on locks available from pg_locks.

Since most of the operations are inserts, it does not look like it is due to locks, unless something has acquired table locks. But just to rule out that possibility, it would be better if you check  pg_locks on the datanodes, if you have checked it only on coordinator so far.

   An strace simply says recfrom(10.

   The are no errors in the logs from gtm, coordinator or datanodes.

   The tables have referential integrity and use a shared sequence to get the next id. Is it possible that something is going on with the logic to retrieve sequence numbers?  The tables are all replicated.

   Unfortunately, we have not been able to reproduce a reliable test case.

   [postgres@gnode0 pgxc]$ /usr/local/pgsql/bin/psql -p 5433 -U postgres -d postgres -c 'select * from pg_catalog.pg_stat_activity;'
 datid |   datname    |  pid  | usesysid | usename  | application_name |  client_addr   | client_hostname | client_port |         backend_start         |          xact_start           |          query_start          |         state_change          | waiting |        state        |
                                                                                                            query
-------+--------------+-------+----------+----------+------------------+----------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+---------------------+--------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 12893 | postgres     | 22330 |       10 | postgres | pgxc             | 192.168.53.109 |                 |       47025 | 2013-03-31 21:42:16.724845-07 |                               | 2013-04-08 15:43:52.313325-07 | 2013-04-08 15:26:11.444754-07 | f       | idle                | COMMIT PREPARED 'T1
32273'
 16393 | master       |  4267 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       54961 | 2013-04-08 15:24:28.668023-07 |                               | 2013-04-08 15:33:17.586836-07 | 2013-04-08 15:33:17.587942-07 | f       | idle                | SELECT count(*) FRO
M ONLY bicommon.account_datasource WHERE true
 16395 | adchemy10013 |  4363 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       55084 | 2013-04-08 15:28:48.822939-07 |                               | 2013-04-08 15:50:21.650727-07 | 2013-04-08 15:50:07.916753-07 | f       | idle                | SELECT prd_id, prd_
semid, prd_name, prd_line, prd_model, prd_brand, prd_image_url, prd_dest_url, created_ts, updated_ts, source_msg_ts, modified_by FROM biods.product
 16393 | master       |  4486 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       55246 | 2013-04-08 15:33:21.019388-07 |                               | 2013-04-08 15:43:51.321376-07 | 2013-04-08 15:43:51.322675-07 | f       | idle                | SET SESSION AUTHORI
ZATION DEFAULT;RESET ALL;
 16393 | master       |  4781 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       55515 | 2013-04-08 15:42:42.122785-07 |                               | 2013-04-08 17:02:21.023713-07 | 2013-04-08 17:02:20.804751-07 | f       | idle                | SET SESSION AUTHORI
ZATION DEFAULT;RESET ALL;
 16393 | master       |  4787 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       55521 | 2013-04-08 15:42:42.142662-07 |                               | 2013-04-08 16:17:19.26364-07  | 2013-04-08 16:17:19.126163-07 | f       | idle                | SET SESSION AUTHORI
ZATION DEFAULT;RESET ALL;
 16393 | master       |  4792 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       55526 | 2013-04-08 15:42:42.159009-07 |                               | 2013-04-08 15:45:11.915026-07 | 2013-04-08 15:45:11.886392-07 | f       | idle                | SET SESSION AUTHORI
ZATION DEFAULT;RESET ALL;
 16393 | master       |  4799 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       55533 | 2013-04-08 15:42:42.678387-07 |                               | 2013-04-08 17:02:21.195332-07 | 2013-04-08 17:02:20.805074-07 | f       | idle                | SET SESSION AUTHORI
ZATION DEFAULT;RESET ALL;
 16393 | master       |  4804 |    16392 | xcadmin  | pgxc             | 192.168.53.109 |                 |       55538 | 2013-04-08 15:42:42.694802-07 |                               | 2013-04-08 15:45:11.904619-07 | 2013-04-08 15:45:11.888493-07 | f       | idle                | SET SESSION AUTHORI
ZATION DEFAULT;RESET ALL;
 16395 | adchemy10013 |  4977 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       55732 | 2013-04-08 15:47:34.901175-07 | 2013-04-08 15:48:08.345331-07 | 2013-04-08 15:48:08.528818-07 | 2013-04-08 15:48:08.410815-07 | f       | idle in transaction | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  4979 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       55734 | 2013-04-08 15:47:35.042778-07 | 2013-04-08 15:48:16.384763-07 | 2013-04-08 15:48:16.506899-07 | 2013-04-08 15:48:16.388503-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  4985 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       55740 | 2013-04-08 15:47:35.235945-07<tel:35.235945-07> | 2013-04-08 15:48:14.38895-07  | 2013-04-08 15:48:14.445351-07 | 2013-04-08 15:48:14.446752-07 | t       | active              | UPDATE biods.featur
e SET feature_semid = $3, feature_name = $2, created_ts = $1, updated_ts = $6, source_msg_ts = $5, modified_by = $4 WHERE (feature_id = $7)
 16395 | adchemy10013 |  4986 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       55741 | 2013-04-08 15:47:35.238843-07<tel:35.238843-07> | 2013-04-08 15:48:18.201043-07 | 2013-04-08 15:48:18.273204-07 | 2013-04-08 15:48:18.205647-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  4998 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       55753 | 2013-04-08 15:47:35.910309-07 | 2013-04-08 15:48:08.412038-07 | 2013-04-08 15:48:08.566945-07 | 2013-04-08 15:48:08.415026-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  6340 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57002 | 2013-04-08 16:31:44.414804-07 | 2013-04-08 16:31:50.293828-07 | 2013-04-08 16:31:50.433988-07 | 2013-04-08 16:31:50.297752-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  6341 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57003 | 2013-04-08 16:31:44.418356-07 | 2013-04-08 16:31:49.450704-07 | 2013-04-08 16:31:49.599946-07 | 2013-04-08 16:31:49.45562-07  | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  6348 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57010 | 2013-04-08 16:31:45.065767-07<tel:45.065767-07> | 2013-04-08 16:31:50.699979-07 | 2013-04-08 16:31:50.817425-07 | 2013-04-08 16:31:50.704669-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  6349 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57011 | 2013-04-08 16:31:45.06926-07  | 2013-04-08 16:31:51.528207-07 | 2013-04-08 16:31:51.582036-07 | 2013-04-08 16:31:51.532618-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  6350 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57012 | 2013-04-08 16:31:45.072711-07<tel:45.072711-07> | 2013-04-08 16:31:50.085336-07 | 2013-04-08 16:31:50.223221-07 | 2013-04-08 16:31:50.088908-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7269 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57774 | 2013-04-08 16:57:15.563006-07 | 2013-04-08 16:57:21.849156-07 | 2013-04-08 16:57:21.978984-07 | 2013-04-08 16:57:21.853289-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7271 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57776 | 2013-04-08 16:57:15.63199-07  | 2013-04-08 16:57:16.575535-07 | 2013-04-08 16:57:17.00605-07  | 2013-04-08 16:57:17.007747-07 | t       | active              | INSERT INTO biods.f
eature_value (feature_value_id, feature_value_semid, feature_value, feature_semid, feature_id, created_ts, updated_ts, source_msg_ts, modified_by) VALUES ($9, $5, $4, $3, $2, $1, $8, $7, $6)
 16395 | adchemy10013 |  7283 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57788 | 2013-04-08 16:57:16.292702-07 | 2013-04-08 16:57:21.849125-07 | 2013-04-08 16:57:21.978824-07 | 2013-04-08 16:57:21.853251-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7284 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57789 | 2013-04-08 16:57:16.295879-07 | 2013-04-08 16:57:24.233166-07 | 2013-04-08 16:57:24.321938-07 | 2013-04-08 16:57:24.237514-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7285 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57790 | 2013-04-08 16:57:16.299271-07 | 2013-04-08 16:57:22.119868-07 | 2013-04-08 16:57:22.197213-07 | 2013-04-08 16:57:22.128357-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7465 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57954 | 2013-04-08 17:01:54.750113-07 | 2013-04-08 17:02:00.17336-07  | 2013-04-08 17:02:00.320469-07 | 2013-04-08 17:02:00.177758-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7466 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57955 | 2013-04-08 17:01:54.753559-07 | 2013-04-08 17:01:59.49003-07  | 2013-04-08 17:01:59.602925-07 | 2013-04-08 17:01:59.493732-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7467 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57956 | 2013-04-08 17:01:54.75699-07  | 2013-04-08 17:01:58.262083-07 | 2013-04-08 17:01:58.349452-07 | 2013-04-08 17:01:58.350822-07 | t       | active              | INSERT INTO biods.f
eature_value (feature_value_id, feature_value_semid, feature_value, feature_semid, feature_id, created_ts, updated_ts, source_msg_ts, modified_by) VALUES ($9, $5, $4, $3, $2, $1, $8, $7, $6)
 16395 | adchemy10013 |  7473 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57963 | 2013-04-08 17:01:55.49134-07  | 2013-04-08 17:02:00.313138-07 | 2013-04-08 17:02:00.420405-07 | 2013-04-08 17:02:00.318887-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 16395 | adchemy10013 |  7474 |    17361 | adchemy  | pgxc             | 192.168.53.109 |                 |       57964 | 2013-04-08 17:01:55.494777-07 | 2013-04-08 17:02:00.514142-07 | 2013-04-08 17:02:00.577239-07 | 2013-04-08 17:02:00.519572-07 | t       | active              | INSERT INTO biods.p
roduct_feature (prd_feature_id, category_id, prd_id, feature_id, feature_value_id, category_semid, prd_semid, feature_semid, feature_value_semid, created_ts, updated_ts, modified_by) VALUES ($12, $1, $9, $4, $6, $2, $10, $5, $7, $3, $11, $8)
 12893 | postgres     |  8517 |       10 | postgres | psql             |                |                 |          -1 | 2013-04-08 17:35:28.217934-07 |                               | 2013-04-08 17:35:28.220366-07 | 2013-04-08 17:35:28.220369-07 | f       | active              | select * from pg_ca
talog.pg_stat_activity;
(30 rows)

________________________________________

Venky Kandaswamy

Principal Engineer, Adchemy Inc.

925-200-7124<tel:925-200-7124>

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

--
Andrei Martsinchyk

StormDB - http://www.stormdb.com<http://www.stormdb.com/>
The Database Cloud

--
Andrei Martsinchyk

StormDB - http://www.stormdb.com<http://www.stormdb.com/>
The Database Cloud

2010	Jan	Feb	Mar	Apr (10)	May (17)	Jun (3)	Jul	Aug	Sep (8)	Oct (18)	Nov (51)	Dec (74)
2011	Jan (47)	Feb (44)	Mar (44)	Apr (102)	May (35)	Jun (25)	Jul (56)	Aug (69)	Sep (32)	Oct (37)	Nov (31)	Dec (16)
2012	Jan (34)	Feb (127)	Mar (218)	Apr (252)	May (80)	Jun (137)	Jul (205)	Aug (159)	Sep (35)	Oct (50)	Nov (82)	Dec (52)
2013	Jan (107)	Feb (159)	Mar (118)	Apr (163)	May (151)	Jun (89)	Jul (106)	Aug (177)	Sep (49)	Oct (63)	Nov (46)	Dec (7)
2014	Jan (65)	Feb (128)	Mar (40)	Apr (11)	May (4)	Jun (8)	Jul (16)	Aug (11)	Sep (4)	Oct (1)	Nov (5)	Dec (16)
2015	Jan (5)	Feb	Mar (2)	Apr (5)	May (4)	Jun (12)	Jul	Aug	Sep	Oct	Nov	Dec (4)
2019	Jan	Feb	Mar	Apr	May	Jun	Jul (2)	Aug	Sep	Oct	Nov	Dec

S	M	T	W	T	F	S
	1 (14)	2	3 (4)	4 (12)	5 (14)	6
7 (1)	8 (7)	9 (10)	10 (7)	11 (8)	12 (6)	13
14 (1)	15 (3)	16 (1)	17 (8)	18 (11)	19 (3)	20
21 (2)	22 (9)	23 (2)	24 (14)	25 (13)	26 (1)	27
28	29 (1)	30 (11)

postgres-xc-developers Mailing List for Postgres-XC

postgres-xc-developers — Postgres-XC hackers and developers