postgres-xc-developers Mailing List for Postgres-XC

I am sending this out to allow for feedback.

This patch adds support for INSERT SELECT (including "multi-step" queries).

This is done by utilizing COPY. We query the data nodes with the tuples
being sent to the Coordinator. These are then converted into COPY lines
and sent back down to the appropriate nodes. (Long term when we add
data node to data node communication, these can be sent directly).

We also optimize for the case when the SELECT is single-step, the
destination table is partitioned, the input column value comes from
the partition column of the source, and there are no limit or offset
clauses. In this case, we just pass down the INSERT SELECT to the data
nodes.

There is one kluge here in that I added a static variable for the
copy state. I did this to avoid having to move the CopyState definition
to a header file (for usage in execMain.c), in order to make merging
with vanilla PostgreSQL easier.

-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

In the document, we installed one coordinator and one data node into
one physical server to make best user of both coordinators and data
nodes, independent from the workload shared by coordinators and data
nodes.

It also make parameters simple to claim the sacalability against the
number of servers involved.   I don't thinks it's a good idea to
install loader and coordinator in the same server.   Loader's work is
simply application-oriented and should not be included as a part of
database performance.

As Mason mentioned, I'm very interested in the spec of servers, number
of servers used and configuration information what components (GTM,
coordinators and data nodes) you installed in what servers, as well as
network connections.

As you might have seen, we need gigabit network links between GTM,
Coordinators and Data Nodes.   I strongly recommend to use good L2
switch to reduce the network worklord too.

Kind Regards;
----------
Koichi Suzuki



2010/11/23 Mason Sharp <mas...@en...>:
> On 11/23/10 3:27 AM, xiong wang wrote:
>> Dears,
>>
>> I tested the postgres-xc DBT1 performance according to the published
>> document. But the result is worse than what the document declares. One
>> loader with one coordinator is much better. One loader with two
>> coordinators is much worse than the former. I don't know which way is
>> right. And I don't know the reason why the latter method is so worse
>> than the former.
>>
>>
> How much worse?
>
> How many physical servers are in each configuration? How is each server
> configured in each, with how many data nodes? What kind of network?
> Gigabit?
>
> Or was everything on one system? With virtual machines or without and
> just using different ports?
>
> Are there errors in the log file (connection limits hit)?
>
> Regards,
>
> Mason
>
>> You reply will be appreciated.
>>
>> Thanks.
>>
>> Best regards,
>>
>> Benny
>>
>> ------------------------------------------------------------------------------
>> Increase Visibility of Your 3D Game App&  Earn a Chance To Win $500!
>> Tap into the largest installed PC base&  get more eyes on your game by
>> optimizing for Intel(R) Graphics Technology. Get started today with the
>> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
>> http://p.sf.net/sfu/intelisp-dev2dev
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>
>
> --
> Mason Sharp
> EnterpriseDB Corporation
> The Enterprise Postgres Company
>
>
> This e-mail message (and any attachment) is intended for the use of
> the individual or entity to whom it is addressed. This message
> contains information from EnterpriseDB Corporation that may be
> privileged, confidential, or exempt from disclosure under applicable
> law. If you are not the intended recipient or authorized to receive
> this for the intended recipient, any use, dissemination, distribution,
> retention, archiving, or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify
> the sender immediately by reply e-mail and delete this message.
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

On 11/23/10 3:27 AM, xiong wang wrote:
> Dears,
>
> I tested the postgres-xc DBT1 performance according to the published
> document. But the result is worse than what the document declares. One
> loader with one coordinator is much better. One loader with two
> coordinators is much worse than the former. I don't know which way is
> right. And I don't know the reason why the latter method is so worse
> than the former.
>
>    
How much worse?

How many physical servers are in each configuration? How is each server 
configured in each, with how many data nodes? What kind of network? 
Gigabit?

Or was everything on one system? With virtual machines or without and 
just using different ports?

Are there errors in the log file (connection limits hit)?

Regards,

Mason

> You reply will be appreciated.
>
> Thanks.
>
> Best regards,
>
> Benny
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App&  Earn a Chance To Win $500!
> Tap into the largest installed PC base&  get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>    


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.

Dears,

I tested the postgres-xc DBT1 performance according to the published
document. But the result is worse than what the document declares. One
loader with one coordinator is much better. One loader with two
coordinators is much worse than the former. I don't know which way is
right. And I don't know the reason why the latter method is so worse
than the former.


You reply will be appreciated.

Thanks.

Best regards,

Benny

On 11/21/10 9:13 PM, Michael Paquier wrote:
>
> I made a couple of times the same tests and it looked to work all the 
> time.
> The logs on each node also looked correct.
>
I just committed, after doing a bug fix. The problem appeared to be that 
the list was getting
reassigned when an item is deleted in the foreach loop that depended on it.
I changed that to be more careful.

You guys probably just got lucky in your environment, but without the 
change it would
always crash under MacOS.

I also made a small change to check if the connection to GTM is NULL 
before rolling back.
(The GTM crash was causing a coordinator crash.)

Thanks,

Mason

> Regards,
>
> -- 
> Michael Paquier
> http://michaelpq.users.sourceforge.net
>


-- 
Mason Sharp
EnterpriseDB Corporation
The Enterprise Postgres Company


This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.