postgres-xc-developers Mailing List for Postgres-XC (Page 4)

PFA revised patch

On Fri, Aug 9, 2013 at 11:32 AM, Masataka Saito <pg...@gm...> wrote:
> I think "order by" in runInfinityTests is unnecessary.
>
> On Thu, Aug 8, 2013 at 6:33 PM, Abbas Butt <abb...@en...> wrote:
>> Hi,
>> PFA path to fix time-stamp tests needing order by
>>
>> --
>> Abbas
>> Architect
>>
>> Ph: 92.334.5100153
>> Skype ID: gabbasb
>> www.enterprisedb.com
>>
>> Follow us on Twitter
>> @EnterpriseDB
>>
>> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
>>
>> ------------------------------------------------------------------------------
>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> It's a free troubleshooting tool designed for production.
>> Get down to code-level detail for bottlenecks, with <2% overhead.
>> Download for free and get started troubleshooting in minutes.
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>



-- 
--
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

On Fri, May 24, 2013 at 1:30 AM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi All,
> While I hear Robert's presentation on query planning go wrong, I heard
> (first time) about MergeAppend plan. It looks like, we have something
> better to do with the way we are pushing the ORDER BY clauses and doing
> merge at the coordinator. Right now we are using Sort node and in ExecSort
> we start merging the rows. Instead, I think we should be using MergeAppend
> node.
>
>
I generally like this idea (though I saw below you did not get performance
numbers as expected). If we do this, it would also make sense to rework
RemoteQuery to use Append nodes for combining unsorted results from the
datanodes. But we will need some kind of parallelism at that node level so
that queries can be started and executed on all the child nodes at the same
time. AFAIK the current Append node executes serially i.e. it will move to
the next node only after finishing one subplan.

Thanks,
Pavan

-- 
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee

On Thu, Aug 8, 2013 at 9:41 PM, Abbas Butt <abb...@en...> wrote:
> Hi,
> PFA patch to fix an XC limitation that update of a replicated table
> created with oids cannot be based on oids.

When did you change distribution rule to REPLICATION from ROUNDROBIN?
I lost your patch.

> i.e. this update supposed to fail
> update rep_table_with_oids set colum = $1 where oid=$2
> saying
> ERROR : Write to replicated table returned different results from the Datanodes
>
> The reason is that we would select oid from the primary datanode and
> that particular oid will not be there on the rest of the datanodes.
> The patch comments explain the complete scenario in detail.
>
> With this patch the JDBC regression runs without any failures or
> errors on my machine.
> Strangely I did not have to change any thing on the server side.

What do you experience the strangeness of?
Your patch eliminated "oid" column from the selection, therefore,
UPDATE query does not contain "oid" anymore. I think it is
straightforward.

Regards.

>
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

I hesitated to accept.

Because original testCopyOutByRow tests COPY from a table but our one
tests COPY from a subquery. But I found that it tests to issue COPY
query and to receive the result. The target of COPY is not important.

Hence, I think this patch is applicable.

On Thu, Aug 8, 2013 at 8:06 PM, Abbas Butt <abb...@en...> wrote:
> Hi,
> PFA patch to fix copy tests needing order by.
>
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

I think "order by" in runInfinityTests is unnecessary.

On Thu, Aug 8, 2013 at 6:33 PM, Abbas Butt <abb...@en...> wrote:
> Hi,
> PFA path to fix time-stamp tests needing order by
>
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

My comment is same as the one for 10_1_date. patch.

On Thu, Aug 8, 2013 at 6:32 PM, Abbas Butt <abb...@en...> wrote:
> Hi,
> PFA patch to fix time tests needing order by.
>
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

It will work. But it took a bit time to understand that
selectSQL("testdate order by i", "dt")) is expanded to "SELECT dt FROM
testdate order by i" because the first argument name of selectSQL is
"table".
You'd better use selectSQL (String table, String columns, String
where, String other) rather than selectSQL(String table, String
columns).

On Thu, Aug 8, 2013 at 6:31 PM, Abbas Butt <abb...@en...> wrote:
> Hi,
> PFA patch to fix date tests needing order by.
>
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

No problem.

On Thu, Aug 8, 2013 at 6:31 PM, Abbas Butt <abb...@en...> wrote:
> Hi,
> PFA patch to fix integer tests needing order by.
>
> --
> Abbas
> Architect
>
> Ph: 92.334.5100153
> Skype ID: gabbasb
> www.enterprisedb.com
>
> Follow us on Twitter
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers and more
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>

Hi,
PFA patch to fix an XC limitation that update of a replicated table
created with oids cannot be based on oids.
i.e. this update supposed to fail
update rep_table_with_oids set colum = $1 where oid=$2
saying
ERROR : Write to replicated table returned different results from the Datanodes

The reason is that we would select oid from the primary datanode and
that particular oid will not be there on the rest of the datanodes.
The patch comments explain the complete scenario in detail.

With this patch the JDBC regression runs without any failures or
errors on my machine.
Strangely I did not have to change any thing on the server side.

--
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

Hi,
PFA patch to fix copy tests needing order by.

--
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

Hi,
PFA path to fix time-stamp tests needing order by

--
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

Hi,
PFA patch to fix time tests needing order by.

--
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

Hi,
PFA patch to fix date tests needing order by.

--
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

Hi,
PFA patch to fix integer tests needing order by.

--
Abbas
Architect

Ph: 92.334.5100153
Skype ID: gabbasb
www.enterprisedb.com

Follow us on Twitter
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars, whitepapers and more

On Thu, Aug 8, 2013 at 3:42 PM, Koichi Suzuki <koi...@gm...> wrote:
> Hello;
>
> Because I had to change my e-mail account and I happened to lose old e-mail
> history, I'm posting this as a separate thread.
>
> It was favored to change BARRIER elog message level to DEBUG1.   Here's a
> patch.  Considering BARRIER is not issued frequently, this will be okay.
LGTM. Thanks.
-- 
Michael

Yes, that's a possibility. Although, we need a robust executor interface,
so that this separation would work without any problem. But, unfortunately,
current RemoteQuery executor is fragile.


On Thu, Aug 8, 2013 at 11:41 AM, Koichi Suzuki <koi...@gm...>wrote:

> Can it be optional so that planner can choose if ExecInitRemoteQuery just
> issue a query or wait for the first row to come?  Then it is a planner
> issue which to choose.
>
>
> 2013/8/7 Ashutosh Bapat <ash...@en...>
>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 1:57 PM, Koichi Suzuki <koi...@gm...>wrote:
>>
>>> Hi;
>>>
>>>
>>> 2013/8/6 Ashutosh Bapat <ash...@en...>
>>>
>>>> Hi All,
>>>> I worked on this (patch attached) but I didn't see any performance
>>>> improvement. In fact, to my surprise the performance actually dropped.
>>>>
>>>> On a true cluster, with 5 coordinators and 5 datanodes, I saw these
>>>> numbers with 1000000 rows in a table with two integer columns. The times
>>>> are average times required for 100 runs of the same query, returning whole
>>>> rows ordered by one of the columns.
>>>> No ORDER BY push-down - 1640 ms
>>>> ORDER BY push-down using Sort plan - 1100 ms (this is the current
>>>> implementation)
>>>> ORDER BY Push-down using MergeAppend plan - 1170 ms
>>>>
>>>> The assumption behind trying this out was explained in my earlier mail
>>>> in this thread. But it looks like we do not get any improvement.
>>>>
>>>> One of the possible reasons, I think, is involvement of multiple
>>>> RemoteQuery nodes in MergeAppend plan. While executing MergeAppend plan,
>>>> the ExecProcNode() is called on the underlying node, one after the other.
>>>> For every RemoteQuery node in MergeAppend plan it waits till that node
>>>> emits a row. This means that we do not execute the queries on datanodes in
>>>> parallel when executing through MergeAppend plan unlike Sort plan where
>>>> there is only one RemoteQuery node involved.
>>>>
>>>> In order to get full advantage of MergeAppend plan (and keep the code
>>>> clean), I think we need to query the datanodes in parallel somewhere
>>>> outside ExecProcNode(). Thoughts? Suggestions?
>>>>
>>>
>>> Can we divide MergeAppend execution into two stage, one is just to issue
>>> a query and then begin to receive rows.   Because this is a change in the
>>> core, we should submit the idea, the usecase and the code to PG community.
>>>
>>>
>> This can be done if we move "issueing query" inside ExecInitRemoteQuery,
>> but I am not sure if that can be applied in all the cases. This wouldn't
>> require any change in PG.
>>
>>
>>> Regards;
>>> ---
>>> Koichi Suzuki
>>>
>>>
>>>>
>>>> On Fri, May 24, 2013 at 1:30 AM, Ashutosh Bapat <
>>>> ash...@en...> wrote:
>>>>
>>>>> Hi All,
>>>>> While I hear Robert's presentation on query planning go wrong, I heard
>>>>> (first time) about MergeAppend plan. It looks like, we have something
>>>>> better to do with the way we are pushing the ORDER BY clauses and doing
>>>>> merge at the coordinator. Right now we are using Sort node and in ExecSort
>>>>> we start merging the rows. Instead, I think we should be using MergeAppend
>>>>> node. The only bad thing about this approach is MergeAppend node expects
>>>>> separate plans for each run of the sorted data. To do this, we need to
>>>>> replicate the RemoteQuery node as many times as there are nodes and put
>>>>> them in the list to the MergeAppend plan. I think that's going to solve the
>>>>> problem with materialisation (see mail thread "Using remote sorting for
>>>>> merge-join") and thus improve performance. It also takes away the need to
>>>>> have xc_node_id in the tupleslot of tuplestore. Obviously, simplifying the
>>>>> code.
>>>>>
>>>>> Any comments?
>>>>>
>>>>> Created 3613815
>>>>> <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232>and
>>>>> assigned to me.
>>>>> <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232>
>>>>>
>>>>> On Mon, Dec 3, 2012 at 3:44 AM, Ashutosh Bapat <
>>>>> ash...@en...> wrote:
>>>>>
>>>>>> On Fri, Nov 30, 2012 at 9:37 AM, Michael Paquier
>>>>>> <mic...@gm...> wrote:
>>>>>> > Hi Ashutosh,
>>>>>> >
>>>>>> > Thanks for the patch.
>>>>>> > I found for the time being the following problems:
>>>>>> > 1) The patch has some whitespaces
>>>>>> > /home/michael/download/xc_sort.patch:16: trailing whitespace.
>>>>>> >
>>>>>> > /home/michael/download/xc_sort.patch:34: trailing whitespace.
>>>>>> >
>>>>>> > /home/michael/download/xc_sort.patch:307: trailing whitespace.
>>>>>> >         return local_plan_left;
>>>>>> > /home/michael/download/xc_sort.patch:356: trailing whitespace.
>>>>>> >
>>>>>> > /home/michael/download/xc_sort.patch:360: trailing whitespace.
>>>>>> >
>>>>>>
>>>>>> Fixed.
>>>>>>
>>>>>> > error: patch failed: src/include/pgxc/execRemote.h:95
>>>>>> > 2) the patch does not apply cleanly even by using "patch -p1"-like
>>>>>> commands
>>>>>>
>>>>>> When I sent the patch (more than a week before you replied) it was
>>>>>> based on the HEAD, but then there were some commits which created the
>>>>>> conflicts. The one attached should apply.
>>>>>>
>>>>>> > 3) No documentation about the new option enable_remotesort.
>>>>>> > 4) enable_remotesort is not listed in postgresql.conf.
>>>>>> > Please provide both documentation and reference in
>>>>>> postgresql.conf.sample
>>>>>> > for transparency of the code. As an open source project it is
>>>>>> important to
>>>>>> > classify all the options available, for both the user and the
>>>>>> developers.
>>>>>>
>>>>>> Added although, I still don't approve of it myself.
>>>>>>
>>>>>> > 5) What is cruelly missing with this patch are numbers. By how much
>>>>>> this
>>>>>> > feature improves performance?
>>>>>> > Simple tests like a ORDER BY on a table with millions of rows would
>>>>>> be
>>>>>> > enough, but as this is a performance feature, you should provide
>>>>>> data about
>>>>>> > how much those cases are improved.
>>>>>> > 6) A complain that I say for each planner-related patch in this
>>>>>> project....
>>>>>> > This patch has a size of 200KB, 65% being updates on the EXPLAIN
>>>>>> queries of
>>>>>> > the XC query planning regression tests. This makes the commit tree
>>>>>> > unreadable and the evolution of the regression tests realy hard to
>>>>>> follow
>>>>>> > because they need to be modified each time a planner optimization is
>>>>>> > introduced. We should move to a set of tests which is far more
>>>>>> > code-modification proof!
>>>>>>
>>>>>> This is planner code, so it's going to affect the plans that it
>>>>>> creates. In the test files we add EXPLAIN (with proper switches) to
>>>>>> check 1. whether the plans are coming as expected at that stage of
>>>>>> development, 2. Whether the clauses expected to be pushed down are
>>>>>> being indeed pushed down 3. the queries to be sent to the datanodes
>>>>>> are being crafted correctly. For every planner change, all the three
>>>>>> things above change, for the operation being planned and hence the
>>>>>> expected output changes are unavoidable. In fact, if you look at how
>>>>>> the expected output changing you get a idea of how correctly/wrong the
>>>>>> planner changes are made. Those diffs if go wrong, are great help to
>>>>>> me in finding problems earlier. During merge as well, if those tests
>>>>>> fail because of EXPLAIN outputs, that's a way to unwanted planner
>>>>>> changes early enough and correct them. Sorry, for the huge change, but
>>>>>> it's unavoidable. The testcases are intentionally prone to (planner)
>>>>>> code modification.
>>>>>>
>>>>>> >
>>>>>> > Regards,
>>>>>> >
>>>>>> > On Mon, Nov 26, 2012 at 8:42 PM, Ashutosh Bapat
>>>>>> > <ash...@en...> wrote:
>>>>>> >>
>>>>>> >> Hi All,
>>>>>> >> PFA patch which enables pushing down the ORDER BY clause or
>>>>>> sorting to the
>>>>>> >> Datanode/s through standard_planner().
>>>>>> >>
>>>>>> >> This patch leverages the already existing merge sort mechanism for
>>>>>> >> RemoteQuery but now outside RemoteQuery using Sort plans.
>>>>>> >>
>>>>>> >> For a Sort plan it checks if all the expressions on which the
>>>>>> result needs
>>>>>> >> to be sorted are shippable and that the underlying tree allows to
>>>>>> push ORDER
>>>>>> >> BY to the Datanodes. If so, it includes the ORDER BY clause in the
>>>>>> query to
>>>>>> >> be sent to the Datanodes. If the number of Datanodes which
>>>>>> participate in
>>>>>> >> RemoteQuery is not 1, it adds a covering Sort node to merge the
>>>>>> results from
>>>>>> >> the Datanodes, otherwise, there is no covering Sort plan.
>>>>>> >>
>>>>>> >> The merge sort mechanism treats every Datanode as a tape and
>>>>>> prefetches
>>>>>> >> the results from the Datanode corresponding to a dwindling tape.
>>>>>> This is
>>>>>> >> enabled by setting the node from where to pull the next row before
>>>>>> calling
>>>>>> >> ExecProcNode on RemoteQueryState.
>>>>>> >>
>>>>>> >> The same mechanism is being used for Sort plan node under Group or
>>>>>> Agg
>>>>>> >> nodes.
>>>>>> >>
>>>>>> >> The patch adds test xc_sort.sql for testing this functionality. It
>>>>>> also
>>>>>> >> removes the now useless members sort and tuplesortstate from
>>>>>> RemoteQuery and
>>>>>> >> RemoteQueryState resp.
>>>>>> >> --
>>>>>> >> Best Wishes,
>>>>>> >> Ashutosh Bapat
>>>>>> >> EntepriseDB Corporation
>>>>>> >> The Enterprise Postgres Company
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> ------------------------------------------------------------------------------
>>>>>> >> Monitor your physical, virtual and cloud infrastructure from a
>>>>>> single
>>>>>> >> web console. Get in-depth insight into apps, servers, databases,
>>>>>> vmware,
>>>>>> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>>>>>> >> Pricing starts from $795 for 25 servers or applications!
>>>>>> >> http://p.sf.net/sfu/zoho_dev2dev_nov
>>>>>> >> _______________________________________________
>>>>>> >> Postgres-xc-developers mailing list
>>>>>> >> Pos...@li...
>>>>>> >>
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Michael Paquier
>>>>>> > http://michael.otacoo.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Wishes,
>>>>>> Ashutosh Bapat
>>>>>> EntepriseDB Corporation
>>>>>> The Enterprise Postgres Company
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Wishes,
>>>>> Ashutosh Bapat
>>>>> EntepriseDB Corporation
>>>>> The Postgres Database Company
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Wishes,
>>>> Ashutosh Bapat
>>>> EntepriseDB Corporation
>>>> The Postgres Database Company
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Get your SQL database under version control now!
>>>> Version control is standard for application code, but databases havent
>>>> caught up. So what steps can you take to put your SQL databases under
>>>> version control? Why should you start doing it? Read more to find out.
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>
>>>> _______________________________________________
>>>> Postgres-xc-developers mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>
>>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

Okay.   Are there any more input to this?   We have to fix the leak anyway
and I'd like to commit this unless there's any more issue.

Regards;
---
Koichi


2013/8/6 Nikhil Sontakke <ni...@st...>

>
>
>
>> One of the reasons why I chose to do it this way is to reduce the impact
>> on the code that I borrowed heavily from Postgres and turned it into
>> multi-threaded architecture. Thats why we also see something like
>> TopMostMemoryContext. I wanted to emulate thread as a process, but a few
>> things were required to be at process level.
>>
>
> Yeah, I understand your point of view. I am just saying that it confuses
> things. I have reported one core earlier just because someone palloc'ed in
> TopMemoryContext "assuming" that it's the top memory context and not the
> memory context of the thread. I am talking about those kinds of confusion.
>
> As I said, the current code looks ok enough and takes care to cleanup
> memory contexts appropriately AFAICS.
>
> Regards,
> Nikhils
>
>
>>
>>
>>> thrinfo->thr_error_context = AllocSetContextCreate(ErrorContext,
>>>
>>> The comment above this line says
>>>
>>>     /*
>>>      * Each thread gets its own ErrorContext and its a child of
>>> ErrorContext of
>>>      * the main process
>>>      */
>>>
>>> However due to the inclusion of gtm.h, ErrorContext becomes a macro and
>>> expands to GetMyThreadInfo->thr_error_context which should be NULL!
>>>
>>> However the code is careful to track and release contexts AFAICS. But we
>>> need to be mindful of this in the future as well.
>>>
>>>
>> Will check that code.
>>
>> Thanks,
>> Pavan
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>

Can it be optional so that planner can choose if ExecInitRemoteQuery just
issue a query or wait for the first row to come?  Then it is a planner
issue which to choose.


2013/8/7 Ashutosh Bapat <ash...@en...>

>
>
>
> On Wed, Aug 7, 2013 at 1:57 PM, Koichi Suzuki <koi...@gm...>wrote:
>
>> Hi;
>>
>>
>> 2013/8/6 Ashutosh Bapat <ash...@en...>
>>
>>> Hi All,
>>> I worked on this (patch attached) but I didn't see any performance
>>> improvement. In fact, to my surprise the performance actually dropped.
>>>
>>> On a true cluster, with 5 coordinators and 5 datanodes, I saw these
>>> numbers with 1000000 rows in a table with two integer columns. The times
>>> are average times required for 100 runs of the same query, returning whole
>>> rows ordered by one of the columns.
>>> No ORDER BY push-down - 1640 ms
>>> ORDER BY push-down using Sort plan - 1100 ms (this is the current
>>> implementation)
>>> ORDER BY Push-down using MergeAppend plan - 1170 ms
>>>
>>> The assumption behind trying this out was explained in my earlier mail
>>> in this thread. But it looks like we do not get any improvement.
>>>
>>> One of the possible reasons, I think, is involvement of multiple
>>> RemoteQuery nodes in MergeAppend plan. While executing MergeAppend plan,
>>> the ExecProcNode() is called on the underlying node, one after the other.
>>> For every RemoteQuery node in MergeAppend plan it waits till that node
>>> emits a row. This means that we do not execute the queries on datanodes in
>>> parallel when executing through MergeAppend plan unlike Sort plan where
>>> there is only one RemoteQuery node involved.
>>>
>>> In order to get full advantage of MergeAppend plan (and keep the code
>>> clean), I think we need to query the datanodes in parallel somewhere
>>> outside ExecProcNode(). Thoughts? Suggestions?
>>>
>>
>> Can we divide MergeAppend execution into two stage, one is just to issue
>> a query and then begin to receive rows.   Because this is a change in the
>> core, we should submit the idea, the usecase and the code to PG community.
>>
>>
> This can be done if we move "issueing query" inside ExecInitRemoteQuery,
> but I am not sure if that can be applied in all the cases. This wouldn't
> require any change in PG.
>
>
>> Regards;
>> ---
>> Koichi Suzuki
>>
>>
>>>
>>> On Fri, May 24, 2013 at 1:30 AM, Ashutosh Bapat <
>>> ash...@en...> wrote:
>>>
>>>> Hi All,
>>>> While I hear Robert's presentation on query planning go wrong, I heard
>>>> (first time) about MergeAppend plan. It looks like, we have something
>>>> better to do with the way we are pushing the ORDER BY clauses and doing
>>>> merge at the coordinator. Right now we are using Sort node and in ExecSort
>>>> we start merging the rows. Instead, I think we should be using MergeAppend
>>>> node. The only bad thing about this approach is MergeAppend node expects
>>>> separate plans for each run of the sorted data. To do this, we need to
>>>> replicate the RemoteQuery node as many times as there are nodes and put
>>>> them in the list to the MergeAppend plan. I think that's going to solve the
>>>> problem with materialisation (see mail thread "Using remote sorting for
>>>> merge-join") and thus improve performance. It also takes away the need to
>>>> have xc_node_id in the tupleslot of tuplestore. Obviously, simplifying the
>>>> code.
>>>>
>>>> Any comments?
>>>>
>>>> Created 3613815
>>>> <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232>and
>>>> assigned to me.
>>>> <https://sourceforge.net/tracker/?func=detail&aid=3613815&group_id=311227&atid=1310232>
>>>>
>>>> On Mon, Dec 3, 2012 at 3:44 AM, Ashutosh Bapat <
>>>> ash...@en...> wrote:
>>>>
>>>>> On Fri, Nov 30, 2012 at 9:37 AM, Michael Paquier
>>>>> <mic...@gm...> wrote:
>>>>> > Hi Ashutosh,
>>>>> >
>>>>> > Thanks for the patch.
>>>>> > I found for the time being the following problems:
>>>>> > 1) The patch has some whitespaces
>>>>> > /home/michael/download/xc_sort.patch:16: trailing whitespace.
>>>>> >
>>>>> > /home/michael/download/xc_sort.patch:34: trailing whitespace.
>>>>> >
>>>>> > /home/michael/download/xc_sort.patch:307: trailing whitespace.
>>>>> >         return local_plan_left;
>>>>> > /home/michael/download/xc_sort.patch:356: trailing whitespace.
>>>>> >
>>>>> > /home/michael/download/xc_sort.patch:360: trailing whitespace.
>>>>> >
>>>>>
>>>>> Fixed.
>>>>>
>>>>> > error: patch failed: src/include/pgxc/execRemote.h:95
>>>>> > 2) the patch does not apply cleanly even by using "patch -p1"-like
>>>>> commands
>>>>>
>>>>> When I sent the patch (more than a week before you replied) it was
>>>>> based on the HEAD, but then there were some commits which created the
>>>>> conflicts. The one attached should apply.
>>>>>
>>>>> > 3) No documentation about the new option enable_remotesort.
>>>>> > 4) enable_remotesort is not listed in postgresql.conf.
>>>>> > Please provide both documentation and reference in
>>>>> postgresql.conf.sample
>>>>> > for transparency of the code. As an open source project it is
>>>>> important to
>>>>> > classify all the options available, for both the user and the
>>>>> developers.
>>>>>
>>>>> Added although, I still don't approve of it myself.
>>>>>
>>>>> > 5) What is cruelly missing with this patch are numbers. By how much
>>>>> this
>>>>> > feature improves performance?
>>>>> > Simple tests like a ORDER BY on a table with millions of rows would
>>>>> be
>>>>> > enough, but as this is a performance feature, you should provide
>>>>> data about
>>>>> > how much those cases are improved.
>>>>> > 6) A complain that I say for each planner-related patch in this
>>>>> project....
>>>>> > This patch has a size of 200KB, 65% being updates on the EXPLAIN
>>>>> queries of
>>>>> > the XC query planning regression tests. This makes the commit tree
>>>>> > unreadable and the evolution of the regression tests realy hard to
>>>>> follow
>>>>> > because they need to be modified each time a planner optimization is
>>>>> > introduced. We should move to a set of tests which is far more
>>>>> > code-modification proof!
>>>>>
>>>>> This is planner code, so it's going to affect the plans that it
>>>>> creates. In the test files we add EXPLAIN (with proper switches) to
>>>>> check 1. whether the plans are coming as expected at that stage of
>>>>> development, 2. Whether the clauses expected to be pushed down are
>>>>> being indeed pushed down 3. the queries to be sent to the datanodes
>>>>> are being crafted correctly. For every planner change, all the three
>>>>> things above change, for the operation being planned and hence the
>>>>> expected output changes are unavoidable. In fact, if you look at how
>>>>> the expected output changing you get a idea of how correctly/wrong the
>>>>> planner changes are made. Those diffs if go wrong, are great help to
>>>>> me in finding problems earlier. During merge as well, if those tests
>>>>> fail because of EXPLAIN outputs, that's a way to unwanted planner
>>>>> changes early enough and correct them. Sorry, for the huge change, but
>>>>> it's unavoidable. The testcases are intentionally prone to (planner)
>>>>> code modification.
>>>>>
>>>>> >
>>>>> > Regards,
>>>>> >
>>>>> > On Mon, Nov 26, 2012 at 8:42 PM, Ashutosh Bapat
>>>>> > <ash...@en...> wrote:
>>>>> >>
>>>>> >> Hi All,
>>>>> >> PFA patch which enables pushing down the ORDER BY clause or sorting
>>>>> to the
>>>>> >> Datanode/s through standard_planner().
>>>>> >>
>>>>> >> This patch leverages the already existing merge sort mechanism for
>>>>> >> RemoteQuery but now outside RemoteQuery using Sort plans.
>>>>> >>
>>>>> >> For a Sort plan it checks if all the expressions on which the
>>>>> result needs
>>>>> >> to be sorted are shippable and that the underlying tree allows to
>>>>> push ORDER
>>>>> >> BY to the Datanodes. If so, it includes the ORDER BY clause in the
>>>>> query to
>>>>> >> be sent to the Datanodes. If the number of Datanodes which
>>>>> participate in
>>>>> >> RemoteQuery is not 1, it adds a covering Sort node to merge the
>>>>> results from
>>>>> >> the Datanodes, otherwise, there is no covering Sort plan.
>>>>> >>
>>>>> >> The merge sort mechanism treats every Datanode as a tape and
>>>>> prefetches
>>>>> >> the results from the Datanode corresponding to a dwindling tape.
>>>>> This is
>>>>> >> enabled by setting the node from where to pull the next row before
>>>>> calling
>>>>> >> ExecProcNode on RemoteQueryState.
>>>>> >>
>>>>> >> The same mechanism is being used for Sort plan node under Group or
>>>>> Agg
>>>>> >> nodes.
>>>>> >>
>>>>> >> The patch adds test xc_sort.sql for testing this functionality. It
>>>>> also
>>>>> >> removes the now useless members sort and tuplesortstate from
>>>>> RemoteQuery and
>>>>> >> RemoteQueryState resp.
>>>>> >> --
>>>>> >> Best Wishes,
>>>>> >> Ashutosh Bapat
>>>>> >> EntepriseDB Corporation
>>>>> >> The Enterprise Postgres Company
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> ------------------------------------------------------------------------------
>>>>> >> Monitor your physical, virtual and cloud infrastructure from a
>>>>> single
>>>>> >> web console. Get in-depth insight into apps, servers, databases,
>>>>> vmware,
>>>>> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>>>>> >> Pricing starts from $795 for 25 servers or applications!
>>>>> >> http://p.sf.net/sfu/zoho_dev2dev_nov
>>>>> >> _______________________________________________
>>>>> >> Postgres-xc-developers mailing list
>>>>> >> Pos...@li...
>>>>> >> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Michael Paquier
>>>>> > http://michael.otacoo.com
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Wishes,
>>>>> Ashutosh Bapat
>>>>> EntepriseDB Corporation
>>>>> The Enterprise Postgres Company
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Wishes,
>>>> Ashutosh Bapat
>>>> EntepriseDB Corporation
>>>> The Postgres Database Company
>>>>
>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Postgres Database Company
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Get your SQL database under version control now!
>>> Version control is standard for application code, but databases havent
>>> caught up. So what steps can you take to put your SQL databases under
>>> version control? Why should you start doing it? Read more to find out.
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>

I'd like to commit this patch unless there're no more discussion or input.

Regards;
---
Koichi Suzuki


2013/8/8 Koichi Suzuki <koi...@gm...>

>
>
>
> 2013/8/6 Nikhil Sontakke <ni...@st...>
>
>>
>>
>>
>>> Can GTM/proxy cleanup the pending list when it detects the death of a
>>> connection, just like GTM does its cleanup?
>>>
>>>
>> Well, the pending requests are not stored in a per client list. We could
>> go and look for this specific client in the list though and clean it up.
>> But remember that the client might have given a set of instructions to the
>> proxy and disappeared legitimately. So processing all requests in order
>> before processing the disconnection seems much cleaner.
>>
>
> Yes, it makes sense.    BTW, all the pending and processed list for each
> select() cycle is cleaned up.
> ---
> Koichi Suzuki
>
>
>>
>> Regards,
>> Nikhils
>>
>>
>>>
>>> On Tue, Aug 6, 2013 at 11:03 AM, Nikhil Sontakke <ni...@st...>wrote:
>>>
>>>>
>>>> A naive question,
>>>>> Can we also cleanup the pending list?
>>>>>
>>>>>
>>>> Umm, the processed commands are put onto the thr_processed_commands
>>>> list and freed up eventually. Is that what you were asking?
>>>>
>>>> Regards,
>>>> Nikhils
>>>>
>>>>>
>>>>>  On Tue, Aug 6, 2013 at 9:56 AM, Nikhil Sontakke <ni...@st...>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We have had a long standing issue with a GTM Proxy in the mix. It has
>>>>>> been reported here:
>>>>>>
>>>>>>
>>>>>> http://sourceforge.net/mailarchive/forum.php?thread_name=CAB7nPqQ51fabopxCOHPrRM%3D2WTJMikfZUPHNEW942DDHmED7bQ%40mail.gmail.com&forum_name=postgres-xc-developers
>>>>>>
>>>>>> I think I found the root cause:
>>>>>>
>>>>>> Here's how things work:
>>>>>>
>>>>>> The client connects to proxy. The proxy internally connects to the
>>>>>> GTM.
>>>>>>
>>>>>> Some commands from clients are sent immediately to the GTM by the
>>>>>> proxy.
>>>>>>
>>>>>> Some commands are put in the pending list on the proxy for batch
>>>>>> processing later.
>>>>>>
>>>>>> Now, if the client goes away before processing pending commands, the
>>>>>> proxy is immediately telling the GTM so. That initiates a cleanup on the
>>>>>> GTM side and removes the XIDs associated with this client. After this
>>>>>> cleanup when the GTM receives a command like "MSG_SNAPSHOT_GET" or
>>>>>> "MSG_TXN_COMMIT_MULTI" etc. from the pending list from the proxy, then it's
>>>>>> going to get confused.
>>>>>>
>>>>>> A way to handle this is to put "disconnected" messages also onto the
>>>>>> pending list. We then process these "disconnected" messages LAST from the
>>>>>> pending list.
>>>>>>
>>>>>> I have tried this fix and run a number of tests (using pgbench) and
>>>>>> things seem to be ok. I do not see the invalid handle and snapshot errors
>>>>>> on the GTM side with this patch in place.
>>>>>>
>>>>>> PFA, patches for 1.0, 1.1 and head attached with this email.
>>>>>>
>>>>>> Regards,
>>>>>> Nikhils
>>>>>> --
>>>>>> StormDB - http://www.stormdb.com
>>>>>> The Database Cloud
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Get your SQL database under version control now!
>>>>>> Version control is standard for application code, but databases havent
>>>>>> caught up. So what steps can you take to put your SQL databases under
>>>>>> version control? Why should you start doing it? Read more to find out.
>>>>>>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> Postgres-xc-developers mailing list
>>>>>> Pos...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Wishes,
>>>>> Ashutosh Bapat
>>>>> EntepriseDB Corporation
>>>>> The Postgres Database Company
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> StormDB - http://www.stormdb.com
>>>> The Database Cloud
>>>>
>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Postgres Database Company
>>>
>>
>>
>>
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>>
>>
>> ------------------------------------------------------------------------------
>> Get your SQL database under version control now!
>> Version control is standard for application code, but databases havent
>> caught up. So what steps can you take to put your SQL databases under
>> version control? Why should you start doing it? Read more to find out.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>

I finally found that missing "break" caused the problem.  It caused extra response to the proxy (only when connected via proxy), causing next round error.

THank you very much for the advice.   I will commit it both to master and 1..1.

Best;
---
Koichi Suzuki
On 2013/08/05, at 22:25, 鈴木 幸市 <ko...@in...<mailto:ko...@in...>> wrote:

Thanks.  I've already tested similar patch but it is not sufficient.

Somehow, response from GTM is received twice in a GTM-Proxy.   Because GTM-Proxy clears up all the response in each round of backend scan, I'm worrying this has be caused by bad response handling in GTM, which does not happen when backgrounds are connected directly to the GTM.

I'm asking Pavan a review on this.

Regards;
---
Koichi Suzuki

On 2013/08/02, at 12:39, Nikhil Sontakke <ni...@st...<mailto:ni...@st...>> wrote:

Hi,

I saw missing breaks for the MSG_BARRIER handling in both GTM and GTM_Proxy.. I wonder if barrier works in the first place without these breaks? PFA, patch to fix this. AFAICS, the 1.1 branch also has similar issues.

Regards,
Nikhils
--
StormDB - http://www.stormdb.com<http://www.stormdb.com/>
The Database Cloud <pgxc_head_barrier_missing_breaks.patch>------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...<mailto:Pos...@li...>
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk_______________________________________________
Postgres-xc-developers mailing list
Pos...@li...
https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

2013/8/6 Nikhil Sontakke <ni...@st...>

>
>
>
>> Can GTM/proxy cleanup the pending list when it detects the death of a
>> connection, just like GTM does its cleanup?
>>
>>
> Well, the pending requests are not stored in a per client list. We could
> go and look for this specific client in the list though and clean it up.
> But remember that the client might have given a set of instructions to the
> proxy and disappeared legitimately. So processing all requests in order
> before processing the disconnection seems much cleaner.
>

Yes, it makes sense.    BTW, all the pending and processed list for each
select() cycle is cleaned up.
---
Koichi Suzuki


>
> Regards,
> Nikhils
>
>
>>
>> On Tue, Aug 6, 2013 at 11:03 AM, Nikhil Sontakke <ni...@st...>wrote:
>>
>>>
>>> A naive question,
>>>> Can we also cleanup the pending list?
>>>>
>>>>
>>> Umm, the processed commands are put onto the thr_processed_commands list
>>> and freed up eventually. Is that what you were asking?
>>>
>>> Regards,
>>> Nikhils
>>>
>>>>
>>>>  On Tue, Aug 6, 2013 at 9:56 AM, Nikhil Sontakke <ni...@st...>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We have had a long standing issue with a GTM Proxy in the mix. It has
>>>>> been reported here:
>>>>>
>>>>>
>>>>> http://sourceforge.net/mailarchive/forum.php?thread_name=CAB7nPqQ51fabopxCOHPrRM%3D2WTJMikfZUPHNEW942DDHmED7bQ%40mail.gmail.com&forum_name=postgres-xc-developers
>>>>>
>>>>> I think I found the root cause:
>>>>>
>>>>> Here's how things work:
>>>>>
>>>>> The client connects to proxy. The proxy internally connects to the
>>>>> GTM.
>>>>>
>>>>> Some commands from clients are sent immediately to the GTM by the
>>>>> proxy.
>>>>>
>>>>> Some commands are put in the pending list on the proxy for batch
>>>>> processing later.
>>>>>
>>>>> Now, if the client goes away before processing pending commands, the
>>>>> proxy is immediately telling the GTM so. That initiates a cleanup on the
>>>>> GTM side and removes the XIDs associated with this client. After this
>>>>> cleanup when the GTM receives a command like "MSG_SNAPSHOT_GET" or
>>>>> "MSG_TXN_COMMIT_MULTI" etc. from the pending list from the proxy, then it's
>>>>> going to get confused.
>>>>>
>>>>> A way to handle this is to put "disconnected" messages also onto the
>>>>> pending list. We then process these "disconnected" messages LAST from the
>>>>> pending list.
>>>>>
>>>>> I have tried this fix and run a number of tests (using pgbench) and
>>>>> things seem to be ok. I do not see the invalid handle and snapshot errors
>>>>> on the GTM side with this patch in place.
>>>>>
>>>>> PFA, patches for 1.0, 1.1 and head attached with this email.
>>>>>
>>>>> Regards,
>>>>> Nikhils
>>>>> --
>>>>> StormDB - http://www.stormdb.com
>>>>> The Database Cloud
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Get your SQL database under version control now!
>>>>> Version control is standard for application code, but databases havent
>>>>> caught up. So what steps can you take to put your SQL databases under
>>>>> version control? Why should you start doing it? Read more to find out.
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Postgres-xc-developers mailing list
>>>>> Pos...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Wishes,
>>>> Ashutosh Bapat
>>>> EntepriseDB Corporation
>>>> The Postgres Database Company
>>>>
>>>
>>>
>>>
>>> --
>>> StormDB - http://www.stormdb.com
>>> The Database Cloud
>>>
>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>
>
> ------------------------------------------------------------------------------
> Get your SQL database under version control now!
> Version control is standard for application code, but databases havent
> caught up. So what steps can you take to put your SQL databases under
> version control? Why should you start doing it? Read more to find out.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Thanks Nikhil, Pavan.

I will commit the change to sig_t both to the master and 1.1.

Regards;
---
Koichi Suzuki


2013/8/7 Nikhil Sontakke <ni...@st...>

> Suzuki san,
>
> This is the standard way of handling handler functions in PG AFAICS.
>
> libpq/gtm also uses this way, so it's pretty portable.
>
> Pavan, yeah sig_t works on OS X as well.
>
> Regards,
> Nikhils
>
>
>
> On Wed, Aug 7, 2013 at 3:04 PM, Pavan Deolasee <pav...@gm...>wrote:
>
>> On Wed, Aug 7, 2013 at 3:00 PM, Koichi Suzuki <koi...@gm...>wrote:
>>
>>> I still have a couple of questions on the patch.
>>>
>>> 1. Is this way portable enough?  It fixes the type of signal handler,
>>> which is originally abstracted.
>>>
>>>
>> FWIW I'd fixed the same issue by using sig_t on darwin.
>>
>> Thanks,
>> Pavan
>> --
>> Pavan Deolasee
>> http://www.linkedin.com/in/pavandeolasee
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>

Suzuki san,

This is the standard way of handling handler functions in PG AFAICS.

libpq/gtm also uses this way, so it's pretty portable.

Pavan, yeah sig_t works on OS X as well.

Regards,
Nikhils


On Wed, Aug 7, 2013 at 3:04 PM, Pavan Deolasee <pav...@gm...>wrote:

> On Wed, Aug 7, 2013 at 3:00 PM, Koichi Suzuki <koi...@gm...>wrote:
>
>> I still have a couple of questions on the patch.
>>
>> 1. Is this way portable enough?  It fixes the type of signal handler,
>> which is originally abstracted.
>>
>>
> FWIW I'd fixed the same issue by using sig_t on darwin.
>
> Thanks,
> Pavan
> --
> Pavan Deolasee
> http://www.linkedin.com/in/pavandeolasee
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud

On Wed, Aug 7, 2013 at 3:00 PM, Koichi Suzuki <koi...@gm...> wrote:

> I still have a couple of questions on the patch.
>
> 1. Is this way portable enough?  It fixes the type of signal handler,
> which is originally abstracted.
>
>
FWIW I'd fixed the same issue by using sig_t on darwin.

Thanks,
Pavan
-- 
Pavan Deolasee
http://www.linkedin.com/in/pavandeolasee

I still have a couple of questions on the patch.

1. Is this way portable enough?  It fixes the type of signal handler, which
is originally abstracted.

#ifdef <macos>
typedef void (*pgsigfunc) (int);
#else
#define pgsigfunc sighandler_t
#endif

...

#ifdef <macos>
typedef void (*pgsigfunc) (int)
#endif

Is the following okay?  Do you have any idea what symbol is used to
identify Mac-OS?   Makefile.global identifies Mac-OS as darwin but I
couldn't find how we can use this in the source code.

2. Are you okay if the patch is applied only to master, or do you need this
for 1.1 as well?

Regards;
---
Koichi Suzuki


2013/8/7 Koichi Suzuki <koi...@gm...>

> Thanks Nikhil for the patch.
>
> I have a couple of questions on this patch.
>
> 1, Did you run only pgxc_ctl on Mac OS and the others (all XC components)
> on Linux?
> 2. Did you really test that pgxc_ctl runs on Mac OS?   pgxc_ctl uses so
> many system() functions so it is shell script dependent.   I do hope all
> the shell scripts used in pgxc_ctl works in Mac OS too.
>
> Regards;
>
>
>
>
> 2013/8/7 Nikhil Sontakke <ni...@st...>
>
>> Hi Suzuki san,
>>
>> PFA, patch which allows pgxc_ctl to compile on OS X.
>>
>> Regards,
>> Nikhils
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>>
>> ------------------------------------------------------------------------------
>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> It's a free troubleshooting tool designed for production.
>> Get down to code-level detail for bottlenecks, with <2% overhead.
>> Download for free and get started troubleshooting in minutes.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>