postgres-xc-developers Mailing List for Postgres-XC (Page 3)

The background of the restriction is apparently PG restriction.   My point is does this issue will not happen in PG.

Regards;
---
Koichi Suzuki



On 2013/05/17, at 13:36, Ashutosh Bapat <ash...@en...> wrote:

> Ok, in that case, I don't think we have any other way but to convert COPY into INSERT between coordinator and datanode when the triggers are not shippable. I think this restriction applies only to the row triggers; statement triggers should be fine.
> 
> 
> On Fri, May 17, 2013 at 10:01 AM, Amit Khandekar <ami...@en...> wrote:
> 
> 
> On 17 May 2013 09:36, Ashutosh Bapat <ash...@en...> wrote:
> 
> 
> 
> On Fri, May 17, 2013 at 9:27 AM, Amit Khandekar <ami...@en...> wrote:
> 
> 
> On 15 May 2013 12:53, Amit Khandekar <ami...@en...> wrote:
> In XC, the way COPY is implemented is that for each record, we read the whole line into memory, and then pass it to the datanode as-is. If there are non-shippable default column expressions, we evaluate the default values , convert them into output form, and append them to the data row.
> 
> In presence of BR triggers, currently the ExecBRInsertTriggers() do not get called because of the way we skip the whole PG code block; instead we just send the data row as-is, optionally appending default values into the data row.
> 
> What we need to do is; convert the tuple returned by ExecBRTriggers into text data row, but the text data should be in COPY format. This is because we need to send the data row to the datanode using COPY command, so it requires correct COPY format, such as escape sequences. 
> 
> For this, we need to call the function CopyOneRowTo() that is being used by COPY TO. This will make sure it will emit the data row in the COPY format. But we need to create a temporary CopyState because CopyOneRowTo() needs it. We can derive it from the current CopyState that is already created for COPY FROM. Most of the fields remain the same, except we need to re-assign CopyState->line_buf, and CopyState->rowcontext.
> 
> This will save us from writing code to make sure the new output data row generated by BR triggers complies with COPY data format.
> 
> I had already done similar thing for appending default values into the data row. We call functions like CopyAttributeOutCSV(), CopyInt32() to append the values to the data row in COPY format. There, we did not require CopyOneRow() because we did not require the complete row, we needed to append only a subset of columns to the existing data row.
> 
> Comments/suggestions welcome.
> 
> I have hit a dead end in the way I am allowing the BR triggers to execute during COPY.
> 
> It is not possible to send any non-COPY messages to the backend when the client-server protocol is in COPY mode. Which means, it is not possible to send any commands to the datanode when connection is in DN_CONNECTION_STATE_COPY_IN state. When the trigger function executes an SQL query, that query can't be executed because it's not possible to exchange any non-copy messages, let alone sending a query to the backend (i.e. datanode).
> 
> Is this an XC restriction or PG restriction?
> 
> The above is a PG restriction.
> 
> Not accpepting any other client messages during a COPY protocol is a PG backend requirement. Not accepting  trigger queries from coordinator has become an XC restriction as a result of the above PG protocol restriction.
> 
>  
> 
> This naturally happens only for non-shippable triggers. If triggers are executed on datanode, then this issue does not arise.
> 
> We need to device some other means to support non-shippable triggers for COPY. May be we would end up sending INSERT commands on the datanode instead of COPY command, if there are non-shippable triggers. Each of the data row will be sent as parameters to the insert query. This operation would be slow, but possible.   
> 
> 
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
> 
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Ok, in that case, I don't think we have any other way but to convert COPY
into INSERT between coordinator and datanode when the triggers are not
shippable. I think this restriction applies only to the row triggers;
statement triggers should be fine.


On Fri, May 17, 2013 at 10:01 AM, Amit Khandekar <
ami...@en...> wrote:

>
>
> On 17 May 2013 09:36, Ashutosh Bapat <ash...@en...>wrote:
>
>>
>>
>>
>> On Fri, May 17, 2013 at 9:27 AM, Amit Khandekar <
>> ami...@en...> wrote:
>>
>>>
>>>
>>> On 15 May 2013 12:53, Amit Khandekar <ami...@en...>wrote:
>>>
>>>> In XC, the way COPY is implemented is that for each record, we read the
>>>> whole line into memory, and then pass it to the datanode as-is. If there
>>>> are non-shippable default column expressions, we evaluate the default
>>>> values , convert them into output form, and append them to the data row.
>>>>
>>>> In presence of BR triggers, currently the ExecBRInsertTriggers() do not
>>>> get called because of the way we skip the whole PG code block; instead we
>>>> just send the data row as-is, optionally appending default values into the
>>>> data row.
>>>>
>>>> What we need to do is; convert the tuple returned by ExecBRTriggers
>>>> into text data row, but the text data should be in COPY format. This is
>>>> because we need to send the data row to the datanode using COPY command, so
>>>> it requires correct COPY format, such as escape sequences.
>>>>
>>>> For this, we need to call the function CopyOneRowTo() that is being
>>>> used by COPY TO. This will make sure it will emit the data row in the COPY
>>>> format. But we need to create a temporary CopyState because CopyOneRowTo()
>>>> needs it. We can derive it from the current CopyState that is already
>>>> created for COPY FROM. Most of the fields remain the same, except we need
>>>> to re-assign CopyState->line_buf, and CopyState->rowcontext.
>>>>
>>>> This will save us from writing code to make sure the new output data
>>>> row generated by BR triggers complies with COPY data format.
>>>>
>>>> I had already done similar thing for appending default values into the
>>>> data row. We call functions like CopyAttributeOutCSV(), CopyInt32() to
>>>> append the values to the data row in COPY format. There, we did not require
>>>> CopyOneRow() because we did not require the complete row, we needed to
>>>> append only a subset of columns to the existing data row.
>>>>
>>>> Comments/suggestions welcome.
>>>>
>>>
>>> I have hit a dead end in the way I am allowing the BR triggers to
>>> execute during COPY.
>>>
>>> It is not possible to send any non-COPY messages to the backend when the
>>> client-server protocol is in COPY mode. Which means, it is not possible to
>>> send any commands to the datanode when connection is in
>>> DN_CONNECTION_STATE_COPY_IN state. When the trigger function executes an
>>> SQL query, that query can't be executed because it's not possible to
>>> exchange any non-copy messages, let alone sending a query to the backend
>>> (i.e. datanode).
>>>
>>
>> Is this an XC restriction or PG restriction?
>>
>
> The above is a PG restriction.
>
> Not accpepting any other client messages during a COPY protocol is a PG
> backend requirement. Not accepting  trigger queries from coordinator has
> become an XC restriction as a result of the above PG protocol restriction.
>
>
>>
>>>
>>> This naturally happens only for non-shippable triggers. If triggers are
>>> executed on datanode, then this issue does not arise.
>>>
>>> We need to device some other means to support non-shippable triggers for
>>> COPY. May be we would end up sending INSERT commands on the datanode
>>> instead of COPY command, if there are non-shippable triggers. Each of the
>>> data row will be sent as parameters to the insert query. This operation
>>> would be slow, but possible.
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> AlienVault Unified Security Management (USM) platform delivers complete
>>> security visibility with the essential security capabilities. Easily and
>>> efficiently configure, manage, and operate all of your security controls
>>> from a single console and one unified framework. Download a free trial.
>>> http://p.sf.net/sfu/alienvault_d2d
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

On 17 May 2013 09:36, Ashutosh Bapat <ash...@en...>wrote:

>
>
>
> On Fri, May 17, 2013 at 9:27 AM, Amit Khandekar <
> ami...@en...> wrote:
>
>>
>>
>> On 15 May 2013 12:53, Amit Khandekar <ami...@en...>wrote:
>>
>>> In XC, the way COPY is implemented is that for each record, we read the
>>> whole line into memory, and then pass it to the datanode as-is. If there
>>> are non-shippable default column expressions, we evaluate the default
>>> values , convert them into output form, and append them to the data row.
>>>
>>> In presence of BR triggers, currently the ExecBRInsertTriggers() do not
>>> get called because of the way we skip the whole PG code block; instead we
>>> just send the data row as-is, optionally appending default values into the
>>> data row.
>>>
>>> What we need to do is; convert the tuple returned by ExecBRTriggers into
>>> text data row, but the text data should be in COPY format. This is because
>>> we need to send the data row to the datanode using COPY command, so it
>>> requires correct COPY format, such as escape sequences.
>>>
>>> For this, we need to call the function CopyOneRowTo() that is being used
>>> by COPY TO. This will make sure it will emit the data row in the COPY
>>> format. But we need to create a temporary CopyState because CopyOneRowTo()
>>> needs it. We can derive it from the current CopyState that is already
>>> created for COPY FROM. Most of the fields remain the same, except we need
>>> to re-assign CopyState->line_buf, and CopyState->rowcontext.
>>>
>>> This will save us from writing code to make sure the new output data row
>>> generated by BR triggers complies with COPY data format.
>>>
>>> I had already done similar thing for appending default values into the
>>> data row. We call functions like CopyAttributeOutCSV(), CopyInt32() to
>>> append the values to the data row in COPY format. There, we did not require
>>> CopyOneRow() because we did not require the complete row, we needed to
>>> append only a subset of columns to the existing data row.
>>>
>>> Comments/suggestions welcome.
>>>
>>
>> I have hit a dead end in the way I am allowing the BR triggers to execute
>> during COPY.
>>
>> It is not possible to send any non-COPY messages to the backend when the
>> client-server protocol is in COPY mode. Which means, it is not possible to
>> send any commands to the datanode when connection is in
>> DN_CONNECTION_STATE_COPY_IN state. When the trigger function executes an
>> SQL query, that query can't be executed because it's not possible to
>> exchange any non-copy messages, let alone sending a query to the backend
>> (i.e. datanode).
>>
>
> Is this an XC restriction or PG restriction?
>

The above is a PG restriction.

Not accpepting any other client messages during a COPY protocol is a PG
backend requirement. Not accepting  trigger queries from coordinator has
become an XC restriction as a result of the above PG protocol restriction.


>
>>
>> This naturally happens only for non-shippable triggers. If triggers are
>> executed on datanode, then this issue does not arise.
>>
>> We need to device some other means to support non-shippable triggers for
>> COPY. May be we would end up sending INSERT commands on the datanode
>> instead of COPY command, if there are non-shippable triggers. Each of the
>> data row will be sent as parameters to the insert query. This operation
>> would be slow, but possible.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>

I'm afraid it's an XC restriction.

Regards;
---
Koichi Suzuki



On 2013/05/17, at 13:06, Ashutosh Bapat <ash...@en...> wrote:

> 
> 
> 
> On Fri, May 17, 2013 at 9:27 AM, Amit Khandekar <ami...@en...> wrote:
> 
> 
> On 15 May 2013 12:53, Amit Khandekar <ami...@en...> wrote:
> In XC, the way COPY is implemented is that for each record, we read the whole line into memory, and then pass it to the datanode as-is. If there are non-shippable default column expressions, we evaluate the default values , convert them into output form, and append them to the data row.
> 
> In presence of BR triggers, currently the ExecBRInsertTriggers() do not get called because of the way we skip the whole PG code block; instead we just send the data row as-is, optionally appending default values into the data row.
> 
> What we need to do is; convert the tuple returned by ExecBRTriggers into text data row, but the text data should be in COPY format. This is because we need to send the data row to the datanode using COPY command, so it requires correct COPY format, such as escape sequences. 
> 
> For this, we need to call the function CopyOneRowTo() that is being used by COPY TO. This will make sure it will emit the data row in the COPY format. But we need to create a temporary CopyState because CopyOneRowTo() needs it. We can derive it from the current CopyState that is already created for COPY FROM. Most of the fields remain the same, except we need to re-assign CopyState->line_buf, and CopyState->rowcontext.
> 
> This will save us from writing code to make sure the new output data row generated by BR triggers complies with COPY data format.
> 
> I had already done similar thing for appending default values into the data row. We call functions like CopyAttributeOutCSV(), CopyInt32() to append the values to the data row in COPY format. There, we did not require CopyOneRow() because we did not require the complete row, we needed to append only a subset of columns to the existing data row.
> 
> Comments/suggestions welcome.
> 
> I have hit a dead end in the way I am allowing the BR triggers to execute during COPY.
> 
> It is not possible to send any non-COPY messages to the backend when the client-server protocol is in COPY mode. Which means, it is not possible to send any commands to the datanode when connection is in DN_CONNECTION_STATE_COPY_IN state. When the trigger function executes an SQL query, that query can't be executed because it's not possible to exchange any non-copy messages, let alone sending a query to the backend (i.e. datanode).
> 
> Is this an XC restriction or PG restriction?
>  
> 
> This naturally happens only for non-shippable triggers. If triggers are executed on datanode, then this issue does not arise.
> 
> We need to device some other means to support non-shippable triggers for COPY. May be we would end up sending INSERT commands on the datanode instead of COPY command, if there are non-shippable triggers. Each of the data row will be sent as parameters to the insert query. This operation would be slow, but possible.   
> 
> 
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

On Fri, May 17, 2013 at 9:27 AM, Amit Khandekar <
ami...@en...> wrote:

>
>
> On 15 May 2013 12:53, Amit Khandekar <ami...@en...>wrote:
>
>> In XC, the way COPY is implemented is that for each record, we read the
>> whole line into memory, and then pass it to the datanode as-is. If there
>> are non-shippable default column expressions, we evaluate the default
>> values , convert them into output form, and append them to the data row.
>>
>> In presence of BR triggers, currently the ExecBRInsertTriggers() do not
>> get called because of the way we skip the whole PG code block; instead we
>> just send the data row as-is, optionally appending default values into the
>> data row.
>>
>> What we need to do is; convert the tuple returned by ExecBRTriggers into
>> text data row, but the text data should be in COPY format. This is because
>> we need to send the data row to the datanode using COPY command, so it
>> requires correct COPY format, such as escape sequences.
>>
>> For this, we need to call the function CopyOneRowTo() that is being used
>> by COPY TO. This will make sure it will emit the data row in the COPY
>> format. But we need to create a temporary CopyState because CopyOneRowTo()
>> needs it. We can derive it from the current CopyState that is already
>> created for COPY FROM. Most of the fields remain the same, except we need
>> to re-assign CopyState->line_buf, and CopyState->rowcontext.
>>
>> This will save us from writing code to make sure the new output data row
>> generated by BR triggers complies with COPY data format.
>>
>> I had already done similar thing for appending default values into the
>> data row. We call functions like CopyAttributeOutCSV(), CopyInt32() to
>> append the values to the data row in COPY format. There, we did not require
>> CopyOneRow() because we did not require the complete row, we needed to
>> append only a subset of columns to the existing data row.
>>
>> Comments/suggestions welcome.
>>
>
> I have hit a dead end in the way I am allowing the BR triggers to execute
> during COPY.
>
> It is not possible to send any non-COPY messages to the backend when the
> client-server protocol is in COPY mode. Which means, it is not possible to
> send any commands to the datanode when connection is in
> DN_CONNECTION_STATE_COPY_IN state. When the trigger function executes an
> SQL query, that query can't be executed because it's not possible to
> exchange any non-copy messages, let alone sending a query to the backend
> (i.e. datanode).
>

Is this an XC restriction or PG restriction?


>
> This naturally happens only for non-shippable triggers. If triggers are
> executed on datanode, then this issue does not arise.
>
> We need to device some other means to support non-shippable triggers for
> COPY. May be we would end up sending INSERT commands on the datanode
> instead of COPY command, if there are non-shippable triggers. Each of the
> data row will be sent as parameters to the insert query. This operation
> would be slow, but possible.
>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

On 15 May 2013 12:53, Amit Khandekar <ami...@en...>wrote:

> In XC, the way COPY is implemented is that for each record, we read the
> whole line into memory, and then pass it to the datanode as-is. If there
> are non-shippable default column expressions, we evaluate the default
> values , convert them into output form, and append them to the data row.
>
> In presence of BR triggers, currently the ExecBRInsertTriggers() do not
> get called because of the way we skip the whole PG code block; instead we
> just send the data row as-is, optionally appending default values into the
> data row.
>
> What we need to do is; convert the tuple returned by ExecBRTriggers into
> text data row, but the text data should be in COPY format. This is because
> we need to send the data row to the datanode using COPY command, so it
> requires correct COPY format, such as escape sequences.
>
> For this, we need to call the function CopyOneRowTo() that is being used
> by COPY TO. This will make sure it will emit the data row in the COPY
> format. But we need to create a temporary CopyState because CopyOneRowTo()
> needs it. We can derive it from the current CopyState that is already
> created for COPY FROM. Most of the fields remain the same, except we need
> to re-assign CopyState->line_buf, and CopyState->rowcontext.
>
> This will save us from writing code to make sure the new output data row
> generated by BR triggers complies with COPY data format.
>
> I had already done similar thing for appending default values into the
> data row. We call functions like CopyAttributeOutCSV(), CopyInt32() to
> append the values to the data row in COPY format. There, we did not require
> CopyOneRow() because we did not require the complete row, we needed to
> append only a subset of columns to the existing data row.
>
> Comments/suggestions welcome.
>

I have hit a dead end in the way I am allowing the BR triggers to execute
during COPY.

It is not possible to send any non-COPY messages to the backend when the
client-server protocol is in COPY mode. Which means, it is not possible to
send any commands to the datanode when connection is in
DN_CONNECTION_STATE_COPY_IN state. When the trigger function executes an
SQL query, that query can't be executed because it's not possible to
exchange any non-copy messages, let alone sending a query to the backend
(i.e. datanode).

This naturally happens only for non-shippable triggers. If triggers are
executed on datanode, then this issue does not arise.

We need to device some other means to support non-shippable triggers for
COPY. May be we would end up sending INSERT commands on the datanode
instead of COPY command, if there are non-shippable triggers. Each of the
data row will be sent as parameters to the insert query. This operation
would be slow, but possible.

Hi Abbas,
I am also seeing a lot of changes in the expected output where the rows
output have changed. What are these changes?


On Thu, May 16, 2013 at 2:55 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi Abbas,
> Instead of fixing the first issue in pgxc_build_dml_statement(), is it
> possible to traverse the Query in validate_part_col_updatable() recursively
> to find UPDATE statements and apply partition column check? That would
> cover all the possibilities, I guess. That also saves us much effort in
> case we come to support distribution column updation.
>
> I think, we need a generic solution to solve this command id issue, e.g.
> punching command id always and efficiently. But for now this suffices.
> Please log a bug/feature and put it in 1.2 bucket.
>
>
>
>
> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt <abb...@en...>wrote:
>
>> Adding developers mailing list.
>>
>>
>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt <abb...@en...>wrote:
>>
>>> Hi,
>>> Attached please find a patch to fix test case with.
>>> There were two issues making the test to fail.
>>> 1. Updates to partition column were possible using syntax like
>>>     WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t
>>>     The patch blocks this syntax.
>>>
>>> 2.  For a WITH query that updates a table in the main query and
>>>      inserts a row in the same table in the WITH query we need to use
>>>      command ID communication to remote nodes in order to
>>>      maintain global data visibility.
>>>      For example
>>>      CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY REPLICATION;
>>>      INSERT INTO tab VALUES (1,'p1');
>>>      WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS
>>> newid)
>>>      UPDATE tab SET id = id + newid FROM wcte;
>>>      The last query gets translated into the following multi-statement
>>>      transaction on the primary datanode
>>>      (a) START TRANSACTION ISOLATION LEVEL read committed READ WRITE
>>>      (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id --
>>> (42,'new)'
>>>      (c) SELECT id, val, ctid FROM ONLY tab WHERE true
>>>      (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) --
>>> (43,(0,1)]
>>>      (e) COMMIT TRANSACTION
>>>      The command id of the select in step (c), should be such that
>>>      it does not see the insert of step (b)
>>>
>>> Comments are welcome.
>>>
>>> Regards
>>>
>>> --
>>> *Abbas*
>>>  Architect
>>>
>>> Ph: 92.334.5100153
>>>  Skype ID: gabbasb
>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
>>> *
>>> Follow us on Twitter*
>>> @EnterpriseDB
>>>
>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>>>
>>
>>
>>
>> --
>> --
>> *Abbas*
>>  Architect
>>
>> Ph: 92.334.5100153
>>  Skype ID: gabbasb
>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
>> *
>> Follow us on Twitter*
>> @EnterpriseDB
>>
>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>>
>>
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-core mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

Hi Abbas,
Instead of fixing the first issue in pgxc_build_dml_statement(), is it
possible to traverse the Query in validate_part_col_updatable() recursively
to find UPDATE statements and apply partition column check? That would
cover all the possibilities, I guess. That also saves us much effort in
case we come to support distribution column updation.

I think, we need a generic solution to solve this command id issue, e.g.
punching command id always and efficiently. But for now this suffices.
Please log a bug/feature and put it in 1.2 bucket.




On Wed, May 15, 2013 at 5:31 AM, Abbas Butt <abb...@en...>wrote:

> Adding developers mailing list.
>
>
> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt <abb...@en...>wrote:
>
>> Hi,
>> Attached please find a patch to fix test case with.
>> There were two issues making the test to fail.
>> 1. Updates to partition column were possible using syntax like
>>     WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t
>>     The patch blocks this syntax.
>>
>> 2.  For a WITH query that updates a table in the main query and
>>      inserts a row in the same table in the WITH query we need to use
>>      command ID communication to remote nodes in order to
>>      maintain global data visibility.
>>      For example
>>      CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY REPLICATION;
>>      INSERT INTO tab VALUES (1,'p1');
>>      WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id AS newid)
>>      UPDATE tab SET id = id + newid FROM wcte;
>>      The last query gets translated into the following multi-statement
>>      transaction on the primary datanode
>>      (a) START TRANSACTION ISOLATION LEVEL read committed READ WRITE
>>      (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id --
>> (42,'new)'
>>      (c) SELECT id, val, ctid FROM ONLY tab WHERE true
>>      (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) --
>> (43,(0,1)]
>>      (e) COMMIT TRANSACTION
>>      The command id of the select in step (c), should be such that
>>      it does not see the insert of step (b)
>>
>> Comments are welcome.
>>
>> Regards
>>
>> --
>> *Abbas*
>>  Architect
>>
>> Ph: 92.334.5100153
>>  Skype ID: gabbasb
>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
>> *
>> Follow us on Twitter*
>> @EnterpriseDB
>>
>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>>
>
>
>
> --
> --
> *Abbas*
>  Architect
>
> Ph: 92.334.5100153
>  Skype ID: gabbasb
> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
> *
> Follow us on Twitter*
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-core mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

On Thu, May 16, 2013 at 12:00 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi Abbas,
> I didn't check every place that needed replacement. But it will be caught
> in compilation I guess. I have few comments
> 1. Instead of name has_to_save_cmd_id, can we use save_cmd_id or pass
> command id?
>

Sure we can use the name you suggest.


> 2. The comment against this variable is still the old, it needs to change,
> with the change in the name?
>

I had modified the comment against the variable to add some more lines in
it. I checked that the change is in the patch I sent earlier. Please see if
those changes are sufficient.


>
> Why that member needs to be in Query structure? I think we have discussed
> this when we added the member, but I think, at that time, we added it there
> to avoid last minute complications. Is there a better place?
>

We catch the statements requiring command id exchange in transformation
stage e.g. the WITH query needing command id exchange is caught in
transformUpdateStmt, and this is required in RemoteQuery, so we had decided
that a member is Query structure will be required. Do you see some other
way of conveying this information from transformUpdateStmt to
create_remotequery_plan?


>
>
> On Wed, May 15, 2013 at 5:30 AM, Abbas Butt <abb...@en...>wrote:
>
>> Adding developers mailing list.
>>
>>
>> On Wed, May 15, 2013 at 4:51 AM, Abbas Butt <abb...@en...>wrote:
>>
>>> Hi,
>>> PFA a patch to change names of a couple of variable to more general
>>> names.
>>> This patch touches some of the areas of command id exchange mechanism
>>> between nodes of the cluster and we can use this thread to discuss whether
>>> we need to enable command id exchange in all cases OR not.
>>> For reference see the patch 38b2b79 committed by Michael for the rest of
>>> the details of the mechanism.
>>>
>>> --
>>> *Abbas*
>>>  Architect
>>>
>>> Ph: 92.334.5100153
>>>  Skype ID: gabbasb
>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
>>> *
>>> Follow us on Twitter*
>>> @EnterpriseDB
>>>
>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>>>
>>
>>
>>
>> --
>> --
>> *Abbas*
>>  Architect
>>
>> Ph: 92.334.5100153
>>  Skype ID: gabbasb
>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
>> *
>> Follow us on Twitter*
>> @EnterpriseDB
>>
>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>>
>>
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-core mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>



-- 
-- 
*Abbas*
 Architect

Ph: 92.334.5100153
 Skype ID: gabbasb
www.enterprisedb.co
<http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
*
Follow us on Twitter*
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars,
whitepapers<http://www.enterprisedb.com/resources-community>and
more<http://www.enterprisedb.com/resources-community>

Hi Abbas,
I didn't check every place that needed replacement. But it will be caught
in compilation I guess. I have few comments
1. Instead of name has_to_save_cmd_id, can we use save_cmd_id or pass
command id?
2. The comment against this variable is still the old, it needs to change,
with the change in the name?

Why that member needs to be in Query structure? I think we have discussed
this when we added the member, but I think, at that time, we added it there
to avoid last minute complications. Is there a better place?


On Wed, May 15, 2013 at 5:30 AM, Abbas Butt <abb...@en...>wrote:

> Adding developers mailing list.
>
>
> On Wed, May 15, 2013 at 4:51 AM, Abbas Butt <abb...@en...>wrote:
>
>> Hi,
>> PFA a patch to change names of a couple of variable to more general names.
>> This patch touches some of the areas of command id exchange mechanism
>> between nodes of the cluster and we can use this thread to discuss whether
>> we need to enable command id exchange in all cases OR not.
>> For reference see the patch 38b2b79 committed by Michael for the rest of
>> the details of the mechanism.
>>
>> --
>> *Abbas*
>>  Architect
>>
>> Ph: 92.334.5100153
>>  Skype ID: gabbasb
>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
>> *
>> Follow us on Twitter*
>> @EnterpriseDB
>>
>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>>
>
>
>
> --
> --
> *Abbas*
>  Architect
>
> Ph: 92.334.5100153
>  Skype ID: gabbasb
> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
> *
> Follow us on Twitter*
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-core mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

Thank you Ashutosh.

----------
Koichi Suzuki


2013/5/15 Ashutosh Bapat <ash...@en...>

> I committed this patch with some changes.
>
>
> On Wed, May 15, 2013 at 1:28 PM, 鈴木 幸市 <ko...@in...> wrote:
>
>> Here's third try.
>>
>> I just left all the explain with (….) option as is and just added
>> (verbose on, num_nodes off, nodes off, costs off) option to only one
>> explain statement which simply issued explain verbose.
>>
>> Hope this patch makes sense.
>>  ---
>> Koichi Suzuki
>>
>>
>>
>> On 2013/05/15, at 16:33, 鈴木 幸市 <ko...@in...> wrote:
>>
>> Verbose appeared in the old files.  Should I remove all the "verbose"?
>>
>> Regards;
>>  ---
>> Koichi Suzuki
>>
>>
>>
>> On 2013/05/15, at 14:58, Ashutosh Bapat <ash...@en...>
>> wrote:
>>
>> Hi Suzuki-san,
>> There are syntax errors in expected output in your patch. The right way
>> to invoke EXPLAIN is
>> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) <query>
>>
>> In your patch it is used as
>> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) verbose <query>
>> that's why it's giving syntax error.
>>
>> Also, you have use cost off instead of cost*s* off.
>>
>>
>> On Wed, May 15, 2013 at 11:01 AM, 鈴木 幸市 <ko...@in...> wrote:
>>
>>> Here's the second fix.
>>>
>>> Regards;
>>>  ---
>>> Koichi Suzuki
>>>
>>>
>>>
>>>
>>> On 2013/05/15, at 13:22, Ashutosh Bapat <ash...@en...>
>>> wrote:
>>>
>>> Again in this case, the right fix is to use the standard EXPLAIN form
>>> EXPLAIN  (verbose, num_nodes off, nodes off, costs off).
>>>
>>> What I don't understand is how come these changes appeared suddenly?
>>>
>>>
>>> On Wed, May 15, 2013 at 7:59 AM, Koichi Suzuki <
>>> koi...@gm...> wrote:
>>>
>>>> Hello;
>>>>
>>>> This failure was caused by the following:
>>>>
>>>> 1. Now FQS pushes down order by and there're no sort at coordinator.
>>>>  Current regression did not reflect this.
>>>>
>>>> 2. Regression test changed the datanode name.  Current regression did
>>>> not reflect this.
>>>>
>>>> Enclosed is a fix.
>>>>
>>>> Regards;
>>>> ----------
>>>> Koichi Suzuki
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> AlienVault Unified Security Management (USM) platform delivers complete
>>>> security visibility with the essential security capabilities. Easily and
>>>> efficiently configure, manage, and operate all of your security controls
>>>> from a single console and one unified framework. Download a free trial.
>>>> http://p.sf.net/sfu/alienvault_d2d
>>>> _______________________________________________
>>>> Postgres-xc-developers mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Postgres Database Company
>>> ------------------------------------------------------------------------------
>>> AlienVault Unified Security Management (USM) platform delivers complete
>>> security visibility with the essential security capabilities. Easily and
>>> efficiently configure, manage, and operate all of your security controls
>>> from a single console and one unified framework. Download a free trial.
>>>
>>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>>
>>
>>
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>>
>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>

Thanks a lot;

----------
Koichi Suzuki


2013/5/15 Ashutosh Bapat <ash...@en...>

> I committed this patch, with slight changes. I tested it on build farm to
> make sure that the test passes there as well.
>
>
> On Wed, May 15, 2013 at 2:07 PM, 鈴木 幸市 <ko...@in...> wrote:
>
>> Sorry for bothering.   This time, I only applied  num_nodes off only to
>> the statement which is shipped to more than one datanode.  There was two of
>> them.
>>
>> Hope this makes sense.
>>
>> Regards;
>>  ---
>> Koichi Suzuki
>>
>>
>>
>> On 2013/05/15, at 15:04, Ashutosh Bapat <ash...@en...>
>> wrote:
>>
>> Hi Suzuki-san,
>>
>>
>>
>> On Wed, May 15, 2013 at 11:15 AM, 鈴木 幸市 <ko...@in...> wrote:
>>
>>> I see.   PFA the revised patch.   Some num_node off was missing in
>>> explain statements.  They're fixed too.
>>>
>>>
>>>
>> You need to apply changes only to the queries which are showing diffs
>> because of difference in number of nodes. But it seems you have changed all
>> the explain commands in the file. That's not needed.
>>
>> As I said in my previous mail, there are some explain outputs where the
>> query is shipped to only one node and the node count in explain output will
>> be always 1 for such queries. That doesn't change with the cluster
>> configuration. For such queries, it's important that we have num_nodes off.
>>
>>
>>>  ---
>>> Koichi Suzuki
>>>
>>>
>>>
>>> On 2013/05/15, at 13:17, Ashutosh Bapat <ash...@en...>
>>> wrote:
>>>
>>> That's not a correct fix. We should turn off the node count in Explain
>>> output. Use EXPLAIN  (verbose, num_nodes off, nodes off, costs off) as a
>>> general rule. In case of single node reduction tests, we may use num_nodes
>>> on, but in that case num_nodes will always be 1, and will not change
>>> depending upon cluster configuration.
>>>
>>>
>>> On Wed, May 15, 2013 at 7:56 AM, Koichi Suzuki <
>>> koi...@gm...> wrote:
>>>
>>>> I found the regression failure was due to the change of explain output.
>>>>   Original expected output uses node count = 3, which is not correct
>>>> because regression test is configured with only two datanode.   Actual
>>>> result says node count = 2 which is correct.
>>>>
>>>> Attached is a patch for the correction.
>>>>
>>>> Regards;
>>>> ----------
>>>> Koichi Suzuki
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> AlienVault Unified Security Management (USM) platform delivers complete
>>>> security visibility with the essential security capabilities. Easily and
>>>> efficiently configure, manage, and operate all of your security controls
>>>> from a single console and one unified framework. Download a free trial.
>>>> http://p.sf.net/sfu/alienvault_d2d
>>>> _______________________________________________
>>>> Postgres-xc-developers mailing list
>>>> Pos...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Postgres Database Company
>>> ------------------------------------------------------------------------------
>>> AlienVault Unified Security Management (USM) platform delivers complete
>>> security visibility with the essential security capabilities. Easily and
>>> efficiently configure, manage, and operate all of your security controls
>>> from a single console and one unified framework. Download a free trial.
>>>
>>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>>
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>

I committed this patch with some changes.


On Wed, May 15, 2013 at 1:28 PM, 鈴木 幸市 <ko...@in...> wrote:

> Here's third try.
>
> I just left all the explain with (….) option as is and just added (verbose
> on, num_nodes off, nodes off, costs off) option to only one explain
> statement which simply issued explain verbose.
>
> Hope this patch makes sense.
> ---
> Koichi Suzuki
>
>
>
> On 2013/05/15, at 16:33, 鈴木 幸市 <ko...@in...> wrote:
>
> Verbose appeared in the old files.  Should I remove all the "verbose"?
>
> Regards;
> ---
> Koichi Suzuki
>
>
>
> On 2013/05/15, at 14:58, Ashutosh Bapat <ash...@en...>
> wrote:
>
> Hi Suzuki-san,
> There are syntax errors in expected output in your patch. The right way to
> invoke EXPLAIN is
> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) <query>
>
> In your patch it is used as
> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) verbose <query>
> that's why it's giving syntax error.
>
> Also, you have use cost off instead of cost*s* off.
>
>
> On Wed, May 15, 2013 at 11:01 AM, 鈴木 幸市 <ko...@in...> wrote:
>
>> Here's the second fix.
>>
>> Regards;
>>  ---
>> Koichi Suzuki
>>
>>
>>
>>
>> On 2013/05/15, at 13:22, Ashutosh Bapat <ash...@en...>
>> wrote:
>>
>> Again in this case, the right fix is to use the standard EXPLAIN form
>> EXPLAIN  (verbose, num_nodes off, nodes off, costs off).
>>
>> What I don't understand is how come these changes appeared suddenly?
>>
>>
>> On Wed, May 15, 2013 at 7:59 AM, Koichi Suzuki <koi...@gm...
>> > wrote:
>>
>>> Hello;
>>>
>>> This failure was caused by the following:
>>>
>>> 1. Now FQS pushes down order by and there're no sort at coordinator.
>>>  Current regression did not reflect this.
>>>
>>> 2. Regression test changed the datanode name.  Current regression did
>>> not reflect this.
>>>
>>> Enclosed is a fix.
>>>
>>> Regards;
>>> ----------
>>> Koichi Suzuki
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> AlienVault Unified Security Management (USM) platform delivers complete
>>> security visibility with the essential security capabilities. Easily and
>>> efficiently configure, manage, and operate all of your security controls
>>> from a single console and one unified framework. Download a free trial.
>>> http://p.sf.net/sfu/alienvault_d2d
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>>
>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
>
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

I committed this patch, with slight changes. I tested it on build farm to
make sure that the test passes there as well.


On Wed, May 15, 2013 at 2:07 PM, 鈴木 幸市 <ko...@in...> wrote:

> Sorry for bothering.   This time, I only applied  num_nodes off only to
> the statement which is shipped to more than one datanode.  There was two of
> them.
>
> Hope this makes sense.
>
> Regards;
> ---
> Koichi Suzuki
>
>
>
> On 2013/05/15, at 15:04, Ashutosh Bapat <ash...@en...>
> wrote:
>
> Hi Suzuki-san,
>
>
>
> On Wed, May 15, 2013 at 11:15 AM, 鈴木 幸市 <ko...@in...> wrote:
>
>> I see.   PFA the revised patch.   Some num_node off was missing in
>> explain statements.  They're fixed too.
>>
>>
>>
> You need to apply changes only to the queries which are showing diffs
> because of difference in number of nodes. But it seems you have changed all
> the explain commands in the file. That's not needed.
>
> As I said in my previous mail, there are some explain outputs where the
> query is shipped to only one node and the node count in explain output will
> be always 1 for such queries. That doesn't change with the cluster
> configuration. For such queries, it's important that we have num_nodes off.
>
>
>>  ---
>> Koichi Suzuki
>>
>>
>>
>> On 2013/05/15, at 13:17, Ashutosh Bapat <ash...@en...>
>> wrote:
>>
>> That's not a correct fix. We should turn off the node count in Explain
>> output. Use EXPLAIN  (verbose, num_nodes off, nodes off, costs off) as a
>> general rule. In case of single node reduction tests, we may use num_nodes
>> on, but in that case num_nodes will always be 1, and will not change
>> depending upon cluster configuration.
>>
>>
>> On Wed, May 15, 2013 at 7:56 AM, Koichi Suzuki <koi...@gm...
>> > wrote:
>>
>>> I found the regression failure was due to the change of explain output.
>>>   Original expected output uses node count = 3, which is not correct
>>> because regression test is configured with only two datanode.   Actual
>>> result says node count = 2 which is correct.
>>>
>>> Attached is a patch for the correction.
>>>
>>> Regards;
>>> ----------
>>> Koichi Suzuki
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> AlienVault Unified Security Management (USM) platform delivers complete
>>> security visibility with the essential security capabilities. Easily and
>>> efficiently configure, manage, and operate all of your security controls
>>> from a single console and one unified framework. Download a free trial.
>>> http://p.sf.net/sfu/alienvault_d2d
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>
>>
>> --
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>>
>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

Sorry for bothering.   This time, I only applied  num_nodes off only to the statement which is shipped to more than one datanode.  There was two of them.

Hope this makes sense.

Regards;
---
Koichi Suzuki


On 2013/05/15, at 15:04, Ashutosh Bapat <ash...@en...> wrote:

> Hi Suzuki-san,
> 
> 
> 
> On Wed, May 15, 2013 at 11:15 AM, 鈴木 幸市 <ko...@in...> wrote:
> I see.   PFA the revised patch.   Some num_node off was missing in explain statements.  They're fixed too.
> 
> 
> 
> You need to apply changes only to the queries which are showing diffs because of difference in number of nodes. But it seems you have changed all the explain commands in the file. That's not needed.
> 
> As I said in my previous mail, there are some explain outputs where the query is shipped to only one node and the node count in explain output will be always 1 for such queries. That doesn't change with the cluster configuration. For such queries, it's important that we have num_nodes off.
>  
> ---
> Koichi Suzuki
> 
> 
> 
> On 2013/05/15, at 13:17, Ashutosh Bapat <ash...@en...> wrote:
> 
>> That's not a correct fix. We should turn off the node count in Explain output. Use EXPLAIN  (verbose, num_nodes off, nodes off, costs off) as a general rule. In case of single node reduction tests, we may use num_nodes on, but in that case num_nodes will always be 1, and will not change depending upon cluster configuration.
>> 
>> 
>> On Wed, May 15, 2013 at 7:56 AM, Koichi Suzuki <koi...@gm...> wrote:
>> I found the regression failure was due to the change of explain output.   Original expected output uses node count = 3, which is not correct because regression test is configured with only two datanode.   Actual result says node count = 2 which is correct.
>> 
>> Attached is a patch for the correction.
>> 
>> Regards;
>> ----------
>> Koichi Suzuki
>> 
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> 
>> 
>> 
>> 
>> -- 
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> 
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company

Here's third try.

I just left all the explain with (….) option as is and just added (verbose on, num_nodes off, nodes off, costs off) option to only one explain statement which simply issued explain verbose.

Hope this patch makes sense.
---
Koichi Suzuki


On 2013/05/15, at 16:33, 鈴木 幸市 <ko...@in...> wrote:

> Verbose appeared in the old files.  Should I remove all the "verbose"?
> 
> Regards;
> ---
> Koichi Suzuki
> 
> 
> 
> On 2013/05/15, at 14:58, Ashutosh Bapat <ash...@en...> wrote:
> 
>> Hi Suzuki-san,
>> There are syntax errors in expected output in your patch. The right way to invoke EXPLAIN is
>> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) <query>
>> 
>> In your patch it is used as 
>> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) verbose <query>
>> that's why it's giving syntax error.
>> 
>> Also, you have use cost off instead of costs off.
>> 
>> 
>> On Wed, May 15, 2013 at 11:01 AM, 鈴木 幸市 <ko...@in...> wrote:
>> Here's the second fix.
>> 
>> Regards;
>> ---
>> Koichi Suzuki
>> 
>> 
>> 
>> 
>> On 2013/05/15, at 13:22, Ashutosh Bapat <ash...@en...> wrote:
>> 
>>> Again in this case, the right fix is to use the standard EXPLAIN form EXPLAIN  (verbose, num_nodes off, nodes off, costs off).
>>> 
>>> What I don't understand is how come these changes appeared suddenly?
>>> 
>>> 
>>> On Wed, May 15, 2013 at 7:59 AM, Koichi Suzuki <koi...@gm...> wrote:
>>> Hello;
>>> 
>>> This failure was caused by the following:
>>> 
>>> 1. Now FQS pushes down order by and there're no sort at coordinator.  Current regression did not reflect this.
>>> 
>>> 2. Regression test changed the datanode name.  Current regression did not reflect this.
>>> 
>>> Enclosed is a fix.
>>> 
>>> Regards;
>>> ----------
>>> Koichi Suzuki
>>> 
>>> ------------------------------------------------------------------------------
>>> AlienVault Unified Security Management (USM) platform delivers complete
>>> security visibility with the essential security capabilities. Easily and
>>> efficiently configure, manage, and operate all of your security controls
>>> from a single console and one unified framework. Download a free trial.
>>> http://p.sf.net/sfu/alienvault_d2d
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Postgres Database Company
>>> ------------------------------------------------------------------------------
>>> AlienVault Unified Security Management (USM) platform delivers complete
>>> security visibility with the essential security capabilities. Easily and
>>> efficiently configure, manage, and operate all of your security controls
>>> from a single console and one unified framework. Download a free trial.
>>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> 
>> 
>> 
>> 
>> 
>> -- 
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
> 
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Verbose appeared in the old files.  Should I remove all the "verbose"?

Regards;
---
Koichi Suzuki



On 2013/05/15, at 14:58, Ashutosh Bapat <ash...@en...> wrote:

> Hi Suzuki-san,
> There are syntax errors in expected output in your patch. The right way to invoke EXPLAIN is
> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) <query>
> 
> In your patch it is used as 
> EXPLAIN  (verbose, num_nodes off, nodes off, costs off) verbose <query>
> that's why it's giving syntax error.
> 
> Also, you have use cost off instead of costs off.
> 
> 
> On Wed, May 15, 2013 at 11:01 AM, 鈴木 幸市 <ko...@in...> wrote:
> Here's the second fix.
> 
> Regards;
> ---
> Koichi Suzuki
> 
> 
> 
> 
> On 2013/05/15, at 13:22, Ashutosh Bapat <ash...@en...> wrote:
> 
>> Again in this case, the right fix is to use the standard EXPLAIN form EXPLAIN  (verbose, num_nodes off, nodes off, costs off).
>> 
>> What I don't understand is how come these changes appeared suddenly?
>> 
>> 
>> On Wed, May 15, 2013 at 7:59 AM, Koichi Suzuki <koi...@gm...> wrote:
>> Hello;
>> 
>> This failure was caused by the following:
>> 
>> 1. Now FQS pushes down order by and there're no sort at coordinator.  Current regression did not reflect this.
>> 
>> 2. Regression test changed the datanode name.  Current regression did not reflect this.
>> 
>> Enclosed is a fix.
>> 
>> Regards;
>> ----------
>> Koichi Suzuki
>> 
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> 
>> 
>> 
>> 
>> -- 
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> 
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company

In XC, the way COPY is implemented is that for each record, we read the
whole line into memory, and then pass it to the datanode as-is. If there
are non-shippable default column expressions, we evaluate the default
values , convert them into output form, and append them to the data row.

In presence of BR triggers, currently the ExecBRInsertTriggers() do not get
called because of the way we skip the whole PG code block; instead we just
send the data row as-is, optionally appending default values into the data
row.

What we need to do is; convert the tuple returned by ExecBRTriggers into
text data row, but the text data should be in COPY format. This is because
we need to send the data row to the datanode using COPY command, so it
requires correct COPY format, such as escape sequences.

For this, we need to call the function CopyOneRowTo() that is being used by
COPY TO. This will make sure it will emit the data row in the COPY format.
But we need to create a temporary CopyState because CopyOneRowTo() needs
it. We can derive it from the current CopyState that is already created for
COPY FROM. Most of the fields remain the same, except we need to re-assign
CopyState->line_buf, and CopyState->rowcontext.

This will save us from writing code to make sure the new output data row
generated by BR triggers complies with COPY data format.

I had already done similar thing for appending default values into the data
row. We call functions like CopyAttributeOutCSV(), CopyInt32() to append
the values to the data row in COPY format. There, we did not require
CopyOneRow() because we did not require the complete row, we needed to
append only a subset of columns to the existing data row.

Comments/suggestions welcome.

Hi Amit,
Here are comment on trig_shippability patch.

1. The function pgxc_trigevent_quickfind() needs a better name like
pgxc_has_trigger_for_event() to convey the functionality clearly.
 2. The prologue of function pgxc_should_exec_triggers() has all the
necessary content, but it needs to be written in a better order. The
prologue talks about a single trigger (to be executed) but in reality the
function is executed for checking whether all the triggers matching the
given criteria and belonging to a given relation are firable or not. The
prologue needs to be corrected. Also, first specify why is such a check
needed (trigger order being alphabetical makes it necessary that all or
none of the triggers execute on the same node).
3. pgxc_is_inttrigger_firable(), the function name needs to change a bit.
The int in inttrigger can be easily associated with integer instead of
internal. Can you please change the name to use internal_ or intern_
instead of int. If the name becomes too long, please use suitable prefix.
4. RemoteQueryNext() is called multiple times; the after statement triggers
too will be called those many times. Right now this does not happen since
there is no way the function will be called multiple times for a DML. But
in future, we will start supporting RETURNING with FQS, in which case
RemoteQueryNext will be called multiple times.

test xc_trigship

1. we are using same name xc_auditlog for table and function. Please use
different names.
2. For every object (table/function etc) that the test creates, please
mention the purpose of that object. That helps validating the object
definition there and there itself.
 3. I couldn't understand, how does the test check where the trigger is
being fired?



On Mon, May 13, 2013 at 2:36 PM, Ashutosh Bapat <
ash...@en...> wrote:

> Hi Amit,
> We now have Query structure in RemoteQuery. This corresponds to the query
> being fired on the datanodes. THe commandType here should tell you whether
> it's DELETE or not.
>
>
> On Mon, May 13, 2013 at 1:14 PM, Amit Khandekar <
> ami...@en...> wrote:
>
>> In case of multiple triggers of the same type for a given table, the
>> requirement is that they should be fired in alphabetical order. To do so,
>> we need to fire either all of them on coordinator, or all of them on
>> datanode. The main changes in the attached patch are related to this
>> requirement.
>>
>> All of the Exec*Trigger() functions now execute should_exec_trigger*()
>> functions that return true if the node on which they are run is the right
>> node to execute those type of triggers.
>>
>> Also for BR triggers, the additional requirement is that they should be
>> run on coordinator if AR triggers are not shippable, regardless of whether
>> BR themselves are shippable or not. This is because, if we fire BR triggers
>> on datanode, they might change the final row updated, and so we need to
>> again fetch the new row back to the coordinator. Instead, if we fire them
>> on coordinator, we already know what's the final row. Also, we would have
>> required additional changes to add RETURNING to the remote query to fetch
>> the final updated row.
>>
>> Constraint triggers are exception; we need to fire them always on
>> datanode. Once we support global constraints, these need not be specially
>> handled.
>>
>> I was trying to avoid the should_exec_trigger*() calls for each of the
>> Exec*Trigger() functions by doing this in TriggerEnabled() because it is a
>> common function called for all triggers. But the issue is, for AFTER
>> triggers, it is called with a different set of  event type values than the
>> usual  TRIGGER_TYPE* values. For AFTER triggers, TRIGGER_EVENT_* values are
>> used. Anyways, TriggerEnabled() is called for each of the trigger.
>>
>> The trigger shippability helper functions are now completely changed.
>> pgxc_find_nonshippable_row_trig() is the key function. The comments in
>> these functions should give a fair idea of their functionality.
>>
>>
>> For stmt shippability, the trigger functions are now explicitly called if
>> it's not an internally generated query. rq_internal_params field is used to
>> know that this DML is user-supplied query (that is, it would be FQS). The
>> functions are called in RemoteQueryNext().
>>
>>
>> THere is another patch (relaccess_type.patch) that you need to first
>> apply before the main patch (trig_shippability.patch) is applied. I had to
>> add another RELATION_ACCESS_DELETE in ExecNodes. To handle the stmt
>> triggers, I had to know whether the statement is DELETE or INSERT or UPDATE.
>>
>> The new test xc_trigship is added.
>>
>>
>> ------------------------------------------------------------------------------
>> Learn Graph Databases - Download FREE O'Reilly Book
>> "Graph Databases" is the definitive new guide to graph databases and
>> their applications. This 200-page book is written by three acclaimed
>> leaders in the field. The early access version is available now.
>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
>



-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

Hi Suzuki-san,



On Wed, May 15, 2013 at 11:15 AM, 鈴木 幸市 <ko...@in...> wrote:

> I see.   PFA the revised patch.   Some num_node off was missing in explain
> statements.  They're fixed too.
>
>
>
You need to apply changes only to the queries which are showing diffs
because of difference in number of nodes. But it seems you have changed all
the explain commands in the file. That's not needed.

As I said in my previous mail, there are some explain outputs where the
query is shipped to only one node and the node count in explain output will
be always 1 for such queries. That doesn't change with the cluster
configuration. For such queries, it's important that we have num_nodes off.


> ---
> Koichi Suzuki
>
>
>
> On 2013/05/15, at 13:17, Ashutosh Bapat <ash...@en...>
> wrote:
>
> That's not a correct fix. We should turn off the node count in Explain
> output. Use EXPLAIN  (verbose, num_nodes off, nodes off, costs off) as a
> general rule. In case of single node reduction tests, we may use num_nodes
> on, but in that case num_nodes will always be 1, and will not change
> depending upon cluster configuration.
>
>
> On Wed, May 15, 2013 at 7:56 AM, Koichi Suzuki <koi...@gm...>wrote:
>
>> I found the regression failure was due to the change of explain output.
>> Original expected output uses node count = 3, which is not correct because
>> regression test is configured with only two datanode.   Actual result says
>> node count = 2 which is correct.
>>
>> Attached is a patch for the correction.
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
>
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

Hi Suzuki-san,
There are syntax errors in expected output in your patch. The right way to
invoke EXPLAIN is
EXPLAIN  (verbose, num_nodes off, nodes off, costs off) <query>

In your patch it is used as
EXPLAIN  (verbose, num_nodes off, nodes off, costs off) verbose <query>
that's why it's giving syntax error.

Also, you have use cost off instead of cost*s* off.


On Wed, May 15, 2013 at 11:01 AM, 鈴木 幸市 <ko...@in...> wrote:

> Here's the second fix.
>
> Regards;
> ---
> Koichi Suzuki
>
>
>
>
> On 2013/05/15, at 13:22, Ashutosh Bapat <ash...@en...>
> wrote:
>
> Again in this case, the right fix is to use the standard EXPLAIN form
> EXPLAIN  (verbose, num_nodes off, nodes off, costs off).
>
> What I don't understand is how come these changes appeared suddenly?
>
>
> On Wed, May 15, 2013 at 7:59 AM, Koichi Suzuki <koi...@gm...>wrote:
>
>> Hello;
>>
>> This failure was caused by the following:
>>
>> 1. Now FQS pushes down order by and there're no sort at coordinator.
>>  Current regression did not reflect this.
>>
>> 2. Regression test changed the datanode name.  Current regression did not
>> reflect this.
>>
>> Enclosed is a fix.
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
>
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EntepriseDB Corporation
The Postgres Database Company

Here's revised fix.   num_nodes off was missing in several explain statements.  This fixes missing options too.

Regards;
---
Koichi Suzuki


On 2013/05/15, at 14:31, 鈴木 幸市 <ko...@in...> wrote:

> Here's the second fix.
> 
> Regards;
> ---
> Koichi Suzuki
> 
> 
> <fix_xc_sort_1.patch>
> 
> On 2013/05/15, at 13:22, Ashutosh Bapat <ash...@en...> wrote:
> 
>> Again in this case, the right fix is to use the standard EXPLAIN form EXPLAIN  (verbose, num_nodes off, nodes off, costs off).
>> 
>> What I don't understand is how come these changes appeared suddenly?
>> 
>> 
>> On Wed, May 15, 2013 at 7:59 AM, Koichi Suzuki <koi...@gm...> wrote:
>> Hello;
>> 
>> This failure was caused by the following:
>> 
>> 1. Now FQS pushes down order by and there're no sort at coordinator.  Current regression did not reflect this.
>> 
>> 2. Regression test changed the datanode name.  Current regression did not reflect this.
>> 
>> Enclosed is a fix.
>> 
>> Regards;
>> ----------
>> Koichi Suzuki
>> 
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>> 
>> 
>> 
>> 
>> -- 
>> Best Wishes,
>> Ashutosh Bapat
>> EntepriseDB Corporation
>> The Postgres Database Company
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers

Ah, I just received the remaining emails of the chain.

On Wed, May 15, 2013 at 10:53 AM, Abbas Butt <abb...@en...>wrote:

> Shouldn't we use
> NODES OFF, NUM_NODES OFF
> so that the test would pass independent of number of nodes in the cluster.
>
> On Wed, May 15, 2013 at 7:26 AM, Koichi Suzuki <koi...@gm...>wrote:
>
>> I found the regression failure was due to the change of explain output.
>> Original expected output uses node count = 3, which is not correct because
>> regression test is configured with only two datanode.   Actual result says
>> node count = 2 which is correct.
>>
>> Attached is a patch for the correction.
>>
>> Regards;
>> ----------
>> Koichi Suzuki
>>
>>
>> ------------------------------------------------------------------------------
>> AlienVault Unified Security Management (USM) platform delivers complete
>> security visibility with the essential security capabilities. Easily and
>> efficiently configure, manage, and operate all of your security controls
>> from a single console and one unified framework. Download a free trial.
>> http://p.sf.net/sfu/alienvault_d2d
>> _______________________________________________
>> Postgres-xc-developers mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>
>>
>
>
> --
> --
> *Abbas*
>  Architect
>
> Ph: 92.334.5100153
>  Skype ID: gabbasb
> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
> *
> Follow us on Twitter*
> @EnterpriseDB
>
> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community>
>



-- 
-- 
*Abbas*
 Architect

Ph: 92.334.5100153
 Skype ID: gabbasb
www.enterprisedb.co
<http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
*
Follow us on Twitter*
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars,
whitepapers<http://www.enterprisedb.com/resources-community>and
more<http://www.enterprisedb.com/resources-community>

Shouldn't we use
NODES OFF, NUM_NODES OFF
so that the test would pass independent of number of nodes in the cluster.

On Wed, May 15, 2013 at 7:26 AM, Koichi Suzuki <koi...@gm...>wrote:

> I found the regression failure was due to the change of explain output.
> Original expected output uses node count = 3, which is not correct because
> regression test is configured with only two datanode.   Actual result says
> node count = 2 which is correct.
>
> Attached is a patch for the correction.
>
> Regards;
> ----------
> Koichi Suzuki
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
-- 
*Abbas*
 Architect

Ph: 92.334.5100153
 Skype ID: gabbasb
www.enterprisedb.co
<http://www.enterprisedb.com/>m<http://www.enterprisedb.com/>
*
Follow us on Twitter*
@EnterpriseDB

Visit EnterpriseDB for tutorials, webinars,
whitepapers<http://www.enterprisedb.com/resources-community>and
more<http://www.enterprisedb.com/resources-community>

I see.   PFA the revised patch.   Some num_node off was missing in explain statements.  They're fixed too.


---
Koichi Suzuki


On 2013/05/15, at 13:17, Ashutosh Bapat <ash...@en...> wrote:

> That's not a correct fix. We should turn off the node count in Explain output. Use EXPLAIN  (verbose, num_nodes off, nodes off, costs off) as a general rule. In case of single node reduction tests, we may use num_nodes on, but in that case num_nodes will always be 1, and will not change depending upon cluster configuration.
> 
> 
> On Wed, May 15, 2013 at 7:56 AM, Koichi Suzuki <koi...@gm...> wrote:
> I found the regression failure was due to the change of explain output.   Original expected output uses node count = 3, which is not correct because regression test is configured with only two datanode.   Actual result says node count = 2 which is correct.
> 
> Attached is a patch for the correction.
> 
> Regards;
> ----------
> Koichi Suzuki
> 
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
> 
> 
> 
> 
> -- 
> Best Wishes,
> Ashutosh Bapat
> EntepriseDB Corporation
> The Postgres Database Company
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d_______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers