Menu
▾
▴
postgres-xc-developers — Postgres-XC hackers and developers
You can subscribe to this list here.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
(10) |
May
(17) |
Jun
(3) |
Jul
|
Aug
|
Sep
(8) |
Oct
(18) |
Nov
(51) |
Dec
(74) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(47) |
Feb
(44) |
Mar
(44) |
Apr
(102) |
May
(35) |
Jun
(25) |
Jul
(56) |
Aug
(69) |
Sep
(32) |
Oct
(37) |
Nov
(31) |
Dec
(16) |
| 2012 |
Jan
(34) |
Feb
(127) |
Mar
(218) |
Apr
(252) |
May
(80) |
Jun
(137) |
Jul
(205) |
Aug
(159) |
Sep
(35) |
Oct
(50) |
Nov
(82) |
Dec
(52) |
| 2013 |
Jan
(107) |
Feb
(159) |
Mar
(118) |
Apr
(163) |
May
(151) |
Jun
(89) |
Jul
(106) |
Aug
(177) |
Sep
(49) |
Oct
(63) |
Nov
(46) |
Dec
(7) |
| 2014 |
Jan
(65) |
Feb
(128) |
Mar
(40) |
Apr
(11) |
May
(4) |
Jun
(8) |
Jul
(16) |
Aug
(11) |
Sep
(4) |
Oct
(1) |
Nov
(5) |
Dec
(16) |
| 2015 |
Jan
(5) |
Feb
|
Mar
(2) |
Apr
(5) |
May
(4) |
Jun
(12) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(4) |
| 2019 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(2) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
|
|
|
|
1
|
2
|
3
|
4
|
|
5
|
6
|
7
(13) |
8
(8) |
9
(12) |
10
(10) |
11
|
|
12
(1) |
13
(19) |
14
|
15
(22) |
16
(6) |
17
(20) |
18
|
|
19
|
20
(1) |
21
(1) |
22
(3) |
23
(4) |
24
(9) |
25
|
|
26
|
27
(4) |
28
(10) |
29
(3) |
30
(3) |
31
(2) |
|
|
From: Koichi S. <koi...@gm...> - 2013-05-27 22:41:31
|
Thank you Andrei for the patch. I took a glance at it and will review it before commit. Best; ---------- Koichi Suzuki 2013/5/27 Andrei Martsinchyk <and...@gm...> > We noticed that transaction handles are not released after direct > connections to datanodes, if they are connecting to GTM through GTM proxy. > So if Datanode is periodically connected directly (ex. for monitoring) > GTM eventually starts throwing error "Max transaction limit reached". > Please find fix attached. > > -- > Andrei Martsinchyk > > StormDB - http://www.stormdb.com > The Database Cloud > > > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may > _______________________________________________ > Postgres-xc-developers mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers > > |
|
From: Abbas B. <abb...@en...> - 2013-05-27 02:53:20
|
On Fri, May 24, 2013 at 7:54 AM, Ashutosh Bapat < ash...@en...> wrote: > > > > On Thu, May 23, 2013 at 10:23 PM, Ashutosh Bapat < > ash...@en...> wrote: > >> >> >> >> On Thu, May 23, 2013 at 10:19 PM, Abbas Butt <abb...@en... >> > wrote: >> >>> >>> >>> On Fri, May 17, 2013 at 1:58 PM, Ashutosh Bapat < >>> ash...@en...> wrote: >>> >>>> >>>> >>>> >>>> On Fri, May 17, 2013 at 2:23 PM, Abbas Butt < >>>> abb...@en...> wrote: >>>> >>>>> >>>>> >>>>> On Thu, May 16, 2013 at 3:13 PM, Ashutosh Bapat < >>>>> ash...@en...> wrote: >>>>> >>>>>> Hi Abbas, >>>>>> I am also seeing a lot of changes in the expected output where the >>>>>> rows output have changed. What are these changes? >>>>>> >>>>> >>>>> These changes are a result of blocking partition column updates >>>>> >>>> >>>> Are those in sync with PG expected output? >>>> >>> >>> No, in PG the update does not fail, in XC it fails. >>> >>> >>>> Why did we change the original expected output in first place? >>>> >>> >>> Do you mean that the changes in expected output due to blocking of >>> partition column updates should only be done in alternate expected output >>> file? >>> >>> >> >> yes, of course. >> >> > > This response can be confusing. If you are talking about the changing of > table distribution, then that has to be changed everywhere. But, I do not > understand, why should we see those many changes. The original output file > must have preserved the correct output, right? > Unfortunately the results in original output were incorrect, especially for the cases where it was possible to update partition column using WITH syntax. The patch fixes two issues 1. Block partition column updates using WITH syntax 2. WITH query that updates a table in the main query and inserts a row in the same table in the WITH query Hence there are more changes in the output files. > > >> >>>> >>>>> and changing the distribution of tables to replication. >>>>> >>>>> >>>> >>>> That's acceptable. >>>> >>>> >>>>> >>>>>> >>>>>> On Thu, May 16, 2013 at 2:55 PM, Ashutosh Bapat < >>>>>> ash...@en...> wrote: >>>>>> >>>>>>> Hi Abbas, >>>>>>> Instead of fixing the first issue in pgxc_build_dml_statement(), is >>>>>>> it possible to traverse the Query in validate_part_col_updatable() >>>>>>> recursively to find UPDATE statements and apply partition column check? >>>>>>> That would cover all the possibilities, I guess. That also saves us much >>>>>>> effort in case we come to support distribution column updation. >>>>>>> >>>>>>> I think, we need a generic solution to solve this command id issue, >>>>>>> e.g. punching command id always and efficiently. But for now this suffices. >>>>>>> Please log a bug/feature and put it in 1.2 bucket. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, May 15, 2013 at 5:31 AM, Abbas Butt < >>>>>>> abb...@en...> wrote: >>>>>>> >>>>>>>> Adding developers mailing list. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, May 15, 2013 at 4:57 AM, Abbas Butt < >>>>>>>> abb...@en...> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Attached please find a patch to fix test case with. >>>>>>>>> There were two issues making the test to fail. >>>>>>>>> 1. Updates to partition column were possible using syntax like >>>>>>>>> WITH t AS (UPDATE y SET a=a+1 RETURNING *) SELECT * FROM t >>>>>>>>> The patch blocks this syntax. >>>>>>>>> >>>>>>>>> 2. For a WITH query that updates a table in the main query and >>>>>>>>> inserts a row in the same table in the WITH query we need to >>>>>>>>> use >>>>>>>>> command ID communication to remote nodes in order to >>>>>>>>> maintain global data visibility. >>>>>>>>> For example >>>>>>>>> CREATE TEMP TABLE tab (id int,val text) DISTRIBUTE BY >>>>>>>>> REPLICATION; >>>>>>>>> INSERT INTO tab VALUES (1,'p1'); >>>>>>>>> WITH wcte AS (INSERT INTO tab VALUES(42,'new') RETURNING id >>>>>>>>> AS newid) >>>>>>>>> UPDATE tab SET id = id + newid FROM wcte; >>>>>>>>> The last query gets translated into the following >>>>>>>>> multi-statement >>>>>>>>> transaction on the primary datanode >>>>>>>>> (a) START TRANSACTION ISOLATION LEVEL read committed READ >>>>>>>>> WRITE >>>>>>>>> (b) INSERT INTO tab (id, val) VALUES ($1, $2) RETURNING id -- >>>>>>>>> (42,'new)' >>>>>>>>> (c) SELECT id, val, ctid FROM ONLY tab WHERE true >>>>>>>>> (d) UPDATE ONLY tab tab SET id = $1 WHERE (tab.ctid = $3) -- >>>>>>>>> (43,(0,1)] >>>>>>>>> (e) COMMIT TRANSACTION >>>>>>>>> The command id of the select in step (c), should be such that >>>>>>>>> it does not see the insert of step (b) >>>>>>>>> >>>>>>>>> Comments are welcome. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Abbas* >>>>>>>>> Architect >>>>>>>>> >>>>>>>>> Ph: 92.334.5100153 >>>>>>>>> Skype ID: gabbasb >>>>>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>>>>> * >>>>>>>>> Follow us on Twitter* >>>>>>>>> @EnterpriseDB >>>>>>>>> >>>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- >>>>>>>> *Abbas* >>>>>>>> Architect >>>>>>>> >>>>>>>> Ph: 92.334.5100153 >>>>>>>> Skype ID: gabbasb >>>>>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>>>>> * >>>>>>>> Follow us on Twitter* >>>>>>>> @EnterpriseDB >>>>>>>> >>>>>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------------------------------------------------------ >>>>>>>> AlienVault Unified Security Management (USM) platform delivers >>>>>>>> complete >>>>>>>> security visibility with the essential security capabilities. >>>>>>>> Easily and >>>>>>>> efficiently configure, manage, and operate all of your security >>>>>>>> controls >>>>>>>> from a single console and one unified framework. Download a free >>>>>>>> trial. >>>>>>>> http://p.sf.net/sfu/alienvault_d2d >>>>>>>> _______________________________________________ >>>>>>>> Postgres-xc-core mailing list >>>>>>>> Pos...@li... >>>>>>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Wishes, >>>>>>> Ashutosh Bapat >>>>>>> EntepriseDB Corporation >>>>>>> The Postgres Database Company >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Wishes, >>>>>> Ashutosh Bapat >>>>>> EntepriseDB Corporation >>>>>> The Postgres Database Company >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> *Abbas* >>>>> Architect >>>>> >>>>> Ph: 92.334.5100153 >>>>> Skype ID: gabbasb >>>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>>> * >>>>> Follow us on Twitter* >>>>> @EnterpriseDB >>>>> >>>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Wishes, >>>> Ashutosh Bapat >>>> EntepriseDB Corporation >>>> The Postgres Database Company >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> AlienVault Unified Security Management (USM) platform delivers complete >>>> security visibility with the essential security capabilities. Easily and >>>> efficiently configure, manage, and operate all of your security controls >>>> from a single console and one unified framework. Download a free trial. >>>> http://p.sf.net/sfu/alienvault_d2d >>>> _______________________________________________ >>>> Postgres-xc-core mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-core >>>> >>>> >>> >>> >>> -- >>> -- >>> *Abbas* >>> Architect >>> >>> Ph: 92.334.5100153 >>> Skype ID: gabbasb >>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>> * >>> Follow us on Twitter* >>> @EnterpriseDB >>> >>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>> >> >> >> >> -- >> Best Wishes, >> Ashutosh Bapat >> EntepriseDB Corporation >> The Postgres Database Company >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
|
From: Abbas B. <abb...@en...> - 2013-05-27 02:18:34
|
On Fri, May 24, 2013 at 7:04 PM, Ashutosh Bapat < ash...@en...> wrote: > > > > On Fri, May 24, 2013 at 9:01 AM, Abbas Butt <abb...@en...>wrote: > >> >> >> On Fri, May 24, 2013 at 7:22 AM, Ashutosh Bapat < >> ash...@en...> wrote: >> >>> >>> >>> >>> On Thu, May 23, 2013 at 9:21 PM, Abbas Butt <abb...@en... >>> > wrote: >>> >>>> Hi, >>>> >>>> While working on test case plancache it was brought up as a review >>>> comment that solving bug id 3607975 should solve the problem of the test >>>> case. >>>> However there is some confusion in the statement of bug id 3607975. >>>> >>>> "When a user does and PREPARE and then EXECUTEs multiple times, the >>>> coordinator keeps on preparing and executing the query on datanode al >>>> times, as against preparing once and executing multiple times. This is >>>> because somehow the remote query is being prepared as an unnamed statement." >>>> >>>> Consider this test case >>>> >>>> A. create table abc(a int, b int); >>>> B. insert into abc values(11, 22); >>>> C. prepare p1 as select * from abc; >>>> D. execute p1; >>>> E. execute p1; >>>> F. execute p1; >>>> >>>> Here are the confusions >>>> >>>> 1. The coordinator never prepares on datanode in response to a prepare >>>> issued by a user. >>>> In fact step C does nothing on the datanodes. >>>> Step D simply sends "SELECT a, b FROM abc" to all datanodes. >>>> >>>> 2. In step D, ExecuteQuery calls BuildCachedPlan to build a new generic >>>> plan, >>>> and steps E and F use the already built generic plan. >>>> For details see function GetCachedPlan. >>>> This means that executing a prepared statement again and again does >>>> use cached plans >>>> and does not prepare again and again every time we issue an execute. >>>> >>>> >>> The problem is not here. The problem is in do_query() where somehow the >>> name of prepared statement gets wiped out and we keep on preparing unnamed >>> statements at the datanode. >>> >> >> We never prepare any named/unnamed statements on the datanode. I spent >> time looking at the code written in do_query and functions called from with >> in do_query to handle prepared statements but the code written in >> pgxc_start_command_on_connection to handle statements prepared on datanodes >> is dead as of now. It is never called during the complete regression run. >> The function ActivateDatanodeStatementOnNode is never called. The way >> prepared statements are being handled now is the same as I described >> earlier in the mail chain with the help of an example. >> The code that is dead was originally added by Mason through commit >> d6d2d3d925f571b0b58ff6b4f6504d88e96bb342, back in December 2010. This code >> has been changed a lot over the last two years. This commit does not >> contain any test cases so I am not sure how did it use to work back then. >> >> > > This code wasn't dead, when I worked on prepared statements. So, something > has gone wrong in-between. That's what we need to find out and fix. Not > preparing statements on the datanode is not good for performance either. > I was able to find the reason why the code was dead and the attached patch (WIP) fixes the problem. This would now ensure that statements are prepared on datanodes whenever required. However there is a problem in the way prepared statements are handled. The problem is that unless a prepared statement is executed it is never prepared on datanodes, hence changing the path before executing the statement gives us incorrect results. For Example create schema s1 create table abc (f1 int) distribute by replication; create schema s2 create table abc (f1 int) distribute by replication; insert into s1.abc values(123); insert into s2.abc values(456); set search_path = s2; prepare p1 as select f1 from abc; set search_path = s1; execute p1; The last execute results in 123, where as it should have resulted in 456. I can finalize the attached patch by fixing any regression issues that may result and that would fix 3607975 and improve performance however the above test case would still fail. > > >> >>> >>>> My conclusion is that the bug ID 3607975 is not reproducible. >>>> >>>> >>> Did you verify it under the debugger? If that would have been the case, >>> we would not have seen this problem if search_path changed in between steps >>> D and E. >>> >> >> If search path is changed between steps D & E, the problem occurs because >> when the remote query node is created, schema qualification is not added in >> the sql statement to be sent to the datanode, but changes in search path do >> get communicated to the datanode. The sql statement is built when execute >> is issued for the first time and is reused on subsequent executes. The >> datanode is totally unaware that the select that it just received is due to >> an execute of a prepared statement that was prepared when search path was >> some thing else. >> >> > Fixing the prepared statements the way I suggested, would fix the problem, > since the statement will get prepared at the datanode, with the same search > path settings, as it would on the coordinator. > > >> >> >>> >>> >>>> Comments are welcome. >>>> >>>> -- >>>> *Abbas* >>>> Architect >>>> >>>> Ph: 92.334.5100153 >>>> Skype ID: gabbasb >>>> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >>>> * >>>> Follow us on Twitter* >>>> @EnterpriseDB >>>> >>>> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Try New Relic Now & We'll Send You this Cool Shirt >>>> New Relic is the only SaaS-based application performance monitoring >>>> service >>>> that delivers powerful full stack analytics. Optimize and monitor your >>>> browser, app, & servers with just a few lines of code. Try New Relic >>>> and get this awesome Nerd Life shirt! >>>> http://p.sf.net/sfu/newrelic_d2d_may >>>> _______________________________________________ >>>> Postgres-xc-developers mailing list >>>> Pos...@li... >>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers >>>> >>>> >>> >>> >>> -- >>> Best Wishes, >>> Ashutosh Bapat >>> EntepriseDB Corporation >>> The Postgres Database Company >>> >> >> >> >> -- >> -- >> *Abbas* >> Architect >> >> Ph: 92.334.5100153 >> Skype ID: gabbasb >> www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> >> * >> Follow us on Twitter* >> @EnterpriseDB >> >> Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> >> > > > > -- > Best Wishes, > Ashutosh Bapat > EntepriseDB Corporation > The Postgres Database Company > -- -- *Abbas* Architect Ph: 92.334.5100153 Skype ID: gabbasb www.enterprisedb.co <http://www.enterprisedb.com/>m<http://www.enterprisedb.com/> * Follow us on Twitter* @EnterpriseDB Visit EnterpriseDB for tutorials, webinars, whitepapers<http://www.enterprisedb.com/resources-community>and more<http://www.enterprisedb.com/resources-community> |
1 message has been excluded from this view by a project administrator.
