postgres-xc-developers Mailing List for Postgres-XC

On Mon, Apr 30, 2012 at 7:07 AM, Amit Khandekar
<ami...@en...> wrote:
>
>
> On 30 April 2012 16:15, Mason Sharp <mas...@gm...> wrote:
>>
>> On Mon, Apr 30, 2012 at 12:41 AM, Amit Khandekar
>> <ami...@en...> wrote:
>> >
>> >
>> > On 30 April 2012 09:55, Ashutosh Bapat <ash...@en...>
>> > wrote:
>> >>
>> >> If it's an SPI interface, PLs can use this hole to exploit security.
>> >> Is that a possibility?
>> >
>> >
>> > Yes that's a possibility. That's what I meant when I said it is not a
>> > security solution:
>> >
>> >>> It is not a full-fledged security solution. A malicious user
>> >>> can even call the new SPI_execute_direct() once he identifies that
>> >>> it's
>> >>> there. We have just made it harder to do so
>> >
>> > This solution is more to avoid normal users to accidentally end up doing
>> > table writes using exec-direct. Once the user starts using
>> > SPI_execute_direct() calls in C functions, he can do so. I don't think
>> > there
>> > is a way to define an SPI function as being internal. Just that we
>> > should
>> > not document this function since it is for our internal purpose.
>> >
>> > More comments inline.
>> >
>> >>
>> >> On Mon, Apr 30, 2012 at 8:56 AM, Michael Paquier
>> >> <mic...@gm...> wrote:
>> >> >
>> >> >
>> >> > On Sun, Apr 29, 2012 at 9:17 PM, Amit Khandekar
>> >> > <ami...@en...> wrote:
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 27 April 2012 16:32, Amit Khandekar
>> >> >> <ami...@en...>
>> >> >> wrote:
>> >> >>>
>> >> >>> >>Comment By: Koichi Suzuki (koichi-szk)
>> >> >>> > Date: 2012-04-27 00:57
>> >> >>> >
>> >> >>> > Message:
>> >> >>> > This is included in V1.0 bucket.   I'm afraid this is a bit more
>> >> >>> > than
>> >> >>> > we
>> >> >>> > can do in 1.0.   I'd suggest to leave this as 1.0 restriction and
>> >> >>> > carry
>> >> >>> > over to the next chance.
>> >> >>> >
>> >> >>> > As Mason suggested, this restriction is from old user assumption
>> >> >>> > by
>> >> >>> > datanode and this restriction is now gone.
>> >> >>> >
>> >> >>> > As to relieving EXECUTE DIRECT privilege, we should be very
>> >> >>> > careful
>> >> >>> > to
>> >> >>> > eliminate a chance to bring whole cluster into inconsistent
>> >> >>> > state.
>> >> >>> >  Using
>> >> >>> > volatile functions in SELECT clause can easily do this.   I think
>> >> >>> > this
>> >> >>> > is a
>> >> >>> > topic for the next major release.
>> >> >>>
>> >> >>> One possibility I am considering is whether we can bypass this
>> >> >>> restriction by creating the execute-direct full-transformed query
>> >> >>> structure by our own, and then planning and executing normally.
>> >> >>> Currently the restriction is placed in transformExecDirectStmt(),
>> >> >>> so
>> >> >>> if we manage to bypass this function call, we will be able to
>> >> >>> bypass
>> >> >>> this restriction.
>> >> >>>
>> >> >>> Currently what happens is, in pg_relation_size(), we execute
>> >> >>> exec-direct using SPI calls. So it goes through all the steps from
>> >> >>> parsing, analysing, rewriting, planning and execution. So, now
>> >> >>> first
>> >> >>> construct a Query structure with fields ending up populated the way
>> >> >>> they are populated after transformExecDirectStmt() statement. And
>> >> >>> then
>> >> >>> call the rest of the usual functions like pg_pan_queries() ,
>> >> >>> PushActiveSnapshot(), and execute(). So a new function
>> >> >>> ExecuteDirectInternal() will have all the above calls, and this
>> >> >>> function would be used for our internal calls like
>> >> >>> pg_relation_size()
>> >> >>> and pg_advisory_lock(). Roughly it will be equivalent of
>> >> >>> SPI_prepare
>> >> >>> plus SPI_execute().
>> >> >>>
>> >> >>> Comments welcome. By the way, this won't be an invasive fix because
>> >> >>> it
>> >> >>> does not affect anything other than the particular system functions
>> >> >>> that propagate to datanodes, namely, pg_relation_*() and
>> >> >>> pg_advisory_lock*() . Just that I am yet o try this fix out.
>> >> >>
>> >> >>
>> >> >>
>> >> >> I have tried to quickly see if it is feasible to implement using
>> >> >> this
>> >> >> approach, and it looks like we can. Attached is a WIP patch. I have
>> >> >> created
>> >> >> a new SPI function SPI_execute_direct(). This internally creates an
>> >> >> ExecDirect statement and then calls a modified version of
>> >> >> SPI_prepare.
>> >> >> This
>> >> >> version accepts a parsed string instead of a raw string. This
>> >> >> enables
>> >> >> us to
>> >> >> bypass parsing stage of EXEC DIRECT. And I have moved the super-user
>> >> >> restriction from analyse to parsing stage. In this way we could
>> >> >> bypass
>> >> >> the
>> >> >> restriction by calling SPI_execute_direct().
>> >> >
>> >> > OK thanks.
>> >> >
>> >> >>
>> >> >>
>> >> >> Currently we have other ways of bypassing the "Cannot use DML using
>> >> >> exec-direct" restriction. For e.g. a user can create an immutable
>> >> >> function
>> >> >> which internally executes inserts.
>> >> >
>> >> > In this case the DBA has all responsability, an immutable function
>> >> > cannot
>> >> > modify the database.
>> >
>> >
>> > Come to think of it, a user can even do this without using exec-direct.
>> > So,
>> > I agree that there's no point thinking about immutable function that
>> > does
>> > writes.
>> >
>> > But even if we identify that the functions in exec-direct sql are
>> > volatile
>> > and restrict the exec-direct, that won't help the pg_relation_size issue
>> > because we do need to propagate such volatile functions in this case.
>> >>
>> >> >
>> >> >>
>> >> >> So in the future there should be a *hard* restriction in the
>> >> >> datanode
>> >> >> side, where it identifies that this is a stmt directly executed on
>> >> >> datanode,
>> >> >> and it is doing user-table writes. So until then, it is not good to
>> >> >> allow
>> >> >> exec-direct for any user.
>> >> >
>> >> > This could be done by some GUC parameter perhaps, just an idea.
>> >> >>
>> >> >>
>> >> >> Note that the above fix just prevents a normal user to accidentally
>> >> >> do
>> >> >> DMLs directly. It is not a full-fledged security solution. A
>> >> >> malicious
>> >> >> user
>> >> >> can even call the new SPI_execute_direct() once he identifies that
>> >> >> it's
>> >> >> there. We have just made it harder to do so.
>> >> >>
>> >> >> Yet to run regression and testing, commenting etc. Just wanted to
>> >> >> have
>> >> >> some comments meanwhile.
>> >> >
>> >> > Now only pg_database_size is changed. You may also have similar SPI
>> >> > call
>> >> > for
>> >> > pg_tablespace_size and pg_relation_size.
>> >
>> >
>> > All these functions directly or indirectly call pgxc_execute_on_nodes(),
>> > which now would call SPI_execute_direct(). So this change affects all
>> > the
>> > relevant system functions.
>> >
>>
>> I think it is best to implement pg_relation_size and similar functions
>> without using execute direct at all, just aggregate the results.
>
>
> Can you please elaborate a bit more? Any particular disadvantage with
> exec-direct that you anticipate? Currently we do not support executing
> directly on multiple nodes at the same time (although it might be no longer
> unsafe), so we need to sum up individual results for object size functions.
>

There seemed to be concern about execute direct and permissions. I was
just saying one could execute on all the nodes without going through
exec direct and manually sum up the results for this and any other
internal functions that need aggregation. I didn't have time to write
a patch, but it looks like you guys already went with another
solution.

> In case of pg_advisory_lock, it is mandatory to execute the calls to other
> coordinators serially, not in parallel; i.e. wait until we finish one
> coordinator, and then goto next coordinator.
>
>>
>> Mason
>
>

On Tue, May 1, 2012 at 11:56 AM, Abbas Butt <abb...@te...>wrote:

> PFA a patch to fix this crash.
> The crash was due to two reasons
> 1. The stmt_name member of CachedPlanSource was left uninitialized
> in FastCreateCachedPlan. (Patch provided by Mason)
> 2. StoreCachedPlan was wrongly typecasting utility statements
> to PlannedStmt *.
>
No problems I think.
Regressions are running without problems here, and patch looks to be clean.
Thanks.

>
> Regression shows no extra failure and the libpq test program and a few
> other variations of it work fine.
>
>
> On Mon, Apr 30, 2012 at 4:23 PM, Abbas Butt <abb...@te...>wrote:
>
>>
>>
>> On Mon, Apr 30, 2012 at 3:57 PM, Ashutosh Bapat <
>> ash...@en...> wrote:
>>
>>> I think, Abbas has looked into this crash. There's a fake Inner/Outer
>>> pointer in the plan tree OR we are mistakenly casting to PlannedStmt.
>>>
>>> Abbas,
>>> Is it the same crash as the one for CREATE TABLE inside the function?
>>>
>>
>> No.
>>
>>
>>> We have to also understand why we are using extended query protocol
>>> for SET command?
>>>
>>
>> This test case is not reproducible using psql, however I am able to
>> reproduce it using the attached libpq program, which explains why we are
>> using extended query protocol. I will look at it after finishing the
>> foreign_key test case.
>>
>> BTW we need to run some basic libpq & jdbc test programs to make sure
>> whatever works through psql works from these connectors too.
>>
>>
>>
>>
>>>
>>> On Mon, Apr 30, 2012 at 4:21 PM, Mason <ma...@us...>
>>> wrote:
>>> > On Sat, Apr 28, 2012 at 8:39 PM, Michael Paquier
>>> > <mic...@gm...> wrote:
>>> >>
>>> >>
>>> >> On Sun, Apr 29, 2012 at 7:22 AM, Mason <ma...@us...
>>> >
>>> >> wrote:
>>> >>>
>>> >>> On Thu, Apr 26, 2012 at 8:00 PM, Michael Paquier
>>> >>> <mic...@gm...> wrote:
>>> >>> > On Fri, Apr 27, 2012 at 6:49 AM, Mason <
>>> ma...@us...>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> There was a seg fault when executing ""SET extra_float_digits =
>>> 3" due
>>> >>> >> to an uninitialized variable.
>>> >>> >>
>>> >>> >> The attached patch fixes it.
>>> >>> >
>>> >>> > I am not able to reproduce that issue.
>>> >>> > Do you have a more complete test case? Does it happen randomly?
>>> >>>
>>> >>> It might be hard to reproduce if you get lucky and that part of
>>> memory
>>> >>> happens to already be 0. Initializing would be safer.
>>> >>
>>> >> By looking at the code it looks to be a problem with former Postgres.
>>> >> Why not solving the problem at the source?
>>> >
>>> > The backtrace shows a problem in XC:
>>> >
>>> > (gdb) bt
>>> > #0  0x00000000005a9b5c in SetRemoteStatementName
>>> (plan=0x7f7f7f7f7f7f7f7f,
>>> >   stmt_name=0x7f7f7f7f7f7f7f7f <Address 0x7f7f7f7f7f7f7f7f out of
>>> > bounds>, num_params=0, param_types=0x0, n=0)
>>> >   at prepare.c:473
>>> > #1  0x00000000005a9da4 in SetRemoteStatementName (plan=0x1f295e58,
>>> >   stmt_name=0x7f7f7f7f7f7f7f7f <Address 0x7f7f7f7f7f7f7f7f out of
>>> > bounds>, num_params=0, param_types=0x0, n=0)
>>> >   at prepare.c:536
>>> > #2  0x00000000005a9de7 in SetRemoteStatementName (plan=0x1f295df8,
>>> >   stmt_name=0x7f7f7f7f7f7f7f7f <Address 0x7f7f7f7f7f7f7f7f out of
>>> > bounds>, num_params=0, param_types=0x0, n=0)
>>> >   at prepare.c:540
>>> > #3  0x0000000000824fc1 in StoreCachedPlan (plansource=0x1f296b50,
>>> > stmt_list=0x1f2969d8, plan_context=0x1f2c16a0)
>>> >   at plancache.c:374
>>> > #4  0x0000000000824e09 in FastCreateCachedPlan
>>> (raw_parse_tree=0x1f295cf0,
>>> >   query_string=0x1f296a10 "SET extra_float_digits = 3",
>>> > commandTag=0x9f165a "SET", param_types=0x0, num_params=0,
>>> >   cursor_options=0, stmt_list=0x1f2969d8, fully_planned=1 '\001',
>>> > fixed_result=1 '\001', context=0x1f2c16a0)
>>> >   at plancache.c:274
>>> > #5  0x00000000007323ce in exec_parse_message (query_string=0x1f289629
>>> > "SET extra_float_digits = 3",
>>> >   stmt_name=0x1f289628 "", paramTypes=0x1f289e58,
>>> > paramTypeNames=0x0, numParams=0) at postgres.c:1460
>>> > #6  0x00000000007360e9 in PostgresMain (argc=2, argv=0x1f1cc8e0,
>>> > username=0x1f1cc690 "web") at postgres.c:4175
>>> > #7  0x00000000006de8ad in BackendRun (port=0x1f1f8030) at
>>> postmaster.c:3769
>>> > #8  0x00000000006ddf7e in BackendStartup (port=0x1f1f8030) at
>>> postmaster.c:3454
>>> > #9  0x00000000006dae24 in ServerLoop () at postmaster.c:1545
>>> > #10 0x00000000006da59a in PostmasterMain (argc=7, argv=0x1f1cba30) at
>>> > postmaster.c:1206
>>> > #11 0x0000000000649323 in main (argc=7, argv=0x1f1cba30) at main.c:199
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Live Security Virtual Conference
>>> > Exclusive live event will cover all the ways today's security and
>>> > threat landscape has changed and how IT managers can respond.
>>> Discussions
>>> > will include endpoint security, mobile security and the latest in
>>> malware
>>> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> > _______________________________________________
>>> > Postgres-xc-developers mailing list
>>> > Pos...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>>
>>>
>>> --
>>> Best Wishes,
>>> Ashutosh Bapat
>>> EntepriseDB Corporation
>>> The Enterprise Postgres Company
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Postgres-xc-developers mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Postgres-xc-developers mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-developers
>
>


-- 
Michael Paquier
http://michael.otacoo.com