"Scalar found where operator expected" on printing to indirect filehandle within s///e

[p5pRT]

Migrated from rt.perl.org#133850 (status was 'pending release')

Searchable as RT133850$

[p5pRT]

From @Corion

Created by corion@corion.net

This is a bug report for perl from corion@​corion.net,
generated with the help of perlbug 1.40 running under perl 5.26.0.

-----------------------------------------------------------------
Hi all,

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

  perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the code​:

  Scalar found where operator expected at -e line 1, near "$fh $1"
  (Missing operator before $1?)
  Name "main​::fh" used only once​: possible typo at -e line 1.
  Use of uninitialized value $_ in substitution (s///) at -e line 1.
  Use of uninitialized value $_ in substitution (s///) at -e line 1.
  Can't use an undefined value as a symbol reference at -e line 1.

The code even outputs 'foo' in the below version​:

  perl -wle "my $fh = \*STDOUT; $_ = ''; s//print $fh 'foo'/e"
  String found where operator expected at -e line 1, near "$fh 'foo'"
  (Missing operator before 'foo'?)
  foo

The "fix" for this (highly unconventional) code is to use curly brackets
around the filehandle, forcing the "complicated expression" version of
print​:

  perl -wle "my $fh = \*STDOUT; $_ = ''; s//print {$fh} 'foo'/e"
  foo

I still think that the original code should parse without error, even
though it
is really weird and I'm no fan of side-effects during regexp evaluation.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.26.0:

Configured by strawberry-perl at Wed May 31 04:46:12 2017.

Summary of my perl5 (revision 5 version 26 subversion 0) configuration:

  Platform:
    osname=MSWin32
    osvers=6.3
    archname=MSWin32-x64-multi-thread
    uname='Win32 strawberry-perl 5.26.0.1 #1 Wed May 31 04:42:31 2017 x64'
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE
-D__USE_MINGW_ANSI_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing
-mms-bitfields'
    optimize='-s -O2'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='7.1.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++'
    ldflags ='-s -L"c:\strawberry-perl-5.26.0.1-64bit\perl\lib\CORE"
-L"c:\strawberry-perl-5.26.0.1-64bit\c\lib"'
    libpth=c:\strawberry-perl-5.26.0.1-64bit\c\lib
c:\strawberry-perl-5.26.0.1-64bit\c\x86_64-w64-mingw32\lib
c:\strawberry-perl-5.26.0.1-64bit\c\lib\gcc\x86_64-w64-mingw32\7.1.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr
-lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl526.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=xs.dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-mdll -s
-L"c:\strawberry-perl-5.26.0.1-64bit\perl\lib\CORE"
-L"c:\strawberry-perl-5.26.0.1-64bit\c\lib"'



@INC for perl 5.26.0:
    C:/strawberry-perl-5.26.0.1-64bit/perl/site/lib/MSWin32-x64-multi-thread
    C:/strawberry-perl-5.26.0.1-64bit/perl/site/lib
    C:/strawberry-perl-5.26.0.1-64bit/perl/vendor/lib
    C:/strawberry-perl-5.26.0.1-64bit/perl/lib


Environment for perl 5.26.0:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\Program Files\Microsoft IntelliType Pro\;C:\Program
Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\bin;C:\Program Files\NVIDIA
GPU Computing
Toolkit\CUDA\v9.2\libnvvp;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
Files\Git\cmd;C:\Program Files\PuTTY;C:\Program Files\MiKTeX
2.9\miktex\bin\x64\;C:\strawberry-perl-5.20.1.1-x64\c\bin;C:\strawberry-perl-5.20.1.1-x64\perl\site\bin;C:\strawberry-perl-5.20.1.1-x64\perl\bin;C:\Program
Files (x86)\Skype\Phone\;C:\Program Files (x86)\NVIDIA
Corporation\PhysX\Common;C:\Program Files
(x86)\GtkSharp\2.12\bin;C:\Program Files
(x86)\Gpg4win\..\GnuPG\bin;C:\Program Files\nodejs\;C:\Program
Files\NVIDIA Corporation\NVIDIA
NvDLISR;C:\Users\Corion\AppData\Local\Programs\Python\Python36\Scripts\;C:\Users\Corion\AppData\Local\Programs\Python\Python36\;C:\Program
Files\Docker
Toolbox;C:\ProgramData\chocolatey\bin;C:\strawberry-perl-5.26.0.1-64bit\perl\bin;c:\strawberry-perl-5.26.0.1-64bit\c\bin;C:\strawberry-perl-5.26.0.1-64bit\c\x86_64-w64-mingw32\bin;C:\Users\Corion\AppData\Roaming\npm
    PERL_BADLANG (unset)
    SHELL (unset)

[p5pRT]

From @tonycoz

On Fri, 15 Feb 2019 11​:37​:16 -0800, corion@​corion.net wrote​:

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the
code​:

Scalar found where operator expected at -e line 1, near "$fh $1"
(Missing operator before $1?)

This isn't a syntax error, but a warning (but still buggy).

The code inside the replacement part of s///e is handled with a sublex, and the code that decides whether an operator or term is expected after C<print $fh> is mostly skipped for a sublex.

The attached fixes that, doing the same checks for C<print $fh> in a sublex within brackets as for a normal parse.

This fixes the same problem for C<< "... ${code} ..." >>.

Tony

[p5pRT]

From @tonycoz

0001-perl-133850-fix-parsing-hints-for-print-fh-foo-in-s-.patch

From 9e0f09ef2afa82ef1a96cba981d4fca094bba484 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Tue, 19 Feb 2019 15:38:39 +1100
Subject: (perl #133850) fix parsing hints for print $fh "foo" in s///e

The replacement code in s///e is parsed using a sublex, similarly
to double-quotish replacement.

The code that handles C< $id > tries to detect what the next token
should be, but only special cased code similar to C< print $fh ... >
in a non-sublex, so s/../print $fh $1/e set PL_expect to XOPERATOR
which caused the parser to complain when the $1 wasn't an operator.

The fix here handles two cases:

 s/.../code here/e

In this case the code is inserted into a do {} block like:

  do { code here }

so PL_lex_brackets ends up non-zero.

The second case is code like:

  "xxx${code here}"

which had the same problem, which this also fixes.
---
 t/lib/warnings/toke | 14 ++++++++++++++
 toke.c              |  2 +-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index 1c85d7bc44..a500856283 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1675,3 +1675,17 @@ Execution of - aborted due to compilation errors.
 use utf8;
 qw���foo ��� ��� bar���
 EXPECT
+########
+# NAME [perl #133850] print $fh $1 in s///e expression
+use warnings;
+my $fh = \*STDOUT;
+$_ = "abc";
+s/(x)/ print $fh $1 /e;
+EXPECT
+########
+# NAME [perl #133850] side case
+use warnings;
+my $fh = \*STDOUT;
+my $y = "";
+my $x = "${print $fh $y; \'x'}";
+EXPECT
diff --git a/toke.c b/toke.c
index 5a3fe78472..0eb63fcb55 100644
--- a/toke.c
+++ b/toke.c
@@ -6772,7 +6772,7 @@ Perl_yylex(pTHX)
 	    }
 
 	    PL_expect = XOPERATOR;
-	    if (PL_lex_state == LEX_NORMAL && isSPACE((char)tmp)) {
+	    if ((PL_lex_state == LEX_NORMAL || PL_lex_brackets) && isSPACE((char)tmp)) {
 		const bool islop = (PL_last_lop == PL_oldoldbufptr);
 		if (!islop || PL_last_lop_op == OP_GREPSTART)
 		    PL_expect = XOPERATOR;
-- 
2.11.0

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @tonycoz

On Mon, 18 Feb 2019 20​:44​:26 -0800, tonyc wrote​:

This fixes the same problem for C<< "... ${code} ..." >>.

There's some other similar cases - i.e. where PL_lex_state is checked for LEX_NORMAL but no other checks are made. The simplest I was able to reproduce was​:

$ ./perl -we '%x = qw(a b c d); $_ = ""; $_ = @​x {a}'
Scalar value @​x{"a"} better written as $x{"a"} at -e line 1.
$ ./perl -we '%x = qw(a b c d); $_ = ""; s/^/ @​x {a} /e'
(no warning)

Kind of unlikely​:

tony@​mars​:.../git/perl$ ./perl -Ilib -we '{ my $x : shared = 1; }'
tony@​mars​:.../git/perl$ ./perl -Ilib -we '$_ = ""; s/^/ { my $x : shared = 1; } /e'
Unquoted string "shared" may clash with future reserved word at -e line 1.
syntax error at -e line 1, near "$x :"

Another​:

$ ./perl -we '$a = ${time}'
Ambiguous use of ${time} resolved to $time at -e line 1.
Name "main​::time" used only once​: possible typo at -e line 1.
$ ./perl -we '$_ = ""; s/^/ ${time} /e'
Name "main​::time" used only once​: possible typo at -e line 1.
Use of uninitialized value $time in substitution (s///) at -e line 1.
(no ambiguous warning)

Tony

[p5pRT]

From @tonycoz

On Mon, 18 Feb 2019 20​:44​:26 -0800, tonyc wrote​:

On Fri, 15 Feb 2019 11​:37​:16 -0800, corion@​corion.net wrote​:

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the
code​:

Scalar found where operator expected at -e line 1, near "$fh $1"
(Missing operator before $1?)

This isn't a syntax error, but a warning (but still buggy).

The code inside the replacement part of s///e is handled with a
sublex, and the code that decides whether an operator or term is
expected after C<print $fh> is mostly skipped for a sublex.

The attached fixes that, doing the same checks for C<print $fh> in a
sublex within brackets as for a normal parse.

This fixes the same problem for C<< "... ${code} ..." >>.

Rebased on blead, and fixes for the related issues.

I'll apply this in a few days.

Tony

[p5pRT]

From @tonycoz

subparse-tokens.patch

From cb43f46efc62573c3470f5c2de01ee0e5b6ffd4a Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Tue, 19 Feb 2019 15:38:39 +1100
Subject: (perl #133850) fix parsing hints for print $fh "foo" in s///e

The replacement code in s///e is parsed using a sublex, similarly
to double-quotish replacement.

The code that handles C< $id > tries to detect what the next token
should be, but only special cased code similar to C< print $fh ... >
in a non-sublex, so s/../print $fh $1/e set PL_expect to XOPERATOR
which caused the parser to complain when the $1 wasn't an operator.

The fix here handles two cases:

 s/.../code here/e

In this case the code is inserted into a do {} block like:

  do { code here }

so PL_lex_brackets ends up non-zero.

The second case is code like:

  "xxx${code here}"

which had the same problem, which this also fixes.
---
 t/lib/warnings/toke | 14 ++++++++++++++
 toke.c              |  2 +-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index 83641e5106..944e5d92f4 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1691,3 +1691,17 @@ EXPECT
 OPTION regex
 Malformed UTF-8 character: .*non-continuation.*
 The eval did not crash the program
+########
+# NAME [perl #133850] print $fh $1 in s///e expression
+use warnings;
+my $fh = \*STDOUT;
+$_ = "abc";
+s/(x)/ print $fh $1 /e;
+EXPECT
+########
+# NAME [perl #133850] side case
+use warnings;
+my $fh = \*STDOUT;
+my $y = "";
+my $x = "${print $fh $y; \'x'}";
+EXPECT
diff --git a/toke.c b/toke.c
index b0fa2c35ba..dc376f6beb 100644
--- a/toke.c
+++ b/toke.c
@@ -6838,7 +6838,7 @@ Perl_yylex(pTHX)
 	    }
 
 	    PL_expect = XOPERATOR;
-	    if (PL_lex_state == LEX_NORMAL && isSPACE((char)tmp)) {
+	    if ((PL_lex_state == LEX_NORMAL || PL_lex_brackets) && isSPACE((char)tmp)) {
 		const bool islop = (PL_last_lop == PL_oldoldbufptr);
 		if (!islop || PL_last_lop_op == OP_GREPSTART)
 		    PL_expect = XOPERATOR;
-- 
2.11.0


From 2b67caa9b5055eb7441998a9538af722639982af Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 29 May 2019 11:46:01 +1000
Subject: (perl #133850) warn on ${time} even in sub-parse

Code like:

  $a = ${time}

warns with:

  Ambiguous use of ${time} resolved to $time at ...

but:

  s/^/ ${time} /e

didn't, since the parser is in a special state in sub-parses.
---
 t/lib/warnings/toke | 8 ++++++++
 toke.c              | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index 944e5d92f4..eab22fbbc9 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1705,3 +1705,11 @@ my $fh = \*STDOUT;
 my $y = "";
 my $x = "${print $fh $y; \'x'}";
 EXPECT
+########
+# NAME [perl #133850] another case
+use warnings;
+my $time = 1;
+$_ = "";
+s/^/ ${time} /e
+EXPECT
+Ambiguous use of ${time} resolved to $time at - line 4.
diff --git a/toke.c b/toke.c
index dc376f6beb..c42d4fd802 100644
--- a/toke.c
+++ b/toke.c
@@ -9660,7 +9660,7 @@ S_scan_ident(pTHX_ char *s, char *dest, STRLEN destlen, I32 ck_uni)
 		PL_lex_state = LEX_INTERPEND;
 		PL_expect = XREF;
 	    }
-	    if (PL_lex_state == LEX_NORMAL) {
+	    if (PL_lex_state == LEX_NORMAL || PL_lex_brackets) {
 		if (ckWARN(WARN_AMBIGUOUS)
                     && (keyword(dest, d - dest, 0)
 		        || get_cvn_flags(dest, d - dest, is_utf8
-- 
2.11.0


From f623089c7b47c9de58667b70063be135948356bd Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 29 May 2019 13:47:36 +1000
Subject: (perl #133850) allow @x{...} with space in sub-parse

---
 t/lib/warnings/toke | 8 ++++++++
 toke.c              | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/t/lib/warnings/toke b/t/lib/warnings/toke
index eab22fbbc9..5dbef1ab62 100644
--- a/t/lib/warnings/toke
+++ b/t/lib/warnings/toke
@@ -1713,3 +1713,11 @@ $_ = "";
 s/^/ ${time} /e
 EXPECT
 Ambiguous use of ${time} resolved to $time at - line 4.
+########
+# NAME [perl #133850] another case
+use warnings;
+%x = qw(a b c d);
+$_ = "";
+s/^/ @x {a} /e
+EXPECT
+Scalar value @x{"a"} better written as $x{"a"} at - line 4.
diff --git a/toke.c b/toke.c
index c42d4fd802..e1b163334f 100644
--- a/toke.c
+++ b/toke.c
@@ -6911,7 +6911,7 @@ Perl_yylex(pTHX)
 	if (!PL_tokenbuf[1]) {
 	    PREREF('@');
 	}
-	if (PL_lex_state == LEX_NORMAL)
+	if (PL_lex_state == LEX_NORMAL || PL_lex_brackets)
 	    s = skipspace(s);
 	if (   (PL_expect != XREF || PL_oldoldbufptr == PL_last_lop)
             && intuit_more(s, PL_bufend))
-- 
2.11.0


From d50d49c97634fa03d472f06bf4e70fe7dfc17fd2 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 29 May 2019 14:37:25 +1000
Subject: (perl #133850) allow attributes in sub-parses

---
 t/op/attrs.t | 3 +++
 toke.c       | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/t/op/attrs.t b/t/op/attrs.t
index 649525ce22..dd230b3fbd 100644
--- a/t/op/attrs.t
+++ b/t/op/attrs.t
@@ -507,4 +507,7 @@ BEGIN failed--compilation aborted at - line 1.
 EOS
               "attribute on our hash with sub of same name");
 
+fresh_perl_is('$_ = ""; s/^/ { my $x : shared = 1; } /e;', "", {},
+              "attributes in sub-parse");
+
 done_testing();
diff --git a/toke.c b/toke.c
index e1b163334f..0b3b76fee3 100644
--- a/toke.c
+++ b/toke.c
@@ -5991,7 +5991,7 @@ Perl_yylex(pTHX)
 
 	switch (PL_expect) {
 	case XOPERATOR:
-	    if (!PL_in_my || PL_lex_state != LEX_NORMAL)
+	    if (!PL_in_my || (PL_lex_state != LEX_NORMAL && !PL_lex_brackets))
 		break;
 	    PL_bufptr = s;	/* update in case we back off */
 	    if (*s == '=') {
-- 
2.11.0

[p5pRT]

From @tonycoz

On Tue, 28 May 2019 22​:08​:50 -0700, tonyc wrote​:

On Mon, 18 Feb 2019 20​:44​:26 -0800, tonyc wrote​:

On Fri, 15 Feb 2019 11​:37​:16 -0800, corion@​corion.net wrote​:

last week I (together with E. Choroba) stumbled on a weird syntax
error (Windows shell quoting)​:

perl -wle "s/(.*)/print $fh $1/e"

This surprisingly gives a syntax error, yet still seems to run the
code​:

Scalar found where operator expected at -e line 1, near "$fh $1"
(Missing operator before $1?)

This isn't a syntax error, but a warning (but still buggy).

The code inside the replacement part of s///e is handled with a
sublex, and the code that decides whether an operator or term is
expected after C<print $fh> is mostly skipped for a sublex.

The attached fixes that, doing the same checks for C<print $fh> in a
sublex within brackets as for a normal parse.

This fixes the same problem for C<< "... ${code} ..." >>.

Rebased on blead, and fixes for the related issues.

I'll apply this in a few days.

Applied as eec8822 through 82de236.

Tony

[p5pRT]

@tonycoz - Status changed from 'open' to 'pending release'