Skip to content

Archaic control character 0x1f causes crash in linecount report #5884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
gvanrossum opened this issue Nov 11, 2018 · 1 comment
Closed

Archaic control character 0x1f causes crash in linecount report #5884

gvanrossum opened this issue Nov 11, 2018 · 1 comment

Comments

@gvanrossum
Copy link
Member

I have this file (foo.py):

def foo():
    # (�)
    return ""

The comment contains a weird control character hex 1f. Running mypy like this:

mypy --xml-report .xml-report foo.py --tb

causes this crash:

foo.py: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.650+dev.94c54aa755f2d7826fed5e0e2276053ba49da8e6
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/__main__.py", line 11, in <module>
    main(None)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/main.py", line 92, in main
    res = build.build(sources, options, None, flush_errors, fscache)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 155, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 204, in _build
    graph = dispatch(sources, manager)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 2172, in dispatch
    process_graph(graph, manager)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 2469, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 2605, in process_stale_scc
    graph[id].finish_passes()
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 1767, in finish_passes
    manager.report_file(self.tree, self.type_map(), self.options)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 1519, in wrap_context
    yield
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 1767, in finish_passes
    manager.report_file(self.tree, self.type_map(), self.options)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 643, in report_file
    self.reports.file(file, type_map, options)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/report.py", line 83, in file
    reporter.on_file(tree, type_map, options)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/report.py", line 450, in on_file
    any_info=self._get_any_info_for_line(visitor, lineno))
  File "src/lxml/etree.pyx", line 3112, in lxml.etree.SubElement
  File "src/lxml/apihelpers.pxi", line 199, in lxml.etree._makeSubElement
  File "src/lxml/apihelpers.pxi", line 194, in lxml.etree._makeSubElement
  File "src/lxml/apihelpers.pxi", line 297, in lxml.etree._initNodeAttributes
  File "src/lxml/apihelpers.pxi", line 311, in lxml.etree._addAttributeToNode
  File "src/lxml/apihelpers.pxi", line 1439, in lxml.etree._utf8
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
foo.py: : note: use --pdb to drop into pdb
Generated XML report: /Users/guido/src/server/.xml-report/index.xml
@gvanrossum
Copy link
Member Author

Also, if the directory .xml-report doesn't already exist, I get additional tracebacks (instead of the final Generated XML report: ... line):

Traceback (most recent call last):
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 1519, in wrap_context
    yield
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 1767, in finish_passes
    manager.report_file(self.tree, self.type_map(), self.options)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 643, in report_file
    self.reports.file(file, type_map, options)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/report.py", line 83, in file
    reporter.on_file(tree, type_map, options)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/report.py", line 450, in on_file
    any_info=self._get_any_info_for_line(visitor, lineno))
  File "src/lxml/etree.pyx", line 3112, in lxml.etree.SubElement
  File "src/lxml/apihelpers.pxi", line 199, in lxml.etree._makeSubElement
  File "src/lxml/apihelpers.pxi", line 194, in lxml.etree._makeSubElement
  File "src/lxml/apihelpers.pxi", line 297, in lxml.etree._initNodeAttributes
  File "src/lxml/apihelpers.pxi", line 311, in lxml.etree._addAttributeToNode
  File "src/lxml/apihelpers.pxi", line 1439, in lxml.etree._utf8
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 204, in _build
    graph = dispatch(sources, manager)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 2172, in dispatch
    process_graph(graph, manager)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 2469, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 2605, in process_stale_scc
    graph[id].finish_passes()
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 1767, in finish_passes
    manager.report_file(self.tree, self.type_map(), self.options)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 1523, in wrap_context
    report_internal_error(err, self.path, 0, self.manager.errors, self.options)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/errors.py", line 615, in report_internal_error
    raise SystemExit(1)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/__main__.py", line 11, in <module>
    main(None)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/main.py", line 92, in main
    res = build.build(sources, options, None, flush_errors, fscache)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 155, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache)
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/build.py", line 214, in _build
    reports.finish()
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/report.py", line 87, in finish
    reporter.on_finish()
  File "/Users/guido/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/report.py", line 666, in on_finish
    last_xml.write(out_path, encoding='utf-8')
  File "src/lxml/etree.pyx", line 2039, in lxml.etree._ElementTree.write
  File "src/lxml/serializer.pxi", line 721, in lxml.etree._tofilelike
  File "src/lxml/serializer.pxi", line 780, in lxml.etree._create_output_buffer
  File "src/lxml/serializer.pxi", line 770, in lxml.etree._create_output_buffer
FileNotFoundError: [Errno 2] No such file or directory

This seems to be unrelated to the control character, it's just due to the situation where on_finish() is called without on_file() ever being called -- there's no ensure_dir_exists() call on on_finish().

gvanrossum pushed a commit to gvanrossum/mypy that referenced this issue Nov 11, 2018
Fixes to reports.py -- don't crash on control characters, ensure dire…
10ff869 
…ctories earlier

Fixes python#5884
@gvanrossum gvanrossum mentioned this issue Nov 11, 2018
Merged
gvanrossum added a commit that referenced this issue Nov 14, 2018
@gvanrossum
Fixes to reports.py -- don't crash on control characters, ensure dire…
bdc5604 
…ctories earlier (#5885)

Fixes #5884 (both parts). Also fixes testcmdline.py to report unexpected stderr or exit code even when there's an expected output file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gvanrossum