Skip to content

ENH: use native filesystem (if available) for read_parquet with pyarrow engine #51609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Mar 14, 2023
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
89e6f9a
ENH: Add filesystem to read_parquet/to_parquet
mroeschke Feb 24, 2023
06a2b18
Add to to_parquet
mroeschke Feb 24, 2023
35d8cbb
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Feb 24, 2023
0b07437
Bump fsspec
mroeschke Feb 24, 2023
f57a195
fix import
mroeschke Feb 24, 2023
6503a7e
Mock gcs to local for parquet
mroeschke Feb 24, 2023
16a3da2
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Feb 24, 2023
dd27604
Fix condidition, add whatsnew
mroeschke Feb 24, 2023
4ce7da8
address tests, bump gcsfs
mroeschke Feb 24, 2023
0b3bd7d
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Feb 24, 2023
e1f8912
bump s3fs
mroeschke Feb 24, 2023
49f2fc8
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Feb 25, 2023
c5166b5
Fix doc issues
mroeschke Feb 25, 2023
7ec7d75
Try without fsspec wrapper
mroeschke Feb 25, 2023
1b9ab77
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Feb 25, 2023
4ce52cd
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 8, 2023
553105c
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 10, 2023
c1161d3
Revert "Try without fsspec wrapper"
mroeschke Mar 10, 2023
ef7095a
Returns a tuple
mroeschke Mar 10, 2023
e6a61a9
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 11, 2023
08f3a30
Don't wrap in fsspec, undo deps bump
mroeschke Mar 11, 2023
7703f0f
Fix whatsnew
mroeschke Mar 11, 2023
e99d234
Add validations for filesystem
mroeschke Mar 11, 2023
f4ef416
Validate that mock filesystem is used
mroeschke Mar 11, 2023
3f0e751
Undo install.rst
mroeschke Mar 11, 2023
970a08f
Try this
mroeschke Mar 11, 2023
d79f29f
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 11, 2023
29bce3b
Make global again?
mroeschke Mar 11, 2023
440eeac
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 13, 2023
0017ae5
Try this
mroeschke Mar 13, 2023
4d3aff9
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 13, 2023
f6ac7ca
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 13, 2023
31593f0
Address review
mroeschke Mar 13, 2023
7f5dad3
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 13, 2023
646dbad
Fix test
mroeschke Mar 13, 2023
70f1e6c
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 13, 2023
96ef2fb
Use localfilesystem correctly
mroeschke Mar 13, 2023
42e614d
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 14, 2023
ae77a26
use absolute
mroeschke Mar 14, 2023
91c9b93
Merge remote-tracking branch 'upstream/main' into enh/parquet/arrow_fs
mroeschke Mar 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address review
  • Loading branch information
mroeschke committed Mar 13, 2023
commit 31593f05f2b38e68f0cbe38ec38b47eb7bf7053d
4 changes: 2 additions & 2 deletions pandas/io/parquet.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,9 @@ def _get_path_or_handle(
)
elif pa_fs is not None and isinstance(fs, pa_fs.FileSystem) and storage_options:
raise NotImplementedError(
"storage_options not supported with a pyarrow Filesystem."
"storage_options not supported with a pyarrow FileSystem."
)
if is_fsspec_url(path_or_handle) and fs is None:
fsspec = import_optional_dependency("fsspec")
if storage_options is None:
pa = import_optional_dependency("pyarrow")
pa_fs = import_optional_dependency("pyarrow.fs")
Expand All @@ -119,6 +118,7 @@ def _get_path_or_handle(
except (TypeError, pa.ArrowInvalid):
pass
if fs is None:
fsspec = import_optional_dependency("fsspec")
fs, path_or_handle = fsspec.core.url_to_fs(
path_or_handle, **(storage_options or {})
)
Expand Down