Use DefaultFileSystemClient to do any of the operations with data in S3 using latest version of hadoop-aws 3.4.2 Delta Lake version: master but also in previous versions. For instance in 3.1.0 the ...
When querying a Delta table via the Python API (deltalake), each call to to_pyarrow_table() (or to_pandas()) causes process RSS to grow by hundreds of MB, even when the result is a single row with a ...