Project Every Record Counts: Data Engineering for Backend Engineers
Part 3: "Bytes on Disk" — File Formats, Parquet and Object Storage
Rohan writes the same six million sales lines as CSV, JSON, Avro and Parquet and asks object storage for them back. Row vs columnar formats, what is inside a Parquet file, the sort order that lets a filter skip 49 of 50 row groups, Apache Arrow, and what object storage gives you instead of a filesystem.