cc2p

CC2P (Convert CSV To Parquet) is a high-performance command-line tool written in Rust that efficiently converts CSV files to the Apache Parquet format. Parquet is a columnar storage file format that offers efficient data compression and encoding schemes, making it ideal for big data processing.

Why Use CC2P?

- Performance: Leverages Rust's speed and multi-threading for fast conversions
- Memory Efficiency: Processes files with minimal memory footprint
- Flexibility: Supports various CSV formats with different delimiters and header options
- Schema Inference: Automatically detects column types from your data
- Batch Processing: Convert multiple CSV files in a single command

Features

- Columnar Storage: Parquet's columnar format provides better compression and faster query performance compared to row-based formats like CSV
- Efficient Compression: Uses Snappy compression for a good balance between compression ratio and speed
- Schema Handling: Automatically infers data types and handles duplicate column names
- Parallel Processing: Multi-threaded conversion using Tokio runtime
- Progress Tracking: Real-time progress indication with indicatif progress bars