Erasure coding en HDFS: cómo b... Note

Erasure coding en HDFS: cómo bajar el costo de replicar de 200% a 50%

Erasure coding offers a more efficient way to tolerate data loss compared to simple replication. Instead of storing multiple full copies of files, erasure coding divides data into chunks and adds parity chunks calculated using finite field mathematics. Only a specific number of these combined data and parity chunks are needed to reconstruct the original file, even if some are lost. This approach significantly reduces storage overhead; for instance, a 6+3 scheme (six data chunks, three parity chunks) uses 50% more space, compared to 200% for triple replication. Systems like HDFS, S3 object storage, and RAID 6 utilize erasure coding to ensure data durability against hardware failures. The underlying mathematics, primarily Reed-Solomon codes, are remarkably robust and have applications in various technologies. Implementing erasure coding involves choosing a k+m scheme based on the desired fault tolerance and understanding the trade-offs. While efficient, erasure coding incurs CPU costs during data reconstruction and can introduce latency when data needs to be rebuilt. It is particularly beneficial for cold or infrequently accessed data due to potential overhead on small files. Careful consideration of fragment distribution is also crucial for effective fault tolerance. The same finite field arithmetic that enables erasure coding also powers error correction in QR codes, CDs, and even deep space communication.