sparklearning

Serialization

Stories about how Spark encodes data: Tungsten’s binary format, Kryo, and the wire format between driver and executors.

Planned stories:

• Why Spark moved to a binary row format (Tungsten)
• Serialization over the wire: task serialization, shuffle, broadcast
• When Kryo (or other serializers) are used

Stories will be added here gradually.