Codepickle: Portable Python Grid Computing

Abstract

We present Codepickle, a groundbreaking solution designed to overcome Python version constraints and enhance portability, especially within distributed Python grids and volunteer computing environments. Unlike existing serialization libraries such as Cloudpickle and Dill, which rely on Python bytecode and are prone to version conflicts, Codepickle offers a robust alternative. Our methodology includes innovative adjustments for function serialization and shared variable management. Experimental results reveal challenges like code sourcing and nonlocal variable handling. Performance benchmarks highlight Codepickle's significant advantages over Cloudpickle, including better portability and reduced message sizes. Notably, Codepickle achieves message sizes that are 84% of those produced by Cloudpickle especially for small code segments, with comparable execution performance. Proposed enhancements target critical issues such as lambda functions and cross-version compatibility. This comprehensive study not only demonstrates Codepickle's transformative potential but also underscores the ongoing quest for advanced serialization techniques in Python's distributed computing landscape.