A dataflow runtime environment and static scheduler for edge, fog and in-situ computing

Abstract

In the dataflow computation model, tasks are executed according to data dependencies, instead of following program order, enabling natural parallelism exploitation. Sucuri is a dataflow library for Python that allows transparent execution of applications on clusters of multicores, while taking care of scheduling issues. Recent trends in edge/fog/In-situ computing assume that storage and network devices will have processing elements with lower power consumption and performance, which would make a good case for runtime environments that deal with the data versus computation movements trade-off in a more transparent and automated way. This work presents a study on different factors that should be considered when running dataflow applications in in-situ environments, using Sucuri to conduct experiments in a small system emulating a smart storage (in-situ device) utilisation. A static scheduling solution is also presented, allowing Sucuri to choose the most suited approach regarding this in-situ trade-off.