It should be possible to mark some datasets to be saved to permanent storage for checkpointing. If a failure occurs, then the master should be able to roll back to the previous checkpointed state and rerun all intermediate datasets.