On Modular Learning of Distributed Systems for Predicting End-to-End Latency