From bc6c8c0e93a79072b94c8c44bec91fbfc65e10c5 Mon Sep 17 00:00:00 2001 From: epriestley Date: Fri, 5 Oct 2018 13:55:10 -0700 Subject: [PATCH] Explicitly shuffle nodes before selecting one for cluster sync Summary: Depends on D19734. Ref T13202. Ref T13109. Ref T10884. See PHI905. See PHI889. We currently rank cluster nodes in three cases: # when performing a write, we can go to any node (D19734 should make our ranking good); # when performing a read, we can go to any node (currently random, but T10884 discusses ideas to improve our ranking); # when performing an internal synchronization before a read or a write, we must go to an up-to-date node. Currently, case (3) is not-exactly-deterministic but not random, and we won't spread intracluster traffic acrosss the cluster evenly if, say, half of it is up to date and half of it is still synchronizing. For a given write, I believe all nodes will tend to synchronize from whichever node first received the write today. Instead, shuffle the list and synchronize from any up-to-date node. (I think we could improve upon this only by knowing which nodes actually have load and selecting the least-loaded -- doable, but not trivial.) Test Plan: Poked at it locally, will deploy to `secure`. This is hard to measure/test terribly convincingly. Reviewers: amckinley Reviewed By: amckinley Maniphest Tasks: T13202, T13109, T10884 Differential Revision: https://secure.phabricator.com/D19735 --- .../diffusion/protocol/DiffusionRepositoryClusterEngine.php | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php b/src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php index f85c7862fe..2e5d4215db 100644 --- a/src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php +++ b/src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php @@ -688,6 +688,9 @@ final class DiffusionRepositoryClusterEngine extends Phobject { 'fetchable.')); } + // If we can synchronize from multiple sources, choose one at random. + shuffle($fetchable); + $caught = null; foreach ($fetchable as $binding) { try {