Scalable network reconstruction in subquadratic time Open Access

Networks encode the pairwise interactions that determine the dynamical behaviour of a broad class of interconnected systems. However, in many important cases, the interactions themselves are not directly observed, and instead we have access only to their indirect outcomes, usually as samples from a multivariate distribution modelled as a probabilistic graphical model [1–3], or from time-series data representing some dynamics conditioned on the network structure [4,5]. Instances of this problem include the inference of interactions between microbial species from co-occurrence data [6], financial market couplings from stock prices [7], protein structure from amino-acid contact maps [8], gene regulatory networks from expression data [9], neural connectivity from fMRI and EEG data [10] and epidemic contact tracing [11], among others.

Perhaps, the most well studied formulation of the network reconstruction problem is covariance selection [12], where it is assumed that the data consist of independent samples of a multivariate Gaussian, and the objective is to infer its precision matrix—often assumed to be sparse. The most widely employed algorithm for this purpose is the graphical LASSO (GLASSO) [13] and its many variations [14,15]. More generally, one can consider arbitrary probabilistic graphical models (also known as Markov random fields) [16], where the latent network structure encodes the conditional dependence between variables. Covariance selection is a special case of this family where the variables are conditionally normally distributed, resulting in an analytical likelihood, unlike the general case which involves intractable normalization constants. A prominent non-Gaussian graphical model is the Ising model [17], applicable for binary variables, which has a wide range of applications.

The vast majority of algorithmic approaches so far employed to the network reconstruction problem cannot escape a complexity of at least $O(N2)$⁠, where $N$ is the number of nodes in the network. The original GLASSO algorithm for covariance selection has a complexity of $O(N3)$⁠. By exploiting properties that are specific to the covariance selection problem (and hence do not generalize to the broader reconstruction context), the faster QUIC [18] and BigQUIC [19] approximative methods have $O(N2)$ and $O(NE)$ complexities, respectively, with $E$ being the number of edges (i.e. non-zero entries in the reconstructed matrix), such that the latter also becomes quadratic in the usual sparse regime with $E=O(N)$⁠. Similarly, for the inverse Ising model [17,20] or graphical models in general [16,21] no known method can improve on a $O(N2)$ complexity, and the same is true for reconstruction from time series [4,22]. To the best of our knowledge, no general approach exists to the network reconstruction problem with a lower complexity than $O(N2)$⁠, unless strong assumptions on the true network structure are made. Naively, one could expect this barrier to be a fundamental one, since for the reconstruction task—at least in the general case—we would be required to probe the existence of every possible pairwise coupling at least once.

Instead, in this work, we show that it is in fact possible to implement a general network reconstruction scheme that yields subquadratic complexity, without relying on the specific properties of any particular instance of the problem. Our approach is simple and relies on a stochastic search for the best update candidates (i.e. edges that need to be added, removed or updated) in an iterative manner that starts from a random graph and updates the candidate list by inspecting the second neighbours of this graph—an approach which leads to log-linear performance [23–25]. Furthermore, every step of our algorithm is easily parallelizable, allowing its application for problems of massive size.

This paper is organized as follows. In §2, we introduce the general reconstruction scenario, and the coordinate descent (CD) algorithm, which will function as our baseline with quadratic complexity. In §3, we describe our improvement over the baseline and analyse its algorithmic complexity. In §4, we evaluate the performance of our approach on a variety of synthetic and empirical data and, in §5, we showcase our algorithm with some selected large-scale empirical network reconstruction problems. We finalize in §6 with a conclusion.

Algorithm 1 has complexity $O(τN2)$⁠; assuming step (1), corresponding to an element-wise optimization, can be done in time $O(1)$ (e.g. using bisection search),¹ where $τ$ is the number of iterations required for convergence—which, in general, will depend on the particulars of the problem and initial state, but typically we have $τ=O(1)$⁠. We observe that in all our analyses we assume that the initial state $W0$ corresponds to an entirely empty network with every entry being equal to zero—corresponding to the worst-case scenario of maximum initial ignorance about the network to be reconstructed.

Note that the CD algorithm does not require a differentiable objective $π(W)$⁠, but convergence to the global optimum is only guaranteed if it is convex and sufficiently smooth [28]. In practice, CD is the method of choice for covariance selection and the inverse Ising model, with a speed of convergence that often exceeds gradient descent (which is not even strictly applicable when non-differentiable regularization is used, such as $L1$ of GLASSO [13]), since each coordinate can advance further with more independence from the remaining ones, unlike with gradient descent where all coordinates are restricted by the advance of the slowest one.

For a non-convex objective $π(W)$⁠, the CD algorithm will, in general, not converge to the global optimum. Nevertheless, it is a fundamental baseline that often gives good results in practice even in non-convex instances and can serve as a starting point for more advanced algorithms. In this work, we are not primarily concerned with issues due to non-convexity, but rather with a general approach that circumvents the need to update all $(N2)$ entries of the matrix $W$⁠.

Our objective is to reduce the $O(N2)$ complexity of the CD algorithm. But, before continuing, we remark on the feasibility of the reconstruction problem, and the practical obstacle that this quadratic complexity represents. At first, one could hypothesize that the size of the data matrix $X$ would need to be impractically large to allow for the reconstruction of networks with $N$ in the order of hundreds of thousands or millions. In such a case, a quadratic complexity would be the least of our concerns for problem sizes that are realistically feasible. However, for graphical models, it is possible to show that the number of samples required for accurate reconstruction scales only with $M=O(log⁡N)$ [16,21,29–31], meaning that reconstruction of large networks with relatively little information is possible. In view of this, a quadratic algorithmic complexity on $N$ poses a significant obstacle for practical instances of the problem, which could easily become more limited by the runtime of the algorithm than the available data.

As mentioned in the introduction, our approach consists of implementing a stochastic second-neighbour search to find the entries of the matrix $W$ that yield the largest improvement of the objective function. We proceed in three steps, we define: (i) a greedy extension of the CD algorithm 1 that updates only the best entries per iteration; (ii) a function that finds the best entries to be updated according to an iterated search for the $k$ nearest neighbours (KNNs) of each node; and (iii) a function that solves the KNN problem in subquadratic time using a stochastic second-neighbour search.

In the greedy extension of the CD algorithm 1 (GCD), we select only the $κN$ entries of the matrix $W$ that would individually lead to the steepest increase of the objective function $π(W)$⁠, as summarized in algorithm 2.

As we shall discuss in a moment, the recursion depth of algorithm 3 is bounded logarithmically on $|S|$⁠, and hence its runtime is dominated by the KNN search. Note that so far we have done nothing substantial to address the overall quadratic performance, since finding the KNNs exhaustively still requires all pairs to be probed. Similarly to the $m$ closest pairs problem, efficient log-linear algorithms exist based on spacial sorting when the distance is Euclidean; however more general approaches do also exist. In particular, the NNDescent algorithm by Dong et al. [23] approximately solves the KNN problem in subquadratic time, while not requiring the distance function to be a metric. The algorithm is elegant and requires no special data structure beyond the nearest neighbour graph itself, other than a heap for each node. It works by starting with a random KNN digraph, and successively updating the list of best neighbours by inspecting the neighbours of the neighbours in the undirected version of the KNN digraph, as summarized in algorithm 4. The main intuition behind this approach is that if $(i,j)$ and $(j,v)$ are good entries to update, then $(i,v)$ is likely to be a good candidate as well—even if triangle inequality is not actually obeyed.

Steps (1) and (2) in the algorithm can be both performed in time $O(1)$ by keeping a (e.g. Fibonacci) heap containing the nearest $k$ neighbours for each node. Thus, each full iteration of algorithm 4 runs in time $O(k2N)$⁠, if the degrees of nodes in $U$ are all $O(k)$⁠, otherwise it would run in time $O(⟨q⟩N)$ where $⟨q⟩$ is the average number of second neighbours in $U$⁠. In case $⟨q⟩≫k2$⁠, the inner loop is then modified to run over only the first $k$ neighbours for each node, to preserve the $O(k2N)$ complexity, regardless of any structure that $U$ may have. Although this algorithm has seen wide deployment, in particular as part of the popular UMAP dimensionality reduction method [34], and has had its performance empirically ‘battle tested’ in a variety of practical workloads, it has so far resisted a formal analysis of its algorithmic complexity. To date, the most careful analyses of this algorithm observe that the number of iterations required for convergence does not exceed $2⌈log2k⁡N⌉$ in empirical settings [25]. The intuitive reasoning is that the initial random graph has as a diameter of approximately $⌈log2k⁡N⌉$⁠, and hence twice this is the number of steps needed for each node to communicate its neighbourhood to all other nodes, and for the updated information to return.² This conjectured bound on the convergence results in an overall $O(k2Nlog⁡N)$ complexity. However, this conjecture can only be rigorously proven on a version of the algorithm where the second-neighbour search is replaced by a range query, and the data are generated by a homogeneous Poisson process [24]. These conditions are not applicable to our problem (and we do not assume them); however as we see in our numerical experiments, we find robust evidence that it holds to the case of network reconstruction as well. This typical log-linear complexity of the NNDescent algorithm is what finally enables us to escape the quadratic complexity of the CD algorithm, as we demonstrate shortly.

Importantly, the NNDescent algorithm is approximative, as it does not guarantee that all nearest neighbours are always correctly identified. Although in practice it often yields very good recall rates [23], its inexact nature is not a concern for our purposes, since it does not affect the correctness of our GCD algorithm: if an element of the best set $Ebest$ in algorithm 2 is missed at a given iteration, it will eventually be considered in a future iteration, owing to the random initialization of algorithm 4. Our primary concern is only with the average speed with which it finds the best update candidates. Nevertheless, as we show later, inaccuracies in the solution of the KNN problem often disappear completely by the time algorithm 3 finishes (as the KNN problem considers a much larger set of pairs than what gets eventually selected), such that the recall rates for the $m$ closest pairs problem approach the optimal values as the parameter $κ$ is increased.

In appendix C, we list some low-level optimizations that can improve the overall performance of the algorithm, including parallelization. A C++ implementation of the algorithm (with OpenMP CPU parallelism) is available as part of the graph-tool Python library [36].

We now obtain the overall algorithmic complexity of our GCD algorithm. Since it consists in repeatedly finding the best $κN$ entries in the matrix $W$ to be updated, its overall runtime will depend crucially on the $FINDBEST(κN,V,D)$ function defined in algorithm 3.

The prefactor in the above expression will, in general, depend on the data and the stage of the algorithm, as we shall see.

We can now obtain a loose upper bound to the running time of FindBest by assuming the worst case where the progression of the algorithm is (in principle) the slowest.

However, although already subquadratic, this upper bound is not tight. This is because it is not in fact possible for the worst case $st=1/2t$ to be realized at every recursion, and the number of nodes being considered will generically decrease faster than exponentially. We notice this by performing a more detailed analysis of the runtime, as follows.

We now consider a few representative cases for how the output of FindBest may be, and consider their respective running times.

We move now to a case with intermediary heterogeneity, with the output following a geometric degree distribution.

This result means that for relatively more heterogeneous degree distributions we accrue only an additional $log⁡N$ factor in comparison to the $d$-regular case, and remain fairly below the upper bound found previously.

Based on the above, we can expect broader degree distributions in $A$ to cause longer runtimes. A more extreme case is given by the Zipf distribution, which we now consider.

We emphasize that the graph $A$ considered for the complexity analysis above is distinct from the final network $W$ we want to reconstruct. The latter might have an arbitrary structure, but the graph $A$ represents only the updates that need to be performed, and it has a density which is controlled by the parameter $κ$ of our algorithm. Thus, even if $W$ has a very broad degree distribution, the one we see in $A$ will be further limited by the parameter $κ$ and for which updates are needed by the GCD algorithm, and in general $W$ and $A$ will be very different from each other (for an example, compare panels (d) and (f) in figure 2).

This kind of log-linear performance is promising for the scalability of the overall algorithm, but it stills needs to be determined if the results of FindBest are sufficiently accurate for a speedy progression of the GCD algorithm. In figure 4, we show the cumulative recall rates of the exact best pairs obtained with an exhaustive algorithm, defined as the fraction of best pairs correctly identified up to a given rank position (with the best pair having rank 1), for a variety of empirical data. We observe in general very good recall rates, compatible with what is obtained with the NNDescent algorithm [23]. Importantly, in every case we considered, we observed that the cumulative recall values start at $1$⁠, meaning that the first best edges are always correctly identified. For some data, the algorithm may falter slightly for intermediary pairs and a low $κ$⁠, but increasing $κ$ has the systematic effect of substantially improving the recall rates (at the expense of longer runtimes). We emphasize again that it is not strictly necessary for the FindBest function to return exact results, since it will be called multiple times during the GCD loop, and its random initialization guarantees that every pair will eventually be considered—it needs only to be able to locate the best edge candidates with high probability. Combined with the previous result of figure 3, this indicates that the fast runtime of the FindBest function succeeds in providing the GCD algorithm with the entries of the $W$ matrix that are most likely to improve the objective function, while avoiding an exhaustive $O(N2)$ search.

We demonstrate the use of our algorithm with a few large-scale datasets, which would be costly to analyse with a quadratic reconstruction algorithm. We showcase on the following:

A collection of crowd-sourced microbial samples from various biomes and habitats across the globe, containing abundances of operational taxonomic units (OTUs), obtained via DNA sequencing and mass spectrometry. An OTU is a proxy for a microbial species, and a single sample consists of the abundances of each OTU measured at the same time. This dataset consists of $M=23 828$ samples involving $N=317 314$ OTUs.

A collection of microbial samples from 300 healthy adults between the ages of 18 and 40, collected at five major body sites (oral cavity, nasal cavity, skin, gastrointestinal tract and urogenital tract) with a total of 15 or 18 specific body sites. The abundances of OTUs were obtained using 16S rRNA and whole metagenome shotgun (mWGS) sequencing. This dataset consists of $M=4788$ samples involving $N=45 383$ OTUs.

For both co-occurrence datasets above, we binarized each sample as $xi∈{−1,1}$⁠, for absence and presence, respectively, and used the Ising model for the network reconstruction (see appendix B). In this case, the matrix $W$ yields the strength of the coupling between two OTUs, and a value $Wij=0$ means that OTUs $i$ and $j$ are conditionally independent from one another.

This database consists of genome-wide gene expression reads for 10 animal species as well as budding and fission yeast. Each gene expression sample was obtained using RNAseq, with read counts converted to base-2 logarithms after adding a pseudo-count of 0.125, and batch corrected using Combat [42]. We used the nematode dataset, consisting of $M=1357$ samples of $N=17 256$ genes. For this dataset, we used the multivariate Gaussian model (see appendix. B), where the matrix $W$ corresponds to the inverse covariance between gene expression levels (also known as precision matrix). In this case, the partial correlation between two genes, i.e. their degree of association controlling for all other genes, is given by $−Wij/WiiWjj$⁠, so that gene pairs with $Wij=0$ are conditionally independent.

The reconstructed networks for all three datasets are shown in figure 7. They all seem to display prominent modular structure. In the case of microbial co-occurrence datasets the clusters correspond mostly to different habitats and geographical regions for the earth microbiome, and to different body sites for the human microbiome.

We have described a method to reconstruct interaction networks from observational data that avoids a seemingly inherent quadratic complexity in the number of nodes, in favour of a data-dependent runtime that is typically log-linear. Our algorithm does not rely on particular formulations of the reconstruction problem, other than the updates on the edge weights being done in constant time with respect to the total number of nodes. Together with its straightforward parallelizability, our proposed method removes a central barrier to the reconstruction of large-scale networks and can be applied to problems with a number of nodes and edges on the order of hundreds of thousands, millions or potentially even more depending on available computing resources.

Our algorithm relies on the NNDescent [23] approach for approximate KNN search. Despite the robust empirical evidence for its performance, a detailed theoretical analysis of this algorithm is still lacking, in particular of its potential modes of failures. We expect further progress in this direction to elucidate potential limitations and improvements to our overall approach.

In this work, we focused on convex reconstruction objectives, such as the inverse Ising model and multivariate Gaussian with $L1$ regularization. More robust regularization schemes or different models may no longer be convex, in which case CD will fail, in general, at converging to the global optimum. However, it is clear that our strategy of finding the best edge candidates in subquadratic time is also applicable for algorithms that can be used with non-convex objectives, such as stochastic gradient descent or simulated annealing. We leave such extensions for future work.

All amplicon sequence data and metadata have been made public through the EMP data portal (http://qiita.microbio.me/emp).

I have not used AI-assisted technologies in creating this article.

T.P.: conceptualization, data curation, formal analysis, funding acquisition, investigation, methodology, project administration, resources, software, supervision, validation, visualization, writing—original draft, writing—review and editing

I declare I have no competing interests.

This work has been funded by the Vienna Science and Technology Fund (WWTF) and by the State of Lower Austria [Grant ID: 10.47379/ESS22032].

Sample processing, sequencing, and core amplicon data analysis were performed by the Earth Microbiome Project (www.earthmicrobiome.org), and all amplicon sequence data and metadata have been made public through the EMP data portal (http://qiita.microbio.me/emp).

Nevertheless, for problems that lie outside of the aforementioned class, if a single coordinate update has an arbitrary algorithmic complexity given by $Δ(N)$⁠, then the CD baseline would have an overall complexity of $O(Δ(N)N2)$⁠, while our improved algorithm would have a typical complexity of $O(Δ(N)Nlog2⁡N)$⁠, thereby providing precisely the same relative speedup.

In both cases, we have an additional set of $N$ parameters $θ$ which we update alongside the matrix $W$ in our algorithms. Updates on an individual entry $Wij$ of $W$ can be computed in time $O(1)$ (independently of the degrees of $i$ and $j$ in a sparse representation of $W$⁠) by keeping track of the weighted sum of the neighbours $mi=∑j≠iWijxj$ for every node and updating it as appropriate.

Below, we describe a few low-level optimizations that we found to give good improvements to the runtime of the algorithm we propose in the main text.

(a) Caching

The typical case for objective functions $π(W)$ is that the computation of $maxWijπ(W)$ will require $O(M)$ operations, where $M$ is the number of data samples available. Since this computation is done in the innermost loops of algorithm 4, it will amount to an overall multiplicative factor of $O(M)$ in its runtime. However, because the distance function $D(i,j)$ will be called multiple times for the same pair $(i,j)$⁠, a good optimization strategy is to cache its values, for example in a hash table, such that repeated calls will take time $O(1)$ rather than $O(M)$⁠. We find that this optimization can reduce the total runtime of algorithm 4 by at least one order of magnitude in typical cases.

(b) Gradient as distance

The definition of $D(i,j)=−maxWijπ(W)$ is sufficient to guarantee the correctness of algorithm 3, but in situations where it cannot be computed in closed form, requiring for example a bisection search, a faster approach is to use instead the absolute gradient $D(i,j)=−|∂/∂Wijlog⁡π(W)|$⁠, which often can be analytically computed or well approximated with finite difference. In general, this requires substantially fewer likelihood evaluations than bisection search. This approach is strictly applicable only with differentiable objectives, although we observed correct behaviour for $L1$-regularized likelihoods when approximating the gradient using central finite difference. We observed that this optimization improves the runtime by a factor of approximately six in typical scenarios.

(c) Parallelization

The workhorse of the algorithm is the NNDescent search (algorithm 4), which is easily parallelizable in a shared memory environment, since the neighbourhood of each node can be inspected, and its list of nearest neighbours can be updated, in a manner that is completely independent of the other nodes, and hence requires no synchronization. Thus, the parallel execution of algorithm 4 is straightforward.

The actual updates of the matrix $W$ in the GCD algorithm 2 can also be done in parallel, but that requires some synchronization. For many objectives $π(W)$⁠, we can only consider the change of one value $Wij$ at a time for each node $i$ and $j$⁠, since the likelihood will involve sums of the type $mi=∑jWijxj$ for a node $i$⁠, where $xj$ are data values. Therefore, only the subset of the edges $Ebest$ in algorithm 2 that are incident to an independent vertex set in the graph will be able to be updated in parallel. This can be implemented with mutexes on each node, which are simultaneously locked by each thread (without blocking) for each pair $(i,j)$ before $Wij$ is updated, which otherwise proceeds to the next pair if the lock cannot be acquired. Empirically, we found that is enough to keep up to 256 threads busy with little contention for $N>104$ with our OpenMP implementation [36].