Hi Michel - You've been busy with this and the Postgres lib. Good work. The Gremlin User Group has been working on a new universal graph DSL and compiler that can compile down to any of the graph backends. GraphBLAS has been part of the discussion since the start. An early draft of mm-ADT was just released last week...
This looks really interesting, potentially makes GraphBLAS much more accessible for exploratory work. A few questions for the author:
Does this work in blocking or non-blocking mode? Naively I imagine there might be more opportunity for the GraphBLAS implementation to optimize execution in non-blocking mode.
Is there a way to efficiently store and load matrices to and from files? Ideally in a such a way that the data is just mmap'ed or copied directly into memory on load?
> Does this work in blocking or non-blocking mode? Naively I imagine there might be more opportunity for the GraphBLAS implementation to optimize execution in non-blocking mode.
Non blocking mode.
> Is there a way to efficiently store and load matrices to and from files? Ideally in a such a way that the data is just mmap'ed or copied directly into memory on load?
The Matrix.from_mm and Matrix.to_mm methods will read and write files in the standard Matrix Market format. They are wrappers around the underlying LAGraph functions.
It uses a lot of the GxB "extensions" that are to my knowledge only present in SuiteSparse. Investigating other implementations has been on my radar but not a high priority, certainly very open to getting PRs for helping with interchangeability!
The GraphBLAS GPU code is in the works [1]. For storage see the new RedisGraph 1.0 implementation released last year and 'michelp has a Python Postgres implementation in development.
For the bioinformatics datasets I work with it is not cost efficient to load everything into a database. For some of these datasets (e.g. GWAS - Genome Wide Association Studies which are essentially sparse matrices) it might be interesting to explore with graph queries. I guess my ideal would be to have a GraphBLAS equivalent to Spark SQL queries working across files in cloud storage / NAS.
There are several teams working on distributed GraphBLAS (the GraphBLAS/D4M model was designed to run on supercomputers [0]). Kepner's team at MIT is one [1].
NB: D4M was the original name before it was changed to GraphBLAS and became a standard.
RAPIDS has picked up Dask for multi-gpu aspects of cudf (think spark/pandas on GPUs), and as cugraph is single GPU (https://github.com/rapidsai/cugraph) for going fast on ~billion row datasets... I'm guessing dask+cugraph will be happening for the next 100-1000X, if not already.
Graph partitioning is a weird world, so will be interesting to see!
Interesting - would be interested in a comparison between GraphBLAS (which I had not heard of until just now) and, for example, graph-tool's (https://graph-tool.skewed.de/) underlying algorithms (Boost Graph Library).
It seems that they serve very different purposes. Graphblas is mostly focused on the processing of real-valued functions defined on the vertices and edges of large-scale sparse graphs, and is optimized for this use case (think graphs with thousands of millions of vertices). The boost graph library is not tailored to numeric functions and it will probably be less efficient for this use case.
GraphBLAS is also the engine behind RedisGraph. Sparse adjacency matrices are interesting from a graph database perspective in particular because they are typically faster and more compact than the mainstream index-free adjacency systems (e.g. Neo4j).
pygraphblas author here. scipy.sparse and graphblas are superficially similar in that they provide a sparse matrix type, but scipy.sparse is oriented more toward mathematical matrix computation and provides a wonderful and huge bag of tools to work with, for example, solving and doing LU decomposition, see scipy.sparse.linalg.
graphblas provides a different bag of tools oriented toward solving graph problems, primarily around semirings and custom graph element operation. I don't think scipy.sparse, for example, provides a means for plugging in different semiring operations for computing shortest paths or max flow. On the other hand, graphblas does not provide decomposition or solving algorithms like scipy.sparse does.
It is my goal to make it so that scipy.sparse matrices, dense numpy matrices, and networkX graphs can be interchangeable used with pygraphblas in the most reasonably efficient way. This way we get all the tools in both bags. Would love to chat with more scipy developers to see where there can be unification.
UPDATE: it looks like there is some support for graph solving with scipy.sparse.csgraph, would be interesting to see how these compare to GraphBLAS based approaches!
mm-ADT: A Multi-Model Abstract Data Type http://rredux.com/mm-adt/