I am happy to present pyibverbs, a minimalistic wrapper for Linux VERBS API for Infiniband.
InfiniBand (abbreviated IB) is a computer-networking communications standard used in high-performance computing that features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also used as either a direct or switched interconnect between servers and storage systems, as well as an interconnect between storage systems.
InfiniBand has no standard API. The standard only lists a set of verbs such as ibv_open_device or ibv_post_send, which are abstract representations of functions or methods that must exist. The syntax of these functions is left to the vendors. Sometimes for reference this is called the verbs API.
libibverbs, developed and maintained by Roland Dreier since 2006, are de-facto the verbs API standard in *nix.
Till now there has been no Verbs API for Python 3.x and the existing libibverbs wrapper for Python 2.x was very complete but at the same time not very pythonic (I didn’t have this batteries-included feel 😉 – please don’t hesitate to compare the two for yourself).
The aim of pyibverbs project is providing a minimalistic wrapper for libibverbs which works in Python 2/3 and allows to exchange NumPy arrays between Infiniband nodes. The style of the API is very pythonic and gives a fully functional application in only a few lines of code. Data rates are easily an order of magnitude higher than using IPoIB.
cython ibverbs.pyx gcc -c ibverbs.c -o ibverbs.o gcc -shared ibverbs.o -o ibverbs.so -lpython3.6 -libverbs cp -iv ibverbs.so /usr/lib/python3.6/site-packages/ibverbs.so
from ibverbs import IBDeviceList, \ IBAccessFlags as acc import numpy as np A = np.zeros((512, 512, 512)) # # ibport, remote_qpn, remote_psn, remote_lid, # local_psn, remote_vaddr and remote_rkey # must be established using a side-channel # with IBDeviceList() as dlst, dlst.open() as ctx, ctx.protection_domain() as pd, pd.memory_region_from_array(A, acc.LOCAL_WRITE | acc.REMOTE_WRITE) as mr, ctx.completion_channel() as chan, ctx.completion_queue() as rcq, ctx.completion_queue(chan) as scq, pd.queue_pair(rcq, scq) as qp: qp.change_state_init(ibport) qp.change_state_ready_to_receive(remote_qpn, remote_psn, remote_lid, ibport) qp.change_state_ready_to_send(local_psn) qp.post_send(mr, 3, remote_vaddr, remote_rkey) scq.wait_complete()
Primitives for side channel data encapsulation and exchange are provided in IBConnSpec class. Examples of usage for data send/receive are placed in test_verbs_cli.py and test_verbs_srv.py. In the examples, ZMQ is used for the exchange of side-channel information but naturally it could be anything.
You will find the code in my GitHub Repository: https://github.com/sadaszewski/pyibverbs.