Skip to main content

How to send packed array from math kernel to 3rd party app via WSTP


Background



I'm developing a small python wrapper PyWSTP. The wrapper is still in very early stage and lacking features. One important feature I'm considering is transferring dense array between math kernel and python. (Yep, the all awesome NumPy) I do realize LibraryLink would be better suited for this, but WSTP has the capability of sending stuff over TCP.


Problem


Whenever kernel sends packed array to a 3rd party app via LinkWrite[_LinkObject, _], it sends unpacked array. On the receiving side, tokens such WSTKPACKED are never seen. Thus WSTP C APIs such as WSGetReal64Array cannot be used.


Here's some simple code to illustrate the problem:


server.m


(* run via "math -script" *)
Needs["Developer`"];

link = LinkCreate["testlink", LinkMode->Listen];


obj1 = Image[RandomInteger[{0, 255}, {4, 4}]];
obj2 = Developer`ToPackedArray[RandomReal[1., {4,4}]];
LinkWrite[link, #]& /@ {obj1, obj2};
InputString["Done, ENTER to exit ..."];

LinkClose[link];

client.py


import wstp


ctx = wstp.WSTPContext()
link = ctx.open_link('testlink', mode='connect')

obj1 = link.recv()
print('got object1:')
print(obj1)
obj2 = link.recv()
print('got object2:')
print(obj2)
input('Done, ENTER to exit ...')


ctx.close_link('testlink')

On PyWSTP, I have not implemented handlers for WSTKPACKED yet. Only basic data types (WSTKSYM, WSTKFUNC, WSTKINT, WSTKREAL, WSTKSTR) are implemented. Yet client side is able to correctly receive the object:


got object1:
Image[RawArray["Real64", List[List[207.0, 142.0, 166.0, 205.0], List[137.0, 115.0, 60.0, 39.0], List[37.0, 110.0, 9.0, 89.0], List[56.0, 162.0, 116.0, 225.0]]], "Real", Rule[ColorSpace, Automatic], Rule[Interleaving, None]]
got object2:
List[List[0.8681530207587866, 0.0734728851659936, 0.9233463976844913, 0.1346824422121633], List[0.7711652733626824, 0.88933420636346, 0.9202474236790081, 0.5288719560153152], List[0.18096422954894353, 0.04400711306110572, 0.2330108388845873, 0.7479540642843265], List[0.4382030298759463, 0.42470372650663424, 0.9718400392443247, 0.6100416353164417]]
Done, ENTER to exit ...


This indicates math kernel unpacks dense array before sending over WSTP.


Question summary




  1. How to make sure kernel sends packed array to 3rd party app over WSTP?




  2. Is it possible to achieve 1. without writing C code?







Comments

Post a Comment

Popular posts from this blog

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

mathematical optimization - Minimizing using indices, error: Part::pkspec1: The expression cannot be used as a part specification

I want to use Minimize where the variables to minimize are indices pointing into an array. Here a MWE that hopefully shows what my problem is. vars = u@# & /@ Range[3]; cons = Flatten@ { Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; Minimize[{Total@((vec1[[#]] - vec2[[u[#]]])^2 & /@ Range[1, 3]), cons}, vars, Integers] The error I get: Part::pkspec1: The expression u[1] cannot be used as a part specification. >> Answer Ok, it seems that one can get around Mathematica trying to evaluate vec2[[u[1]]] too early by using the function Indexed[vec2,u[1]] . The working MWE would then look like the following: vars = u@# & /@ Range[3]; cons = Flatten@{ Table[(u[j] != #) & /@ vars[[j + 1 ;; -1]], {j, 1, 3 - 1}], 1 vec1 = {1, 2, 3}; vec2 = {1, 2, 3}; NMinimize[ {Total@((vec1[[#]] - Indexed[vec2, u[#]])^2 & /@ R...
Post a Comment
Read more

How to remap graph properties?

Graph objects support both custom properties, which do not have special meanings, and standard properties, which may be used by some functions. When importing from formats such as GraphML, we usually get a result with custom properties. What is the simplest way to remap one property to another, e.g. to remap a custom property to a standard one so it can be used with various functions? Example: Let's get Zachary's karate club network with edge weights and vertex names from here: http://nexus.igraph.org/api/dataset_info?id=1&format=html g = Import[ "http://nexus.igraph.org/api/dataset?id=1&format=GraphML", {"ZIP", "karate.GraphML"}] I can remap "name" to VertexLabels and "weights" to EdgeWeight like this: sp[prop_][g_] := SetProperty[g, prop] g2 = g // sp[EdgeWeight -> (PropertyValue[{g, #}, "weight"] & /@ EdgeList[g])] // sp[VertexLabels -> (# -> PropertyValue[{g, #}, "name"]...
Post a Comment
Read more