mathlink or wstp - Asynchronous Programming

Consider that you would like to listen for Raw WiFi Packets coming into your NIC. Since Mathematica doesn't have built-in functions for such low level programming (as far as I know), one alternative would be following the steps:

1. launch an executable file through Mathematica: Run["ReceivePacket.exe > packet.tmp"]


2. read output file "packet.tmp"



3. do simple data analysis


4. iterate loop (go back to step 1)

The principle of the executable file "ReceivePacket.exe" is simple. It waits for a single Raw WiFi Packet to arrive. When the packet arrives, the data is written to a dump file and the executable ends. Fortunately, the hard work has already been done by WinPcap, so all that is necessary is to compile the tutorial saving packets to a dump file (with minor modifications).

However, there is a significant drawback in this process. By the time Mathematica read the output file "packet.tmp" and re-launch "ReceivePacket.exe", several packets would be 'lost'. Anticipating possible comments: This happens not because the data analysis is time consuming (on the contrary, the data analysis is very simple) neither because window pop-up is slow (the process is in batch mode with no print to screen).

1st Issue
Is it possible to improve this process in order to decrease the amount of packet loss?

2nd Issue
Is it possible to use MathLink to improve this process?

Answer

Let me give you a very basic example, how you can employ an asynchronously running LibraryLink function for this specific task. I will not do any real packet listen, but only explain the general setup.

Usually, a LibraryFunction in Mathematica is written in C and wrapped with a very small boiler plate code that is needed to attach the function directly to the Mathematica Kernel. Once this library is compiled, it can be loaded by LibraryFunctionLoad which gives you a Mathematica function that can be called like a normal Mathematica function. Exactly like other functions, the library function blocks the Kernel until it is finished.

This would be a problem, because what you want is a packet listener that runs in the background, catching all network packets while you can analyse them one after another. Therefore, your library packet listener must not block your Kernel. For this purpose, Mathematica provides you with a heavily undocumented API for asynchronous tasks. What I tell you here can only be found out by trying and reading example code.

You need to study the C file very carefully

FileNames["async-tasks-repeating.c", {$InstallationDirectory}, Infinity]

and I will use the function IntBackgroundTask as an example here. That means, that our network packets are simple integers. Let me stop here and just present you a running example. After this, I will try to answer questions you might ask yourself. A very uncommon form of an answer, but lets give it a try:

libFile = First@FileNames["async-tasks-repeating.c", {$InstallationDirectory},
     Infinity];
libMain = FileNameJoin[{DirectoryName[libFile], "async-examples-libmain.c"}];


<< CCompilerDriver`
lib = CreateLibrary[{libFile, libMain}, "asyncLib", 
  "IncludeDirectories" -> {DirectoryName[libMain]}];
packetListener = LibraryFunctionLoad[lib, 
  "start_int_repeating_background_task", {Integer, Integer}, Integer];

pauseInMilliseconds = 500;
packetCounterInit = 0;


taskID = Internal`CreateAsynchronousTask[packetListener, 
  {pauseInMilliseconds, packetCounterInit}, (packet = {##}) &]

Dynamic[packet]

Let's explain what we did:

What is this libMain file? It is common code that is used in many examples. Therefore, they put it into a separate file once and you have to compile it as well. Btw, giving "IncludeDirectories" in the CreateLibrary call is important so that "async-examples.h" can be found.

• Attention: In version 9.0 the file "async-examples.h" seems to be missing completely. Therefore, please create a file "async-examples.h" in the same directory where libMain is located and copy the following content into it.

How do I know the argument types inside LibraryFunctionLoad? (1) Look into the C code of "start_int_repeating_background_task". You see that Args[0] and Args[1] (line 43-44) are accessed and both with getInteger. In the end, you see that the result is set as Integer as well (line 46). (2) You study carefully the Examples.m file in the following directory

FileNames["LibraryLink", {$InstallationDirectory}, Infinity]

This will help to clear a lot other questions too!

How does this CreateAsynchronousTask work? The basic approach is the following: Your library function sets up the background task by initializing the arguments for it. Then you need to call createAsynchronousTaskWithThread (line 45) which returns an unique taskID that you return as result. You don't just call this library function! You need to register it by calling Internal`CreateAsynchronousTask which will directly start the background task for you. The last argument there is an event function that is called, every time your C loop calls raiseAsyncEvent (line 29).

But I have network packets and not only integers. What should I do? If you look in the C code, you see that a DataStore is used (line 21,27,28) to store the result in each iteration of the loop. Please look into the file

FileNames["WolframLibrary.h", {$InstallationDirectory}, Infinity]

starting in line 133 you find all available functions for asynchronous tasks and DataStore. Instead of adding only an integer to your data store, you can add all basic types and even give them names for easier access. Therefore, let's say you do something like this with your data store

ioLibrary->DataStore_addNamedInteger(ds, "Data", eventdata);

ioLibrary->DataStore_addNamedString(ds, "Name", "Packets");

then the data you receive in Mathematica will look like this

{"Data" -> 8, "Name" -> "Packets"}

This is a very convenient way to send important information of your network packet back.

The last argument of Internal`CreateAsynchronousTask is confusing. Where do I put my packet analyzing code? The last argument of CreateAsynchronousTask is the call-back function that is called when a new result is pushed by raiseAsyncEvent in your asynch. task. This callback function gets 3 arguments and the correspond directly to the 3 arguments you have given raiseAsyncEvent. Let us look at its declaration in WolframLibrary.h

void raiseAsyncEvent(mint asyncTaskID, char* eventType, DataStore data)

Therefore, your call-back function should get 3 arguments:

• First, the AsynchronousTaskObject itself that can be used to stop the running background task (or start it again).


• The second argument is a string that you could use to e.g. indicate whether a bad packet arrived. The eventType can therefore be used to handle different situations in your call-back function.


• The third argument is always a DataStore that contains data that can be used in Mathematica.

Therefore, instead of using (packet = {##}) & like I did which simply sets the variable packet for dynamic displaying, you could define a real function:

    analyzePacket[task_, "Offline", _] := StopAsynchronousTask[task];
    analyzePacket[_, "Packet", data_] := DoSomethingWithData[data];
    analyzePacket[_, "BadPacket", _] := DoNothing[];

In your C code packet sniffing loop, you make a discrimination when e.g. your WIFI is offline. Then you send something like raiseAsyncEvent(asyncObjID, "Offline, NULL), or when a normal packet arrived, you use raiseAsyncEvent(asyncObjID, "Packet", packetData), etc.

What else can I do with the AsynchronousTaskObject that you saved in the variable taskID? Well, we have 4 highlevel Mathematica function that you need to really control starting and stopping of asynchronous tasks:

Additionally, you should look at

Options[Internal`CreateAsynchronousTask]
(* {"PostRemoveAsynchronousTask" -> None, 
 "PostStartAsynchronousTask" -> None, 
 "PostStopAsynchronousTask" -> None, 
 "PreRemoveAsynchronousTask" -> None, 
 "PreStartAsynchronousTask" -> None, 

 "PreStopAsynchronousTask" -> None, "Repeating" -> False, 
 "TaskDetail" -> "", "UserData" -> None, "Visible" -> True} *)