September 02, 2019

Connecting programming languages with sockets

How exactly gets Python programmer access to existing C++ libraries? On the first look the question is easy to answer because Python can import C libraries easily. A more detailed look into the problem will show, that the existing wrapper and interface generators like SWIG and ctypes are not enough. And even in the case of success, they only allow to Python programmers to get access to a c library, but they do not provide a universal language independent format.

Every new programming language which was invented has the same problem. Inside the ecosystem all the features are working great, the only problem is to convince other programmers to use the same language. What programming languages are not designed for is to communicate with the outside world. Even Python is not prepared to include existing c libraries or Java programs.

A more recent technique for a communication standard is not working on the sourcecode level and is not part of a language compiler, but has to do with TCP/IP connection. An early attempt in connecting different programming languages was SOAP, a more recent development is RESTful and sockets.

Let us take a look into the gaming library SFML what they know about sockets, https://www.sfml-dev.org/tutorials/2.5/network-socket.php According to the documentation a socket is network connection which allows different applications to send and receive data. The interesting point is, that first the socket works with different languages and secondly, the code is running while he communicates. The SFML library was written in C++ but if a SFML application has opened up a socket, other languages like Java, Python and Javascript can get access to the connection.

In contrast to a SWIG like language interface which is working on a lower level, TCP/IP sockets have a poor performance. According to some tests, it's possible to get around 300 requests per seconds on the localhost interface. With some newer techniques like non-blocking RESTful interfaces the traffic will increase a bit.

The advantage of sockets over import a c library is, that the amount of needed explanation are shorter. Suppose, programmer A has written a game engine which allows other programmer to write plugins. In the past, programmer A has to document his sourcecode, because the plugin interface is working on the language level. For example, the game engine was written in C++, so the other programmer need the API and the documented C++ sourcecode. This allows them to write a plugin which can utilize the existing game engine.

The disadvantage is, that the number of programmers how are interesting in analyzing existing sourcecode, especially if it was written in C++ is low. The problem is not located in the game engine itself, because C++ is the perfect choice for creating such a project. It is ultrafast and most game engines are written in C++. The problem is, that it's very complicated to analyze an existing project. The more easier way is, if programmer B can use his language of choice. For example he is trying out the C# language, because he likes the syntax very much. And this opens the question how to connect a C# frontend with a C++ backend?

The sad news is, that C# doesn't provide access to existing C++ code. Sure, Stackoverflow is answering the problem with the reference to the COM interface https://stackoverflow.com/questions/3029031/connecting-c-to-c-sharp and the DllImport library, https://stackoverflow.com/questions/2958416/call-c-library-from-c-sharp But none of these techniques are making sense. And what's more important, they fail if the languages are slightly different.

On the first look this sounds paradox, because C++ and C# are both mainstream languages which a huge userbase and lots of energy which was invested in the improvement of the compilers. So the naive impression is, that there is a technique to connect a normal C# frontend with a C++ backend. But it seems, that the problem is too complicated even for the C# community. A closer look into modern programming languages, for example Java, Python, C and Javascript will show, that no one is mastering the issue. All of the languages are struggling if the user tries to connect different languages.