Showing posts with label Database. Show all posts
Showing posts with label Database. Show all posts

May 23, 2026

Lessons learned from Douglas Lenat's Cyc

 During the late 1980s the Cyc project was a large scale AI project. The promise was to create a database with handcrafted Lisp rules which is able to reason about the world. The attempt has failed but that is no problem because it its possible to analyze the reason why.

From today's perspective Cyc was an early attempt to create a dataset. A dataset is a .csv file but doesn't contain of computer code. Datasets are storing numbers and text. During the 1980s it was unknown how to create large scale datasets and Cyc had some builtin mistakes:

a) there was no word2vec algorithm which allows to convert the textual information into numerical representation
b) Cyc was encoded with rules but not with question answer pairs

A modern dataset which is superior over cyc would solve these mistakes. A common dataset used for training neural networks contains of a simple Q&A structure like "What is the capital of france? -- Paris". and it would use a word embeddings algorithm to project the information into a numerical space which can be parsed by neural networks.

The Cyc knowledge base was a combination of Lisp software and textual information. It was a hybrid of computer code and a dataset. Such kind of knowledge base was replaced by data only datasets which have become popular since the deep learning boom. In a data only dataset there is no computer code but only data itself which can be text or images. The computer code which is searching in the data is externalized in a deep learning library.

March 07, 2020

Small introduction into the Kexi software

The amount of tutorials about the Kexi-project is very low. The official forum has around 200 postings, and even youtube provides less than 5 videos about the topic. At the same time, there is a need for an Open Source desktop database, and Kexi is one of the most promising examples in that direction. The sad news for the beginning is, that the project is in early stadium. It can be called an alpha version which doesn't provide anything. Even MS-Access 2.0 has more features to offer and this program was realized 25 years ago.

But it makes no sense to be too critical. Because all the advanced existing database applications are close source projects and so called web databases on top of PHP and Ajax are very complicated to use and can't replace a desktop database. So i'd like to introduce Kexi to the newbies with some screenshots and explain what the developers have programmed so far.

The program is available in Fedora and Debian as well. According to the official description its an “integrated database environment for the Calligra Suite” and an “visual database applications builder” [1] The program has only a small size. The kexi sourcecode consists of 150k lines of code which is around 4 MB for the program.



The main menu looks exactly like a small project. The user is asked to create tables, forms and reports. The menu bar on top of the screen have no purpose, all the features have to do with tables, forms and reports. At first the user can create a new table. I have done so and the table stores the firstname and the last name of fictional students. Additional a primary key was defined which is incremented in the auto mode.



In the next screen the form editor is shown. Which is a minimalist one. There are some widgets available which can be dropped with the mouse into the form The most important one is the text field which holds the data of the underlying table. The data source can be provided right the properties menu. In contrast to the Gambas software (which is a visual basic clone) the connection between the database table and the form is working great. The user is able to see the data in the form.



The last item in the list is the report generator. This module has less features than the minimalist form generator but the user is able to create simple reports. He can drag and drop text fields, specify the underlying data source and with the printing driver of the Linux operating system it's possible to create a pdf document.



More features are not available. The official Kexi handbook explains mostly what is missing in the project and indeed the software is in an early stage. But compared to other software open source projects, Kexi is my personal favorite. It has to two main advantages. First one is, that the project is going into the right direction. The project goal is to program in C++ and integrated database software which contains of tables, forms, reports and scripting features. The second advantage is, that the code written so far is stable and the user can create a small but working database.

Let us make a simple thought experiment. Suppose enough manpower is provided for this project, and the amount of codelines grows by the factor 10 from today 150k to 1.5 million. Additionally some example .kexi files are provided and the forum gets more traffic. The prediction is, that Kexi will become one of the most interesting Open Source projects since decades. The reason is, that the normal user can do a lot of things with a database RAD tool. The options are endless. If such a software is programmed as open source with open standards in mind it can change the software industry.

I would guess that the next milestone for the project is to bring kexi on the same level like the outdated MS-access 2.0 software. I would guess that today kexi provides around 20% of the features from Access 2.0. Most of the interesting features are missing. But from a technical perspective it's possible to improve the software.

Importing CSV

Under the tab “external data” the user can import existing csv data. I have tested out the feature with an example csv file. [2] It works great. The csv file is converted into a kexi table.

disadvantages

Some major problems are visible with the software. The first one is, that after clicking with the right mouse button on a table and select “Export table as csv” the kexi program crashes without warning. It's not possible to export the table at all. This makes it hard to use the data for external purposes. The second problem is, that kexi has some preinstalled plugins which includes an sqlite driver, but it's not possible to connect to a sql lite database.

It's not clear what exactly the reason is why the important features are not working, but suppose the connection to a sqlite database works, and suppose the export filter gets improved, then the normal user would be able to use Kexi for some smaller databases. It has some advantages over LibreOffice calc to create a dedicated database, and create queries with the SQL language. As far as i can see the sourcecode is available so it would be possible either to contribute to the existing kexi project or fork the project and start a clone.

sources

[1] Debian kexi package, https://packages.debian.org/search?keywords=kexi

[2] CSV sample data, https://people.sc.fsu.edu/~jburkardt/data/csv/csv.html

Building a minimalist database from scratch

From a historic point of view, many database management system were developed over the years: dbase was used in the 1980s, MS-Access in the early 1990s, since the 2000s MS-Access used in combination with SQL server and since 2010 the situation has become very complex, because many companies are experimenting with Java, Linux, C# and PHP.

The idea is to throw away everything and reinvent a database from scratch with Open Source software. A good starting point is the sqlite software which is available out of the box in all Linux distributions. The problem is, that sqlite is only a small part of an overall database management system. To make things more comfortable a middle layer and a frontend is needed:

backend sqlite -> middleware python -> frontend python

Programming a python frontend is not very complicated. It has to do with drawing windows on the screen the wxwidget library and add some animations plus sounds to make the game more pleasant. The more complicated part is the middleware. To introduce the business logic layer we have to analyze who to interact with a normal sqlite database.

Suppose the user has started the python3 interpreter and is connected to the sqlite database. What the user can do is to submit an SQL statement. He has to type in the SQL request into the command line and gets the feedback from the database. A naive assumption is that the interaction can be improved with the help of a GUI frontend. On the longrun not a GUI frontend is needed but a middlelayer. A business layer is working on the textual layer and allows the user to enter high level textual commands. Instead of typing in:

SELECT * FROM Customers WHERE First_Name='John'

The user enters the command:

middleware.showcustomer(“john”)

The mdidleware program code converts the high level statement into low level SQL commands which are submitted to the sqlite database. The interesting point is, that no GUI interface is needed and the user can interact with the database very comfortable.

REST middleware is the future of databases

The common database application in the 1990s was based on the backend frontend philosophy. The SQL server from Microsoft was used as a backend and the MS-access database is realizing the frontend. The disadvantage is, that most of the logic is located in the frontend. The answer to the problem is to introduce a new layer called middleware.

Microsoft has invented the .net C# platform which is directed towards this goal. But the term middleware can be realized with other programming languages as well. A famous one is Javascript which can be used for creating RESTful middleware api. To understand how this is realized in reality, we have to take a look at outdated MS-Access applications.

The idea is, that the user is pressing graphical buttons and this activates tasks in the software. THe better alternative is to provide a textual interface. Bascially spoken, the programmer has the obligation to convert an existing MS-access application into a text-only interface. This text-only interface is the REST API. Programming a graphical interface on top of an existing REST API is very easy.

The REST specification has become popular in web-oriented development. But is Javascript needed for classical desktop databases? Suppose the tables are stored in a sqlite database. The SQLlite database can't be called a middleware, but it's equal to the backend. Also the SQL language is located on the backend. What is needed an additional layer ontop of SQLlite. This middleware layer can be realized with python. The python software doesn't display the GUI interface for the user, but it connects an existing GUI with the SQLlite database.

sqlite -> python middleware -> python GUI

In the context of REST, the term business logic layer describes the middleware. Let us investigate how REST is working internally. REST means, that the methods of a Javascript class are mapped to external function calls. It has much in common with a programming library API which is realized with the HTTP interface. Suppose the idea is to program a desktop database but not a web-database. What is needed here is a Python library.

The python library contains of 100 lines of code, which has one class and can be imported from another python program. The python middlelayer connects two layers:

sqlite -> python middleware -> python frontend

The communication between python program1 and program2 is realized with the normal function calls. In the python frontend the functions of the middleware are called with:

import middleware

middleware.updatedate()

March 06, 2020

Linux can replace MS-Access

The main reason why Linux struggles on the desktop and especially on the desktop in the business world is because Linux developers didn't listen carefully what business users are trying to do with a computer. The most important software used on PC in a company isn't the LaTeX software, it's not Python and it's not the webbrowser. All of these programs are running great in Linux but the average business user has no need for such software. What the user need instead is MS-Access, MS-Access and nothing else but MS-Access.

The first step to bring Linux on the desktop of companies is to abstract a bit from the user needs and describe in detail what the mentioned software from Microsoft is doing and why it is so important. The main issue why computers are used in the business world is to store and retrieve table-related data, for example inventory lists, product lists, customer lists, accounting tables and so on. 90% of the IT infrastructure in the business world has to do with database management, and GUIs for database management software. The reason why the MS-Access software has become so popular is because it consists of two things:

- a database engine known as the Jet Engine
- a Rapid Application Development software in which the normal user can create forms, reports, views and macros

The combination of both fulfills the needs in the business reality great. The next step is to ask what the Open Source replacement will look like. The good news is, that the technology is available but it's not documented well enough. The database engine can be realized with sqlite and the Rapid application development software can be found under the name Gambas basic.

Let us describe the details of an open source MS-access clone. A single sqlite file stores the data of many database tables. The information is created and updated with sql statements. Sqlite is a very powerful software which is used for anything. The main problem is, that on the command line it's hard to interact with the tool. To make the life easier a rapid application development software is needed. This is a program which allows to create multi document interface applications. That is a single application in which different windows are opened at the same time. For example three tables, plus 2 forms. The Gambas IDE https://en.wikipedia.org/wiki/Gambas was initiated in 1999 and provides such graphical environment. Very similar to MS-Access, Foxpro and other database frontends the user is allowed to create forms and reports. And all the windows are connected to an underlying Sqlite database. Some examples projects have shown, that in theory the combination of Sqlite plus gambas is able to replace MS-Access databases in a business context.

To understand why desktop databases are used everywhere in a business context it's important to analyze the workflow of creating new software in a business context. In context to classical programming project a business software is never written in C++ or in Python. It would take too long to program 20 forms in this language. And normal text oriented programming languages are not flexible enough to change an existing layout. What business users are prefering instead are non-programming options to create applications which has to do with MS-Excel sheets and Visual basic layout software.

These non-classical programming system are ignored by classical programmers. It has nothing to do with programming language development but it's called a rapid application development system. A rad tool is some kind of painting software for creating lots of windows, and in the second step the window buttons are connected with macros. The concept was realized first in the MS-Access software, but many other MS-Windows programs are available for this need.

From a technical point of view it's possible to realize a rad tool in Linux with Open Source as well. It's only a question how many manpower is provided to create such software. The Gambas tool is on the same level like early versions of MS-Access it allows to create smaller business applications and for most purposes this fulfills the needs of the user.

What is missing are documentation and practical examples in which gambas is used in the real world. The amount of literature in which MS-windows based databases were created in the past is much higher. Linux in general and RAD tools for business application in detail are a very new development. In theory this will increase the market share of Linux massive. The reason is, that apart from creating GUI-database, the average business user has no further needs. If a software or an operating system fulfills this single task, the user is happy. He will use the application all the time, and he will recommend it to other.

Dive into Gambas3

After installing the software package the user can create forms. The software has much in common with visual basic. After doubleclicking on a button it's possible to enter basic sourcecode which opens new forms. Creating a complex GUI with different windows and tabs is pretty easy with gambas and some youtube tutorials are explaining the details.

The bottleneck is to establish a database connection. In theory the user has to create a connection first, then a datasource widget is created in a form and then a databrowser will display the content of the sqlite database in the form. The problem is, that it doesn't work. THere is an error message: connection can't be established and the help section has no answer to it.

It seems that Gambas is in an early development status. But the assumption is, that the databse connectivity will work in future versions and apart from Gambas there are similar projects started. One of them Camelot RAD which is a RAD tool around the python language. The idea is similar to Gambas because the user has to create forms and puts some glue code into the application.



sqlite and RAD tools

Let us describe the picture in general. What is available in Linux is the database backend. The user can create a single file in the sqlite format which holds all the tables. sqlite meets without problems every requirement. What is missing is a rad tool for developing forms and reports ontop of an existing sqlite database. Existing projects like gambas, Openoffice base and Camelot RAD are in an early development stage. They are working in theory but it's to early to recommend them as alternative to MS-access.

Suppose the team behind the gambas project is improving the database connection. Then this tool will be used very often in a business context. What the user can do today is to create GUI prototype application.

From an abstract point of view, the key requirement for business application is a RAD tool which allows to create forms, reports which are connected to an underlying sqlite database. In the context of web development the missing RAD tool isn't a problem, because with PHP it's possible to output HTML code. But if somebody likes to create a desktop application he needs such a tool. Programming all the forms in Python with the tkinter framework won't work. It will take too much time. The prediction is, that in 5 years from today a fully working RAD tool is available as Open Source which is using a sqlite database as backend. Such a tool will motivate many million of users to switch to the Linux operating system as their main desktop environment. A desktop database is the missing piece of software to make Open Source a success.

March 05, 2020

Making LInux compatible with MS-Access

The main reason why Linux has struggled on the desktop PC of companies is because of a single program, which is MS-Access. This single program has top priority for companies but it has a low priority with the Open Source ecosystem. A naive approach to adress the issue is to recommend to replace existing MS-access database with modern backend databases. Under Linux operating systems it can be realized with mysql for the server and PHP as a frontend programming language.

Unfurtunately, this advice works not very good in reality. Converting existing MS-access database into PHP scripts which are displayed in the browser is equal to start a large scale software project. It will take million of manhours and will costs million of US$. No company in the world is rich enough in doing so. And what the users are doing instead is not to use Linux in general and stay on the well known MS-Windows plus MS-Access software.

Suppose a company doesn't has existing .mdb files on the harddrive but they need the computer only for browsing in the internet, write some python scripts and create LibreOffice textfiles. Then it's pretty easy to switch from the Windows 10 operating system to a modern Debian Linux system. The problem is, that in reality, the average company has a need for a desktop database.

There is an Open Source software available which can maybe adress the problem. It's called mdbtool and was started 15 years ago. MDBtools is a software for reading existing Microsoft Access databases. https://github.com/brianb/mdbtools/tree/master/doc The format is outdated because the current format is called ACCDB and mdbtool can't write in the format. But the tool goes into the right direction. Suppose the software is improved a bit. The idea is not to create a database with Linux, but the idea is, that the real need is to read and write the MDB fileformat. In theory, this will help to bring LInux to the desktop.

Let us go a step back and analyze why the LibreOffice suite is a good replacement for existing Microsoft Office software. Because the tool is able to open and write in the same file format. If the user has any Word document he can open the file easily with Libreoffice. For the MS-Access software such a converter isn't available. Instead the normal Linux user is asked to write and read from different file formats like json, csv files and mysql databases. But these formats are different from the existing one. The MS-access fileformat is too important for the desktop PC to ignore it.