How to write longer technical documents

 

There is some sort of mystery available about how to create longer documents and especially academic books. On the first look the problem is easy to manage because there are word processing tools like MS-Word available plus dedicated publishing tools like Framemaker. The problem is that most authors are not interested in using a certain software but they want to know in general how to write a document.

The first thing which is important to know that writing a document has nothing to do with formatting it. Formatting a document can be realized by pressing the CTRL+A key to select the document and then select a font in the menu. So the formatting task is the easier one. The more serious question is how to create the text. This is done with a software tool called outline editor. Without an outline editor it is not possible to write longer technical documents.

Let us take a closer look how the existing programs are working. Software used for writing books are MS-Word, framemaker, libreoffice, latex, lyx and many other. What all these programs have built in is an outline editor. The interesting situation is, that this feature isn't mentioned in the handbook and most authors doesn't know what an outline editor is. But they are using this mode all the time. Let me give an example.

The famous program texniccenter is an IDE for the LaTeX program and according to the amount of documention, texniccenter is used to write papers in the reality. There is a single feature available in the program which is the ability to manage a pane on the left side. This pane can be used to use texniccenter in the outline mode. That means, the user can create notes or longer text and order them in a hierarchical fashion.

The interesting situation that other state of the art programs like MS-Word, lyx or emacs have s similar built in feature. Writing a text means basiically to type in the text into an outline editor. How this editor is called doesn't take matter. And it is also not very important if it is using LaTeX or not. The only thing which is simportant that some sort of outline editor is available. Let us take a look into a seldom mentioned software available as opensource: Cherrytree.

Cherrytree is a normal outline editor without additional formatting options. it can be seen as a tripped down version of Lyx. Lyx needs around 10 GB on the harddrive which includes the TeX back end, but cherrytree works fine with 10 MB space. According to the self description in the debian repository, “CherryTree is a hierarchical note taking application.” And yes the program is more than suited to created longer technical documents.

The interesting fact is, that Cherrytree doesn't belongs to the categories of word processing software. It is not a WYSIWYG software like Word, and it is not LaTeX clone for text based markup. At the same time, the program is the core component of a technical authoring software.

Writing documents for the Academia.edu website

 

The Academia.edu website provides lots of academic papers for downloading. But sometimes there is need to upload something. The reason is because it makes fun, because somebody can learn how to write a paper or because somebody likes to share it's knowledge. The only problem is that creating an academic paper from scratch is a bit hard. The following style guides explains the basics for newbie authors and somebody who likes to behave in such a way.

First thing which is important is, that a complicated two column layout is not the best option for the Academia.edu website. The reason is, that documents are displayed as default in the HTML mode and something with the rendered images is wrong, so the chance is high that precise formatted documents won't like great on the screen. The better idea is to use a minimalist layout which contains of a single column and a larger fontsize. Also it is recommended to deactivate any sort of kerning and fully justified text and use only normal wrapping mode from the document software. Such a layout can be realize with many ways for example, Word, markdown and LaTeX My personal favorite is the Lyx software but it depends on the user's experience.

The more serious problem is what to write in the document. Academia.edu is a scientiic website so it makes sense to choose on of the disciplines like mathematics, physics, literature or psychology as subject. Also it is recommended to reference to external sources which is a sign that the document was created for academic purposes. Writing the text itself is much harder to explain. A good workflow is to use existing content written in the past which was stored on the local harddrive and then convert it into an academic paper. If the user has not written any draft document in the past and the local harddrive doesn't contain self written notes or documents than it is not possible to write an academic paper.

 

After uploading a document to Academia.edu something interesting will happen next. In contrast to the expectation the amount of readers for academic documents is low and even zero. Some blog authors are asking themself why their traffic is so low. In Academia.edu there is no problem with traffic at all because the average traffic for a document is 0 views after 6 months. That means, the author will put a lot of effort into writing a document and then it is ignored by the world.

The problem is not located in the website itself but it has to do with academic content in general. That means academic papers in general are producing no traffic at all. Even Acadedmia.edu has an advanced tool for measuring the daily traffic the chance is high that the charts are showing nothing but zero rreadership in the world.

Understanding the line breaking algorithm in TeX

 

In the default setting, TeX is formatting a text with the fully justified mode. In contrast, the screenshot shows the opposite which is called left-justified text. The open question is how big is the difference? On the first look there are some white spaces at the right edge, but not very much. It seems that a normal line wrapping algorithm makes sure that in average the right side looks smooth. But let us take a closer look into the example.

The paragraph on the left are mostly correct formatted. That means, no visible space or only a smaller one is available at the right edge. In contrast the left page has some visible spaces. The cause was that the word was wrapped to the next line and no hyphenation or character kerning was applied. But this affects only 2-3 paragraphs and not all. So we have to ask what exactly is the advantage if the entire text gets formatted fully justified?

The interesting situation is that the situation will become more complicated if the text gets blured. If the reader will look fro the distance to the layout it is harder to see the advantage of fully justified text.

Is fully justified text really needed?

The LaTeX community claims indirectly that only fully justified text looks clean. The TeX engine tries to create such appearance really hard. Advanced hyphenation, spacing and even intra word spacing is used to make sure that each line has a straight edge on left and the right side as well.

The interesting situation is that the average reader doesn't care about this feature and a causual look at the document can't tell if it was formatted in fully justified mode or not. The example screenshot shows a document which was formatted left justified without any hyphenation. In spite of this absence of high quality formatting, the right edge looks reasonable well adjusted. Sure. It is not forming a perfect line but the document doesn't look very uncommon or even chaotic. The thesis is that the goal of creating fully justified text is maybe no longer important?

In the current example an unsharp fiilter was applied to the image so that it is no longer possible to recognize each words. Instead the reader will perceive the document from a distant view. The interesting situation is the right edge looks reasonable well formatted, even no formal fully-justified mode was applied. It seems that on the average a simple left-justified line wrapping algorithm results into a near right edge even without adjusting each line with inter word spacings. Is the TeX internal algorithm for word spacing an example for over engineering in a sense that in the reality nobody needs such formatted documents?

From a technical perspective, a left-justified document has the same appearance like a text rendered in a browser. That means, the spacing between the words is always the same and if the line is full, the word gets wrapped into the next line. This sounds a bit un-typographic but it results into an easy to read text.

Fully justified text in LaTeX documents

 

The LaTeX software is known as a high quality typesetting program. The reason why LaTeX generated documents are described as high quality is because they are looking the same like hand-typeset documents created 200 years ago. To understand LaTeX better we have to describe what classical hand typesetting is about.

Typessetting 200 years ago was at first the art of creating fully justified text. The interesting fact is, that this formatting style is only seldom described in the literature. MS-Word has a simple button to activate it and LaTeX is using this formatting style as default. So the user has no further explanation why this is needed.

Fully justified text means basically to adjust the words on a line so that the right and left edge are straight. In case of hand typesetting this art was very complicated to realize and the main reason why it takes so long until the metal characters are sorted on a page. The main trick for created adjusted text lines is to use glue between the words.

In the past, all the newspapers were typeset with this formatting style. What the manual typesetter were trained is to realize this unique shape. From a technical perspective it is possible to create flush left pages with manual typesetting as well. It would be even a bit more economical. But nobody has done so. A closer look into old newspapers will show that 100% of them were typesetted with fully justified text.

The idea of LaTeX is to imitate this formatting style. A standard LaTeX generated document will look like a newspaper which was manual typesetted in the past.

It is not very hard to guess what the opposite of fully justified text is. Flush left text is equal to missing typesetting. A flush left formatted text looks very different from what is used in newspapers in the past. The main reason why LaTeX generated texts are looking all the same is because of missing flush left formatting. The typical latex user assumes that it is prohibited to use a flush left formatting style.

Programming larger projects with Forth

 

The sad news is that it technical not possible to run existing c source code on a forth cpu. The reason is, that Forth cpus have no registers but only a minimal stack. And no c compiler in the world is able to generate the assembly instructions for such a machine. Even for mainstream CPUs like the 6510 it is hard to write c compiler because this CPU has also a low amount of registers. And in a forth cpu the situation is more dramatically.

So the only practical method to write software for the GA144 and other CPU is to hand code the software in Forth. Unfortunately, Forth is known as one of the hardest programming languages ever next to Dyalog APL. An easy way to learn the Forth language should be explained in the following blog post.

Instead of trying to understand the idea of a push down stack the more beginner friendly approach to write something in Forth is to use modular programming. Modular programming is a powerful software engineering method which is supported by most languages like Pascal, C, Python, C++ and of course Forth. The idea is that each file contains of 4-5 functions plus a handful of variables. The functions can access the variables and the module can be included by other modules. The concept is some sort of strip down version of a class.

Modular programming in Forth is a bit more complicated than doing the same in C. but it is not impossible in writing such programs. It is mostly a question of getting experience in writing stack based functions. The concept of modular programming is the same. It allows that the source code can grow to a size of 1000 lines of code and even more. And each single file has a limited size of not more than 100 lines of code. If the file is getting bigger, the programmer has to outsource some of the routines and creates some interfaces to communicate with other modules.

The interesting situation is that Forth based modular programming works well with existing Forth cpus. The only thing what a certain Forth system needs to support is the ability to execute a word, and also the ability to read and write to variables.

Understanding the 1990s in terms of AI

 

In case of normal compurer technology the 1990s and the year 2021 are not very different. In the 1990s wokstation computers were invented already which includes the Unix operating system. Video gaming consoles were widespread available and letter were created of course with a computer but not with a typewriter. So the only new thing today is, that the CRT monitores were replaced by flat screens and the computer is a bit faster. But this is not a revolution but only a normal minor improvements of the last 30 years.

In the context of Artificial Intelligence the situation is much more different. The situation in 1990s and the situation today is nearly the opposite. The people in the 1990s had only a rough understanding what AI is, and no practical demonstrations are available. Instead of describing what the situation today is let us take a closer look into the past.

The first thing to mention is, that in the 1990s the internet was in an early stage. It wasn't the dominant media which includes all the other media but existing media technology like the cinema, television, books and printed journals were used instead. What the people have learned about Artificial Intelligence were published in these media. That means, the understanding of robots and AI were influenced by movies and books about the subject. Robots played only a small role in all the existing media, and it was unclear if such technology can be realized soon. There was a big gap between robots in the movies and robots realized in reality. And this gap was projected into the future. Basically spoken AI was something not available in the 1990s. It was used as part of a science fiction plot in which robots take over the world or it was explained in a computer science book, that solving the np hard problem is not possible.

This public understanding is very different from today's situation. Today the average user of the internet will see lots of videos with working robots including the source code for the software to recreate them from scratch. This includes walking robots, self driving cars and Mario AI autonomous bots. Such a library of existing software which includes textual tutorials wasn't available in the 1990s. In the past, it was difficult to find an entire book about the subject Artificial intelligence and even the advanced one have described only a very rough situation. Another problem in the 1990s was, that modern programming languages like python wasn't invented. That means it was not possible for a newbie to create a line following robot because for doing so he had to learn languages like C++ first. Basically spoken, creating such a robot was in the 1990s a multi million dollar project for a research lab at the university but not an amateur project for a weekend to learn something.

Perhaps these examples have shown, that since the 1990s many things have changed. The dominant new situation is, that the former np hard problem seems to be solved. That means the amount of papers in which mathematicians are able to proof that AI can't be realized within the next 1000 years has reduced drastically. It was replaced by the understanding, that each month a new robot is presented which is a bit more powerful than the previous one.

What would happen if a person from the 1990s gets teleported into today's time? In case of normal computing the situation would be relaxed. He can use his former knowledge about how to use a mouse and a keyboard to control a computer. He will find the same MS-DOS prompt and the same Unix command line and sending e-mails is the same like 30 years ago. But in one subject the teleported individual gets confused strongly which is the current state of AI. After sitting down in front of a computer and watching a youtube playlist he won't believe what he will see. The shown examples in robotics are so much different than what he knows that the person wouldn't understand it anymore. It would be simply too much to see a biped robot, self driving cars, autonomous drones or the poor hitchbot who is waiting on a park bench at night.

The development in robotics wasn't an abrupt event but a continuous flow of changes. But at the end, the year 2021 and 1990 have nothing in common in terms of AI. There is a huge gap between both years and not a single revolution but many of them have occurred. Most of today's AI technology like biped robots, deeplearning, and game playing AI wasn't available in the 1990s. It has entered the world without any warning and there is no sign that the development will stop someday. It seems that the subject of AI was the single driver which has changed the world into a futuristic one.

Modular programming with any programming language

In the past it was some kind of game to compare different languages against each other. C programmers are convinced that their language is the fastest one, python programmers emphasizes how easy it is to write code and Forth programmers are proud of low energy consumption for a single instruction. It is hard or even impossible to bridge these communities.

On the other other hand there is a unique element which have all the programming languages in common. This feature is more powerful than stack based computing, and easier to create than object oriented programs. The single useful feature is the ability to write programs in a modular fashion. The term modular programming is a bit uncommon and sounds outdated so it makes sense to explain it further.

A module is a file which contains of 5-8 procuedures and 5-8 variables. The procedures are allowed to manipulate the variables and the idea has much in common with a class. A module was used in programming before the advent of oop. The pascal language knows the unit statement, the c language can include prototype files, and fortran porgrammer can create modules as well. So we can say that modular programming is the single paradigm which is available in all the languages including forth which has also modular capabilities.

Modular programming looks not very interesting on the first look but it is the key element to write larger program. A larger programs contains of 1000 and more lines of code. Such projects can only be realized in a modular fashion because it allows to structure the program into logical chunks which are programmed independent from each other.

The perhaps most interesting situation is that with modular programming all the languages including forth and C++ are easy to master. What the programmer has to do is to follow strict the rules. That means, if the program gets bigger he has to create a new file and put the procedures into this file. Programming means to manage procedures and variables. This paradigm allows to solve any problem in software.

Let us take a closer look into some larger Forth projects at github. Instead of explaining how a stack based language works let us focus only on modular programming. What we can see in these projects is that some files are there and each file has the same structure. At top there are some variables initialized and in the bottom some functions are written down which are accessing to the variables. So the concept is the same what c programmers are using and java programmers if they are creating new files for a new class. And yes the principle makes sense because it allows to write longer programs which has more features.

The interesting situation is, that modular programming has no limit in the code size. Because newly created submodules can be included in other modules and at the end there are 50 and more files available which have each 100 lines of code. So the overall structure is highly hierarchical. It is less powerful than real object oriented programming but it comes close to the idea.

It sounds a bit trivial to expain this programming style in detail because c programmers are doing so since the 1970s and it is the most common coding style ever. On the other hand the amount of books how are describing the principle is low. Most C introductionary books doesn't even have a chapter about the subject of creating programs with more than a single sourcecode file. Similar to other languages the main focus of a tutorial is to explain the language itself, which includes the statements, but this knowledge doesn't allow to create useful software.

So we can say that an individual programming language is less important. The fact that the for loop in forth works different from a for loop in python can be ignored. What the programmer needs to know instead is how to write programs distributed in units.