HTML test

 This post is experimenting with the HTML syntax.

section

The quick brown fox. 

 

1 section

the

1.1 subsection

quick

2 section

Academic publishing in the humanities

The main reason why the Arxiv preprint server has accumulated around 2 Mio papers is because of the LaTeX typesetting program. Other preprint servers like Socarxiv which are preferring MS-Word as default format have a lower submission rate. It might be an open problem if LateX is producing a higher typographic quality than other publishing tools, but what can be measured is, that authors who are using this software are confident that their paper is written very well. This makes it likely that LaTeX papers are submitted to an online only preprint server, while MS-Word generated documents never exit the local hard drive. This strange situation can be described in detail. What happens after the pdflatex compiler was applied to a document is, that the document is ready. The pdflatex compiler checks for possible overfull hboxes, wrong style sheets and correct referenced literature and if he doesn't found a mistake the resulting pdf file is created int he working directory. According to the pdflatex program this file is ready to publish. it was formatted with the most advanced typesetting software available which is used by thousands of other researchers worldwide. So there is no reason left why this document should be revised or formatted something else. Technically it is possible to reformat the pdf paper because the pdflatex program has done all the work. All what the author is allowed to do is to print it out or better, to submit it to a preprint server. Basically spoken, a latex first pipeline makes it easy to publishing something. Authors are encouraged to type in a manuscript and then they are encouraged to submit the document to a webserver. This is why the latex software was created. In contrast, possible alternative word processing software are not forcing such a mandatory behavior. A simple outline tool or a literature database was programmed to support a local user during his work. And publishing the results is not recommended. It doesn't make much sense to print out a “ work in progress” manuscript. The reason is that such draft file has to be reformatted before it can be published somehow in the future – maybe. Let us try to investigate the hypothesis, that the LaTeX community is overconfident about the quality of the generated pdf files. Suppose we are asking in one of the latex forums, if latex is superior to MS-Word. The answer is: yes it is. And if we are asking if a latex generated pdf document is ready to print it out or ready for uploading it to arxiv, then the answer is the same. It is pretty hard to find someone who explains that latex generated papers have a low quality and it is not recommended to publish them because they need to be formatted somehow else.

Some applause for the Times new roman font

In the TeX community and in the Scribus community as well there is the shared understanding available that the default font Times new roman should be replaced by something which looks better. In the forum there are many individual recommendations available which alternative type1 font may look smoother, easier to read and more professional. A typical description is, that the Times font looks bad, but the recent discovered Pallatine / Utopia or Latin modern roman fonts looks so much better designed.

On the first look there is a clear difference between the default times font and other serif fonts. For most users, the times font looks to narrow and they are preferring a more wider font. The interesting situation is that subjective description has nothing to do with the Times font itself but width its current parameter. In the character setting of Libreoffice or any other word processor it is possible to change parameters like font size and very important the width scaling.

In the lower part of the screenshot, the times font was made wider while the latin modern font was reduced in the width and the surprising situation is, that all the fonts are looking equal. Not a certain font looks familiar but a certain scaling factor of this fonts. Basically spoken the Times font looks great always and if the user likes to get a new look he can make the font more wider.

 

Perhaps some detail information are helping to repeat this experiment. The default serif font in the postscript language is Times new roman. Apart from Times there are only Helvetica, and Courier available but both are not proportional serif fonts. With this restricted post script situation the user has no choice. He can either use the Times font or doesn't print the document at all.

If the user likes to replace Times with a different font there is no need to search for a new fontface, but the user has to adjust the scaling parameter in the postscript language. In the lower example, the same liberation font was used. The reason why it looks different is because the width was scaled up by 10%. That means all the characters are stretched by 10% and this make the font look more like Pallatino, Garamond or any other Times alternative.

Scribus for LaTeX experts

 

The assumption is, that the user is already familiar with the LaTeX software package, has used the texlive distribution many times and is able to create an academic paper which includes a table of contents, and a larger amount of included float figures. And now the user likes to know how to master the publishing workflow outside the TeX ecosystem and he is prefering especially the Scribus software.

The advantage of Scribus is, that it has much in common with the Adobe pagemaker software but is is available as open source. So let us start the journey.

First thing to mention is, that the workflow needs to be planned much better. If LaTeX and especially modern frontend likes Overleaf and Lyx have much in common with the python scripting language, the Scribus software is using a more linear workflow. That means at first the text is written in a word processor which is Libreoffice writer and then the document gets imported into Scribus. After the text was imported it is not possible to modify the text anymore so the content has to be froozen first.

Scribus works with the page concept. That means, the designer defines each page for it self. On page 1 it is possible to activate a twocoumn layout and on page 2 only a one column layout is visible. It is difficult to ochange a page later, so there is a need to plan the process. At first, the layout designer imagines a rough idea for the document. He makes decision about the amount the column and where the pictures should be placed. Then he is realizing the idea in the scribus software.

A common task is to create a new text frame or a new graphics frame. These functions are used so frequently that a shortcut is available. The grid system on each page helps to position a frame and then the content is send to this box.

Especially the import filter in Scribus is an early stage. In most cases the importet stylesheets have to be assigned again. This is done manually over context menu. The duration until a short document was created is long. That means, the layout process will take endless amount of hours, and then the designer will recognize that major components like the page number or the table of contents are missing. So these registers have to be created manual or half automatic.

The perhaps most obvious difference between latex and scribus has to do with changing an existing document. Suppose the well written report contains of 8 pages. The document was rendered once into the pdf format and it looks well formatted. Then, the author likes to add a single paragraph on page 4 somewhere in the middle of the text. Such a request is denied. The reason is, that after inserting a paragraph, the existing text will move to a different position and the position of the images won't match to the text anymore. Because of this reason, Scribus requires more pre planning than latex. The content needs to be written already, has to be proofread and it is not possible to change it drastically. The only thing what can be realized within scribus are smaller adjustments, for example adding a word into a section or adding a coma in a sentence.

If the author of a text likes to add an entire paragraph or likes to rewrite larger parts of the text a new project has to be started. That means, the text has to be written again in libreoffice and then it can be imported into a new scribus document.

In contrast, it is much easier to change the graphics. The reason is, that scribus creates in the document only a link and the graphics is stored in the working directory. After adapting the graphics, it gets rendered automatically in Scribus.

Perhaps it makes sense to explain which kind of workflow can be realized in scribus very well. The idea is, that that creation of the text is done by different users and at a different date in the calendar than formatting the layout. For example, in January person A and B are writing the text in libreoffice and in Febrauary this document gets imported by Person C and D into Scribus. So the final pdf file can be printed in March. In contrast the interactive single person workflow of LaTeX can't be realized with Scribus. The typical Latex project is done by a single person in a single month and if a mistake was found in the text it gets fixed on the fly. So we can say latex has much in common with the Python programming language and Scribus has much in common with the Cobol language printed on punch cards.

Text publishing in the Linux operating system

 The short summary is, that Lyx is the most advanced typesetting software. The details why this makes sense is given next.

The perhaps obvious attempt to create a text is a text editor, for example gedit. The problem with plain text editors is that they are not able to handle images and tables which are both needed for creating pdf documents. So the question is, how to create such complex documents? There is more than a single option for doing so:

- latex
- libreoffice, abiword
- HTML
- pandoc

All these options have advantages and disadvantages. Typeseeting is a complex domain and many books and papers were written about it. The reason why latex outperforms all the other software packages is not because of the software itself, but because of the communit yaround it. If someone enters in a search engine of choice a typographic related question he can be sure, that this question was discussed already in a tex related content. That means the tex community is the largest typographic oriented community in the internet.

In theory it is possible to format a document with abiword or HTML. From a technical perspective both programs are well suited for the job. The abiword software is a lightweight program which supports twocolumn mode, can generate pdf files and has access to all the build in fonts. Also the HTML language is well suited for created documents. The largest HTML standard supports hyphenation, justified text and of course PNG images. Or let me explain it a bit more direct. A bad formatted latex document and a well formatted abiword document are looking nearly the same.

In the latex community there is a single tool available which has a unique position: lyx. In contrast to all the other frontends like Scientific word, texmaker or emacs, lyx combines many features. It allows to create easily complex documents and the main strength is perhaps the ability to insert linked images plus the outline window. So lyx has much in common with an outline editor but it is producing TeX quality documents.

The famous typesetting system from knuth was mentioned already. The reason why this program is used frequently is because the ability to produce justified text. Perhaps it makes sense to explain this feature in detail. Information in the internet are usually displayed in a sans serif font with left justified paragraphs. This is the most efficient way to present text on low resolution monitors. Sans serif fonts are easier to read, and left justified text makes it easier for a browser to reformat the content if the size of the windows is changed.

In contrast printed text needs a different format. The common standard for printed documents is to use a serif font in combination with a fully justified paragraph. Creating fully justified paragrpahs is more complex because the change is high that longer white spaces are visible. To prevent these white gaps, TeX is using the Knuth Plass line wrapping algoirthm. Tex and Adobe Indesign are the only programs available which are able to do so.

MS-Word, Abiword, libreoffice writer and HTML are not using a global optimization technique and the result is, that the justified paragraphs has a lower quality. In theory there are some github repositories available with the aim to implement knuth plass line breaking algoirthm in the HTML rendering engine. But until now this isn't possible. So we can say that from an objective standpoint, tex is the  most advanced typesetting software available and has the largest community. So it is a here to stay if someone likes to create documents with a computer.

Having fun with the TeX typesetting system

The best way in figuring out what TeX is can be realized by deactivating all the advantages of TeX. The idea is to produce documents in the same quality like MS-Word is able to realize. From a technical perspective this is realized with the following commands in the preamble:

\usepackage{times}
\usepackage[none]{hyphenat}
\sloppy
\setlength{\parskip}{1pt}
\setlength{\topskip}{0pt}

These commands are activating the Times font, deactivating the hyphenation, switch off the inter word spacing optimization and deactivating the vertical space optimization. The resulting text will have lots of white gaps in the text and the paragraphs are positioned as they are on the page.

The interesting situation is, that such a setting is not recommended but the output looks ok. That means, graphics and text can be read easily and the only thing which is missing is the additional 1% of typographic quality provided by the TeX engine. If in addition a Times fonts was used the difference to an Abiword generated document is very low.

The mdtopdf tool

 There is an interesting upcoming github project available under the URL https://github.com/mandolyte/mdtopdf which has to do with converting a markdown file into the pdf format.

Somebody may argue that such a converter is already available and it is built in into the pandoc tool. But, pandoc is using the external latex program while the previously mentioned tool was written in go and doesn't need TeX at all. There is a reason why such tool wasn't available in the decades before, because the subject is difficult to understand.

On the first look a markdown to pdf converter has to do with converting format A into format B. But a closer look towards the problem will show that it more complicated. The markdown format is similar to HTML a markup language. It contains of text which is enriched with additional information. In contrast, a printer and the printer format PDF doesn't contain of text but it is action oriented. And this makes it harder to write a converter.

Basically spoken, the pdf format is not a text format similar to a .html but it is printer command language. It contains of actions like "drawtext()". So the converter has to read a plain text file as input and generates a script which is executed on the printer as output. So it is a text to program converter. And this makes it hard to write such a converter.

Let us give an example. In the markdown syntax a bold formatting is realized by "**hello**". This is a typical example for a markup syntax. The text author writes something and then the word should be printed. But, a printer doesn't work in this way. A printer doesn't need content but a command. What a printer can execute is a command like:

    setfont(bold)
    drawtext("hello")

That means, the original markdown file holds textual information, while the printer API is working with action commands. Or let me explain it the other way around. A markdown file can't be executed on a shell because it contains no commands, while a printer command language script doesn't holds information but it contains of statements.

Typesetting with a command language

 There are endless amount of software available which are promising to create high quality pdf documents. Some of them can be used in reality while other have become obsolete. The main problem is not how to program these tools but the bottleneck is to understand what word processing is in general.

A common assumption is that there are two different approaches available namely WYSIWYG programs and markup oriented tools. Software programs for each categories are widely known so there is no need to reference them again. The reason is that typesetting is used by a large amount of users and the PC is well suited for such a purpose.

First thing to do is to understand what the average user is doing with a word processor. In most cases the starting position is a zip file which contains of text files and image files. And the goal is to convert this zip file into a pdf document. The steps in between are realized with the typesetting program.

To understand what the task in between is, we have to focus on the technical needs. The step in between is realized with a printer command language. A printer command language is a programming language used to send commands to a plotting device. And this description is maybe a bit unusual. Because the starting text file is not a program and what the user likes to do is print out this text file. But a printer can't handle plain text files, what a printer is a command file.

In the past, lots of command languages were created. The concrete specification is not important at all. It is even possible to invent a command language for a display. Suppose there is a pixel map on the screen which has 700x300 pixels. A printer command language is used to draw on this screen. It is relatively easy to imagine what the commands are. In the easiest case the printer command language contains of commands like drawtext(x,y,text) and drawimage(x,y,image).

The output device is not a file, similar to a text file on the computer, but the output device is a computer, which accepts commands but not content. Word processing means to convert a zip file which contains of content into a command language which contains of actions. Let me give an example:

  # command language example
  title="hello world"
  drawtext(0,0,title)

  abstract="Lorem ipsum"
  drawtext(0,20,abstract)

  image=screenshot.png
  drawimage(0,50,image)

  paragraph="lorem ipsum"
  drawtext(0,0,paragraph)


This mini program has much in common with a python script but it is printing something to a screen. The interesting situation is such a file doesn't contain of normal text and it is not a markup language, but it is sequence of action words, formulated in a printer command language. Such a script can be executed on a printer or a graphic display. So the question is how to create such a command language script? One option is enter to commands manually, or to use a generator. This would result into a multi step process.

1. input: textfile, images
2. printer command language
3. output: pdf file

 

Generating a pdf paper with a command language

The interesting situation, that even the pdf format is known since years and around 100 different programs are available to create such a file format, it remains unclear what exactly word processing is about. The reason is, that the subject is mostly defined from a users perspective which makes it hard to grasp the underlying technical pipeline.

From a users perspective there are two sorts of programs available: WYSWYG programs and markup languages. Instead of arguing which of these tools are working better the idea is to describe in general what typesetting and printing is about. Typesetting means to convert content into commands. An example for content is a text file or a png image. And an example for a printer command is "drawtext()".

So what is the difference? It has to do with different understanding of the same subject. An author of a text things in categories of content. He writes a text or he draws an image. All these content is stored in files. A text file contains of information but it can't be executed. In contrast, an output device like a printer or a graphic display have a different perspective to the world. They are operating with commands. That means, a printer provides to the outside world a API (application programing interface). Printing something out means to translate between content into commands. This is inbetween step is realized by word processor.

After this introduction let us take a low how to print out something from a low level perspective. The idea is, that not the content stands in the focus but the needs of the output device. In the concrete case the output device is a pdf file which is created by the FPDF libray in the python language.

    import fpdf,os

    width=190
    pdf = fpdf.FPDF()
    pdf.add_page()

    title="lorem ipsum"
    pdf.set_font('Arial', '', 32)
    pdf.multi_cell(width,12,title,1,"C")
    pdf.ln()

    text="""Longer text
    newline
    lorem ipsum."""
    pdf.set_font('Arial', '', 12)
    pdf.multi_cell(width,6,text,1,"J")
    pdf.ln()

    pdf.output('1.pdf', 'F')
    os.system("evince 1.pdf")


After running this Python script it will create a pdf file and show it on the screen. The interesting situation is, that the python script contains of commands. That means, the printer doesn't understand a text file but it needs a commands to print a certain line. Suppose the idea is to print out a longer document which has 20 pages. This is realized by providing lots of commands after each other. At the beginning there is a command for printing the headline, then a command for the abstract, then a command for a picture and so on.

The problem is, that most authors are not trained to formulate a text in this way. The author doesn't like to program a printer, but he likes to print a text file. So the question is how to convert a ".txt" file into a ".py" script which contains of printer commands? This is indeed an interesting problem and it is handled by word processing software. A word processor takes content from the user and converts it into a command language.

In theory it is possible to convert the markdown format into a printer command language. Such a tool would work the following way. The text author writes something into th markdown format which is a plain text format. Then the converter is started. The converter takes the markdown content and generates a python script formulated in the printer command language. The python script contains the syntax of the fpdf library which is mostly the command multi_cell() and set_font(). These commands are send to the virtual printer and are generating the pdf file.

The challenge is to convert the markdown content into an action language. For example in the markdown syntax a new section is started with "## section". But this markup syntax contains only of content but it is not a command. A command which can be interpreted from a printer would look like:
   pdf.set_font('Arial', '', 12)

There is a typical communiation problem. The author of a text things in categories of content, while the output printer needs dedicated commands to operate. The interesting situation is, that with the correct commands the printer device can do anything. That means, it will output a single character or a longer text line. The only problem is how to convert content into action.