How complicated is the workflow to create a social network from scratch?

All the major social networks have something in common. Facebook groups, Reddit groups and Google+ groups are working with the same philosophy in mind. The groups containing something which is described as news aggregation. But how exactly is the groups game working?

Let us simplify the world a bit and suppose that in a intranet the only website which is available is a mediawiki installation. Mediawiki has the advantage that the software is open source and all activities in the wiki are tracked in realtime. It's easier to watch inside the box to describe what news aggregation is about. We need apart from the wiki itself, some motivated people who are creating new articles in the wiki and modifying existing one. Around 10 users are doing so and the wiki growths.

Content aggregation means to observe the tab with the last changes. One category of changes is more important it is the newly created articles. An example logfile could look the following:

2019-05-01 N linefollowingrobot, URL

2019-05-01 N programmingexample, URL

2019-05-03 N profilpage of user3 was created, URL

2019-05-04 N neural networks tutorial in python, URL

The N stands for new page and is used in most wikis to tag the edits. Each added article has a creation date, a title and a URL. News aggregation means to monitor, comment and repost this list. That means a news wiki page is created called “news of the day” and if in the logfile something relevant happens the URL is copied to the news page. The “news of the day” section has to look the following way:

2019-05-01 linefollowingrobot, URL, boring article

2019-05-01 programmingexample, URL, sounds nice, comments needed

2019-05-03 profilpage of user3 was created, URL, hello user3

It is the same list, with some minor modification. The last item was forget to mention, because the aim is to get the news list as short as possible.

With this background knowledge it is possible to explain what content aggregation means and how complicated the process is for doing so. The amount of work which has to be invest to convert the existing logfile into a news section is very small. If the editor is lazzy he copies all the entries and adds some context information. This can be done in under 1 minute.

Social networks like Facebook, Reddit and Google+ are doing the same thing. They are creating a news section of the entire internet which contains of an URL and a title. The idea is, that the user will read the news and visit the url. How many users are doing so in reality is unclear, what we can say for sure is, that creating a news section for a wiki is an easy task. A single person can do so for a large Mediawiki installation with investing less than 1 minute work. The more demanding task is to create all the content which is done by the 100 users. For example if somebody want's to write the linefollowingrobot wiki page he has to invest many hours before the text was written.

Redesign of this blog

I've discovered that in the blogspot menu it's possible to adjust the navi-bar on the right side. The first thing to do is to add a gadget for showing the traffic counter. And like all traffic counter the numbers are fake. The correct amount of pageviews right now is not 887 but it is zero. That means no one has visited the blog so far. The reason why is simple. Because Google ranked my blog at the last place, no comments were made in the blog, the content is not attractive for a mainstream audience and so on.

But i think it's funny to see the current pageview counter on a prominent position, it helps to get a realistic impression of what the difference is between a great blog read by many and a newbie blog read by nobody.

Building a social network website from scratch

Many people are fascinated by social network websites like Facebook, Twitter and Reddit and they want to understand how to clone such a website. If somebody is able to clone a product or a website he has understood the inner working. Cloning is some kind of reverse engineering with the aim to open the black box and analyze what is hidden inside.

The first step in building a social network from scratch is the technical side. What is needed here is a software like Elgg or Humhub which are open source projects with the aim to build a social network in the intranet. That means, the users are not dependent from large scale websites like Facebook but they are creating their own space hosted on the own server.

But is the Elgg software similar to a social network? After installing the program on a server the system is technically working fine but something is missing which are users and traffic. From an abstract perspective a social networking site allows to post comments similar to a forum website. But what exactly is the difference between a forum software like phpbb a blogging software like Wordpress and a social networking software like Elgg? From a technical perspective all of them are working similar. In the backend there is a SQL database and in the frontend is a PHP script which generates the HTML form shown on the screen. The user types in something and the information is stored in the database.

This allows to reduce a social network tool to it's core feature. There is no need to install a software like Elgg, but a simple python script which has access to a database can do the same job. It seems, that social networks are not based on a certain scripting technology, but it's about the user's behavior.

Suppose it's possible in an intranet to make the own rules how the users should act. What are the soft rules everybody has to follow to build a social network? I would guess this is the more important recipe. To moderate the users into a certain direction. The good news is, that in an intranet the technical admin is able to ban users if they are ignoring the rules. That means, the admin is in the comfortable position that his rules have to be fulfilled. The follow up question is which kind of rules are needed for a social network? If the admin doesn't know the project won't work successful.

I have researched the topic a bit. The common social rules which are equal to Twitter, Facebook and Reddit is, that these websites understand themself as a blog aggregator. This self-description allows to formulate the rules for the Reddit clone:

rule 1: it is forbidden to post longer content. That means, if somebody tries to upload a 1 MB pdf file or is posting a 100 kb textfile to the social network he gets banned.

rule 2. What the users are allowed to post are URLs, plus a short description whats behind the URL.

rule 3: the posted URL should fit to the section for example Artificial Intelligence, and the URL should be fresh. That means it is referencing to content created one month ago.

rule 4: if somebody posts URLs which are not fitting to the domain or the URL is outdated the user gets banned.

These simple 4 rules are able to replicate a social network in the intranet. If the admin checks if the rules are respected by the users, the newly created social network will look similar to the large websites (Facebook, Reddit, Twitter). Perhaps the intranet social network will not be exact the same, because the number of users is smaller and because the real reddit website is more attractive, but the general idea is the same.

Sometimes, social networks are described as social tagging websites in which the users can post links. This description is not sufficient. Because a social network doesn't has an internal structure, but what the users are doing in the network depends on the content located outside the network. Before somebody can post a URL he has to know the URL first. And before somebody finds a URL in the Internet, somebody else has to put content behind the URL. The better description of what social networks are about is a blog aggregator. It is some kind of overview websites which is monitoring the blogs which are updated recently.

Detail features like upvoting, downvoting and commenting links from other are additional feature which make the social network more interesting. Also the feature to preview the content in a smaller window and a strong focus on entertainment / funny subject will result into an improved social networking site.

What we can say for sure is, that a social network contains URL to external content which is posted by a group of people and sorted into domains like “fun, games, party pictures and so on”. IN case of Facebook the situation is not so strict, because the user can upload images directly to facebook, so it has the features of an image hosting website, while at twitter the postings are not organized in groups but around people who have written a message. What the smalles common definition is, that social networks are monitoring the blogosphere. It is some kind of realtime search engine created by humans. Usually, the amount of interaction on these websites is higher than in a normal forum. A forum collects information posted by the users. A forum is not monitoring external content.

In the academic community social networks are called an overview paper. An overview paper doesn't present new information about a topic, but it is reviewing exsting content which is already there. The typical overview paper contains of 300 references, but sometimes up to 1000 references. In contrast to a social network the overview paper can't commented in realtime. The shared similarity is, that social networks and overview papers are trying to stand on top of the community. They are in a higher position and make jokes and recommendations about existing information. In most cases, overview papers are written by experts of an domain who has read all the information and is able to give the context information which makes it easier for newbies to identify relevant information.

The interesting point is, that even in the age of search engines, overview papers remain important. They will answer the question which topic is interesting and why. In contrast, a search engine makes only sense if the user is an expert already and knows the keywords to enter.

Cloning Reddit

In the introduction it was mentioned, that before a website can be cloned it has to be understood. Suppose the idea is to create a Reddit like website in the own intranet which is generating a lot of traffic. The rules for doing so are:

1. the users should post URLs plus small descriptions

2. the links should be fresh and referencing to content created less than a month ago

3. the only allowed topic is “funny”, “jokes” and “computergames”

4. many users can login into the site, discuss and upvotes the postings of other

If all these rules are respected strictly the intranet website will develop into the same direction like the Reddit example. It will become a high traffic hub. The reason why is of the mixture of some features. At first, it is not very complicated to post a URL into a website. It can be done in under 2 minutes. This motivates the users to do it very often. Secondly, the topic is jokes, fun and computergames which is attractive to 100% of the users. and third the users can post without asking before so they are in control of the website.

All these features combined makes it likeley, that the website is perceived as useful and will attract new users to participate. The success can be increased if a bit marketing is done for the project, if the technical infrastructure runs stable and if some starting posts are available before the first users are invited to become active.

Now it is possible to describe some counter rules which will result into lower traffic on the social network. Counter rules are:

1. funny topics and jokes are forbidden, computergames too. Only hard scientific topics and programming URLs are welcome

2. the need for posting fresh url is reversed. Instead it is only allowed to post URLs older than 1 year.

Both rules combined make the website less attractive for the community. Perhaps some of the users will participate in the project too, because they have no need to talk about funny thinks or jokes. And perhaps they find the idea interesting that the URL should be not fresh. But in general the amount of users will become slower. The result will look different from reddit. It will become a low frequency social network which is missing something.

The difference between a frontpage wiki and URL spam

A frontpage wiki is referencing to external blog content hosted somewhere else, similar to what Reddit is doing. The term URL spam is referencing to a behaviors of persons who are editing wiki pages with the aim to place advertaisment links on that wiki. The funny point is, that both are the same. Every frontpage wiki will look like linkspam. But without a frontpage wiki it's not possible to connect the community and to aggregate existing content.

The answer to the problem is to be aware that an aggregator wiki similar to Reddit will look as default like spam. It is only a question of the judgement if this perceived as a problem or as helpful. Let us analyze a neutral website like Planet gnome http://planet.gnome.org/ What is seen on the site is a collection of URLs which are referencing to wordpress, blogspot, google and gnome.org content. Planet gnome is basically an advertaisment website, founded with the aim of linkbuilding and traffic increasement of the referenced websites. Trying to explain to the newbies why planet gnome is different from URL spam / webring is not possible. It fulfills exactly this definition. The reason why the links doesn'T get deleted by the admin of the project is because the advertaisment feature is wanted. That means planet gnome was created exactly for the purpose of linkbuilding and rank manipulation in Google. The funny thing is, that at the same time, no ads are visible at the website. The user gets a clean interface combined with valuable content. That means, the website has reached the maximum amount of spam which is possible.

Sometimes the Open Source community is described as the opposite of modern capitalism. It's exactly the opposite. Linux and the ecosystem around it, sees advertaisment as their core ideology. Each blogger who is writing something about Gnome is trying to make advertaisment. He will increase the market share of Linux and the aim is to redirect the readers to other websites in which they can download the latest iso file from Red Hat or Ubuntu. The LInux ecosystem is overpopulated with ads. The idea of selling something is everywhere. The only exception to the normal world is, that the desired actions are not connected to money. The user can read the linked tutorial for free, the user can download the latest Fedora software without paying anything, and the updated podcast about a new Linux program is provided without charge in the Ogg theora format.

How to build a news site in the internet

Sometimes, Reddit was called similar to Facebook an overhyped website which is populated by bots. But what is an alternative? Let us analyze the idea of Reddit, Hackernews and Facebook groups to understand how to create such websites from scratch.

The internet contains of two elements, the first one is the World wide web which contains of URLs. If the user types in the URL into the browser he sees the content. The second important element of the Internet is a fulltext search engine, namely Google which is able to provide URLs. If a user types in a keyword in Google he gets a list of clickable links and this directs him to the content.

The problem with this kind of knowledgedatabase is, that the user must know the right keyword to find the URL. He can only see a piece of information, if he knows for what is searching for. What most users are prefering is a browsable list of URL which are ordered similar to the headline of a newspaper. Websites like Hackernews and Reddit are providing such list.

The ticker contains of at minimum of a headline plus an underlying URL. And the ticker is updated constantly, if new content was published in the internet. From a technical perspective such a ticker is monitoring the activities of the Google Crawler bot. The Google robot is searching for new websites and updated websites and puts them into a fulltext database. Each second many thousands websites gets updates. A short part of these updates is redirected to the Reddit ticker.

Such a minimal web aggregator can be improved with upvotes, user comments, preview pictures and categories. This is what all the large new aggregation websites are doing. An enriched new aggregator is more interesting for the users because it allows them to get the best content faster.

Let us imagine we want to build a news aggregator and a search engine from scratch so we can ignore the existing infrastructure of Google and Reddit. Taking both authorities as a work hypothesis offline allows to see the filter problem by itself. The assumption is, that in the internet are million of websites. These websites can be blogposts, forums, homepages, podcasts or sourcecode repositories. The websites do not linking to each other because they are not aware of their neighbors. A very large problem in this knowledge collection is to find the information which is the best one for a user. From a technical side the first thing which is needed is a fulltext search engine. It contains of a crawler bot who is indexing the fulltext and a database which allows a search in the content.

Such a search engine is only the first step for getting access to the information. What is missing is a ticker. A ticker works without entering a search term, similar to the table of contents of a magazine. It presents all the information available. The ticker is equal to the menu in a restaurant the user gets an impression what is available even if he has no idea what he likes most.

Creating a live ticker can be made in the easiest form with an automatic algorithm. The ticker presents all the updated websites in a linear list and the user can scroll from top to bottom. The disadvantage is, that even in a small internet, the amount of recent changes is large. SImilar to what WIkipedia presents under the tab recent changes. The better idea is to organize the ticker into categories like sport, funny, cars and to allow annotations to decide which of the content updates are important and which not.

What Reddit and other content aggregators are doing is to create/monitor the live ticker of the Internet. In contrast to the Wikipedia last changes section, not only a single websites is on the ticker, but the complete Internet.

The edit counter of Wikipedia per minute is known, because the data are available in the public domain. An estimation of the complete Internet was made in discussion https://www.quora.com/How-many-websites-are-created-each-year-month-week which comes to the conclusion that 10 websites are added each minute. It is measuring only new domains, if the same websites posts something new the number is higher. From another statistics it is known, that each day 4 million blogposts are created which is equal to 2778 postings per Minute. And blogs are only a small part of the internet. If we would monitor all the edits in a lifeticker, the number of changes each minute would be higher.

So called news aggregation websites are trying to monitor, evaluate, categorize and contextualize the changes in the Internet. The reason why so many bots are available who are posting URLs into aggregator websites is because humans are overwhelmed by such a task. Which human can evaluate 1 million updates each minute?

Let us simplify the process a bit. The assumption is that in the intranet the only website which is available is a wiki which is filled with information. Apart from the wiki no other domain is available. So the only URL which contains content is https://localhost/wiki/

If the wiki is updated this is made visible in the “last changes” section. To make the Wiki more transparent in the wiki an article is created called “News”. Humans are monitoring the last changes section and put the URL of the change into the news section. Sometimes they add a small comment.

On the first look, it is some kind of extra work. Because the wiki can monitor itself in the version history. The problem is, that this doesn't provide the context. A new user who is not familiar with the wiki has no entry point to get an overview. If he can read through the news section, he get a better impression how the wiki is working internally.

Summarize this concept is easy. Knowledge contains of three parts: the fulltext, a search engine, and annotated news created by humans to give an overview in a chronological order.

Blog aggregators in science

Creating a science blog is not very complicated. This can be done by any amateur writer who has access to an internet PC. If some content was put on the blog, the Google crawler bot will scan the content and now the world can find the information. But who determines, how much traffic will be directed to the blog?

The good news is, that apart from science blogs, this question was answered in the mainstream internet already. The amount of websites who are explaining who to rank the own blog higher in the google list is endless. The concept can be adapted to science blogs as well. A well known technique is called a blog aggregator. Typical examples from the science community are: The Early Modern Commons, http://researchblogging.org/ and http://cstheory-feed.org/ What these websites have in common is, that they are meta-blogs. They are not containing information but they are collecting URLs to other blogs. In case of “The early modern commons” the topic is about the middle age. The advantage of a blog aggregator over a single authored blog is, that each day lots of news are shown, similar to a large scale newspaper. These article were created by a distributed authorship who have individual blogs. The same is done in the CS-Theory blog, but not for history topics but for mathematics. And the last Blog aggregator on the list, contains a collection of misc science topics from all domains like biology, physics, medicine and computer science.

The basic idea of a blog aggregator is the driving force behind large scale websites like Reddit. Reddit is not a scientific website, but has it's root in mainstream entertainment. Reddit sees themself as an aggregated internet. Similar to a search engine but with handcollected links. In contrast, the CStheory website contains a smaller amount of hand-selected posts. Instead an automatic feed aggregator was used to combine all the RSS feeds. It's not possible to upvote or downvote articles and the amount of interaction of the website is smaller.

Fulltext search engines like Google and Blog aggregators like RB are playing an important role in making existing content visible. The internet contains of two seperate parts. The first one are the information itself, which are the text, images, videos and forum discussions. This kind of data needs a lot of discspace and it is growing fast. The second part of the internet contains of infrastructure to search and catalog the information. This can be done automatically with search engines and semi-automatically with blog aggregators. A blog-aggregators needs less storage capacity but generates more traffic. For example the RB aggregator is visited much often by the users, while a single blog within the network gets only a little amount of traffic.

Blog aggregators are playing the role of a gate keeper. They are the first information point for users who doesn't know which information they need, and then the user can click on one of the links to read the fulltext.

Reddit

The perhaps most famous blog aggregator in the world is reddit. The website provides some unique features which are explaining the high amount of traffic. The first one is, that Reddit is handcrafted. Instead of using the planet software for automatically combine RSS feeds, the users have to post the URLs manually. Secondly, Reddit isn't compiled by a single person but by a group of persons. They are monitoring their activities on the site by upvoting and downvoting the reputation. And third, Reddit has a strong focus on mainstream relevant topics like entertainment, computergames and fashion which makes it attractive for a large audience to participate.

Reddit shouldn't be misinterpreted as a blogging website. A blog can be referenced from Reddit, but the content itself is hosted elsewhere. Reddit should be understood as an alternative to Google. A website which comes close was dmoz.org but with the exception that Reddit is working with timebased URLs.

Possible pitfalls

Around blog aggregator some legal questions are obvious. In the case of a german blog aggregator “planet history” the project was stopped because of copyright issues. http://carta.info/leistungsschutzrecht-der-zweite-streich-steht-im-koalitionsvertrag/ The case was, that in the RSS stream an embedded copyright protected picture was delivered. The same problem is visible for Google News, because the original authors of the content were not asked if the snippet of the information can be shown in the internet.

How the legal discussion ends is unclear, what we can say for sure is, that copyright related issues are handled comfortable if the blog aggregator lists only content which has a Creative commons license. Even if the original content is distributed through the RSS feed to a larger audience it's not a problem because Creative commons means, that the author has made clear that he is interested in distribution over the internet.

In contrast, creating a blog aggregator for copyright protected material, especially if it's located behind paywals can be problematic. The debate of the lawyers goes around the question what the difference is between a normal textlink, a nofollow link, an embedded picture and shared content.

The good news is, that Google has in the advanced section a filter to reduce the hits only to creative commons licenses. It's a good idea to prefer these content in the blog aggregator. According to the creative commons license, it's allowed not only to link to such material, but to copy the entire fulltext, aggregate it to a larger corpus and even replay the information in a meeting.

The difference between a blog and a newsletter

On the first look, both sort of media are sounds the same. Sometimes a blog is described as a website while a newsletter is send in the plain text format. But suppose, the newsletter is published in a Blog, there is no visible difference.

But a blog is very different from a newsletter. I would like to explain the details. A blog is something which is created by an individual. Oh i see, this definition doesn't work, because newsletters are created by individuals as well. So it seems, that it's easier to say what the shared principle is over the difference. Perhaps we should go a step backward before the internet age and using pen and paper as the prefered tool. Suppose somebody writes an essay about a topic he likes, for example he invents a short story. This is called a blog. The public won't have any interested in reading such information.

In contrast, a newsletter is created with a different target audience. No, this definition is also not correct because a newsletter can be addressed as a single person as well. I see, it's very hard to define what the difference is. Let me give a third trial.

A blog is feed by a single person, a newsletter is feed by a community. Even if a single person has created the newsletter he is using the input of a larger group. A newsletter is similar to a blog aggregator. Which means that RSS streams are mixed together. The newsletter is not discussing a certain topic, but it explain about which topic the community is talking. A typical example for an entry in a newsletter is, to explain that in the last week the stackoverflow community has solved a major problem in C++. The question is not which kind of problem they have solved, but the news was, who has discussed the issue.

A newsletter is about the social status of group members. It has to do with projects who are done by many. A newsletter is a typical management tool it is used to control larger amount of groups. If somebody questioning if a newsletter is right he is asking about a social structure, management and projects. This explains why newspapers and TV news have an important role into modern society. They are not only posting information, but their role is to manage groups. The funny thing is, that so called alternative newsletter have the same job. They are not an alternative to the newsletter concept but they are trying to manage groups in a different way.

The only real alternative to a newsletter is normal information which is not provided on a meta level but it is going into the details. If in a paper it's written down that 1+1=2 then this information stands alone. There is no need to ask which group is interested also in this problem. An important criteria for a newsletter is, that it's depended on time ad location. All newsletter are grounded at a certain date and a certain place. A typical ticker message starts with the introduction, “According to the embedded journalist in the sims 4, who is sitting in the garden, the party on monday was great”.

A typical example for a newletter was posted to the reddit community. The user has announced that he has build his first house which is a caffe and a techstore in the same building. https://www.reddit.com/r/thesims/comments/a8erpb/built_my_first_house_in_the_sims_4_today_a_caf%C3%A9/ The posting has become 58 upvotes so far, and there is need to forward this story to Wikinews to put it on the frontpage of the new issue. If the journalists at Wikinews are doing their job well, they will verify the source first and ask the Original poster for a small interview in which he can explain the details. For example the question is open if the house contains of two floors or only one.

Solving the peer review issue

The open access movement has one major bottleneck, it is called missing peer review. In the classical academic publishing system a manuscript is send to a ghostwriter, the ghostwriter provides peer review, then the journal is send to the journal and the journal publishes the paper. Other content is referencing to the peer reviewed paper and this produces the overall gutenberg galaxy. It is an unsolved question how to replicate this workflow without using a paid peer review service like Enago and without using a quality journal which accepts only peer reviewed papers.

The answer to the problem can be explained in a single sentence. A higher instance is need to evaluate the content which can be realized with a reddit clone which is working with the wiki principle. But let us sort the information a bit and describe the workflow step by step.

At first the academic author needs his own blog. On the blog a short paper is posted which contains of a single page. It can be formatted in the normal wordpress syntax which is markdown or as a pdf document. All authors are doing so, if we have 100 authors, each of them his its own blog and is posting papers regularly.

The second step is to evaluate the content with a higher instance. A common misconception is that a Google like search engine can do this task. If the content of the 100 blogs was indexed the public is able to search and retrieve the content. The problem is, that if somebody doesn't understand the paper he is not able to use the fulltext search engine the right way. The better alternative is to a use a reddit like blog aggregator for content evaluation. In the famous wikinews project this was demonstrated a while ago for evaluating newspaper articles. The same principle can be adapted to any other topic. The best way in doing so is not reddit but a normal wiki is great for the task. The idea is, that the users of the wiki are scanning the 100 blogs with the papers and put the URL to the wiki page.

This collaborative content filtering results into an edit war, because user1 will add a url to a paper which is not wanted by a second user. The conflict is solved within the wiki. That means, on the talk page, with edit wars and with the admin. If the wikigroup has made a decision the wiki acts as a content aggregator. It doesn't contains the fulltext of the papers, but only the url to the external blogs of the authors.

The most important aspect is, that the content creation is separated from the evaluation process. That means, the authors can create new content and flood their individual blog no matter what the users in the Wiki are doing. Also the wiki-users doesn't need to create content by it's own, but their job is to identify high quality URLs to existing content and write a small description teaser for it. A second task of the wiki users is to do edit wars with other users if a certain URL fits into the wiki or not.

The overall structure of 100 wordpress blogs plus 1 wiki provides a great hierarchy which is open to new contributors and establish a high quality standard at the same time. Some aspects of this idea are realized in the today's internet. For example the WIkinews project is known, blog aggregators are sometimes realized (the famous example is planet.gnome.org). Using this technology for the Open Access to realize a distributed peer review system is not realized yet.

The key concept is to understand what Reddit and WIkinews are doing in the current internet. They are not providing new content but they are peer reviewing existing content. In case of Reddit this is done by million of people while on Wikinews a high end Wiki system is used which is superior to Reddit's proprietary website. Unfortunatly, the number of users within Wikinews is small and the teaser texts before a URL are too long.