Designing a Generative Pictographic Language

Abstract

The ability to express our thoughts is a very powerful tool in our society. Being able to write is more difficult than being able to read, and this is especially for the Alphabetical languages/scripts. From personal experience, being able to write in Latin/Arabic/Chinese is a lot more difficult than just being able to read them and requires a greater understanding of the language.

We now have machines that can help us accurately classify images and read handwritten characters. However, for machines to gain a deeper understanding of the content they are processing, they will also need to be able to generate such content. The next natural step is to have machines draw simple pictures of what they are thinking about, and develop an ability to express themselves. Seeing how machines produce drawings may also provide us with some insights into their learning process.

In this project/paper, a machine will be trained to learn pictographic scripts by exposing it to a database of selected ancient and modern pictographic scripts. The machine learns by trying to form invariant patterns of the shapes and strokes that it sees, rather than recording exactly what it sees into memory, a simulation of how our brains operate. Afterwards, using its neural connections, the machine attempts to write/construct something out, stroke-by-stroke. A technique that could be applied and used on different platforms, opening the door for a language or means of communication for the future.

1 The Significance of Pictographic Language

1.1 The Origins of Writing

Described by the Swiss designer Adrian Frutiger (1928–2015) as “An early attempt to visualize language and make a record of linguistic discourse”, ice-age wall carvings date back to 27,000 to 40,000 years ago. (2006, p. 55). It is said that these drawings were often conveyed along a series of gestures, as a ritual or further elaboration of their meaning. While the drawings serve as a trace of such ancient times, the speech and gestures have long been lost among the years. Moreover, throughout the Sumerian and the Ancient Egyptian periods, the human body was often regarded as a reference point for these drawings, in which men and women featured were distinguished by either drawings of genitalia or full body figures. These significant wall drawings are pictographic signs and can be defined as “proto-writing”.

As the evolution of man civilizations progressed, proto-writing systems -featuring ideographic and/or mnemonic symbols- paved the way to the development of today’s writing systems. Representation systems of language that utilized visual means ultimately transformed into actual writing, allowing the reader to construct, reconstruct, and derive meaning from a linguistic utterance that is encoded in writing. Writing became a fundamental method of data documentation and information storage, in a tangible sense.

The exact time at which proto-writing developed into true writing systems is greatly debated among scholars. According to recent archaeological research, the origin and spread of writing is quite complex to identify, in which recent findings suggest that proto-cuneiform writing on clay tablets may have existed during the mid-forth millennium BC, in the Middle East. Since clay was inexpensive and famous for its longevity, both pictographic and abstract signs were carved into dampened clay using a reed or stick (Fig. 1).

Fig. 1.

Sumerian-proto-writing, clay tablet inscribed with details of food rations, dating from c.3300–3100 BC from southern Mesopotamia.

Studies suggest that the idea of writing gradually spread from a culture to another. The transform of pictograms to ideograms is considered the first stage of development, an effective move from iconic representation to symbolism. The following stage of development was the introduction of the Rebus. A need to communicate a higher level of detail was developed, and hence; the “Rebus” came to mean that the pictographic icon incorporated a phonetic sound associated with the icon.

Scholars acknowledge the independent development of true writing of language in two locales; in Mesopotamia around 3200 BC, and in Mesoamerica around 600 BC. The conventional phase of development from proto-writing to true writing systems of language, suggest the following stages of progression:

• 1. Pictorial (picture-based) writing system: Glyphs directly represent objects and concepts. In relation to this, the following sub-stages may be distinguished:

  • Mnemonic: Glyphs

  • Pictographic: Glyphs that represent an object or a concept

  • Ideographic: Graphemes (abstract symbols) that represent an idea or concept

• 2. Transitional system: A grapheme refers to the object or idea that it represents, and the name as well.

• 3. Phonetic system: Graphemes refer to sounds or spoken symbols, and the form of the grapheme is not related to its meanings. This is explained through the following sub-stages:

  • Verbal: A grapheme (logogram) that represents a whole word

  • Syllabic: A grapheme that represents a syllable

  • Alphabetic: A grapheme that represents an elementary sound.

1.2 Pictographic/Ideographic/Logographic Writing Systems

Writing systems are commonly classified into four categories: pictographic, logographic, syllabic, and alphabetic. Yet, any writing system may feature a combination of some or all of the four categories in varying ranges. Hence, attempting to ultimately classify a writing system might in fact be problematic. In light of this, a writing system is defined as “a complex system”. In pictographic scripts, graphemes are iconic images, while in ideographic scripts, graphemes represent concepts and/or ideas, rather than representing a specific word in language. In logographic writing systems, glyphs represent words or morphemes (a meaningful unit of a word that cannot be divided further).

2 Machine Learning

Our lives have become extremely digitally oriented at a very wild pace, ever since the dawn of computers in the middle of last century. Online databases and digital media have been regarded as a substitute for printing on paper as the main method of storing information. Today, all sorts of material, be it numbers, text, images, video, or audio, are stored, processed, and even transferred digitally, thanks to online connectivity. Such great level of digital processing results in an immense amount of data, what we can refer to as a “dataquake”, is the primary reason behind the prevalent interest in data analysis and machine learning.

For many applications that differ in nature (from vision to speech, from translation to robotics), man was not able to devise sufficient algorithms, despite extensive research dating back to the 1950s. Yet, for all these tasks, data collection was quite easy, and so the current aim is to automatically learn the algorithms from the collected data, substituting programmers with learning programs. This is the fundamental niche of machine learning. Not only has data increased tremendously over the years, but also the extent to which data is successfully transformed into knowledge through machine learning, has significantly advanced.

In essence, machine learning, a branch of artificial intelligence, is a key method of data analysis that utilizes analytical model building. It is based on the core idea of systems/machines are capable of learning from data they are exposed to, accurately identifying and recognizing patterns upon which an algorithm is applied with minimal or almost complete absence of human intervention.

3 Generated Language

In light of the concept of machine learning, the prospect of generating a novel language becomes a certain scenario. Relying on pattern recognition and the theory that computers can learn by merely being exposed to data, without the necessity of being programmed to perform specific tasks, machines can indeed offer mankind a newly developed language (writing system) that is conceived from its processed language(s).

After becoming exposed to a set of characters and/or symbols, a machine becomes capable of independently adapting, learning from acquired computations to produce reliable, repeatable results on a very large scale as it weaves the similarities amongst the data it has been exposed to.

4 Selected Generated Language Projects

4.1 A Book from the Sky 天书: Exploring the Latent Space of Chinese Handwriting (Xu Bing) (1988)

Based on Xu Bing’s book titled “A Book from the Sky”, the project features a huge collection of characters created by a deep convolutional generative adversarial network (DCGAN) that is instructed on a database of handwritten Chinese characters. DCGAN is a form of convolutional neural network capable of learning abstract representations after being exposed to a collection of images. It achieves this by balancing between functioning as a “generator” that fabricates fake images and a “discriminator” that attempts to detect whether the generator’s images are authentic. After training, the generator is guaranteed to accurately generate samples of images that are resonant of their originals.

Below is a set of fake images of characters produced by a DCGAN that was exposed to a labeled subset of ~1 M handwritten simplified Chinese characters. The generator was then capable of producing fake images of characters that were not present in the original dataset (Fig. 2).

Fig. 2.

Generated characters produced by DCGAN.

Xu Bing’s Book.

The original book upon which this project was built, is created by Chinese artist Xu Bing who featured the styles of fine editions from the Song and Ming dynasties, yet filled entirely with meaningless glyphs that are designed to look like traditional Chinese characters. The book consists of four volumes, forming 604 pages in total, and was printed in a single print run of 126 copies between 1987 and 1991. It was composed incorporating a set of 4,000 characters, which is approximately the number of characters commonly used in modern written Chinese. The characters were engraved into single pieces of movable type made from pear wood, in a style slightly thicker than that of Song typefaces. Firstly, Xu himself typeset samples of the book’s pages and then took them to a factory in the village of Hányíng for printing. Workers at the factory then typeset the pages by following a “model book” prepared by Xu as a reference. The book’s first public exhibition was in October of 1988, at the Beijing’s China Art Gallery (Fig. 3).

Fig. 3.

Installation view of Book from the Sky (1987–91) in “Xu Bing: A Retrospective” at Taipei Fine Arts Museum, 2014.

4.2 Alphabet Synthesis Machine (2001)

Created by both Golan Levin, Jonathan Feinberg, and Cassidy Curtis, and commissioned by PBS and Art21.org, the “Alphabet Synthesis Machine” is an interactive online artwork that allows one to develop and evolve potential writing systems of one’s own imaginary civilizations. The software produces abstract alphabet that can be downloaded later as TrueType fonts, and are registered into a complete archive of user creations.

The final products of the “Alphabet Synthesis Machine” investigate the liminal boundaries between familiarity and chaos, language and gesture. The project came to be as both an interactive installation and an online tool. From 2001 to 2006, users who visited the website created more than 20,000 abstract writing systems (Fig. 4).

Fig. 4.

“Alphabet synthesis machine” project

4.3 Random Radicals: Kanji Machine Learning (2012)

In this project, a machine was trained to learn Chinese characters by being exposed to a Kanji (Japanese term for Chinese Characters) database. Similar to how a human brain functions, the machine learned by trying to formulate invariant patterns of the shapes and strokes it is exposed to, rather than merely recording the data in its memory. Then, utilizing its neural connections, the machine became successfully capable of generating characters onto a screen, stroke by stroke (Fig. 5).

Fig. 5.

Kanji generated script

After acquiring sufficient stroke-level data from the Kanji database, the machine became able to categorize certain strokes together on its own, eventually forming more abstract concepts of basic radicals and components that initially make up the original Kanji language, such as 口, 豆, 辶. Furthermore, the machine also learned to write these radicals in the correct number of strokes, and proper sequence. For instance, 口 must be written featuring three distinct strokes,, in this order, and cannot be drawn at once as a square ⃞ with a single stroke. Despite how one may arrive at a final character after following an incorrect stroke order, the result is also regarded as an incorrect Kanji. Hence, although the machine attempts to construct fake Kanji, it preserves the internal logical order of Japanese Kanji, at both the stroke level and radical level, as it has established a logical pattern about the relative location, and relative sequence to structure its abstract concepts.

Kanji has existed for thousands of years, and hence the writing system has developed numerous branches and divisions. For example, there are two different methods to write 逗, either by using the simplified or the traditional 辶. Accordingly, the machine also processed the different variations of writing certain Kanji components. The eventual outcome of the machine, is a set of vector lines which suits Kanji much more adequately than many other recent machine learning image generation methods that are primarily pixel based.

Considering how the ordering of strokes is of fundamental importance to Kanji writing system, using vectors to model the ordered strokes is much more significant than any pixel based generation technique. Remarkably, this approach may pave the way to future developments of facilitating machines to sketch images.

4.4 Asemic Languages

Created by So Kanno and Takahiro Yamaguchi, the project “Asemic Languages”, directs particular attention towards the form of characters rather than their embedded meanings. Historical civilizations throughout the world have acquired numerous characters as part of their identities, conveying their distinctive culture and history. Accordingly, emphasizing the role of characters as a core means of visual communication and documentation of language, the project features drawn lines that resemble characters but do not hold any meaning.

The project was publicized in 2016 at the international art festival “Aichi Triennale”. The design implementation process incorporated a collection of handwritten statements by artists. By learning the handwriting with one writer in each language, the machine collected visual data on the shapes of each character system, as well as the idiosyncrasies of each writer. The lines were generated via a plotter, and written as if they reflect something of great importance, visually deceiving the audience (Fig. 6).

Fig. 6.

“Asemic Languages” project

5 Project Concept

The pictography existing in all early scripts of mankind is a crucial cornerstone in the theoretical argument of universal iconography common to all writing systems. The fact that all independently derived writing systems came to be as arrangements of pictograms before their evolution into sophisticated forms, serves as evidence to the significant iconographic nature of writing, as a notion.

In light of what has been raised and examined above, the aim of my project revolves around the basic idea of introducing a designed pictographic generative language utilizing machine learning. The machine would be exposed to a database of vector-based ancient pictographic scripts, ranging from Sumerian Cuneiforms, Egyptian hieroglyphs, Dongba and Nsibidi symbols, Testerian Catechism scripts, to Chinese characters. By forming consistent patterns of the shapes and strokes it processes, the machine utilizes its neural connections in attempting to produce new pictographic characters, stroke by stroke, onto a digital screen.

Ultimately, by recognizing and grouping similar patterns and pinpointing the similarities amongst these scripts in relation to style of strokes, complexity of figures, and proportions, the machine becomes capable of generating a firsthand pictographic language reflecting the homogenous characteristics of each of the writing systems, combined. Writing systems created civilizations, hence; the final result produced, would serve as a unique investigation of the existing, yet unconsciously neglected, relations among the diverse cultures of many civilizations (Figs. 7, 8 and 9).

Fig. 7.

Hieroglyphs from the tomb of Seti I, Valley of the Kings, Luxor.

Fig. 8.

Detail from a Naxi manuscript, displaying pictographic Dongba.

Fig. 9.

Testerian catechism

6 Conclusion

In conclusion, writing systems are instruments of the various languages that have came to exist throughout the long history of mankind. They are considered a historical necessity of the emergence of our countless languages. Accordingly, an investigation of ancient pictographic languages/scripts utilizing machine learning would offer insight into the visual language of the cultures of ancient civilizations, proposing a modern pictographic language of whose traits are drawn by the machine directly from the data it has acquired and processed.