Rich Datatypes Without Datatype Fascism

Introduction

The basic idea of the YOSH shell is to introduce a richer set of datatypes to the command shell and the programs that run in it. Some people who I've discussed the idea with have assumed that this means that programs that would operate in this environment must communicate in a common data format mandated by the shell. In a sense this is true: but the key idea is that programs must use some common format to identify how their data is encoded: it is not necessary that the programs actually change how they encode their data.

Conformity vs. Inspectability

Basically there are two ways to make different tools work together automatically... The first is to make them use the same datatype, while the second is to make them be very explicit about what data type they're communicating with, so their peers will know how to translate the data, if necessary...

Conformity is generally a problem because people using old tools don't want to rewrite them to use a different data format, and people writing new tools want to choose (or design) the format that's best for the job, rather than getting roped into some less optimal format just because it's the local standard.

What interests me more than conformity is inspectability: the power to know just enough about the formatting of a piece of data to know what to do with it, if it's not in the format you expect. This accomplishes a number of useful things:

• A process receiving a piece of data can recognize situations where it's being handed incompatible data formats
• A process or a user with a piece of data in an unrecognized format can reliably identify both the format
• By identifying the data format, a facilitating library or process may be able to translate the datastream to a format the process can work with

Of course, this can't be accomplished without a certain level of conformity. Processes may communicate the data itself however they like, but they must agree to a standard format for identifying the data structure they've used.

YOSH approaches to providing inspectability

YOSH-native communication format

Programs targetted at the YOSH environment may, of course, be specifically written for it. Though naturally I know there won't be many programs written like this for a long time, if ever... But this capability is an important building block for other approaches. Naturally, this native format will be written such that, to the extent possible, it will never be misidentified as another file format and vice versa. I also figure that the format should be written such that, when data of some other format is wrapped in it, it's fairly easy to extract again.

Type Identification Channels

YOSH will provide ways to describe the behavior of a program statically: this information about the behavior and data formats used by the program (which may include calling arguments, input and output formats, etc.) could be attached to the executable itself through file extended attributes, through a set of centrally-located rules, through an interface script on the path, or could possibly be embedded into ELF binaries themselves.

Default Assumptions

For processes which are not written for or adapted to the YOSH environment, some baseline assumptions will still be available to bind these programs to YOSH in useful ways:

• Output Form Recognition and Sanctioned Data Types: In lieu of any other data being provided about the output of a program, if the output is of one of these "sanctioned" forms then the shell will reliably recognize the output format on the fly, and add explicit type identification to the data. In order to be able to reliably perform this level of auto-detection, "sanctioned data types" may only contain those datatypes which can be reliably autodetected: meaning that they can't be mistaken for another datatype, and no other datatype will ever be mistaken for one of them. The mechanism for identifying types is similar to the /etc/magic database, but a key difference is that the set of YOSH-sanctioned datatypes will be smaller, limited to the set of formats most commonly encountered, with an emphasis on the most universal ones and those for which auto-detection is most reliable.
• Default Input Forms: By default it is assumed that any data passed to a command as an argument must be text, and any data fed into a program's input stream is an undecorated binary representation of the data. For foreign types the type identification data is simply stripped away. For YOSH-native datatypes, the data is delivered in YOSH-native binary format.
• Interactive Application Detection: Generally YOSH commands are functions, objects or libraries. The expectation is that a program run from YOSH won't do something like take over the console unless YOSH knows that it's an interactive program. YOSH commands are run in a read-eval-print loop, which means that if the shell isn't aware that a program's output is intended for the terminal, then it will be interpreted and presented as result data. If it is necessary to auto-detect cases where a program operates interactively on the terminal, there may be some ways to do it. For instance, a program that links to a curses or slang library is likely interactive. Alternately, the shell could simply guard against console applications by clearing the TERM environment variable when running programs it can't account for. For sensible applications, this would result in a recognizable error message, and the user would know to either map behavior information to the application or to run an alternate command that would explicitly contain the new application in a new terminal window (for instance, xterm). Currently, handling these scenarios is a weak point of the YOSH design...