There's another feature of unix processes: they isolate the user code
from the system code, notably in terms of user/supervisor modes, and
memory protection. (Even if as an optimization a unix process can run in
user mode, from kernel code mapped in the user address space, it's code
that is mapped read-only and entirely under control of the kernel).

There's one feature of pipes we don't find in normal chains of function
calls:  they synchronize parallel processes.

So a PIPE operation can still be useful in a lisp REPL, that would run
producers/consumers in parallel, and synchronize them according to the
speed they produce/consume lisp objects.

Concerning the hierarchical file system, the proposed alternative to use
keywords to classify automatically the items goes too far in the other
direction: you have to remember the keywords!

They don't often show the list of keywords that have been used so far,
and neither do they propose easy ways to navigate them.

On the other hand, what works well with humans, is the spatial memory.
It is even used as a mnemotechnic trick to remember large quantities of
data.  That's where a hierarchical file system is good, when you use it
to build a spatial structure from as high a level you want down to as
low a level you want (eg. planet, country, city, street, house, room,
cabinet, drawer, file (the real one ;-)).  cf. also Apple's idea of the
desktop metaphor, where document icons are placed in _physical_
locations (the coordinates are recorded in the files metadata and are
never lost). This gives a tangible way to find documents, geographically
based, in each folder window.  However, this fails when the number of items
become too big.  (Personnaly I can find more easily files left unused
for a long time by walking back to their path than by searching them
graphically, or by keywords, but clearly we can't generalize).

While google sees and shows the web as a bunch of links, indexed by
keywords, the resources are still organized hierarchically:

          protocol:          http
          tld:               com
          domain             metamodular
          (no sub*domain)
          hierarchical path: /Common-Lisp
          file:              lispos
          type:              html

For some reason, it made more sense to use this hierarchical pathname,
rather than to let google index it and giving me three keywords to find
this document thru it.

In conclusion, I would not reject the hierarchical file system, but
rather provide various paradigms of file classification.

In EROS, a capability based system, there's no predefined file
system. There's a root object, and domain objects that can let the user
connect to them and objects that will manage the keys for the user, and
some keys may let the user access an object that manage files, letting
the user retrieve them according to their own schemes and rules.   So:

1- file systems organization are entirely "virtual", managed by
   application level objects,

2- there can be several different kind of file system managing objects,

3- the user only has access to the file system objects or other data
   objects to which he has access.  Systematic "chroot".

Indeed, it's more like a persistent object database than a file system

With 64-bit addressing spaces, the address can also identify network
objects, with the disk playing the role of a cache for the accessible
network objects.  cf. eg. DropBox, or web caches, etc.

Against the loss of data, journaled disk I/O works well enough.
cf. journaled file systems like ext3; in the case of EROS, the system
keeps a journal, and has an optimized background task to update the
disk, which archived better I/O throughput than unix file systems.

Of course, the presence of batteries in a lot of modern computer system
helps a lot.

I agree that eventually a LispOS will have an archicture closer to
capability based OSes than to unix.

One important question is the role of the language or language run-time
in the architecture of the system, notably the status of the code
generated by the compiler.

1- Do we allow other code generators? Such as compilers of other languages
generating native code?  If we do, it means that the level at which we
have to enforce controls is that of the processor, with user
mode/supervisor mode and MMUs, etc.  And we are considering spreading
elements of the Common Lisp system across those two parts.

2- Or do we enforce that all code that runs on the system goes thru the
system compiler?  After all, Common Lisp provides a low level language
(the 25 special operators), and we could reject execution of data
generated outside of the system compiler.  Or like Java, we could have
a validator of code generated, before switching it to the executable

In the case of EROS, while there's a single addressing space,
accessibility of address ranges depends on the capabilities the domain
has.  There's memory protection enforced with the usual hardware
features (supervisor mode vs. user mode, MMU, etc).  So it's more like
option 1-.

One objective for me would be to have a system that is introspectable at
the source level and which could be modified while running (like a
Smalltalk system or Lisp Machine systems).  This should not be too
difficult to attain with the option 2- and the management of the
environments.  Assuming the system and CL package are protected, we
could provide an operator to make a writable copy of the parts of the system or CL
environment the user wants to modify and there on, the user would be
working with a forked "virtual" machine.  Actually, all domains (EROS
terminology = set of capabilities ≅process) are already like virtual
machines, since each has its own environment.

Considering this notion of lisp environment, with the need for
protection,  I would say that the unit in such a system would be the CL
COMPILATION-UNIT.  Which can be as small as a single function, or as big
as a whole (asdf) system compiled and optimized in a single

Let's say that we give names to environments or subenvironments,
hierarchically at the superior level to the package names, like:

  (mount beach::: pjb:::com.informatimago.common-lisp.cesarum
         :access '(:read :execute))

would make the package COM.INFORMATIMAGO.COMMON-LISP.CESARUM from the
environment PJB readable and executable in the environment BEACH.
That gives access to all the units linked to symbols in that package.

  (mount pjb:::cl (copy-package system:::cl) :access '(:read :write :execute))

would replace in the PJB environment CL package by a copy of the CL package
that is modifiable.

We'll need to list all the elements to put in these environment.

- packages (mapping of package names to packages),
- global bindings of symbol in the packages:
    - macro-function,
    - fdefinition,
    - symbol value (therefore *readtable*, therefore reader macros),
    - symbol macro.
- sub-environments (so a user may create his own specific use environments),
(non exhaustive list)

We would have to add a token binding for special operators, to allow a
(any) CL package to be used to read sources to be compiled by the system
compiler.  It could be a set of special macros bound to macro-function.

(setf (macro-function 'si) (macro-function 'system:::cl:if))
would copy the special operator status of IF to the symbol SI, so that:

   (compile '(lambda (a b) (si (zerop a) b (1- b))))

would compile just like:

   (compile '(lambda (a b) (if (zerop a) b (1- b))))

but also pjb:::cl:if,
 when (not (eql (system:::system:find-package "CL" :environment "PJB")
                (system:::system:find-package "CL" :environment "SYSTEM")))

About generic functions, I'm wondering if we couldn't rely on the
symbols naming the specialisers?

If beach:::example:toy  and  pjb:::example:toy are two different symbols
naming a class, then (defmethod print-object ((self toy) stream) …) read
and evaluated in the beach::: environment where beach:::cl:*package* is
bound to the package beach:::example, or the pjb::: environment where
pjb:::cl:*package* is bound to the package pjb:::example would define
two different methods on two different classes, even if print-object is
system:::cl:print-object in both cases. (CL already forbids definition
of standard methods on standard classes (named with symbols in CL,
ie. in system:::cl).

And I'm wondering because there could be specialization/generalization
relationships between classes across environments, so a call to
(print-object my-toy *standard-output*) could involve methods in
different environments in the (call-next-method) chain.

Now, the thing is that each time the control goes from a unit in one
environment to a unit in another environment, capabilities and
environments must be checked managed. (cf. capability OSes).

I'm not sure capability management can be reduced to some bits in
addresses.  IIRC, in EROS, there's some more sophisticated processing done, to
ensure correct security rules.

One thing: if the system:::cl package is compiled as a single
compilation unit, then it's a single environment unit, which means that
when the function system:::cl:load calls system:::cl:read, it jumps to
the code in the same unit.  If I take a writable copy of that package,
and redefine pjb:::cl:read, system:::cl:load will not call my new unit.
On the other hand, if we provide system:::cl with independent
compilation units for each function and macro, when the function
system:::cl:load calls system:::cl:read it goes thru the symbol cl:read
in the current environment, so when I redefine pjb:::cl:read, and call
system:::cl:load in the pjb::: environment, will call system:::cl:load
my pjb:::cl:read unit.  Both options may be useful.

So we'd have a system buit upon:

- units of code (one or more functions with an ABI defined by the system

- environments (a set of bindings: package, symbol functions, symbol
  values, etc).

- persistent objects.

- capabilities to manage access to the objects and units of code.

Are the special operators (and the few other notions like bindings,
closures, etc) sufficient an API to the system compiler?

Do we need new special operators (eg. to manage capabilities)?

Do we need a lower level API (eg. LLVM)?

In a system with persistent objects, there's a system garbage collector
(even at 4TB hard disks fill quickly).  Is this garbage collector able
to manage the memory for user programs too, or do we need or want to
have application level garbage collectors?  (I'd guess that
generationnal garbage collectors would work at all those levels).  Also,
we may want a parallel and/or real-time garbage collector (just for the
competition ;-)).

Similarly, in eg. EROS, IIRC each object brings along set of
capabilities.  That would make some overhead on cons cells and other
small lisp objects.  Perhaps we need to keep a notion of lisp image,
with an internal garbage collector, and those lisp images would be the
persistent data unit with its list of capabilities.  But that sounds
like quite a complication: the objects inside the lisp image wouldn't be
system objects accessible from the other environments.

Let's imagine a list of objects (eg. windows) managed by the system.

   (system:::display:all-windows) -> (#<window "emacs"> #<window "repl"> …)

assume the system collects those windows object from all the
environements, and builds a list of which it gives our environment read
access: we can read the car and the cdr of each cons cell.  However,
some windows may be in some environment which don't give us access to
their window objects.

Calls such as (system:::display:window-position #<window "login">) or
(system:::display:send-key-event #<window "login"> #\x) would not be
able to read the login window position or write its event queue.

But other windows in the list (ours) would be accessible.

That said, when implementing a different compiler, or a virtual machine,
or a specific application, it may also be an implementation choice to
put all the data of a program in such a big "image" object, and produce
a single unit of code to manipulate these image objects.

So for example, a device driver written in C could be compiled into a
single unit of code (with lisp entry points and using system lisp
objects externally), addressing its C buffers and data structure
internally in a single "C" image object.

Perhaps we should use the word "domain" instead of "environment", since
"environment" already means something in CL, and "domain" is used in
some capability OSes to denote something similar?  (But there are also
some differences, we'll see).

Too bad Jonathan Shapiro has been hired during a few years by Microsoft
so he didn't work much on coyotos, and he's somewhat sidetracked
designing BitC, his system programming language.  He hasn't made much
progress on coyotos.

In any case:

1- it looks like it could work,

2- the crucial part is the design of the environments and the bindings we
   put in it, along with the system compiler support for them,

3- this could be added to a CL implementation to explore without needing
   the rest of the system (persistance, capability access control,
   etc).  Actually with techniques like IBCL (shadowing packages) and
   some reader macro hackery, a prototype could be done without even
   having to hack the entrails of a CL compiler.

4- apart from those OS level developments, CL applications would still
   have their familiar CL environment, so there would be minimal work to
   "port" them; we can provide them a posixy virtual file system and the
   needed features to make them happy.

I would have to re-read the EROS and coyotos papers, (and perhaps some
other capability OS stuff), to make a summary of the relationships
between code units, data objects, environments and capabilities.   And
we could hack a CL implementation to add those environments and see what
comes out of it.

Unfortunately, since I'm currently salaried, I'm left with only at most
a couple of hours a week to consacre to this project.

Basically, the idea of my LEP project was just to gather existing
lisp applications and lisp servers in a system, and see what a purely
CL environment would be able to archieve nowadays; not much I expect, but
there is some code already in a lot of application domains, so gathered
in a single place, that could be a critical mass to attract some
interest?  But I'm also worried that starting with an architecture too
similar to unix systems, the user experience wouldn't be that different,
just a little more inconvenient.

Crash proof (maybe)

One aspect is that of the consistency of the data structures.

One thing that fsck does, is garbage collection: it checks the link
count of each node, and re-attach in /lost+found those that have a
positive link count with no directory entry, and otherwise, it removes
the nodes that have a link count of 0.

Otherwise, fsck checks and corrects the structure of two kinds of unix
"objects": the hierarchical directory, and the file blocks.

One thing that is not done by fsck, is to check the data structures
_inside_ the files.

If we want a crash proof system, with unification of primary and
secondary memory, we must be able to ensure consistency of all data in

The lisp garbage collector performs part of it: if we lose a reference
to a lisp object, then it's collected, and that's it. Perhaps it would
be a good idea to keep several references to the objects close to the
root; if the root pointer was mangled (nullified) in a crash, the
garbage collector could free everything when we restart.  Otherwise
there could be an option to scan memory and recover unreferenced
objects, but this sounds very dangerous.

But the delicate part is what fsck doesn't do: checking the
consistency of data.  If the crash occurs when partial bit patterns
have been commited to memory, then we may read back inconsistent or
invalid data.  I guess invalid data would be caught by the run time as
type errors.  Inconsistent could be more problematic, and would even
be a security risk, eg. if a reference to a string is replaced by
hasard of the bit patterns by a reference to an object belonging to a
different environment.

In the absolute, we can't count on the data written to persistent
secondary memory to be 100% consistent, after a crash.

And as you wrote, there can always be defects in software, including
system software.

We could have tools that would do consistency checks beyond what the
garbage collector would do, for example, checking for type patterns
over data structures.  And depending on the security level (the
environment in/capability with  which we run the tool), we could
either allow modification of the bad data structures, or cutting them
out (resetting the reference to the bad data structure so it gets
garbage collected).  This would rely on an effective type system (with
satisfies functions to perform deep checks, validate the capabilities,