From nobody Sun Oct 27 17:14:08 2013
X-From-Line: nobody Thu Oct 24 00:24:57 2013
From: "Pascal J. Bourguignon" <pjb@informatimago.com>
To: Robert Strandh <robert.strandh@gmail.com>
Subject: Re: Initial version of a LispOS web page.
References: <CADUeJY7rGGhCm4ba3RAeCK7vtqZSiC=jwJtz3ZBTs3=fPthn3A@mail.gmail.com>
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwAQMAAABtzGvEAAAABlBMVEUAAAD///+l2Z/dAAAA
oElEQVR4nK3OsRHCMAwF0O8YQufUNIQRGIAja9CxSA55AxZgFO4coMgYrEDDQZWPIlNAjwq9
033pbOBPtbXuB6PKNBn5gZkhGa86Z4x2wE67O+06WxGD/HCOGR0deY3f9Ijwwt7rNGNf6Oac
l/GuZTF1wFGKiYYHKSFAkjIo1b6sCYS1sVmFhhhahKQssRjRT90ITWUk6vvK3RsPGs+M1RuR
mV+hO/VvFAAAAABJRU5ErkJggg==
X-Draft-From: ("nnimap+voyager.informatimago.com:INBOX" 27157)
X-Accept-Language: fr, es, en
Date: Thu, 24 Oct 2013 00:24:56 +0200
In-Reply-To: <CADUeJY7rGGhCm4ba3RAeCK7vtqZSiC=jwJtz3ZBTs3=fPthn3A@mail.gmail.com>
(Robert Strandh's message of "Mon, 21 Oct 2013 11:06:47 +0200")
Message-ID: <87hac7fww7.fsf@informatimago.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Lines: 369
Xref: kuiper sent.2013-10:42
X-Gnus-Article-Number: 42 Thu, 24 Oct 2013 00:24:57 +0200
Robert Strandh <robert.strandh@gmail.com> writes:
> http://metamodular.com/Common-Lisp/lispos.html
> Let me know what you think.
There's another feature of unix processes: they isolate the user code
from the system code, notably in terms of user/supervisor modes, and
memory protection. (Even if as an optimization a unix process can run in
user mode, from kernel code mapped in the user address space, it's code
that is mapped read-only and entirely under control of the kernel).
There's one feature of pipes we don't find in normal chains of function
calls: they synchronize parallel processes.
So a PIPE operation can still be useful in a lisp REPL, that would run
producers/consumers in parallel, and synchronize them according to the
speed they produce/consume lisp objects.
Concerning the hierarchical file system, the proposed alternative to use
keywords to classify automatically the items goes too far in the other
direction: you have to remember the keywords!
They don't often show the list of keywords that have been used so far,
and neither do they propose easy ways to navigate them.
On the other hand, what works well with humans, is the spatial memory.
It is even used as a mnemotechnic trick to remember large quantities of
data. That's where a hierarchical file system is good, when you use it
to build a spatial structure from as high a level you want down to as
low a level you want (eg. planet, country, city, street, house, room,
cabinet, drawer, file (the real one ;-)). cf. also Apple's idea of the
desktop metaphor, where document icons are placed in _physical_
locations (the coordinates are recorded in the files metadata and are
never lost). This gives a tangible way to find documents, geographically
based, in each folder window. However, this fails when the number of items
become too big. (Personnaly I can find more easily files left unused
for a long time by walking back to their path than by searching them
graphically, or by keywords, but clearly we can't generalize).
While google sees and shows the web as a bunch of links, indexed by
keywords, the resources are still organized hierarchically:
protocol: http
tld: com
domain metamodular
(no sub*domain)
hierarchical path: /Common-Lisp
file: lispos
type: html
For some reason, it made more sense to use this hierarchical pathname,
rather than to let google index it and giving me three keywords to find
this document thru it.
In conclusion, I would not reject the hierarchical file system, but
rather provide various paradigms of file classification.
In EROS, a capability based system, there's no predefined file
system. There's a root object, and domain objects that can let the user
connect to them and objects that will manage the keys for the user, and
some keys may let the user access an object that manage files, letting
the user retrieve them according to their own schemes and rules. So:
1- file systems organization are entirely "virtual", managed by
application level objects,
2- there can be several different kind of file system managing objects,
3- the user only has access to the file system objects or other data
objects to which he has access. Systematic "chroot".
Indeed, it's more like a persistent object database than a file system
properly.
With 64-bit addressing spaces, the address can also identify network
objects, with the disk playing the role of a cache for the accessible
network objects. cf. eg. DropBox, or web caches, etc.
Against the loss of data, journaled disk I/O works well enough.
cf. journaled file systems like ext3; in the case of EROS, the system
keeps a journal, and has an optimized background task to update the
disk, which archived better I/O throughput than unix file systems.
Of course, the presence of batteries in a lot of modern computer system
helps a lot.
http://coyotos.org/docs/index.html
http://eros-os.org/eros.html
I agree that eventually a LispOS will have an archicture closer to
capability based OSes than to unix.
One important question is the role of the language or language run-time
in the architecture of the system, notably the status of the code
generated by the compiler.
1- Do we allow other code generators? Such as compilers of other languages
generating native code? If we do, it means that the level at which we
have to enforce controls is that of the processor, with user
mode/supervisor mode and MMUs, etc. And we are considering spreading
elements of the Common Lisp system across those two parts.
2- Or do we enforce that all code that runs on the system goes thru the
system compiler? After all, Common Lisp provides a low level language
(the 25 special operators), and we could reject execution of data
generated outside of the system compiler. Or like Java, we could have
a validator of code generated, before switching it to the executable
status.
In the case of EROS, while there's a single addressing space,
accessibility of address ranges depends on the capabilities the domain
has. There's memory protection enforced with the usual hardware
features (supervisor mode vs. user mode, MMU, etc). So it's more like
option 1-.
One objective for me would be to have a system that is introspectable at
the source level and which could be modified while running (like a
Smalltalk system or Lisp Machine systems). This should not be too
difficult to attain with the option 2- and the management of the
environments. Assuming the system and CL package are protected, we
could provide an operator to make a writable copy of the parts of the system or CL
environment the user wants to modify and there on, the user would be
working with a forked "virtual" machine. Actually, all domains (EROS
terminology = set of capabilities ≅process) are already like virtual
machines, since each has its own environment.
Considering this notion of lisp environment, with the need for
protection, I would say that the unit in such a system would be the CL
COMPILATION-UNIT. Which can be as small as a single function, or as big
as a whole (asdf) system compiled and optimized in a single
with-compilation-unit.
Let's say that we give names to environments or subenvironments,
hierarchically at the superior level to the package names, like:
BEACH:::CL:CAR or PJB:::CL:DEFUN.
(mount beach::: pjb:::com.informatimago.common-lisp.cesarum
:access '(:read :execute))
would make the package COM.INFORMATIMAGO.COMMON-LISP.CESARUM from the
environment PJB readable and executable in the environment BEACH.
That gives access to all the units linked to symbols in that package.
(mount pjb:::cl (copy-package system:::cl) :access '(:read :write :execute))
would replace in the PJB environment CL package by a copy of the CL package
that is modifiable.
We'll need to list all the elements to put in these environment.
- packages (mapping of package names to packages),
- global bindings of symbol in the packages:
- macro-function,
- fdefinition,
- symbol value (therefore *readtable*, therefore reader macros),
- symbol macro.
- sub-environments (so a user may create his own specific use environments),
(non exhaustive list)
We would have to add a token binding for special operators, to allow a
(any) CL package to be used to read sources to be compiled by the system
compiler. It could be a set of special macros bound to macro-function.
(setf (macro-function 'si) (macro-function 'system:::cl:if))
would copy the special operator status of IF to the symbol SI, so that:
(compile '(lambda (a b) (si (zerop a) b (1- b))))
would compile just like:
(compile '(lambda (a b) (if (zerop a) b (1- b))))
but also pjb:::cl:if,
when (not (eql (system:::system:find-package "CL" :environment "PJB")
(system:::system:find-package "CL" :environment "SYSTEM")))
About generic functions, I'm wondering if we couldn't rely on the
symbols naming the specialisers?
If beach:::example:toy and pjb:::example:toy are two different symbols
naming a class, then (defmethod print-object ((self toy) stream) …) read
and evaluated in the beach::: environment where beach:::cl:*package* is
bound to the package beach:::example, or the pjb::: environment where
pjb:::cl:*package* is bound to the package pjb:::example would define
two different methods on two different classes, even if print-object is
system:::cl:print-object in both cases. (CL already forbids definition
of standard methods on standard classes (named with symbols in CL,
ie. in system:::cl).
And I'm wondering because there could be specialization/generalization
relationships between classes across environments, so a call to
(print-object my-toy *standard-output*) could involve methods in
different environments in the (call-next-method) chain.
Now, the thing is that each time the control goes from a unit in one
environment to a unit in another environment, capabilities and
environments must be checked managed. (cf. capability OSes).
I'm not sure capability management can be reduced to some bits in
addresses. IIRC, in EROS, there's some more sophisticated processing done, to
ensure correct security rules.
One thing: if the system:::cl package is compiled as a single
compilation unit, then it's a single environment unit, which means that
when the function system:::cl:load calls system:::cl:read, it jumps to
the code in the same unit. If I take a writable copy of that package,
and redefine pjb:::cl:read, system:::cl:load will not call my new unit.
On the other hand, if we provide system:::cl with independent
compilation units for each function and macro, when the function
system:::cl:load calls system:::cl:read it goes thru the symbol cl:read
in the current environment, so when I redefine pjb:::cl:read, and call
system:::cl:load in the pjb::: environment, will call system:::cl:load
my pjb:::cl:read unit. Both options may be useful.
So we'd have a system buit upon:
- units of code (one or more functions with an ABI defined by the system
compiler).
- environments (a set of bindings: package, symbol functions, symbol
values, etc).
- persistent objects.
- capabilities to manage access to the objects and units of code.
Are the special operators (and the few other notions like bindings,
closures, etc) sufficient an API to the system compiler?
Do we need new special operators (eg. to manage capabilities)?
Do we need a lower level API (eg. LLVM)?
In a system with persistent objects, there's a system garbage collector
(even at 4TB hard disks fill quickly). Is this garbage collector able
to manage the memory for user programs too, or do we need or want to
have application level garbage collectors? (I'd guess that
generationnal garbage collectors would work at all those levels). Also,
we may want a parallel and/or real-time garbage collector (just for the
competition ;-)).
Similarly, in eg. EROS, IIRC each object brings along set of
capabilities. That would make some overhead on cons cells and other
small lisp objects. Perhaps we need to keep a notion of lisp image,
with an internal garbage collector, and those lisp images would be the
persistent data unit with its list of capabilities. But that sounds
like quite a complication: the objects inside the lisp image wouldn't be
system objects accessible from the other environments.
Let's imagine a list of objects (eg. windows) managed by the system.
(system:::display:all-windows) -> (#<window "emacs"> #<window "repl"> …)
assume the system collects those windows object from all the
environements, and builds a list of which it gives our environment read
access: we can read the car and the cdr of each cons cell. However,
some windows may be in some environment which don't give us access to
their window objects.
Calls such as (system:::display:window-position #<window "login">) or
(system:::display:send-key-event #<window "login"> #\x) would not be
able to read the login window position or write its event queue.
But other windows in the list (ours) would be accessible.
That said, when implementing a different compiler, or a virtual machine,
or a specific application, it may also be an implementation choice to
put all the data of a program in such a big "image" object, and produce
a single unit of code to manipulate these image objects.
So for example, a device driver written in C could be compiled into a
single unit of code (with lisp entry points and using system lisp
objects externally), addressing its C buffers and data structure
internally in a single "C" image object.
Perhaps we should use the word "domain" instead of "environment", since
"environment" already means something in CL, and "domain" is used in
some capability OSes to denote something similar? (But there are also
some differences, we'll see).
Too bad Jonathan Shapiro has been hired during a few years by Microsoft
so he didn't work much on coyotos, and he's somewhat sidetracked
designing BitC, his system programming language. He hasn't made much
progress on coyotos.
In any case:
1- it looks like it could work,
2- the crucial part is the design of the environments and the bindings we
put in it, along with the system compiler support for them,
3- this could be added to a CL implementation to explore without needing
the rest of the system (persistance, capability access control,
etc). Actually with techniques like IBCL (shadowing packages) and
some reader macro hackery, a prototype could be done without even
having to hack the entrails of a CL compiler.
4- apart from those OS level developments, CL applications would still
have their familiar CL environment, so there would be minimal work to
"port" them; we can provide them a posixy virtual file system and the
needed features to make them happy.
I would have to re-read the EROS and coyotos papers, (and perhaps some
other capability OS stuff), to make a summary of the relationships
between code units, data objects, environments and capabilities. And
we could hack a CL implementation to add those environments and see what
comes out of it.
Unfortunately, since I'm currently salaried, I'm left with only at most
a couple of hours a week to consacre to this project.
Basically, the idea of my LEP project was just to gather existing
lisp applications and lisp servers in a system, and see what a purely
CL environment would be able to archieve nowadays; not much I expect, but
there is some code already in a lot of application domains, so gathered
in a single place, that could be a critical mass to attract some
interest? But I'm also worried that starting with an architecture too
similar to unix systems, the user experience wouldn't be that different,
just a little more inconvenient.
--
__Pascal Bourguignon__
http://www.informatimago.com/
From robert.strandh@gmail.com Mon Oct 21 11:07:01 2013
Return-Path: <robert.strandh@gmail.com>
X-Original-To: pjb@informatimago.com
Delivered-To: pjb@informatimago.com
Received: from mail-oa0-f52.google.com (mail-oa0-f52.google.com [209.85.219.52])
(using TLSv1 with cipher RC4-SHA (128/128 bits))
(No client certificate requested)
by voyager.informatimago.com (Postfix) with ESMTPS id 3669D43C208
for <pjb@informatimago.com>; Mon, 21 Oct 2013 11:06:50 +0200 (CEST)
Received: by mail-oa0-f52.google.com with SMTP id n10so2712519oag.11
for <pjb@informatimago.com>; Mon, 21 Oct 2013 02:06:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:date:message-id:subject:from:to:content-type;
bh=Z0uE7sRRhCmQx9NQtZEVm8YVrLIKr05M8wA86NcUWTw=;
b=W//s++5zdKZI71TuHYu+SZ5UDSnzDGPnXiD0DhBIYdNH3Rjbomc94qigwc8nk829tg
hKIoAfBzE+wMWp8HgPdaE/tN8UkaYTVOlWtazkfZ2XDgkmHP/UNHNlqv1JEkU38SY7mu
wIVPHsQC9p3m+4pPmo0+Sc8CTbLeTDIprJ6uqIBtVOe1CfA6utPPoMFp4svnHaVgKjKn
oHaxCrv24Bbtc12Rf7eRthqwSLZ8GFeQh4B+HnqNBCt52nXoptH00fMTK9kVE1WQi/52
+wvYVp05eECpTZzwLZj1aQBFpMighgbI8F9gFKnQi/REV0/8gD3CKCqxhO5d33/cVahF
2UIg==
MIME-Version: 1.0
X-Received: by 10.182.24.105 with SMTP id t9mr8463obf.86.1382346407199; Mon,
21 Oct 2013 02:06:47 -0700 (PDT)
Received: by 10.182.76.38 with HTTP; Mon, 21 Oct 2013 02:06:47 -0700 (PDT)
Date: Mon, 21 Oct 2013 11:06:47 +0200
Message-ID: <CADUeJY7rGGhCm4ba3RAeCK7vtqZSiC=jwJtz3ZBTs3=fPthn3A@mail.gmail.com>
Subject: Initial version of a LispOS web page.
From: Robert Strandh <robert.strandh@gmail.com>
To: pjb@informatimago.com
Content-Type: text/plain; charset=UTF-8
Hi,
This is what I wrote this morning.
http://metamodular.com/Common-Lisp/lispos.html
Let me know what you think.
Take care,
--
Robert Strandh
From robert.strandh@gmail.com Sun Oct 27 05:25:24 2013
Return-Path: <robert.strandh@gmail.com>
X-Original-To: pjb@informatimago.com
Delivered-To: pjb@informatimago.com
Received: from mail-pb0-f42.google.com (mail-pb0-f42.google.com [209.85.160.42])
(using TLSv1 with cipher RC4-SHA (128/128 bits))
(No client certificate requested)
by voyager.informatimago.com (Postfix) with ESMTPS id A8B7BE90099
for <pjb@informatimago.com>; Sun, 27 Oct 2013 05:25:12 +0100 (CET)
Received: by mail-pb0-f42.google.com with SMTP id jt11so5097332pbb.1
for <pjb@informatimago.com>; Sat, 26 Oct 2013 21:25:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=mBSfHO9FYJuWBbxjWk59sHIRtSZ2IKpK8rslOmP8X0c=;
b=XsC1jLmnrL10vbDBqIdcf5xyAKXfAvFH17OO05RodkpJ2VIcil7hcH2FLsA9vng3Op
7NNPZnaKO/Kw1s6DJg2j+E9/0xdIvxiDLo22NQxBMYq0Vvqz7nyjLkmEcs8OZyEED8ys
wsqutcoZvO/7UmddoxREPo4zc/CwdJrnyJUqy5Vv5D6q+YUIGPcMWVUYPmpB0aAZWNT8
2yKymgHGE2+OlJRatnpHF4ftzdPfRNbJFoCURSeNHYde6BteeBvdtlGQSbnblYjvsY8m
mg20Ca+ZdZcNjCQp9hZrJ5qA95e7Z4rCIGOw15jMhlfirGyc6DXVPNbX/1OHC26PUSoy
Rvxg==
MIME-Version: 1.0
X-Received: by 10.68.35.229 with SMTP id l5mr243552pbj.134.1382847908470; Sat,
26 Oct 2013 21:25:08 -0700 (PDT)
Received: by 10.70.66.37 with HTTP; Sat, 26 Oct 2013 21:25:08 -0700 (PDT)
In-Reply-To: <87hac7fww7.fsf@informatimago.com>
References: <CADUeJY7rGGhCm4ba3RAeCK7vtqZSiC=jwJtz3ZBTs3=fPthn3A@mail.gmail.com>
<87hac7fww7.fsf@informatimago.com>
Date: Sun, 27 Oct 2013 05:25:08 +0100
Message-ID: <CADUeJY75Pp-tJRPjgGOZ+bhV1aaHM3PjVS+X2cDkBY8MLQidmw@mail.gmail.com>
Subject: Re: Initial version of a LispOS web page.
From: Robert Strandh <robert.strandh@gmail.com>
To: "Pascal J. Bourguignon" <pjb@informatimago.com>
Content-Type: text/plain; charset=UTF-8
Hello,
> There's another feature of unix processes: they isolate the user code
> from the system code, notably in terms of user/supervisor modes, and
> memory protection. (Even if as an optimization a unix process can run in
> user mode, from kernel code mapped in the user address space, it's code
> that is mapped read-only and entirely under control of the kernel).
Good point.
I'll see what I can add to the page to address this issue.
Take care,
--
Robert Strandh
From robert.strandh@gmail.com Sun Oct 27 06:28:27 2013
Return-Path: <robert.strandh@gmail.com>
X-Original-To: pjb@informatimago.com
Delivered-To: pjb@informatimago.com
Received: from mail-oa0-f41.google.com (mail-oa0-f41.google.com [209.85.219.41])
(using TLSv1 with cipher RC4-SHA (128/128 bits))
(No client certificate requested)
by voyager.informatimago.com (Postfix) with ESMTPS id 5490C43C1C8
for <pjb@informatimago.com>; Sun, 27 Oct 2013 06:28:08 +0100 (CET)
Received: by mail-oa0-f41.google.com with SMTP id o9so2455964oag.0
for <pjb@informatimago.com>; Sat, 26 Oct 2013 22:28:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=o4oAT8I7OqYc/lYawOx+4EWesGlYu5qrmorhyXsGR88=;
b=LK9SyMfD1ghkWwXRxCOkFpJZA4vR5NqyCvt9GSYDdqEZVONXqrFPSdb7o9Ko6WdxIO
cuHNUvPLZl6HWlFHIYZ03bWsWtkqYJ4sKoo1rvM0r7w6E5OB7PPtPcFC9FvBHJ2iv11k
B7bbSmua2vxhWED5pEtzCycMl9qTf0Cd8Dut9GTYFgJGH9MnB4RPmrLY0cMsyfnzoT7P
2v7oMmDB4Bu+CiHZfoozt6kVYi00zG3Ixm7Il72VmJ+bad8IS2ka7wx9KWSskfcWc/WZ
fYfR32On153ggu3SVvy4xyWSqWUv9ZdAkqyNRlF98aE74FXT7FgEr4iHCoh1u+zpkIXb
mfcg==
MIME-Version: 1.0
X-Received: by 10.182.48.130 with SMTP id l2mr9614696obn.44.1382851684759;
Sat, 26 Oct 2013 22:28:04 -0700 (PDT)
Received: by 10.182.76.38 with HTTP; Sat, 26 Oct 2013 22:28:04 -0700 (PDT)
In-Reply-To: <87hac7fww7.fsf@informatimago.com>
References: <CADUeJY7rGGhCm4ba3RAeCK7vtqZSiC=jwJtz3ZBTs3=fPthn3A@mail.gmail.com>
<87hac7fww7.fsf@informatimago.com>
Date: Sun, 27 Oct 2013 06:28:04 +0100
Message-ID: <CADUeJY5RD7a9_hBqqVdvZHra1NHmUoV-A0E0zxEA8sXE57TotA@mail.gmail.com>
Subject: Re: Initial version of a LispOS web page.
From: Robert Strandh <robert.strandh@gmail.com>
To: "Pascal J. Bourguignon" <pjb@informatimago.com>
Content-Type: text/plain; charset=UTF-8
Hello again,
> Concerning the hierarchical file system, the proposed alternative to use
> keywords to classify automatically the items goes too far in the other
> direction: you have to remember the keywords!
>
> They don't often show the list of keywords that have been used so far,
> and neither do they propose easy ways to navigate them.
I did not mean to say that the classification was going to be made
automatically in all cases. SOME things can be automatic, like date,
subject, etc. But most keywords will be carefully chosen by the user,
just as the user carefully chooses directory names and file names
in a traditional operating system.
I imagine that at any point in time, a user has a set of keywords that
can be used to classify items. The set can be modified of course, but
it is important that keywords are chosen from this set so that an object
won't be eternally lost in case the user makes a simple typo.
Now, I haven't decided the exact nature of these keywords. Sometimes
I think they are just Lisp keywords. Sometimes I think they can be
arbitrary objects. Sometimes I think they could be optionally related by
some inheritance-like mechanism.
Let me give this some more thought so that I can come up with plausible
examples. I think such examples would clarify things a bit.
Take care,
--
Robert Strandh