Scheme in Python

The Scheme in Python series has been completed! Since there are still a few readers of this blog and I haven’t yet setup domain forwarding, I’ll post the links page here.


The Scheme in Python project is a port of the Scheme in Scheme project. The goal is to implement a small subset of Scheme as an interpreter written in Python.

There are a number of goals for this project. First, implementing Scheme in Scheme allowed us to “cheat” a bit by having access to the Scheme reader and data structures. Using Python as the implementation language will force us to code the reader by hand and create new data structures where there isn’t a one-to-one mapping from Scheme to Python.

There are also two auxiliary goals to this project. Using Python should make this more accessible to programmers who are interested in language development, but are unfamiliar with Scheme. Also I’m using this project as a way to familiarize myself with branching and merging in git, so each post will correspond to a branch in the repository.

All the code for this project will be hosted on GitHub. The code is licensed under a BSD license if you are interested in forking it for any reason.

This series will focus on building a very simple interpreter for the purpose of learning the steps involved in building one. For this reason there will be no/very little error checking or optimization. This port will be slightly more complicated than Scheme in Scheme so if you are interested in an even simpler interpreter look here.

Part 0 | Reading from stdin

Part 1 | Numbers

Part 2 | Extending the read layer

Part 3 | Pairs and linked lists

Part 4 | Self-evaluating values

Part 5 | Assignment and define

Part 6 | scheme-syntax macro

Part 7 | Refactor and load

Part 8 | Primitive procedures and apply

Part 9 | Environments

Part 10 | lambda

* Most implementations of Scheme in Python use regular expressions in the reader. I chose to write a parser by hand so I could explain some of the details of parsing. As this is an educational exercise I think this is appropriate.

Other resources for writing a Scheme in Python

lis.py and lispy.py
Simple Schemes written by Peter Norvig.

Psyche
pyscheme
I haven’t got the chance to look at Psyche or pyscheme, but you may be interested in them as well.

Other resources for writing a Scheme in Scheme or other languages

Structure and Interpretation of Computer Programs
Chapter 4 onward covers designing and implementing a number of interpreters and was the inspiration for this interpreter.

An Incremental Approach to Compiler Construction (PDF)
Great paper on building a Scheme compiler from the ground up. Each step is simple and results in a fully working compiler.

Scheme from Scratch
The blog series that inspired and guided the development of the original Lispy. Even if you don’t know C (I didn’t at the time) you will still be able to follow along and construct your own Scheme. Peter’s coding style is easy and pleasant to read and he mentions tips for going in different directions for your own implementations.

A Self-Hosting Evaluator using HOAS (PDF)
An interesting implementation of Scheme using a Higher-Order Abstract Syntax representation. This paper, An Incremental Approach to Compiler Construction and SICP were the primary motivating forces behind my interest in PL design and implementation. The author, Eli Barzilay, has many other interesting papers at his site.

Chai – Math ∩ Programming
A series detailing the development of Chai (what appears to be a Scheme-like language). It is well written and currently in development. I’ll post more information when it’s available.

Scheme in Scheme
Another series that is just beginning about writing a bytecode interpreter. It appears to be put on hold as of April 2011.

Lisp in Scheme
An implementation of McCarthy’s original Lisp in Scheme.

Offline

Lisp in Small Pieces
Great book. Contains code for 11 interpreters and 2 compilers. Source code from the book available here.

Koding in Kate

I’ve recently been preparing to begin looking for my first programming job. As such I’ve been reading a number of “what employers are looking for” type posts. A while ago I read a comment that one interviewer asks candidates what IDE(ditor) they use and uses that information, along with some other things I consider just as trivial to judge candidates. Remembering that post (I think it was in comments on a reddit submission, but I’m not sure) led me to write this one.

I use Kate and I’m proud of it.

Yes Kate, the KDE Advanced Text Editor. Kate is more than a text editor but not quite an IDE, yet is very versatile. From what I’ve read Kate is not used very much by developers, I personally love it and here’s a few reasons why…

Kate works with just about anything I’ve thrown at it:

I do a decent amount of Scheme development and Kate does a surprisingly good job at both highlighting and indenting Scheme. Parenthesis are colored at all times and are highlighted as a matching set when the cursor is over them. It is one of the few editors that I have used that not only works adequately well with Scheme but also has a modern interface and handles other languages very well.

I also do a lot of Python, and Kate has phenomenal support for that. When I did projects in C, Javascript, HTML and even plain text files, Kate supported them well. Kate might not be the best at any one thing, but since it’s usually close to the top, I don’t have to waste any time learning yet another editor/IDE just because I’m doing a project in a new language. Even though it doesn’t take that long, I’d rather focus on developing (and writing blog posts about my editor).

Sessions

Kate has really good support for sessions. You can choose to have your last session open automatically, or use a Kate launcher that lets you choose sessions. Any files you open will be automatically saved to the current session (unless you’re using the anonymous session feature). I rarely “use” sessions in Kate they just do their thing with little interference from me, except for the occasional “Sessions” -> “Quick Open Session”.

Tabs, Documents and the File Manager

Managing the documents in your current session is both easy and customizable. There is a toolbar on the left hand side that has both an open documents manager and a filesystem browser. Both are easy and intuitive to use, but if that’s not enough there are extensions that offer tabs as well. In fact there are two, an older extension “Tab Bar” and a newer and sleeker looking “Tabify”.

Terminal and External Tools

Most of the languages that I use have some sort of REPL. In almost all these cases using the REPL from the command line is not only easy, but is “the way to do it”. Kate offers an embedded terminal that allows quick testing of code snippets in a REPL*. In addition the terminal can be set to auto-sync with the current document, which comes in handy in a number of situations.

With External Tools, Kate can be set up to compile/execute your files just about any way you see fit. External tools can be used for lots of other scripting tasks too, though I haven’t played with it much and can’t comment further.

Find and Regular Expressions

Kate has a pretty powerful search/replace feature including highlighting of all found terms. It’s not just for the document you have open though, Kate can search through multiple files or the entire hard drive if you want. You can also type your query as a regular expression giving you that much more power to find what you’re looking for.

Code Folding, Bookmarking and Split Views

Common IDE features such as code folding and bookmarking are also present in Kate. You can also split the view pane of the editor, horizontally or vertically, allowing multiple documents to be viewed and edited at once.

Shortcuts and VI mode

Just about every command in Kate can be set as a keyboard shortcut. This makes things like the terminal plugin feel right at home. In addition there is also a VI mode that provides Vi(m) like editing. Though not 100% compatible with VI(m) commands, it is an interesting feature. Unfortunately not one that I have any experience using yet, so you can check here and here for more info.

Word/Code Completion and Auto-Braces

One of the things I really like in Kate is word completion. It’s a bit different from the code completion that is commonly found in IDEs. The word completion in Kate is based off other words that are present in the file. This means that you won’t be able to do someObject.met and see all the attributes/methods of that object that start with “met”. However you can use auto-completion when you’re writing anything from C to Scheme to HTML to README files.

Code Tools

Kate also comes with a number of tools for working with code. A comment/uncomment command that works with just about any language and style of comment. Indent/Dedent, support for Unix, Windows and Mac newlines and tons of character encodings, word wrap and spell checking are available and that just barely scratches the surface.

Plugins, Extensions and Scripting

Over 20 plugins and extensions are available (some for KatePart and some for Kate) that add more functionality. Plus you can write KatePart scripts in JavaScript that can add tons of features or just implement a quick hack to get things done.

 

There are many good things that coding in Kate brings.
These are but a few of my favorite things!

 

Links

The Kate Editor Homepage

Using Kate As a Web Editor

Scheme Macros, syntax-rules, Explicit Renaming and Links to More Insanity

This is a response to a question on Yahoo Answers about macros in Scheme. Apparently this was too long for YA, so I decided to post it here. Eventually I’ll rewrite this and make it more coherent, for now please bear with me. Note that there may be some errors here so please point them out if you find them. I am not an expert macrologist

Here’s the original question

I’m new to scheme/racket macros and here is a simplified version of the problem I’m having:

> (define-syntax let-five
(syntax-rules ()
[(let-five expr …)
(let ((five 5)) expr …)]))
> (let-five (+ five 2))

This should return 7 but instead I get an error saying that the identifier “five” is unbound.
All I want let-five to do is to bind the identifier “five” to the number 5 so that I can use “five” within let-five. The reason I want to do this is more complicated than this example so don’t tell me that this macro is doesn’t do anything useful.
I am fairly sure that I know what is wrong, but my question is what would be the correct way to write this code so that it does what I intend?
Additional Details
Jack Trades, I don’t understand what x, r, and c are and what the % and @ are. It would be nice if you could explain that.

And here’s my response…

I don’t use Racket, but Chicken gives the same error (but probably a different error message).

The problem here is that syntax-rules is a hygienic macro transformer. Which basically means that the symbol five is being renamed by syntax-rules to avoid name clashes during expansion. You can see this clearly by looking at the error message (like I said they’re probably different from yours)…

    <syntax>          (##core#let ((five119 5)) (+ five 2))

As you can see syntax-rules renames five to five119 (or something completely different). So when you try to use five it is unbound.

In Chicken I would use explicit renaming macros which preserve hygiene but allow you to break it where necessary. I’m not sure if Racket has this option (check into an implementation of syntax-case, that will allow you to do something similar).

(define-syntax (let-five x r c)
  (let ((%let (r 'let)))
    `(,%let ((five 5)) ,@(cdr x))))

(let-five (+ five 2))
;===> 7

***** EDIT *****

As I already said I don’t think Racket has explicit renaming macros so this probably won’t work for you in that case. You need to look into an implementation of syntax-case for Racket. This will provide you with the ability to do what you want. To the best of my knowledge this is not possible with syntax-rules.

That being said here’s the explanation you asked for…

x, r and c are parameter names the same as if you defined a function (lambda (x r c) …).

The x parameter is the expression, which would be (let-five (+ five 2)) in your example.

The r parameter is a renaming procedure. This is the “explicit renaming” part of the explicit renaming macro. To maintain hygiene you must use this procedure to rename every symbol that you use (except for the ones you don’t want renamed). You do this by calling (r ‘let) for example.

The c parameter is a comparison function that can compare renamed symbols for equality.

You do not provide any of those parameters. Rather the implementation of ER macros does that for you. All you need to do is provide a function that accepts 3 parameters. Another equivalent way to write the ER macro above would be this…

(define-syntax let-five
  (lambda (expression rename compare)
    (let ((%let (r 'let)))
      `(,%let ((five 5)) ,@(cdr x))))

The % sign is just an ordinary character no different from the #\l #\e or #\t. The reason I use that is to remind the reader that “%let” is the renamed symbol “let”. You could use (renamed-let (r ‘let)) if you want instead of (%let (r ‘let)).

The @ is the syntax for unquote-splicing. If you look at the last line in the ER macro above you’ll notice three forms of symbols (` , @).

` the backquote is syntax for (quasiquote (expression)). quasiquote is similar to the regular quote, except that instead of simply returning the whole expression unevaluated, it allows you to evaluate some things using the (,) syntax.

, the comma is syntax for (unquote expression) this tells Scheme to evaluate the expression that immediately follows it. So if we had the list `(1 (+ 1 1) ,(+ 1 2)) when it is evaluated it would return the list (1 (+ 1 1) 3). The expression (+ 1 1) was not evaluated because it was never unquoted, however the expression (+ 1 2) was unquoted and so it was evaluated and returned 3.

@ is syntax for splicing which is a way to evaluate an expression that returns a list and then put that list into the quasiquoted list. In other words it weaves a list into the quasiquoted list.

Here is some more information on Scheme macro systems. I read each of these articles at least a dozen times and eventually it started to come together. Even now though, I’d say that I only really understand about 10-20% of it.

Scheme macros are difficult to understand, but coming to that understanding (even if it is limited) is incredibly rewarding. It will definitely make you a better Scheme programmer and will probably help you to realize why the vast majority of other languages are simply inadequate.

This is the easiest read and the highest level overview.
A Scheme Syntax Rules Primer

This is probably the most complete approach to macros from the relatively simple to the complex. Read this until you don’t understand what it’s talking about, then play around with some of the things you learned. Then read it again from the beginning until you don’t know what it’s talking about, then play around. Then read it again… Eventually you’ll make it at least half way through and by then you’ll know more about macros than you ever thought possible.
define-syntax Primer

Here’s an interesting (and long) post on Chicken’s macro system. This covers ER macros.
Macro Systems and Chicken (long)

And finally here’s an advanced discussion of macros that may just leave you without hair.
An Advanced Syntax-Rules Primer for the Mildly Insane