New Features in upcoming Neko 1.1

posted on 2005-10-29

A few months ago, Neko 1.0 was released. It was a good starting point : a fully specified language, a compiler for it (written in OCaml), a virtual machine to run the bytecode, and a C FFI API to extend the language and write libraries.

Since then, a lot have been done. Here’s some new things in Neko 1.1 which will be released soon :

NekoML : one of the most big feature is that Neko does not need OCaml anymore. A new language have been added, which is similar to OCaml but have a different syntax. It can also generate Neko sourcecode. I started by writing a NekoML “compiler” (parser + typer + neko generator) in OCaml and when that was done, I rewrote first the Neko compiler from OCaml to NekoML and then the NekoML compiler from OCaml to NekoML. Now that both compilers are written in NekoML, the Neko language is bootstrapped.

NekoML is good tool for writing additional languages for Neko, since it includes a Lexer library (which can define Lexers at runtime, not using a separate compilation phase such as Flex or Ocamllex), a Parser library (directly integrate LL(k) Ocaml stream parsers in the language), and a ML type system with pattern matching which is quite a good thing to write a compiler.

The NekoML compiler is trying to generate efficient Neko code, for example in pattern matching to goal is to reduce the number of tests. It was needed to add several features to Neko in order to make it work efficiently, which can then be used for other languages :

  • gotos and labels, in order to do non-local jumps, have been added to the language specification.
  • switchs are a way to branch on a given value. If all switch cases are small constant integers, then the Neko compiler will directly generate a dispatch table so only one jump will be done.
  • a new builtin have been added : $apply, which enable partial application. In the case the number of parameters needed is reached, the function will be called. This is useful for currying.

Apart from theses few addins, the Neko language was enough powerful to express all the NekoML data structures and mechanisms, which is good news. NekoML comes with several additions compared to OCaml. For exemple each type instance carry its own printer to you can actually “print” anything and it will display the tagged constructors names. Useful, very useful.

After I was done with bootstrapping, the next goal was to improve “NekoML compiler compiling itself” speed. The start was quite high, with more than 3 minutes and half needed to recompile the compilers (NekoML + Neko). After some perfs improvements, GC corrections, better Neko code generation and VM optimization, the time is now less than 20 seconds on my box which is reasonable (remember that the compiler is running on dynamicly typed-safe VM).

Other improvments were added to the VM, such as callstack traces and exception traces like in Java so it’s a lot more easy to debug a complex program.

Flexible Specs

posted on 2005-10-23

The new language I was talking about yesterday is starting to get a specification. Read it if you’re interested…

NekoML / OFLA / MTASC

posted on 2005-10-22

Today was quite a rough day !

  • I finished bootstrapping Neko and NekoML, so I have now Neko and NekoML compilers written in NekoML and compiling themselves.
  • OFLA was a great online conference ! We had more than 350 users login at peak and more than 200 make it till the end of the conference. Hope to have some recordings available soon to link to.
  • I took time then to announce that MTASC will not support ActionScript3 language and will follow another way.

Posted on the OSFlash mailing list

As announced at OFLA today MTASC will unlikely support ActionScript3.
The reason is quite simple. MTASC supporting AS3 + AVM2 will be very
comparable to Macromedia Compiler provided with FlexBuilder2. And only
having the “opensource” difference is not enough.

My proposal is then a new language (name still unknown) that will support
several platform :

  • the Flash platform of course, with in the beginning current FlashVM and later AVM2. The new language will try to make it easy to port your existing AS2 code to it (or at least as much difficult as porting your AS2 to AS3).
  • the Brower platform, by allowing Javascript code generation. So you can write all your DHTML and AJAX *strongly-typed* with this language, and interact seemlessly with Flash.
  • the Server platform, since it will be able to run on the NekoVM, and then you can write web pages and standalone executable using this language. It’s also great since you’ll be able to have to communicate with the same-language from client to server, and you can access all the NekoVM libraries (sockets, databases, file system, ….)

So it’s really one language, which syntax will be near AS2 and Java but
still different and more flexible (more powerful also) and which evolution
will be driven by the community. By you.

NekoML done !

posted on 2005-10-15

This week I was quite busy working on NekoML. NekoML is a programming language with an ML type system using type inference and unification. It is quite similar to OCaml but have the following differences :

  • the syntax is different, and more like C or Java syntax, using curly braces for blocks
  • several hacks have been added, especially for the operation + which can take two integers or two floats or two strings
  • there is a lexer library (based on Alain Frisch ULex) that you can use to define lexers at runtime using pure NekoML syntax (ocaml uses ocamlllex which is a compile-time preprocessor with enhanced syntax)
  • streams and streams parsers are directly in the language (ocaml uses camlp4 preprocessor)
  • all values carry their own printers so you can print any value and it will be displayed nicely using NekoML constructors

And most important point is that NekoML compiler generates Neko source code. So you can use Neko Virtual Machine to run NekoML, or write some libraries in NekoML and some others in MotionTypes or other Neko-supported languages.

So the people that know about OCaml might ask what is the point of having a staticly typed programming language running on top of a dynamicly typed VM ? That’s true that’s not the best choice for speed-oriented applications since you can’t generate very efficient native code such as ocaml, but otoh you can use the dynamic features of the Neko language, which are really useful when you want for example :

  • interact with the outer world : databases, serialization…. since you always have the types at runtime, you can handle things better (without crashes)
  • interact with other languages, since you have a common representation and low-level semantics
  • use dynamic linking in order to load plugins at runtime
  • do “low level dynamicly yped” stuff by directly embedding Neko source in a NekoML program

All this together makes NekoML the language of choice for writing language compilers targeting Neko. Next step is to rewrite the NekoML compiler… in NekoML. Once this is done the language will be fully bootstrapped and after I take some time reorganizing and completing standard libraries I can make a Neko 1.1 release that will not use ocaml anymore.