Friday, October 12, 2007

Getting started with Factor - Easy FFI

When I found Factor, a few months back, I was using Ruby for three years and Erlang for one year. I was looking for a language with a combination of the features of both languages but with a simpler FFI. Don't get me wrong, Ruby and Erlang are great languages with many features I then liked but I was looking for the "perfect" programming language (as we all do) and FFI wasn't up to my expectations with either of these.

What's FFI, by the way, and why is it so important for a real-world programmer. FFI (short for Foreign Function Interface) is a synonym for C extensions or C bindings and gives you access to a wealth of standard, universal, and platform independent libraries such as OpenSSL or OpenLDAP. If you were to write them from scratch in the programming language of your choice it would otherwise cost you a lengthy and uncertain development. Ruby is reputed for its simple C extensions but when I tackled these for the first time, they were far from being as simple as advertised. The same goes for Erlang where writing a C driver is a dreaded experience (I already know about Erlang Port and Erlang Interface but they are slow and limited to bridging your code with a huge switch case statement in C).

Let's get started now. If you've already installed Factor, skip this paragraph. Otherwise download the latest stable release from http://factorcode.org/download.fhtml (0.90 at the time of this writing).

This tutorial focuses on Mac OS X and I'm using a PPC, so I downloaded Factor-0.90-ppc.dmg (the same web page also has binaries for Intel Mac and Windows as well as source downloads for other platforms) and dragged the Factor folder to the default Applications folder. 'Allow user to administer this computer' should be checked. As an alternative you can drag the Factor folder to your Desktop or to the user's own ~/Applications folder.

Drag Factor to your Applications folder



The Applications folder is C:\Program Files on Windows but you can just as well unpack the Factor distribution to your Desktop. Most of the commands will run unchanged on Linux, but you'll have to adapt them on Windows. Mainly for building the .dll since there are many compilers that can do the job (please paste your solution in the comments).

As many Ruby veterans, I'll be using TextMate for the tutorial examples but Factor comes with editor hooks for editpadpro, emacs, gvim, jedit, scite, textmate, and vim. You can't say you don't have the choice. The TextMate Factor Bundle (Factor.tmbundle) is under the misc/ folder.

You can at any point edit a specific word inside Factor with \ word edit. This will pop-up the file containing that word in TextMate with the edit cursor positioned at the beginning of the line where the word is defined. You need to modify line 7 in extra/editors/textmate/textmate.factor replacing "mate -a -l " with whatever the which mate command gives you as a location in your terminal (e.g. "/usr/bin/mate -a -l "), for this to work in Factor version 0.9 (it has been corrected in the upcoming version).

As an example of the above functionality, open the installed Factor folder and start Factor by double-clicking on the Factor icon (the raptor) then try the following in the Input area of the listener:

USE: editors.textmate

\ + edit

The Edit Hook in action



For this tutorial's purposes we're going to build a small C library. Paste the following code into a file called SampleDylib.c:

8<------------------------SampleDylib.c----------------------------------

#include

int addFunction( int a, int b ) {
return a + b;
}

int stringLength( char *str ) {
return strlen(str);
}


8<------------------------------------------------------------------------

Now to compile it into a dynamic library on Mac OS X, run the following three commands in a terminal shell:

(Mac OS X)

cc -fno-common -c SampleDylib.c

cc -dynamiclib -install_name SampleDylib.dylib \
-o SampleDylib.dylib SampleDylib.o

(Linux)

cc -fPIC -c SampleDylib.c

cc -shared SampleDylib.so SampleDylib.o

(Mac OS X, Linux)

sudo mv SampleDylib.dylib /usr/local/lib

The last command puts the dynamic library in the library search path defined on Mac OS X by LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, DYLD_FALLBACK_LIBRARY_PATH ($HOME/lib:/usr/local/lib:/usr/lib), and the process's working directory (this is usually done through make install with standard libraries). Linux has the LD_LIBRARY_PATH environment variable in case you move your .so to some other location than $HOME/lib, /usr/local/lib, or /usr/lib. The DLL would go typically under windows/system32 on Windows but since the DLL search path on Windows looks in the same directory as the executable, the system directory, and directories specified in the PATH environment variable, you can put your .dll anywhere under this path.

When you'll start contributing libraries to Factor these will go into the extra/ folder but for the time being create a new work/ folder under Factor's root folder where you can do your own experimentation. You can find out where Factor searches for vocabularies with vocab-roots get .. This is an example output:

V{ "resource:core" "resource:extra" "resource:work" "." }

where "." is your installed Factor's root folder.

Create a work/ folder



Let's organize the folder structure of our sample vocabulary. Under work/ create a new sample/ folder (mkdir sample) and inside sample/ create a libsample/ folder (mkdir libsample). Under sample/ create two files named sample.factor and sample-tests.factor respectively (touch sample.factor sample-tests.factor). Open the libsample/ folder and create a libsample.factor file (touch libsample.factor). This is how your file/folder structure should now look like:

-Factor/
+core
+extra
+fonts
+misc
+unmaintained
+vm
-work/
-sample/
sample-tests.factor
sample.factor
-libsample/
libsample.factor

Your file/folder structure in TextMate



Now let's start implementing our binding. If any of the words' usage is not clear you can get extensive help with:

\ word help

Here's the code for libsample.factor:

8<-------------------------libsample.factor-------------------------------

USING: alien alien.syntax ;

IN: sample.libsample

: add-sample-library
"libsample" "SampleDylib.dylib" "cdecl" add-library ; parsing

add-sample-library

LIBRARY: libsample

FUNCTION: int addFunction ( int a, int b ) ;

FUNCTION: int stringLength ( char* str ) ;

8<------------------------------------------------------------------------

You could also give the full path of SampleDylib.dylib to the add-library word above but you don't need to since you already moved the library to the Mac OS X's library search path. Later when the libraries you're binding to are located in unusual locations like /opt/local/lib (e.g. MacPorts), you can export their path with LD_LIBRARY_PATH (e.g. export LD_LIBRARY_PATH=/opt/local/lib) in order to avoid having to hard-code the path name (check \ add-library help for more details).

In a typical multi-platform implementation, you'll let Factor detect the OS:

"libsample" {
{ [ win32? ] [ "SampleDylib.dll" "stdcall" ] }
{ [ macosx? ] [ "SampleDylib.dylib" "cdecl" ] }
{ [ unix? ] [ "SampleDylib.so" "cdecl" ] }
} cond add-library

Notice how FUNCTIONs are declared. They reproduce the function prototype in C with simple differences such as char* str instead of char *str. You need to watch for the white spaces after the function name as parentheses and commas "are only syntax sugar" to make the declaration easier to read.

That's all it takes to implement a C binding in Factor!

Now we can add words that use the above FUNCTIONs in sample.factor:

8<---------------------------sample.factor--------------------------------

USING: sample.libsample ;

IN: sample

: add-function ( x y -- z )
addFunction ;

: string-length ( str -- n )
stringLength ;

8<------------------------------------------------------------------------

Let's test our new words in the listener. In the Input area enter:

USE: sample

Factor will find sample.factor since work/sample/ is under the vocabularies path and will compile the words. Now enter:

3 4 add-function .
7
"Factor: a practical stack language" string-length .
34

Calling string-length in the Listener



Ok, it works as expected. To make sure we have no regression with further development, we add some unit tests in sample-tests.factor:

8<-------------------------sample-tests.factor----------------------------

USING: sample tools.test ;
IN: temporary

[ 7 ] [ 3 4 add-function ] unit-test

[ 34 ] [
"Factor: a practical stack language" string-length
] unit-test

8<------------------------------------------------------------------------

You run the unit-tests with:

"sample" test

You must have called either USE: sample or "sample" require beforehand.

Well, that's it for this tutorial. If you have questions or comments, please feel free to post them below.

6 comments:

Anonymous said...

Excellent FFI tutorial. Thanx!

austin said...

Good tutorial; liked it a lot. :) Maybe I can do something interesting with factor now, rather than just trivial things...

pate said...

Not to bash factor, I'm learning it too and really having fun doing so, but rubinius should give hope for ffi on Ruby. See http://redartisan.com/2007/10/11/rubinius-coding for an example.

Anonymous said...

There is an FFI extension for Erlang that allows to achieve the same results that you've obtained with Factor. It is being discussed for inclusion in Erlang. See http://muvara.org/crs4/erlang/ffi

esper said...

I'm getting an error message:
"a parsing word cannot be used in the same file it is defined in."

Elie Chaftari said...

esper,

In Factor 0.92 the add-library code doesn't work anymore as in the above example.

The following post http://factor-language.blogspot.com/2008/01/compiler-overhaul.html on Slava's blog explains, in its "Number three: words defined in a compilation unit cannot be called until after the compilation unit" section, why you're getting "a parsing word cannot be used in the same file it is defined in." error.

Here's an excerpt of that post with an example:

This idiom is used with alien bindings:

: load-foo-library
"foo" "libfoo.so" "cdecl" add-library ; parsing

load-foo-library

LIBRARY: foo
FUNCTION: int blah ( float x ) ;
...

The add-library call has to happen at parse time, before the compilation unit ends and compiles bindings, because compiling bindings looks up symbols in the library. However, now in Factor 0.92 the above code doesn't work because load-foo-library is being called before it is compiled. The workaround is to use parse-time evaluation, which results in clearer code because we're no longer defining a parsing word to only call once:

<< "foo" "libfoo.so" "cdecl" add-library >>

LIBRARY: foo
FUNCTION: int blah ( float x ) ;
...