From the Burrow

Delays in Pushing Code

2014-07-28 09:55:06 +0000

I have been hoilding off pushing the new features to Cepl recently as I have one bug in defglstruct that really is confusing me.

At first it seems very simple, it is complaining that certain functions (A) dont exist with compiling functions (B). Now (A) do exist by the time (B) is called so it doesnt cause any immediate problems, but the compiler wanrings are very annoying.

The weird bit is that (A) do exist at that time. For example if I expand the defglstruct macro and then run the resulting code it works with no warnings.

There is something I’m just missing but it is the reason I havent merged into master yet.

I may give this some more attention tonight.


Buffer Backed Textures

2014-07-28 09:26:48 +0000

Last night I was musing over rendering graphs into textures and I was interested in how to write just a couple of pixels to the texture without reuploading the image. Now I can use gl’s ‘copy sub image’ functions but looking through my copy I saw I hadn’t implemented buffer backed textures; this gave me a great excuse to finally get it done!

In cepl buffer backed arrays would more accurately be named array-backed-textures as we already have the gpu-array abstraction over buffers. This works in my favour though as currently the abstraction for textures in cepl is as follows:

  • A texture is a structure that holds a number of images. You use texref (like aref) to get at one of the images.
  • Images are just arrays so they are exposed as texture backed gpu-arrays.

So calling texref on a regular texture gives you a texture-backed gpu-array and calling texref on a buffer-backed texture give you a buffer-backed gpu-array.

So to get change one pixel of a texture to the color red we can do this:

(with-gpu-array-as-c-array ((texref texture) :tempname arr)
  (setf (aref-c arr 10 10) (v! 1 0 0)))

I have been coding a lot of c++, java and c# recently and so coming back to lisp and doing things like the above just makes me so damn happy! I love this language.

Varjo and Multiple Value Return

2014-07-24 15:46:00 +0000

Hi folks, having recently got married (yay!) and got a flat (another yay!) I have been getting back into lisp coding again.

The latest addition to the Varjo compiler is support for ‘values’ and ‘multiple-value-bind’. Now as glsl doesn’t have support for multiple value returns we have to use something else. The mechanism I have now works something like this:

  • If you have a ‘values’ form within a multiple-value-bind form the multiple-value-bind form turns into a ‘let’ and the ‘values’ becomes ‘setf’s to the variables in the let.

  • If you have a values form and it isnt within a multiple-value-bind then one of the following happens ** If the values form can propagate to a ‘return’ form then the function is given out-vars and they are set from the place where the values form was originally. ** If it cant reach a return then it collapses to a progn.

If a varjo function with out-vars is used within a multiple-value-bind then the multiple-value-bind becomes a let statement and the first var is set by the function return and the rest are passed as out-vars to the function.

If the function is used outside of a multiple-value-bind then the form is surrounded in a ‘let’ as the outvars still need to be set.

This results is something like the following.

VARJO> (defshader test ()
         (labels ((thing ((x :int)) (return (values x (* x 2)))))
           (multiple-value-bind (x y) (thing 1)
             (+ x y)))))

;; Gets compiled to

#version 330

int thing_26v(int x_23v, out int return1);

int thing_26v(int x_23v, out int return1) {
    return0 = x_23v;
    return1 = (x_23v * 2);
    return return0;

void main() {
    int a_27v0;
    int a_27v1;
    a_27v0 = thing_26v(1,a_27v1);
    (a_27v0 + a_27v1);

I still have a few bugs to iron out but it is very nearly there. One step closer to lisp!

Eclipse project overlaps error

2014-05-05 14:02:42 +0000

I tried adding an existing android project today and got this error from Eclipse.

“<name of project> overlaps the location of another project..”

It was reported as a project desciption problem. Turns out eclipse doesnt like importing projects from the direcotry it has designated as the workspace. Move the project outside that folder (or change the workspace directory) and try again.

What a pain in the arse!

Android Ndk Build Error Multiple Targets

2014-05-05 13:57:56 +0000

Had this error today:

* multiple target patterns. Stop.

(Turns out I needed to delete the obj file from the android project)[]

This post is mainly to help me remember this but hopefully it helps someone else too.

Cepl on OSX

2014-04-15 17:29:22 +0000

This has been a massive pain in the arse and still isn’t working in the way I would like it but there is some progress so it is worth testing.

First off, while I love sbcl, I cannot get sdl2 to create a fully working window. I get a window with no border or one which does not receive input events.

So instead I tried clozure (ccl) and have had some success.

First cepl:repl now wraps it’s contents in an sdl2:in-main-thread form. So you can continue to use this as usual.

Next you need to wrap the code that contains your main loop in an ‘in-main-thread’ form. In the case of the provided examples you can just write (sdl2:in-main-thread () (run-demo)).

This gives you an active window that is decorated and accepts events which is wonderful. The only problem so far is that #’update-swank doesn’t work and so you cant use slime’s repl while the demo is running. This is a deal breaker for me, as that ruins my workflow so I need as solution to consider using osx for any cepl development.

Hopefully a solution to this will emerge soon. I then really need to go clean up the code surrounding this fix.

Here’s hoping

[edit] Seems the repl in the inferior-lisp buffer (or shell) is active and usable. However the repl buffer doesnt yet. Can anyone with more experience with ccl help me out here?

Fps Counter with Temporal Functions

2014-04-14 00:00:46 +0000

(let ((count 0))
  (tdefun fps ()
    (incf count)
    ((each (seconds 1)) (print count) (setf count 0))))

It’s simple but it makes me happy simply because it is written in term of time.

p.s. Oh yeah, as you may have guessed from this, temporal functions work now!

Followup on the Temporal Functions

2014-04-12 04:04:04 +0000

Just thought I’d warn anyome that cares that the current implementation is terrible!

It was good enough to prove a point to myself but I am currently cleaning up the code and fixing the bugs. The result is quite nice but it is bending my mind and it is time for a sleep.


Cepl Foreign Data

2014-04-11 10:21:27 +0000

Up until now cepl has been putting it’s own wrapper around cffi data in order to make it easier to manage and play with in lisp.

I wanted to start supporting more complicated foreign data and so was just about to start writing when I remembered cl-autowrap. If you are a lisp person and you haven’t checked autowrap out yet then please do, it is a fantastic wrapper generator which given this..

(c-include "file.h")

..Will write an entire lisp wrapper around the foreign library. It also provides hooks (via trivial-garbage ) to allow the CL’s garbage collector to manage your foreign data.

A little digging around in the back-end (with some great help from rpav, cheers man!) of autowrap has shown me that there is more than enough data back there for me to be able to write something like this:

(make-available-for-gpu 'some-type)

And then have the type fully integrated into my glsl compiler.

I do need to augment the c-autowrap slightly as I want to be able to recieve cepl’s own c-array objects instead of autowrap’s. But this is certainly do-able and I’ll have a good look at it next week.

Righto folks, I need coffee


Cepl in Time

2014-04-04 18:23:46 +0000

This post is about time and my ongoing experiments of trying to find a way of representing it in cepl. If you aren’t interested in the journey but are interested in the solution then jump down to the ‘So what’s the result?’ section.

###Why Bother?

It would be very valid to raise the fact that for real-time applications there are some very well-know and suitable ways of handling time (e.g. fixed timestep ) so why wouldn’t we use these techniques?

Often when coding interactively you just want to run something in the repl in pure isolation. These kind of experiments can give rise to new techniques and allow you to toy with idea more easily as you don’t have to worry about any other part of your program. Lisp support this style of programming by having axiomatized some of the core concepts of the language[1] and I wanted that for time as well. I didn’t want to have to take some time delta from another part of my program in order to use it in some function or concept I was developing, I want to focus purely on the task at hand.

###What should time look like?

So after I had decided I wanted some kind of primitive for time in cepl I needed to decide what this would look like. As usual I stole from lisp. Cons cells encapsulate the most basic nature of a list: It is some ‘thing’ followed by some other ‘thing’. So what is time?

Well first off I decided not to look at it from a physics point of view, as this would be unnecessarily complicated, and stick to how time seems to be. Well time is continuous, it doesn’t seem to ‘arrive’ in lumps… but computers by their nature are incremental. So if time is flowing and we treat it like a stream of water, we want to capture time in buckets and then use the captured time to drive some function. OK, that’s weird but time-buffers are kind of what fixed-time-steps implement so lets have ‘time-buffers’ as a concept.

Next…where does the time come from? Well in computing we are used to having streams of data coming from some data source so let’s have a time-source. It also makes sense that we might want to speed up and slow down time so let’s have a time-source be able to be based on another time source with a modifier. We could use this to make a time source that is running 10 times slower than real time. Now if we allow passing of time sources to our time buffers we have a side effect free way of implementing slow motion…even if this only temporary it could be useful for observing some effect slowly to see it is behaving correctly.

When we speak about time we often use some condition to qualify when we are talking about. For example:

  • BEFORE tomorrow do x
  • BETWEEN 10am and 2pm do x
  • AFTER 10 minutes from now do x

These predicates should take a time an return t or nil.

The ‘AFTER’ example is interesting as it includes some syntax for talking about time relative to another point in time. So we will need some kind of ‘from-now’ function.

###The weird bit…

OK so this next part of the process was not reasoned about or based on logic. I was writing down the above and trying to work out other time predicates when I heard a voice in my head say ‘LEXICAL TIME’. This was an odd thing to hear as I had no idea what my brain was talking I stopped for a bit the more I thought about it the more I started to like the idea, we have a temporal-function so you define functionality that implicitly understood time.

I thought it could be some macro that wrapped up the elements we had already created and let the user write code knowing that time was being taken care of.

;;this was the first pseudo-code of a temporal-lambda
(tlambda (x) (within 0 5000) (print x))

###The problems

As usual ideas burst into flame on contact with reality. Here are the major points that arose.


The first problem I noticed was in a tlambda that uses the ‘before’ predicate e.g:

(tlambda () ((before (from-now (seconds 10))) (print "hi")))

Notice that after 10 seconds from now this function will never run again. It essentially has expired. Now I could return some kind of value but this sucks as I was these temporal functions to be identical to regular functions. A friend then recommended using conditions, which turned out to be perfect. So we now have an ‘expired’ condition that can be caught by time aware code.

But when we look at this again we notice that really expired isnt limited to time…it is part of a more general set of conditional-functions. Which crudely stated as a macro could be.

(defmacro defcfun (name args condition &body body)
  `(defun ,name args
     (if ,condition
         (progn ,@body)

So temporal functions are a type of conditional function. I’m not sure how useful this is…but it was interesting to me.

####Composition and Syntax

The next big issue was in composition and what the resulting code looked like. I was starting to get something working with functions and closures but I started running into a problem as I wanted a predicate that returned t once every x seconds/milliseconds.. something like:

(tlambda () ((each (second 2)) (print "hi"))

But what we actually need is to create a closure with a stepper in the closed vars.

(let ((stepper (make-stepper (seconds 2))))
   (tlambda () (when (funcall stepper) (print "Hi")))

Given that tlambdas were meant to implicitly understand time this seemed ugly.

This was compounded by the next problem…

####Time overflow

Let us define an imaginary tlambda with a nice syntax:

(tlambda ()
  (then ((before (from-now (seconds 5))) (print "hi 1"))
        ((before (from-now (seconds 5))) (print "hi 2"))
        ((before (from-now (seconds 5))) (print "hi 3"))))

What this is meant to do is:

  • run the first before clause until it expires and then
  • run the next clause until expires and then
  • run the third clause until it expires and the signal that the whole tlambda has expired.

Now let’s imagine that we create this tlambda but wait 12 seconds before calling it. The first time we call it should now recognize that the first two clauses have already expired and jump straight to 2 seconds into the third clause.

At this point I could not think of a way of doing this that didnt invlove a DSL. This became even more apparent when I thought about optimizing as I can’t inline the stepper closures and inside the tlambda there would be A LOT of signals being fired and caught.

I was quite annoyed that my knowledge of lisp was potentially limiting me here but at least I could have a go at making a macro version which could be more optimized (if perhaps not as functional as I would originally have liked)

####So what’s the result?

Well the result is a very nice DSL which creates lambdas that know time and can mix temporal syntax with default lisp syntax very easily.

Here are a few examples:

;; when you use default lisp code, the result is a regualr lambda
(tlambda () (print "Hi"))

;; inline lambdas are fine too, again this is just a regular
;; lambda
(tlambda ()
  (let ((text "hello"))
    ((lambda (x) (print x)) text)))

;; This function will for the first 10 seconds print "hi jim" each
;; time it is called. After 10 seconds it will print "jim" each
;; time it is called.
;; Temporal forms are conditional, the predicate must be true for
;; the rest of the form to be evaluated. For this reason we borrow
;; the syntax from the cond macro.
;; Also notice that unlike cond every line is evaluated. So the
;; (print "jim") line always works. This also means this function
;; never actually expires. regular lisp forms can be seen as
;; eternal.
(tlambda ()
  ((before (from-now (seconds 10))) (princ "hi "))
  (print "jim"))

;; time syntax must be either at the root of the tlambda or inside
;; one of the special forms defined for time functions.
;; the following is not valid, as the temporal condition form is
;; within a standard lisp form
(tlambda ()
  (print ((before (from-now (seconds 10))) "hi")))

;; tlambda introduces 2 new special forms: then & repeat
;; they each run each containing form until it expires and
;; only then move to the next form. The difference is that
;; once all the forms have expired 'then' will signal it has
;; expired, whereas repeat will start from the first form again
(tlambda () (then ((before (from-now (seconds 10))) (print "hi"))
                  ((once) (print "allo")))

(tlambda () (repeat ((before (from-now (seconds 10))) (print "hi"))
                    ((once) (print "allo")))

;; progn is also modified so that it can contain the temporal
;; condition forms
(tlambda ()
   (progn ((before (from-now (seconds 2))) (print "hi"))))


Here is an example of what is generated by tlambda (I have tidied it up a little and changed gensym names for ease of reading but functionally it is the same):

;; This:
(tlambda ()
  (then ((before (from-now (seconds 3))) (print "hi"))
        ((before (from-now (seconds 4))) (print "bye"))))

;; expands too:
(let ((counter 0)
      (deadline-1 (from-now (seconds 3)))
      (deadline-2 (from-now (seconds 4))))
  (lambda ()
    (if (= counter 2)
        (when (and (< counter 2))
          (if (afterp deadline-1)
                (when (= counter 0)
                  (incf counter)
                  (setf deadline-2 (- (from-now (seconds 4))
                                      (- (funcall *default-time-source*)
                                         (afterp deadline-1)))))
                (if (afterp deadline-2)
                    (when (= counter 1) (incf counter))
                    (when (beforep deadline-2) (print "bye"))))
              (when (beforep deadline-1)
                (print "hi")))))))

####What is missing?

  • Tests and Bugfixing: There are still bugs in this and need to squash them
  • Generate cleaner code: Currently it works well but could be nicer for human reading
  • Generate type declarations: For speed optimization

Well that is all for now. Thanks for reading. The interesting thing now will be playing with this and seeing if the abstraction is actually useful!


1. The simplest example of this is lists in lisp. Rather than just being given a list type, with it’s implementation hidden away, we get the cons cell. The cons cell is the smallest abstraction of a linked list and with it we can build lists, tree and other graphs and then use the techniques we have for navigating cons cells (car and cdr (and friends)) to managed all of these different data structures. Also see Paul Graham’s ‘The Roots of Lisp’. It’s a short article, but it hows how with 7 basic primitives you can write an interpreter for an entire lisp language.