I’ve just had the joy of stumbling onto this series by Jean Niklas L’orange about clojure persistent datastructure. I had gained a basic understanding of the theory of these before now but this series of posts break it all down really well, including the optimizations that make this datastructure (more) practical.

Understanding Persistent Vector - Part 1 Understanding Persistent Vector - Part 2 Understanding Persistent Vector - Part 3 Understanding Clojure Transients

Not only is fantastically written but Ι believe you could work through this series and write a decent persistent vector implementation by the end of it.

Top notch!

This is just a quick check in to log some progress.

Tonight I’ve been poking around in the current event system, mainly it was cleanup but Ι also added the ability to have weak & strong subscriptions to a event node. The weak ones were neccesary to allow GC to cleanup things that were created at the REPL and are now orphaned. However for well written code that guarentees nodes are unsubscribed, the overhead of weakrefs is undesirable.

Currently nodes can also have two functions applied to them, a filter (which can stop an event propagating) and a body (which is just run with the event as an argument). These were added in a hurry so I need to go back and turn these into something sensible. I guess I will just combine them and have the result of the body be an event or nil, events get propagated, nil doesn’t.

We shall see.

Right, in this post I want to go over what I’m going to be making this month, but first a summary of day 0.

Rewriting backscatter in cepl turned out to be a really good way to find bugs in the compiler, it was depressingly good at finding them actually. The changes have made the struct handling in varjo much better and made the function dependency code a lot better in cepl.

Whilst the compiler is now producing good glsl, the pipeline crashes hard as soon as you use it :p. Oddly enough commenting out a couple of the uniforms being passed to the pipeline stops it crashing, so I’m assuming I have either screwed up the uniform uploading, or it didn’t work before in examples like this.

Either way it is not the primary focus for this month so I am forcing myself to leave it as it is for now and concentrate on nanowrimo.

On that note let’s talk about that :)

The project for this month is to add a repl friendly, gc’able, optimizable way of managing hierarchies of transforms in cepl. If this sounds like a scene-graph then that’s no surprise, but I’m consciously trying to avoid putting game objects in the graph directly. By this Ι mean Ι don’t want to have some ‘game-object’ type that you have to subclass and that then goes in the graph.

What I would prefer is some means of defining spaces and then having objects subscribe to them. This decouples objects from the means of organizing them.

In this system it is normally wise to have some kind of caching system. This pays off when there are multiple spaces that share a parent. For example if you are calculating the position of each finger on a hand you dont need to recompute the position of the hand each time. Another property that melds well with this is that in many cases you can compute the transforms lazily. You dont need to calculate a parent transform unless the child needs it (or it is needed itself).

Slam this all together and you get a kind of event system. One that caches the latest event rather than propagating them and can be resolved by pulling.

So far so good, but there is one was case I’m interested in. It may be necessary to walk many of the spaces. To that end it would be good to have a flush event that would push all the pending events through the system.

The finale: Input events are just events which always flush. This means it may be possible to merge these event systems to produce a unified interface. One which would allow subscribing to any event. This is basically some odd rx hybrid (with less state safety). Care will have to be taken around the boundaries between different style nodes but more safe systems can be built on top of this (or it can be ignored and just used to feed some higher level system)

This may end up not working out but it’s the experiment for the month and we shall see how it goes :)

Time to go watch a bunch of talks on streams/frp/etc to steal ideas. Wooo!

Got him! turns out there was a bug in the code that merged compilation environements. One of those things telling me Ι need to have better tests in this code.

Anyhoo, that’s done. back to business.

Given this gpu struct:

(defstruct-g material ()
  (additive-color :vec3 :accessor additive-color)
  (diffuse-color :vec3 :accessor diffuse-color)
  (specular-color :vec3 :accessor specular-color)
  (specular-exponent :float :accessor specular-exponent)
  (background-ammount :float :accessor background-ammount))

Only the call the #'yy breaks

(varjo::defshader marg ((y material))
  (labels ((xx () (additive-color y))
	   (yy ((x material)) (additive-color x)))
	(additive-color y) ;; this works
	(xx) ;; so does this
	(yy y)) ;; but this doesnt
  (v! 0 0 0 0))

Wat?!

In real news I’m catching a good few bugs in varjo today. Can’t wait to squash this one though :p

First day of November, first day of my first NanoWriMo. Traditionally this is a challenge to write a 50000 word novel in a month but I’m going to be coding along with this instead. As word/line counts are pointless in code I’m going to try do some significant work on cepl.

I’m going to aim to make a space-graph (analogous to a scenegraph) I’ll be writing this up as I go so Ι won’t burden this post with it.

Today however Ι am bug hunting by taking a rather nice shader called backscatter from shadertoy and remaking it in cepl. It was written by a good friend who tells me it’s too slow, which may be fair but it is also clearly written and pretty enough to be motivating.

writing it has already let me find a few bugs, including one in Varjo’s structs so I am off to fix that now :)

Phew, now Fuse has shipped the beta I’m not crunching so much at work so I’m back working on cepl again. (Check out Fuse btw, it’s early days but theres some kickass shit in there, I’ll be doing some posts on it in future)

I’ve not a nice branch full of changes to merge in soon in which I’ve been working on two main things:

1. Better error messages This took a lot more work than just tweaking the strings. I’ve gone though varjo (the lisp->glsl compiler) trying to find places where I can make suggestions of possible correct alternatives. The obvious example for this is variable names so if you type the wrong name varjo will make suggestions based on names that are significantly similar (I’m just using jaro-winkler-distance for this). In cepl it also involved going through the more hairy code, trying to seperate it into more sensible functions and also making sure I give more specific errors especially around the code that handles converting between texture-formats & lisp types.

2. New event system So I changed this yet again :p. The reasons for doing this have mainly been around limiting scope. I dont want unnecesarily high level abstractions in cepl core, I want ‘low enough’ abstractions that are easy to build on.

So this time the main change is that event listeners are remvoed in favour of event-nodes. Event nodes can subscribe to other event nodes to recieve their events. Any event node can also be subscribed to. Each node holds a list of weak refs to the node that are subscribed to it, and a list of strong refs to the nodes that it subscribes to. In this way we have a event graph that the garbage collector can clean up if you don’t unsubscribe explicitly.

“Why add this weak ref complexity” would be a fair question to ask, and the reason is making a balance between good practice and pragmatism. Common Lisp allows awesome levels experimentation with the repl and incremental compiling. It is inevitable that, whilst explicitly disconnecting event-nodes is the correct thing to do, managing that is a PITA when you are just trying to play with ideas. To this end we make sure we can GC away parts of the event graph that are no longer needed, letting you focus on the important stuff. This kind of thing is what I mean by ‘low level enough’, I’m not trying to bake in some glitch free reactive programming framework (though those are awesome), but I still want to make the experience good and lispy.

Less importantly, with the new changes events are now structs with read only fields.

The driving force for the above changes has been that I now have event sources other than SDL2. I have made a simple android app that sends touch events over sockets to cepl to allow more direct interaction with whatever I’m making. It’s early days but the events from the cepl remote-control are now flowing through the event system just like mouse & keyboard events so I’m pretty stoked :)

This physical hardware business makes recording youtube videos more tricky so I’m gonna go buy some extra cameras today so I can try show this stuff off.

Hopefully I’ll be back soon with a new video for you. Ciao

Not sure how long until we will be programming biology but it doesnt hurt to start knowing about the technology which will make it robust. CRISPR is an interesting topic in how it works and also with it’s reliability (relative to previous methods).

Andrew Gibiansky has writting an excellent post on CRISPR which was very easy to understand, check it out here

…the most terrifying thing code can do is run whilst you looks at some part of it so terrible that you cant understand how it even started. It’s like cutting someone in half and they walk off and try and carry on their day.

Been a bit quiet recently but yesterday I spent the day working on conditionals in cepl.

There are simply too many cases where the error is cryptic, or fails way later in the process that it should have.

Defpipeline should now through more sensible errors now and varjo will try and suggest types if you use one it doesn’t know. e.g.

VARJO> (defshader test ((x sampler2d))
         (texture x (v! 0 0)))

gives

Varjo: Varjo: Could not find the correct type for type-spec SAMPLER2D

Perhaps you meant one of these types?:
:usampler-2d
:sampler-3d
:sampler-2d-shadow
:sampler-2d-rect
:sampler-2d-ms
:sampler-2d-array
:sampler-2d
v-sampler-2d
:sampler-1d
:isampler-2d
:sampler
   [Condition of type UNKNOWN-TYPE-SPEC]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1005078003}>)

I have finished cleaning up type errors for textures and will be focusing on c-arrays & fbos after this.

That should hopefully stop the cases where Ι assume that cepl is buggy rather than checking my own typos :D