When I gave that talk a few weeks back I said, rather naively in hindsight, that geometry & tessellation shaders should work.. man was I wrong there. It turned out to be rather fiddly to find a balance that felt lispy, worked with my current analogies and worked across all GLSL version (or at least failed gracefully).

Lets start at the beginning.

Passing values between GLSL stages

To pass values from stage to stage in GLSL starts simply, you declare something as out in one stage and in in the next e.g:

out vec4 foo; // in the vertex shader

in vec4 foo; // in the fragment shader

Nice and simple. It also works for arrays of values.

Next we add a geometry (or tessellation) shader. Now we are working with primtives (or patches) so the outs from the last stage become an array of ins in the geometry stage.

out vec4 foo; // in the vertex shader

in vec4[2] foo; // in the geometry shader

The length of the array is dictate by the size of the primitive, so lines are length 2, triangles are length 3, etc.

But how about if the out was an array?

out vec4[10] foo; // in the vertex shader

in vec4[2][10] foo; // in the geometry shader

Simple right? I thought so and I updated my compiler to work with this. I kept having this nagging feeling though, something about interface blocks. Turns out I should have listened to the feeling sooner. Support for arrays of arrays only arrive in GLSL in v4.3 and before that you needed to use an interface block:

out VertexOuts
{
	vec4[10] foo;
}

out VertexOuts
{
	vec4[10] foo;
} gs_in[2];

Which seems ok, but it has subtleties to it that make this hard to abstract for all GLSL versions. Lets have a look at the lisp.

First a vertex shader:

(defun-g test-vert ((position :vec4) (uv :vec2))
  (values position normal))

the outputs from the above are a vec4 which is used as the gl-position and a vec2 which is passed to the next stage. Here is a valid fragment shader to go with this:

(defun-g test-frag ((uv :vec2) &uniform (tex :sampler-2d))
  (texture tex uv))

So far, so simple. Next let’s look at a geometry shader that would match that vertex shader’s outputs. We will assume we are rendering triangles.

(defun-g test-geom ((uv (:vec2 3)))
  ...)

Makes sense right? We just array the inputs. However we know that our interface block is going to cause problems. In this case uv isn’t really vec2[3] it’s in blockName { vec2 }gs_in[3]. We have a mismatch between the abstraction and reality. However it’s a useful lie, it takes the GLSL behavior and makes things more consistent and thus easy to understand, so if we can keep it I’d prefer to.

The answer (at least for now) is to make an ‘ephemeral’ type, this is a type that can’t exists in GLSL for real, but one that our compiler can handle.

So this:

(defun-g test-geom ((uv (:vec2 3)))
  (let ((a uv)
        (index 1))
    (aref uv index)
	..))

becomes something like:

..
in IN_BLOCK
{
   vec2 uv;
} gs_in[3]

void main()
{
    int index = 1;
    gs_in[index].uv;
    ..
}

Notice that int index = 1 ended up in the GLSL but there is no a = <something>, that is because of that ephemeral type. The reason that it has to be ephemeral is that until we use aref we can’t access the uv slot inside the block. Also you cant write IN_BLOCK tmp = gs_in[1] or IN_BLOCK[3] tmp = gs_in as whilst interface blocks may look like structs, but they dont’ behave like them and those two example as simply illegal in GLSL. It’s a damn shame really as otherwise this would be much easier!

One other option we could have gone for instead of this ephemeral array business we could invent a ‘primitive’ type. So the geom shader could be:

(defun-g test-geom ((uv triangle))
  (let ((a uv)
        (index 1))
    (aref uv index)
	..))

But then test-geom is no longer a stand alone function, we would need to wait until test-geom was used in a pipeline to know what slots triangle contained. One of the beauties of CEPL is that all gpu functions are simply functions until they are used in a pipeline, they can be used as stages or just be called from other gpu functions. If we use triangle then we can’t compile this gpu function ahead of time, which would suck as that feature currently lets you find and fix bugs faster. On top of this triangle would also have to be ephemeral as there is no triangle type in GLSL

The same also goes for requiring the user to define the interface between vertex & geom shaders as structs. You still need the ephemeral & you require the user to repeat themselves.

This is one of those cases where every option is kinda sucky but the first one described feels the least sucky, and that is the one I have gone for.

Implement it

The above took a few days of experiments and pain to get ironed out, and then it needed to be implemented. As usual working in areas of the compiler that haven’t been touched for a while uncovered bugs and general weaknesses that needed fixes.

Another thing that was playing a lot on my mind was that one of the features CEPL has is support for is defining a stage as raw GLSL and using it in a pipeline, so whatever I made needed to not break that. Having users is awesome and I really want to make sure I don’t fuck up their workflow if I can help it. For example one person is already using the GLSL stage support in CEPL to use Tessellation shaders, which is something I’ve never used! The fact that they can do this makes me happy and eases the pain whilst I slowly find nice lispy ways of doing the same.

Oh, I almost forgot..

Primitives!

As we are in the world of Geometry & Tessellation we need to talk about primitives. One of the things I love about Varjo is that it can pass information from stage to stage so the user doesn’t have to write the same stuff multiple times, we should be able to do the same with primitives. I wanted to be able to take the OpenGL draw-mode and pass it through the compilation. This allows Varjo to not only write some of the GLSL automatically (e.g. layout(triangles) in; in the Geometry stage) but also to know the length of the array’d interface block going into the geometry stage. Having little details like this checked is worth my time as then Varjo can give much nicer and more targeted errors that GL can.

I’ll skip the details on this for now as this post is already getting long.

Where are we now?

We are here

I finally got these damn things working. Here we are using a geometry shader to draw the normals of the sphere.

There is still a bunch of stuff to do and there are some aspects that really pushed the limits of how ‘lispy’ stuff could be made. I’ll save that for another post. I’ll simply say thanks for reading and leave you with the pipeline for drawing the lines in the above.

Ciao

This lisp code:

(defun-g normals-vert ((vert g-pnt) &uniform (model->clip :mat4))
  (values (* model->clip (v! (pos vert) 1))
          (s~ (* model->clip (v! (norm vert) 0)) :xyz)))

(defun-g normals-geom ((normals (:vec3 3)))
  (declare (varjo:output-primitive :kind :line-strip :max-vertices 6))
  (labels ((gen-line ((index :int))
             (let ((magnitude 0.2))
               (setf gl-position (gl-position (aref gl-in index)))
               (emit-vertex)
               (setf gl-position
                     (+ (gl-position (aref gl-in index))
                        (* (v! (aref normals index) 0f0)
                           magnitude)))
               (emit-vertex)
               (end-primitive)
               (values))))
    (gen-line 0)
    (gen-line 1)
    (gen-line 2)
    (values)))

(defun-g normals-frag ()
  (v! 1 1 0 1))

(def-g-> draw-normals ()
  :vertex (normals-vert g-pnt)
  :geometry (normals-geom (:vec3 3))
  :fragment (normals-frag))

makes this glsl:

("#version 450

in vec3 fk_vert_position;
in vec3 fk_vert_normal;
in vec2 fk_vert_texture;

out _FROM_VERTEX_
{
    out vec3 _VERTEX_OUT_1;
};

uniform mat4 MODEL_62CLIP;

void main() {
    vec3 return1;
    vec4 g_G1550 = (MODEL_62CLIP * vec4(fk_vert_position,float(1)));
    return1 = (MODEL_62CLIP * vec4(fk_vert_normal,float(0))).xyz;
    gl_Position = g_G1550;
    vec3 g_G1552 = return1;
    _VERTEX_OUT_1 = g_G1552;
    return;
}

#version 450

layout (triangles) in;

in _FROM_VERTEX_
{
    in vec3 _VERTEX_OUT_1;
} inputs[3];

layout (line_strip, max_vertices = 6) out;

void GEN_LINE(int INDEX);

void GEN_LINE(int INDEX) {
    float MAGNITUDE = 0.2f;
    gl_Position = gl_in[INDEX].gl_Position;
    EmitVertex();
    gl_Position = (gl_in[INDEX].gl_Position + (vec4(inputs[INDEX]._VERTEX_OUT_1,0.0f) * MAGNITUDE));
    EmitVertex();
    EndPrimitive();
}

void main() {
    GEN_LINE(0);
    GEN_LINE(1);
    GEN_LINE(2);
}

#version 450

layout(location = 0) out vec4 _FRAGMENT_OUT_0;

void main() {
    vec4 g_G1553 = vec4(float(1),float(1),float(0),float(1));
    _FRAGMENT_OUT_0 = g_G1553;
    return;
}

From Sunday to Wednesday I was away on Brussels at the European Lisp Symposium. It’s an opportunity to see good talks but more importantly to me it’s a catch up with some folks I haven’t seen in a year and nerd out.

Aside from that I’ve been cracking away at getting sketch working using CEPL under the hood. It’s rendering using lisp shaders now and live recompile works great. There has been a slight performance drop as sketch used buffer object streaming to upload the vertices and I’m not doing that yet. In fact CEPL doesn’t expose glMapBufferRange at all so first order of business is fixing that. After that I want to make a vertex-ring it’s going to be an object that encapsulates the buffer object streaming pattern and will make using it seamless.

I have also been working on fixes to Varjo that will finally get Geometry shaders working. The task for the last few days has been rewriting how Varjo handles return & making Varjo use interface blocks for passing data between stages. The reason for working on return (apart from it being buggy) was that in geometry shaders your primary task is to ‘emit’ extra geometry rather than ‘returning’ transformed data via the out vars.

I’m hoping that this weekend I can get those two things coded and tested. After that I need to look at the Geometry shaders themselves, I know there are some ugly details around ins & outs so wish me luck.

I’m tired right now but also really happy with the reception the video got.

I’ve had a number of PRs to CEPL & Varjo with fixes for various things. Of special note is a chap called djeis97 who has been given me a pile of help with issues in my geometry shader generation. He’s been using the inline glsl features of CEPL too which is awesome.

In order for geometry shaders to work properly I need to do the following:

• Improve CEPL’s handling of arrays
• Generate interface blocks between shader stages
• Fix some hacks around return & out values
• Add some helpers for emitting vertices.

I’ve made some progress on the arrays in the last week and will be starting on interface blocks asap.

I’ve also had one chap who works on a project called sketch decide that he has had enough with dealing with opengl directly and wants to build his project on top of CEPL. This is exactly the kind of project I’ve wanted to see built with CEPL, stuff that has a clear set of trade-offs and makes making stuff super simple. His current approach supports multiple windows however, and CEPL did not, so I have been getting that in place. This has meant (even though Im not implementing yet) looking at multiple GL contexts and threading.

CEPL has no business with your threads and expects you to use them wisely, sketch however was built on top of sdl2kit which handles threading for you. This mismatch with CEPL is a little problematic and, as I don’t yet have any libraries for helping with the threading situation, I’m going to make a shim to make sdl2kit drive CEPL. It’ll be butt ugly but will hopefully be enough for now.

It’s the European Lisp Symposium in Brussels next week so it would be awesome if I had something new to give a lightning talk on..or even just to show of with :)

At some point I need sleep too.

Ciao!

Turns out I booked a bunch of small trips last year and they all are around now. Sorry for the late update.

I got back to coding yesterday and have been working on the compiler. Since the talk I have some great support from djeis97 and jobez on issues in CEPL & Varjo. One big thing I realized was more broken that expected was Geometry Shaders, so that is current top priority.

The prerequisites for fixing this are better array support in Varjo & support for interface blocks, so whilst traveling I was reading the GLSL spec again. Last night I added checks to make sure you are not trying to access an uninitialized variable and then started on adding lisp’s make-array function.

As I currently don’t track const‘ness in Varjo I can’t validate a bunch of the array rules that the GLSL requires. However those are errors that will be caught by the driver’s own glsl compiler so I’m not worrying about those too much yet.

That’s all for now Peace

As I wrote so late last week I haven’t got too much to add here, however I have made progress on tweak.

The goal is to be able to inspect & change data easily while the program is running. You wrap a form in (tweak ..) and you get a small window (rendered using gl via nuklear) which allows you too mess with the value.

For example given this:

(defparameter *some-val* 10)

You can write this in the main body code

(tweak *some-color*)

And you get this:

And if you scrub the value it immediately applies to the form you are tweaking. You can define different tweak ui’s for different lisp types so extending will be easy. I still need to add a couple of helpers for that though.

Other than that I made a little companion library to SDL-TTF which allows people to render text straight to CEPL textures. So this:

(with-font (font "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf")
  (text-to-tex "Yo!" font (v! 1 0 0 1)))

Gives you a texture with the text.

That’s the lot I’m afraid. Time for sleep.

I have traveled to see family this week so sorry for the late writeup.

This week went pretty well. I went through the PBR example I had been using line by line making notes and then tried to match my implementation to theirs.

I had made some progress but still the colors looked washed out, something must have been putting the color values way out of wack. Finally I saw it, I had been adding a color I should have been multiplying. A facepalm and a tweak later color started looking like this

I’m still not convinced that the example’s approach for creating the final color is correct, however this is plenty good enough for now.

With that done I revisited my UI code (which uses nuklear) and found that it was broken…shits.

A bunch of git archaeology told me the issue was related to my GL context caching. Double shits

..and then that the crash occurred because I had made CEPL’s handling of the context more robust. At this point a drink was in order, but after that, fixing!

The result was that I hadn’t been unbinding VAOs correctly when rendering and so then subsequent buffer unbindings had been captured in the VAO. This sucks but was a 1 line fix and CEPL is much better for it.

Theres other stuff going on buts they are not worth yakking about right now.

Seeya

As I was so late writing last week I don’t have a lot that’s new, but I need to keep the rhythm of writing these.

I have been focused on getting my maths library up to scratch. I did a bunch of work on getting the non-consing api cleaned up and merged. This involved fixing a whole host of optimization notes from the compiler. The compiler I use is pretty good at explaining where it is forced to ‘waste’ speed and why. The biggest single change was changing the return type of the vector length & distance functions to show that it could only return positive floats. That was important as the square root of a negative number is a complex number, so by indicating the functions could only be greater than 0 the compiler could ignore the complex number cases.

I also found a branch where I had started adding support for lines, rays, line-segments & axis aligned bounding boxes in ℝ3. I fixed up the line-segments & AABBs and got that merged too.

Finally I started the non-consing (destructive) version of the matrix apis. This is also on master.

Tonight I want to test the non-consing api against the consing version to make sure I haven’t introduced any bugs. I’ll then redefine the consing version using the non-consing functions to do the work. At that point I’m probably ready to stop scratching this itch and focus on graphics stuff again.

Peace

This week (and a bit) has felt like a bit of a blur as I’ve mainly been working on boring stuff that just needed doing. I’ll be honest, I can’t remember what I did so I’ll just go spelunking through my commits and see what I find.

lisp-games-cross-test

I started yet another project :p. This came out of me feeling the uncertainty you get when you have written a good chunk of that stack you use (and don’t have enough tests). It’s stupid to have written this thing which is meant to be nice for creating things and then be stymied by and lingering sense that something, somewhere must be breaking.

One thing I really wanted to be sure about was that my maths library rtg-math was solid.

As I didn’t want to rely on my own (very limited) knowledge I decided to make a basic fuzz testing setup where I compare my library against other math libraries written in Common Lisp.

I built on top of the excellent fiveam testing library and made a fairly nice system for defining comparative tests across several libraries. It takes into account that a library’s features will not exactly overlap with another. I’m happy with it so far but it’s not ready for github yet.

Anyhoo I set it to work and found that a lot of stuff is working just fine. One function (the matrix to quaternion function) was broken. At first I stupidly blamed the c++ I based the function on until, after many re-reads, I noticed that the [] operator had been overloaded to index from x and ignore the w component altogether. This meant I had a bunch of ‘off by one’ indexing errors in that function :facepalm:.

However I did find a typo in the book’s code which has now been fixed! So that’s a nice side effect of this work.

I then got massively mistake about the api of another library I was comparing to (sorry axion). The process of being gently schooled by psilord on the #lispgames irc was incredibly helpful but did send me into frequent ‘What if my entire api is wrong’ panics.

Luckily he provided me with some excellent links and I’m slowly getting a better grasp of the terminology.

Luckily, as my library is heavily based on the c++ code from the text book I learned from, my library is fine and then only problems were (and to some extent remain) with my brain :D

RTG-Math

Whilst working on the above I also got a friendly jab that my vector math was too slow. It turns out the ‘jab-er’ in question was comparing his destructive (mutable) api with my non-destructive one. This is kind of understandable as I didn’t have a destructive api to compare against.. but of course I fixed that >:)

Below is the (very basic) test I did to check how it performs:

;; First we make a list of 1000 random vector3s
;;                                    ↓↓
RTG-MATH> (let ((to-add (loop :for i :below 1000 :collect 
                           (v! (- (random 2f0) 1f0)
                               (- (random 2f0) 1f0)
                               (- (random 2f0) 1f0))))
                (res (v! 0 0 0)))
            ;;
            ;; Here we make the list circular
            ;;     ↓↓
            (setf (cdr (last to-add)) to-add) 
            ;;
            ;; Then we iterate 100 million times adding the elements of
            ;; the circular list into the variable 'res'
            ;;    ↓↓
            (time
             (loop :for i :below 100000000 :for e :in to-add :do
                (v3-n:+ res e)))
            res)
            
Evaluation took:
  0.748 seconds of real time
  0.748000 seconds of total run time (0.748000 user, 0.000000 system)
  100.00% CPU
  2,297,194,923 processor cycles
  0 bytes consed 
  
#(-1248991.9 2382024.5 358581.84)

Not too shabby! bytes consed in lisp means ‘memory allocated’ so we see this only mutating the vectors already created.

The test is pretty rough and ready however the list is long enough that it wouldn’t just fit in a cache line, and the random calculates are outside the loop as they the slowest part of my original test.

In the process of making these destructive versions of the API I have been going through the majority of the codebase adding type declarations to everything I could. I still have some optimization work and cleanup to do but it’s mostly giving the compiler the information it needs to do it’s job.

Optional typing is fucking great

Nineveh

Nineveh is intended to be a ‘standard library’ of gpu functions and helpers for CEPL. It has code that doesn’t belong in a GL abstraction but is really useful to have. Currently it’s not in quicklisp (the lisp package manager) as it’s still too small & too in flux however progress is being made.

The main addition this week was a draw-tex function which will take any texture or sampler and draw it to the screen. Simple stuff but it has a bunch of options for drawing in certain portions of the screen (top-left, bottom-right etc) scaling, rotation, drawing cubemaps in a nice ‘cross’ layout and maintaining aspect ratios while doing this.

Also Ferris helped me find some (really nasty) bugs in one of the functions in the library which was ace.

PBR

I spent yesterday evening at Ferris’ place where I got a bunch of help with some artifacts I was seeing in my code. It helps so much to have someone who knows what certain things should look like. When it comes to graphics coding I really lack that.

In the process of this I found the previous mentioned bug in nineveh, a bug in my compiler (let wasn’t throwing an error for duplicate var names) and added a way to support uint literals.

This has made me much more confident in the code, and with where the rest of artifacts I’m seeing are probably coming from. HOPEFULLY will have some new screenshots soon.

Sleep

I haven’t been getting enough, so I’m going to bed.

Seeya!

It’s done, finally merged. Hundreds of commits, 15 issues closed, a monster is dead.

The biggest thing I’m noticing right now is how slow the evening goes when you aren’t crunching :D

I was looking at last week’s post to see what I was up to and I realize that although I’ve been feeling like I was going really slow I’ve had 52 commits in the last week which includes the following:

Added a simple api for user’s to be able to inspect the compilation environment in their macros

Common Lisp allows you to access the environment in the macros, however the interface to interact with that object didnt make it into the standard. It was, however, documented so I can still read it [0]. I decided not to implement that api yet though and instead I just made the functions I felt made sense. It will be easy to revisit this when I feel the drive to.

Added ‘declare’

Lisp allows you to declare information about things in your code, whether it is whether a function should be inline or the type of a value, it’s handled through declare

Once again, although it didn’t make it into the standard, extending the kinds of things that can be declared has been thought about and documented. For now I just hooked it into my compile-time-metadata system. A little reading reveals that the proposed system is a rough superset of what I offer so at some point I will implement that and redefine by metadata using it.

Added ‘deftype’

This is one place I left the lisp spec entirely. deftype in lisp is mad, types are seen as sets of objects (makes sense) and they don’t restrict you in how you define that. For example I can say:

;; a function that returns true if given an array where
;; all it's dimensions are of equal length
;;
(defun equidimensional (a)
   (or (< (array-rank a) 2)
       (apply #'= (array-dimensions a))))

;; the type of all square arrays
;;
(deftype square-matrix (&optional type size)
  `(and (array ,type (,size ,size))
        (satisfies equidimensional)))

This is super expressive but totally impossible to check at compile time as you can have arbitrary function calls in there. The compilers do a great job on using the flexibility in some cases though, which I won’t yak about now.

In GLSL things are very different, we don’t want to do anything at runtime unless we can help it. We can’t make any types except structs and I already have a nice mechanism for that.

However there are so things that may be interesting.

• Making a ‘special kind of’ some exisiting type. For example a rotation type that is essentially a vec3. In the GLSL it will be a vec3 however by making the type checker see it as a different type we could make it impossible to pass a position as the rotation argument to a rotate function by accident. I call these shadow types (I probably need to rethink this name :p).

• Making a type that only exists at compile-time for the purpose of holding metadata. This ‘ephemeral’ type never appears in the resulting GLSL and is an advanced feature but can be useful. I use these to represent vector spaces in my code and then use macros to inject code to create the matrices that transform between the spaces.

To make a shadow type I write (deftype rotation () :vec3)

To make an ephemeral type I write (deftype space () ())

In the case of shadow functions I also allow you to easily ‘copy’ functions that are defined for the type you are shadowing. This means you don’t have to redefine addition, subtraction etc for your rotation type, you just copy it from vec3

Mooore Types

One problem with how I used to write lisp is that too often I would use lists when I should have used classes. This has bitten me in the arse one to many times and so I refactored a bunch of stuff to have their own types.

This will help with future refactoring and also as I start working out the ‘final’ public api for my compiler.

Rewrite Space Feature in CEPL

Spaces are a feature I have spoken of before but I will quickly recap. They are a fairly simple idea, you define a scope with a vector space and inside that scope all vectors have to be in that space. If you try and use a vector from another space it will be automatically converted to the correct space before anything else happens. This lowers the chances of messing up your vector math and makes the code more self-documenting.

Previously this was implemented by:

• compiling the code once
• analysing the AST for places that needed conversions
• patching the code with the needed conversions
• recompiling and using the new GLSL

This felt super hacky as all of the process was specific to my compiler. I have now rewritten it using macros, there is much less that will feel alien to a lisper[1] and it all happens in one pass now.

Change the default minify-filter for samplers in CEPL

I thought I had a bug in the textures or samplers in CEPL. I was using the textureLod function in my shaders but no matter what mipmap level I chose nothing would change.

I spent hours looking at the code around mipmaps and finally found out that nothing was wrong, but I had misunderstood how things were meant to work. I had imagined that textureLod allowed you to query the colors from a texture’s mipmaps regardless of the filter settings, whereas actually you need to minify-filter to either linear-mipmap-nearest or linear-mipmap-linear before ANY querying using textureLod is possible.

CEPL defaulted to just linear as it felt the simplest and I didnt want to confuse beginners (irony goes here) but instead I created a behaviour that was surprising and required knowledge to understand. This interests me and I’ll muse over this in future when designing apis.

Bugfixes EVERYWHERE

Just whole bunch of other stuff. Not much to note here other than how quickly trying to make nice errors seems to balloon code.

For example if you are processing 10 things, and one fails, it is easy to detect this in the process_thing function and throw an error. However this means that the user may fix the one thing retry and hit the same error on the next thing. Really we want to aggregate the errors and present an exception saying ‘these 15 things failed for these reasons’. This would give the user more chance of seeing the patterns that are causing problems. This is especially true with syntax error in your code. You don’t want them one at a time, you want them on aggregate in order to see what you are doing wrong.

This also means that your error messages should be able to adjust depending on the number of things that went wrong. For example when 15 things break the ‘these 15 things failed for these reasons’ message is cool, but if only 1 thing broke we want ‘this thing failed for this reason’. These subtle language differences can make a huge difference to how the user feels about the tool they are using. Pluralizing words when there is only one thing sounds dumb and just adds to the annoyance of the tool not working.

I don’t have ideas for a clean solution to this yet, but one day I want something for this.

Sleep

I’m damn tired now. I’m off to bed.

Peace

[0] https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html [1] and once I have the stuff from [1] it will be even better

I’ve been back in the compiler this week and it’s just been constant refactoring :p

I kicked of the week with tweaks & bugfixes to the first-class function code. I then started on adding some compile-time metadata to Varjo.

The idea is super basic: We want to associate some data with the values flowing through the shader.

We already have flow-ids, which represent the values in the shader, so we could just add a hashtable that maps these ids to the data. Simple, and it seems to work!

At the very least it will work enough to make some progress on the idea anyhoo. You are able to add, but never remove metadata. Metadata can also be combined, for example:

(let ((x (v! 1 0 0 0))  ;; ← lets assume that both these have metadata
      (y (v! 0 1 0 0))) ;; ← with a key called 'foo'
  (if something
      x
      y))

What should be the metadata for the result of the if? Clearly this depends on what the metadata is for. So in the case of conditionals like if or switch we call a method called combine-metadata defined for that metadata type, and it is it’s job to decide the metadata for the new result.

This mechanism is very simple but should be enough to get some useful results.

So with this in place I could dive back into re-implementing my space analysis feature! Progress.. excitement… Ah fuck it, I really need symbol-macros.

Symbol macros are pretty simple:

(let ((x some-object))
  (symbol-macrolet ((foo (slot-value x 'foo))) ;; You specify a symbol 'foo' and what it will expand into
    (print foo)   ;; the compiler then replace every instance of 'foo' in your code with the expansion
    (setf foo 20) ;; before it compiles to code
    (print foo)))

So what actually ends up getting compiled is:

(let ((x some-object))
  (print (slot-value x 'foo))
  (setf (slot-value x 'foo) 20)
  (print (slot-value x 'foo)))

So that should be simple to implement.

SHOULD BE

Alas when I made the macroexpander[0] for Varjo I was super lazy. I didn’t implement any lexically-scope macros and just expanded everything in a prepass. This really hurt as I suddenly needed to go change how all this was written and mix the expansion into the compile pass.

This refactor took all of Sunday and it was one of those ones where you are changing so much fundamental stuff that you can’t compile for an hour or so at a time. As someone who compiles many times a minute this just left me feeling very unsettled. However when it was done the little test suite I have made me feel fairly confident with the result.

Another nice thing about all this is that my macros will now be able to take the compilation-environment as an argument just like in real common lisp. This means I can provide some of the environment introspection features that are described in CLTL2 but sadly didn’t make it into the standard. This together with metadata make the macros extremely powerful.

Next week

The first order of business is compiler-macros, which should be easy enough. Then I want to add support for &rest (vararg) arguments to the compiler. Then I will get back intothe spaces feature and see how far I can get now.

Peace

[0] back in 2013, holy shit!