TechSnuffle2023-12-09T12:21:25+00:00http://www.techsnuffle.comBaggerstechsnuffle.gmail.com2023-12-09T12:21:25+00:00http://www.techsnuffle.com/2023/12/09/2023-12-09-talespire-dev-log-424--creature-modding-package-available-for-testing<p>Heya folks!</p>
<p>In preparation for creature modding feature shipping, we are making the tool available so you can try it out, and we can fix any egregious bugs you find.</p>
<p>You can find a guide on how to get the tool and how to use it over here:</p>
<p><a href="https://talespire.com/taleweaverlite-guide">https://talespire.com/taleweaverlite-guide</a></p>
<blockquote>
<p>NOTE: You cannot load the mods into TaleSpire today. This is simply a build of the tooling used to make the mods.</p>
</blockquote>
<h2 id="what-is-this-and-what-is-it-not">What is this (and what is it not)</h2>
<p>TaleSpire is going to have a couple of ways for you to add new creatures to the game:</p>
<ul>
<li>In-game creature creator: A way of building creatures out of a palette of configurable parts</li>
<li>Mods: mod-files are composed of pre-made assets outside of TaleSpire</li>
</ul>
<p>TaleWeaverLite is for the second approach. It is a very minimal tool for composing pre-made assets into a creature mod that is ready to be used by TaleSpire.</p>
<p>It is not a modeling or posing tool. It expects content in specific formats and with specific layouts.</p>
<p>The expected usage is that the mesh and textures are prepared in other tools, such as Blender, and using TaleWeaverLite is the final step in making a mod.</p>
<h2 id="the-good">The good</h2>
<p>Once you are producing meshes and textures in the correct format, it’s trivial to turn them into a mod. Simply drag them into the Unity project, set the fields in the creature panel, and save.</p>
<h2 id="the-meh">The meh</h2>
<p>We’ve been using Unity for our asset tools for a long time. It handled processing assets for us and gave us a UI we could use to make tools.</p>
<p>This means that to use TaleWeaverLite, you need to install the free version of Unity.</p>
<p>We have mixed feelings about this, as packaging this stuff for public use has not been as easy as we had hoped. Your feedback will help us decide whether to stick with this in the long term or to start moving to a more standalone tool.</p>
<h2 id="the-future">The future</h2>
<p>We are busy working on the backend changes needed to support the new creatures. Once we have that done, we’ll wrap up the needed changes in TaleSpire and get ready to ship. We are pushing hard for this and hope we can get this in your hands very soon.</p>
<p>By getting TaleWeaverLite out now, there might even be some new creature mods ready to ship the moment that TaleSpire gets the feature!</p>
<p>Until then, thanks for stopping by.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4222023-11-13T23:33:56+00:00http://www.techsnuffle.com/2023/11/13/talespire-dev-log-422<p>Heya folks,</p>
<p>Today was a tedious one.</p>
<p>I finished up last week by moving a whole bunch of code out of our Unity project and into a separate C# project that referenced the Unity DLLs. We are doing this as we want to ship TaleWeaver as compiled code[0].</p>
<p>I finished that task, and the TaleSpire project compiled, but when I ran it, it crashed immediately. The reason is that by moving all the code to a separate project, all of the references from prefabs and scenes to that code broke.</p>
<p>To understand why, we have to know that Unity wants developers to be able to rename and move things without breaking all those references. So, rather than referring to the classes by name[1], they create a GUID for everything in the project. As they can’t push those GUIDs into the files they are identifying, they instead make a separate file called a <code class="language-plaintext highlighter-rouge">meta</code> file, which stores the ID and any other metadata for that thing[2].</p>
<p>As meta files are a Unity concept, I can’t include them in our new C# library. So, by moving the code, I broke all the references!</p>
<p>Of course, Unity does need a way to identify types inside a C# library, and I found this unofficial info talking about how it works: <a href="https://www.robinryf.com/blog/2017/10/30/unity-behaviour-in-dlls.html">https://www.robinryf.com/blog/2017/10/30/unity-behaviour-in-dlls.html</a></p>
<p>I now have the tools I need to fix this issue.</p>
<ul>
<li>First, I will check out a build from before I moved the code out to the separate library.</li>
<li>Then, I will write a script to collect the GUID for every script in the project[3]. I will make a lookup table between the GUIDs and the fully-qualified type names.</li>
<li>I can then check out the most recent code (the one with all the broken references), scan the project for any use broken references, and use the GUID to find the fully qualified type name.</li>
<li>With the type and the sample code from the link above, I can calculate the new reference information and replace the broken reference.</li>
</ul>
<p>If I’m correct, I can automate the entire repair, which is a relief as I have done fixes for issues like this before, and they SUCKED.</p>
<p>In the future, I might even be able to use this knowledge to make tools to help fix issues that occur during merges sometimes.</p>
<p>Well, wish me luck for tomorrow. We’ll see how this all turns out.</p>
<p>Peace.</p>
<p>p.s. The MD4 hashing code from the linked blog post was helpful but performed a bunch of unnecessary heap allocations. I ended up procrastinating earlier today by tweaking it to be more efficient. You can find that here: <a href="https://gist.github.com/baggers-br/48bba51a9e0b1c5a0632c57537f84534">https://gist.github.com/baggers-br/48bba51a9e0b1c5a0632c57537f84534</a>
It is almost entirely untested, so please beware.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] I know, I know. Byte-code. But still, you know what I mean.
[1] Fully-qualified or otherwise
[2] If you are very curious, you can find some info about this here: <a href="https://unityatscale.com/unity-meta-file-guide">https://unityatscale.com/unity-meta-file-guide</a>
[3] In fact I’m only including subclasses of MonoBehaviour</p>
TaleSpire Dev Log 4212023-11-10T12:07:23+00:00http://www.techsnuffle.com/2023/11/10/talespire-dev-log-421<p>Heya folks!</p>
<p>After a few weeks of public activity, it feels weird to be quiet again, but things are progressing well.</p>
<p>My big task is restructuring our projects so we can ship the creature modding tools as DLLs.</p>
<p>This task involves some decoupling from Unity, so I’ve been removing the dependency on Unity.Entities. We didn’t use the ECS they provided at all, but they had a zero-copy data format, which was pretty handy.</p>
<p>I wrote our own format the other day, so for the last couple of days, I have been updating all the code to use that. I’ll admit that a lot of that time was me trying to find what I thought were bugs but turned out to be me screwing up.</p>
<p>My task for today is to work out how I will handle conditional compilation (e.g. ConditionalAttribute and #if) with this library. We’ll see how that goes.</p>
<p>Catch ya soon.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4202023-11-02T00:11:39+00:00http://www.techsnuffle.com/2023/11/02/talespire-dev-log-420<p>‘Allo folks.</p>
<p>Today went well. I’ve got a texture format I think we can work with, and I’ve made a de/serializer for it. Here is a quick pic.</p>
<p><img src="/assets/images/tex0.png" alt="textured mesh loaded from our formats" /></p>
<p>There isn’t really a way to tell it’s from our format, but I guess that’s the point :P</p>
<p>I need to put this all together into a file format, but that is easy with the work from the other day.</p>
<p>The next unknown I need to tackle is building dlls from Unity projects. This is required for the project we will ship to let you make creature mods.</p>
<p>We’ll keep you posted on the progress.</p>
<p>Peace</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4192023-10-31T18:13:46+00:00http://www.techsnuffle.com/2023/10/31/talespire-dev-log-419<p>Hi again folks,</p>
<p>Today, I <a href="https://bouncyrock.com/news/articles/talespire-dev-log-418">continued</a> working on our custom mesh format. The image below shows some of the progress.</p>
<p><img src="/assets/images/mesh0.png" alt="first mesh loaded from custom format" /></p>
<p>This is the first mesh I’ve taken from Unity to our format and back again.</p>
<p>With that working, my next task for creature modding is to work on how we will store textures on disk. I should be able to work on that tomorrow, although there may be other Bouncyrock work that will take precedence. We shall see :)</p>
<p>That’s all for today,</p>
<p>Ciao.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4182023-10-30T19:38:26+00:00http://www.techsnuffle.com/2023/10/30/talespire-dev-log-418<p>Heya folks! I’ve not written in the last couple of weeks as we’ve been going hell for leather, trying to ship two patches a week.</p>
<p>That’s been very successful, and it’s a lot of fun to do from time to time, but it is pretty unsustainable when you hit bigger features. So this week, we are only aiming to ship the board folders. The good news, however, is that the feature I’m chipping away at now is creature mods.</p>
<p>So, where are we at?</p>
<p>Unity can pack stuff into things called AssetBundles. They are very flexible, but one downside is that you need to build one for each platform you are targeting. That isn’t nice for modding, and it also makes upgrading the Unity version stressful, as the AssetBundle format can change over time.</p>
<p>As we don’t need[0] all the features AssetBundles provide, we are opting to make our own cross-platform format that only supports the very few things we need.</p>
<p>The first task was defining a low-level format that only specifies blocks of bytes and an index. That was nice and simple to design, and I’ve left the save/loading code for another day, as it should be pretty straightforward.</p>
<p>From there, I’m making a format for storing meshes that is compatible with Unity’s low-level mesh APIs. We want to use that low-level format as it allows us to create meshes across multiple threads and also do so with a minimal number of copies. I got a nice draft of that format finished in the morning and then started fleshing out the implementation of a zero-copy de/serializer[1].</p>
<p>Until now, we have used Unity’s <a href="https://docs.unity3d.com/Packages/com.unity.entities@1.0/manual/blob-assets-create.html">blob asset reference</a> system. However, it means we need a dependency on an additional Unity library, and I’m no longer comfortable with any unnecessary dependency on Unity systems (due to how Unity has acted this year[2]).</p>
<p>I think I’m about 70% of the way through writing the zero-copy stuff. Tomorrow, I’ll start pushing meshes through this thing and see how it handles it. I’m sure I’ll hit plenty of things I forgot up to that point.</p>
<p>As soon as meshes are working, I’m looking into how we will handle textures. I’ve been giving it some thought already, so it should be quick to make some progress.</p>
<p>I’ll be sure to come back and update you once I’ve got some news to share on the texture progress. Especially as, from there, I should be able to dive into the UI that you folks will be interacting with!</p>
<p>I hope you’re having a good day,</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] And actually we actively do not want all the features</p>
<p>[1] Feels weird to call it that when the point is that you don’t need an explicit deserialize step as you just load it into memory and use it.. but meh. I can’t recall a better term right now.</p>
<p>[2] This might seem moot after they rolled back their proposed fee changes for many folks, but I disagree, and I furthermore think the same applies to WOTC. The company is still run by mostly the same people, they still want money, and it stands to reason that sooner or later, they are going to try and extract it from us again. “Fool me once” can be overused, but it feels appropriate here.</p>
TaleSpire Dev Log 4172023-10-13T01:53:47+00:00http://www.techsnuffle.com/2023/10/13/talespire-dev-log-417<p>Well yesterday was nice! Borodust migrated the database without any major hitches and Golbin shipped another asset pack. I’ve been helping out but my main focus has been on board folders and the slab-browser.</p>
<p>My first pass at my server-side code was a little more broken than I thought, but all the fixes were easy enough. The UI side is also coming along.</p>
<iframe width="800" height="600" src="https://www.youtube.com/embed/lm3goiu5jjs?si=jNML8RCONLAtEbhl" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<p>That’s all from me for today, see you folks around.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4162023-10-11T02:35:24+00:00http://www.techsnuffle.com/2023/10/11/talespire-dev-log-416<p>Heya folks,</p>
<p>It felt good to get the slab-browser beta out, and while I do have things to fix before it is ready for prime time, I couldn’t help but take a swing at a different feature. One we have all wanted for a while. Board folders.</p>
<p><img src="/assets/videos/boardFolder0.gif" alt="janky unfinished folder with a board in it" /></p>
<p>Today was mostly spent writing server code for this feature. I tested some of it but wanted to get some UI in before trying the rest. To that end, I’ve been hacking the basics together.</p>
<p>For the first version, changing a board’s folder will be done via a dropdown. Really, I’d prefer to make it possible to just drag boards into folders, but I’m not familiar with how to do that yet, and I don’t want to spend too much time on this as I NEED to get moving with the creature modding support. So, the first version will be functional, if a bit basic. I’m also not adding arbitrary nesting of folders for now.</p>
<p>Jumping back to the creature modding, I read more about what kinds of access Unity gives us to mesh data. I’m satisfied that I can make something to cover all the bases very quickly. It’s also tempting to look into Zeux’s excellent <a href="https://github.com/zeux/meshoptimizer">mesh optimizer</a>.</p>
<p>That’s enough for today.</p>
<p>See you in the next one.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4152023-09-28T19:47:05+00:00http://www.techsnuffle.com/2023/09/28/talespire-dev-log-415<p>Heya folks,</p>
<p>For the last two days, I’ve been working on the slab upload. It’s going very well.</p>
<p>I’m currently cleaning up a lot of little issues with it, and then it’ll be ready to show it off. The main focus the last two days was being able to edit the details of mods you have already published. That was a slog until about six hours ago when it all started to click.</p>
<p>I’m also pleased to say we support multiple kinds of authentication with mod.io. This means you can sign in via Steam, email, browser, and even QR code.</p>
<p>This is all very meh without some video to show it working, however, so I’ll come back when I have fixed the remaining visual issues.</p>
<p>The big issue right now is that the list that shows published mods doesn’t like certain kinds of filtering and thus can leave gaps between items. I know what to do, but it’s gonna take some hours to code.</p>
<p>Right, seeya folks soon.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4142023-09-26T23:19:20+00:00http://www.techsnuffle.com/2023/09/26/talespire-dev-log-414<p>With yesterday’s realization that some of Unity’s handling of overlapping keybindings is broken, I needed to add our own workaround.</p>
<p>I knew I didn’t want to try and build a complete solution, so the first version was just to handle the issue that caused undo/redo on Mac to be broken.</p>
<p>To recap on Mac, Undo is <code class="language-plaintext highlighter-rouge">Command+z</code>, and Redo is <code class="language-plaintext highlighter-rouge">Command+Shift+z</code>. This is an overlap; in order to hit the keys for Redo on Mac, you also need to hit the keys for Undo.</p>
<p>I’ve added a simple handler that detects multiple actions happening on the same frame and picks the one with the longest chain of modifiers. This works for the Mac undo/redo issue as both actions will fire on the same frame as <code class="language-plaintext highlighter-rouge">z</code> is the last key pressed and is the one that causes both to fire.</p>
<p>We will need to handle more cases in the future, but that’s a problem for another day.</p>
<p>I’ve got some new builds progressing through the build pipeline, so tomorrow, I should be able to ship a fix for that. Then, I can get back to business.</p>
<p>Hope you folks are well,</p>
<p>Seeya in the next log.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4132023-09-26T00:46:18+00:00http://www.techsnuffle.com/2023/09/26/talespire-dev-log-413<p>Evening all.</p>
<p>All I wanted to do today was finish off the slab-sharing UI, but instead, I spent the day looking into an input bug.</p>
<p>It goes like this. Sometimes, you have keybindings that overlap. For example, on Mac, Undo is <code class="language-plaintext highlighter-rouge">Command+z</code>, and Redo is <code class="language-plaintext highlighter-rouge">Command+Shift+z</code>. This is an overlap; in order to hit the keys for Redo on Mac, you also need to hit the keys for Undo.</p>
<p>This was a problem we hit when releasing on Mac, so we turned on a setting in Unity’s input system called <code class="language-plaintext highlighter-rouge">shortcutKeysConsumeInput</code></p>
<p>Here’s how Unity describes it:</p>
<blockquote>
<p><strong>improves shortcut key support by making composite controls consume control input</strong></p>
<p>actions are exclusively triggered and will consume/block other actions sharing the same input. e.g. when pressing the ‘shift+b’ keys, the associated action would trigger but any action bound to just the ‘b’ key would be prevented from triggering at the same time. please note that enabling this will cause actions with composite bindings to consume input and block any other actions which are enabled and sharing the same controls. input consumption is performed in priority order, with the action containing the greatest number of bindings checked first. therefore actions requiring fewer keypresses will not be triggered if an action using more keypresses is triggered and has overlapping controls. this works for shortcut keys, however in other cases this might not give the desired result, especially where there are actions with the exact same number of composite controls, in which case it is non-deterministic which action will be triggered. these conflicts may occur even between actions which belong to different action maps e.g. if using an uiinputmodule with the arrow keys bound to the navigate action in the ui action map, this would interfere with other action maps using those keys. however conflicts would not occur between actions which belong to different action assets.</p>
</blockquote>
<p>Seems ideal, but I think it has a bug. When <code class="language-plaintext highlighter-rouge">shortcutKeysConsumeInput</code> is enabled, we see a problem by following these steps:</p>
<ul>
<li>User selects a region of tiles</li>
<li>User cuts with <code class="language-plaintext highlighter-rouge">Ctrl+x</code>
The user dismisses the slab from the hand with a right-click of the mouse</li>
<li>The selection mode is stuck on.</li>
</ul>
<p>What is happening is that when the user presses <code class="language-plaintext highlighter-rouge">Ctrl+x</code>, Unity fires an event for Cut and for the selection modifier (the <code class="language-plaintext highlighter-rouge">x</code> key by default), but it doesn’t fire when the key is released.</p>
<p>I’ve prodded a bunch of settings, but it does seem to be a bug. My plan is to give up on <code class="language-plaintext highlighter-rouge">shortcutKeysConsumeInput</code> and handle the collision cases myself. This sucks, but we really need to get this fixed so we can get back to everything else on the todo list.</p>
<p>It’s rather late here, so I’m gonna get to bed. Hopefully, this isn’t too painful to deal with tomorrow.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 411 - The Steam UI choice that is war crimes2023-08-29T23:24:43+00:00http://www.techsnuffle.com/2023/08/29/talespire-dev-log--the-steam-ui-choice-that-is-war-crimes<blockquote>
<p>This post is about releasing for Mac</p>
</blockquote>
<p>We were ready dammit. We had done all the stuff. The build was signed and tested. We had been open about goals and limitations, and we were counting down the minutes to release time.</p>
<p>We. Were. Ready.</p>
<p>Take a look at this:</p>
<p><img src="steamUiExpectation.png" alt="" /></p>
<p>Now, obviously, we all know from the title that something here is wrong. But take a second, clear your mind, and pretend that the world isn’t a vortex of discount madness. What should this UI do?</p>
<p>To give you a moment and to give some distance before the answer, here are two distractions.</p>
<p><img src="/assets/videos/slabInLibrary.gif" alt="" /></p>
<p>Hey, look! Slabs in the library. We’ve been meaning to do this for a while. Content-packs in TaleSpire will be able to contain slabs. That will let us ship bigger building blocks for speedy building.</p>
<p><img src="/assets/videos/roomDragTest.gif" alt="" /></p>
<p>Distraction the 2nd. This is a test I’ve been wanting to do for a bit. It doesn’t feel as good as it should, as it feels like it should have smarts to handle intersecting rooms. However, it was a neat test, and it might find its way into the Symbiote API.</p>
<p>ALRIGHT. So you’ve decided what a sane system would do. Now let’s see what we saw 20 minutes before we shipping the Mac release:</p>
<p><img src="/assets/videos/steamUiWat.gif" alt="" /></p>
<p>Yeah… YEAH.</p>
<p>Twenty minutes before release, we find out that we can’t choose not to ship on Intel Macs, even though none of the Steamworks documentation I’ve found says that is a restriction, and the UI definitely doesn’t suggest it.</p>
<p>As you folks know, we chose not to ship on Intel Macs as we don’t have the resources, as a team, to make the game run well on those machines and support any idiosyncrasies that come up.</p>
<p>Suddenly, we are missing our deadline because of… assuming Steamworks checkboxes work as checkboxes.</p>
<p>If only someone had invented some kind of UI gizmo for this very situation. Maybe even inspired by those old buttons on radios.</p>
<p><img src="assets/images/SteamUiSane.png" alt="" /></p>
<p>Ah, a man can dream.</p>
<p>Back in reality, we were in trouble. We either had to cancel or try and get a valid build ready for intel that day. Naturally, we had to at least attempt the second option.</p>
<p>Telling Unity to build for Intel Macs is simple enough, but when I tried to run it on my 2015 Macbook, it quickly reminded me that we have native library dependencies, too.</p>
<p>Some of those libraries are our own, so we rebuilt those as universal binaries for those. An easy fix. But for some, we didn’t have the binaries required.</p>
<p>For one, we were able to find the Intel version on Nuget, and I tried to tell Unity which one was for what architecture. Sadly, I couldn’t get that to work. I’m 100% sure it was my fault, but we were short on time, and I was stressed. Almost as a joke, I googled how to combine two dylibs into a universal dylib, and bloody hell, <a href="https://ss64.com/osx/lipo.html">there is a way</a>. <a href="https://ss64.com/osx/lipo.html">Lipo</a> is a program that lets you do exactly that!</p>
<p>By simply running <code class="language-plaintext highlighter-rouge">lipo libminiz-intel.dylib libminiz-as.dylib -output libminiz-universal.dylib -create</code>, we suddenly had what we needed. I slapped that into the build and got an intel-compatible build.</p>
<p>Now, we still cannot support these platforms, so we whipped up some UI to inform the user of that fact. We saw Unity’s <a href="https://docs.unity3d.com/ScriptReference/SystemInfo-processorType.html">SystemInfo.processorType</a> property, and the docs even showed how to detect ARM. Perfect. We slapped that in, but it didn’t work as, even though Apple’s M processor range is ARM architecture, <code class="language-plaintext highlighter-rouge">processorType</code> doesn’t return <code class="language-plaintext highlighter-rouge">"ARM"</code> for them. On any typical day, this would be a momentary annoyance, but when the day has already gone so sideways, it’s just an extra kick in the knackers.</p>
<p>Compounding the stress is that a complete clean release build and signing takes a couple of hours, so little issues can really compound.</p>
<p>In the end, as you know, we shipped on Mac a few minutes before midnight in Norway. So we had met our public deadline, but a stiff drink was in order after all that nonsense.</p>
<p>To end, I want to say this to the ether:</p>
<blockquote>
<p>Steam peeps, please invest in radio buttons. I no longer know how to trust, but others could be spared.</p>
</blockquote>
<p>And to the rest of you, have a good one folks :P</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
Scheduling TaleSpire downtime for maintenance2023-08-29T15:26:39+00:00http://www.techsnuffle.com/2023/08/29/scheduling-talespire-downtime-for-maintenance<p>Hi folks.</p>
<p>Tomorrow at 11:00 a.m. UTC, we will be undertaking a significant upgrade of our backend infrastructure. This will be our biggest one for a long time, so we are scheduling four hours for the maintenance.</p>
<p><a href="https://www.timeanddate.com/worldclock/fixedtime.html?msg=TaleSpire+downtime+for+backend+upgrade&iso=20230830T13&p1=187&ah=4">Click here to see the time in your timezone</a></p>
<p>As mentioned, this is a big one for us. Behind the scenes, we’ve been working on the changes we would need to not only scale the game to much larger numbers of concurrent players but to improve our uptime while doing so. Tomorrow, we land a chunk of those improvements. With these changes, we’ll be able to perform backend upgrades just as significant as tomorrow’s one, with zero downtime.</p>
<p>See you on the other side!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4102023-08-23T23:18:24+00:00http://www.techsnuffle.com/2023/08/23/talespire-dev-log-410<p>Well, we are nearly there. Depending on where you are, we are shipping Mac support today or tomorrow. I’ve just signed a release candidate build and have it primed on our pre-release Steam branch.</p>
<p>Here’s a fun shot from our recent testing. TaleSpire running on Windows, Mac, and Linux, side-by-side and in the same board.</p>
<video controls="" autoplay="" loop="">
<source src="/assets/videos/threePlatforms.webm" type="video/webm" width="100%" />
</video>
<blockquote>
<p>There simply must be a “three men walk into a pub” joke here somewhere, but I can’t find it.</p>
</blockquote>
<p>And here’s a slightly clearer shot</p>
<p><img src="/assets/images/threePlatforms.png" alt="" /></p>
<p>In the last week, I’ve been working on:</p>
<ul>
<li>Platform-specific keybinding support</li>
<li>Updating the cut volume to work with all tiles and props</li>
<li>Tweaks to improve reconnection behavior</li>
<li>Allowing Symbiotes to subscribe to creature selection changes.</li>
</ul>
<p>These will all ship alongside the Mac release, so even pure Windows parties will get something out of it :)</p>
<p>That last one can potentially allow some very cool Symbiote integrations as creators can now react to the player switch between their characters in real-time. It’s pretty speedy with group selections too.</p>
<video controls="" autoplay="" loop="">
<source src="/assets/videos/symbSubSelection.mp4" type="video/mp4" width="100%" />
</video>
<p>Right, I best get ready and rested for the release.</p>
<p>Have fun folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Update - Focus and Effect2023-08-16T16:41:48+00:00http://www.techsnuffle.com/2023/08/16/talespire-update--focus-and-effect<p>In this update, we have a fix and a feature.</p>
<p>GMs can now right-click on rulers made by other players and turn them into AOEs.</p>
<p>We have also fixed a bug where switching back to the TaleSpire window could cause ‘jumping’ due to inputs that occurred while TaleSpire wasn’t focused. Thanks to all the folks who provided info on this, it’s been a long-term bug, and it’s good to be squashing this one finally.</p>
<p><img src="/assets/videos/playerAoe.gif" alt="turning player rulers into aoes" /></p>
<p>That’s it for today. We’ve got lots more coming this year, but we’ll save that for the dev-logs.</p>
<p>Ciao</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 409 - Jumping through hoops for Apple2023-08-09T15:09:51+00:00http://www.techsnuffle.com/2023/08/09/talespire-dev-log-409<p>Heya folks,</p>
<p>I’m still around and still pushing toward the macOS release. The last big technical hurdle before we can release is the digital equivalent of paperwork we must do for Apple to allow the game to run.</p>
<p>On Windows and Linux, you can (pretty much) just download programs and use them. Not so on Mac[0]. Apple requires that apps are “signed” and “notarized.”</p>
<p>Roughly speaking, “signing” is a process that makes it easier to know who created an app and to validate that the contents of the app are unchanged. “Notarization” is where Apple scans the app for common vulnerabilities and checks that the app was signed correctly. Apple can then keep that information around so they can check that the app you have downloaded hasn’t been messed with.</p>
<p>This wasn’t always the case, but they’ve got stricter over the years, and now, even though we aren’t using their App Store, we need to do these things.</p>
<p>So I’ve been trying to understand this process so I can add it to our build pipeline, and it has been a draaaag!</p>
<p>I started off looking into ways we could do the signing and notarizing from Windows (as our build agent is a Windows machine), and while there are some <a href="https://github.com/anchore/quill">interesting options</a> we would have to read all their code to do a security review to make sure they aren’t doing sketchy things with the certificates.</p>
<p>Given that we need to ship as soon as possible, we put that aside and looked into how to do this on Mac. We could buy another Mac, put it on one of our homes, and set up our build infrastructure talk to it. This makes us reliant on that person’s power and internet working and would complicate things when they go on holiday. We could rent a Mac in a data center, but that is a non-trivial cost compared to other bare-metal servers. For now, we chose to use a <a href="https://github.com/features/actions">github action</a> instead.</p>
<p>Github actions are, as always, just code running on someone else’s computer, but it is quite convenient. We can choose to run on a Mac and then define a number of steps to run our scripts. We can also easily drop this if we decide to move to another approach. Borodust set all this up for us. With that, I just needed to make the shell script that would (finally) jump through Apple’s hoops.</p>
<p>Now Apple documentation is a special beast. Whereas Microsoft documentation specializes in blunt unhelpfulness via tautology, Apple has seemingly assembled a team of lawyers and psyche-warfare specialists to remove anything that would allow too much certainty in your life. Apple forums are generally unhelpful, and due to being more of a walled garden, stackoverflow simply has fewer people posting Apple stuff.</p>
<p>It took me an embarrassingly long time to work out why a certificate I had installed and validated was not working (TLDR: it needed to be made from my signing request, not Ree’s). I take some of the blame, but I could install the certificate, right-click on it and validate it for code-signing, and the Mac said, “Yup, this is good to go.” But then, when trying to use it, the Mac would say, “Oh, I can’t find that certificate.”</p>
<p>I don’t have the skill to explain how life-sapping debugging Apple shit is, but if you know an Apple dev, simply mention signing issues while looking in their eyes, and you’ll see the hollow space where hope used to be.</p>
<p>ANYHOO, I finally got it going. If you ever need to do something similar, I’ve put our script <a href="https://gist.github.com/baggers-br/58f103f375778176d36a8d1025138d6e">online here</a> so you can use it for inspiration [1].</p>
<p>I’m now testing it. Hopefully, this part is almost over.</p>
<p>Once it is, we should be able to give you a release date for Mac. My hope would be only a week or two later. That would give us time for a Beta and to deal with anything else that comes up.</p>
<p>Alright, enough of all that! I hope you are doing well or have something nice on the horizon.</p>
<p>Until next time,</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] Yeah, of course, you can change settings so that it only tells you it’s not trusted and then forces you to go allow the program. But that isn’t a good experience and is beyond the skill level of many casual computer users.</p>
<p>[1] At the time of this dev log going out, the script is still being tested. I’ve left a warning comment on the gist and will remove it when it works.</p>
TaleSpire Dev Log 4062023-07-27T09:30:08+00:00http://www.techsnuffle.com/2023/07/27/talespire-dev-log-406<p>Heya folks.</p>
<p>I’ve narrowed the macOS issue down to something happening to the library we use for the Symbiotes web view.</p>
<p>The library itself seems to be innocent, however. My current assumption is that our build process accidentally does <em>something</em> to the library that stops it from loading. The good part, of course, is that we now know where the problem is, so all that is left is the “why.”</p>
<p>That was a bit of a short update, so let’s have a look at something unfinished Chairmander and I have been looking at between other jobs. Symbiote Login Issues.</p>
<p>The speedy explanation is: Lots of websites use other services to handle their login. That often requires some kind of popup where you log in to that service (e.g. google), and then you get redirected back to the site you cared about.</p>
<p>We didn’t have support for that in Symbiotes, and so creators have hit a bit of a roadblock.</p>
<p>Luckily Vuplex (the web view library we use) has a hook for sites trying to open popups, so I hacked together a little test.</p>
<iframe class="video" src="https://www.youtube.com/embed/LJPKTJ8LVwo" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<p>The popup has some minor limitations compared to regular Symbiotes. But for login, this will probably do the job just fine. It still needs a bunch of testing and likely some tweaks, but it looks like even more useful Symbiotes will be coming soon!</p>
<p>I’m gonna grab a coffee and get back to it.</p>
<p>Thanks for stopping by.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4052023-07-26T11:51:12+00:00http://www.techsnuffle.com/2023/07/26/talespire-dev-log-405<p>Heya folks!</p>
<p>Coding days like yesterday are great. Issues are clear and don’t get weird when prodded, and fixing them is smooth.</p>
<p>I fixed up a bug around the positioning of bookmark names, font issues on Mac, some bugs on scene transition, and a mistake in our Mac build process.</p>
<p>Today my focus is on the Mac builds produced by our build server. The first issue was that I was putting the assets in the wrong place, which was an easy fix. It also let me dip my toes into writing Jenkins files which was… well, I won’t say nice, but it’s good for me to do!</p>
<p>Now I’m looking into a Mac bug where it seems like the root camera is null. Oddly, I’ve not had this in any of my Mac testing so far. It’s likely that I’ve only been deploying dev builds though so this may be specific to release builds. I’m gonna check that first. Hopefully, this is simple.</p>
<p>Alright, I’m too excited to keep writing. I gotta get back to this.</p>
<p>Seeya!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4032023-07-25T09:14:26+00:00http://www.techsnuffle.com/2023/07/25/talesprire-dev-log-403<p>What the hell. Days go too fast. Alright, let’s get caught up.</p>
<p>We are getting the Mac build ready. Testing got delayed by several days because the delivery of the m2 Mac mini took longer than quoted. That was disappointing, but eh, it happens. It was also a good opportunity to work on some bugs.</p>
<p>We’ll have a patch out today containing a group of little fixes and a fix for group selection. The group selection issue was very embarrassing as it was much worse than I expected. It was broken in a couple of ways. First off, the stencil buffer wasn’t being cleared, which meant every lasso shape drawn would effectively be combined with every lasso shape drawn that session.
Next, the depth buffer was empty, which resulted in group selections selecting creatures through walls.
Both of these things got in when we switched out where we were rendering to allow independent scene and UI resolutions.</p>
<p>I went a little crazy looking at the empty depth buffer bug, as it really looked like it shouldn’t have been the case. We were rendering into a RenderTexture with a correctly set up depth buffer, and it was attached to the camera’s <code class="language-plaintext highlighter-rouge">targetTexture</code> property (which is, in our case, how Unity wants you to render into these targets).</p>
<p>Ree saved me by making a minimal test case and finding out that if we ran the commands <a href="https://docs.unity3d.com/ScriptReference/Rendering.CameraEvent.html">AfterFinalPass</a> instead of <a href="https://docs.unity3d.com/ScriptReference/Rendering.CameraEvent.AfterEverything.html">AfterEverything</a> the depth buffer was still populated. Unfortunately, this is before the image effects are run, which means things like depth of field affect the lasso.</p>
<p><img src="/assets/images/dofGs.jpg" alt="depth of field interacting poorly with lasso" /></p>
<p>It’s not game-breaking, and that’s not the usual angle to be commanding larger groups from, so we aren’t going to hold back the patch for this issue, but we will resolve it in a future patch.</p>
<p>Next up on the Mac front, I’ll look into a weird bug affecting overlay text positioning and our Mac build pipeline.</p>
<p>Have a good one folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 4022023-07-14T16:45:08+00:00http://www.techsnuffle.com/2023/07/14/talespire-dev-log-402<p>Hey folks,</p>
<p>As the macOS build is rapidly approaching being shippable, I decided to spend the day looking back into how Steam handles multiplatform releases.</p>
<p>I set up the requisite depots and, after a bit of poking around, got the build script how I’d like it. That let me do a manual push, and so now we have:</p>
<p><img src="/assets/images/macSteamInstall.png" alt="TaleSpire installing on mac" /></p>
<p>Of course, this is not available for everyone yet. We still have a bunch of things to do before this can go public. Including but not limited to:</p>
<ul>
<li>Changing our keybinding handling so we can make sensible platform-specific defaults</li>
<li>Tracking down an error that happens on exit</li>
<li>Performing heavy testing and fixing any critical bugs that appear</li>
<li>Fixing up Symbiote issues</li>
<li>Adding in-game info to inform players about the limitations of the current Mac build</li>
<li>Updating the info on Steam and prepping for release.</li>
</ul>
<p>Due to not knowing what bugs we will find, we are not giving a release date yet. But if all goes well, it will be soon.</p>
<p>Big thanks to all the macOS folks who have been so patient with us. We hope it’s worth the wait!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 401 - Macinations2023-07-13T17:02:00+00:00http://www.techsnuffle.com/2023/07/13/talespire-dev-log-401<p>Heya folks!</p>
<p>Today I come with a screenshot.</p>
<p><img src="assets/images/macNoErrors.png" alt="mac build without errors" /></p>
<p>That screenshot shows the macOS build working without errors.
This is great news in that we are again one more step toward shipping this blighter.</p>
<p>The tradeoff is that, for the first release, we (bouncyrock) will need to build the assets separately for Windows and Mac. That is very normal, but it’s not something we want for modding. We want you to be able to build your mods once and have them work on any platform we support.</p>
<p>To that end, I think we will need to make a custom asset format before we ship creature, tile, or prop mods. That’s fine though. I was expecting that to be the case. Another upside of this is that we untie our assets from the Unity version, so upgrades will be less risky for us in the future[0].</p>
<p>Once we have our own format, we can use that in place of Unity’s asset-bundles and remove these separate Windows/macOS asset builds.</p>
<p>Alrighty, time to go get these changes merged in. Hopefully, next week we can get back to looking at Steam’s process for shipping games on macOS.</p>
<p>Until then,</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] The worry is that a new version might require that assets be rebuilt to use some feature, and once the community makes mods, that becomes nearly impossible to ensure. This makes us less likely to take that risk, which in turn means TaleSpire can miss out on fixes and feature improvements in later versions of Unity.</p>
TaleSpire Dev Log 4002023-07-08T20:41:02+00:00http://www.techsnuffle.com/2023/07/08/talespire-dev-log-400<p>Hey folks,</p>
<p>Well, the Symbiotes release was great! There was one simple bug we needed to patch, but aside from that it’s been going very well. We had crossed 10k Symbiotes downloads before Friday ended and it’s been a delight to start seeing community symbiotes rolling in.</p>
<p>With Symbiotes out I could start looking at other things and so I upgraded Unity to see if the macOS graphics fix worked. The answer is: YES!</p>
<p><img src="/assets/images/macWaterFixed.png" alt="TS on macOS with working water" /></p>
<p>For context this is what water looked like before.</p>
<p><img src="/assets/images/macWaterBroken.png" alt="how water looked before the fix" /></p>
<p>You might have spotted the little red errors in the first screenshot, so clearly not everything is working, but it’s a great start.</p>
<p>The plan is that @Chairmander and I will begin thoroughly testing the Windows build on Monday. If that passes then we will hopefully get this merged very soon.</p>
<p>The little errors mentioned above are shader related. I’ve started trying to hunt them down but I’m not sure what they are yet. Once they are sorted we’ll get macOS builds set up in our CI and start the testing process.</p>
<p>I’m not happy with the performance on m1, but it might be worth it to mac players to ship sooner and fix the perf issues later. We’ll see.</p>
<p>macOS support has been hanging over me for nearly a year now and I can’t wait to get this out. Let’s hope it’s soon.</p>
<p>I’ve got plenty more little things in the works but I’ll talk about them another day.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3992023-07-05T01:30:57+00:00http://www.techsnuffle.com/2023/07/05/talespire-dev-log-399<p>Heya folks,</p>
<p>No surprise here. We are fixing bugs and getting ready to push Symbiotes public. The focus is naturally the mod.io integration.</p>
<p>Here is the process of installing and using Symbiotes in our latest internal builds</p>
<iframe class="video" src="https://www.youtube.com/embed/hOmvycKin6o" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<p>I don’t want to jinx it, but a release looks pretty close!</p>
<p>Hope you are doing good,</p>
<p>Seeya.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3982023-07-04T00:01:48+00:00http://www.techsnuffle.com/2023/07/04/talespire-dev-log-398<p>Hey folks,</p>
<p>As expected, I’ve been working on the community-mod browser so we can have an in-game way to install Symbiotes when we make that feature public this week.</p>
<p>Here is a little clip of that in action:</p>
<p><img src="/assets/videos/modioSymb0.gif" alt="gif of installing and uninstalling symbiotes using the in-game integration" /></p>
<p>I’m currently fixing smaller bugs and UX things, and that’s what I’m going to get back to now.</p>
<p>Have fun folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3962023-06-25T17:34:20+00:00http://www.techsnuffle.com/2023/06/25/talespire-dev-log-396<p>Heya folks,</p>
<p>Work has continued integrating mod.io (so we can download Symbiotes from within TaleSpire), albeit sporadically, as I’ve had a bunch of folks visiting for midsummer (no ritual deaths involved, in case you wondered :p). The first few days of this week are gonna be a bit hectic for me, so we are shooting for Symbiotes to come out of Beta sometime next week (as in between the 3rd and 7th). I am always hesitant to give dates, as life has a habit of kicking off when I do. But hey, it’s been a while, and I’m ready to make that mistake again!</p>
<p>As you’ve probably seen, we’ve also been adding logs and gathering details on a bug that has given some folks sporadic difficulties logging in. We’ve narrowed it down to the request that fetches the game-environment details[0], but we are still working on minor fixes to address the cases that have been reported.</p>
<p>I hope you’ve had a good weekend.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] The game-env tells TaleSpire the address for the backend along with some extra stuff like maintenance times</p>
TaleSpire Dev Log 3952023-06-22T08:05:40+00:00http://www.techsnuffle.com/2023/06/22/talespire-dev-log-395<p>Heya folks,</p>
<p>Currently, I’m looking into whether we can have mod.io support ready in time for Symbiotes leaving beta.</p>
<p>I know we won’t have the in-game upload done for that time. But what if we allow uploads on the website and downloads in-game? That still should be a much better experience for regular players.</p>
<p><img src="/assets/images/symbBrowser0.png" alt="in-game symbiotes browser" /></p>
<p>It took minimal effort to wire up the community-mod-browser to show the symbiotes, so now I need to handle the subscriptions to mods themselves.</p>
<p>mod.io does have its own mechanism for subscribing to mods, but we would prefer to handle that for a couple of reasons:</p>
<ol>
<li>It makes the code that handles mod.io more sharable with community-run repositories</li>
<li>We can let you decide who knows what you subscribe to.</li>
</ol>
<p>Number two feels important. With that TaleSpire can give you the option of whether to only store your mod list locally or on our servers so that it is automatically available wherever you are playing. If we can provide that choice, we would prefer to.</p>
<p>Right, that’s all from me for today. This week is pretty hectic as I have a bunch of guests over for midsummer, and I need to make sure I don’t just hide away coding :p</p>
<p>Have a great one folks. More coming soon!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3942023-06-16T23:23:01+00:00http://www.techsnuffle.com/2023/06/16/talespire-dev-log-394<p>Heya folks,</p>
<p>Yesterday we were getting ready for the Symbiotes Beta. That involved writing up the release texts, recording gifs and all that jazz. We also tested the build and found a few more bugs that needed investigating, so that’s a pain, but better than we find it than you folks :P</p>
<p>The new backend made it through the first 24 hours just fine, which is a lovely first milestone. It’s great to start seeing the ebb and flow as people play across various timezones. That’s only going to get more useful over the weeks and months ahead.</p>
<p>That’s all for me for now. Back to the bugs!</p>
<p>Ciao!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3932023-06-15T15:54:22+00:00http://www.techsnuffle.com/2023/06/15/talespire-dev-log-393<p>Heya folks, today is going very well so far.</p>
<p>You may have seen us talk about switching to our new backend infrastructure today. That went relatively smoothly, with only a few things to tweak for next time.
I don’t want to steal @Borodust’s thunder by going into all the details of what he is doing, but I certainly sleep more easily these days seeing what is being built. We are on the road to a much more maintainable system, and the news insights we have into our servers mean we finally have some tools to start doing a decent job of seeing problems coming rather than reacting to everything. Of course, it’s still early days, but I’m naturally pumped about it.</p>
<p>Next up, Ree, myself, and Chairmander have all been merging in bits and pieces for Symbiotes. I’ve squashed the bugs I needed to handle before the beta, Ree merged in the UI, and Chairmander has the latest improvement to the docs (and supporting code) based on your feedback thus far.</p>
<p>I’ve got a candidate build running through our build system now, so I will go kick the tires on that and see how it goes.</p>
<p>Have a good one!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3922023-06-14T09:18:33+00:00http://www.techsnuffle.com/2023/06/14/talespire-dev-log-392<p>Whoops :D</p>
<p>I released a little more than I expected to yesterday. Let’s talk about that.</p>
<p>I aimed to release some UI we could use during maintenance to make the process nicer for players. However, I forgot to disable a few unfinished things before shipping, so you folks got a little peek.</p>
<blockquote>
<p>Big thanks to Ree, who cleaned up my mistakes as I was on the road when you alerted us to this issue.</p>
</blockquote>
<p>The first thing that you might have seen is this:</p>
<p><img src="/assets/images/communityMods0.jpg" alt="community mod browser" /></p>
<p>This is the first version of the ‘community mod browser.’ It’s where you’ll be able to find community-made slabs, symbiotes, etc, hosted on mod repositories. It’s currently only pulling from our mod.io repo, but that will be easy enough to expand once the core is complete.</p>
<p>As some of you saw, you can click on a slab to bring it immediately into the game, making the flow of building with community creations even smoother.</p>
<p>Aside from some more testing and adding separate categories for symbiotes and boards, this part is relatively close to shipping. The reason we haven’t is due to the next thing we accidentally shipped.</p>
<p><img src="/assets/videos/slabShareUI.gif" alt="ui for uploading slabs from inside TaleSpire" /></p>
<p>This is the unfinished UI for sharing slabs from inside TaleSpire. Very soon, it will be as easy as selecting your creation and hitting a few buttons to share your slabs with the world. We are starting with uploading to mod.io but then want to expand to uploading to other mod repositories</p>
<p>That said, it’s clear why this hasn’t shipped yet. The camera controls are borked, the UI is unfinished, and there is no way to set up your mod.io account.</p>
<p>There is <em>one</em> part that works, however. You’ll notice that, when entering the upload UI, TaleSpire isolates just the selection. This is very handy for seeing what is selected, which can be tricky in busy scenes and can help avoid sharing more than you mean to.</p>
<p>This “isolation mode” is a helpful tool in its own right, so we have made it available in the building tools menu.</p>
<p><img src="/assets/videos/isolationMode1.gif" alt="isolation mode" /></p>
<p>Also, this feature already works despite being an accidental release, so we’ve left it in. At least that way, you folks get something useful out of my screw-ups!</p>
<p>Today I’m back squashing bugs in Symbiotes. I have an annoying exception occurring when switching to and from dev-mode.</p>
<p>Catch ya in the next dev log.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3912023-06-13T11:31:58+00:00http://www.techsnuffle.com/2023/06/13/talespire-dev-log-391<p>Morning all!</p>
<p>I waited until today to put out the maintenance ui update as I wanted to do a bit more testing. I’m prepping that release right now, and it should be out within the hour.</p>
<p>We are also scheduling downtime on Thursday so we can move to our new server infrastructure. We are super excited about this as it will give us far more stability, insight into our systems, and new ways to scale.</p>
<p>You can find the maintenance time in your timezone at this link: <img src="https://www.timeanddate.com/worldclock/fixedtime.html?msg=TaleSpire+Server+Maintenance&iso=20230615T12&p1=187&ah=1" alt="maintenace time" /></p>
<p>I’ve got to rush off now to work on the release, but here is a little clip of the maintenance UI before I go.</p>
<iframe class="video" src="https://www.youtube.com/embed/jJsAwLBk4zU?feature=oembed" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<p>Ciao!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 390 - Symbiotes Documentation2023-06-12T16:35:37+00:00http://www.techsnuffle.com/2023/06/12/talespire-dev-log-390--symbiotes-documentation<p>Heya folks,</p>
<p>Part of getting ready for the first Symbiotes release is finalizing the documentation. Or rather, I should “was” as that is now done!</p>
<p>We know that a bunch of you in the community are developers who might be interested in peeking at this even before the feature is available, so we have put the docs and examples public early.</p>
<p>You can find the documentation at <a href="https://symbiote-docs.talespire.com/">https://symbiote-docs.talespire.com/</a> and the examples are at <a href="https://github.com/Bouncyrock/symbiotes-examples">https://github.com/Bouncyrock/symbiotes-examples</a></p>
<p>The documentation is also a valid Symbiote. That can be found here: <a href="https://github.com/Bouncyrock/symbiotes-docs">https://github.com/Bouncyrock/symbiotes-docs</a></p>
<p>We have intentionally not added calls that modify board or campaign state in this first API version. That allowed us to avoid the impact of such changes on the backend. However, we will be significantly expanding the API into those areas in the future.</p>
<p>Major props go to @Chairmander, who has been driving the entire documentation process. Left to my own devices, the first version would have been much more bare-bones!</p>
<p>You should also see a little patch going out in the next 12 hours. This will be the patch that adds UI that we will use during future maintenance periods.</p>
<p>Well, that’s all for today!</p>
<p>Ciao</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3892023-06-10T10:44:58+00:00http://www.techsnuffle.com/2023/06/10/talespire-dev-log-389<p>Morning all!</p>
<p>It’s Saturday, so this will just be a short log.</p>
<p>Yesterday I completed the implementation of the maintenance UI. It needs a little more testing, but I expect to be shipping that Monday. This won’t affect you yet, but we’ll get to use it during the next server upgrade, which is also coming very soon. Pavel has done fantastic work taking the very basic thing I had built and putting us on the road to serious infrastructure that will stand up to the tests ahead.</p>
<p>I also got to chat with some lovely folks in the broader ttrpg community both about Symbiotes and what they were up to. it’s always a blast to see how much is happening out there.</p>
<p>Alright, time to get some work done in the garden.</p>
<p>Ciao!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3882023-06-09T09:58:03+00:00http://www.techsnuffle.com/2023/06/09/talespire-dev-log-388<p>‘Allo folks.</p>
<p>Yesterday I continued working on the update UI and some details with Symbiotes that drag us that last little distance towards release.</p>
<p>Symbiotes has to be the most feature with the most creep we’ve ever had. It started with “let’s just get a web-view in here,” evolved to “let’s add a basic API in here so creators can interact with TS” to a place where we were really drilling down into the details of the programmer’s experience, writing extensive documentation, and making something we can truly support and grow over the years[0].</p>
<p>I’m much happier with this result than I would have been with the original plan, but I regret how much time it took. However, we are wrapping up the last few outstanding things:</p>
<ul>
<li>Finishing the v1 UI</li>
<li>Prodding the legal folks to get the TOS updated.</li>
<li>Fixing little bugs and annoyances as testing reveals them</li>
</ul>
<p>We have also been chatting and working with other creators to be sure that the first version is helpful for real projects. I’ll talk more about that another day.</p>
<p>Anyhoo, back to what I was actually doing.</p>
<p>First off, I expanded the manifest to support icons for the symbiotes. The manifest is a file you write to tell TaleSpire about the symbiote, including what features to inject.</p>
<p>Here is a sample manifest:</p>
<p><img src="/assets/images/manifest0.png" alt="manifest with some details on what does what and an indicator showing that I was working on icon support yesterday" /></p>
<p>As you can see, I was focused on icons. We currently support a couple of kinds:</p>
<ul>
<li>A 64x64 pixel icon which is used for the symbiote in menus and such</li>
<li>A smaller, greyscale icon that will be used for notifications</li>
</ul>
<p>We can add more size options in the future as we need those, whether for nicer integration in mod-stores or higher DPI displays. But for now, this will work just fine.</p>
<p>I also fixed a bug in the formatting of <code class="language-plaintext highlighter-rouge">talespire://</code> URLs generated by symbiotes. Let’s take a second to talk about that feature.</p>
<p>You folks probably have already seen that external programs can do certain things via <code class="language-plaintext highlighter-rouge">talespire://</code> URLs. Importing boards, setting the dice tray, and so on have URLs that integrate with TaleSpire. It’s super helpful, and we wanted you to have that too, so we allow you to generate a URL specifically for your symbiote like this:</p>
<p><img src="/assets/images/genSymbUrl0.png" alt="chrome dev tool console showing a call to TS.urls.createUrlPrefixForThisSymbiote" /></p>
<p>You can then put whatever text you like after the final slash (as long as it’s a valid URL) and it will be delivered to your symbiote when the URL is processed.</p>
<p><img src="/assets/videos/sendSymbUrl0.gif" alt="gif showing a url message being delivered to a symbiote" /></p>
<p>The console you are seeing is part of the dev tools in Chrome. When you enable dev mode in the Symbiote settings, you can connect to and debug your symbiotes trivially. I’ll talk more about this in another dev log.</p>
<p>Right! I gotta get back to work, so I’ll stop writing for now.</p>
<p>Ciao.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] Not just supporting more features but supporting entirely different kinds of symbiotes.</p>
TaleSpire Dev Log 3872023-06-08T07:04:12+00:00http://www.techsnuffle.com/2023/06/08/talespire-dev-log-387<p>Hi folks,</p>
<p>I’ve been enjoying a dip out of public view for the last month. I had two weeks off (which I spent fixing up stuff at home), and then I’ve been quietly plugging away at TaleSpire. It’s been nice, but I’m back to the dev logs again.</p>
<p>A <em>lot</em> has been happening, but today’s dev log is short to get me back into the rhythm.</p>
<p>Yesterday was spent working on UI, which will show before and during maintenance. We’ve got a lot of stuff to ship this year, both front-end and back-end, and we want to work towards that being less janky.</p>
<p>It’s nothing exciting, but here is the ticker that appears periodically before maintenance to let you know what is upcoming:</p>
<p><img src="/assets/images/maintenance0.jpg" alt="in-game upcoming maintenance ticker" /></p>
<p>The ticker shows briefly once an hour for the few hours leading up to the event. It transitions to a permanent message bar in the last 20 minutes and then finally into a modal window for the duration of the update itself.</p>
<p>Naturally, there is also UI on the login screen.</p>
<p>Symbiotes are dangerously close to being released. We will have an alpha as soon as we can pull together the last UI and legal bits. If you are wondering how it took so much longer than we were saying previously… yeah, that’s a tale too. I’ll get that down in a dev-log in this next week.</p>
<p>Anyhoo, I best get back to it.</p>
<p>Ciao</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3792023-03-08T09:56:07+00:00http://www.techsnuffle.com/2023/03/08/talespire-dev-log-379<p>Heya folks!</p>
<p>I’m back again with an update on what I’ve been touching this last week.</p>
<h3 id="symbiotes">Symbiotes</h3>
<p>Chairmander and I kicked off with a long meeting to design the manifest. We took some inspiration from other projects but ensured the file grew naturally in step with what Symbiote features are being used. This amounted to talking through user stories, though we didn’t think explicitly about that then.</p>
<p>After the designing, it was time to work!</p>
<p>I knew quickly that if a Symbiote failed to load, I wanted some UI in the panel to show that. I’ve previously mentioned that I wanted to support different kinds of Symbiote (as in, not just web-based ones), and this felt like an excellent place to use another type.</p>
<p>Knowing what I wanted and getting it were two different beasts, though. I spent many hours going in circles until I found an approach I liked. The details of the solution aren’t particularly exciting. Still, the upshot is that we can now support not just different user-made symbiotes, but we can technically support hot-loading the infrastructure for entirely different kinds of Symbiote. This won’t matter for a while, but I’m happy to have that worked out.</p>
<p>We are still improving the API Symbiotes have access to. As of today, we have calls for the following:</p>
<ul>
<li>fetching the initiative list and the currently active member</li>
<li>sending chat messages both as players and as creatures</li>
<li>putting dice into the tray</li>
<li>fetching info about the currently active ruler</li>
<li>fetching a slab string based on the current selection</li>
<li>putting a slab into the hand (switching into build mode if necessary)</li>
<li>unpacking slab strings to binary representation</li>
<li>packing binary slab representation to slab string</li>
<li>getting the data size of a given slab string</li>
<li>sending a network message to the same Symbiote running on other clients</li>
<li>fetching the ids of any parties</li>
<li>fetching the ids of creatures in a given party</li>
<li>fetching info about the active campaign</li>
<li>fetching the board list for the active campaign</li>
<li>fetching a list of the bookmarks in the active campaign</li>
<li>fetching the distances units for the active campaign</li>
<li>fetching the names of creature stats used in the active campaign</li>
<li>fetching a list of the players in the campaign</li>
<li>fetching a list of the players in the board</li>
<li>fetching the id of your player</li>
<li>checking if a given client-id is yours</li>
<li>fetching info about a given player</li>
<li>fetching a list of the connected clients</li>
<li>fetching the id of your client</li>
<li>checking if a given client-id is yours</li>
<li>fetching info about a given client</li>
<li>fetching the list of loaded asset-packs</li>
<li>fetching info about the asset pack (including all tile/prop/creature info)</li>
<li>fetching the list of unique creatures</li>
<li>fetching a list of creatures owned by a given player</li>
<li>fetching a list of currently selected creatures</li>
</ul>
<p>We also let you subscribe to events such as:</p>
<ul>
<li>unique creature created/deleted</li>
<li>dice roll results</li>
<li>slab copy events</li>
<li>asset pack loaded/unload events</li>
<li>client join/left board events</li>
<li>player joined/left campaign events</li>
<li>player permissions changed events</li>
<li>initiative list changed events</li>
</ul>
<p>Next up, I’ll be experimenting with exposing a picker tool that sends the kind and id of the picked <em>thing</em> to the Symbiote.</p>
<p>We also will be handling documentation generation soon. With that and another big pass on the API, we will be nearing something we can ship.[0]</p>
<h3 id="community-mod-repository">Community mod repository</h3>
<p>I just put this section here to reassure you that slab sharing and mod.io integration have not been forgotten. They have had to wait a bit as pushed to find out what the Symbiote feature needed to be.</p>
<h3 id="performance">Performance</h3>
<p>Ree is working on a cool feature that requires scanning over any tiles or props in a given area. We have some code for this that is used by the hide plane. However, it was not currently Burst compiled due to limitations on how generics work in HPC#.</p>
<p>I changed the logic so that a caller-defined visitor was passed chunks of data. The caller could then return jobs that processed these chunks with the help of some methods on the chunk.</p>
<p>The result was a significant performance improvement to the hide plane and code that could be used in more cases.</p>
<h3 id="gameobject-alternative">GameObject Alternative</h3>
<p>The perf work above got me very motivated for more. So I sat with a pad of paper and started trying to design a jobified alternative to GameObjects we could use in some places.</p>
<p>I’m not looking to cover all the cases GameObjects do, but I want something that is more familiar than ECS (and more compatible with our setup) while being easy to integrate with jobs.</p>
<p>Not much to report here yet, but you’ll be hearing more about this in the future.</p>
<h3 id="physics">Physics</h3>
<p>Next up, I worked on a bug in our physics library helpers which resulted in sphere-overlap checks not working.</p>
<p>My reason for the bug was that, at the time, the library we wrapped had no documentation, and the name of the field I was incorrectly using was bafflingly named[1]. That’s my excuse, and I’m sticking to it :P</p>
<h3 id="macos-news">macOS News</h3>
<p>Some good news on the macOS front. The bug that has been blocking us has been fixed in Unity 2023.2 (which we don’t use) and is in review to be released in 2021.3, which is one we can upgrade to[2].</p>
<p>This is exciting! When that lands, we can finally get a good look at what we have and work out how to get the alpha of macOS support into your hands.</p>
<h3 id="backend">Backend</h3>
<p>Work progresses rapidly on the backend. I was in some discussions about how we will be handling certificates and looked into our options. Nothing interesting to report, but it took some time, so it gets a spot here!</p>
<h3 id="wrapping-up">Wrapping up</h3>
<p>More than ever, my dev-logs fail to capture the breadth of what is going on behind the scenes. Seeing so much moving from backend code stuff to community outreach is a joy.</p>
<p>Until next time!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] You may have noticed that no calls change the state of the game directly or require server requests. This is very intentional for the first version. We will look into state-modifying functions in future updates to symbiotes (an example of such a function would be setting the initiative list)</p>
<p>[1] The field is called <code class="language-plaintext highlighter-rouge">MaxFraction</code>. Normally it holds a fraction of another distance. However, in some cases, this same field holds the distance directly and not a fraction. Very annoying.</p>
<p>[2] Those not in software development might wonder why we can’t just upgrade to 2023.2. The answer is that changes between major revisions can be substantial, so while technically we <em>could</em> switch, it means putting a lot of time into repairing systems broken by those changes. Also, software always has bugs, so we would be trading a known set of bugs for an unknown set, so the time it would take to get back up to speed is uncertain.</p>
<p>It’s very likely that we will make that jump eventually, but we will let you know when, as that will likely stop us from shipping new features for the duration of that transition.</p>
TaleSpire Dev Log 3782023-02-26T02:28:34+00:00http://www.techsnuffle.com/2023/02/26/talespire-dev-log-378<p>‘Allo!</p>
<p>The latter half of the week has been 100% focused on Symbiotes.</p>
<p>In <a href="https://bouncyrock.com/news/articles/talespire-dev-log-377">my last post</a> I talked about starting to make a code generator that would take a JSON spec for the API and generate the C# and JS plumbing code required for communicating between the WebView and TaleSpire. That is now working, and it’s such a help.</p>
<p>It all but removes a class of user errors from the interconnect code as the boring stuff is generated. This also makes having slightly tighter but ugly encodings easier, as you don’t need to update all the plumbing code directly. Also, by generating C# structs, we get type-checking, further reducing the number of ways for me to mess up[0].</p>
<p>Once the generator was working, I was able to write the implementations for about a dozen of the required API functions for the first version in a couple of hours. This really reassured me that it was worth the work.</p>
<p>After this, I merged the slab-repository branch into the Symbiotes one. This is because one of the Symbiote API functions allows you to get the slab string for the current selection, and some of the work on the slab branch made that much easier.</p>
<p>We then ran into an issue we knew was coming: sometimes, the user’s JS can load in before the TaleSpire API has loaded. I added an option to the Symbiote manifest that allows the creator to specify a function that will be called on initialization and updated our setup code to support that.</p>
<p>With this all working, I got a build to Chairmander so he could continue his work, and I doubled back to the C# side of the Symbiotes feature. I spent a chunk of Friday cleaning up code so that all the JS and WebView-specific code lived together. As previously mentioned, I would like to see other kinds of Symbiote in the future, and the cleanup serves that potential future.</p>
<p>The next big todo is working on threading. I also need to get stuck back into slabs.</p>
<p>Anyhoo, that’s enough rambling for now.</p>
<p>I hope you are having a lovely weekend.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] Just as well, I’m a master at breaking code :P</p>
TaleSpire Dev Log 3772023-02-21T20:02:25+00:00http://www.techsnuffle.com/2023/02/21/talespire-dev-log-377<p>Heya folks.</p>
<p>Since we last spoke, I’ve carried on with my main two tasks, slab upload, and symbiotes.</p>
<h2 id="slab-upload">Slab Upload</h2>
<p>Slab upload is going well. It’s always nice when something starts feeling like a feature and not a hack.</p>
<p>The progress isn’t exciting to show, though. It’s been things like:</p>
<ul>
<li>Side-load the slab data into the cache, so it doesn’t have to be redownloaded</li>
<li>Show an hourglass cursor when the slab tool is waiting on a slab to download</li>
<li>Tuning camera behavior so that transitioning in and out of publish mode does what is expected.</li>
<li>Get tabbing between fields working as expected</li>
<li>Subtle visual changes to screenshot view to improve the experience</li>
<li>Add loading graphics to the entries in the community-mod browser to show that thumbnails are downloading</li>
</ul>
<p>And so on.</p>
<p>I’ve still got a list of tickets to get through to get a good first version, but there don’t seem to be any show-stoppers.</p>
<p>The one (happy) distraction has been helping with the Symbiotes feature.</p>
<h2 id="symbiotes">Symbiotes</h2>
<blockquote>
<p>Context: Symbiotes is an upcoming feature allowing community-made mods to dock on the right side of the TaleSpire window.</p>
</blockquote>
<p>Our first version of this feature supports Symbiotes powered by WebViews[0].</p>
<p>We are providing an API that allows communication with TaleSpire. The messages between TaleSpire and the Symbiote travel over a simple interconnect provided by the WebView.</p>
<p>The code on either side of such a bridge needs to match and, from my experience, are places where simple user errors result in extremely annoying bugs. To deal with that, I prefer to write the API specification as a simple document, which is then used to generate the plumbing code for either side.[1]</p>
<p>And so that’s what I’m making. I have a JSON document specifying types, calls to TaleSpire, and the arguments and return types of those calls. I load that document, type-check it, and produce an intermediate tree of objects which describe the bridge.</p>
<p>Next, I’ll write code to walk over that tree to spit out the JavaScript and C# boilerplate code required.</p>
<p>Some of you may be asking, “why aren’t you using <favorite-serialization-library> for this?". The answer is slightly dumb and slightly sensible, and it's "I can make this approach more quickly as I haven't used that library yet." I'm actually quite interested in
[flatbuffers](https://google.github.io/flatbuffers/) for symbiotes and server communication. But that said, I am very used to writing these kinds of generators. I know how to make something that solves our exact problem and do it in a short amount of time.</favorite-serialization-library></p>
<p>Given that we are trying to ship mods quickly and that this approach doesn’t stop us from using a different serialization approach in the future, this is the way we are going for now.</p>
<p>Alright, enough rambling for today. Looking forward to sharing more with you soon.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] In the future, I want to explore supporting other languages/environments.
[1] This is exactly what we do with our server-side API. The API is represented as an erlang data structure, and from that, we generate C# and erlang plumbing code. It’s been a huge boon.</p>
TaleSpire Dev Log 3742023-02-10T16:35:56+00:00http://www.techsnuffle.com/2023/02/10/talespire-dev-log-374<p>Heya folks!</p>
<p>I’m away from my main dev machine today, so no gifs for ya, but there is some progress.</p>
<p>I’ve got to the point where I can select a region of the board, submit it as a slab to mod.io and bring it back into the game. So the essentials are there. It’s still janky, but I know what I need to do.</p>
<p>We’ve also continued working on Symbiotes (our mods that slot into the right-hand side of the game). We’ve got messages passing between mods and the game, so you can do things like send chat messages, set up dice rolls, and listen to the results.</p>
<p>We are busying ourselves experimenting with the API to work out what is clean and practical to get started with. We have more to show, but as that work wasn’t done by me, I’ll let that person write their own dev log about it :)</p>
<p>We definitely are pushing for mods to be out as soon as possible. Expect to see plenty of logs about this in the coming weeks.</p>
<p>Hope you’re doing well.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3732023-02-02T20:41:36+00:00http://www.techsnuffle.com/2023/02/02/talespire-dev-log-373<p>Heya folks!</p>
<p>Last week was spent on the backend. A lot is going on, but I want to save that update for when the governmental paperwork for our potential new hire is complete. It’s taking ages.</p>
<p>Work has continued on Symbiote, our modding feature allowing you to dock webview-based mods on the right-hand side of TaleSpire. Luis and I have been testing the web-view library under Linux (via Proton) to ensure we can provide the same experience across Windows, Linux, and Mac. So far, we’ve got it working if we disable sandboxing, which isn’t ideal but might be how we ship the beta.</p>
<p>Continuing with Linux, we have progressed in getting talespire:// URLs to work out of the box. Until now, we’ve been relying on a great script from community members, but in some cases, it requires user input [0]. This was because some execution flags[1] involved are system dependent.</p>
<p>Luckily for us, after a bunch of digging, Luis realized they were in the environment variables! So now we are pretty sure we have what we need to make this work. Hopefully, more on this soon.</p>
<p>From Linux to Mac we also have some good news. Unity has replied to Ree, saying that the bug he submitted (and that is blocking us from shipping mac support) has been fixed and is rumbling through whatever internal process is needed before it is released. When the fix eventually ships, we will have to upgrade our Unity version, which hopefully won’t be too painful [2].</p>
<p>I’m mostly done with a feature that allows you to only show the tiles/props you are currently selecting. This can be handy for precise selections, especially when working out what to upload to a slab repository. You can see it in action here:</p>
<p><img src="/assets/videos/isolationMode0.gif" alt="a mode which isolates what you see to the active selection" /></p>
<p>You may also have spotted this icon:</p>
<p><img src="/assets/images/publishSlabButton.png" alt="the publish slab button" /></p>
<p>This is the “publish slab” button. Clicking it will take you to a mode where you can frame the screenshot for your slab and provide the required information. That mode will also be used for editing the info of the slabs you’ve already uploaded. It’s looking rough right now, but we have plenty of time to improve it. For now, I want to focus on shipping this stuff as soon as possible.</p>
<p><img src="/assets/images/slabPublishWip.png" alt="publish mode" /></p>
<p>I’m running a bit low on caffeine, so other things will have to wait for another dev log.</p>
<p>Have a good one folks.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] Also because it is run by Proton, so needs to be a Windows program.
[1] ESync or FSync
[2] There is a good chance that it will also fix the long-standing bug in the input system that stops us binding <code class="language-plaintext highlighter-rouge">e</code> to actions. That one is so weird!</p>
TaleSpire Dev Log 3712023-01-19T19:19:43+00:00http://www.techsnuffle.com/2023/01/19/talespire-dev-log-371<p>It’s time for a video dev log.</p>
<iframe class="video" src="https://www.youtube.com/watch?v=PGAjRoe7n4U?feature=oembed" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<iframe width="560" height="315" src="https://www.youtube.com/embed/PGAjRoe7n4U" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>While I have you here, I’m going to abuse this reach I have to pimp something non-talespire, and even non-ttrpg related.</p>
<p>A good friend of mine released a game last week, and I’m so stoked for him. It’s called <a href="https://store.steampowered.com/app/1261430/Kandria/">Kandria</a>, and it’s a 2d open world hack-n-slash platformer, with celeste’esce smoothness of control, and a story laden post-apocalyptic environment. If that sounds at all interesting please give it a look here.</p>
<iframe src="https://store.steampowered.com/widget/1261430/" frameborder="0" width="646" height="190"></iframe>
<p>And with that I’m off,
Cheers folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3692023-01-10T00:30:07+00:00http://www.techsnuffle.com/2023/01/10/talespire-dev-log-369<p>‘Allo again!</p>
<p>I was on the road most of today, but I have a little progress to show. Here is the first slab coming from mod.io directly into the game:</p>
<p><img src="/assets/videos/slabIo_0.gif" alt="slab being spawned from community mod library" /></p>
<p>As mentioned in the last post, unlike chonkier mods like creatures and tiles, we want you to be able to spawn slabs without subscribing to them first. However, I don’t like the idea of waiting for the web request each time, so naturally, we’re doing some caching.</p>
<p>It works like this. When you first use a slab, it downloads it and saves it as a temporary file (that is deleted on exit). Once again though, loading a file does take a little bit of time, so we also keep the ten most recently used slabs strings in memory.</p>
<p>So when spawning a slab, TaleSpire can use the in-memory cache, fall back to the local file, or just download it.</p>
<p>In the gif above, you can <em>just</em> about see the timing difference between the first and second spawn of the slab.</p>
<p>While the code is still currently full of todos and stuff to clean up, I am going to switch to upload now as I want to start filtering by tags, and those are most easily added via the upload API. I’ll start tomorrow and see how quickly I can hack something in.</p>
<p>Until then, have a good one folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3682023-01-08T22:55:25+00:00http://www.techsnuffle.com/2023/01/08/talespire-dev-log-368<p>Hey folks,</p>
<p>In the last dev stream, we stated that we would start our foray into mods by making an official slab repository (along with an API for user-run sites to hook in too). This gives us some experience using mod.io’s API. As of a few days ago, I had this bit working:</p>
<p><img src="https://imgur.com/RHv1FPt" alt="showing a mod in the browser and inside TaleSpire" /></p>
<p>Since then, I’ve worked on the manager that handles downloading and caching slabs. It’s coming along well.</p>
<p>While I expect UX for creature, tile, and prop mods to be similar to our HeroForge integration[0], I think slabs should behave slightly differently. Rather than needing to link/subscribe to a slab to use it, I want you to be able to simply search, then click the icon to spawn the slab. If you want easy access, you can bookmark it to save the slab to your library for future use.</p>
<p>The way things are looking, I should have the manager working mid-week, at which point I’ll switch to working on the in-game slab uploader. Woop!</p>
<p>Right, that’s all from me for now.</p>
<p>Ciao.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] And also share a lot of the same code</p>
TaleSpire Dev Log 3672022-12-28T23:15:59+00:00http://www.techsnuffle.com/2022/12/28/talespire-dev-log-367<p>Hey again folks,</p>
<p>I hope you’ve had a good holiday season. I’ve finished Christmassing it up, so I am getting some work done before the close of the year.</p>
<p>I would have loved mac support to be wrapping up, but I’m waiting on news from Unity regarding the shader bug.</p>
<p>Instead, I got back to persisting initiative. I had it mostly working before Christmas, but trying to ship right before a holiday is a fool’s game, so I left it.</p>
<p>Today started suboptimally as the internet was down. Turns out a local distribution box[0] had frozen. Luckily the ISP was quick at getting out, so by mid-afternoon, I was back in business.</p>
<p>A little bug-fixing later and I have this:</p>
<p><img src="/assets/videos/persistInitiative.gif" alt="initiative order persisting" /></p>
<p>Which looks pretty decent. I’ll prod it some more tomorrow and see if it’s ready to ship.</p>
<p>That’s all for today,</p>
<p>I’ll be back with more soon,</p>
<p>Seeya</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] Or whatever they call them</p>
Talespire Dev Log 3662022-12-17T00:00:00+00:00http://www.techsnuffle.com/2022/12/17/talespire-dev-log-366<p>Heya folks!</p>
<p>I’m dropping in to let you know what I’ve been up to these last few days.</p>
<h2 id="hiding-board-list-from-players">Hiding board list from players</h2>
<p>I’ve done some work on the option for hiding the board list from players. I wired up the behind-the-scenes stuff and then handed it over to Ree as I was having some trouble with the UI portion.</p>
<p>All went well, so we expect this to be the next feature that ships.</p>
<h2 id="mac-bug">Mac bug</h2>
<p>In <a href="https://bouncyrock.com/news/articles/talespire-dev-log-364">log 364</a>, Ree talked about a bug blocking mac support and raised a ticket with Unity. I decided to spend a couple of days looking at it in case I could make a workaround. Spoiler, I couldn’t.</p>
<p>To trigger the bug, you have a light (which renders its volume without writing to depth buffer), and then you have something modify the depth in that area. What is extra odd is that it requires that you forward-render something for it to show up. It’s super weird. Check out this daft shit:</p>
<p><img src="/assets/videos/macGlitch.gif" alt="wat" /></p>
<p>Looking at the Metal frame trace in XCode, the most obviously wrong thing is that the GPU actions from later command-buffers seem to be executed in an earlier command encoder. I don’t know enough about Metal to work out why this is causing the errors, but it is feasible that it’s related.</p>
<p>Unity has confirmed the issue, so we are waiting to see what comes next.</p>
<h2 id="persisting-initiative">Persisting initiative</h2>
<p>Today I spent my time working on the code to allow us to persist data from board-wide systems. This is going well. It’s hard to write much about as it’s mainly me poking around to find something I like. I will need to make another pass on this sometime as patterns are emerging between the different managers that persist board data.</p>
<p>Once I’ve got this done (I’m thinking at least one more day of work), hooking up the initiative manager should be trivial. I’m currently expecting this feature to ship late next week.</p>
<h2 id="and-thats-ya-lot">And that’s ya lot</h2>
<p>Despite the mac bug, it feels good to be making steady progress. I don’t foresee any issues with persisting the initiative list. I’ll get right on to the Mod.io slab integration as soon as that works!</p>
<p>I hope this finds you all well.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3652022-12-08T16:28:46+00:00http://www.techsnuffle.com/2022/12/08/talespire-dev-log-365<p>Heya folks!</p>
<p>It’s a short one today. Since the release of group-movement, I’ve taken a moment to relax. We are seeing a few crashes that we assume are group-movement related, so we’ll look into those asap.</p>
<p>I’ve driven east today to meet up with Ree for a catchup and the opportunity to work together in person. Our potential server/dev-op hire is also in the area, so we get to hang out and talk shop, which is lovely! [0]</p>
<p>My goal is to ship persistence for the initiative list and the option for GMs to hide the board list from players next week. Thanks to those server updates from earlier this week, this shouldn’t be too complicated.</p>
<p>And then I’ve gotta get cracking on the mod.io slab store integration. If all goes to plan, that will set us up perfectly for distributing creature mods!</p>
<p>Needless to say, I’m excited. Here’s to a productive run-up to Christmas!</p>
<p>Have a good one folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] We are still waiting on feedback from the Norwegian government on work status. Bureaucracy is a pain.</p>
TaleSpire Dev Log 3632022-11-28T00:30:29+00:00http://www.techsnuffle.com/2022/11/28/talespire-dev-log-363<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>Let’s look at macOS</p>
<p>Work has continued at a good clip behind the scenes. I’ve got server updates ready for group move, new vertical controls, and the option to hide the board list from players.</p>
<p>Besides a few bug fixes, I’ve been working on support for apple silicon macs.</p>
<h2 id="apple-silicon">Apple Silicon</h2>
<p>We have started our macOS support focusing on the new apple silicon macs as GPU feature support is spotty across older models, and the push towards the new architecture seems very strong. Steam’s hardware survey is already showing that over 49% of macs with Steam installed are apple silicon macs [1].</p>
<p>We are using the m1 iMac as our baseline for apple’s new lineup. If we can run well there, we can run well on the rest.</p>
<h2 id="final-pre-show-caveats">Final pre-show caveats</h2>
<p>I remember an aphorism that goes roughly:</p>
<blockquote>
<p>If you get a speedup of 100 or 1000 times, you probably didn’t start doing something smart but instead stopped doing something dumb.</p>
</blockquote>
<p>While we are not seeing anything in the realm of 100 times speedups, we are looking at the “stop doing something dumb” category of fixes in this post.</p>
<p>“Dumb” code often arises simply from needing to implement <em>something</em> to discover what the feature would become. (But sometimes, it can also just be dumb)</p>
<p>Next, we are going to show timings in some parts of this post. While they were taken from an m1 iMac, they are intended to be purely illustrative. We are only showing timings from single frames and under specific loads. This is not a good representation of the performance in general.</p>
<p>With that said, we hope this is still interesting to some!</p>
<h2 id="part-1---the-start-point">Part 1 - The start point</h2>
<p>As I had recently been working on group movement, I decided to use some of the optimizations I had made for the lasso to improve culling for picking single creatures. This benefits all platforms.</p>
<p>I made a board with 600 creatures and got to work.</p>
<p><img src="/assets/images/macperf00.jpg" alt="" /></p>
<p>The picking code requires rendering the creatures into a buffer. With a mild amount of grief, we replaced the old code with one that culls the creatures and only draws the ones whose bounds intersect a ray from the camera.</p>
<p>This gave a 24x improvement for the code that dispatched the creature drawing for the pixel picker. It also reduces the amount of work the GPU needs to do, discarding irrelevant vertices.</p>
<p><img src="/assets/images/macperf01.jpg" alt="" /></p>
<h2 id="part-2---uploading-data">Part 2 - Uploading data</h2>
<p>Next, I had this fun one. In the image below, check out the timing differences between the green and red arrows for each platform.</p>
<p><img src="/assets/images/macperf02.png" alt="" /></p>
<p>The green arrows show when we start culling and dispatching render tasks to the render thread.
The red arrows show when the render thread dispatches the render tasks to the GPU.</p>
<p>After a bunch of digging (and convincing XCode to let me profile a frame[2]), I saw that every call to <a href="https://docs.unity3d.com/ScriptReference/ComputeBuffer.SetData.html">ComputeBuffer.SetData</a> was recreating the buffer. I changed the mode of the <code class="language-plaintext highlighter-rouge">ComputeBuffer</code> to <a href="https://docs.unity3d.com/ScriptReference/ComputeBufferMode.SubUpdates.html">SubUpdates</a>, which had this effect.</p>
<p><img src="/assets/images/macperf03.jpg" alt="" /></p>
<p>Clearly, this has an effect.</p>
<p>However, with this setting, we can now write over data currently being used, which is not ok. So we have some work to do around here. This leads us neatly to…</p>
<h2 id="part-3---the-elephant-in-the-room">Part 3 - The elephant in the room</h2>
<p>I’ve been working heavily on speeding up CPU tasks. That’s all well and good, but the problem we really have is the time it takes to render things.</p>
<p>This is the scene I’ve been using for testing (ignore the graphical glitches, those are shaders that still need fixing and are unrelated to this case).</p>
<p><img src="/assets/images/macPerf04.png" alt="" /></p>
<p>This is a lot to render. Currently, it’s taking over 25ms to do so, which is far too much. 30fps is playable, but we need to dig into how the shaders work if we hope to reach 60fps. XCode’s metal frame capture seems pretty temperamental, but I’ve got some captures we can work with.</p>
<h2 id="part-4---ignoring-the-elephant">Part 4 - Ignoring the elephant</h2>
<p>That is not to say we shouldn’t work on the CPU side. By doing so, we can improve all platforms at once.</p>
<p>Given that rendering is always going to take some time, what can we do at the same time? By default, the answer in Unity seems to be “not much.” This is because their MonoBehaviours <a href="https://docs.unity3d.com/Manual/ExecutionOrder.html">don’t expose a callback</a> for all interesting points in a frame. However, Unity did add a <a href="https://docs.unity3d.com/ScriptReference/LowLevel.PlayerLoop.html">low-level API</a> to completely control the system that runs in a frame, which is terrific! [3]</p>
<p>With this new tool in hand, I decided to move the code that applies build changes to the data-model to just after render tasks have been pushed to the render thread.</p>
<p>This meant that, while we still needed to do the work, we hid it behind the work on the render thread.</p>
<p><img src="/assets/images/macPerf05.png" alt="" /></p>
<h2 id="part-5---what-elephant">Part 5 - What elephant?</h2>
<p>While I’m pretty happy with hiding work, I was looking at this capture and thinking…</p>
<p><img src="/assets/images/macPerf06.png" alt="" /></p>
<p>… what are these chunky jobs, and why are they taking so long?</p>
<p>The answer was that they are jobs that copy data from dynamic physics objects (like creatures) into the physics system before the simulation runs, and then copies the data back out afterward. The data came from managed objects, so the jobs could not be Burst compiled.</p>
<p>Instead, I moved this data into a native collection that our physics MonoBehaviour could index into when they needed the values. This probably makes main-thread access slightly slower, but this is not where bulk lookups happen.</p>
<p>This allowed for this change.</p>
<p><img src="/assets/images/macPerf07.png" alt="" /></p>
<p>Pretty nice!</p>
<h2 id="part-6---what-next">Part 6 - What next</h2>
<p>The honest answer is “make the above work” :P</p>
<p>The ComputeBuffer data uploads aren’t safe, and the physics changes have some bugs. However, this feels tractable.</p>
<p>On the CPU side, some mesh generation code could be turned into jobs and burst compiled, which could give some notable wins. Also, creatures and dice currently take way too long to update, so I want to look into those too.</p>
<p>Really though, for mac, I need to look at rendering. Any improvements we can make to the shaders would be a big deal, and ideally, we would add LOD support to our batcher. I would love LODs to be a quick win, but because our code had to become so custom, this is not the case. I’ll take a look, though.</p>
<p>Anyhoo, that’s enough for tonight. I’ll try to post a few more dev logs this week, as I have fewer releases to do.</p>
<p>Peace.</p>
<p>[1] A huge caveat for this number is that Steam’s hardware survey is guaranteed to be biased due to the chicken-and-egg nature of gaming on mac. Without a big game market, game makers can’t afford to focus on mac, and without that focus, the market stays small. (In my opinion) Apple also doesn’t seem to care about games outside of gloating at conferences, which makes adoption harder. However, that’s a grumble for another day.</p>
<p>[2] No matter how long I work with XCode, it is still exhausting.</p>
<p>[3] You can find some ace articles about the PlayerLoop API here:</p>
<ul>
<li>https://medium.com/@thebeardphantom/unity-2018-and-playerloop-5c46a12a677</li>
<li>https://www.grizzly-machine.com/entries/maximizing-the-benefit-of-c-jobs-using-unitys-new-playerloop-api</li>
<li>https://www.patreon.com/posts/unity-2018-1-16336053</li>
</ul>
TaleSpire Dev Log 3622022-11-14T09:39:29+00:00http://www.techsnuffle.com/2022/11/14/talespire-dev-log-362<p>Heya,</p>
<p>I’m hopefully wrapping up work on the “move whole AOE” feature today. There are only a couple of things left.</p>
<p>The first is we need a better visual. The boxy thing I have in the clip below won’t do.</p>
<p>The second is testing network play. This is the only one that could delay things. I don’t anticipate any problems, but that is often the case :)</p>
<p><img src="/assets/videos/aoeMoveWhole1.gif" alt="aoe-wip" /></p>
<p>After that, I’m switching to two tasks:</p>
<ul>
<li>Catching up with some poor folks I’ve left in the lurch with bugs for a long time.</li>
<li>Performance fixes required for the macOS beta</li>
</ul>
<p>Those two will likely fill my next week, but I’ll drop in to let ya know how it’s going.</p>
<p>Ciao</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3612022-11-09T17:08:17+00:00http://www.techsnuffle.com/2022/11/09/talespire-dev-log-361<p>Allo!</p>
<p>After releasing the AOE markers feature, the number one request we got was to add a way to move an entire AOE in one go. This week I’ve been working on just that.</p>
<p>In the clip below, you can see a square that you can grab that lets you move the whole AOE. Of course, these are dummy visuals that are just there to help me develop the feature.</p>
<p><img src="/assets/videos/moveWholeAoe0.gif" alt="moving the aoe" /></p>
<p>Our plan is to make the visual a ring surrounding the handle. That way, you can grab the handle to adjust just that point or the ring to move the whole AOE.</p>
<p>The implementation so far has gone well. It requires a slight adjustment to the save format to support the feature, but that is already underway. I’d love to get this out this week, but</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3592022-10-22T00:08:46+00:00http://www.techsnuffle.com/2022/10/22/talespire-dev-log-359<p>Hey folks.</p>
<p>After the push to get group movement into Beta, I’ve taken this week a little easier.</p>
<p>I’ve been quietly working away on the performance of the lasso picking, and I wanted to show some profiler screenshots, but I’m still chasing down some bugs, so I’ll save that for another day. The issue goes like this:</p>
<ul>
<li>We need to render creatures for the lasso picking</li>
<li>When we enqueue them using a <a href="https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.html">CommandBuffer</a>, Unity doesn’t “batch” them into an instanced draw call, so we end up with a lot of draw calls</li>
<li>We can perform a <a href="https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.DrawMeshInstanced.html">DrawMeshInstanced</a> call ourselves, but then we have to handle the per-instance data ourselves as we are drawing a mesh, not a Unity <a href="https://docs.unity3d.com/ScriptReference/Renderer.html">Renderer</a></li>
</ul>
<p>We could use a <a href="https://docs.unity3d.com/ScriptReference/MaterialPropertyBlock.html">MaterialPropertyBlock</a>, but doing this is extra work. How about we do something nastier. We already have to provide a Matrix4x4 transform for every instance, but only three rows of that matrix are being used. So let’s sneak the extra data in the last row. This will absolutely break the world-to-local matrix Unity will generate, but we aren’t using that, so hopefully, we can ignore it. With this, and some bodging in the shaders, we get the data we need to the GPU and MASSIVELY reduce the number of draw calls.</p>
<p>For example, this screen goes from over 400 picking related draw calls to about four.</p>
<p><img src="/assets/images/someCreatures.png" alt="some guys" /></p>
<p>The scene might seem contrived, but higher numbers of the same creature are expected when dealing with armies.</p>
<p>During the above work, I also spotted that I could accelerate the rough culling we do when picking creatures normally. Most won’t notice this, but it definitely makes a difference on lower-end machines. This is part of the required work for the macOS release, so I’m happy to be doing it.</p>
<p>Alright, I’m very sleepy, so I’m heading off now.</p>
<p>Have a lovely weekend.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3582022-10-13T09:20:49+00:00http://www.techsnuffle.com/2022/10/13/talespire-dev-log-358<p>Morning all,</p>
<p>I’m currently working full-steam on getting a beta build of the group-movement feature to all of you.</p>
<p>Yesterday I gave the lasso prototype to some folks for private testing. The feedback was generally positive, but naturally, they found some new bugs and highlighted some annoying behaviors.</p>
<p>I’m now working through that list, fixing the critical stuff. Hopefully, I can get those done today.</p>
<p>I’ll then write up the usual beta guides and tutorial videos so we can hit the publish button!</p>
<p>I won’t promise a date, but we are close.</p>
<p>One big spanner in the works is that I need to have some work done on my car, which may keep me out of the office most of the day. We’ll just have to see how that goes. If I’m stuck in town, I’ll use the day to work on server code, which is easier to work on from my laptop.</p>
<p>Have a good one folks,
Ciao</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3572022-10-11T11:29:04+00:00http://www.techsnuffle.com/2022/10/11/talespire-dev-log-357<p>Yesterday my goal was testing the group movement with multiple clients and, as mac support is progressing, I decided to use that for the testing.</p>
<p>To do this I updated our native plugin to support the lasso and broke out the profiler to check that the implementation was going to be viable on Apple Silicon. Luckily enough it ran perfectly, but as I looked at the profiler I got curious about the somewhat poor performance on medium sized boards. This led me into a day of prodding, profiling, and remembering how much I hate dealing with XCode.</p>
<p>The TLDR is that the M1 iMac currently struggles to render the shere amount of stuff we regularly put in our boards.</p>
<p>That of course won’t be the end of the story. I bumbled my way through getting an XCode build of TaleSpire that allowed for GPU profiling and the captures contains reams of data giving clues as to where we can speed things up.</p>
<p>We are going to need to do a lot of experiements but the work we do there will probably benefit lower-end PCs too.</p>
<p>There is more I can ramble about, but I’ll get to that when I’m working on mac support again so I’ll spare you that for today.</p>
<p>Alright folks, now I should get back to testing.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3562022-10-10T00:49:43+00:00http://www.techsnuffle.com/2022/10/10/talespire-dev-log-356<p>Hey folks.</p>
<p>I’m back again with a group-selection update. In short, progress is very good.</p>
<p>I spent a couple of days finding a control scheme that felt good. I wanted the interaction to be based around a single modifier. You hold down a key (in my test, <code class="language-plaintext highlighter-rouge">x</code>), then you left-click and drag to draw the lasso. You can also single-click on a creature to add/remove them to/from the selection. It sounds simple, but I managed to go in circles a few times, ensuring that switching between single creatures and groups felt natural. It’s easy to add extra modifiers and make things feel like a tool. I’m happy to have somewhat avoided that.</p>
<p>Having spoken to Ree, we’ve also discussed trying another approach to the lasso control. I’ll talk about that more as it develops.</p>
<p>With the behavior taking shape, I turned to performance. I knew from the start that the regular C# implementation would be too slow. I wrote the code assuming that I’d be switching to compiling with Burst and using native collections. My initial tests[0] showed that updating and meshing a sizable wobbly lasso took between 14 and 17 milliseconds, which is hilariously bad[1]. By compiling with Burst, that was chopped down to a few milliseconds.</p>
<p>From there, I bucketed the lasso segments spatially so that intersection checks had to consider fewer segments. With this and a few more techniques, I dropped the time required to around 0.3 milliseconds.</p>
<p>The next part of the optimization was moving the whole process (including setting up the Unity mesh) into jobs. I spent quite a while playing with different configurations[2] until settling on what we have now. The lovely thing is that, in the typical case, the jobs are finished before the main thread needs the result. This results in the main thread only having to spend ~0.01ms on updating the lasso and mesh.</p>
<p>While there is definitely more I can do to optimize the lasso code, it’s plenty good enough for now[3]. It was then time to work on other actions that groups of creatures could perform. If you’ve been on our discord, you’ll have seen some clips of that work. If not, then here you go!</p>
<p><img src="/assets/videos/groupBaseColor.gif" alt="group base color" />
<img src="/assets/videos/groupMisc.gif" alt="various group actions" />
<img src="/assets/videos/groupStatus.gif" alt="persistent emotes" /></p>
<p>I’m not trying to support everything out of the gate, but some things just felt like they needed to be there.</p>
<p>Well, that’s all from me for now. While I have started looking into performance improvements needed for the macOS release, I’ll leave the details of those for another day.</p>
<p>Have a good one!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] I’m talking about timings, but it is very misleading. The time needed to update the lasso depends on many things, such as the lasso length, the number of self-intersections, etc. For narrative reasons, I’m going to mention numbers, but the only thing to really take away is that they got better :)</p>
<p>[1] For those not familiar. To run at 60fps, you need to be done with <em>everything</em> for a frame in 16.6ms. Spending that time just for a lasso is naturally unworkable</p>
<p>[2] For example, I experimented with running the meshing of separate lasso chunks in parallel. This was promising, but the overheads undid the benefits I saw from the concurrency.</p>
<p>[3] There are lots of other things to optimize before this one creeps back to the top of the list :)</p>
TaleSpire Dev Log 3552022-10-04T21:40:48+00:00http://www.techsnuffle.com/2022/10/04/talespire-dev-log-355<p>Hey everyone.</p>
<p>As hoped, a lot of things came together today, so I’m thrilled to be able to show you this:</p>
<iframe class="video" src="https://www.youtube.com/embed/tP8fdK1G90E?feature=oembed" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<p>The lasso is finally working!</p>
<p>I’ve still got plenty to do, but I’m very confident about what remains.</p>
<p>First off, I need to clean up the implementation. Some things are still hacky and/or inconsistent with other tooling[0].</p>
<p>After that, the big priority will be performance. The current implementation is abysmally slow, but this was expected. I wrote it in an exploratory manner but made sure not to write code that would be hard to optimize later on[1]. If the code is slow in the ways I expect, then a combination of Burst, threading, and some spatial hashing will solve it. But we’ll see once the code is profiled.</p>
<p>After that, the feature should be in the cleanup phase. Tweaks, bug fixes, and testing will be the order of the day[2]. Of course, the visuals for the lasso need making, but they should not impact any other part of this feature.</p>
<p>And that’s the lot for today. Tomorrow is going to be another fun one for me, though not as visually dramatic :)</p>
<p>Have a great one folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] I’m being a bit nebulous as explaining it would be clunky without code examples
[1] To me, this mostly means keeping to simple constructs and not writing “clever” code.
[2] I’ll also be porting the c++ portions to macOS. But I think there is less than an hour of work to do there.</p>
TaleSpire Dev Log 3542022-10-03T21:21:48+00:00http://www.techsnuffle.com/2022/10/03/talespire-dev-log-354<p>Heya folks! I’ve been quiet this last week, so let’s remedy that now.</p>
<h2 id="group-movement">Group Movement</h2>
<p>The hard work for the lasso is now done! By this, Ι mean that we can draw out a lasso and get pixel-perfect info of what creatures are underneath it back from the GPU.</p>
<iframe class="video" src="https://www.youtube.com/embed/UGhJw-Ym_DM?feature=oembed" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<blockquote>
<p>The text at the bottom of the video shows the IDs of creatures as they get selected. I was trying to show the pixel precision, but it’s not sure clear. The red lasso graphic is not final (which is probably obvious)</p>
</blockquote>
<p>I am now resurrecting Ree’s group-movement code and wiring up the lasso selection. If all goes well, Ι hope to have something to show by the end of tomorrow.</p>
<p>After that, the focus is on cleanup, testing, and a proper shader for the lasso itself.</p>
<p>At that point, we’ll get this into a Beta. While in Beta, I’ll be focussing on bugs and performance.</p>
<h2 id="macos-support">macOS support</h2>
<p>This is looking good. Ree has worked out the shader problem, but some legwork is still needed to fix it.</p>
<p>I’m pretty confident we’ll have macOS support in Beta within the next couple of months.</p>
<h2 id="other-stuff-and-things">Other stuff and things</h2>
<p>Based on this dev-log, you’d be totally justified in thinking not much is going on. While the opposite is the case, it would, unfortunately, be unfair to natter about it just now, so Ι won’t. I am really hoping to be more forthcoming in the next dev stream.</p>
<table>
<tbody>
<tr>
<td>As for the dev-stream, we can’t set a date yet as Ree is currently off sick. So no news on that yet either :</td>
</tr>
</tbody>
</table>
<p>Not the most satisfying way to wrap up a dev log Ι know. That’s why I haven’t written until Ι could at least show the lasso progress.</p>
<p>Can’t wait to share the rest with you,</p>
<p>Have a good one.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3522022-09-21T02:57:06+00:00http://www.techsnuffle.com/2022/09/21/talespire-dev-log-352<p>Hi again,</p>
<p>This is a quick one to show you progress with the lasso.</p>
<p><img src="/assets/videos/lassoMeshing2.gif" alt="meshing progress" /></p>
<p>As you can see, we can now generate the mesh that fills the lasso. The visuals are all for debugging and not how it will look in-game.</p>
<p>Now I have this, we can start using it with the rendering portion of the selection code.</p>
<p>I would be starting on this first thing tomorrow. However, I had a nasty discovery today at the dentist, so I’m back there tomorrow to get it fixed. The odds are that I will be in a decent amount of pain afterward, so I’m not expecting to get any work done.</p>
<p>Sucks, but the alternative is worse.</p>
<p>Alright, I’ll check back in when I’m coding again.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3512022-09-19T21:03:30+00:00http://www.techsnuffle.com/2022/09/19/talespire-dev-log-351<p>Hey again folks,</p>
<p>With AOEs shipped, I’ve returned to the lasso tool for the group-movement feature.</p>
<p><a href="https://bouncyrock.com/news/articles/talespire-dev-log-350">Previously</a> I worked on decomposing the lasso path into <a href="https://en.wikipedia.org/wiki/Simple_polygon">simple polygons</a>[0]. This was required for the next step, which is using a technique called “ear clipping” to make a mesh of the selected area.</p>
<p>If you are interested in how it works, then <a href="https://www.youtube.com/watch?v=QAdfkylpYwc">this video</a> is an excellent place to start. It’s very simple, but I still managed to make it take a day :P</p>
<p>I made a separate project for the experiment to speed up compile times. This video shows the test code in action:</p>
<p><img src="/assets/videos/lassoMeshing0.gif" alt="lasso meshing" /></p>
<p>An additional requirement for ear-clipping to work is that no three consecutive vertices are colinear. It’s a bit faint, but this clip shows that when the verts are in a line, the result is one triangle rather than two.</p>
<p><img src="/assets/videos/lassoMeshing1.gif" alt="colinear vertex removal" /></p>
<p>Now I have this working, I will clean up the code, so it is ready to combine with the previous lasso work. I’ve been careful to write the code in a fashion that lends itself to porting to <a href="https://docs.unity3d.com/Packages/com.unity.burst@0.2/manual/index.html">Burst</a>. As a single lasso path can result in multiple polygons, I will use Unity’s job system to mesh them concurrently.</p>
<p>That’s the lot for today. It’s been wonderful seeing the response to AOEs. Hopefully, it won’t be too long until I can get this into Beta.</p>
<p>Ciao.</p>
<p>[0] I am not yet satisfied with my approach, but it’s good enough to allow me to write the rest of the lasso implementation.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3502022-09-11T18:53:22+00:00http://www.techsnuffle.com/2022/09/11/talespire-dev-log-350<p>Heya folks!</p>
<p>It’s been a good week for feature work, so let’s get into it.</p>
<h2 id="aoes">AOEs</h2>
<p>With Ree back from vacation, he has been let loose on making AOEs. He’s made a more low-key variant of the ruler visual for the AOEs that is tintable. So yes, we will have color options for the AOEs on launch (this is landing in Beta in an hour or so). He’s made it so that, as well as the keybinding, you can right-click on a ruler and click a menu option to turn it into an AOE marker. He’s also made it so that the distance/angle indicators only appear when hovering the cursor over the AOE. This keeps the visual noise to a minimum</p>
<p>You can see a little of it in action here (although the visuals are not finished yet)</p>
<iframe class="video" src="https://www.youtube.com/embed/4cSBeUWueWU?feature=oembed" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<p>The latest round of playtesting has revealed a couple of things we don’t like:</p>
<ul>
<li>
<p>The pixel picking for the AOE handles works great, but it’s too easy to get handles stuck inside tiles. So we are looking at augmenting the AOE picking with something that feels more forgiving. I’m currently prototyping this and will be continuing tomorrow.</p>
</li>
<li>
<p>For rulers, it feels right to see the handles through walls. For AOEs, it rarely does. We need to change that behavior before release.</p>
</li>
</ul>
<h2 id="group-movement">Group movement</h2>
<p>As for me, I spent most of the last week staring at squiggly lines diagrams like these:</p>
<p><img src="/assets/images/squiggles.png" alt="squiggles" /></p>
<p>My goal has been working on a good way to divide them into simple polygons (see <a href="https://bouncyrock.com/news/articles/talespire-dev-log-349">my last dev log</a> for the overview of what we need this for).</p>
<p>I have an approach that is quick and works in many cases…</p>
<p><img src="/assets/videos/lasso1.gif" alt="lasso working" /></p>
<p>…but not all of them</p>
<p><img src="/assets/videos/lasso2.gif" alt="lasso with hole" /></p>
<p>The above gif shows two holes between the two polygons. Now, this isn’t uncommon. You are probably very familiar with what happens in graphics applications when you use lasso-select and self-intersect a bunch.</p>
<p><img src="/assets/videos/lassoGimp0.gif" alt="lasso tool in gimp" /></p>
<p>This behavior sucks for TaleSpire though. Actions like this should have 100% predictable behavior and trying to anticipate self-intersection behavior should be unnecessary.</p>
<p>This means I need to have another go at the polygon decomposition. However, what we have is plenty good enough for continuing with our experiments, so I’m going to leave it for later.</p>
<p>The next step is to write the ear clipping code that creates the mesh from the simple polygon. I’m back working on AOEs for now, but I think I’ll be done there soon.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>Well, that is probably the best place to leave it for now.</p>
<p>Hope you are having a great week. Hope to see you in the next dev-log.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3492022-09-06T21:28:58+00:00http://www.techsnuffle.com/2022/09/06/talespire-dev-log-349<p>Heya folks!</p>
<p>With AOEs in Beta, and Ree working on the visuals, I have turned to the next feature we want to ship: group-movement.</p>
<p>A good while back, Ree got this feature pretty much to the finish line. We were just missing a selection method that felt good. For our tests, we used the same rectangle section tool from the building tools.</p>
<iframe class="video" src="https://www.youtube.com/embed/KLNrGoxkX8I?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>As much as you lovely folks tried to tell us the selection would be fine, we are still very stubborn about the feel of TaleSpire, and so you’ve had to wait. More recently, however, we think we’ve figured out how to make a pixel-perfect lasso tool that updates in real-time. So we are currently working to find out if it’s correct. The general idea is that we can render the on-screen creatures to the picking buffer, reusing the depth buffer from the scene render. By setting the depth test set to “equals” and turning depth-write off, we early out of all hidden fragments (which helps with performance).</p>
<p>I’ve tested the rendering portion, and it’s doing what I expect, so I’ve turned to the lasso itself. We need to make a mesh that is like this…</p>
<p><img src="/assets/videos/lasso0.gif" alt="lasso" /></p>
<p>…but filled in. This means triangulating an arbitrary polygon. Luckily there are techniques for this, such as <a href="https://www.youtube.com/watch?v=QAdfkylpYwc">ear clipping (the linked video is ace)</a> but many only work on <a href="https://en.wikipedia.org/wiki/Simple_polygon">“simple” polygons</a>. Simple, in this case, means no “holes” and no self-intersection. There are algorithms out there that can handle these complex cases, but after drawing a lot of squiggles in my notebook, I <em>think</em> Ι can make something more efficient[0]. That is my goal for the next few days. After that, it should be mostly copying and modifying existing code to get the results back from the GPU.</p>
<p>That’s all from me for now. I’ll be back when there is more to show :)</p>
<p>Ciao.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] In the game, we are building the polygon incrementally over many frames and don’t have to handle “holes.” So I think a decomposition to simple polygons can also be spread over many frames. After that, the ear clipping approach can be applied.</p>
TaleSpire Dev Log 3482022-08-31T19:43:26+00:00http://www.techsnuffle.com/2022/08/31/talespire-dev-log-348<p>Heya folks,</p>
<p>Work on AOE[0] feature has been going well. In fact, if all goes to plan, we should have a Beta in your hands by the weekend.</p>
<iframe class="video" src="https://www.youtube.com/embed/FcWRFhAe5MU" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<p>The goal for the Beta is to test the basic functionality. What we have is as follows:</p>
<ul>
<li>While using a ruler, a GM can press a key[1] to turn that ruler into an AOE</li>
<li>Players and GMs can use the Tab key to view the names of the AOEs</li>
<li>GMs can right-click on the handles of an AOE or bring up a menu that lets them delete or edit the AOE</li>
<li>AOEs are saved in boards.</li>
<li>They are also included in published boards.</li>
</ul>
<p>For now, the visuals of the AOE are exactly the same as the rulers. This will probably change before the feature leaves Beta.</p>
<p>Yesterday was spent making a fast way to place and update the names above the AOEs. The result is a slightly generalized version of the code I use for bookmarks and creatures. Now that I have this, I’d like to go back and try replacing the creature and bookmark implementation with this new version, but that’s a task for another day.</p>
<p>As you can see, the names in the above videos are just dummy values. Today I’ll be adding the code for setting and changing names. I expect that to be a quick job so getting the Beta out by Saturday seems feasible.</p>
<p>I guess we’ll know soon enough!</p>
<p>Until next time,</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] AOE = area of effect
[1] The keybinding is configurable in settings</p>
TaleSpire Dev Log 3462022-08-16T20:45:15+00:00http://www.techsnuffle.com/2022/08/16/talespire-dev-log-346<p>Hey again folks,</p>
<p>As planned today, I focussed on saving/loading and getting cone AOE to work correctly.</p>
<p><img src="/assets/videos/aoeWip1.gif" alt="load working" /></p>
<p>That worked.</p>
<p>Next up will be getting pixel picking working with the AOE handles. This will take a bit of work to do efficiently. I can’t just reuse the ruler code as that only needed to handle up to sixteen rulers at a time, whereas this should be made to handle thousands efficiently. I know what tools should allow me to do this, so I just need to find the form I like.</p>
<p>I’ll keep you posted :)</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3452022-08-16T01:24:17+00:00http://www.techsnuffle.com/2022/08/16/talespire-dev-log-345<p>Hey folks,</p>
<p>With things calming down after the releases, I’m back working on area-of-effect markers again.</p>
<p>Today has gone well. I’ve been playing around getting the basics working and then focused on the serialization code later in the afternoon.</p>
<p>This is a bit early to show, but you can see something coming together:</p>
<p><img src="/assets/videos/aoeWip0.gif" alt="aoe wip" /></p>
<p>Tomorrow I’ll be testing the serialization code and looking at some bug in the cone AOE.</p>
<p>After that, I’ll start on the radial menu stuff. The big task there is hooking it up to the pixel-picking system in an efficient way. I expect that to take a day or so.</p>
<p>Alright, that’s all for tonight.</p>
<p>Ciao</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3442022-07-20T01:53:11+00:00http://www.techsnuffle.com/2022/07/20/talespire-dev-log-344<p>Heya folks. Today I’ve been working on the feature to allow region changing. The basics now work:</p>
<p><img src="/assets/videos/regions0.gif" alt="login menu" /></p>
<p>I’m still working on the code that handles what happens for players who are currently in the campaign. It technically works, but it’s ugly right now.</p>
<p>Seeya!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3432022-07-03T23:44:54+00:00http://www.techsnuffle.com/2022/07/03/talespire-dev-log-343<p>Polymorph beta and Dev stream coming this week</p>
<p>Hey folks,</p>
<p>This week polymorph is going to become available for testing. We are rolling this feature out differently than usual, so I thought it would be good to explain the process and why we are doing it.</p>
<p>Also, now that I have moved house, I finally have decent internet speeds again, so it’s time to bring back developer streams. Details are towards the end of this log.</p>
<h2 id="polymorph-beta">Polymorph Beta</h2>
<p>The short version is that we will be making polymorph available on a beta branch of TaleSpire so that the brave and wreckless can try it out and report bugs before we push it out to everyone else.</p>
<p>We’ve wanted to start doing this as it allows us to catch critical bugs without doing damage to existing creations.</p>
<p>-AND-</p>
<p>We also know that even though they are unsupported, there are folks out there hacking the game to make mods. We’d like to give them a bit of time to update their work to fix anything that isn’t compatible with changes to the game.</p>
<p>To this end, the polymorph beta will be opt-in and take place on an entirely different backend from the one you are currently playing on. And that backend will only last for the duration of the testing. That means that when we release the feature to everyone, the test backend will be shut down, and everything that was on that branch will be cleared.</p>
<p>The process to opt-in will be something like this:</p>
<p><img src="/assets/images/steam_beta.png" alt="opt-in" /></p>
<ul>
<li>You will open the Steam beta properties for TaleSpire</li>
<li>You’ll enter a password we give out later this week</li>
<li>You’ll be given the option to opt into the beta</li>
<li>If you do, the game will update, and then you will be on our testing build and backend</li>
</ul>
<p>Opting out is as simple as setting the beta back to “None.”</p>
<p>We’ll have full instructions and warnings out on the day.</p>
<iframe class="video" src="https://www.youtube.com/embed/2C49FD3e0_w?feature=oembed" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<h2 id="dev-stream">Dev Stream</h2>
<p><a href="https://www.timeanddate.com/worldclock/fixedtime.html?msg=TaleSpire+Dev+Log&iso=20220707T23&p1=187&ah=2">Dev Steam World Clock</a></p>
<p>Yup, we are finally doing a dev stream again. We will be live on <a href="https://www.twitch.tv/bouncyrock">twitch.tv/bouncyrock</a> this Thursday at 11pm Norway time (click the <a href="https://www.timeanddate.com/worldclock/fixedtime.html?msg=TaleSpire+Dev+Log&iso=20220707T23&p1=187&ah=2">link</a> above to see the time in your timezone.</p>
<p>This will be a general catchup. As usual, you’ll have the opportunity to fire questions at us too.</p>
<h2 id="thats-all-for-today">That’s all for today</h2>
<p>That’s all the news for today, there has been plenty of code and art work going on, but we can save that for another dev-log.</p>
<p>Ciao!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3422022-06-09T11:00:11+00:00http://www.techsnuffle.com/2022/06/09/talespire-dev-log-342<p>Heya folks,</p>
<p>Last night I pushed a server update containing code needed for polymorph, HeroForge pack grouping, and tracking photon errors. Aside from one hiccup, it all went well.</p>
<p>The last few days have been a mix. My Sunday and Monday were mainly working on moving house stuff. I’ve got a little cabin thing I’m using as an office, so I dug a trench and laid conduit to get internet out there. That is working great, so I’ll need to work out my streaming setup again so we can do some catchups.</p>
<p>I’ve just merged a pile of polymorph work into master. It’s slowly getting more stable, although I still have some pressing bugs to fix before we can push it out.</p>
<p>I’ve got to head out now to get a vaccine for tick-borne encephalitis, as the area I’m in is pretty forest’y. When I get back, I’ll be working on HeroForge. I need to focus on the biggest pain points there so we can get it out of beta status. I’d like to get file hash validation done, and then I’ll see how far I can get on texture rescaling today[0].</p>
<p>The next week or so will still be a bit hectic for me (due to life stuff), but everything is going in a great direction.</p>
<p>See you around.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] Currently, our assets file sizes are way bigger than they need to be due to working around a Unity bug where their texture quality setting breaks DXT textures. I’m hoping I can rescale the textures so that sizes at each mip level will be a multiple of four (required for DXT) regardless of the texture scaling their quality setting uses.</p>
TaleSpire Dev Log 3412022-06-03T00:54:39+00:00http://www.techsnuffle.com/2022/06/03/talespire-dev-log-341<p>Heya folks.</p>
<p>I’ve now packed up my computer, ready for the move, but I did get some stuff done before then.</p>
<p>I’ve made the first pass at including groups in the HeroForge browser.</p>
<p><img src="/assets/videos/hf-folder-wip.gif" alt="wip" /></p>
<p>I’ve merged the polymorph UI branch into my dev branch and am currently working on bugs. Currently, I’m updating the code to push scale along with the morph data so that players get the correct result even if the asset hasn’t finished loading yet.</p>
<p>The scale work required back-end updates to make unique-creatures handle this correctly. That is done and is looking good so far in testing.</p>
<p>I think that’s the interesting stuff for now. I should be set up for work by Monday, so I’ll see you then.</p>
<p>Ciao</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3402022-04-29T17:47:02+00:00http://www.techsnuffle.com/2022/04/29/talespire-dev-log-340<p>‘Allo folks!</p>
<p>We’ve been chugging away on fixes and features as usual. With a long weekend coming up, we’ve been working hard to make sure things are as stable as possible for those days.</p>
<p>HeroForge made minis from packs available to TaleSpire yesterday. This rollout was not perfect, and we uncovered some details that had slipped by us. We’ve been chasing down issues that resulted from that, and once we push out a small patch later today, we should have workarounds for the most egregious bugs.</p>
<p>Next week I will be looking to improve the speed of fetching the HeroForge data. Currently, it’s pretty slow for large collections.</p>
<p>While fixing the above and the other TaleSpire bugs, I’ve also started work on “move-rulers.” These will be a feature you can control locally, and the first version looks like this:</p>
<p><img src="/assets/videos/moveRulers1.gif" alt="move rulers wip" /></p>
<p>This should be a nice, lightweight complement to tactical rulers.</p>
<p>Not much work remains for the move-rulers feature. I need to add some UI for toggling it on or off and to clean up a few edge cases.</p>
<p>Until next time,</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3392022-04-28T10:32:34+00:00http://www.techsnuffle.com/2022/04/28/talespire-dev-log-339<p>Hi again folks,</p>
<p>Work continues at speed over here. We will have another patch out for you pretty soon that fixes some mouse glitching, scroll speed issues, and a bug that meant people were stuck on the main menu.</p>
<p>As well as that, work has restarted on other features. I’ve been working on one nicknamed “tactical rulers.” The idea is that in tactical play, it is often nice to be able to leave the ruler on the table for a minute while doing things like moving creatures. This feature seeks to support that.</p>
<p><img src="/assets/videos/tacticalRulers0.gif" alt="tactical ruler wip" /></p>
<blockquote>
<p>Note: this is an early test, hence the unused icon in the menu</p>
</blockquote>
<p>Another benefit is that, in the future, we can add more features to the ruler right-click menu, such as turning them into markers for area-of-effects spells and such.</p>
<p>That’s all for the moment,</p>
<p>I hope you have a great day.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3382022-04-27T01:09:18+00:00http://www.techsnuffle.com/2022/04/27/talespire-dev-log-338<p>Hellooo!</p>
<p>We have passed the first 24 hours of having shipped the HeroForge integration, and things are going well!</p>
<p>I’ll be honest that there were so many moving parts to this that I was expecting everything to be on fire at this point. It’s nice to be wrong.</p>
<p>Yesterday’s release included not only HeroForge support, but also:</p>
<ul>
<li>the vast majority of the code for the polymorph feature</li>
<li>a jump to the latest version of Unity</li>
<li>changes we needed to revisit creature copy/paste.</li>
</ul>
<p>Today I’ve been working on bugs, a couple of which I’ll highlight here:</p>
<p>One is a regression to some of the sign-in code, which has left a few people stuck on the main menu.</p>
<p>Thanks to a community member, we’ve been able to get some very helpful logs, and we are testing some possible fixes over the coming days.</p>
<p>Another issue was that minis with transparency were failing to convert. While we aren’t supporting translucent HF minis, we don’t want the conversion to simply fail. It turned out that the issue was that those minis were missing a texture that we otherwise expected to be there. I added a new code path to handle this case.</p>
<p>In the near future, HeroForge will be updating their site to give more information about what is supported. So hopefully, there will be much less chance of unwelcome surprises then.</p>
<p>That’s probably the most interesting stuff I have to say for now. I’m diving back into feature work now! I think I will continue working on the next version of the rulers feature. Until next time.</p>
<p>Peace</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3372022-04-25T01:00:57+00:00http://www.techsnuffle.com/2022/04/25/talespire-dev-log-337<p>Long time no dev-log!</p>
<p>Just a quick heads up to let you know that there will be brief server maintenance on Monday, 25 April 2022, 10:00:00 UTC</p>
<p>You can see what time that is in your region here: <a href="https://www.timeanddate.com/worldclock/fixedtime.html?msg=TaleSpire+Server+Maintenance&iso=20220425T12&p1=187&ah=1">https://www.timeanddate.com/worldclock/fixedtime.html?msg=TaleSpire+Server+Maintenance&iso=20220425T12&p1=187&ah=1</a></p>
<p>We have scheduled one hour for the work, but we will only be using all of that if something goes wrong. The expected downtime is under a minute.</p>
<p>This maintenance is in preparation for an upcoming feature release :)</p>
<p>Have a good one folks.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3362022-04-07T12:42:12+00:00http://www.techsnuffle.com/2022/04/07/talespire-dev-log-336<p>Hi folks,</p>
<p>You may have noticed that I’ve been gone for about a week. The reason for that was that I caught the flu, and it’s been doing a merry dance on me since then.[0]</p>
<p>I’m not entirely back to normal, I still run out of energy after about five hours of being awake, and my coughing sounds like some satanic accordion. But I am on the mend.</p>
<p>My sincere apologies to those of you who I’m meant to have contacted. It simply hasn’t been possible. I’ll catch up when this passes.</p>
<p>For those wondering, yes, this did affect the HeroForge release date, although it has not changed the month it is happening. Also, this has not given us extra time for polish, as I’d haven’t been able to code at all while ill (which has been driving me crazy!). That said, other TaleSpire work has continued, and <a href="https://bouncyrock.com/news/articles/talespire-dev-log-335-weekend-experiments">Ree’s latest post</a> is mind-blowingly exciting.</p>
<p>Here in Norway, we have Easter break rapidly approaching, so we will be away for that. Ree will be along with another dev-log talking about that, though.</p>
<p>Thanks for bearing with us, folks. I can’t wait to be back working while not rattling.</p>
<p>Take care</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>[0] I did test that it wasn’t COVID, which was a relief.</p>
TaleSpire Dev Log 3342022-03-31T02:13:36+00:00http://www.techsnuffle.com/2022/03/31/talespire-dev-log-334<p>Hey folks, I’m quite frazzled today so no proper update of what’s going on. I can say though, that it’s all going well, and we are still expecting the beta launch to happen on the same date that we still aren’t telling you :P</p>
<p>In lieu of text content, have some video content!</p>
<iframe class="video" src="https://www.youtube.com/embed/IJ-rDzGrKhg?feature=oembed" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0"></iframe>
<p>This is a speedy video showing the process of connecting a HeroForge account, linking some creatures to the campaign, and showing how things respond to being unlinked and relinked.</p>
<p>We are still working on bugs[0], and the UI is still changing. However, we are excited to make the experience as smooth as possible. So that this could be done effortlessly, even in the heat of battle.</p>
<blockquote>
<p>Please note that you do not need a HeroForge account to play with miniatures created by other players, even if you are the GM. An account is only required for sharing your creations with a campaign.</p>
</blockquote>
<p>Alright, I’m heading out.</p>
<p>Have a good one!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] that ugly lag when opening the HeroForge menu is already fixed, for example.</p>
TaleSpire Dev Log 3332022-03-24T15:59:14+00:00http://www.techsnuffle.com/2022/03/24/talespire-dev-log-333<p>So while we wait to announce the release date, let’s talk about some other details.</p>
<h2 id="the-road-to-heroforge-release">The road to HeroForge release</h2>
<p>The release itself is going to be a two-parter. First, there will be a short open beta, and then the more public “launch.”</p>
<p>Now, this is a bit unusual for us, so why are we doing it?</p>
<p>In short, HeroForge has a much larger community than ours[0], and when they publicize a new feature, a lot of folks will rock up to try it out. By having a short beta before pushing announcements everywhere, we all collectively get to kick the tires and find any scary bugs that snuck in before any bigger rush of people.</p>
<p>There are no restrictions to participate in the beta beyond having a HeroForge account[1]. The doors will be open, and you are all welcome in, but we won’t be doing a big PR push immediately, and the word “beta” will be slapped on a bunch of places.</p>
<p>Once both teams are happy, we will remove the “beta” label and then fire up the social medias, blast out announcements, and see how it goes!</p>
<p>The fact there is no restriction will confuse a few people, but after having you wait for so long, we couldn’t stand the idea of only letting some of you in for the beta. This feels like a decent trade-off.</p>
<h2 id="coming-soon">Coming soon</h2>
<p>Already we are turning our eyes to what we will be working on next. The art team will be busy with the cyber-punk and scifi pack, so it’s definitely a good time to ship some features.</p>
<p>This is ideal as we have a ton of stuff that feels 80% done and just needs that work to get it out the door.</p>
<p>Even if we marked it experimental and required turning it on in the settings, it would be good to get this stuff out.</p>
<p>One example of this is the change to rulers I’ve been calling “tactical rulers” internally. That change means a ruler isn’t cleared the instant you click on something else. This makes moving creatures (especially troops) pre-measured distances much easier. It is also a requirement for us to be able to start on area-of-effect markers (but that’s a different story).</p>
<p>Another thing that marries well with tactical rulers is group movement. Ree has already put a lot of work into that, and getting the last 20% feels doable.[2]</p>
<p>I feel like I’m tempting fate if I start teasing the other features in that “80%” category, so I’ll leave it for now, but I’m feeling very optimistic about what comes next.</p>
<h2 id="thats-all-for-today">That’s all for today</h2>
<p>So yeah, that’s the lot for this log.</p>
<p>More news in the next one.</p>
<p>Have a good day!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] But hey, no trades, our community rules!</p>
<p>[1] Of course, you or one of your party will need to have bought some digital minis to be able to bring them into your campaign.</p>
<p>[2] We will probably limit the number of uniques that can be added to a group initially. This is due to the amount of data that must be sent to the backend on each movement. I know what changes I need to make on the backend to make this more efficient, but that will take some more time.
Non-unique creatures are handled totally different and so won’t have that restriction.</p>
TaleSpire Dev Log 3322022-03-23T10:13:35+00:00http://www.techsnuffle.com/2022/03/23/talespire-dev-log-332<p>Hey folks!</p>
<p>Things are going great over here. Here’s the latest from me.</p>
<h2 id="server-stuff">Server stuff</h2>
<p>Last week was given almost entirely to server work, continuing the work from <a href="https://bouncyrock.com/news/articles/talespire-dev-log-331">the previous log</a>. I was able to get our <a href="https://www.packer.io/">Packer</a> scripts up to the point where it completely replaces how we were building our old docker containers and Amazon AMIs. They’re also way easier to maintain.</p>
<h2 id="heroforge">HeroForge</h2>
<p>We had a planning meeting with the HeroForge team last Friday, and we now have our internal timeline, which is super exciting for us. I say ‘for us’ because we aren’t telling you the date yet so as not to put extra pressure on both our teams.</p>
<p>While that date is ‘soon,’ I’ve probably said ‘soon’ in so many different ways already that it doesn’t really convey how soon we are talking, but you’ll see pretty quickly from what I’m up to how close things are.</p>
<p>Speaking of which!</p>
<p>I am currently working through my last coding tasks marked <strong>mandatory</strong> for release. All of these are related to handling failure cases and how to expose the issues to the user.</p>
<p>Then I’m onto the tasks marked as things we really should have. This currently includes:</p>
<ul>
<li>Adding a panel for managing errors (which will be essential for modding)</li>
<li>Changing where the HeroForge thumbnail data is stored</li>
</ul>
<p>Ree has been working on the UI, and as soon as that lands, I’ll merge my stuff and start recording videos of the entire flow of importing minis. It’s a very short process, but we need tutorial content for TaleSpire and HeroForge’s sites.</p>
<p>That takes us to the rest of the work leading up to release. We need to:</p>
<ul>
<li>Produce various texts, videos, and images for documentation</li>
<li>Set up a new Steam branch for testers</li>
<li>Chase things up with lawyers (for the wording of the ToS around HeroForge)</li>
<li>Work out our point of contact for HF if they need support</li>
<li>Write up our PR stuff for the release</li>
<li>More testing</li>
</ul>
<p>Writing stuff for humans always takes way longer than I think, so I’m aiming to have as much of the coding done by the middle of next week as possible.</p>
<p>Aaaaaaaa it’s exciting!</p>
<p>We have also been preparing for the feature work we will be doing immediately after this ships. I think we’ll be able to have some cool features landing before sci-fi ships :)</p>
<p>Alright, enough vague info for one day! I’ll be back soon with as many extra details as I can share.</p>
<p>Peace.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3312022-03-17T10:10:30+00:00http://www.techsnuffle.com/2022/03/17/talespire-dev-log-331<p>Hi folks,</p>
<p>Let’s start with HeroForge news. We’ve been working closely, iterating, and finding shortcomings in the format for some time now. As of yesterday, we have finalized the format and conversion process and are looking to the next milestone. We obviously wouldn’t announce any dates for a potential release without HeroForge’s involvement, as there are naturally a lot of overlapping concerns. Not just technically, but from the PR side too.
So today, all I’ll say is: Things are looking good!</p>
<p>I’ve also continued work on the backend. I have been working heavily using <a href="https://www.terraform.io/">terraform</a> to get all our backend infrastructure defined as code. In the future, we want to be able to spin up services[0] in whole new regions trivially, and this should let us do that.</p>
<p>Currently, I’m experimenting with an unfinished setup in the EU. This never went live but is a reasonably clean setup and makes for a good testbed.</p>
<blockquote>
<p>Please note: I’m not experimenting with live infrastructure. None of this is currently used by TaleSpire, and so any bugs you hit are not related to this :)</p>
</blockquote>
<p>As part of this process, I’m working with another tool from HashiCorp, <a href="https://www.packer.io/">packer</a>. Packer is letting me define our docker containers and AMIs from common scripts.</p>
<p>Previously we have run docker containers on our EC2 servers. However, the separation from the host has caused complications in the past, and, personally, I prefer just using docker to make development cleaner. My goal with packer is to make our AMIs and containers as close to functionally identical as is reasonable. This will remove a small overhead[1], simplify some edge cases, and make building AMIs waaaay easier[2].</p>
<p>Today I’ll continue with that backend work.</p>
<p>Seeya folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] Using this term generally, we don’t use a microservice architecture
[1] Which won’t matter now, but that I believe certainly would if we start taking on more real-time traffic
[2] I could probably argue that it will make the dev environment closer to the running environment, but that’s another ramble.</p>
TaleSpire Dev Log 3302022-03-15T22:11:30+00:00http://www.techsnuffle.com/2022/03/15/talespire-dev-log-330<p>Hi folks,</p>
<p>After the latest Photon issue, I decided to look into our options for our real-time traffic going forward. We don’t have any immediate plans to change our setup, but it was bothering me that I didn’t have a good overview.</p>
<p>That took all of Monday and half of today, and it led neatly into reading about tooling for helping the portion of the backend we host.</p>
<p>It has all been research, so I don’t have anything specific to talk about yet, but you’ll have a blog post about it as soon as that changes!</p>
<p>Hope you are doing well, folks
Chris</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>p.s: Hats off to the art team for the latest patch!</p>
TaleSpire Dev Log 3292022-03-09T16:46:38+00:00http://www.techsnuffle.com/2022/03/09/talespire-dev-log-329<p>Yesterday the code gods conspired against us, so we didn’t get to start on the latest iteration of the HeroForge mini specification.</p>
<p>Instead, I focused on packages.</p>
<p>We use a bunch of experimental packages, and naturally, these have bugs from time to time. Yesterday we had such a case with <code class="language-plaintext highlighter-rouge">Unity.Entities</code>. The package had already been fixed in the latest update, but that fix was incompatible with the latest version of another package with its own bug (and so on it goes).</p>
<p>This led to me shuffling around packages versions until I got a setup that will work for now.</p>
<p>The amusing part is that we will be removing the Entities package soon anyway. We currently have it because:</p>
<ol>
<li>Unity.Physics depends on it</li>
<li>We use a couple of serialization tools that happen to live in that package.</li>
</ol>
<p>Neither of these reasons will be valid for long, however. We are forking <code class="language-plaintext highlighter-rouge">Unity.Physics</code> to optimize it for TaleSpire’s specific use case. As we don’t use Unity’s ECS, we don’t need the ECS integration in the physics engine.[0]</p>
<p>As for Serialization, I have already started working on internal tooling for defining data formats and convertors between them. This will help in a few ways:</p>
<ul>
<li>It will reduce the amount of hard to maintain serialization code.</li>
<li>It will make writing & upgrading binary formats more approachable to more of the team.</li>
<li>It should make it easier to iterate on the formats used in mods without worrying about breaking what is already out there.</li>
</ul>
<p>I only need the basics of this to allow us to drop <code class="language-plaintext highlighter-rouge">Unity.Entities</code>. That, in turn, will let us to pull the latest version of the conflicting packages and put us in a great place to improve the performance of physics.</p>
<p>And that, for today, is that!</p>
<p>Seeya you around.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] In fact, I did this today! It’s sitting on a branch, ready to merge</p>
TaleSpire Dev Log 3282022-03-07T18:29:39+00:00http://www.techsnuffle.com/2022/03/07/talespire-dev-log-328<p>Heya!</p>
<p>Mac support has been going surprisingly well. Let’s natter about it!</p>
<h3 id="progress">Progress</h3>
<p>The first order of business was pixel-picking. This requires custom native code as we need to use features of the graphics API not exposed by Unity. I had written the DirectX version <a href="https://bouncyrock.com/news/articles/talespire-dev-log-254">over a year ago</a>, but this time I needed something for <a href="https://en.wikipedia.org/wiki/Metal_(API)">Metal</a>. I haven’t written any Metal code before, but I found <a href="https://developer.apple.com/documentation/metal/reading_pixel_data_from_a_drawable_texture">this article</a> and <a href="https://github.com/Unity-Technologies/iOSNativeCodeSamples/blob/0c155b56b79649ed840e6cf35608040828e6d687/Graphics/MetalNativeRenderingPlugin/Assets/Plugins/iOS/MetalPlugin.m#L176">this code sample</a> and felt pretty confident I could bodge something useful.</p>
<p>This turned out to be correct. I am very lucky that, at my previous job, I suffered through so much ObjectiveC development as it made the process much easier. After a bit of experimenting, the solution turned out to match the form of the DirectX one very closely, so the API for our plugin could remain the same.</p>
<p>It was fantastic to see picking come to life!</p>
<p><img src="/assets/videos/macPick0.gif" alt="picking on mac" /></p>
<p>As you’ll have just seen, the cursors are HUGE! It turns out that Unity will try to match the cursor size as close to the source image as possible, so I need to go adjust the import settings.</p>
<p>Next up, I set myself to fixing a mistake in the build scripts[0], and then I needed to look at some more platform-specific code.</p>
<p>In Unity, if the game is in fullscreen mode, the reported resolution of the display is that of the fullscreen game. This is handy in some cases, but not for us. We want the full pixel resolution of the monitor that the game is currently on (according to the OS). This is easy to get from the <a href="https://en.wikipedia.org/wiki/Cocoa_(API)">Cocoa API</a>, and so this was another job for some native code.</p>
<p>With the pixel-picker under my belt, it was extremely simple to get this written.</p>
<p>Another thing that turned out to be super easy was supporting the <code class="language-plaintext highlighter-rouge">talespire://</code> URLs. Unity has a <a href="https://docs.unity3d.com/2021.2/Documentation/Manual/deep-linking-macos.html">simple setting</a>, allowing you to set up custom URL schemes. It was so easy that there is nothing else to say about it. So here’s a picture instead:</p>
<p><img src="/assets/images/macCustomUrls.png" alt="custom url support" /></p>
<h3 id="issues">Issues</h3>
<p>There are plenty! The big one is that lots of shaders are broken. As an example, here is water:</p>
<p><img src="/assets/images/macBrokenWater.png" alt="refreshing glitches" /></p>
<p>The fps is also bizarrely low. Below I have a screenshot of the profiler. You can see that the work on the main thread is done in under 6ms, and the render thread is done by ~9ms. VSync is enabled, so I expect it to wait until 16ms before starting the next frame; however, it is waiting until ~25ms. Clearly, something is wrong or is missing from the graph.</p>
<p>This could be anything from a two-minute fix to a two-month one. We’ll have to see!</p>
<p><img src="/assets/images/macProfileWeirdness.png" alt="odd" /></p>
<h3 id="whats-next">What’s next?</h3>
<p>Clearly, there is still plenty to do before mac support can ship, but tomorrow I need to switch back to HeroForge and modding. I am also no shader expert, so I’ll definitely have to kick this to Ree so he can work out what is breaking there.</p>
<p>I’m delighted with the 10Gb network gear we purchased to link my dev PC and the Mac. It has made cross-platform development so much nicer than what we have had in the past and has helped Mac support progress smoothly.</p>
<p>So that’s it. I left out a bunch of smaller things for the sake of time, but I hope you’re as excited as I am for where this is going.</p>
<p>Have a good one folks!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] If <code class="language-plaintext highlighter-rouge">BuildPlayerOptions.locationPathName</code> does not end with <code class="language-plaintext highlighter-rouge">.app</code>, the dll containing the compiled burst jobs is not being copied to the plugins folder.</p>
TaleSpire Dev Log 3272022-03-04T23:15:41+00:00http://www.techsnuffle.com/2022/03/04/talespire-dev-log-327<p>‘Allo folks.</p>
<p>Yesterday seemed to want to stop me from working on Mac support, and it won. Today I turned the tables.</p>
<p><img src="/assets/images/macAgain.png" alt="running on m1" /></p>
<p>Before we get into that, let’s skim through some tasks from earlier in the day.</p>
<p>The first order of business was handling the renewable of some server certificates. This led neatly into one of the problems from yesterday, certificate authentication callbacks failing on the latest Unity version.</p>
<p>It turned out that, while the <a href="https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.servercertificatecustomvalidationcallback?view=netframework-4.7.2">HttpClientHandler.ServerCertificateCustomValidationCallback</a> callback is now broken, the global <a href="https://docs.microsoft.com/en-us/dotnet/api/system.net.servicepointmanager.servercertificatevalidationcallback?view=net-6.0">ServicePointManager.ServerCertificateValidationCallback</a> one still functions.</p>
<p>I hate this global event as it is fragile to modification by other code[0] and is called for all requests when we only need it for specific ones. We only need to check requests to the backend against our pinned certificate, so I have to filter those cases in the callback. However, at least we can continue work.</p>
<p>I then updated our build scripts, so we could perform mac builds. As I previously had issues building directly to the network share, I now build to a temp directory and copy the files after completion.</p>
<p>Next was updating platform-specific code:</p>
<ul>
<li>Reliably obtaining the monitor size on Windows required some calls to the win32 API. For Mac, I’m starting by assuming the Unity commands will suffice and will write a native alternative if needed</li>
<li>I updated the dylib for <a href="https://github.com/richgel999/miniz">miniz</a> to one that supports M1</li>
<li>Updated a couple of shaders <code class="language-plaintext highlighter-rouge">#ifdef</code>s to include them for Metal</li>
<li>Disabling the pixel-picker on mac</li>
<li>A whole slew of code house-keeping stuff</li>
</ul>
<p>That was enough to get TaleSpire to start, login, and load a board. This felt huge as the last time I tried this, I couldn’t make a standalone build, steam’s API didn’t support M1 mac, and I had run a hacker server locally just to get something running.</p>
<p>With all of this, I am finally ready to start coding the thing I tried to start midday yesterday :D</p>
<p>Well… almost ready. I just need to download XCode, and I sure do hate me some XCode. That’s just the price of Mac support, I guess.</p>
<p>Anyhoo, that’s enough for today.</p>
<p>Hope you have a great weekend.
Ciao.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] because, as the docs say, “Despite being a multicast delegate, only the value returned from the last-executed event handler is considered authoritative”</p>
TaleSpire Dev Log 3262022-03-03T23:35:01+00:00http://www.techsnuffle.com/2022/03/03/talespire-dev-log-326<p>Hey everyone.</p>
<p>Today has not wanted me to make progress!</p>
<p>I have a few working days before the new HeroForge asset changes are available, so I thought I’d look at other upcoming features. So I:</p>
<ul>
<li>reviewed the roadmap</li>
<li>reread the server chat code</li>
<li>got an overview of rulers</li>
<li>came up with more emote ideas</li>
<li>made some more notes for some potential gm tool for previewing what you players can see</li>
</ul>
<p>But these are just notes. I wanted to dig into something. I did some reading around Metal (Apple’s graphics API), specifically how to read back data from textures. It looked fun, so I thought I’d dig into Mac support.</p>
<p>So I:</p>
<ul>
<li>updated the Unity version</li>
<li>worked around a new bug</li>
<li>pulled the latest version of Steamworks.NET</li>
</ul>
<p>…and then promptly ran into a bug where https requests weren’t working. It’s looking like <a href="https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.servercertificatecustomvalidationcallback?view=netframework-4.7.2">HttpClientHandler.ServerCertificateCustomValidationCallback</a> is not working, and so certificate pinning is broken.</p>
<p>This is maddening, and I’ve spent a good few hours poking around code to see if another code path would work. So far, it seems not.</p>
<p>This means I should probably roll back to earlier Unity builds to see if they work. However, while this was going wrong, I found some server-side work requiring prompt attention. So it looks like that’s where my time is going next.</p>
<p>Yeah, so it was a bit of an annoying day. I was pumped to make stuff, and I ran into walls instead.</p>
<p>No worries, though. That’s just how it goes some days.</p>
<p>Have a good one.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
TaleSpire Dev Log 3252022-02-28T19:53:17+00:00http://www.techsnuffle.com/2022/02/28/talespire-dev-log-325<p>Hey folks,</p>
<p>How the hell has it been a week since the last one of these?! Let’s remedy that.</p>
<h4 id="assetdb">AssetDb</h4>
<p>The <code class="language-plaintext highlighter-rouge">AssetDb</code> is a chunk of code that handles the loading of assets and indexes. It keeps track of metadata such as tags and groups and provides hooks for other systems.</p>
<p>Up until now, we haven’t needed to unload packages at runtime. With modding, this all changes, so I’ve been adding that functionality.</p>
<p>There were a few false starts and dead-ends as I tried to find a balance between simplicity and speed. I’m still trying to learn the skill of choosing the approach that is just dumb enough to work while still being amenable to development over time. Getting stuff working faster and iterating is so valuable.</p>
<p>Asset code obviously touches many things in TaleSpire, so you find yourself in bits of the codebase that are quite old (like the inventory bar at the bottom). Once again, we don’t want to spend too much effort on these things as we already have proper improvements planned for the future. So I find myself holding off any <em>smart</em> fix that would be too invasive.</p>
<p>All that rambling aside, it’s going well. Unloading is working great. Next up will be testing and fixing systems that don’t know how to react to assets becoming unavailable dynamically.</p>
<h4 id="fix-tesla-coil--blue-fire">Fix tesla-coil & blue-fire</h4>
<p>We have had an intermittent bug with tesla-coils occasionally not showing their lightning for a long time now. It’s always been tricky to track down as I didn’t have a board that could reliably reproduce the issue. Every time I would test, the damn thing would work, so the ticket would keep falling further back in the bug list.</p>
<p>However, last week the aberration pack shipped, and we had reports of the blue fire asset not working for some too. Ree quickly managed to track down one issue[0], but the fire behaved suspiciously like the tesla-coils even with that patched.</p>
<p>Unlike previous efforts, this time, we got lucky. One of our moderators had a board where they could semi-reliably get the tesla-coil bug. She published it, passed me the link, and we were off!</p>
<p>I was able to find out the following:</p>
<ul>
<li>If I imported the board and then placed a coil, it would fail</li>
<li>If I reloaded the board, the previously placed coil and all new ones would work</li>
</ul>
<p>This is precisely the kind of place I love to start, and I quickly tried to delete parts of the board until I could get the smallest number of tiles that triggered the issue. This can yield a minimal test case, but this time it was very stubborn. If I removed any significant chunk of the board, the tesla coils just started working again.</p>
<p>That in itself is data, of course, so I changed tactic and started stepping through the batching code in the debugger.</p>
<p>I should take a tiny diversion to explain that the commonality between the coils and fire is that they are both ‘multi-mesh’ animations. To get the stop-motion-like effect of the flames, the spaghet script that runs the fire is swapping out the active mesh over time[1]. I quickly confirmed that the multiple meshes existed but that the batching code didn’t think any of them was current due to an incorrect index.</p>
<p>It’s a weird one to explain, but in short, the index should have been an int, but instead, it was a byte.</p>
<p>The index was into our global collection of meshes[2], and this meant that if the index was <256, the mesh would work. If not, it wouldn’t.</p>
<p>It worked when loading a board that already has the tesla-coil as (purely by chance) it was one of the first 255 meshes added to the global collection.</p>
<p>It didn’t work when importing the mod’s board and THEN adding the tesla-coil because that board had a lot of stuff loaded already, and so the index of the tesla-coil’s mesh ended up being >255</p>
<p>This also explains why I couldn’t reduce the board easily. It wasn’t the number of assets in the board that mattered, but the variety. When I tried to trim the board to make it smaller, I was removing just enough kinds of assets that the tesla mesh would end up with a valid index again.</p>
<p>Looking at our git history, I think I was going to use a local mesh index at first (and so chose byte), but partway through, I switched to indexing the global pool, evidently without updating the type from byte to int.</p>
<p>So! Now to ship it. I’ve checked out the TaleSpire code from before all the HeroForge work, and I have cherrypicked the fix commits over. I hope to ship these fixes tomorrow. It’s tempting to rush it out tonight, but I’d rather have an extra pair of eyes on it first.</p>
<h4 id="build-id">Build-Id</h4>
<p>Until now, we tried to include the Steam build-id in the UI to help with bug reports. However, the steam command to get the build-id is super weird:</p>
<blockquote>
<p>int GetAppBuildId();</p>
<p>Gets the buildid of this app, may change at any time based on backend updates to the game.</p>
<p>Returns: int
The current Build Id of this App. Defaults to 0 if you’re not running a build downloaded from Steam.</p>
</blockquote>
<p>Did you spot it? It returns the latest build-id on Steam, not the build-id of what you have installed!</p>
<p>Why? I don’t know. But I finally got annoyed enough to fix the problem.</p>
<p>I’ve modified our build scripts to use git to get the short hash of the current TaleSpire commit and include it as a <code class="language-plaintext highlighter-rouge">const string</code> in the c# code.</p>
<p>It’s dead simple and makes it trivial to ensure we are testing the build the players were playing.</p>
<p>For you lovely folks out there who are dll hacking and making unofficial mods, we append an extra <code class="language-plaintext highlighter-rouge">_M</code> to the end of the build-id when the <code class="language-plaintext highlighter-rouge">AppStateManager.UsingCodeInjection</code> flag is set.</p>
<p>This flag has been hugely helpful for helping sort bugs reports on the server side, so thank you so much for setting it. By including it in the build-id, we can speed up triaging some of the bug reports we’ve seen so far that have been mod-related.</p>
<h4 id="in-house-tooling">In house tooling</h4>
<p>Today I picked up a task that has been on my todo list for ages. Improve in-editor tools for working with tiles and props.</p>
<p>Tiles and props in TaleSpire are not made with Unity’s GameObjects. We made enormous performance improvements by creating a custom approach [3], but in doing so, we have lost the fantastic tooling we got trivially with GameObjects.</p>
<p>To improve this situation, I wrote a custom inspector that uses our pixel-picking code to display a bunch of our internal asset data. It’s ugly, but it’s already proving useful.</p>
<p>As I was apparently in the mood to work on stuff that had been waiting forever, I decided to optimize the physics debug drawing code. Until now, we used Unity’s <a href="https://docs.unity3d.com/ScriptReference/Handles.html">Handles</a> API, and it is slow. Slow to the point that I can’t visualize physics info for large boards.</p>
<p>Luckily the <code class="language-plaintext highlighter-rouge">Debug.DrawLine</code> method was updated to work from <a href="https://docs.unity3d.com/Packages/com.unity.burst@1.7/manual/index.html">Burst</a> compiled jobs, so I decided to rewrite <code class="language-plaintext highlighter-rouge">Handles</code> to use <code class="language-plaintext highlighter-rouge">Debug.DrawLine</code> instead.</p>
<p>This only took an hour or two, and the result was a HUGE boost in performance. However, there is a cap on the number of lines that can be drawn per frame, so my next step will be to replace <code class="language-plaintext highlighter-rouge">Debug.DrawLine</code> with my own approach. However, for now, this is great.</p>
<p>With that done, I tweaked the new inspector to enable visualizing the physics data for the selected asset. This used to be pretty tedious to debug, and now it’s two clicks :)</p>
<h4 id="heroforge">HeroForge</h4>
<p>The AssetDb work was required for HeroForge as well as modding, so we’ve covered the bulk of the news. However, the HeroForge team has really gone to town packing extra data into their models so that we can hopefully support ‘Dark Magic’ as well as a bunch of other things.</p>
<p>As just one example, they now include metadata about their emissive textures, which helps us process them much faster in some cases. This is all stuff they didn’t have to do, so we’re super grateful!</p>
<h4 id="and-thats-the-lot">And that’s the lot</h4>
<p>It’s been a busy time, and it is not slowing down. The next asset pack is well underway, the next patch is ready for testing, and I’ll start integrating all the new stuff from HeroForge tomorrow.</p>
<p>Until next time. Stay safe.</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] A missing texture causing odd behavior on some GPUs</p>
<p>[1] We also use standard vertex animations.</p>
<p>[2] This is an approximation, but it will do.</p>
<p>[3] For a reminder of how slow it was, even with lots of caching and work, check out the video in this dev log: https://bouncyrock.com/news/articles/talespire-dev-log-202</p>
TaleSpire Dev Log 3242022-02-21T14:06:28+00:00http://www.techsnuffle.com/2022/02/21/talespire-dev-log-324<p>Hi folks!</p>
<p>The HeroForge work is rapidly solidifying. We have had some testers playing with it and finding plenty of bugs, but nothing looks too scary. We have been able to try out some more normal creations…</p>
<p><img src="/assets/images/hf_0.png" alt="examples_0" />
<img src="/assets/images/hf_1.png" alt="examples_1" /></p>
<p>…and some that are a little more out there.</p>
<p><img src="/assets/images/hf_2.png" alt="examples_2" /></p>
<p>In recent days I’ve been working on:</p>
<ul>
<li>Support for multiple creatures on a single base</li>
<li>A mistake in mipmap generation</li>
<li>A Linux (via Proton) specific bug due to how I was calling some WinAPI functions for random data</li>
<li>Issues with miniature placement</li>
<li>A line-of-sight bug which arose because of the changes needed to the creature code.</li>
<li>Better use of head/hand/etc positions from model, and sensible fallback values;</li>
</ul>
<h4 id="handling-scale">Handling scale</h4>
<p>The eagle-eyed among you may have noticed some large creatures with surprisingly small bases. Until now, the base size has matched the creature size (0.5x0.5 -> 4x4). After some discussions with HeroForge, we decided that the scale of the base relative to the creature should match the HeroForge editor as closely as possible.</p>
<p>One problem that immediately arose was that you can’t just use the base size as the default scale. Otherwise, this chap</p>
<p><img src="/assets/images/hf_6.png" alt="big with small base" /></p>
<p>who is a 1x1 creature can be scaled up to 4x4, which gives you this</p>
<p><img src="/assets/images/hf_5.png" alt="far too big" /></p>
<p>Which is unplayable[0].</p>
<p>This meant decoupling base size and creature size. We use the total size of the mesh to pick a sensible default size and let the base be rescaled to keep the proportions required. This keeps things looking good while also being playable[1]</p>
<p><img src="/assets/images/hf_3.png" alt="decoupled" /></p>
<h4 id="today">Today</h4>
<p>So far today, I’ve hooked up the emissive map, which is where HeroForge bakes its lights[2].</p>
<p><img src="/assets/images/hf_4.png" alt="emissive" />
<em>Note: As you can see there is still work to do on the strength of the emissive</em></p>
<p>It seems that the emissive map is included even if it is empty, so I am updating the asset conversion process to check for this. We can then exclude blank maps, saving some GPU memory in the process.</p>
<p>I am then back on bugs and loose ends. I have a lot of ‘todos’ in the code where I need to handle failure cases and decide how to relay them to the user. This will be fine, but it takes a little time. All of these cases will be relevant for the creature modding too.</p>
<p>I won’t finish all the fixes today naturally, but it’s all looking very positive. Getting a mini from HeroForge into TaleSpire takes about ten seconds, depending on download time (mine is pretty slow).</p>
<p>Until next time!</p>
<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>[0] I’m trying not to be hyperbolic here. I tried playing with it, and it was utterly useless at those scales.</p>
<p>[1] Even today, scale can be tricky in TaleSpire. I definitely don’t want to claim that our approach with HeroForge is ideal, but this feels in line with the current experience of the built-in assets.</p>
<p>[2] Except for the “dark magic” lights. We need to work out how/if we can support those.</p>
TaleSpire Dev Log 3232022-02-15T13:00:39+00:00http://www.techsnuffle.com/2022/02/15/talespire-dev-log-323<p><em>Disclaimer: This DevLog is from the perspective of one developer. It doesn’t reflect everything going on with the team</em></p>
<p>Hey everyone,</p>
<p>Last week was quite a doozy. I didn’t mention it at the time, but I was in the process of trying (and failing) to buy a house, so lots of extra distractions.</p>
<h4 id="photon-issues">Photon issues</h4>
<p>A few of you also got bitten by a Photon issue earlier in the week so let’s talk about that first.</p>
<p>We’ll kick off with a bit of background. Photon is a service we use for handling real-time network communication in TaleSpire. By adding their SDK to your game (which we will call “the client”) can create “rooms.” Other clients can then join the same room. When multiple clients are in the same room, they can send data to each other. The data can be transmitted reliably (via RPC) or unreliably[0]. Photon also has a lot of functionality to help with managing networked GameObjects to trivially keep them in sync. When a room is closed, no information persists.</p>
<p>Photon operates servers worldwide so that the time to send messages between clients is minimized. Running such a service is a huge undertaking, and even the <a href="https://www.youtube.com/playlist?list=PLzVi6Kh_HMIWKS1aXOV4XobuymoF0P-_S">pros struggle with it</a>. Without something like Photon, TaleSpire wouldn’t have happened as we would not have been able to make something similar in the time our funds allowed.</p>
<p>We do have our own backend and data message service, but it is not designed to handle real-time traffic. Instead, we focus on what you can’t do with Photon: communication between rooms and persisting data.</p>
<p>Right, that’s a lot of background, but hopefully, Ι got across that this service is valuable and is not trivially replaced[1]. But when Photon has issues, we feel it.</p>
<p>That was what happened earlier this week. For about an hour, some people whose campaigns are hosted in the EU were having issues connecting. We were confused as Photon’s <a href="https://www.photonengine.com/status">status page</a> was not showing a problem. We have confirmed with other developers using Photon that the event occurred and are starting a conversation with Photon’s support to see what we can learn.</p>
<p>It’s somewhat disconcerting to see such issues after Photon being solid for so long, but we aren’t keen to throw out the baby with the bathwater, so we take these things slowly.</p>
<h4 id="voice-chat">Voice chat</h4>
<p>As we are talking about network stuff, let’s ramble about voice chat.</p>
<p>Voice chat is an interesting challenge. In principle, we can <em>just</em> find a suitable[3] webrtc client and hook it up. Of course, hooking it up is where it gets fun.</p>
<p>WebRTC is principally p2p. Setting up the connection requires exchanging some messages, so there does need to be a way to do that. Of course, we have our backend, so we can use that to handle the setup (of course, then it’s not <em>pure</em> p2p, but the call will be).</p>
<p>Next, you have <a href="https://en.wikipedia.org/wiki/NAT_traversal">NAT travelsal</a>. Basically, it is not always possible, so sometimes (let’s say 15% of the time), you can’t have a p2p connection. What to do? Well, WebRTC allows for relays that the problematic client can connect to in those cases. Of course, these relays have to be hosted somewhere (not p2p), and that costs money. You can certainly find people who host free ones (google does this), but as you can imagine, you need to trust them.</p>
<p>And we are far from done. Let’s talk traffic. We will assume that we have five clients connected, and that all managed to connect peer-to-peer. That means that each person has to send their voice and video data to all four other machines. More players? More overhead. And all of those clients also have to receive data from every other client too.</p>
<p>That traffic adds up real fast! So being ingenious, you decide to change the approach. How about we have some central machine that all the clients send their voice and video to. That machine then has a stream for each player that you can connect to. So if there are ten clients, each client is uploading one stream and receiving nine. That is a significant improvement over the nine outbound and nine inbound approach when communicating p2p. That does mean we just lost p2p. Worse, we now need servers around the world for relaying this information.</p>
<p>Ten media streams per client are still quite a lot. Instead, you could have the previously proposed central server take all the media streams, combine them into a single stream, and make that available to the clients. Now each client only needs two streams, one inbound and one outbound. However, now the servers behind the scenes aren’t just relaying data; they are re-encoding audio and video, which is much more resource-intensive. So the cost of your backend just went up a lot!</p>
<p>The above is a clumsy, simplified view of some of the problems in adding voice and video chat. However, these are precisely the kinds of tradeoffs discord, zoom, and others make. It is not viable to hand-wave these problems away if you are selling a product. Your customers will care.</p>
<p>And so we go back to TaleSpire. We need voice and video chat. And we are going to use WebRTC to provide p2p calls. However, this will take time. Unity also has a service for voice chat called <a href="https://unity.com/products/vivox">Vivox</a>. I will look into this as a means to bring voice chat to TaleSpire faster without us having to provide extra infrastructure.</p>
<p>I don’t yet know if it will be a good fit, but it is definitely worth a look. It could even be that we use it as the default for those who don’t want to trade off the overhead for the additional security of p2p.</p>
<h4 id="heroforge-integration">HeroForge integration</h4>
<p>In the last dev-log, I talked about how Ι had the core HeroForge functionality working, but also that I had written some of it quickly rather than robustly so we could start testing other parts of the system.</p>
<p>My job since then has been rewriting to make things solid. The short version of the problem is that we have code that:</p>
<ul>
<li>queries which miniatures we are meant to have</li>
<li>downloads miniatures</li>
<li>converts miniatures</li>
<li>caches miniature data</li>
<li>deletes miniatures that are no longer needed from the cache</li>
</ul>
<p>All of these processes are asynchronous, so things can get fun when operations overlap.</p>
<p>For example, let’s say that someone links a miniature to their campaign, and then when trying to link a different mini, accidentally unlinks the first one, realizes their mistake, and links it again. Let us also say they have a good internet connection and that by the time they correct their error, the miniature data has been downloaded and that the conversion has begun (this is a realistic situation).</p>
<p>Now! They unlinked, so we should delete the mini from the cache, but it’s not there yet as it’s still converting. So we can try cancel the conversion process, but it’s not guaranteed that the message will be processed before the other thread has finished. Instead, let us assume we mark somewhere to delete the mini when it hits the cache.</p>
<p>We can also recall they have now relinked while this was happening. So we do need the mini. So if we delete it, we need to download it and convert it again. And if we do try and download it, we need to make sure we download it to a different folder as deleting the local data takes time too.</p>
<p>I’m belaboring the point a bit, but Ι hope it’s clear that by assuming different timings of user input, downloads, and processing, we can end up with various different situations to handle. [4]</p>
<p>After a good few flow diagrams and pacing around my room, we have something that behaves reliably. Ι can’t think of a way to describe it without making this post much longer. The short version is that there is a strict hierarchy between the HeroForge account handling code, the cache manager, and the download manager. The cache also needs to handle the possibility of multiple versions of the same mini being in flight at once (rare case but needs handling).</p>
<h4 id="caches-and-running-multiple-copies-of-talespire">Caches and running multiple copies of TaleSpire</h4>
<p>Now I hear you calling out, “caching is just so fun, tell us more!” and to that Ι say, get help. But also… sure.</p>
<p>Some of you out there run multiple copies of TaleSpire on the same machine. It’s a relatively uncommon case, but it can be helpful for streamers or TV play.</p>
<p>The problem is that TaleSpire caches files (parts of boards, HeroForge minis, etc.) in a specific directory. If we have two copies of TaleSpire running, both are trying to write to those folders simultaneously. Worse, they might try and delete the same stuff too.</p>
<p>So far, we have ignored the problem as conflicts were rare, and the use case was niche. However, these problems will become mainstream with the upcoming package manager.</p>
<p>To handle this, we have to detect whether a cache is in use and, in such a case, make a new cache (or reuse an old one). And we have to do this in a cross-platform fashion.</p>
<p>This is really a variant of the “detect whether my program is already running problem,” so let’s look into it.</p>
<p>We need some construct to signal that a resource is taken across processes. Another wrinkle in the problem is that this not only needs to be cross-platform but ideally needs to be supported in MS and Mono’s DotNet implementations.</p>
<p>If this were an interthread problem, we would use a mutex, but your traditional mutex doesn’t communicate across processes. Windows does have the <a href="https://docs.microsoft.com/en-us/windows/win32/sync/using-named-objects">named mutex</a>, which might work, but isn’t supported on ‘Nix[5]. Linux does support named semaphores, so there might be a route there.</p>
<p>Windows also has file locking, and while Unix OS’ don’t tend to (afaik), they usually have advisory locking. This soft construct doesn’t prohibit overwriting but instead indicates that you shouldn’t. As we can query this information for a given file, we can use this as our communication construct.[6]</p>
<p>With this tool in hand, Ι made a system that does the following on TaleSpire start:</p>
<ul>
<li>tries to acquire the primary cache</li>
<li>if it succeeds, it also tries to clean up any old caches which are not in use[7]</li>
<li>if it fails, it looks to see if there is an older temporary cache it can claim else, it makes a new one</li>
</ul>
<p>Thanks to the locking, we can trust that multiple clients can’t claim the same cache simultaneously.</p>
<h4 id="wrap-up">Wrap up</h4>
<p>That’s most of the news from me Ι think. Naturally, everyone else in the team has plenty going on too, but this log is long enough :)</p>
<p>Hope you are all doing well,</p>
<p>Until next time!</p>
<p>[0] If you are unused to network programming, you might be wondering why unreliable messages would be desirable. The reason is that for many cases, the latest values are all you care about, and older values are nearly useless. Player position is often a good example of this.</p>
<p>Making messages reliable takes a lot of work and sleight of hand performed either by your program or the network protocol. This adds a lot of overhead that you simply can’t afford in most cases in games.</p>
<p>This is a big subject, and I’m butchering it in this little comment, but hopefully, this helps a little.</p>
<p>[1] peer-to-peer is its own challenge, and it has plenty that makes it significantly trickier than the client-server approach. We will tackle it in the future, but it’s not something I expect to go smoothly :D</p>
<p>[2] And still isn’t, which is annoying</p>
<p>[3] It’s got to work and not be a resource hog.</p>
<p>[4] I’ve also left out a bunch of fun details such as file locking and OS/filesystem-specific considerations</p>
<p>[5] We are using Proton on Linux for the foreseeable future, so that case should be easy, but we still need a native client for Mac</p>
<p>[6] This approach is hairy but commonly used.</p>
<p>[7] It leaves a few around as the small cost in disk space is worth it to allow multiple clients to work faster. There would only be multiple caches if there was a time when multiple copies of TS were open at the same time.</p>
TaleSpire Dev Log 3212022-02-08T14:16:09+00:00http://www.techsnuffle.com/2022/02/08/talespire-dev-log-321<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>The weekend was very successful. Ree made significant progress with the UI for the HeroForge integrations, and I worked on the implementation.</p>
<p>I started by fixing an issue causing minis not to be positioned correctly relative to the bases. Next up was scaling. I surveyed our current minis to find reasonable thresholds for picking different ‘default scale’ values for the miniatures. I need to test more minis with this. It will not be perfect, but it should get us started.</p>
<p>Next, I wrote the client-side code that handles unlinking minis from a campaign. Technically it works, but I noticed potential race conditions between asset conversion and cache clearing. This is obviously unsuitable, so I’m writing a manager to mediate access to the cache.</p>
<p>Usually, today would be a workday, but as I worked all Saturday and Sunday, I will take today as a break and get back into this tomorrow morning.</p>
<p>Have a good one folks!</p>
TaleSpire Dev Log 3212022-02-05T07:42:54+00:00http://www.techsnuffle.com/2022/02/05/talespire-dev-log-321<p>Hi again,</p>
<p>As expected, today has been about HeroForge. I’ve added the manager that handles pulling down thumbnails and have been reworking parts of the conversion process.</p>
<p>Hmm, it’s very annoying how little text my day compressed down to :D</p>
<p>I’ve now merged our HeroForge branch into master. This is great as we are forced to deal with all the broken things. My plan for today is to help out with the UI where I can and otherwise to find and fix bugs.</p>
<p>The first order of business is definitely fixing some incorrect texture formats we are picking on load.</p>
<p>I’ll keep you posted,</p>
<p>Peace.</p>
TaleSpire Dev Log 3202022-02-03T23:15:30+00:00http://www.techsnuffle.com/2022/02/03/talespire-dev-log-320<p>Good-evening all,</p>
<p>I am exhausted right now but didn’t want to leave another day without a dev log, so here is a tiny one.</p>
<p>I’ve continued work on trying to make some support tooling. It’s clearly one of those less fun things that my brain doesn’t want to do, so it’s been a bit of an uphill struggle.</p>
<p>I’m over at Ree’s for a few days now, and tomorrow we dive into HeroForge stuff again. I’m very excited about this and will let you know how it goes.</p>
<p>Right, I’m off to bed.</p>
<p>Seeya folks</p>
TaleSpire Dev Log 3192022-02-02T00:53:38+00:00http://www.techsnuffle.com/2022/02/02/talespire-dev-log-319<p><em>Disclaimer: This DevLog is from the perspective of one developer. So it doesn’t reflect everything going on with the team</em></p>
<p>Hi folks!</p>
<p>I’m back from my break and have been getting stuck into things for the last two days.</p>
<ul>
<li>I’ve updated the HeroForge convertor to use some metadata embedded in the model (previously, this was in a separate file)</li>
<li>Thanks to some changes by HeroForge, I can now separate the ‘extras’ from the base so we can use our own base (which we need for the indicators)</li>
<li>I’ve tested our code with the production version of their API (I have been using their api sandbox until recently)</li>
<li>I tried making some very large minis to explore edge cases of the conversion process.</li>
<li>I’ve merged the latest work from the master branch into the HeroForge branch.</li>
<li>Worked on some server stability improvements (still got some code-gen things to finish there)</li>
</ul>
<p>Tomorrow I’ll be focusing on making tools to help the team with support tickets. Currently, things like restoring deleted boards rely on some hacky scripts I wrote ages back and are not in a state useful for anyone else.</p>
<p>I’ll be visiting Ree from Friday to Monday to work on HeroForge integration together. I’m hoping to deal with most of my loose ends on the tech side during that time. It’s optimistic, but we’ll see how it goes.</p>
<p>I had a lovely break and am glad to be back into the swing of things again.</p>
<p>I should have another dev log for you tomorrow.</p>
<p>Peace.</p>
<p><img src="/assets/images/jotunheimen.png" alt="jotunheimen" /></p>
TaleSpire Dev Log 3162022-01-14T00:21:34+00:00http://www.techsnuffle.com/2022/01/14/talespire-dev-log-316<p>I’ve been bamboozled, hoodwinked, and led astray.</p>
<p>I was tricked into adding a feature I was not ready to add, but we will be shipping anyway.</p>
<p>Today we are talking about creature copy/paste!</p>
<p>So, creature copy/paste is a bit of a minefield. The reason it has been delayed is that supporting everything we want is hard. For example, if you can copy/paste a creature, you expect it to work with slabs, which means selection bounds should work with them, which really implies multi-delete should work too… and undo/redo.</p>
<p>This is not trivial. Multiplayer undo/redo gets hard in ways that you cannot get around with quick fixes. Explaining why would turn this post into an essay, so I’m going to ask for your forgiveness in not covering this and instead ask you to trust me on that point. [0]</p>
<p>I did try to get it working before the Steam release, but it got nasty, and it was a case of pushing back the release or delaying the feature. I’ll admit that I didn’t expect I wouldn’t have revisited it yet, but c’est la vie.</p>
<p>Just a warning, this story doesn’t end with all of the above being supported, but you will be able to copy and paste creatures in (and out of) the game soon.</p>
<p>This all started with me trying to add “creature blueprints.” The idea of creature blueprints is to let you right-click a creature and generate a talespire:// URL containing all the setup info for that creature. This would allow you to share these blueprints with others. It was a simple, neat idea.</p>
<p>So, in short order, I had it working. You could right-click and click a button to get the URL. For convenience, we allow people to paste talespire:// URLs into TaleSpire to get the same effect as following the link [1]. It seemed like the correct behavior, so I added that too.</p>
<p><img src="/assets/videos/first-prototype.gif" alt="first prototype" /></p>
<p>I showed it to some folks, and they said it would be neat if Ctrl-C did the same as the button. It sounded good to me, and, dumb as I am, I didn’t notice until I tested it that I had just been prodded into adding creature copy/paste [2].</p>
<p><img src="/assets/videos/bamboozled.gif" alt="bamboozled" /></p>
<p>Of course, it lacked all the complimentary features that I listed above. But this time I looked at it, I felt a bit differently. No, it does not work with undo/redo yet, but neither does spawning creatures right now. This means it isn’t out of place in the current build. It also is only about copying single creatures at a time, so none of the expectations around selections apply yet.</p>
<p>Honestly, I should have realized this a lot sooner. I just have to chalk it up to experience and remember to challenge my assumptions more. “Perfect is the enemy of good” seems relevant here.</p>
<p>The next thing to crop up was that when you paste a creature, it comes with all its stat values. However, the names of the stats in the campaign the creature was copied from might not match yours. In the future, the stat names should probably belong to a rule system, and the blueprint should hold an id to the rule system in use. However, I don’t want to repeat the same mistake by waiting for future features. So let’s get something simple out.</p>
<p><img src="/assets/videos/stat-names.gif" alt="stat names" /></p>
<p>We are adding a copy button to the stats settings. Clicking it adds a URL to the clipboard, which contains the names. You can then ship these around to allow others to quickly apply the same stat names as you.</p>
<p>While not being the smoothest journey, it is working, and I want to get this out to you asap. We are still testing and ironing out some of the uglier bugs, but we haven’t seen any scary release blockers so far.</p>
<p>More news on this coming soon,</p>
<p>Peace.</p>
<p>[0] Even just selecting and deleting is scary. What if you select 100 unique creatures, delete them, and then hammer undo/redo a little? The server has to be informed of all these changes, and that is no small amount of data. And yes, it’s trivial to come up with ideas for handling this, but being good enough 90% of the time is not good enough. In short, this feature needs to be written slowly, carefully, and with testing.</p>
<p>[1] We added this when we saw confusion around how to use talespire://published-board URLs. Probably due to being used to how slabs worked.</p>
<table>
<tbody>
<tr>
<td>[2] It really is staggering how blind one can be when being single-minded. I <em>knew</em> what I was making so clearly that what I was actually making was a surprise :</td>
</tr>
</tbody>
</table>
TaleSpire Dev Log 3152022-01-11T21:18:47+00:00http://www.techsnuffle.com/2022/01/11/talespire-dev-log-315<p>Heya folks!</p>
<p>As you’d expect, I’ve been working on HeroForge stuff.</p>
<p>I carried on with metadata integration into the conversion process. Most of Monday was messing around with HeroForge and looking at what kinds of scales, bases, etc., we get for different builds.</p>
<p>I had a few questions about some values in the metadata, so I sent off an email to HeroForge and, as always, got a quick response. So I know what to mess with next.</p>
<p>I had some time before the response came, however, so I did some work on internal tools. I resurrected some unfinished code for managing the news feed.</p>
<p>Both the news feed tooling and the HeroForge code required patching the backend, so I did that too (yay erlang live patching)</p>
<p>That’s all from me for today,</p>
<p>Have a good one!</p>
<p>p.s. Amongst a ton of other work, Ree has got a few bug-fixes/tweaks written, so I’ll probably put out an update tomorrow with those and the fix to the “names of hidden creature incorrectly visible to players” bug.</p>
TaleSpire Dev Log 3142022-01-07T17:37:56+00:00http://www.techsnuffle.com/2022/01/07/talespire-dev-log-314<p>Hi folks,</p>
<p>Work continues well on the HeroForge Integration.</p>
<p>In the last dev-log, I wrote about a specific issue related to perception updates. I was pretty confident with the solution I had come up with. But then I had a sleep and didn’t like it anymore :D</p>
<p>So I reworked that code again, and I’m now much happier with the approach. The TLDR is that in cases where the creature index (some metadata) isn’t yet available, we queue up the perception updates. There are, of course, details that made it easier said than done, but the result should serve us well.</p>
<p>Next, I started integrating the miniature metadata from HeroForge. We can pull metadata for each miniature as well as the asset file itself. This contains a bunch of info that we need, such as points of interest like head/s, chest/s, hand positions, etc.</p>
<p>I made a quick first pass that used this data, and it looked good. The first draft helped me spot places that were sub-par[0], so I’m currently rewriting it.</p>
<p>Technically this is all going well.</p>
<p>On the more human side: I needed to visit the police today to finalize my permanent residency. The Norway side of this has been flawless, but dealing with it put me back in the Brexit headspace, along with other thoughts I was happier leaving back in England. This distracted me today, and I decided to work Saturday instead and take the afternoon for some rest and games.</p>
<p>Hope you are having a good day,</p>
<p>Seeya!</p>
<p><em>[0] I was only pulling and processing the metadata when pulling the mini itself. If I, instead, handled it before downloading the model data, the game could start using that data without waiting for the whole mini to download.</em></p>
TaleSpire Dev Log 3132022-01-04T20:44:40+00:00http://www.techsnuffle.com/2022/01/04/talespire-dev-log-313<p>Hi folks,</p>
<p>To support HeroForge and creature modding, we had to start supporting the spawning of creatures before their meta-data is available locally.</p>
<p>This means (amongst other things) that we don’t know the head position and thus can’t correctly calculate vision when a creature moves.</p>
<p>As getting this right is one of my blockers, I decided I would take today very easy and not force a solution. I wanted to play around and give myself time away from the screen to think about it.</p>
<p>Initially, I thought that I might just sync the head position to other clients when a creature is spawned and when a board is synced. However, this fails in the following scenario:</p>
<ul>
<li>Player A joins a board</li>
<li>The board contains a creature they don’t have locally</li>
<li>Player A’s client starts downloading the creature</li>
<li>Player B joins same board before Player A’s download completes</li>
<li>Player A is now responsible for syncing the board to Player B, but doesn’t have the data for the creature</li>
</ul>
<p>Perfectly reasonable, but messes up the approach. After thinking it over and discounting another idea or two, I decided to sync the head position along with position and rotation when a creature is dropped (dropped in this case, meaning released from the player’s “hand”).[0]</p>
<p>This had me revisiting the code that handles syncing the creature drop, which was a little dated. I cleaned this up, and it should now be trivial to send this data. I’ll add that tomorrow and start testing.</p>
<p>So that’s it, not much for a day, but I’m glad I wasn’t rushing this.</p>
<p>Have a good one.</p>
<p><em>[0] The idea being that the player should have a creature loaded before being able to move it. I’ll have to add some UX to communicate this clearly.</em></p>
TaleSpire Dev Log 3122022-01-03T21:03:51+00:00http://www.techsnuffle.com/2022/01/03/talespire-dev-log-312<p>Hi everyone,</p>
<p>The server upgrade today went well. Since then I’ve switched to our HeroForge branch of TaleSpire and was able to pull my first mini from production, which was very satisfying.</p>
<p>Now to do all the remaining stuff. After the butt-ugly merge I did yesterday, I have to test everything slowly to ensure I haven’t broken anything. I took an hour or so to test basic creature behavior and started writing a manually testing script.</p>
<p>Another big issue I want to deal with early is that we can no longer be sure creature meta-data is loaded before joining a board. This means (amongst other things) that we don’t know the head position and thus can’t correctly calculate vision when a creature moves. At the very least, this means that we need to push the head position to other clients when a creature is spawned and when a board is synced.</p>
<p>That might be enough, but I need to test the theory first.</p>
<p>After that, it’s UI, base conversion, stuff, and also things. But we’ll get to all those in future logs :D</p>
<p>Have a great evening folks.</p>
<p>Ciao.</p>
TaleSpire Dev Log 311 - Scheduled Downtime2022-01-01T23:51:35+00:00http://www.techsnuffle.com/2022/01/01/talespire-dev-log-311--scheduled-downtime<p>Happy new year!</p>
<p>We will be performing server upgrades at 11:00am UTC on Monday, the 3rd of January. You can see what time that is in your region here https://www.timeanddate.com/worldclock/fixedtime.html?msg=TaleSpire+Sever+Maintenance&iso=20220103T12&p1=187&ah=2</p>
<p>We have scheduled two hours for the maintenance, but if all goes smoothly, it will take much less than that. During the update, the servers will be unavailable or unreliable.</p>
<p>The upgrades aim to fix a logging issue we have had for a while and add the code for the upcoming HeroForge integration.</p>
<p>Have a great weekend folks.</p>
<p><em>p.s. Please note, this does not mean that HeroForge support will be available immediately after this upgrade. This is just one step in the process. We will keep you posted with news as it develops</em></p>
TaleSpire Dev Log 3102021-12-31T00:13:48+00:00http://www.techsnuffle.com/2021/12/31/talespire-dev-log-310<p>Evening all!</p>
<p>Today has been focused on the upcoming HeroForge integration.</p>
<p>I’ve tested the server code with the current build, and all is looking good so far. I will put out an announcement to schedule the server as soon as we have decided when that will be. This will also land some other bug fixes I’ve been waiting on, so I’m eager for this.</p>
<p>I’ve also started merging the master branch into the HeroForge branch. Both have significant changes, and so it’s quite a beast. I could see this taking me a day or two.</p>
<p>Also, the new year is here, so work will be a little broken up around that.</p>
<p>That’s all the news for today,</p>
<p>Ciao</p>
TaleSpire Dev Log 3092021-12-29T22:40:07+00:00http://www.techsnuffle.com/2021/12/29/talespire-dev-log-309<p>Hey folks!</p>
<p>It’s my first day back to work, and I’m happily settling back in. I only tackled two things today:</p>
<ul>
<li>I added some options for use in the Unity Editor to allow username + password login instead of Steam. This lets us test interactions between GM & player clients more easily on one machine</li>
<li>I fixed a regression where names of hidden creatures have become visible to players again.</li>
</ul>
<p>Tomorrow I’ll probably tackle another smaller bug, but I’d like to focus on getting the next server patch ready. That one lands fixes and the bulk of the HeroForge backend code. It’s working with my HF branch of TaleSpire, but I know there are some places where the master branch doesn’t play nice with it.</p>
<p>That’s all for now. Hope you had a lovely Christmas.</p>
<p>Peace.</p>
TaleSpire Dev Log 3052021-12-03T11:42:55+00:00http://www.techsnuffle.com/2021/12/03/talespire-dev-log-305<p>Morning all,</p>
<p>Since last time I’ve worked on the polymorph feature, which is going quite nicely.</p>
<p><img src="/assets/videos/polymorph1.gif" alt="" /></p>
<p>We still need UX, better transitions, etc. But the direction feels good.</p>
<p>Photon suffered what seems to be another attack. As the attacks affect specific regions, we pushed a patch to TaleSpire to redirect users to different regions. It is currently just a simple lookup table, but it was the fastest way to progress. The next step on that issue is making server changes to allow region remapping per campaign session.</p>
<p>A couple of weeks back, I did some performance work to improve the handling of large boards. I pushed a build of this to Steam for some internal testing. An issue was quickly found in hide-volumes, so I’ve fixed that and pushed another build so we can continue the testing. After that, we’ll merge that branch and ship it to you folks.</p>
<p>I also found a bug that hide-volumes don’t save when placed in a zone with no tiles or props. This should be an easy fix.</p>
<p>Continuing the subject of bug fixing, I am spending this next week fixing reported bugs. I have no strict criteria for which I’ll be tackling. If I can replicate the problem, I’ll try fixing it :)</p>
<p>That’s all for now. I hope you have a good day.</p>
TaleSpire Dev Log 3032021-11-26T14:38:54+00:00http://www.techsnuffle.com/2021/11/26/talespire-dev-log-303<p>Hey everyone,</p>
<p>I’ve continued my HeroForge work and got to the point where spawning HeroForge minis works. Which is great. In this video, you will see both the asset-pack files appearing after conversion and the creature being spawned.</p>
<p>So that I didn’t get distracted by UI, I just knocked up an inspector in Unity to list and spawn the linked minis.</p>
<p><img src="/assets/videos/hf_wip.gif" alt="hf asset loading" /></p>
<p>The next problem is that this code has a data race. It only works if the HeroForge mini information comes back from the server before the board loads. We could try delaying everything, but this is a symptom of a larger issue, how to handle missing creatures.</p>
<p>Modding and HeroForge integration both expose this problem. Up until now, asset packs have had an index, which we can load fast and gives us creature info, and asset-bundles which hold the mesh, textures, etc., and load slower. We were okay with asset-bundles loading relatively slowly as we made systems that could work with only the data from the index. However, we can no longer rely on the index being available, or at least not in a timely fashion.</p>
<p>To this end, we have to take our code one step further. Instead of just handling the delayed arrival of information from bundles, now we need to handle even more basic information being missing. Things like default-scale, name of the kind of creature, head-position, and many more.</p>
<p>Head position is actually a critical one as, without it, we can’t do line-of-sight checks or fog-of-war updates. We have to be very careful here to minimize the chances of differing results on different clients. I have some ideas, but I’ll save them for another day.</p>
<p>Today I am doing the big refactor to the creature management, spawning, and vision code. Hopefully this goes smoothly.</p>
<p>Until next time,</p>
<p>Peace</p>
TaleSpire Dev Log 3012021-11-19T19:43:51+00:00http://www.techsnuffle.com/2021/11/19/talespire-dev-log-301<p>Today I’ve continued work on HeroForge, but first, let’s have a little multiplatform update.</p>
<h3 id="mac">Mac</h3>
<p>The 10Gb ethernet gear arrived and worked brilliantly out of the box. The reason we bought these is two-fold:</p>
<ul>
<li>Our m1 iMacs only have 8GB of RAM, and that makes for a poor dev environment [0]</li>
<li>We already have a nice setup on our windows machines. Given that we can cross-compile, it makes sense to build for Mac from there.</li>
</ul>
<p>The big problem with building on Windows, however, is getting the result to the Mac. Even though it’s not big by modern standards, TaleSpire still takes a while to copy over a 1Gb network. In my experience so far, that delay is enough to hamper mental flow.</p>
<p>Re-enter the 10Gbe kit! I have now got the Windows box and the Mac directly connected via DAC, and I can copy an 8GB file between them faster than I can copy it from HDD to SSD on the same machine. [1]</p>
<p>I tested building directly to the Mac from Unity, and there was no drop in build time compared to building to the local one [2]. This test was the Windows build, however. So the next step was to start experimenting with builds for Apple Silicon.</p>
<p>The TLDR is there are still things to work out. For some reason, while building for Mac locally works, building directly to the remote folder fails silently[3]. Next, my build scripts are not producing the dll containing the Burst compiled code, whereas building from the Unity GUI does.</p>
<p>These aren’t too worrying though. Soon enough, we’ll have something that allows for fast iteration across these platforms. If it holds up under actual use, we’ll order some more for the team members who need them,</p>
<p>Oh, and Unity’s profiler works great across the link too!</p>
<h3 id="linux">Linux</h3>
<p>Steam support has been back in touch and answered my remaining queries. The key points are:</p>
<ul>
<li>Steam’s store has no feature for us to treat running with Proton as the official Linux build (as I understand it)</li>
<li>It is “not feasible” to bundle Proton with TaleSpire as a means to make a ‘native’ Linux build, which will show in the store as such.</li>
</ul>
<p>What that means is that new Linux players will be buying the Windows version. Which is the same as it is now. Naturally, this works fine, but it’s a little less official than we would like.</p>
<p>We will simply have to make sure our messaging makes it clear what we officially support. It looks like the store page makes that easy.</p>
<h3 id="heroforge">HeroForge</h3>
<p>Alright, back to the real work of the day.</p>
<p>The focus of today’s work has been two classes.</p>
<ul>
<li>HeroForgeDownloadManager: Which is responsible for downloading minis, converting them for TaleSpire, and writing the resulting asset packs to disk.</li>
<li>HeroForgeManager: Which communicates with HeroForge (via our servers) and handles information about which minis are attached to the campaign.</li>
</ul>
<p>The complexity comes from the number of moving parts:</p>
<ul>
<li>Anyone can add or remove minis from their HeroForge account</li>
<li>They give/revoke TaleSpire access to talk to their HeroForge account</li>
<li>They can give/revoke specific campaigns access to specific minis</li>
<li>This information needs to be available on all clients regardless of whether they are actively playing or arrive later.</li>
<li>The assets will download at different rates for different people</li>
<li>There are systems (like fog-of-war) that need specific information about creatures to be on all clients at the same time to ensure that all clients compute the same results</li>
</ul>
<p>My work today has been focused on making sure the flow of information guarantees that the systems have what they need at the correct times. So that, for example, the FoW system can correctly update vision for a creature even if the asset itself has not yet been downloaded.</p>
<p>It’s going well. I’ve got probably half a day more of wiring things up before I can start testing with real data and find all the places that are totally wrong :D [4]</p>
<h3 id="wrapping-up">Wrapping up</h3>
<p>I’m pretty tired now, so I’m calling it a night.</p>
<p>Have a great weekend folks!</p>
<p><em>[0] Unity complains about low memory when trying to do something as simple as profile a dev build while the project is open in the editor. This clearly is unworkable. I don’t blame Unity too much for this, to be honest. 8GB is just not enough for development tools and games running on the same machine.</em></p>
<p><em>[1] The network copy took about 12 seconds, IIRC.</em></p>
<p><em>[2] In fact, it was 8 seconds faster, but I’m assuming that was an anomaly.</em></p>
<p><em>[3] The build appears to succeed, but there is no file.</em></p>
<p><em>[4] This will happen. The interconnectedness of all of this has meant it’s been a while since the code has been running. The compiler can only catch so much!</em></p>
TaleSpire Dev Log 2992021-11-15T03:57:01+00:00http://www.techsnuffle.com/2021/11/15/talespire-dev-log-299<p>Hey folks,</p>
<p>Before we get started, the number of the dev-logs is not something we sync up with features or other announcements. We aren’t that media-savvy :P</p>
<h2 id="rendering-nerdage">Rendering nerdage</h2>
<p>Ok, let’s get into things. In my last log, I was working away on HeroForge integration. However, on Thursday, I <a href="https://xkcd.com/356/">nerd sniped</a> myself by watching <a href="https://www.youtube.com/watch?v=eviSykqSUUw&t=519s">A Deep Dive into Nanite Virtualized Geometry</a> from Unreal and finally understanding some details of occlusion culling. It was just a little detail about building the early Hi-Z, but it got my mind wiring.</p>
<p>This, in turn, spun me off to this <a href="https://www.youtube.com/watch?v=uEtI7JRBVXk">phenominal talk on clustered shading and shadows</a> by Ola Olsson[0]. Even though we’ve managed to replicate Unity’s lighting approach without GameObjects, it simply doesn’t handle thousands of lights efficiently. This feels like something we should be able to integrate with Unity’s built-in rendering pipeline to improve non-shadow-casting lights.</p>
<p>On the subject of the built-in rendering pipeline, I should take this quick detour. The built-in rendering pipeline (sometimes called the BIRP) is the rendering pipeline that is enabled out of the box in Unity. They have, much more recently, added the SRP (scriptable rendering pipeline) that exposes much more control to developers. However:</p>
<ul>
<li>Porting a project to these is non-trivial and would require a lot of engineering and work on both TaleSpire and TaleWeaver</li>
<li>They are not finished. In fact, SRP has been on hold for a while now as Unity works on the internals of DOTS and which we won’t go into here[1]</li>
</ul>
<p>Because of this, we are looking at sticking with the BIRP until we simply outgrow it and have to invest in SRP. The good news is that it’s looking like we might be able to do more than I expected before having to take that plunge.</p>
<p>Ok back to code.</p>
<p>With the clustered shading talk fresh in my mind, I looked around to find more info about the technique. In the process, I found <a href="http://www.aortiz.me/2018/12/21/CG.html#determining-active-clusters">this post</a>, which was a great read on clustering in itself and, I’m sure, will be very valuable in the future.</p>
<p>While musing back on the occlusion culling, it got me thinking about early-z passes in general, and I found this <a href="https://interplayoflight.wordpress.com/2020/12/21/to-z-prepass-or-not-to-z-prepass/">post</a>. I don’t have anything to add about it, but it sparked plenty of ideas off in my head.</p>
<p>While the lights on our tiles and props currently don’t cast shadows, the sun, on the other hand, does, and rendering it is a significant portion of frame-time. Even if we could make our own version and use occlusion culling to optimize the process, I didn’t think we could integrate it without SRP [2].</p>
<p>After reading these two <a href="https://shahriyarshahrabi.medium.com/custom-shadow-mapping-in-unity-c42a81e1bbf8">excellent</a> <a href="https://catlikecoding.com/unity/tutorials/rendering/part-7/">articles</a> on how Unity handles its cascaded shadow maps, I spotted something that might allow us to hook our own approach in. As shown in cutlikecoding’s article, Unity collects the shadows into a screen-space map after making the cascades. Not only is the shader that does that <a href="https://github.com/TwoTailsGames/Unity-Built-in-Shaders/blob/master/DefaultResourcesExtra/Internal-ScreenSpaceShadows.shader">available</a> [2] but Unity also gives an option in the project settings to replace it.</p>
<p><img src="/assets/images/unityShaderStages.png" alt="ui to replace the shader" /></p>
<p>I’m hoping that this means we could make a version of this that integrates both the Unity shadow map and our own.</p>
<p>This is, of course, speculation at this point. But it was rather exciting nevertheless.</p>
<h2 id="performance">Performance</h2>
<p>The above research included plenty I haven’t mentioned, and Friday had come to a close when I wrapped that up.</p>
<p>However, I was in a performance frame of mind, and I had a hankering to see what I could improve. To this end, I broke out the profiler and went looking.</p>
<h3 id="physics">Physics</h3>
<p>One experiment I wanted to do was on how we use Unity’s <code class="language-plaintext highlighter-rouge">Unity.Physics</code> engine[3]. The engine is considered stateless, meaning you have to load in all the rigid-bodies you want to include in the simulation, each frame. There is an optimization, however, where you can tell it that the static objects are the same as last time, and thus it doesn’t have to rebuild the BVH (bounding volume hierarchy) or other internal data for statics.</p>
<p>As you can imagine, in a board with hundreds of thousands of tiles, loading that data into the physics engine starts taking a non-trivial amount of time.</p>
<p>The test board I was using was a <a href="https://discord.com/channels/262552432968466432/575333285719310376/836588588900548700">monster that was shared on the discord</a> and has many thousands of zones. We spawned a job for each copy, and I was concerned that the overhead was hurting the speed. To that end, I gathered all the pointers and lengths into an array and spawned fewer jobs to do the copying. It didn’t make a big difference, however. What did, was calling <code class="language-plaintext highlighter-rouge">JobHandle.ScheduleBatchedJobs</code> after scheduling every N jobs. It kept the workers fed well and gave enough of a speedup that I moved on to other things.</p>
<p>The next thing I wanted to test was, very crudely, would only copying the data from nearby zones be a win, given that the physics engine would have to rebuild the internal data for the statics more often? I quickly hacked this in and, to get a nice worst case, I forced the static rebuild on every frame.</p>
<p>I was, as expected, able to see an improvement, even with the rebuild costs. What I rediscovered, though, was how nice double-right clicking is to zip around the board. I noticed this because I wasn’t including zones from far away, so I couldn’t click them. This is not a show stopped by any means, but it is something I’d need to keep in mind when making a proper solution.</p>
<p>The proper solution would need to work out which zones need physics[4], and then minimize the number of times we need to rebuild the statics by being smart about when to drop zones from simulation.</p>
<p>During all this, I finally spotted something which had been bugging me for ages. I had never understood why we needed to copy the statics in every frame when, as far as I could see, they weren’t being modified. On re-reading their code, I finally noticed that they store dynamic and static rigid-bodies in the same array and that statics always came after dynamics. This meant that the indices from the BVH to the static bodies became incorrect whenever the number of dynamic rigid-bodies changed.</p>
<p>The upshot of this is that we can skip copying static objects if the zones have not been modified and the number of dynamic bodies has not changed since the last frame.</p>
<p>This gave a noticeable improvement! In the future, we’d probably want to fork their code and lay the data out in a way that lets us avoid copying more often. I’m also curious if we could develop a way to rebuild portions of the static data instead of all of it. Very exciting if possible.</p>
<h3 id="general-optimization">General optimization</h3>
<p>I remember an aphorism that goes roughly:</p>
<blockquote>
<p>If you get a speedup of 100 or 1000 times, you probably didn’t start doing something smart, but instead stopped doing something dumb.</p>
</blockquote>
<p>While we are definitely not seeing anything in the realm of 100 times speedups, I think we are still well in the land of ‘stopping doing dumb things’ when it comes to performance.</p>
<p>Over the course of the weekend, I did the following:</p>
<ul>
<li>Added an early out to culling when we are given a frustum which will definitely cull all assets</li>
<li>Changed all Spaghet managers to share the ‘globals’ data for scripts. This meant we only had to update one.</li>
<li>Noticed that zones checked every frame to see if they had missing assets that had finished loading. The zones now register for updates and unregister once all are loaded.</li>
<li>Learned that calling <a href="https://docs.unity3d.com/ScriptReference/ComputeBuffer.GetNativeBufferPtr.html">GetNativeBufferPtr</a> causes a synchronization with the rendering thread and that we should cache it instead.</li>
<li>Refactored a bunch of update logic zones push data into collections for use in the next frame rather than scanning the zones to collect it</li>
<li>Started using culling info to avoid uploading data for dynamic lights.</li>
</ul>
<p>And other little things.</p>
<p>The result is that I was able to get a 50% improvement in framerate in my test on the monster board.</p>
<p><img src="/assets/images/aLittleBitBetter.png" alt="fps improvment" /></p>
<p>Please note: This does not mean 50% improvement in all boards. This is a result of a single test.</p>
<p>The speedups are probably more noticeable in large boards. However, even small improvements in smaller boards are worth it.[5]</p>
<h2 id="linux">Linux</h2>
<p>Since our <a href="https://bouncyrock.com/news/articles/the-road-to-multiplatform">announcement that we would support macOS and Linux</a>, I’ve been trying to get info from Steam regarding how best to ship with Proton.</p>
<p>For context, and as <a href="https://itsfoss.com/steam-play/">itsfoss explains</a>, on Linux, you can choose to enable SteamPlay for ‘supported titles’ or ‘all other titles’.</p>
<p>We inquired about two things:</p>
<ul>
<li>How to be a supported title</li>
<li>Whether we could <em>pin</em> a particular proton version as default so we could test against a known software</li>
</ul>
<p>There has been a lot of back and forth over the last month, but I think we have the info now.</p>
<p>As to the version pinning question, the answer was no.</p>
<p>As to the first question, we got this feedback from support:</p>
<blockquote>
<p>No problem, Chris! It’s a confusing piece slightly because of past history.</p>
<p>Before SteamDeck drove a bunch of additional effort on Proton, games did need to be “whitelisted” so to speak, and a customer playing games on Linux could take the step to say “let me try Windows games via proton even if they haven’t been whitelisted”</p>
<p>So, those customer-facing settings in the Steam client are still visible. But because we’ve made so much progress with Proton, the notion of that whitelisting doesn’t really make sense– we don’t update it, and we may phase it out of the Steam Linux client settings altogether at some point.</p>
<p>For you as a developer, there’s no longer a list to be added to– you’re good to go!</p>
</blockquote>
<p>This is good and… well, not bad but interesting. We would like to make TaleSpire on Linux as obviously supported as on Windows. However, it seems like it will be behind an option for now, and we don’t have a way to indicate when we officially support it.</p>
<p>It might be a non-issue, as we Linux folks are used to having to jump through some hoops, but it is a little disappointing.</p>
<p>One option would be to bundle Proton with TaleSpire ourselves and ship it as a Linux native game. This would then appear in the store as such. This is a bunch more work, and feedback has indicated that it would still be good to allow people to use their own proton install if desired.</p>
<p>It’s an interesting situation, to be sure. We are still a little ways off from looking into the glaring Linux-specific bugs, so we don’t have to worry about an official release yet.</p>
<p>We’ll definitely keep you posted.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>That’s enough for today. Congrats if you made it all the way to the end!</p>
<p>Hope you have a lovely week, and I’ll be back soon with more dev-logs</p>
<p>Ciao.</p>
<p>[0] The lighting in the video isn’t great, and so I recommend following allow with the slides you can find here https://efficientshading.com/2016/07/12/game-technology-brisbane-presentation-2016/</p>
<p>[1] Not least because there is so little public information about it, and far too much conjecture online.</p>
<p>[2] Officially available from Unity Hub or <a href="https://unity3d.com/get-unity/download/archive">here</a></p>
<p>[3] This is not the default physics engine in Unity but one which shipped with DOTs. It is stateless and impressively fast given what we throw at it.</p>
<p>[4] These places include:</p>
<ul>
<li>wherever a player is pointing with their cursor</li>
<li>wherever any dice are</li>
<li>wherever any moving creatures are</li>
<li>etc</li>
</ul>
<p>[5] Physics really shows this to be true. The longer a frame takes, the more time the physics engine has to simulate the next frame, and simulating more time takes more time. This means that speeding up non-physics code can mean the physics is using less time per frame.</p>
TaleSpire Dev Log 2982021-11-09T23:40:28+00:00http://www.techsnuffle.com/2021/11/09/talespire-dev-log-298<p>Hi folks, time for another dev-log :)</p>
<h2 id="heroforge-work-continues">HeroForge work continues</h2>
<p>The first task was to replace the use of BlobAssetReference for serializing the HeroForge asset data, as yesterday we discovered that it was far too slow (~100ms on my machine).</p>
<p>That went smoothly and removed the most significant lag in our code. This allowed us to see all the stuff that was still too slow. From there, the bulk of the work was moving things to jobs. And making sure that, whenever we did have to do something on the main thread, we did it fast.</p>
<p>With asset processing/saving improved, I moved on to asset loading. The work was very similar until something cropped up that was rather surprising…</p>
<p><img src="/assets/images/slowUpload.png" alt="slow texture upload" /></p>
<p>…Unity was taking a very long time to upload the textures. At first, I thought we were trying to access the texture’s data too soon after creation, so I put a delay between those steps. I think it helped a little, but the spike in the perf graph still remained.</p>
<p>I’ll spare you the details of all the tests and head-scratching, but I eventually stumbled on something by accident:</p>
<p><img src="/assets/images/confusingTimings.png" alt="fast texture upload when busy" /></p>
<p>The first spike is the load with slow texture upload. All the little spikes are from running the same code in rapid succession. It seems that, after a delay, the first texture upload is slow, but uploads within a couple dozen frames of each other are fast.</p>
<p>I could make some wild guesses about what is happening, but to be honest, I just don’t know. For now, it’s enough to know that it’s not going to choke when batch processing, but it is rather unsettling.</p>
<p>There are two other spots where the loading code is slower than I’d like.</p>
<p>The first is that opening the asset file takes a couple of milliseconds. We can just move that out to the job that reads from that file.</p>
<p>The second is that the Texture2D constructors take a few milliseconds to run. Unfortunately, there isn’t I can do about that one as it’s not our code :(</p>
<p>Tomorrow I’ll start moving this into TaleSpire.</p>
<h2 id="macos-support">macOS support</h2>
<p>About two weeks ago, Unity 2021.2 shipped. This is the first build of the editor to officially support Apple Silicon.</p>
<p>We had already tested TaleSpire in the 2021.2 beta and, on Windows, all looks good. This means we will probably upgrade the project in the coming weeks.</p>
<p>TaleSpire on macOS is still, of course, totally broken. But this was already known, and we covered the details <a href="https://bouncyrock.com/news/articles/the-road-to-multiplatform">back in this dev-log</a>. We can, however, prepare our development environment. We are currently waiting on a fix to <a href="https://youtrack.jetbrains.com/issue/RSRP-486608">a Rider bug</a> before we can code comfortably there[0]. However, a more ideal solution would be cross-compiling from our primary dev environment on Windows.</p>
<p>I’ve done my fair share of pushing builds across the network to other Windows test machines, and the time it takes to move a few GB really makes iteration times painful. If you’ve done much coding, I’m sure you know just how much difference iteration time makes to flow and thus how fast you can get things done.</p>
<p>Rather than replicate that <em>joy</em> on macOS, we have ordered some Sonnet 10Gb gear and will be testing to see if that makes for a tenable experience.</p>
<p>I’m rather excited about that!</p>
<h2 id="thats-your-lot">That’s your lot</h2>
<p>Ok, that’s all for tonight.</p>
<p>Hope you’ve had a good day.</p>
<p>Peace.</p>
<p><em>[0] I’m a big ol’ emacs nerd and would have loved to stay, but omnisharp or its integration are just too slow. I tried all the usual suspects, but Rider has unfortunately been the best for me by a fair margin so far.</em></p>
TaleSpire Dev Log 2972021-11-08T23:23:50+00:00http://www.techsnuffle.com/2021/11/08/talespire-dev-log-297<p>Today my goal was to take the HeroForge asset processing code and turn it into the state machine used in TaleSpire.</p>
<p>This has gone reasonably well. The code is split up into a few steps, and the texture processing jobs seem to be running across many frames as hoped. This should have meant that our code was barely blocking the main thread. However, I got to learn something new instead!</p>
<p>It turns out that <a href="https://docs.unity3d.com/Packages/com.unity.entities@0.7/api/Unity.Entities.BlobBuilder.html#Unity_Entities_BlobBuilder_CreateBlobAssetReference__1_Allocator_">CreateBlobAssetReference</a> is slow, really slow, like 100ms to allocate slow. This really surprised me until I realized that, while we load BlobAssetReferences in TaleSpire (and they load fast), we only create them in TaleWeaver, where subsecond delays don’t feel bad.</p>
<p>It’s only a minor issue, though, as we can just knock together our own little format. This will take a few hours tomorrow, and then I can look at the performance again. It remains to be seen how these long-running jobs will interact with the per-frame work that TaleSpire already has to do, but we’ll be able to check that once we get this all hooked up.</p>
<p>The really slow part is not our code, though, but <a href="https://github.com/atteneder/glTFast">GLTFast</a>. On my machine, it’s taking 54ms per frame over 3 frames for it to load the data, and my machine is no slouch. I’ll definitely be looking into what we can quickly do to improve this[0], but not yet. I want to get all this code into TaleSpire and start wiring things up so I can see the gaps that still need filling.</p>
<p>Although Ree and I usually blog about our own stuff, I’m going to shout out his work today as I find it exciting. Ree’s currently refactoring the camera controller code, which is the kind of nuts-and-bolts improvement that has been stuck on our todo lists for ages. Not only does it make the codebase easier to work with, but it also takes us measurably closer to being able to give you all access to the cutscene tools that we use to make our trailers. The idea naturally being that you would be able to play these to your players in a session.</p>
<p>Alrighty, that’s the lot from me today.
Seeya tomorrow.</p>
<p><em>[0] Hopefully, I’ve just not read the docs properly, and there’ll be some “skip horribly expensive operation” option. But if not, I’ll have to look at having it create views into the data rather than Unity objects and handle the raw data myself. Then at least we can kick it off to a separate thread and wait for it to finish.</em></p>
TaleSpire Dev Log 2962021-11-07T15:31:37+00:00http://www.techsnuffle.com/2021/11/07/talespire-dev-log-296<p>Blimey, what a week.</p>
<p>I’ve been looking into converting heroforge models into a format we can consume. For our experiments, we were using some APIs that were either only available in the editor or not async and thus would stall the main thread for too long.</p>
<p>We get the assets in GLTF format, so we used <a href="https://github.com/Siccity/GLTFUtility">GLTFUtility</a> to do the initial conversion. We then needed to:</p>
<ul>
<li>pack the meshes together (except the base)</li>
<li>resize some textures</li>
<li>pack some texture together</li>
<li>DXT compress the textures</li>
<li>Save them in a new format which we can load quickly and asynchronously</li>
</ul>
<p>The mesh part took a while as I was very unfamiliar with that part of Unity. Handling all possible vertex layouts would be a real pain, so we just rely on the models from HeroForge having a specific structure. This is a safe assumption to start with. Writing some jobs to pack the data into the correct format was simple enough, and then it was on to textures.</p>
<p>We are packing the metallic/gloss map with the occlusion map using a shader. We also use this step to bring the size of these textures down to 1024x1024. To ensure the readback didn’t block, I switched the code to use <a href="https://docs.unity3d.com/ScriptReference/Rendering.AsyncGPUReadback.html">AsyncGPUReadback</a>.</p>
<p>This did get me wondering, though. The GLTFUtility spends a bunch of time, after loading the data, spawning Unity objects for Meshes and Textures. Worse, because then use <a href="https://docs.unity3d.com/ScriptReference/ImageConversion.LoadImage.htlm">Texture.LoadImage</a> it has to upload the data to the GPU too, which is totally unnecessary for the color and bump maps as we save those almost unchanged.</p>
<p>So I started attempting to modify the library to avoid this and make it more amenable to working with the job system.</p>
<p>Images in the GLTF format are (when embedded in the binary) stored as PNGs in ARGB32 format. LoadImage previously handled that for us, so I added <a href="https://github.com/StbSharp/StbImageSharp">StbImageSharp</a>, tweaked it so as not to use objects, and wired that in instead.</p>
<p>Unfortunately, the further I went, the more little details made it tricky to convert. Even after de-object-orienting enough of the code and making decent progress, I was faced with removing functionality or more extensive rewrites. I was very aware of the time it was taking and the sunken cost fallacy and didn’t want to lose more time than I had to. I also noticed that some features in GLTF were not yet supported, and integrating future work would be tricky.</p>
<p>As I was weighing up options, I found <a href="https://github.com/atteneder/glTFast">GLTFast</a>, another library that supports 100% of the GLTF specification purports to focus on speed. I had to rejig the whole process anyway, so it was an ok time to swap out the library.</p>
<p>In the last log, I talked about porting stb_dxt. stb_dxt performs compressions of a 4x4 block on pixels, but you have to write the code to process a whole image (adding padding as required). I wrote a couple of different implementations, one that collected the 4x4 blocks one at a time, and one that collected a full row of blocks before submitting them all. The potential benefit of the latter is that we can read the source data linearly. Even though it looked like I was feeding the data correctly, I was getting incorrect results. After a lot of head-scratching, I swapped out my port of stb_dxt for <a href="https://github.com/StbSharp/StbDxtSharp">StbDxtSharp</a> and was able to get some sensible results. This is unfortunate, but I had already reached Friday and didn’t want to waste more time. If we are interested, we can look into this another day.</p>
<p>Over the weekend, I did end up prodding this code some more. I was curious about generating mipmaps as the textures included didn’t have any. Even though the standard implementation is just a simple box filter, it’s not something I’ve written myself before, so I did :)</p>
<p>A bit of profiling of the asset loading shows mixed results. Reading the data from the disk takes many milliseconds, but we’ll make that async so that won’t matter. The odd thing is how long calls to <a href="https://docs.unity3d.com/ScriptReference/Texture2D.GetRawTextureData.html">Texture2D.GetRawTextureData</a> are taking. I’m hoping it’s just due to being called right after creating the texture. I’ll try giving it a frame or two and see what it looks like then. The rest of the code is fast and amenable to being run in a job, so it should mean even less work on the main thread.</p>
<p>The processing code is going to need more testing. GLTFast is definitely the part that takes the longest. Once again, the uploading of textures to the GPU seems to be the biggest cost and is something we don’t <em>need</em> it to do… unless, of course, we want to do mipmap generation on the GPU. It’s all a bit of a toss-up and is probably something we’ll just leave until the rest of the HeroForge integration code is hooked up.</p>
<p>So there it is. A week of false starts, frustrations, and progress.</p>
<p>Have a good one folks!</p>
TaleSpire Dev Log 2952021-10-27T09:41:11+00:00http://www.techsnuffle.com/2021/10/27/talespire-dev-log-295<p>Hey folks,</p>
<p>This morning I’m continuing my work on the HeroForge integration.</p>
<p>My current tasks revolve around loading the assets. For Dimension20, we had knocked together an importer which did the job but has two issues that don’t make it suitable for use inside TaleSpire:</p>
<ul>
<li>It didn’t need to worry about blocking the main thread</li>
<li>It used some functionality from <a href="https://docs.unity3d.com/ScriptReference/UnityEditor.html">UnityEditor</a>, which is not available at runtime.</li>
</ul>
<p>This means we need to get coding :)</p>
<p>The first target to replace was <a href="https://docs.unity3d.com/ScriptReference/EditorUtility.CompressTexture.html">EditorUtility.CompressTexture</a>, which we use to compress the textures to DXT5/DXT1. You might at first think that you could just replace it with <a href="https://docs.unity3d.com/ScriptReference/Texture2D.Compress.html">Texture2D.Compress</a>, but it doesn’t have the same quality settings and (much worse) is not async.</p>
<p>So I went looking elsewhere. Luckily for us, the fantastic <a href="https://github.com/nothings/stb">stb single file library project</a> has <a href="https://github.com/nothings/stb/blob/master/stb_dxt.h">just what the doctor ordered</a>. stb_dxt is small, fast, and easy to read; however, it is in C, and while I could just include a dll, it looked like it would be easy to port to Burst’s HPC#.</p>
<p>So that’s what I got up to the other day. It was a straightforward task[0], and now I just need to write the code that drives the compression process[1].</p>
<p>Today my focus is on converting the format we get from HeroForge into something suitable for TaleSpire. We need to extract what we need, apply compression and store everything in a format suitable for fast async loading[2].</p>
<p>That’s it from me for now. I’ll be back with more as it develops.</p>
<p>Peace.</p>
<p>[0] I’ll probably release that code once I can confirm it’s all working.</p>
<p>[1] DXT compresses 4x4 pixel clusters, so stb_dxts’ API takes one cluster and appends the results to a buffer. The code to provide the clusters (with correct padding) is not part of stb_dxt.</p>
<p>[2] A small amount of work still needs to be done on the main thread as creatures use Unity’s GameObjects. However, most of it can be done without blocking.</p>
TaleSpire Dev Log 2942021-10-25T18:12:34+00:00http://www.techsnuffle.com/2021/10/25/talespire-dev-log-294<p>Hey folks,</p>
<p>There has been a lot going on this past week, but just for fun, this morning, I took a few hours to look at performance.</p>
<p>The motivation came from a community member who posted some thoughts on performance issues they were seeing and shared the relatively <a href="https://talestavern.com/slab/dark-souls-megadungeon/">beefy board</a> they were testing.</p>
<p>I cracked out the profiler and saw what I expected. The frame-time was dominated by shadow rendering. Due to <em>reasons</em>[0], the shadows are culled on the CPU, and because of the sheer number of tiles, this was taking a long time[1].</p>
<p>Poking around, I saw lots of things that are already on my todo list for future improvements. What I didn’t expect was to spot something dumb.</p>
<p><code class="language-plaintext highlighter-rouge">BatchRendererGroup</code>’s <a href="https://docs.unity3d.com/ScriptReference/Rendering.BatchRendererGroup.AddBatch.html">AddBatch</a> method takes a <a href="https://docs.unity3d.com/ScriptReference/Bounds.html">Bounds</a> which encapsulates all the instances. I had assumed that it would be used during culling to exclude batches that clearly didn’t need culling. However, this wasn’t the case.</p>
<p>Armed with this knowledge, I simply tweaked the culling job to check the bounds for the entire batch first, and only if it intersected the view frustum, to cull the individual instances. Naturally, this had a big effect.</p>
<p>When I first tested the board linked earlier, I was getting ~28fps. After this change, I was getting ~58fps. It dipped in some places in the board but never below 40fps, so this was still a nice win. [2]</p>
<p><img src="/assets/images/dark60.png" alt="quite a few assets" /></p>
<p>This will go out in a patch later this week.</p>
<p>While I was in the headspace, I also added some coarse culling to dynamic lights. It helped a little (and nudged the test board up to ~60fps), but doing more optimizing can wait for another day. [3]</p>
<p>Have a good one folks!</p>
<p>[0] Initially we tried to render all tiles via <a href="https://docs.unity3d.com/ScriptReference/Rendering.BatchRendererGroup.html">BatchRendererGroup</a>s. This failed, however, due to the (undocumented) face that `BatchRendererGroup’s were never meant to be used with Unity’s built-in render pipeline, and per-instance data was simply not supported.</p>
<p>To tackle this, we use compute shaders to perform frustum culling and populate the draw lists for <a href="https://docs.unity3d.com/ScriptReference/Graphics.DrawMeshInstancedIndirect.html">DrawMeshInstancedIndirect</a>. However, when using <code class="language-plaintext highlighter-rouge">DrawMeshInstancedIndirect</code> Unity doesn’t have enough information to do culling for you, and there are no hooks for doing this (In the built-in render pipeline, which we use).</p>
<p>So! We opted for a hybrid monstrosity. Shadows are handled via <code class="language-plaintext highlighter-rouge">BatchRendererGroup</code>, and we use our custom code to do the primary rendering.</p>
<p><code class="language-plaintext highlighter-rouge">BatchRendererGroup</code> gives us nice hooks to perform culling, and we do these in Burst compiled jobs.</p>
<p>[1] This code no-doubt needs optimization too, but that’s for another day.</p>
<p>[2] Naturally, your mileage will vary. The effect will be most visible in larger boards where a higher percentage of the board is off-screen at any given time. Also, I’m running an AMD threadripper on my dev machine, so it inflates numbers a bit. However, this change will improve performance on all machines regardless of CPU as it’s simply doing less work :)</p>
<p>[3] The next big candidate for performance improvement is physics. I’m pretty confident that we can be smarter about what assets are involved in the simulation of each frame. Cutting down the number of assets included has the potential to help quite a bit.</p>
TaleSpire Dev Log 2912021-09-20T11:43:43+00:00http://www.techsnuffle.com/2021/09/20/talespire-dev-log-291<blockquote>
<p>This dev-log is an update from just one of the developers. It does not aim to include all the other things that are going on.</p>
</blockquote>
<p>Hi everyone!</p>
<p>Just a quick warning. This is just a regular dev log, no big news in here.</p>
<p>Things have been good recently, but there are a lot of spinning plates to keep track of.</p>
<h2 id="server-issues">server issues</h2>
<p>We had a server issue last week that stemmed from a database setting I hadn’t realized was enabled. The setting was to apply minor DB patches automatically. I had missed this, and so when the system obediently upgraded the DB, the server lost connection and got a tad confused.</p>
<p>Naturally, we want to schedule these patches explicitly so that the setting has been corrected.</p>
<p>Backend isn’t my strongest suit, so I’m reading and looking into the best way to handle this in the future. One likely portion of this is adding a ‘circuit breaker’ in front of the DB connection pool.</p>
<h2 id="webrtc">WebRTC</h2>
<p>Unity has a rather interesting WebRTC package in the works, so I’ve been studying this topic again. We know we’d like to have audio and video chat in the future. Ideally, this would be p2p, but (IIRC) you’d be lucky to get NAT traversal to work for more than 85% of people, so this is usually paired with a TURN server to act as a relay for those folks.</p>
<p>That, of course, means handling the cost of such a server, and more importantly, the fees for the data going through it.</p>
<p>By default, p2p would imply a bidirectional connection between pair of players. So if you have ten people, you are sending the same data 10 times to different places. What many video providers opt for instead is to have servers that mix the feeds for you, so you only have one bidirectional connection. However, naturally, that means you are no longer p2p, and the server’s requirements (and thus costs) are MUCH higher than a simple relay server.</p>
<p>Lots to think about here. We’ll likely focus on p2p (with TURN fallback) when we start, but we’ll see how it evolves.</p>
<h2 id="performance">Performance</h2>
<p>Performance work is never done, and we know that we need to do a lot to try and support both larger maps and lower-end machines. This past week my brain had latched onto this, so I spent a good deal of time reading papers and slides from various games to try and learn more of what contemporary approaches are.</p>
<p>Not much concrete to say about this yet. I know I have a lot to learn :P</p>
<h2 id="other">Other</h2>
<p>We had an internal play session the other day, which was a lot of fun and resulted in another page of ideas of potential improvements.</p>
<p>I’ve been working on HeroForge a bit, too, of course. I’m not satisfied with our backend design, so I need to focus on that more this week.</p>
<p>Hmm yeah, I think that’s most of it. Last week was very research-heavy, so I hope I end up coding more in this one.</p>
<p>The above is, of course, just me. The others have been making all manner of assets for future packs, which is both super exciting to see but also agonizing as it’s not time to show them publicly yet.</p>
<p>I hope you are all well. Have a good one folks!</p>
TaleSpire Dev Log 2902021-08-19T22:48:27+00:00http://www.techsnuffle.com/2021/08/19/talespire-dev-log-290<p>Hi folks, just a quick log today to say that I’m [Baggers] taking a week off to relax. The rest of the team are still around, of course, so you’ll still be getting your regular scheduled programming :)</p>
<p>The last few weeks have felt great as bookmarks, links, and bugfixes have been shipping. Work on persistent emotes has been picking up again behind the scenes, and the assets in the pipeline are super exciting.</p>
<p>I hope you all have a great week.</p>
<p>Seeya!</p>
TaleSpire Dev Log 2892021-07-29T02:14:48+00:00http://www.techsnuffle.com/2021/07/29/talespire-dev-log-289<p>Well, naturally, the biggest excitement in my day has been seeing the Dimension20 trailer go public, but code is also progressing, so I should talk about that.</p>
<p>Buuuuut I could watch it one more time :D</p>
<iframe width="600" height="450" src="https://www.youtube.com/watch?v=0bJfIZyEP98" frameborder="0"> </iframe>
<p>RIGHT! Now to business.</p>
<p>I started by looking into markers. Oh, actually, one little detail, soon you will be able to give markers names, at which point they are known as bookmarks. I’m gonna use those names below, so I thought I should mention that first.</p>
<p>Currently, markers are pulled when you join a specific board, and we only pull the markers for that board. To support campaign-wide bookmark search, we want to pull all of them when you join the campaign and then keep them up to date. This is similar to what we do for unique creatures, so I started reading that code to see how it worked.</p>
<p>What I found was that the unique creature sync code had some legacy cruft and was pulling far more than it needed to. As I was revisiting this code, it felt like time for a bit of a cleanup, so I got busy doing that.</p>
<p>As I was doing that, it gave me a good opportunity to add the backend data for <em>links</em>, which soon will allow you to associate an URL with creatures and markers. So I got stuck in with that too.</p>
<p>Because I was looking at links, it just felt right to think about the upcoming <code class="language-plaintext highlighter-rouge">talespire://goto/</code> links, which will allow you to add a hyperlink to a web page that will open TaleSpire and take you to a specific marker (switching to the correct campaign and board in the process). After thinking about what the first version should be, I added this into the mix.</p>
<p>So now things are getting exciting. I’ve got a first iteration of the board-panel made for gms…</p>
<p><img src="/assets/images/bookmarks.png" alt="gm-board-panel" /></p>
<blockquote>
<p>NOTE: I’ll be adding bookmark search soon</p>
</blockquote>
<p>We can add names to markers to turn them into bookmarks. And you can get a <code class="language-plaintext highlighter-rouge">talespire://goto</code> link from their right-click menu.</p>
<p>I’ve got switching between campaigns working, but I need to do some cleanup on the “login screen to campaign” transition before I can wire everything up.</p>
<p>TLDR:</p>
<p><img src="/assets/videos/together.gif" alt="it's all coming together" /></p>
<p>It would be great to ship all this late next week, but I’m not sure if that’s overly optimistic. We’ll see how the rest of it (and the testing) goes.</p>
<p>Until next time,
Peace.</p>
TaleSpire Dev Log 2872021-07-22T11:20:17+00:00http://www.techsnuffle.com/2021/07/22/talespire-dev-log-287<p>Hellooooo!</p>
<p>It’s time for another dev log, and things are moving along well.</p>
<p>Since the last patch, I’ve fixed touched a few things.</p>
<h2 id="links-in-and-out-of-boards">Links in and out of boards</h2>
<p>We are working on two new and complementary features. The first is a <code class="language-plaintext highlighter-rouge">talespire://</code> URL that takes you to a specific position within a board, and the second, which allows you to attach a URL to creatures and markers.</p>
<p>Between these, you could do things like include links into boards from campaign management software like WorldAnvil, or link directly from your creature to their D&DBeyond page.</p>
<p>I’m currently working on the database, backend, and TaleSpire patches to make this work.</p>
<h2 id="download-validation">Download validation</h2>
<p>We have hashes of all board and creature data, but we hadn’t been using it until now. We now hash the data of downloads and use that to validate the download result. This should reduce the cases we’ve seen of invalid cache files.</p>
<h2 id="pixel-perfect-camera-focus">Pixel-perfect camera focus</h2>
<p>Moving the camera using double right-click is incredibly common; however, it used to only use physics ray-casts to work out where to move to. This didn’t work well in cases like the portcullis, where the collider definitely <em>should</em> be a single cuboid, but that doesn’t let you pick through the gaps.</p>
<p>I tried using the depth information from the pixel picker to get the position, but the accuracy was too low.</p>
<p>The new approach is a hybrid. We use the pixel picker to identify the thing under the cursor, and then we cast a ray only against the colliders in that object. This gives us the expected result and will be in the next patch.</p>
<h2 id="network-tester">Network Tester</h2>
<p>We have some users with issues connecting to the game. To help with these cases, I’m updating a little tool we made to test and log the connection process. I might end up shipping this with the game or maybe build it into the game and using command-line arguments to switch to the tester on launch.</p>
<h2 id="performance-improvement">Performance improvement</h2>
<p>I also just have to shout out some work Ree did the other day. To get shadows with reasonable performance in TaleSpire, we misuse <a href="https://docs.unity3d.com/ScriptReference/Rendering.BatchRendererGroup.html">BatchRendererGroup</a>s. Due to them not being designed to work with Unity’s built-in rendering pipeline[0], they seem to have a bug where the layer information is not respected. In practice, this means that we end up running culling routines for tiles and props even when the camera has specified that it doesn’t need to include those assets (using layers). Because of this, culling code was running whenever the reflection probe was updating, even though only a couple of objects were being rendered.</p>
<p>Ree made a replacement probe that totally avoids this bug. It’s really cool as the bigger the board, the more time this can save.</p>
<p>This will be shipping in the next patch.</p>
<h2 id="and-the-rest">And the rest</h2>
<p>As always, there are bugs to fix. One that jumps to mind was that, since the last patch, creatures can be unnamed. In those cases, the name displayed should be the name of the kind of creature. However, that wasn’t happening.</p>
<p>That’s all from me. Of course, the rest of the team is plowing ahead with all sorts of exciting things. It’s gonna be great to see those land :)</p>
<p>Have a good one folks</p>
<p>[0] We did not know about this limitation when we started as the documentation was very lacking.</p>
TaleSpire Dev Log 2862021-07-15T00:03:32+00:00http://www.techsnuffle.com/2021/07/15/talespire-dev-log-286<p>Hi everyone!</p>
<p>First, to business!</p>
<h2 id="server-maintenance">Server Maintenance</h2>
<p>At 3am PT, we will be taking down the servers for maintenance (<a href="https://www.timeanddate.com/countdown/generic?p0=187&iso=20210715T12&msg=TaleSpire%20Server%20Maintenance">click here for the time in your timezone</a>).</p>
<p>We are scheduling one hour for the work, but it is likely to be less than that.</p>
<p>This patch will complete the changes we needed to make on the backend for upcoming creature features.</p>
<h2 id="dev-log">Dev-log</h2>
<p>Today has gone well</p>
<p>I’ve added a way to change your TaleSpire username…</p>
<p><img src="/assets/images/userName.png" alt="username" /></p>
<p>…and a button to rename campaigns is also complete.</p>
<p><img src="/assets/images/campaignName.png" alt="campaign" /></p>
<p>Both of those will be in the next patch to the game.</p>
<p>Ree and I spent a bunch of time testing the current TaleSpire build with the upcoming backend patch. So that <em>should</em> go smoothly. We rediscovered a couple of existing bugs in the process, so we’ll try to get some fixes for those in too.</p>
<p>I’ve also been doing some experiments with erlang so that, hopefully, more of the server updates in the future can be done with zero downtime. We’ll see how that goes :)</p>
<p>I think that is everything. Doing updates properly is slow as hell, but it’s fun to be getting closer.</p>
<p>Have a good one folks.</p>
TaleSpire Dev Log 2852021-07-13T22:13:56+00:00http://www.techsnuffle.com/2021/07/13/talespire-dev-log-285<p>Here is a fun little tweak that is coming in an update soon.</p>
<iframe width="600" height="450" src="http://www.youtube.com/embed/V5RarAY0Za8" frameborder="0"> </iframe>
<p>In TaleSpire, we love the tile-based approach to things. We also love what can be achieved with clipping but, due to tiles have similar sizes, it’s easy to cause z-fighting.</p>
<p>Yesterday, Ree suggested I try to add a tiny offset to the positions, so they were less likely to line up. But there is a caveat… we want the offset for a given tile to be the same every time you load. The reason for that is that it would suck if each time you joined the board, things looked very slightly different.</p>
<p>Tiles and props don’t have unique IDs (as that would be way too much data), and we can’t use their index in the arrays (as that changes when boards are modified), but we do have their position.</p>
<p>The position isn’t necessarily unique, though, but when batching, we also know the UUID that identifies the ‘kind’ of thing being batched. This is ideal, so we mix some bits from the id, some bits from the position, feed it through a bodged noise function, and scale the result waaaay down.</p>
<p>This tiny offset is stable and is enough to improve z-fighting in a bunch of cases.</p>
<p>The eagle-eyed among you will notice that this doesn’t fix cases where two of the same kind of asset exactly overlap. This is correct and is not something we are looking to improve as it’s not a useful tile configuration anyway.</p>
<p>So that’s that. It’s a blast to occasionally get these little things where a ten minute experiment can give such a cool result.</p>
<p>Aside from this, I’ve also pushed a patch to the database, which allows us to store data for a bunch of upcoming creature features (polymorph and persistent emotes among them).</p>
<p>Hope you are having a good day.</p>
<p>Peace.</p>
TaleSpire Dev Log 2842021-07-11T03:16:02+00:00http://www.techsnuffle.com/2021/07/11/talespire-dev-log-284<p>Hi folks!</p>
<p>The last few days of works have felt really good. I’ve been back in the flow as I have got through the ‘working things out’ stage.</p>
<p>The first thing I did was write the serialization code for the new creature data. With this done, I could hook up the code to upgrade from the old format, refactor a whole bunch of stuff and get creature saving to the backend again[0].</p>
<p>With that taking shape, I switched to the backend.[1]</p>
<p>We have an API description as erlang data, and we use that to generate both the erlang and c# code for communicating with the server. I extended this generator so it can make c# structs as previously it only made classes[2]. I also improved the serializer on the erlang side, so it needed a little less hand-holding.</p>
<p>I then used the generator as I defined the new API for creating and updating unique creatures.</p>
<p>My old code for updating unique creatures was lazy/expedient (you decide :P). I previously pushed all of a creature’s data to the server on any change. This was obviously much more data than was needed in almost all cases, but it worked. This time I made many more entry points, each of which applied smaller changes. This will reduce work for the server and hopefully make things a little faster.</p>
<p>With a draft of the API, I could generate the c# code and then add it to TaleSpire to see if I had missed anything. Once I was satisfied with that, I head back to erlang to implement the server code.</p>
<p>While working with the database, I hit an error when calling a SQL function where one of the arguments took a <a href="https://www.postgresql.org/docs/9.5/sql-createtype.html">user defined type</a>. It turned out that epgsql, the library I use, didn’t support auto-conversion of erlang data to custom SQL types. They did, of course, have a way for you to add your own ‘codec’ to do this, so it was time for me to learn about that.</p>
<p>The <a href="https://github.com/epgsql/epgsql/blob/devel/doc/pluggable_types.md">manual was helpful</a> but didn’t provide concrete examples for what I was trying to do. I read the <a href="https://github.com/epgsql/epgsql/tree/devel/src/datatypes">codecs included with the library</a>, and that helped me to draft out the basics. What is nice is that you get to write and read the binary data directly, and erlang’s binary pattern matching is wondeful[3], so I didn’t worry about that.</p>
<p>What was not clear from the existing codecs was how to pass data for a user-defined type rather than a built-in one. I first tried just sending the data for the two fields of the type, but I got an error along the lines of “-34234324 columns specified, 2 expected”, which told me I needed to pass the number of columns, and that it knew what I was trying to send. A bit of googling led me to the <a href="https://doxygen.postgresql.org/rowtypes_8c_source.html">source code for the part of postgres throwing this error</a>. What was great was that at line 520, we can see that it reads a four-byte int to get the number of columns. We can then see exactly what else we need to provide, the most interesting of which is the type-oid for each field we are sending.</p>
<p>At first, I just muddled through as the errors I got told me which type-oids it was expecting[4]. However, this felt very fragile, so I jumped back into the code for the library to see what I was meant to do[5]. What was lovely was that on the init of your codec, they pass you an object you can use to query the type-db for your connection. This let me cache the type ids on init, which was ace.</p>
<p>As I didn’t see an example of this anywhere else, I’ve put up <a href="https://gist.github.com/cbaggers/0bce6c3d08c7a5d9441210af507afe1e">this gist</a> of what I came up with. Maybe it can help someone, or perhaps someone will correct me and show me how it’s done :)</p>
<p>With that hurdled officially jumped, I got back to the slog of writing and testing the code.</p>
<p>I finally got unique-creatures working again[6], so it was time to start work on how we will upgrade to this new code. This involved the following SQL code:</p>
<ul>
<li>Code to apply the changes to the DB</li>
<li>Code to initialize all the new columns with data from the existing columns (where sensible)</li>
<li>Make the old unique-creature SQL code forward compatible. This will let you keep playing on an older version without the new and old data going out of sync.</li>
</ul>
<p>With that DB patch looking promising, it was time to test on my local setup. This means:</p>
<ul>
<li>Starting a build of the old DB and server</li>
<li>Building some stuff and placing some creatures in TaleSpire</li>
<li>Applying the SQL patch</li>
<li>Checking that nothing has broken</li>
<li>Shutting down TaleSpire and the server but leaving the DB and files up.</li>
<li>Switching TaleSpire and server to the new branches</li>
<li>Starting up the new TaleSpire and server builds.</li>
<li>Checking that everything still works</li>
</ul>
<p>This is now working well, so I’m feeling pretty happy. I still need to finalize some details with Ree when I meet up with him[7], but maybe we can push the DB patch next week. This gives us what we need to progress on things like polymorph, persistent emotes, 8 stats per creature, and more.</p>
<p>Right, I should get some sleep.</p>
<p>Have a good one!</p>
<p>[0] All of this has been done on a local build of our backend, so I wasn’t risking messing stuff up.
[1] Uniques creatures are stored in the DB instead of being packed with the non-uniques in a file. So to change what data is stored for creatures means changing both places.
[2] In some cases, it’s nice not to be allocating another object.
[3] Seriously - Low-level languages need to steal it. There is some general erlang’y ugliness to it, but the idea is excellent.
[4] I also verified them using <code class="language-plaintext highlighter-rouge">select typname, typelem from pg_type</code>
[5] Erlang, like many venerable languages, grew up before the modern culture around documentation. Common-lisp felt very similar, and you have to get decent at reading other people’s code to learn how to use many things.
[6] with a few hacks in TaleSpire as I haven’t finished the polymorph code yet.
[7] We have to nail down the data we need for persistent emotes.</p>
TaleSpire Dev Log 2832021-07-07T22:56:30+00:00http://www.techsnuffle.com/2021/07/07/talespire-dev-log-283<p>Hi again folks. I’m just dropping in to give an update on my last few days of work.</p>
<p>My current work is focused on changes to support upcoming improvements to creatures. These include polymorph, eight stats per creature, and persistent emotes.</p>
<p>Creatures are a pain to change as unique creatures are stored in the database, whereas non-uniques are stored in a binary format per board [0].</p>
<p>Beyond that, polymorph means we are storing up to ten asset-id and scales per creature. When creature copy/paste eventually lands, it will make it easier to unnecessarily bloat the non-unique data. Because of that, I’m now aggregating the ids into an array and storing indices into that array [1].</p>
<p>Given that we are having to make this new format, it gets very tempting to make other changes too. I spent some time playing around with some ideas here but decided against doing anything more dramatic for now.</p>
<p>Polymorph is one of those features that touches a lot of code. Up until now a creature doesn’t change its visual once it has been spawned. This assumption is pretty baked into the creature code, so changing this is more fiddly than it might seem. Currently, I’m writing the server code for adding/updating the new creature data, but after that, I’ll have to bite the bullet and update that part of the client-side code.</p>
<p>If all goes well, I’ll be visiting Ree next week to work on the game in person. It’s been quite a while, so that’s pretty exciting.</p>
<p>Have a good one!</p>
<p>[0] The uniques being in the database makes it easy to search across the entire campaign. The format for non-uniques is more efficient for bulk updates.
[1] As an example, let’s say you took a creature with three morphs and used copy/paste to make an army of 50 of them. In that case, the naive approach would take 2400 bytes to store the asset-ids. The aggregate approach would use 148 bytes.</p>
TaleSpire Dev Log 2822021-07-01T12:01:32+00:00http://www.techsnuffle.com/2021/07/01/talespire-dev-log-282<p>Heya folks. For the last five days or so, I’ve kept saying to myself that i’d write the next dev-log “as soon as I finish <em>this</em> task,” aaaand here we are :P</p>
<p>So let’s dig into some of the stuff I’ve been poking at.</p>
<h2 id="feature-request-site">Feature-Request Site</h2>
<p>The #feature-requests channel on our discord has been incredibly useful. Still, a while ago, we gained enough of you lovely folks that we’ve outgrown it as a viable solution for managing feature requests.</p>
<p>Since then, the moderators have done a ton of work exploring options we have for moving that somewhere more appropriate. We are not opening it today, but we’ll soon be moving it and the roadmap over to HelloNext.</p>
<p>We’ve going through every request from talespire.com/faq and the last feature roundup and entering them into that system.</p>
<p>We are not ready yet, but we will put the #feature-request channel in read-only mode when we are. This way, nothing is lost, but new folks won’t be confused about where to post.</p>
<p>We will have more news about this in the coming weeks.</p>
<h2 id="bookmarks">Bookmarks</h2>
<p>Back in the code, I’ve been experimenting with bookmarks. Bookmarks are markers that have been given a name. We want you to be able to easily search and jump to any bookmark in the campaign.</p>
<p>I started with making a basic panel to hold the bookmarks so we could start experimenting with behavior. In the pic below, neither the logic nor the graphics are final. But you can see I was messing around with how to visualize bookmarks in the current board.</p>
<p><img src="/assets/images/wipBoardPanel0.gif" alt="wip0" /></p>
<p>From this, we decided that the bookmarks should be integrated into the boards panel, with the boards acting like folders. You can see the WIP of this in these images.</p>
<p><img src="/assets/images/wipBoardPanel1.gif" alt="wip1" /></p>
<p><img src="/assets/images/wipBoardPanel2.png" alt="wip2" /></p>
<p>This is going well. The next step is to add some new functionality to the server to support things like campaign-wide bookmark search.</p>
<h2 id="internal-systems-documentation">Internal systems documentation</h2>
<p>Along with this, we’ve also needed to start looking at housekeeping.</p>
<p>We have a growing codebase, and to keep things running smoothly, we need both Ree and I to be able to drop in easily and get working. To help with this, we are making some internal documentation.</p>
<p>While doing this, I’ve found stuff that I’m reluctant to document as they require some cleanups. Some of which have been on my todo list since the release.</p>
<p>To that end, I finally dove back into the serialization code and got the API to a nice place. This touched a <strong>large</strong> amount of code and so that it took a couple of days to update everything and get it passing all the tests again. We really don’t want to start breaking boards at this point.</p>
<p>From now on, I’m going to try to put aside one day a week where I only work on things that benefit us behind the scenes.</p>
<p>To get good performance, we have had to make alternatives to some of Unity’s systems, and our versions lack the visuals and tools that make working with them enjoyable. Focusing here will make our lives easier and thus make it easier to get fixes and features to you.</p>
<h2 id="ndi">NDI</h2>
<p>One day last week, I needed a change of scene, so I decided to look back into <a href="https://en.wikipedia.org/wiki/Network_Device_Interface">ndi</a> support. I was hopeful that my knowledge of Unity has increased since last time, and I thought I could get it closer to shippable.</p>
<p>Alas, the plugin now requires .NET Standard 2.0, and for various reasons, TaleSpire uses .Net Framework 4.* (Damn Microsoft’s naming schemes to the pits of hell). Switching TaleSpire to .NET Standard 2.0 broke several things[0], so suddenly, this task stopped being a nice way to unwind.</p>
<p>This will have to wait for another day.</p>
<h2 id="unity-on-linux-warning-not-official-support---please-read-below">Unity on Linux [WARNING: NOT OFFICIAL SUPPORT - PLEASE READ BELOW]</h2>
<p>Multi-platform support is NOT coming yet. But it is our long-term goal, and when it does, we want all your mods to work there too. Part of this means getting an understanding of how Unity’s AssetBundle format works on these platforms.</p>
<p>Also, I prefer working on Ubuntu, so it was a great time to try out the Linux port of Unity.</p>
<p>The tests went well. I was able to get a rather broken version of TaleSpire running on Ubuntu and prove that we can use AssetBundles that were made for Windows in the Linux build (because of how we handle shaders internally). Next, we’ll have to do a similar test on macOS and see how it behaves.</p>
<p>There is still a lot of work to do to make the build work properly. I’ll mention just two for now.</p>
<p>We use Window’s ‘named pipes’ feature to handle custom URL schemes. We will need to make something similar. This was a real pain to get working on Windows, so I expect no less elsewhere :P</p>
<p>The pixel-picking system makes use of a custom c++ plugin we made to be able to get results back from the GPU without hangs. We will need to write variants of this for each platform.</p>
<p>Now, I’m sure a few of you have been yelling at the screen to just use wine/proton to support Linux, and we are well aware of these. TaleSpire seems to work great out of the box with Proton, except for the custom URL scheme. This, too, can be fixed, but we’d want to fix that wrinkle in the experience before we’d consider supporting it.</p>
<p>Also, we simply aren’t ready to take on the extra work that supporting multiple platforms requires. We will get there, but it’s something for another day. For now, I’m excited to see the path ahead.</p>
<h2 id="bugs">Bugs</h2>
<p>As usual, there are also bugs to be looked at. We still have cases of board corruption being reported, and so these take top priority. Not all of them turn out to be corruption, but they still are worth squashing.</p>
<h2 id="i-think-thats-most-of-it">I think that’s most of it</h2>
<p>We naturally have plenty of other things brewing. This post was just my stuff, and the rest of the team has been just as busy. We also have things cooking that we aren’t ready to talk publicly about. It’s gonna be fun :)</p>
<p>Until next time folks,</p>
<p>Have a good one!</p>
<p>[0] Including the custom URL scheme and some HTTP connection stuff</p>
Talespire Dev Log 2812021-06-23T00:00:00+00:00http://www.techsnuffle.com/2021/06/23/talespire-dev-log-281
<hr />
<p>layout: post
title: TaleSpire Dev Log 281
description:
date: 2021-06-23 10:09:09
category:
tags: [‘Bouncyrock’, ‘TaleSpire’]
—</p>
<p>Hi folks.</p>
<p>At the time of writing, I’m about to push an update. In the changelog, there is a line that looks like this:</p>
<blockquote>
<p>This patch also contains a change that will hopefully improve performance for machines with high numbers of cores.</p>
</blockquote>
<p>I wanted to talk about this change as I think it will be interesting (and maybe counterintuitive) to some non-programmers out there.</p>
<p>Here is the change: I’ve told TaleSpire to use fewer cores.</p>
<p>!(dun Dun DUUUUUUUUn)[/assets/videos/gopher.gif]</p>
<p>Let’s set things up so I can explain why.</p>
<p>So TaleSpire uses a job-system. A job-system (in this case) is something that you give a chunk of code to run, and it runs it on one of your machine’s cores[0].</p>
<p>This is good as we can queue up lots of work and let the job-system work out where and when to run that code. This often gives us nice performance improvements as we are using all the cores on our machine.</p>
<p>As a programmer, your job then involves writing code in such a way that it can be run concurrently. For example, let’s say you work in catering and you need to make 20 of a certain kind of sandwich, and you have 5 workers. In this case, the ‘job’ is making one sandwich, and with a bit of orchestration, we can have all 5 workers making sandwiches concurrently.</p>
<p>Recently I upgraded my PC to keep up with how TaleSpire is growing[1]. I was very lucky to get a Threadripper CPU with 24 cores[2] which you would think would make TaleSpire much faster, but in some places, it didn’t. In this next image, you can see that running the physics got slower.</p>
<p>!(overhead)[/assets/images/overhead.png]</p>
<p>Weird right?! The version running on more cores took over three times longer.</p>
<p>Why? The overhead of orchestration.</p>
<p>Let’s imagine our sandwich situation again, but let’s say we broke down the steps of making a sandwich even further. So one job is to fetch the bread, another job is to butter the bread, another job is to bring the filling, etc. Now, imagine we still need to make 20 sandwiches, but we have 100 workers. As you can imagine, things get messy.</p>
<p>Not only does instructing each worker take time, but the workers also share access to limited resources (like access to the one fridge). Furthermore, the workers need to sync up to actually assemble the sandwich, and this coordination also takes time.</p>
<p>This is a long way of saying that, at some point, the overheads of managing workers can outweigh the benefits of having more. And so we come right back round to the physics situation.</p>
<p>I like that we can spread the physics work over multiple cores, but I need to limit how many so we don’t get lost in the overhead. As I don’t yet know the right way to restrict the worker count just for the physics, I’ve <strong>temporarily</strong> lowered the max worker count.</p>
<p>This will be increased again when I understand how to control this stuff for each system I care about.</p>
<p>That’s all I’ve got for this post. Thanks for stopping by, and I hope you have a great day.</p>
<p>[0] This is an approximate description. We could say it runs it on one of the worker threads. But given that they are locked with affinity, and we don’t want to have to explain software threads, this will do for now.
[1] I run dev builds of TaleSpire, which gives more information to me, but the cost is that these builds run much slower.
[2] With hyperthreading, that’s 48 logical processors</p>
TaleSpire Dev Log 2802021-06-21T21:52:07+00:00http://www.techsnuffle.com/2021/06/21/talespire-dev-log-280<p>Heya folks,</p>
<p>We are currently working on the last things before we’ll be ready to ship the ‘props on bases’ feature. This is a super exciting one, and I expect it to be used and abused in very interesting ways.</p>
<p>Last night, however, I took a little detour to look at a bug that has been around since the chimera build. Lights that turn off when you get close to them.</p>
<p>Our lights attempt to be the same as the standard Unity lights, except without using Unity’s GameObjects for performance reasons. This means that, like Unity’s lights, ours use a different material depending on whether the camera is inside or outside the light’s area of influence[0].</p>
<p>The lights seemed to be turning off because we were not always updating the material at the right time. Let’s get into the weeds a little.</p>
<p>To avoid using GameObjects, we use <a href="https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.html">CommandBuffers</a> to render the light meshes at the right point during the frame. For dynamic lights[1] we have to rebuild the queue each frame[2] as the light’s position or properties are being changed, but for static lights, it’s different. Static lights, by definition, aren’t changing, so we have a separate CommandBuffer for them which we only update when the board is modified.</p>
<blockquote>
<p>An important detail here is that you can’t just update an element of the CommandBuffer. It has to be cleared and rebuilt</p>
</blockquote>
<p>This approach has a very dumb mistake in it, though. When the camera moves, the material the light is using might need to be changed. I’m not sure why I didn’t notice that while writing the system, though I expect I was rushing for Early Access and forgot. Regardless, this needed fixing.</p>
<p>Internally the TaleSpire board is split into zones, and we apply operations across these zones in parallel. Each zone communicates with the light-manager to enqueue its lights into the CommandBuffers of lights to be rendered[3]. What I tried to do was have one CommandBuffer for static-lights per zone and then only update them dependent on the camera position[4].</p>
<p>This worked, but I noticed something annoying when I looked at the profiler. The update time for the static lights was fine, but the rendering took a significant hit.</p>
<p>This is my test scene. It’s 4096 static lights. I’ve not added anything else so as not to confuse things.</p>
<p><img src="/assets/images/lightPerfScene.png" alt="lights" /></p>
<p>Here is what we have for light rendering before the fix (so with one CommandBuffer for static lights):</p>
<p><img src="/assets/images/lightPerf0.png" alt="old version" /></p>
<p>And here is the same scene, same camera angle, etc, but with one CommandBuffer per zone:</p>
<p><img src="/assets/images/lightPerf1.png" alt="new version" /></p>
<p>Doesn’t that suck? Even though the amount of work to do is the same, the overhead from many CommandBuffers made things take significantly longer[5].</p>
<p>I still hope to ship the fix for the lighting bug this week, but I’m going to have to look at it after we ship the ‘props on bases’ feature, as it’s clear I’ve got more experiments to do.</p>
<p>There is still plenty to try. What I’ll probably start with is switching to three CommandBuffers. One for dynamic lights, one for static lights from zones that have had to update recently, and a final one for static lights from zones that haven’t been updated in a while. This way, we minimize the overhead from CommandBuffers while also minimizing the number of lights being rewritten to the CommandBuffer each frame.</p>
<p>Alright, that’s all for now. Can’t wait to be back with more.</p>
<p>This is gonna be a fun week.</p>
<p>Seeya!</p>
<p>[0] More or less. This is close enough to be able for this discussion.
[1] Dynamic lights are lights being moved or animated by scripts on the tile/prop
[2] Yup, there are places we can optimize here, but this log skims over that detail as we are focused on static lights
[3] This isn’t the exact architecture, so don’t sweat these details. We just want to talk about the issues.
[4] Of course, this is really about the camera position in relation to the area influenced by any of the lights in the zone.
[5] I’d recommend not caring too much about the wall-clock time in this case. Of course, 2 ms matters, but this is also running on a fast CPU. What really stings to me is that it was significantly faster before.</p>
TaleSpire Dev Log 2792021-06-10T14:30:28+00:00http://www.techsnuffle.com/2021/06/10/talespire-dev-log-279<p>While we work on the ‘props as creatures’ feature we talked about the other day, I’ve switched tasks to look at some bugs hitting folks in the community. To that end, the next patch will have the following:</p>
<ul>
<li>A fix for a memory leak in copy/paste</li>
<li>Cases where exceptions should have caused TaleSpire to leave the board but didn’t.</li>
<li>Fixes in the board loading code to make it more robust</li>
<li>A fix to the campaign upgrader, which was having errors</li>
</ul>
<p>The campaign upgrade issue was a regression caused by me when I changed how some internal data was structured. I should have double-checked the upgrader before pushing.</p>
<p>I also am tracking another regression that is causing issues with picking tiles and props when in certain positions. I’ve got a solid idea of where these issues lie, so I’m hopeful that I can get this fixed today.</p>
<p>Once that is done, I’ll push out a patch.</p>
<p>Hope you are all doing well,</p>
<p>Peace.</p>
TaleSpire Dev Log 2772021-06-01T22:00:22+00:00http://www.techsnuffle.com/2021/06/01/talespire-dev-log-277<p>Heya folks,</p>
<p>For the last few days, I’ve had an urge to work on bugs, so I put down the creature improvements and dipped back into the issue tracker.</p>
<p>The first bug I ended up fixing was one I spotted while working on other things. It turned out that long error logs were not being truncated when uploaded to the server, which caused the transfer to fail. This was a nice quick one to fix.</p>
<p>Next up, I was curious about a long-time bug where static lights would flicker briefly when placing new static lights[0]. The code was fiddly as the way I was iterating through the assets was good for performance but not for readability. In the end, it came down to some mistakes where I was writing the GPU data for the lights[1].</p>
<p>The last thing I did yesterday was to fix a bug in hide-volumes where, if you make them too small in any dimension, they became unclickable. This was simply that the physics engine doesn’t handle boxes where the size in any given dimension is zero. I’ve modified the tool so that you can’t make hide volumes that small anymore.</p>
<p>Today I focused on <a href="https://github.com/Bouncyrock/TaleSpire-Beta-Public-Issue-Tracker/issues/964">one specific bug</a>. As the excellent report shows, there are cases where copying large slabs caused the game to crash. A board was provided, which, along with the instructions, made it trivial to reproduce the issue. The problem came down to an oversight I made while trying to reuse code.</p>
<p>The batching for slabs is heavily based on the code for batching a single zone. This worked great, but I had missed the fact that the batches allowed a max of 13000 instances per kind of object. This was far more than is needed per zone, but for certain slabs it’s not hard to go over that limit (50x6x50 single grass tiles, for example). To handle this, I wrote a new struct where the internal array did not have this limit at the cost of some additional allocations.</p>
<p>All these fixes are now merged, and so they should ship in the next update.</p>
<p>Until next time,
Peace.</p>
<p>[0] Static lights are ones that do not animate. The crystals are good examples of this.
[1] I tried to expand on the explanation here, but it got too unwieldy to explain without lots of surrounding code</p>
TaleSpire Dev Log 2762021-05-22T19:20:49+00:00http://www.techsnuffle.com/2021/05/22/talespire-dev-log-276<p>Hi folks!</p>
<p>After the Norwegian national holidays, I poked around fixing the bugs which caused the board corruption that hit some folks a few weeks ago. Part of the fix involved adding an extra piece of runtime data per tile/prop in the board.</p>
<p>This did not affect the size of the saved data, slab size, etc. However, as this did mean an extra 12 bytes of data per tile/prop at runtime, I spent a while looking at reducing that. From now on, we’ll refer to the tile/prop as a placeable.</p>
<p>The data in question was the world position of the origin of the placeable. Positions of props are snapped to the nearest hundredth of a unit, so we don’t need the full range of float values. This means we could opt for a different representation. Half-float does not have the precision we need when values are greater than around 1000, but we could multiply the value by 100 and cast to and int (essentially storing as fixed-point), the values happily fit within the range of a <code class="language-plaintext highlighter-rouge">short</code> so that would take the size increase down to 6 bytes.</p>
<p>Also, all placeables exist inside zones, so we could store the position relative to the zone. A zone is a cube 16 units across, so that means we only need <code class="language-plaintext highlighter-rouge">16 * 100 = 1600</code> values per component or 11 bits per component. That’s ~5 bytes for the three components.</p>
<p>I also looked into other runtime values stored placeable, which could be stored differently now that we have the origin position per placeable. However, I won’t overload this update by going into all that.</p>
<p>After all the poking, I could make things smaller, but it didn’t result in more placeable data per cache-line, so I didn’t think the extra costs would be worth it. I’ll definitely re-examine this when I switch to more of an SOA format for some of the data.</p>
<p>Other than this, Ree and I took some time to plan some changes to the persisted per-creature data. We are adding the data for a bunch of features in one go, even though it’ll take a while to implement the rest of each feature. This is because upgrading the file format takes a lot of care in order to avoid screwing things up.</p>
<p>We will be adding the data for:</p>
<ul>
<li>Persistent emotes</li>
<li>Four more stats per-creature n</li>
<li>The ability to add props to bases and use them as creatures</li>
<li>Polymorph</li>
<li>Extra color data for things like chat bubbles</li>
</ul>
<p>You’ll be hearing a lot more about those as work continues :)</p>
<p>Have a good one folks!</p>
TaleSpire Dev Log 2742021-05-12T13:49:09+00:00http://www.techsnuffle.com/2021/05/12/talespire-dev-log-274<p>Hey again folks!</p>
<p>Progress is slow but steady for me this week.</p>
<p>On Monday, I started adding support for multiple asset-packs to TaleSpire. This is required for future modding support. We knew this was coming, of course, and so a lot of the work had already been done. The main task was deciding how the asset-pack id would be stored in TaleWeaver, writing it into the correct places in the index, and updating TaleSpire to search for packs in a given directory and load them.</p>
<p>The TaleSpire part will still be improved as we don’t want to load all packs the game can find immediately. Each campaign might be using different packs, and loading unnecessary ones waste local and GPU memory.</p>
<p>Other than some board restores, my focus for this week is on the layout code in TaleSpire. The runtime data has a known issue that can make it very fragile to mistakes in asset-packs. The wrong change there can currently corrupt boards. It’s been easy enough to tiptoe around it for now as it’s only us providing assets, but this is unacceptable for modding, so it needs to be fixed. [0]</p>
<p>Naturally, this touches a whole mountain of code, so it’s going to keep me busy for a bit, but it’s gonna be a huge relief to get it done.</p>
<p>That’s all for me for today.</p>
<p>Seeya around folks :)</p>
<p>[0] The changes should not have any visible impact to the game or to the board format. It’s only changing how we juggle the data behind the scenes.</p>
TaleSpire Dev Log 2732021-05-01T19:38:27+00:00http://www.techsnuffle.com/2021/05/01/talespire-dev-log-273<p>Heya folks.</p>
<p>I spent the end of the week looking into the bug with markers which meant they didn’t spawn when joining the board.</p>
<p>This went well and allowed me to clean up a bit of the implementation behind the scenes. However, this also meant I bumped into a bunch of other bugs related to GM blocks. I’ve got through a few of those now, with only one remaining that is stopping me from shipping[0].</p>
<p>The patch fixing markers will come soon. One thing that will be missing from the initial patch is that published boards won’t include markers. This will be fixed later shortly (probably later in the week).</p>
<p>I would also like to start hashing the board state that lives in the database (markers, unique creatures, etc.) and skipping the download if the local cache has the latest data. It’s not critical, but every little helps.</p>
<p>In the last update, I mentioned that I was sent a published-board and some instructions to replicate a nasty board-breaking bug. Once I had this, I was able to repeatedly delete parts of the board and then test if the bug still occurred. I was able to get the large board down to a small chunk of tiles that still triggered the issue. I then started seeing something odd. Occasionally the delete would corrupt the tiles, and occasionally it wouldn’t. I slowed down and took more note of how I was following the steps, and I realized that if the selection box only just enclosed the tiles, the delete didn’t corrupt anything. However, if I used a huge selection box that encompassed the slab, then delete would corrupt the slab. This told me exactly where the problem was.</p>
<p>The board in TaleSpire is divided into 16x16x16unit zones. When we apply changes to the board, we often do so in parallel across the zones. A delete typically needs to scan through all the tiles in the zone to see which ones intersect the selection bounds. However, if the entire zone is enclosed by the selection bounds, we know that every tile must be too. This allows us to implement delete more efficiently in those cases. The bug was in this optimized version of delete. It wasn’t an exciting bug, just a simple case where I wasn’t incrementing an index at the right time[1], but plenty enough to cause havoc.</p>
<p>I then had to update the tests. First, I added a test for this specific case. But then I updated the fuzzer as that should have been capable of finding this on its own. The problem was simply that the bounds for the deletes being generated were not large enough in all dimensions to enclose zones[2]. With these fixes, we are now covered if some regression were to recreate this problem again in the future.</p>
<p>That’s all for now, folks. I’m excited to get the marker fixes out as I’d really like to get working on the marker panel soon.</p>
<p>Peace.</p>
<p>[0] it’s a bug which means that if you modify a sector of a board, then the gm-blocks in that sector aren’t synced to other gms when they join the board.</p>
<p>[1] There was a similar mistake in the undo code for this kind of delete.</p>
<p>[2] The model we fuzz against is a 1d board, this meant that we only ever needed thin selection bounds. It doesn’t hurt to use larger ones so we do that now.</p>
TaleSpire Dev Log 2722021-04-27T10:44:57+00:00http://www.techsnuffle.com/2021/04/27/talespire-dev-log-272<p>Hi everyone,</p>
<p>I had somewhat of an extended weekend as I had forgotten I had some real-life things happening on Monday. All in all, it was lovely to decompress few days, and I’m stoked to get started again.</p>
<p>And we are in luck. A wonderful member of the community has managed to create a board that reproduces one of the nastiest bugs we have right now. For a long time, there has something that can trigger data corruption when cutting a board region and then undoing that cut. However, what has been confounding us is that it has not been possible to replicate the problem consistently. Several lovely folks have got us close, but there was always something that made it very trial and error.</p>
<p>We now have been given a published board with three-step instructions on how to break it.</p>
<p>Needless to say, this now makes this bug tractable. The start will be a lot of trial and error as I try and trim this board down to the smallest thing that still shows the bug. Then I’ll dive into the data and see what is going on.</p>
<p>Board corruption bugs are the ones that upset me more than any others, so I can’t wait to squash this one.</p>
<p>Hope you are all doing well!</p>
<p>Ciao</p>
<p>p.s. In writing this, I’ve already been able to find out that the issues are not related to cut specifically, but delete. I’ve also halved the size of the board that produces the issue. We are on our way :D</p>
TaleSpire Dev Log 2712021-04-24T05:52:28+00:00http://www.techsnuffle.com/2021/04/24/talespire-dev-log-271<p>Heya folks,</p>
<p>A quick update for today. It’s been a great week for bug fixes and content.</p>
<p>The art team just landed the ‘Siege of the Cackling Horde’ pack. I saw the trailer the same time as you folks, so I was giddy :D Sometimes being involved in a thing means you have seen the whole messy process of bringing it to life, and it’s hard not to see the flaws. Just seeing the result is wild as you get to be surprised by it all. I love it.</p>
<p>The next bugs I’ll be tacking are around markers as they are not syncing correctly at the moment. I hope to have them working again early next week.</p>
<p>I’ve also got a server patch to finish, improving how we handle changelogs and client error logging. The latter is just about organizing the data we already have, but it should mean I can track down erroring boards without having to bother as many players.</p>
<p>However, first, it’s the weekend. So I wish you well and will see you next week with more fixes and features!</p>
<p>Peace.</p>
TaleSpire Dev Log 2692021-04-21T13:25:51+00:00http://www.techsnuffle.com/2021/04/21/talespire-dev-log-269<blockquote>
<p>This dev-log is an update from just one of the developers. It does not aim to include all the other things that are going on.</p>
</blockquote>
<p>Hi everyone!</p>
<p>This is my first one of these post-release, and it’s been a weird first week, that’s for sure. In some ways, the game has been much more stable than expected. The servers have held up well, and, from watching twitch, it seems a good bunch of people have been able to play, which is great. This meant I could take the weekend off and start the process of catching up on sleep. This was very welcome after the nervous lack of sleep caused by the approval delay.</p>
<p>Now I’m a bit more compos mentis I’m tackling bugs, focussing primarily on any that cause crashes or stop people joining boards.</p>
<p>One bug in line-of-sight is a bit of a blighter and which I’m now waiting on more information on. We have players for whom <a href="https://docs.unity3d.com/ScriptReference/SystemInfo-supportedRandomWriteTargetCount.html?_ga=2.214863124.2064915093.1619004981-1439509852.1590509612">supportedRandomWriteTargetCount</a> is one, this means that when calling <a href="https://docs.unity3d.com/ScriptReference/Graphics.SetRandomWriteTarget.html?_ga=2.150390197.2064915093.1619004981-1439509852.1590509612">SetRandomWriteTarget</a> you’d think you want to call with index zero. However, as the docs explain:</p>
<blockquote>
<p>The UAV indexing varies a bit between different platforms. On DX11 the first valid UAV index is the number of active render targets. So the common case of single render target the UAV indexing will start from 1. Platforms using automatically translated HLSL shaders will match this behaviour. However, with hand-written GLSL shaders the indexes will match the bindings. On PS4 the indexing starts always from 1 to match the most common case.</p>
</blockquote>
<p>However, <code class="language-plaintext highlighter-rouge">SetRandomWriteTarget</code> checks if the index is greater-than or equal to <code class="language-plaintext highlighter-rouge">supportedRandomWriteTargetCount</code>, and if so throws an out of range exception.</p>
<p>I’m fairly sure this means that the one random-write-target that <code class="language-plaintext highlighter-rouge">supportedRandomWriteTargetCount</code> is referring to must be the render-target. If so, it means that I’ll need a new approach for line-of-sight on these GPUs. What a pain :D [0]</p>
<p>That aside, the next patch will have (amongst other things) a few small fixes to issues stopping players from joining boards.</p>
<p>Have a good one folks</p>
<p>[0] In future, we will be focusing on supporting lower-end machines. It is its own art, so we are trying to get the game proper functioning well first.</p>
TaleSpire Dev Log 2672021-04-09T23:40:40+00:00http://www.techsnuffle.com/2021/04/09/talespire-dev-log-267<p>Heya folks, just a quick one to let you know that work is progressing well. I’m currently fixing the last annoying little bugs in party line-of-sight, and then I’ll be wiring all this up to the fog-of-war code.</p>
<p>I can’t wait to be pushing this all out in an upcoming patch</p>
<p>Peace.</p>
<p>p.s. Just a reminder that fog-of-war will be super experimental. Both visually and functionality wise it is not ready for use in real campaigns. However, it’s gonna be great to have it in your hands so you can start kicking the tires :)</p>
TaleSpire Dev Log 2662021-04-06T15:12:59+00:00http://www.techsnuffle.com/2021/04/06/talespire-dev-log-266<p>Hi folks.</p>
<p>This dev-log is relatively niche but will be helpful to the few who are into it.</p>
<p>In the beta, all tiles had an associated <code class="language-plaintext highlighter-rouge">.boardAsset</code> JSON file which held information about that tile. Since then, we have replaced the multiple JSON files with one index file with a binary encoding. Although better for the game in every measurable way, the binary format makes it hard for the community-driven sites to get info on the tiles, props, and creatures in the game. To that end, we have added an <code class="language-plaintext highlighter-rouge">index.json</code> file which holds a useful subset of the binary file.</p>
<p>You can find this at <code class="language-plaintext highlighter-rouge"><path to the steam files for talespire>/Taleweaver/index.json</code></p>
<p>You can find an overview of the layout here https://github.com/Bouncyrock/TaleWeaver-Community-AssetPack-Index-Format/blob/master/format.cs</p>
<p>It is written in a C#-like pseudo-code but should be enough to get curious folks started.</p>
<p>We also now pack the asset icons into atlases, so the JSON file includes the per asset information of where in the atlas the icon resides.</p>
<p>Have a great day!</p>
TaleSpire Dev Log 2652021-04-04T23:05:18+00:00http://www.techsnuffle.com/2021/04/04/talespire-dev-log-265<p>Heya folks,</p>
<p>It’s been a pretty heavy few days since the release. The beta went from feeling stable back to feeling very beta, and so we’ve been working away trying to get things back to stable as quickly as possible.</p>
<p>The upgrader and crashes have been number one on my plate. It is a gut-punch to see people not be able to play and, so I’ve been pushing hard to get those fixed. I introduced new bugs in the process, which really compounded that gross, ‘pit of the stomach’ feeling. No data was lost, but it sucks to worry people that they might lose their creations.</p>
<p>The bug was in the upgrader and is now fixed. The next update will re-introduce the downgrade button.</p>
<p>As before, the downgrader is only helpful for restoring certain things. As of today, it can fix:</p>
<ul>
<li>Missing hide volumes: in the case you upgraded and they were missing</li>
<li>Boards which were empty after upgrade (potentially except for creatures)</li>
</ul>
<p>The next patch will also fix the bug where, if a board crosses a certain number of lights, all the lights go dark.</p>
<p>Next, I’ll be looking into some of the layout bugs resulting in particular objects (like doors) being at the wrong angles. I’ll also be continuing work on any issues related to the upgrader.</p>
<p>I hope you are all keeping well,</p>
<p>Ciao.</p>
<p>p.s. Oh, and Happy Easter for those of you that celebrate it :)</p>
TaleSpire Dev Log 2642021-04-01T11:57:00+00:00http://www.techsnuffle.com/2021/04/01/talespire-dev-log-264<p>Phew! Well, that was an ‘exciting’ start.</p>
<p>The big upgrader bug was a use-after-release of some malloc’d memory (seriously, who allowed me to have pointers). Took a little while to track it down as different code would trigger the final crash. After that, it was about 3am so I slept like a log.</p>
<p>Back today to start looking at all sorts of stuff. I need to take a quick look at a LoS issue while I’m with Ree, and collaboration is easier, but then I’ll be back on a raft of bugs (especially upgrader-related ones).</p>
<p>If you’ve had issues with the upgrader, don’t worry, the original board data is safe. I’ve written some routines that should also let us roll back and retry the upgrade if needed. We’ll see how it plays out.</p>
<p>It’s gonna be a heavy couple of weeks as we need to land more stuff for the EA release. Nothing to do but do it!</p>
<p>Special thanks to the bug reporters and folks digging through all this.</p>
<p>Seeya around the Discord :)</p>
<p>p.s. The new format for the slabs is coming. While encoding them in the slab and pasting them was simple, undo/redo brought with it a world of pain. Maybe something to chat about on a stream sometime.</p>
TaleSpire Dev Log 2632021-03-13T03:17:06+00:00http://www.techsnuffle.com/2021/03/13/talespire-dev-log-263<p>I wanted to write an update, but it’s all particular, so sorry this one isn’t exciting.</p>
<p>I’m working on persistence again. A while back I decided that, to start with, we will have two files per sector. One which everything will be persisted into and one that will only hold some kind of changes. The reasoning for this is that when a person copies a board to play, they will be moving around creatures and using objects (chests/doors/etc), and we don’t want that to invalidate the whole sector and cause a copy of the tile/prop data on the backend. So instead, the first file acts as a base, and the second optional file is applied on top.</p>
<p>I’ve updated the backend to support this.</p>
<p>I also set things up to keep the three most recent saves for each sector on the server. This will allow for a rollback in case of bugs. However, I’m still not sure what form the UI should take for this in TaleSpire itself.</p>
<p>I’m now focused on updating code to keep track of the board’s modified regions (using the masks from yesterday’s post) and changing how the non-unique creatures are persisted.</p>
<p>Tomorrow will be lots more of this, so I’m gonna go get some sleep.</p>
<p>Seeya around!</p>
TaleSpire Dev Log 2622021-03-11T04:12:46+00:00http://www.techsnuffle.com/2021/03/11/talespire-dev-log-262<p>Hi again, time to recap what I’ve been up to today.</p>
<p>Aside from fixing a bug in picking, my focus has been on board sync.</p>
<p>We will soon be syncing sections of the board on demand rather than downloading the whole board. This means a bunch of communication between the client and server requesting information about the board’s state.</p>
<p>Naturally, instead of sending long lists of sections in the requests, we would prefer a more compact approach. To that end, we use masks a lot. Our regions of the board form a hierarchy:</p>
<ul>
<li>a board contains sub-boards</li>
<li>sub-boards contain sectors</li>
<li>sectors contain sub-sectors</li>
<li>sub-sectors contain zones</li>
</ul>
<p>Using sub-sectors is optional, and so I’m going to skip over those today. All the requests I’m interested in are about sectors.</p>
<p>Each sub-board contains 64 sectors, so a 64bit mask paired with a sub-board id is a very convenient representation for identifying a bunch of sectors. I had written the code to produce and manipulate these masks, but I had seen an issue a while back that suggested that the math was incorrect. Because of that, I put today aside for testing.</p>
<p>It’s rather tricky to stare at a 64bit number and correctly picture the 3D mask it represents, so I knocked up a little project for visualizing them (Unity makes this very easy)</p>
<p>First a visualization for a single sub-board:</p>
<p><img src="/assets/videos/subBoardMask0.gif" alt="single sub-board" /></p>
<p>Each spherical indicator represents one sector. One the center is red, the bit is set in the mask.</p>
<p>From there, we build up to multiple sub-boards so we can test crossing boundaries:</p>
<p><img src="/assets/videos/subBoardMask1.gif" alt="two sub-boards" /></p>
<p>And then finally to testing a full board’s worth.</p>
<p><img src="/assets/videos/subBoardMask2.gif" alt="full board" /></p>
<p>We can now be much more sure about the masks being produced. This is a relief as beyond server requests, they are also used to track modified board regions and whether they have loaded.</p>
<p>Tomorrow I’ll probably be looking set separating the storage of script & non-unique creature state from the rest of the board data. This will minimize the amount of data changed when people are playing boards shared by others. Once the non-unique creature state has moved, I can at long last add copy/paste of creatures. I have held off on that for AAAAAGES as it makes it far too easy to add hundreds of creatures in a single action, and previously all creatures lived in the database, meaning lots of server requests.</p>
<p>Once creatures are added to copy/paste, I’ll publish the new format so the community can be ready to support it when it lands.</p>
<p>Anyhoo I’ll keep ya posted with all that in the coming logs.</p>
<p>Goodnight.</p>
TaleSpire Dev Log 2612021-03-09T09:40:12+00:00http://www.techsnuffle.com/2021/03/09/talespire-dev-log-261<p>Howdy folks!</p>
<p>We had a round of testing to see where we are at, and it’s pretty positive. There are significant bugs to squash, and the new backend hasn’t landed yet. However, it took a little while to start getting crashes, so that was positive. It is expected at this point for it to be somewhat broken, we just needed to see if there were things we weren’t expecting (for the record, the first time we did this for the beta, it was a nightmare :D)</p>
<p>I’ve spent this last week on bugs:</p>
<ul>
<li>an issue with assets jumping when being picked up making tooling feel bad</li>
<li>a crash when an asset is missing from the pack rather than using the dummy asset</li>
<li>atmosphere not being correctly applied when joining the board</li>
<li>fix sync of spaghet scripts</li>
<li>get gm requests working again</li>
<li>fix a bug where a scripted asset with no MeshRenderer freaked out my batcher</li>
<li>an issue where static lights were not being invalidated on change</li>
<li>fix an incorrect vertical offset of slabs that were elevated when copied</li>
</ul>
<p>And then a stinker: Assets using our fake-translucency plastic shader don’t look correct with the new lighting system.</p>
<p>This thing has dogged me for days now. It’s on the critical path as this shader is fundamental to how TaleSpire looks and, more immediately, we can’t record the trailer until it’s fixed.</p>
<p>Here’s what the problem looks like:</p>
<p><img src="/assets/images/fireWrong.png" alt="incorrect fire effect" /></p>
<p>It is happening because we had to write our own lighting to move tiles and props away from Unity’s GameObject system (because they were too slow). However, this means that a lot of lighting magic that Unity did for us is our problem instead.</p>
<p>Fire, for example, used to render in multiple steps.</p>
<p><img src="/assets/images/fireOld.png" alt="incorrect fire effect" /></p>
<p>First, the base object is rendered. Then it is rendered once for each nearby light, with the result being additively blended with the previous pass.</p>
<p>I had a look at the shaders to see how things were split up, and I figured that we could replicate the effect in a single pass if we could get the light data in a buffer along with some info on where in the buffer to find the lights for the current rendering asset.</p>
<p>But first I needed to make the shader. This took me a day and a fair bit of swearing (visuals are not my strong suit), but I did get a proof of concept working, enough to get back to the renderer.</p>
<p>This also proved to be a real pain in the ass. The batching code that collects light information was written to group lights of the same kind, regardless of what tile or prop they belong to. This is done to ensure that, when rendering, there are a few unnecessary state changes as possible. However, we now need the information grouped per-tile/prop and in a ComputeBuffer. Swearing and coding continued until I had a nice little allocator for the ComputeBuffer, and all the batching jobs were updated to use it.</p>
<p>Finally, we could port over the new shader and wire everything up.</p>
<p><img src="/assets/images/fireNew.png" alt="incorrect fire effect" /></p>
<p>At last! We have something looking correct.</p>
<p>We still need to wire up the changes to the other plastic shaders used for crystals and the like, but that is trivial now that we know the approach works.</p>
<p>Today I’m back on backend work, and I will be for the next week. This thing is shaping up fast now.</p>
<p>Seeya in the next one folks.</p>
<p>p.s. This dev-log is rather late so the next one will be coming very soon</p>
TaleSpire Dev Log 2592021-02-15T19:07:20+00:00http://www.techsnuffle.com/2021/02/15/talespire-dev-log-259<p>Hi everyone!</p>
<p>Progress is excellent behind the scenes. Unfortunately for the dev-logs, my recent work hasn’t made for very interesting writeups.</p>
<p>My focus has been on and around the new board sync.</p>
<p>In TaleSpire, each change to the board is applied in the same order on all clients. Naturally, this means we need to start from a common state. So if you join a board after someone else, how do we catch up? We will need to fetch the board from the GM, but that is not enough as, in the time we are waiting for the board to transfer, the board could have changed. In the current build, we use the following approach:</p>
<ul>
<li>When you join the board, start recording all board change operations which arrive (each has an id)</li>
<li>Ask the GM’s client for the board. The client will write the ID of the most recent board change operation that had been applied before the sync along with the board data.</li>
<li>When the board is finally delivered, deserialize it and discard all the recorded board change operations with IDs older than the one in the board sync file.</li>
<li>Apply the remaining board change operations in order</li>
</ul>
<p>We now have a complication. Each client may only have a portion of the board locally. Ideally, we only want to sync the parts of the board that have changed this session and load the rest from the server. It took me a bit to work out that we should only need to pause applying changes if an operation intersects a zone that exists but is not local yet. We don’t need to do the same ‘id juggling’ as the former approach (although it’s still required for the parts coming from the GM).</p>
<p>Part of the above is about keeping info on what zones or sectors are loaded, modified, etc. I spent some time making sure we do this efficiently.</p>
<p>When we release this, we will need to upgrade all the boards that are already out there. We usually do this transparently on load; however, this time, the changes are so fundamental that I want to make it a dedicated step to reduce the chance for error. This means that once the update is out, players won’t be able to connect to a campaign until it has been upgraded by the campaign owner. The owner will simply have to click an update button and allow the process to run to completion[1].</p>
<p>I’ve added a new board format to my branch where I am putting the new logic, and I’ve started writing the system that will handle the update itself. This change touches so much code that I find myself jumping back and forth doing small parts on one system to allow progress in another.</p>
<p>As we are making changes to the board format again, I made sure all the tests were still passing. We don’t have a CI server, so these tests are run less frequently than they otherwise would be [0]</p>
<p>As I was working on the format, I decided to move board-wide settings like the water height and most recent atmosphere to the database. As I was implementing it, I got really annoyed at some poor JSON was for encoding the data, and, against my better judgment, I decided to spend the weekend on a little diversion.</p>
<p>That diversion was to implement binary messaging to the server. In previous posts, I had talked about experimenting with <a href="https://erlang.org/doc/apps/erts/users_guide.html">ERTS</a>, erlang’s built-in binary format, and how I had already written some experimental code for making ERTS messages from C#. The plan was to update the code generator on the backend to support custom types in requests and replace the JSON serializer with ERTS.</p>
<p>This was going pretty well, it was tricky, but by late Sunday, I had the first messages set from TaleSpire to the backend. However, when I thought I had about an hour of work remaining, I spotted a critical issue.</p>
<p>I had wanted to use specific sized integers for certain parts of the message. When sending messages from C#, this was easy to control as it was my code. However, on the server, I used a method called <code class="language-plaintext highlighter-rouge">term_to_binary</code>, which is analogous to <code class="language-plaintext highlighter-rouge">Json.Encode</code> or similar from your favorite language. What that method helpfully does is to look at the value it’s encoding and pick a suitable encoding, which could be bigger or smaller than I needed[2].</p>
<p>This sucked. I didn’t want to handle those different possibilities on the client-side, but there wasn’t a way to control it on the server side without writing a custom ERTS encoder. Of course, that defeats the point of using ERTS as I wanted something to reduce the amount of work I needed to do.</p>
<p>Apparently, I refuse to pay heed to the sunken cost fallacy. If I were going to write an encoder, I would make something much less dynamic than ERTS that would make smaller messages. For example, there is no point in encoding the values’ types if the message has a fixed layout. And so that’s what the next 8 hours were — a dogged hammering at the keyboard until the format came to life.</p>
<p>So we now have our own format. Because it’s a binary format, the data is usually smaller than the equivalent JSON text representation, and writing the messages is simply pushing bytes into an array rather than conversion to a more human-readable format.</p>
<p>An awesome side-effect is that it’s now easy to add support for more types. For example, I added a dedicated UUID type that sends them in their usual 16byte representation rather than the 36 chars needed for the text version. Given how many UUIDs we pass up and down, this alone is a win.</p>
<p>I cannot stress enough how awesome erlang’s syntax is for handling binary data. It boggles my mind that low-level languages haven’t jumped to add something similar. For example, let’s start with 4 bytes of data in binary.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Data = <<255,255,255,255>>
</code></pre></div></div>
<p>Now let’s say we want to unpack 3 coordinates from those 4 bytes, where the X and Z coords are 11 bits, and the Y coord is 10:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><<X:11, Y:10, Z:11>> = Data
</code></pre></div></div>
<p>That is binary pattern matching in action. X is bound to the first 11 bites, Y to the next 10, etc.</p>
<p>Obviously, we often want to read out specific values where we care about endianness. Below we see how to read the first 20 bits as a little-endian integer and the final 12 bits as a big-endian integer.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><<X:20/little-integer, Y:12/big-integer>> = Data
</code></pre></div></div>
<p>The result from the above is X=2047, Y=1023, and Z = 2047.</p>
<p>You can also bind the rest of the data to a variable:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><<ArrayLen:16/little-unsigned-integer, RestOfTheData/bytes>> = SomeLargeBinary
</code></pre></div></div>
<p>The above binds a <code class="language-plaintext highlighter-rouge">ushort</code> to <code class="language-plaintext highlighter-rouge">ArrayLen</code> and the rest of the bytes in <code class="language-plaintext highlighter-rouge">SomeLargeBinary</code> to <code class="language-plaintext highlighter-rouge">RestOfTheData</code>. This made chaining patterns trivial and allowed me to make a tiny binary encode/decode library in no time.</p>
<p>So yeah, erlang is still a great choice :D</p>
<p>That’s enough for now. This week I’ll keep working on board sync.</p>
<p>See you around</p>
<p>p.s. Although I don’t write for other team members, there is plenty of cool stuff happening. Although we are not changing our release date from ‘Early 2021’ we are still on track for our own internal goals. TaleSpire is coming. The release is gonna be fun!</p>
<p>[0] Tests run with a specific build flag as we have to track additional data that we do not want in normal builds.</p>
<p>[1] The reason is that some data which used to live in the database (non-unique creatures) is moving to the board files and the board data, which used to be one file, is now many. I am nervous about an error or network failure resulting in partially updated boards, and so making it a big, dedicated step seems safest.</p>
<p>[2] I want to stress again that this is an excellent decision for how ERTS is used within erlang. It was just a failure on my part that led to this issue.</p>
TaleSpire Dev Log 2582021-02-01T02:59:01+00:00http://www.techsnuffle.com/2021/02/01/talespire-dev-log-258<p>Hi folks!</p>
<p>Small update from me tonight.</p>
<p>After much wrestling with SQL quoting rules, I’ve got the partitions for board-data set up in the DB. I can now reserve an entry for a file, commit that once an upload has completed, insert the sector info into the correct place, and then query the board hierarchy to find out what files I need to pull for a given area of the board. To be clear, we aren’t uploading real per-sector data yet, but the database doesn’t know that :P.</p>
<p>With that looking promising, I started thinking about the upload/download managers again. To support resuming incomplete downloads, I knew I’d be using the HttpWebRequest API. However, I don’t have that much experience with it so, as an exercise, I rewrote the upload/download routines that we currently use when syncing the entire board. This was great as I didn’t get distracted overthinking the API or wondering how to get a realistic data-set. I could just use TaleSpire as it stands on our branch.</p>
<p>This went well, and now I feel ready to sink my teeth[0] into the real work of the managers.</p>
<p>One data-set I haven’t talked about much recently is the non-unique creatures. We want to remove their data from the database to support more of them, which means they need to be stored somewhere else. I’ll work out how I want this for the Early Access release, and then I can get coding.</p>
<p>Goodnight folks.</p>
<p>[0] No, not <em>sync my teeth</em>, and how dare you pun in this place.</p>
TaleSpire Dev Log 2572021-01-29T02:18:17+00:00http://www.techsnuffle.com/2021/01/29/talespire-dev-log-257<p>Hi folks, the last week has been one where giving updates would have been challenging as most days have consisted of me grumbling at a whiteboard, but let’s do it anyway :p</p>
<p>At least half the week was spent working out how I wanted to store board data on the backend. We knew we didn’t want sharing boards to result in everyone having a duplicate copy of the board on the backend as this would mean 1000 people using a shared board would result in 1000x the storage requirement. Obviously, that means we want different people’s boards to refer to the same data. However, almost all play sessions involve small changes to the board (for example, if a fireball destroys a wall), and we don’t want that first change to result in duplicating the whole board, so we want to be able to store only what has changed.</p>
<p>This has been long-planned under the title ‘per-zone sync’, the implementation details are a little trickier as we start looking at worst-case numbers, however. A board (today) is 2048x1024x2048 units in size, and we divide that into 16x16x16 unit zones. That means the worst-case number of zones is 128x64x128=1048576 per board. Not only is that a lot, but each zone isn’t (usually) much data, and we definitely don’t want the book-keeping to be larger than the zone data. Furthermore, TaleSpire needs to be told what to download, so more separate files mean more things the backend has to tell the frontend. Let’s do something simple to help this; let’s serialize zones together. We will pack zones in 2x2x2 zone chunks (which will later be called sub-sectors), meaning the worst case is now 64x32x64=131072. Not great, but better.[0]</p>
<p>However, these chunks are still reasonably small. They are great for localized changes, but it’s overkill for the board regions that aren’t changing frequently. So let’s make an alternate chunk size of 8x8x8 zones. We can then choose between these larger ‘sectors’ and the smaller ‘sub-sectors’, which gives us options when dividing up the world[1].</p>
<p>So we have 4x4x4=64 sub-sectors to a sector, but quite a few more sectors to the board. If we follow the same 4x4x4 pattern of sub-sectors to sectors, we get 4x2x4=32 regions of 32x32x32 zones per board. Having a consistent branching factor (64) is nice, and 32 regions will be handy in a couple of paragraphs time.</p>
<p>Now we have something that can work as a sparse hierarchy of spatial data, let’s look at copies again. Rather than tracking ancestry at the board level, We can have each sector point to the sector it is descended from. This lets us easily walk backward and find all the files contributing to your version of that sector.</p>
<p>One concern arising from recursively walking the sector data is query time. We directly address rows, so the main factors to query speed (I think) will be table size and index kind. Focusing on size, we can actually do something to help here. The first thought is to use <a href="https://www.postgresql.org/docs/10/ddl-partitioning.html">table partitioning</a>, but I didn’t like it for two reasons:</p>
<ul>
<li>It’s built on table inheritance, and that doesn’t seem to maintain the reference constraints in the child tables</li>
<li>It looks tricky to ensure we keep sectors from the same region in the same partition</li>
</ul>
<p>However, we previously mentioned those 32 regions of the board. What if we treated those regions as our partitions, made 32 tables, and handled it manually. Apart from it being a lot of work, it seems promising. However, most boards aren’t going to use every region, so we’d still get clustering around the origin. Unless, of course, we introduce some factor to more uniformly spread the data across the partitions. The way I’m looking at doing it is taking the region-id, adding the low bits from the board-id of the oldest board ancestor, and then modulo by 32. This spreads the data across partitions and means that regions of copied boards live in the same partition as regions of the boards they were copied from[2].</p>
<p>With the DB side starting to make sense, I had hoped to find a way to trivially dedupe the uploaded data itself in S3. The original idea was to name the uploaded chunks by their hash. That way, two identical chunks will get the same name and thus overwrite each other. The problem is that you can get hash collisions, but these can be minimized by hashing the data with one hash[3], prepending that to the data, and then hashing with a second hash. S3 will validate an upload with md5 if you request it to, so that seemed a logical choice.</p>
<p>However, a huge issue is that we have only made it hard to accidentally get a hash collision; it doesn’t stop people from being malicious and trying to break other peoples’ boards. There is no great answer to this. The only thing to do is upload with a unique name and then have a server-side process validate the file contents before allowing it to be de-duplicated. I’ll be adding this after Early Access has shipped.</p>
<p>Speaking of md5, now that compression is jobified, the md5 calculation is one of the only parts of saving a board that we don’t have in job yet. To remedy this, I’m looking at porting the md5 implementation from https://github.com/kazuho/picohash to Burst. This looks straightforward enough and will be a nice change from the planning of the last week.</p>
<p>While planning, I’ve been weighing up different potential implementations, and this has required reading into a bunch of SQL and Erlang topics. One neat diversion I went on was looking at binary serializing for our websocket messages to the server. Previously I had looked at things like protobufs and captnproto. However, I had ignored the fact that Erlang has <a href="http://erlang.org/doc/apps/erts/erl_ext_dist.html">ETS</a>. ETS is a simple binary format which it uses when communicating between nodes. Some of the message kinds only make sense within Erlang, but one possible subset was identified for external use called <a href="http://bert-rpc.org/">BERT</a>.</p>
<p>I played around with BERT, but I didn’t like how floats were encoded. ETS has a newer, nicer way of encoding them, so I took some inspiration from BERT and started writing a simple Burst compatible way to write ETS messages from C#. I was quickly able to encode maps, tuples, floats, and strings, and sending them as binary messages to the server was also trivial.[4]
The next step would be decoding and updating the backend’s code generator to use this. It should result in faster encoding/decoding with no garbage and smaller messages over the wire. Still, I’m going to leave this until after the Early Access release to avoid risking delaying the release even further.</p>
<p>I think that is the lot for this last week! I’m hoping to start making progress more quickly now that the plan has taken shape. I’ve already started writing the SQL and aim to start the erlang and c# portions as soon as possible.</p>
<p>Hope you are all well,
Peace.</p>
<p>[0] Of course, we can also add limits of how many zones can be in each board, and we’ll almost certainly do that.</p>
<p>[1] This hierarchical spatial subdivision is probably making some of you think of octrees. Me too :P however, this approach will give us much shallower trees, which will be good for limiting the depth of the recursive SQL queries to get the data.</p>
<p>[2] Naturally, this still means that popular boards will introduce a lot of extra rows to the partition in an unbalanced way. Still, each popular board will be uniformly spread across the partitions too, so it all should even itself out.</p>
<p>[3] I’m looking at xxhash3 as it’s super fast and has a Burst compatible implementation already made by Unity</p>
<p>[4] Given that we are now not using BERTS, we could go another step further and make our own binary format with only what we need. This is definitely an option, but moving to ETS first is a decent step that means we don’t have to write an encoder/decoder for the erlang side. We can swap out the implementation once we have moved to binary messaging and ironed out the kinks.</p>
TaleSpire Dev Log 2562021-01-22T00:55:46+00:00http://www.techsnuffle.com/2021/01/22/talespire-dev-log-256<p>Heya all!</p>
<p>Today and tomorrow are somewhat taken up by what I’m hoping is the last of the moving business. However, I’ve had some time this evening to start digging into the per-zone sync.</p>
<p>I started off working out what we would need in a download manager. As I was doing this, I got distracted thinking about the compression and decompression of zones. Currently, we use GZipStream, which has been fine, but it would be way better if it played nicer with Unity’s job system and if it could work without creating garbage. I also love that <a href="https://docs.unity3d.com/ScriptReference/Unity.IO.LowLevel.Unsafe.AsyncReadManager.Read.html">AsyncRedManager.Read</a> gives a job handle as that would allow us to chain the load and decompress if we could work out how to decompress from a job.</p>
<p>I knew that Burst compiled jobs can call native code, so I searched for a simple zlib implementation written in C. I found a library called <a href="https://github.com/richgel999/miniz">miniz</a>, which looked ideal if you use the low-level functions as it doesn’t allocate any memory when de/compressing. I then found a .Net wrapper called <a href="https://github.com/jsm174/net-miniz">net-miniz</a>, and after a read of its code, I forked it so we could make a version that works with Unity’s job system.</p>
<p>It was pretty easy work, and we now have exactly what we wanted. I’ve not settled on an API yet, but I’ll publish the fork when I’ve worked that out and tested it a bit.</p>
<p>So it’s been a very successful evening. A mention I have more work to do at the old flat tomorrow, but I expect I’ll get some time to start looking into database changes we’ll need for per-zone-sync and eventual support for sharing boards :)</p>
<p>Ciao</p>
TaleSpire Dev Log2021-01-19T14:20:47+00:00http://www.techsnuffle.com/2021/01/19/talespire-dev-log-xff<p>Heya folks,</p>
<p>Lots of coding has been going on behind the scenes as usual. A lot of that work has been centered around the picker.</p>
<p>In the STAGING buffer technique we were using, I knew that we were meant to pull the results on the following frame. However, I had been cheeky and tried to pull it on the Unity CommandBuffer event <code class="language-plaintext highlighter-rouge">AfterEverything</code>. This didn’t work well, and I saw significant delays on the render thread; it makes sense and felt like something I would have to change.</p>
<p>We added support for pixel picking to the dice and made it play nice with hide volumes (so you can’t pick things that should be invisible). While coding that I noticed that hide volumes were not applied correctly to pasted tiles or tiles created by redoing placing tiles. I chased this down to a simple case where we were not setting up a tiles state before trying to set apply the hide volumes to that state.</p>
<p>On the subject of picking: After a little experimentation, we switched from using the Depth buffer from the currently rendering scene to giving the pixel picker it’s own depth buffer. This way, we could easily exclude things we didn’t want to be pickable. We decided to try this after noticing the tile/prop/creature preview would be detected by the picker when placing them, as that would have been a bit annoying to work around.</p>
<p>The next task was simple but takes a little context to explain. The picker and physics raycasting code returns a reference to tiles/props, which is basically just a fancy pointer[1]. Tile & Prop data can be moved when a zone is modified so, if you want to store a reference to a tile/prop that is valid for multiple frames, we make a handle that has a safe way to lookup an asset at the expense of some performance. Previously you would create a new handle whenever you wanted to hold on to a tile/prop, but allocating this object means that the GC has to clean it up at some point[2]. To mitigate this, we turned the handle into a sort of container. You create it once and then push a tile/prop reference into it. You can then replace the reference in the handle or clear it.</p>
<p>Then there were a bunch of physics-related tasks:</p>
<ul>
<li>A couple of bugs in our wrapper around <code class="language-plaintext highlighter-rouge">Unity.Physics</code> resulted in ray-casts hitting things that had already been destroyed.</li>
<li>Replace the mesh colliders around the creature base with a single CylinderCollider</li>
<li>Dice has not been set up to interact with creatures</li>
<li>Improving handling of framerate spikes</li>
<li>My code for creating the cylinder geometry was off by 90degrees, so we ended up with stuff like this happening
<img src="/assets/images/cylinder.png" alt="wat" /></li>
</ul>
<blockquote>
<p>Note: It is super obvious what is happening in this picture, but when I noticed it, the only cylinder collider was around the creature’s base, and so I thought I still was messing up creature/dice interactions. I had a good solid facepalm when I finally saw what was up.</p>
</blockquote>
<p>The following item on the bug list was that shadows were not respecting hide volumes. This was harder than it should have been due to hiccups we’d had while rewriting the engine. Let’s get through the basics:</p>
<ul>
<li>GameObjects were too slow</li>
<li>Unity has calls to render objects using instancing without using GameObjects</li>
<li>Those calls cause issues with shadows as Unity can’t them per-light and so does way too much work</li>
<li>Unity added a new thing called a BatchRendererGroup, which lets you make big batches of things to render and implement the culling yourself (Yay!)</li>
<li>We implemented our new code based on the BatchRendererGroup</li>
<li>What wasn’t documented was that specifying per-instance data only works if you are using their new Scriptable-Rendering-Pipeline (ARGGGHH)</li>
<li>We do not have the time to write our own rendering-pipeline and port all the shaders</li>
<li>We decided to implement our own GPU culling and use indirect rendering to draw the board with shadowcasting turned off and keep using the BatchRendererGroup for shadows.[3]</li>
</ul>
<p>This has worked well, but as we can’t get per-instance state to the shadow-shaders, we can’t use our usual approach to cull things in hide volumes. Instead, we now check the hide volume state when doing the CPU-side culling of shadows. This would require too many pointer lookups, so we added an extra job to copy the data into another buffer for fast lookup.</p>
<p>Dice ‘sleep’ was next on the agenda. When an object has stopped moving in most physics engines, the engine puts the object to sleep and stops simulating it. The new Unity.Physics library does not have support for sleep built-in as the library is intended as something to be built upon, and different sleep implementations have various tradeoffs. Instead, what we are doing is simply snapping the die to a stop when it’s velocity and position has stayed at a low value for a given period of time. This stops the dice from drifting but doesn’t stop the physics simulation. To prevent things from (computationally) getting out of hand, we are adding an auto-despawn of dice rolls once they have been at rest for a given duration. This also starts addressing some feature-requests about dice management, so that’s nice.</p>
<p>One thing that is not resolved is that we are seeing inconsistent execution time for some operations related to picking. It’s a bit confusing as it’s the enqueuing of the task that is taking a long time rather than the task itself. I need to collect some more data and post a question on the Unity forums as I’m a bit lost as to why this is happening.</p>
<p>Amongst all this is the usual slew of small issues and fixes, but this is getting plenty long enough already.</p>
<p>I’m immediately switching tasks from front-end work to the per-zone sync that I’ve been talking about for months. The next updates will be all about that.</p>
<p>Until then :)</p>
<p>[1] We implement them as readonly ref structs, so we don’t accidentally store them
[2] GC really is the enemy in games. You might be surprised how badly timed it hurts performance even when using an incremental GC.
[3] Another approach would have been to write our own lighting/shadow system, but that’s also terrifying given how late we already are with the Early Access :/</p>
TaleSpire Dev Log 2542021-01-10T19:02:31+00:00http://www.techsnuffle.com/2021/01/10/talespire-dev-log-254<p>Helloooo!</p>
<p>It’s been an intense few days of coding, and I’m back to ramble about it.</p>
<p>The issue that arose was the behavior of ‘picking’. Picking is the act of selecting something in the 3D space of the board. The current implementation of this uses the raycasting functions of the physics library. For this to work well, you need the collision mesh to match the object as well as possible. However, in general, the more detailed it is, the more work the CPU needs to do to check for collision [0].</p>
<p>Here is an example where we are striking a balance between those two requirements:</p>
<p><img src="/assets/images/colliderTradeOff.png" alt="tree collider tradeoff" /></p>
<p>Most of the time, it works just fine; however, it makes the game feel unresponsive when it doesn’t. Worse, it could take you out of a critical moment in the story, and that would be a real failure on our part. Here is an example of where this can go wrong:</p>
<p><img src="/assets/images/oldPickingIssue.png" alt="bear/tree issue" /></p>
<p>In this case, the bear is selectable just fine near the paws, but higher up, the tree’s simplistic collider is blocking the selection.</p>
<p>What would be ideal is to pick the creature based on the pixel your cursor is over. But how to do that?</p>
<p>The basic approach is very simple. Render everything again to an off-screen buffer, but instead of the typical textures, you give each object a different color based on their ID. Then you can read the color at the pixel of the screen you are interested in, and whatever color is there is the ID of the thing you are picking. Sounds neat, right? But I’m sure you’re already wondering if this might be very costly, and the answer would be yes!</p>
<p>We can do a lot to make it less costly though. We know we only need to consider assets in zones[1] that are under the cursor. Next, what if we have a cheap way of checking if they <em>might</em> be the asset under the cursor? In that case, we could greatly reduce the amount to draw.</p>
<p>Indeed we can do both those things. For the first, it’s done pretty much as you would expect; check if the ray from the cursor intersects each zone. For the second, our culling is done on the GPU, so we add some code to our compute shader to also check if the AABB (Axis aligned bounding box) is intersected by the ray.</p>
<p>That’s not bad, but we still need to render the whole of each asset. That’s a lot of pixels to draw, given that we only care about one. So we use a feature called the ‘scissor rect’ to specify which portion of the screen we want to draw to[2]</p>
<p>Again this is good, but it would be good to limit the number of those fragments that remain that we need to run the fragment shader for. To do this, we simply render the assets for picking after the buffer has been populated and use the screen’s depth buffer. As long as we keep our fragment shader simple, we can get <a href="https://www.khronos.org/opengl/wiki/Early_Fragment_Test">early z-testing</a>[3] which does precisely what we want.</p>
<p>So now we’ve limited the zones involved, the assets from the zones involved, the triangles in the assets involved, and the fragments involved. This is probably enough for the rendering portion of this technique. Now we need to get the results back.</p>
<p>And getting the data back, as it turns out, is where this gets tricky. Ideally, we want it on the next frame; however, we also need it without delaying the main thread, which is hard as reading back means synchronizing CPU and GPU.</p>
<p>To start, I simply looked at reading back the data from the portion of the texture we cared about. But Unity’s GetPixels doesn’t work with the float format I used for the texture. So after some faffing around, I added a tiny compute shader to copy the data out to a ComputeBuffer. It was immediately evident that using GetData on the ComputeBuffer caused an unacceptable delay, between 0.5 and 0.6ms in my initial tests. That is a long time to be blocking the main thread, so instead, I tested <a href="https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.RequestAsyncReadback.html">RequestAsyncReadback</a>, it does avoid blocking the main thread, but it does so by delivering the result a few frames later. This <em>could</em> work, but it’s a shame to have that latency.</p>
<p>After a bit more googling, I learned about the <a href="https://docs.microsoft.com/en-us/windows/win32/api/d3d11/ne-d3d11-d3d11_usage">D3D11_USAGE_STAGING</a> flag and how we could use it to allow us to pull data on the following frame without blocking the main-thread. Soon after I stumbled on <a href="https://www.gamedev.net/forums/topic/691724-gpu-write-cpu-read/5354495/">this comment</a>.</p>
<p>MJP is an excellent engineer who’s blog posts have helped me to no end, so I was excited to see there might be an avenue here. There was only one sticking point, Unity doesn’t expose <code class="language-plaintext highlighter-rouge">D3D11_USAGE_STAGING</code> in it’s compute buffers. This meant I needed to break out c++ and learn to write native plugins for Unity.</p>
<p>Thanks to the examples, I was able to get the basics written, but something I was doing was crashing Unity A LOT. For the next <code class="language-plaintext highlighter-rouge">< insert embarrassing number of hours here ></code> I struggled with this that mapping the buffer would freeze or crash Unity unless done via a <a href="https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.IssuePluginEvent.html">plugin event</a>. In my defense, this wasn’t done in the example when writing to textures or vertex buffers, but I’m a noob at D3D, so no doubt I’m missing something.</p>
<p>Regardless, after all the poking about, we finally get this:</p>
<p><img src="/assets/videos/pixel_picking.gif" alt="Pixel Picking" /></p>
<p>The numbers on the left-hand side are the IDs of the assets being hovered over.</p>
<p>I don’t have the readback timing for the final version; however, when the prototype wasn’t crashing, the readback time was 0.008ms. Which is plenty good enough for us :)</p>
<p>I’ve still got some experiments and cleanup to do, but then we can start hooking this up to any system that can benefit from it[4]</p>
<p>I hope this finds you well.</p>
<p>Peace.</p>
<p>[0] In fact, using multiple collider primitives (box, sphere, capsule, etc) together is much faster than using a mesh, but the assets we are talking about today are the ones that can’t be easily approximated that way.</p>
<p>[1] In TaleSpire, the world is chunked up into 16x16x16 unit regions called zones</p>
<p>[2] IIRC it also affects clearing the texture, which is handy.</p>
<p>[3] I used the GL docs as I think they are clearer, but this is also true for DirectX.</p>
<p>[4] Especially for creatures. We only used the high-poly collider meshes there as accurate picking was so important.</p>
TaleSpire Dev Log 2532021-01-07T01:53:21+00:00http://www.techsnuffle.com/2021/01/07/talespire-dev-log-253<p>Evening all.</p>
<p>It’s been a reasonably productive day for me today:</p>
<ul>
<li>I fixed two bugs in the light layout code</li>
<li>Fixed pasting of slabs from the beta [0]</li>
<li>Fixed a bug in the shader setup code</li>
<li>Enabled use of the low poly meshes in line-of-sight (LoS) and fog-of-war (FoW)</li>
<li>Started writing the server-side portion of the text chat</li>
</ul>
<p>I have not made any UI for chat in TaleSpire yet. My focus has been on being able to send messages to different groups of players. Currently supported are:</p>
<ul>
<li>all players in the campaign</li>
<li>all players in the board</li>
<li>specific player/s</li>
<li>all gms</li>
</ul>
<p>Next in chat, I need to look at attachments. These will let you send some information along with the text. Currently, planned attachment kinds are:</p>
<ul>
<li>a specific position in the board</li>
<li>a particular creature in the board</li>
<li>a dice roll</li>
</ul>
<p>We will be expanding these, but they feel like a nice place to start and should help with the flow of play in play sessions.</p>
<p>We’ll show you more of the UI as it happens.</p>
<p>One cute thing I added for debugging is the ability to render using the low-poly occluder meshes instead of the high-poly ones. Here is an example of that in action. You can see that, in this example, the decimator has removed too much detail from the tiles. This is handy for debugging issues in FoW and LoS.</p>
<p><img src="/assets/videos/renderOccluders.gif" alt="occluder bug example" /></p>
<p>Alright, that’s all for today,</p>
<p>Seeya</p>
<p>[0] The new format is a little different, and I’ll publish it a bit closer to shipping this branch</p>
TaleSpire Dev Log 2522021-01-06T01:05:49+00:00http://www.techsnuffle.com/2021/01/06/talespire-dev-log-252<p>Heya folks! Yesterday I got back from working over at Ree’s place, which was, as always, super productive.</p>
<p>Between us, we have:</p>
<ul>
<li>props merged in</li>
<li>copy/paste working correctly with combinations of props/tiles together</li>
<li>Fix parts of the UI to configure keybindings</li>
<li>More work on the internationalization integration</li>
<li>Progress on the tool hint UI and video tutorial system</li>
<li>Updated all shaders to play nice with the new batching system</li>
<li>Fixed some bugs in the math behind animations and layout of static colliders</li>
<li>Rewritten the cut-volume rendering</li>
<li>Fixed a bug that was stopping us spin up staging servers</li>
<li>Generated low-poly meshes for use in shadows, line-of-sight, and fog-of-war</li>
</ul>
<p>That last one is especially fun. Ree did a lot of excellent work in TaleWeaver hooking up Unity’s LOD mesh generator (IIRC, it was AutoLod). The goal is to allow creators to generate or provide the low poly mesh used for occlusion. This allows TaleSpire to reduce the number of polygons processed when updating line-of-sight, fog-of-war, or when rendering shadows.</p>
<p>Most of the above still need work, but they are in a good place. For example, the cut shader has a visual glitch where the cut region’s position lags behind the tile position during camera movement. However, that shouldn’t be too hard to wrangle into working.</p>
<p>Today I moved the rest of my dev setup to the new house. Lady luck decided it was going to well and put a crack in my windscreen, so I’m gonna have to get that replaced asap.</p>
<p>The next bug for me to look at is probably one resulting in lights being positioned incorrectly.</p>
<p>Anyhoo that’s the lot for tonight.</p>
<p>Seeya!</p>
TaleSpire Dev Log 2512020-12-29T01:17:34+00:00http://www.techsnuffle.com/2020/12/29/talespire-dev-log-251<p>Heya peoples! Today was my first day back working after the Christmas break, and my goal was to add prop support for copy/paste.</p>
<p>Previously I was lazy when implementing copy/paste and stored the bounding box for every tile. This data was readily available in the board’s data, so copying it out was trivial, as was writing it back on paste. However, this is no good when artists need to change an existing asset (for example, to fix a mistake). Fixing this means looking up the dimensions and rotating them on paste, but that is perfectly reasonable.</p>
<p>Looking at this issue reminded me that we have the exact same problem with boards. As we are looking at this, we should probably fix this for board serialization too. This does make saving aboard much more CPU intensive, however. The beauty of the old approach was that we could just blit large chunks of data into the data to save; now, we have to transform some data on save. We will almost certainly need to jobify the serialize code soon[1].</p>
<p>The good news in both the board and slab formats, we will be removing 12 bytes of data per tile[0]. In fact, as we have to transform data when serializing the board, why not make the positions zone-local too. That means we can change the position from a float3 to a short3 and save an additional 6 bytes per tile[2]</p>
<p>A chunk of today was spent umm’ing and ah’ing over the above details and different options. I then got stuck into updating the board serialize code. Tomorrow will be a late start as I have an engineering installing internet at my new apartment at the beginning of the day. After that, I hope to get cracking on the rest of this.</p>
<p>Back soon with more updates.</p>
<p>Ciao</p>
<p>[0] <code class="language-plaintext highlighter-rouge">sizeof(float3) => 12</code></p>
<p>[1] Or perhaps burst compile it and run it from the main thread.</p>
<p>[2] technically we only really need ceil(log((zoneSize * positionResolution), 2)) => ceil(log(16 * 100, 2)) => 11 bits for each position component, which would mean 33 bits instead of 48 for the position. However short3 is easier to work with so will be fine for now.</p>
TaleSpire Dev Log 2502020-12-24T16:47:52+00:00http://www.techsnuffle.com/2020/12/24/talespire-dev-log-250<p>Heya folks, I’ve got a beer in hand, and the kitchen is full of the smell of pinnekjøtt, so now feels like an excellent time to write up work from the days before Christmas.</p>
<p>This last week my focus has been on moving (still) and props.</p>
<p>As mentioned before, I’ve been slowly moving things to my new place, but things like the sofa and bed wouldn’t fit in our little car, so the 18th was the day to move those. I rented a van, and we hoofed all the big stuff to the new place. The old flat is still where I’m coding from as we don’t have our own internet connection at the new place yet, so I’m traveling back and forth a bunch.</p>
<p>Anyhoo you are here for news on the game, and the props stuff has been going well. Ree merged all his experiments onto the main dev branch, and I’ve been hooking it into the board format. The experiments showed that the only big change to the per-placeable[0] data is how we handle rotation. For tiles, we only needed four points of rotation, but props need 24. We tested free rotation, but it didn’t feel as good a rotating in 15-degree steps.</p>
<p>The good news is that we had been using a byte for the rotation even before, so we had plenty of room for the new approach. We use 5 bits for rotation and have the other 3 bits available for flags.</p>
<p>I also wanted to store whether the placeable was a tile or prop in the board data as we need this when batching. Looking up the asset each time seemed wasteful. We don’t need this per tile, so we added it to the layouts. We again use part of a byte field and leave the remaining bits for flags. [1]</p>
<p>There are a few cases we need to care of:</p>
<ol>
<li>Tiles and Props having different origins</li>
<li>Pasting slabs that contain placeables which the user does not have</li>
<li>Changes to the size of the tiles or props</li>
</ol>
<p>Let’s take these in order.</p>
<h3 id="1---tiles-and-props-having-different-origins">1 - Tiles and Props having different origins</h3>
<p>All tiles use the bottom left corner as their origin. Props use their center of rotation. The board representation stores the AABB for the placeable, and so, when batching, we need to transform Tiles and Props differently.</p>
<h3 id="2---pasting-slabs-that-contain-placeables-which-the-user-does-not-have">2 - Pasting slabs that contain placeables which the user does not have</h3>
<p>This is more likely to happen in the future when modding is prevalent, but we want to be able to handle this case somewhat gracefully. As the AABB depends on the kind of placeable, we need to fixup the AABBs once we have the correct asset pack. We do this in a job on the load of the board.</p>
<h3 id="3---changes-to-the-size-of-the-tiles-or-props">3 - Changes to the size of the tiles or props</h3>
<p>This is a similar problem to #2. We need to handle changes to the tiles/props and do something reasonable when loading the boards. This one is something I’m still musing over.</p>
<h3 id="progress">Progress</h3>
<p>With a couple of days of work, I started being able to place props and batch them correctly.</p>
<p>I spotted a bug in the colliders of some static placeables. I tracked down a mistake to TaleWeaver.</p>
<p>With static props looking like they were going in a great direction, I moved over to set up all doors, chests, hatches, etc., with the new Spaghet scripts. This took a while as the TaleWeaver script editor is in a shockingly buggy state right now. However, I was able to get them fixed up and back into TaleSpire. I saw that some of the items have some layout issues, so I guess I have more cases miscalculating the orientation. I’ll look at that after Christmas.</p>
<p>In all, this is going very well. It was a real joy to see how quickly all of Ree’s prop work could be hooked up to the existing system.</p>
<p>After the break, I will start by getting copy/paste to work with props, and then hopefully, we’ll just need to clear up some bugs and UI before it’s ready to be tested.</p>
<p>Hope you all have a lovely break.</p>
<p>God Jul, Merry Christmas, and peace to the lot of ya!</p>
<p>[0] A ‘placeable’ is a tile or a prop
[1] I’m going to review this later as it may be that we need to look up the placeable’s data during batching anyway and so storing this doesn’t speed up anything.</p>
TaleSpire Dev Log 2492020-12-16T18:19:53+00:00http://www.techsnuffle.com/2020/12/16/talespire-dev-log-249<p>Good-evening folks,</p>
<p>I’m happy to report that the new lighting system for tiles and props is working! We now have finally moved away from GameObjects for the board representation.</p>
<p>As it looks identical to the current lights, I’m not posting a clip, but I’m still happy that that bit is done[0]. In time, we will want to revisit this code as there is still more room for performance improvements. However, there are bigger fish to fry for now.</p>
<p>After finishing that, I squashed a simple bug in the physics API where I was normalizing a zero-length vector (woops :P).</p>
<p>Next, I’ll probably be looking at the data representation for props. I will be a bit distracted though, as I’m moving house and in the next few days, I need to get a lot done.</p>
<p>I think that’s all I have to report for today.</p>
<p>Seeya around :)</p>
<p>[0] Technically I still haven’t worked out Unity’s approach to tell if a camera is inside the volume of a spot light. This means my implementation is a little incorrect, but it’s not going to be an issue for a while.</p>
TaleSpire Dev Log 2482020-12-15T01:13:22+00:00http://www.techsnuffle.com/2020/12/15/talespire-dev-log-248<p>Today I’ve been working on the new light system.</p>
<p>The basics of this are that I am porting our <a href="https://bouncyrock.com/news/articles/talespire-dev-log-243">previous experiments allowing lights without gameobjects</a> to our new branch and hooking it into the batching code. There are, of course, lots of details to work out when trying to make something shippable.</p>
<p>As usual, we need to think a little about performance. There are often many lights on screen, and each light in Unity takes a one draw call (when not casting shadows, which these don’t). Each frame, we need to write each light into a CommandBuffer[0]. With Unity’s approach to deferred lights, A light may be rendered with one of two shaders based on where the camera is inside the light’s volume or not. Two matrices need to be provided, and the light color seems to need to be gamma corrected[1]. As CommandBuffers can only be updated on the main thread, I want to do as little of the calculation there as possible. Instead, we will calculate these values in jobs we have handling batching, and then the only work on the main thread is to read these arrays and make the calls to the CommandBuffer. This fits well as the batching jobs already calculate the lights’ positions and have access to all the data needed to do the rest.</p>
<p>If all goes well, I hope to see some lights working tomorrow.</p>
<p>Seeya then.</p>
<p>[0] we need to use a command buffer as the light mesh has to be rendered at a specific point in the pipeline, specifically <a href="https://docs.unity3d.com/ScriptReference/Rendering.CameraEvent.AfterLighting.html">CameraEvent.AfterLighting</a></p>
<p>[1] This one was kind of interesting. When setting the intensity parameter of the light, the color passed to the shader by Unity changed. In my test, the original color was white, so the uploaded values were <code class="language-plaintext highlighter-rouge">float4(1, 1, 1, 1)</code>, where the <code class="language-plaintext highlighter-rouge">w</code> component was the intensity. However, when I changed the intensity to 1.234, the value was <code class="language-plaintext highlighter-rouge">float4(1.588157, 1.588157, 1.588157, 1.234)</code>. It’s not uncommon for color values to be premultiplied before upload (see premultiplied alpha), so I just made a guess that it might be gamma correction. To do this, we raise to the power of 2.2. One (white) raised to the power of anything will be 1. So instead, we try multiplying by the intensity and raising that to the power of 2.2. Doing that gives us <code class="language-plaintext highlighter-rouge">1.58815670083..etc</code> bingo!</p>
TaleSpire Dev Log 2472020-12-13T07:25:55+00:00http://www.techsnuffle.com/2020/12/13/talespire-dev-log-247<p>‘Allo all!</p>
<p>I’m on a high right now as I just fixed a bug thats been worrying us for a while.</p>
<p>TLDR: On our new tech branch of TaleSpire, physics is now stable at low framerates</p>
<p>Now for the director’s cut!</p>
<p>We have been working hard for a while on a big rewrite of the codebase, which gives significant performance improvements. Amongst the changes was a switch to Unity’s new physics engine (DotsPhysics). For the dynamic objects in the game, we still wanted to use GameObjects, so I made a wrapper around DotsPhysics, which made this feel very similar to the old system.</p>
<p>It’s been working well enough since the last <a href="https://bouncyrock.com/news/articles/talespire-dev-log-233">batch of fixes</a>; however, we had noticed that it got very unstable at low framerates. I was nervous about this as, if it wasn’t my fault, I’d have no idea how to fix it (Spoiler: It was totally my fault :P)</p>
<p>It couldn’t be avoided for much longer, though, so for the last couple of days, I’ve been looking into it. Yesterday was exceedingly painful. I read and re-read the integration code Unity use for their ECS, and simply couldn’t find anywhere where we should have been messing up. I tried a slew of things with no results. That day ended on a low note for sure.</p>
<p>Today however, I started fresh. For the sake of the explanation, let’s assume the code looks roughly like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Update()
{
RunPhysics();
}
RunPhysics()
{
LoadDataIntoPhysicsEngine();
while(haveMoreStepsToRun)
{
RunPhysicsStep();
}
ReadDataBackOutOfPhysicsEngine();
}
</code></pre></div></div>
<p>I had noticed yesterday that the first physics step of every frame worked fine. So I knew it had to be related to how I was handling the <a href="https://gafferongames.com/post/fix_your_timestep/">fixed timestep</a>.</p>
<p>So I changed the code to look like this.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Update()
{
RunPhysics();
RunPhysics();
}
RunPhysics()
{
LoadDataIntoPhysicsEngine();
// while(haveMoreStepsToRun)
{
RunPhysicsStep();
}
ReadDataBackOutOfPhysicsEngine();
}
</code></pre></div></div>
<p>And low and behold, it was still stable. The speed was totally incorrect, of course, but it was clear that <code class="language-plaintext highlighter-rouge">RunPhysicsStep</code> was missing something that was handled in the setup or teardown code.</p>
<p>I made small change after small change and finally was able to isolate the issue. One big difference between my code and how Unity was doing things was that they read out the data from the physics engine at the end of each step and then read it back in at the start of each step. This is desirable for them, but it was not something I was doing as for our use-case, it was just overhead. However, what I had forgotten was that when we load the data into the physics engine’s data structures, there are two places you have to put the transform for the bodies being simulated. After each physics step, only one has the updated transform[0], so I needed to make sure that the transform was written back to the other.</p>
<p>And that was it! Suddenly the simulation is solid as a rock at 20fps, and I could breathe a sigh of relief.</p>
<p>I’m now going to do some performance tests again, tweak my fixed-timestep implementation, and then move on to rewriting our board representation to use our new <a href="https://bouncyrock.com/news/articles/talespire-dev-log-243">light implementation</a>.</p>
<p>Hope you have a great day,
Seeya!</p>
<p>[0] This makes sense when you look at it and is not an issue with how Unity made this.</p>
TaleSpire Dev Log 2462020-12-10T17:01:06+00:00http://www.techsnuffle.com/2020/12/10/talespire-dev-log-246<p>Hey again folks.</p>
<p>Progress has been good on the new mesher for the fog of war. The new mesh is much more regular, which should help if we use vertex animation on the mesh.</p>
<p>In the clip below, you will spot two significant issues (Ignoring the shader as we haven’t started on that yet):</p>
<ul>
<li>The seams in the fog at the edges of zones (16x16x16 world-space chunks)</li>
<li>The lighting seems weird</li>
</ul>
<iframe width="560" height="315" src="https://www.youtube.com/watch?v=TXqvB1vlpw8" frameborder="0" allowfullscreen=""></iframe>
<p>The lighting oddness is just that, for now, all the normals are set to straight up. I’m going to make them per-face after this. I could compute the normal per-vertex, but as that slightly more work, I’m holding off until we have some ideas about the visuals.</p>
<p>The seams are an artifact of marching cubes and keeping zones separate from each other. As much as possible, we want to keep zones independent from other zones, and in this case, it means we don’t know if the neighboring zones have fog or not. This, in turn, means we assume there is none so that marching-cubes generates a face. However, marching-cubes can’t do sharp corners, so you get this chamfer.</p>
<p>I’m not sure if we can fix this by modifying the geometry generated on the edges. We’ll have to see.</p>
<p>This morning I was fighting with Unity, trying to get it not to look for things to cull in cases when we are handling the culling. Without moving to their new “scriptable rendering pipelines” (SRP), there doesn’t seem to be a way to do it.</p>
<p>I will also look at the shadow culling jobs as I think the overhead from dispatching them might be larger than the time the job is taking [0]. In that case, I could Burst compile the culling methods and call them from the main thread instead. It’s a trade-off, but it might work.</p>
<p>That’s all for now. Hopefully, I’ll be back tomorrow with some new data :)</p>
<p>Peace.</p>
<p>[0] The profiler seems to suggest this is happening, but I’m not sure how much of the overhead is avoidable and how much is just part of the BatchRendererGroup’s own code.</p>
TaleSpire Dev Log 2452020-12-09T03:27:25+00:00http://www.techsnuffle.com/2020/12/09/talespire-dev-log-245<p>Hi folks. I’ve been a bit quiet recently for a couple of reasons, but I doubt they are that intriguing, so do feel free to skip this next paragraph.</p>
<p>The first reason is easy to explain, I’m moving house, which has been stealing my time. The second reason is more personal. I go through slow swings between needing to produce and needing to absorb content. Recently I switched over to the ‘absorb’ phase, but stupidly I didn’t notice, which meant I was getting very frustrated with my productivity. I don’t know why this blindsided me as it’s happened a fair few times before, but that’s for me to muse over, I guess. The upshot of this was I didn’t want to hang out much as I was not happy with what I was producing. However, this last week, I’ve been coming back out of that funk, and work has been going very well.</p>
<p>Three days ago, I finally got the new cubemap capture system working. This had a lot of false starts as I explored implementing it with CommandBuffers. We needed a custom approach for this because we handle all the culling and rendering ourselves. At first, I thought we could use a replacement shader and the that BatchRendererGroups would render to the camera, but of course, we had set those up in ShadowsOnly mode, so that didn’t work. A while back, Ree and I experimented with shadow-only shaders, which could be overridden with replacement shaders; however, the performance hit was significant.</p>
<p>All this meant I needed to write a culling compute shader to fill per-cube-face batches and then to dispatch them. It took a couple of tries to get something I was happy with. What I have should be adequate for rendering the limited regions we need for the character vision.</p>
<p>With that working, I merged my old fog-of-war (FoW) experiments and got them working with the new system. It’s not networked synced yet, but that is on the todo list soon.</p>
<p>Over the last couple of days, I’ve been rewriting the line-of-sight (LoS) system to use the cubemap capture system results. The cubemaps now only store the distance to the nearest occluder, and creatures are not occluders. This means you can now see a gnome behind a giant, for example. It also means creatures don’t block FoW reveal, which was a bit annoying in my experience.</p>
<p>The way our LoS works is to render all creatures to a cubemap using a shader that discards all the fragments. This sounds pointless, but what we do instead is to compute the direction from the camera to the fragment and use that direction to look up the distance to the nearest occluder from the cubemap made by the capture system. We can then see if the creature fragment is closer to the camera than the occluder and if so, we record that creature’s id in a buffer. We use a compute shader to collate that buffer’s contents, and we read it back to the CPU-side asynchronously. What we end up with is a buffer of visible creature ids.</p>
<p>The cool thing with the new system is that we have separated the LoS step from the view cubemap capture. This lets us recompute the LoS multiple times without having to update the view. This happens a lot when the GM or other players are moving their creatures, but yours is stationary.</p>
<p>As a 512x512 cubemap is not an insignificant amount of GPU-memory, I experimented with different schemes of when to free the cubemaps. There are a few tradeoffs:</p>
<ul>
<li>
<p>Capturing the view is relatively expensive, so you want to do it as infrequently as possible. Ideally, only when the board changes or when the creature, whose view it is, moves.</p>
</li>
<li>
<p>If a GM right clicks to check a specific creature’s LoS, it’s highly likely they will do it again. So we should keep the data around</p>
</li>
<li>
<p>However, it’s also likely that they will check the LoS of a few different creatures. So we have a risk of making too many cubemaps</p>
</li>
</ul>
<p>The approach I have now will hand out the cubemaps but, assuming they don’t become invalid for other reasons, they will expire after some time. Also, if we have more than sixteen cubemaps alive, the manager will mark them for disposal. The amount of time given is less for creatures who are not members of a party[0].</p>
<p>There are a few other details, but that was the meat of it :)</p>
<p>I am now writing a new mesher for the FoW. The Minecraft style mesher we have been using was fine, but it almost does too good a job at reducing the polycount. To experiment with the visuals, we want a finer mesh. As marching-cubes is easy to implement, I’m going to use that for now[1].</p>
<p>I should have that done tomorrow.</p>
<p>Seeya then!</p>
<p>[0] Parties are a concept that will be getting a lot of attention before the Early Access release. LoS and FoW are both party-wide, so you will see what other party members see. I’ll talk more about this as we implement it.</p>
<p>[1] Surface-nets would probably be another good option.</p>
TaleSpire Dev Log 2432020-11-16T23:36:41+00:00http://www.techsnuffle.com/2020/11/16/talespire-dev-log-243<p>Hi!</p>
<p>On Sunday, I was reading about <a href="https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.html">Unity CommandBuffers</a> as I need to rewrite the code that handles the cutaway effect. At the bottom of the page was a link to <a href="https://blogs.unity3d.com/2015/02/06/extending-unity-5-rendering-pipeline-command-buffers/">this blog post</a> where something caught my eye. They were talking about custom deferred lights. This was exciting as lights are the one place where tiles still need to use Unity GameObjects.</p>
<p>In our quest to improve TaleSpire’s performance, GameObjects have long been our nemesis. They are (relatively) slow to spawn, use managed memory, and only interactable from the main thread. To avoid lag, we have to spread light spawning over multiple frames and work pretty hard to apply updated transforms to the GameObjects from the Spaghet scripts that can modify them.</p>
<p>If we had a way to draw lights without GameObjects, we could significantly simplify a lot of code. The temptation was too great, so I’ve spent the whole day working out how to do this.</p>
<p>The good news is that it worked. Here is an ugly test.</p>
<p><img src="/assets/images/firstCustomLight.png" alt="firstCustomLight" /></p>
<p>We use a standard Unity light on the left, and on the right, the new approach. They should look the same. The reason for the pattern in the light is that I knew that we needed to support <a href="https://docs.unity3d.com/Manual/Cookies.html">light cookies</a>, and so I included that in the test. The non-cookie case should be more straightforward.</p>
<p>Ok, back to the blog. The author has included a fantastic example project, which was the first concrete proof that this should be possible. I had one extra complicating factor; however, I didn’t want our lights to look any different from the Unity ones. To me, this meant trying to drive the built-in Unity Shaders somehow. Due to my inexperience with this, I had a bunch of false starts, but I eventually understood the general flow of the rendering process. There is a good, though terse, a summary of the approach <a href="https://kayru.org/articles/deferred-stencil/">here</a>.</p>
<p>Things were looking promising but, try as I might, I could never get the built-in shaders to have the ZWrite or Stencil settings I needed. After many failed experiments, I simply copied the bits I needed into a separate shader to apply the flags myself. This process was easier said than done, and a lot of time was spent in the frame-debugger.</p>
<p>It’s all looking very promising, but there is still time for it to all go wrong :P</p>
<p>The next step is to make sure this works with multiple lights. Then I can extend this to support spot-lights and more of the standard point light settings. Once those are done, we will change how TaleWeaver packs light data, and then finally, we can refactor the light batch in TaleSpire to use this new system. I wonder if we can get some idea of performance without doing all of this… we shall see.</p>
<p>Alright, that’s all for today.
Seeya around!</p>
TaleSpire Dev Log 2402020-11-02T16:37:33+00:00http://www.techsnuffle.com/2020/11/02/talespire-dev-log-240<p>Heya folks!</p>
<p>As you may have seen, the dice URL support just shipped today.</p>
<p>This post will be in two parts. The first is about how dice URLs are formatted, and the second part will be the usual grumbling about bugs :P</p>
<h3 id="url-anatomy">Url Anatomy</h3>
<p>All TaleSpire URLs start with <code class="language-plaintext highlighter-rouge">talespire://</code>. From there on, we have some number of path segments.</p>
<p>We call the first path segment the “behavior identifier”. The behavior identifier tells TaleSpire how to interpret the rest of the segments.</p>
<p>For example, in <code class="language-plaintext highlighter-rouge">talespire://dice/d12</code>, <code class="language-plaintext highlighter-rouge">dice</code> is the behavior identifier and tells TaleSpire to interpret the rest of the path as a dice roll.</p>
<blockquote>
<p>Currently, we only support basic dice rolls with the <code class="language-plaintext highlighter-rouge">dice</code> behavior. In the future, we will support more complex dice rolls as well as other different behaviors.</p>
</blockquote>
<p>Next, let’s take a more complex URL and break it down. We’ll use this as our example: <code class="language-plaintext highlighter-rouge">talespire://dice/1d12/4d6-1d4/4d10+2/3d20+1-d6+2</code></p>
<p>First, we’ll ignore the scheme and behavior identifier, which leaves us with these path segments: <code class="language-plaintext highlighter-rouge">1d12/4d6-1d4/4d10+2/3d20+1-d6+2</code></p>
<p>Each path segment specifies a “dice group”. The roll results of each dice group get totaled separately. You can see an example result from the above roll here:</p>
<p><img src="/assets/images/diceGroupsResults.png" alt="multiple results" /></p>
<p>A dice group is parsed, right-to-left, case insensitively, with the following regex: <code class="language-plaintext highlighter-rouge">(\+|\-|)\d*D\d+(\+\d+|\-\d+|)</code></p>
<p>The C# code to do this is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>var diceGroup = Regex.Matches(pathSegment, @"(\+|\-|)(\d*)D(\d+)(\+\d+|\-\d+|)", RegexOptions.RightToLeft | RegexOptions.IgnoreCase);
</code></pre></div></div>
<p>The extra brackets in the C# regex above were added so that the <code class="language-plaintext highlighter-rouge">Groups</code> field contains the most useful data already separated. Naturally, you will need to reverse the results to get them in the correct order.</p>
<p>The regex is the most accurate specification, but here is a simpler (though maybe less precise) version with extra details.</p>
<blockquote>
<p>[optional operator][optional count]D[sides][optional modifier]</p>
</blockquote>
<p>where:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">operator</code>: currently, the only operators supported are <code class="language-plaintext highlighter-rouge">+</code> and <code class="language-plaintext highlighter-rouge">-</code>.</li>
<li><code class="language-plaintext highlighter-rouge">count</code>: is the number of dice of that kind. Zero is not valid, but if it is not specified at all, then the count is one.</li>
<li><code class="language-plaintext highlighter-rouge">D</code>: is only uppercase in the regex to clearly distinguish it from the <code class="language-plaintext highlighter-rouge">\d</code>s. The regex is always case insensitive.</li>
<li><code class="language-plaintext highlighter-rouge">sides</code>: is the number of sides of the die. Currently, we only support the standard TaleSpire dice. We will need to expand to support dice modding.</li>
<li><code class="language-plaintext highlighter-rouge">modifier</code>: here, you can specify an integer to add or subtract. It is always of the form <code class="language-plaintext highlighter-rouge">+N</code> or <code class="language-plaintext highlighter-rouge">-N</code> where <code class="language-plaintext highlighter-rouge">N</code> is a positive integer</li>
</ul>
<p>Three points of interest:</p>
<p><em>Negation</em></p>
<p>As the operator is part of the group, a valid dice URL is <code class="language-plaintext highlighter-rouge">talespire://dice/-d6</code>. This results in a negated dice roll.</p>
<p>Note that <code class="language-plaintext highlighter-rouge">talespire://dice/d4-d6</code> is supported but <code class="language-plaintext highlighter-rouge">talespire://dice/d4--d6</code> is not.</p>
<p><em>Righ-to-left</em></p>
<p>The right-to-left matching is critical to get the correct result:</p>
<p>Parsed left-to-right <code class="language-plaintext highlighter-rouge">talespire://dice/d6-2d12-1</code> matches as <code class="language-plaintext highlighter-rouge">d6-2</code> and <code class="language-plaintext highlighter-rouge">d12-1</code>, rather than what we want which is <code class="language-plaintext highlighter-rouge">d6</code> <code class="language-plaintext highlighter-rouge">-2d12-1</code></p>
<p>You can experiment with the right-to-left option by browsing here https://www.regexplanet.com/share/index.html?share=yyyyd9vu5ar and clicking the <code class="language-plaintext highlighter-rouge">.Net</code> button.</p>
<blockquote>
<p>Note: remember to example case insensitive on regexplanet before clicking the ‘Test’ button</p>
</blockquote>
<p><em>Garbage is ignored</em></p>
<p>Due to the lack of strictness in the specification, it is valid, although ugly, to have ignored text in the URL. For example: <code class="language-plaintext highlighter-rouge">talespire://dice/horsed6horse-2d12-1horse</code> is parsed to <code class="language-plaintext highlighter-rouge">d6</code> and <code class="language-plaintext highlighter-rouge">-2d12-1</code>.</p>
<p>Arguably this should be made stricter.</p>
<h3 id="normal-dev-log">Normal dev log</h3>
<p>With that out of the way, time for me to grumble. Getting this feature out has been a huge pain in the ass. To explain why I need first to describe how this is set up, don’t worry, it’s super simple.</p>
<ul>
<li>TaleSpire adds a registry key on install that tells windows what exe to run when a <code class="language-plaintext highlighter-rouge">talespire://</code> URL is launched.</li>
<li>We don’t want lots of copies of TaleSpire starting, so instead, we have <code class="language-plaintext highlighter-rouge">TaleSpireUrlRelay.exe</code>, which either
<ul>
<li>launches TaleSpire and passes the URL as a command-line argument</li>
<li>use some form of <a href="https://en.wikipedia.org/wiki/Inter-process_communication">IPC</a> to send the URL to the running instance of TaleSpire</li>
</ul>
</li>
</ul>
<p>One of the testers had an issue where the URLs would not arrive unless the TaleSpireUrlRelay was set to run as administrator. This was clearly a permissions issue, and so I started exploring the security options of the IPC I was using. This is where things get screwy.</p>
<p>But first, let’s wind back and look into how we got here.</p>
<blockquote>
<p>Note: this account is intended as entertainment. I likely missed very obvious information along the way and misinterpreted errors that would have told me the real issue. I can’t accurately say which systems actually had bugs, but I can tell the little tale of how my week has <em>felt</em>. Enjoy!</p>
</blockquote>
<p>So back in the day you might use <a href="https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendmessage">SendMessage</a> (or PostMessage maybe) with <a href="https://docs.microsoft.com/en-us/windows/win32/dataxchg/using-data-copy">WM_COPYDATA</a>. However, since Vista, this <code class="language-plaintext highlighter-rouge">WM_COPYDATA</code> is blocked for security reasons. However, <a href="https://www.codeproject.com/Tips/1017834/How-to-Send-Data-from-One-Process-to-Another-in-Cs">there seem to be workarounds</a> but this looks nasty, and it would be great to avoid this if possible.</p>
<p>Also, we want to port to Linux and Mac in the future, so something in the .Net framework would be much better. You could use a socket, but that’s very low level for what should be a straightforward thing.</p>
<p>How about websockets? We already have a client in TaleSpire, and we have stated we would like to expose a websocket API in the future. However, that has almost the opposite problem. It’s a huge piece of extra ‘machinery’ running in the game, and we don’t know how performant it would be. It would be much nicer to have something lightweight and then launch the websocket server only when needed. So let’s try and avoid the websocket approach for now.</p>
<p>Next, you land on <a href="https://docs.microsoft.com/en-us/dotnet/standard/io/how-to-use-named-pipes-for-network-interprocess-communication">Named Pipes</a>. They look ideal, and in fact, you get them working fine for you. However, this is when that tester I mentioned reported the permissions issue. Dang.</p>
<p>You have a read of the documentation and see the <a href="https://docs.microsoft.com/en-us/dotnet/api/system.io.pipes.pipesecurity?view=dotnet-plat-ext-3.1">PipeSecurity</a> argument to the server constructor. “Yay,” you say, “this should be easy.” You take the example code from the docs, but it uses <code class="language-plaintext highlighter-rouge">TokenImpersonationLevel.Impersonation</code>, which seems to error in your setup. Ugh, maybe it’s best to read some more rather than copying more code.</p>
<p>A quick google, and you start hitting some <a href="https://stackoverflow.com/questions/50711518/attempted-to-perform-an-unauthorized-operation-when-calling-namedpipeserverstr/54896975#54896975">scary</a> <a href="">posts</a> which seem to suggest that it just won’t work… and you miss the fact that they seem to be about <a href="https://docs.microsoft.com/en-us/dotnet/standard/net-standard">.NET standard</a> so maybe it’s not relevant.</p>
<p>However, you have a lot of experience in cases where Mono doesn’t implement something normally available in .Net, so it all seems feasible.</p>
<p>A bunch more testing later, and the only pattern you have found is that the Named Pipes just don’t work if you try and specify the PipeSecurity. It doesn’t help that every tutorial you find has a different way of setting things up. Is the SID meant to be <code class="language-plaintext highlighter-rouge">WellKnownSidType.AuthenticatedUserSid</code>? or <code class="language-plaintext highlighter-rouge">WellKnownSidType.BuiltinUsersSid</code>? or <code class="language-plaintext highlighter-rouge">WellKnownSidType.WorldSid</code>?, or <code class="language-plaintext highlighter-rouge">"Everyone"</code>, etc, etc.</p>
<p>Ok, so now we are worried that Named Pipes won’t work, what other options do we have? Hmm, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.runtime.remoting.channels.ipc.ipcchannel?view=netframework-4.7.2">IpcChannel</a> looks promising. But, surprise surprise, when you copy over the code, it hangs. Now you have two implementations you don’t understand. Do you really want to keep down this road?</p>
<p>Maybe you do for a couple more hours, but “damn,” you think, “I just need something simple.” Perhaps you could dump the URL into a file and have a file-watcher in TaleSpire pick it up. Ugly as hell, but at least it could work.</p>
<p>You keep that in mind and look at the next option in the list, MSMQ. Oh wait, according to a blog, it’s <a href="https://particular.net/blog/msmq-is-dead">“dead”</a>… well, not actually dead, but you don’t need more confusion in your life, so after a quick read, you try to find something else.</p>
<p>Maybe <a href="https://en.wikipedia.org/wiki/.NET_Remoting">.Net Remoting</a>? It’s old as hell, but at least that means it isn’t changing all the time. You dive into some examples, and it’s built around pretending that the same object exists in both programs simultaneously. This model is gross, but hey, if we can get an example working, then maybe it’s… nah, we are running into issues connecting too.</p>
<p>There is this horrible balance you are trying to strike. You can try and implement a system using the tech you don’t understand, knowing that everyone gets stuff wrong the first few times, OR pull in some code someone else wrote and then have to learn that. Either way, this feature is taking way too long, and you desperately want to get it working, so you don’t want to put days into exploring each of these options.</p>
<p>Terrifyingly the <code class="language-plaintext highlighter-rouge">SendMessage</code> stuff is seeming like something you might have to reconsider. However, after checking out some example code and some <a href="https://github.com/wgraham17/simple-ipc">fairly</a> <a href="https://github.com/maintainnothing/NLocalIpc">promising</a> <a href="https://github.com/cgillum/pipe-server">libraries</a> you say “screw it” and go back to Named Pipes as that is the closest you’ve gotten so far.</p>
<p>BUT WAIT. There is something called AnonymousPipes, specially made for only local connections, with a simple API and. NOPE.
It only works if the server process launched the client process, which is not the case for us. Ah well, back to Named Pipes</p>
<p>Now you are back where you started, poking at seemingly random things to find out what will happen. While flailing around, you decide to give the connecting client <a href="https://docs.microsoft.com/en-us/dotnet/api/system.io.pipes.pipeaccessrights?view=netframework-4.8">FullControl</a> of the pipe. You didn’t try this before because you never want to grant anything higher permissions that it needs but suddenly, IT CONNECTS! You franticly whittle down the permissions until it’s the smallest set that allows the connection to work and then make a new build and give it to the tester who had the issues.</p>
<p>Same problem.</p>
<p>Just as you are about to pour a giant glass of the most potent drink you own, you casually ask them if there is any way they know that the game could be running as administrator.</p>
<p>“Oh…” they say.</p>
<p>It turns out that when they were helping test things for you half a year ago, they tried setting the exe to run as admin, and Steam, being a good little soul, maintained this flag through every update until now.</p>
<p>They remove the “Run as administrator” flag, and immediately the dice-URLs start working.</p>
<p>Now. This is not a story about where the tester screwed up; these things happen. This is a story of the sea of nonsense you sail through trying to work stuff out when almost no information you find online is truly reliable, and you don’t have time to learn it all from first principles.</p>
<p>In conclusion, this is why the URLs are three days late :D</p>
<p>That’s enough stupidity for one night. Seeya folks!</p>
TaleSpire Dev Log 2392020-10-28T01:45:16+00:00http://www.techsnuffle.com/2020/10/28/talespire-dev-log-239<p>Heya everyone,</p>
<p>I got back from my break on Sunday, fully rested, and ready to build again!</p>
<p>For the last two days, I’ve been getting back up to speed by looking at the custom URL scheme code again. <a href="https://bouncyrock.com/news/articles/talespire-dev-log-234">The last time I was working on this</a>, I forgot that the custom URLs always try to open a new copy of the application they are linked too. That’s fine when TaleSpire is closed but no good if you already have it open.</p>
<p>To handle this, I wrote a little program that acts as a middleman for the custom URLs. It is called when one is used and then looks to see if TaleSpire is running. If it is, it connects to it using a <a href="https://docs.microsoft.com/en-us/dotnet/standard/io/how-to-use-named-pipes-for-network-interprocess-communication">named pipe</a> and passes along the URL. If TaleSpire is not running, it launches it passing the URL as a command-line argument, and that argument is processed once the user has logged in.</p>
<p>In the future, we still think that we will expose an API for tools to hook into TaleSpire via websockets. However, that is a bigger task than I wanted to take on today, and I wasn’t sure of the performance implications. By default, I think that the websocket server would be disabled, and we’d use one of our <code class="language-plaintext highlighter-rouge">talespire://</code> URLs to enable it and to fetch the connection details (like the port number).</p>
<p>As a test of this new code, I knocked together a simple (and rather unfinished) extension to TS that uses the custom URLs to spawn dice. It supports multiple totals within one roll and modifiers on every dice descriptor. For example, <code class="language-plaintext highlighter-rouge">talespire://dice/2D12+2+1D20</code> spawns 2D12 with a +2 modifier and 1D20 and will sum them. <code class="language-plaintext highlighter-rouge">talespire://dice/2D12+2/1D20</code> spawns the same dice, but the totals for 2D12 with the +2 modifier are calculated separately from that of the 1D20.</p>
<p><img src="/assets/videos/customUrlDice.gif" alt="custom url spawning dice" /></p>
<p>As you can see from the gif, there are still lots of things to fix. But it’s coming together. Once it’s working and we’ve play-tested (and we are sure we want to ship it), we’ll publish the specification for the dice URLs.</p>
<p>That’s all from me for today.
Seeya!</p>
TaleSpire Dev Log 2362020-10-12T12:13:51+00:00http://www.techsnuffle.com/2020/10/12/talespire-dev-log-236<p>Today in “I’ve not been working on line-of-sight but…”, we’ll be talking about things that ended up higher priority than starting on line-of-sight :P</p>
<p>On Friday, I was chatting to Ree about the next tasks on the list, and he suggested that it could be helpful to convert the boards from the beta format sooner rather than waiting for the per-zone-sync. At first, I balked at this, but after a sleep, I realized that much of the work would be valid even when we come to rewrite the new persistence system.</p>
<p>And so that’s where I’ve been. Format conversion is always sensitive, but Ι took the opportunity to simplify how we handle it. Now, each time we need to make a new version, we’ll:</p>
<ul>
<li>copy all the current types into a new numbered ‘legacy’ namespace</li>
<li>strip out everything beyond the deserializing code</li>
<li>write a converter from that version to the latest version.</li>
</ul>
<p>It’s dumb and involves more leg work than fancier solutions, but it works. It’s reassuring to know that you aren’t accidentally messing with something needed by an older version of the board you still need to support when making a change. We’ll see how this feels to work with, but I’ve tried a couple of other approaches before and have found them lacking. Let’s see how Ι do with this one.</p>
<p>After getting conversion working, I realized the codebase was in a good place for some cleanup. I had a bunch of naming (of variables and types) that needed updating, and there is not likely to be another good time to do it before the Early Access. So Ι threw a few hours at this.</p>
<p>I noticed a case where, on photon failing to connect, we wouldn’t retry, essentially leaving you stuck with no board. I’ve put a simple retry in on my branch and will cherrypick this into master tonight. This will be available in the next release.</p>
<p>During testing of board format conversions, I found a case where the physics wrapper would incorrectly schedule some jobs. This is now also fixed.</p>
<p>That’s the lot for now. Back soon with more</p>
TaleSpire Dev Log 2352020-10-10T02:50:09+00:00http://www.techsnuffle.com/2020/10/10/talespire-dev-log-235<p>Hi again,</p>
<p>Today I started out thinking I’d be working on line-of-sight, but instead, I dug into some issues we’ve had with builds since the Unity version upgrade.</p>
<p>First, one was that the dummy-asset resource wasn’t loading reliably on game start. This is very odd, and it feels like it’s a Unity bug. As we are currently using the 2020.2 beta, this is entirely possible. For now, I’ve worked around it by simply exposing a SerializableField for the prefab, which, to be honest, is probably a better way to do it anyway.</p>
<p>Next, I noticed that even though we had supposedly moved away from our old JSON asset format, I hadn’t converted the music asset files. This work began on the TaleWeaver side, and it was straightforward to get the music info into the new binary asset format. As I was upgrading the TaleSpire side to use this new data, I was able to do some cleanup, which totally separated the old asset-data classes from TaleSpire and so now TaleWeaver can own those. This is nice as both sides can evolve their formats in response to their own needs.</p>
<p>I then went on the hunt for an error that occurred when deserializing hide volumes. The TLDR was that, due to having moved to C# 8, a different method overload was being selected for a specific call. It took me a little while to spot, but it was a very easy fix. The C# change that caused this is actually the one I’ve been most excited for over the last year or two, <a href="https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#unmanaged-constructed-types">unmanaged-constructed-types</a>.</p>
<p>The code cleanup was really enjoyable, and, given that it was nearly the end of the day, I decided to do some more cleanup that had been waiting for the engine to get to this point. In this case, it was on the manager that handles asset loading and the manager that handles tiles and props. They were a bit too bound together, and their respected roles weren’t always clear enough. The changes themselves aren’t worth enumerating, but I’m happy with the result.</p>
<p>Tomorrow I might look at line-of-sight, or I might look at laying the groundwork for the new board format and the code to handle upgrading the current boards.</p>
<p>Seeya!</p>
TaleSpire Dev Log 2342020-10-09T02:30:26+00:00http://www.techsnuffle.com/2020/10/09/talespire-dev-log-234<p>Hey folks,</p>
<p>Once again, today was mostly spent on physics. Having solved the dice ‘popcorning,’ we talked about yesterday, I was left with an issue that dice would slide around as if they were on ice. It looked like this:</p>
<p><img src="/assets/videos/iceDice.gif" alt="ice dice" /></p>
<p>I had assumed this was material related and, sure enough, I found several places where I had not been setting the materials properly when generating colliders. However, even with those fixed, the problem still remained.</p>
<p>This led me into a whole bunch of tests and tweaks which weren’t getting me very far. However, after some of the previously mentioned material fixes, I saw that things worked correctly on one of my test scenes. That scene was only using GameObjects with driven by BouncePhysics wrapped around Unity.Physics. The sliding must have been related to tile colliders, which are managed by the board and written into the physics engine separately. However, I had already set a material for all these colliders.</p>
<p>At some point, I finally noticed that the material I was using for the tiles (<code class="language-plaintext highlighter-rouge">Unity.Physics.Material.Default</code>) had a FrictionCombinePolicy than in the material I was using for other objects. For BouncePhysics, I was using a material converted from the default material found in Unity’s classic physics engine. The result of that conversion used ‘arithmetic mean’ for friction combination, whereas the <code class="language-plaintext highlighter-rouge">Unity.Physics.Material.Default</code> used geometric mean. I switched the tiles to use the converted classic material, and the sliding stopped.</p>
<p>This is good, of course, but what a silly mistake. I had just assumed the two would play well together.</p>
<p>Anyhow for completeness, here are the dice. They won’t behave like this when they ship. This is just to show the difference from the gif above.</p>
<p><img src="/assets/videos/iceDiceFixed.gif" alt="ice dice" /></p>
<p>With that done, I knew the next task was to rewrite line-of-sight, but that seemed like a pretty overwhelming way to end the day, and so I decided to play with something much more straightforward.</p>
<p>I’ve really wanted to experiment with adding a custom URL scheme for TaleSpire. This would mean opening a link that looked like <code class="language-plaintext highlighter-rouge">talespire://<something></code> would open TaleSpire and do something useful. My first thought was that you could get a URL for a specific point in a particular board in your campaign. You could then share that with your party or even use it in external tools (World Anvil comes to mind), so you could link points in your documents directly to the TaleSpire maps that represent them. Also, if we support links out of TaleSpire, we could get a very basic but rather compelling way to move in both directions, which isn’t tied to any specific tool or system.</p>
<p>To add a custom URL scheme to Windows, I needed to write some registry keys. Steam <a href="https://partner.steamgames.com/doc/sdk/installscripts">seems to support this</a>, but when I tested the registry options of the install script, I saw only partially made registry subkey trees. This was disappointing, but they also support executing other processes as part of setup. I assumed they would need to run as administrator, so I made a little c# script to add the required registry entries. This worked great. Here it is in action:</p>
<p><img src="/assets/videos/customUrl.gif" alt="custom url scheme" /></p>
<blockquote>
<p>Note: I have chopped out a bit of the middle of this video as loading takes much longer in a developer build, and it’s boring to watch</p>
</blockquote>
<p>I’m not going to do any more work on this for now, as there are many more important things to do, but this was a nice change of pace for a few hours.</p>
<p>That’s all for now.</p>
<p>Peace.</p>
TaleSpire Dev Log 2332020-10-08T02:23:32+00:00http://www.techsnuffle.com/2020/10/08/talespire-dev-log-233<p><img src="/assets/videos/popcornDice.gif" alt="popcord dice bug" /></p>
<p>Internally we have been calling this bug “popcorn dice”, and it’s been driving me crazy for a while.</p>
<p>As we have our own wrapper around Unity’s new physics engine, I had assumed that I was just ‘driving’ it incorrectly. So today, I sat down and read how Unity’s ECS hooks into the physics. I had hoped the mistake was how I was loading in the physics-body data, or when reading it out after the physics step. Alas, no such luck.</p>
<p>I saw that they kept track of their fixed steps in a slightly different way than I did. It should have resulted in identical behavior, but just to be sure, I followed their example. Again no dice (har har).</p>
<p>I read that damn code forwards and backward and finally remembered that I had a test scene that had been surprisingly stable this whole time. It’s the one I showed in an update a few weeks back. It looked like this:</p>
<p><img src="/assets/videos/physicsGoofyTest2.gif" alt="popcord dice bug" /></p>
<p>So what could this possibly have in common with the ECS tests I’d been doing, but that was also different from the main board scene. I suddenly got a hunch and couldn’t help but laugh when it proved to be correct.</p>
<p>The floor was too big.</p>
<p>YUP. So we’d been lazy and just made the board 6000x6000 units in size a looooong time ago and promptly forgot about it. The classic physics engine (bless its soul) had been doing a great job, and so we’d never seen the issue. However, with the new system, if you have mesh colliders interacting with a single box collider more than ~2000 units across, it gets very unstable and starts popcorning like we see above.</p>
<p>This find is a huge relief. TaleSpire obviously relies on the dice behavior being reliable and, if it were a problem with the engine itself, I’d have no idea of how to fix it. This should be trivial. We can make a smaller collider and just teleport it around where we need it.</p>
<p>Alright, that’s a great place to end the day.</p>
<p>I’ll seeya all tomorrow with something else!</p>
TaleSpire Dev Log 2322020-10-06T10:09:36+00:00http://www.techsnuffle.com/2020/10/06/talespire-dev-log-232<p>Updates from the land of code.</p>
<p>The batching/rendering code has reached where we need it to for the Early Access. We see a three times speedup on some of our stress tests, and while there are very clear ways to get more, this endeavor has already left us two months behind schedule, so we need to press on into other features. That said, it is lovely to see all this work pay off, and I’m very excited by what we can do to get the rest of the perf that is still on the table.</p>
<p>During this, I hit one very annoying bug. It seems that Unity does not handle writing into IndirectArguments buffers properly on all GPUs[0]. The GPU-frustum-culling system I wrote recently worked great on my home PC but failed silently and strangely on my laptop. I ended up using a more naive approach and having the culling compute shaders pushing values into an append-buffer and using the CopyCount method to transfer the final length to the IndirectArguments buffer afterward. This still kept things on the GPU but significantly increased the number of compute shader dispatches and buffers required to work. The good news is that it does open up some opportunities for removing some indirection in the vertex shaders, which should give some performance wins later. I <em>think</em> I should also be able to make a system on top of this that can reduce the number of final draw calls, but we’ll see how that goes.</p>
<p>Ree and I did some digging into the physics instabilities we have been seeing. Along with some simple mistakes, we remembered that the Unity.Physics API allows you to swap out the engine used behind the scenes, and that Havok has such a package. We had heard rumors that it was just plug-and-play, but when is programming ever that simple?</p>
<p>It was that simple. To be honest, I was floored. I changed maybe ten lines of code, and TaleSpire was using Havok. We saw that, while the stability was a little better, we were getting uncannily similar issues to using Unity’s DotsPhysics. This made us confident that the problem was how my code is driving the physics rather than the instability coming from the engine.</p>
<p>It should be noted that DotsPhysics was significantly faster than Havok in the stress test we threw at it. TaleSpire doesn’t have many dynamic objects, but we have hundreds of thousands of small static objects. DotsPhysics seems to be heading a very compelling direction, and I think it will carve out a very successful niche for itself.</p>
<p>We then upgraded to the latest version of Unity, which, in turn, allowed us to move to the latest builds of various packages, including physics. The improvements in DotsPhysics was again, notable. It seems that our strange blend of using old and new systems in Unity has paid off well as we haven’t yet run into any painful bugs on this new build. Unless anything major happens to make us have to downgrade, 2020.2 will be the version of Unity we will ship TaleSpire using.</p>
<p>We also merged master into the branch with all the new tech. This includes the props system. To let prop development be done in parallel, Ree has been serializing the props separately from the tile data. Now that props are shaping up well, we can see what data is needed and so we are in a good place to bring props into the same system as tiles. This will let sync, undo/redo, and all the other normal stuff work with props too.</p>
<p>A short time back, discussions with the community on how line-of-sight should work led me to re-think how that system should be implemented. Yesterday, Ree helped me prototype one possible version of this. It relied on us not using the BatchRendererGroup in ShadowsOnly mode but instead discarding fragments in passes where we didn’t want to write to depth. It worked well in small scenes, but we took a significant FPS hit in our stress-test. I’m happy that we found this out very quickly and can now start the version that is a little more work but will not put additional costs on unrelated systems. That version will require us to dispatch the DrawMeshInstanced calls to render the scene to individual faces of the cubemap. We will use jobs to handle the selection & per-face culling of the tiles for these draw-calls.</p>
<p>Aside from this, I’ve continued work on the character controller. It’s not there yet, but we are relatively confident we can fix up the remaining issues, so it’s currently a lower priority.</p>
<p>That wraps up everything for last Wednesday to now. My next immediate task is to hunt down what I’m doing wrong in Physics as currently, it’s the biggest thing that stops us from play-testing this branch.</p>
<p>Hope you folks are doing good.
Peace.</p>
<p>[0] It could be that the IndirectArguments issue was a graphics API limitation rather than a Unity one. I’d need to do more reading to find out.</p>
TaleSpire Dev Log 2312020-09-30T02:55:33+00:00http://www.techsnuffle.com/2020/09/30/talespire-dev-log-231<p>Hey folks, I was meant to write this in the morning, but code is distracting :P</p>
<p>Yesterday I worked on dice physics. I had been concerned by the behavior of dice and wanted to find a good path forward.</p>
<p>To be able to see what was going on, I made this test scene. I can reset the dice by clicking the reset ‘button.’</p>
<p><img src="/assets/videos/physicsGoofyTest.gif" alt="goofy test scene" /></p>
<p>This is one place where Unity absolutely excels. Being able to knock up tests and tooling in minutes so often can help you make headway on otherwise stubborn issues.</p>
<p>There are a few points that I find interesting:</p>
<p>DotsPhysics and ClassicPhysics do not line up in behavior. This is really important as this means that I shouldn’t expect my conversion routines to give the same results as classic as mine are directly based on the ones built into DotsPhysics.</p>
<p>The BouncePhysics wrapper around DotsPhysics lines up with the behavior of DotsPhysics. In that, the way they bounce and come to rest is similar. This lends support to the point above and gives some evidence that the conversion is ok.</p>
<p>DotsPhysics falls faster. This is because they don’t handle time properly, so faster framerate means faster simulation.</p>
<p>Mine falls at the same speed as ClassicPhysics. This is very important. It suggests my fixed-timestep code and use of Unity’s simulations values (like gravity) are in the right ballpark.</p>
<p>Once I realized that DotsPhysics doesn’t give the same bounce, drag, etc, for the same input values, I felt comfortable playing with values to find something that felt right. This clip is not final behavior but shows how a couple of minutes of tweaking can help.</p>
<p><img src="/assets/videos/physicsGoofyTest2.gif" alt="a little better" /></p>
<p>Afterward, I set up some ramps and D12s, so we had a decent place to start testing when we sit down to dial this in for real.</p>
<p>I also saw that my interpolation code was incorrect. I found some issues, but I was struggling to match all of Unity’s behavior. I then noticed that we don’t use interpolation in the current build and so this stopped being a priority. However, it still took me a couple of hours to quit playing with it. It’s so tempting to keep going when you know you have the math right, and it’s the surrounding plumbing that is the issue.</p>
<p>Anyhoo that was that. Today I’ve been working on the character controller again, results are promising, but I’ll leave that until tomorrow.</p>
<p>Goodnight</p>
TaleSpire Dev Log 2302020-09-28T09:41:09+00:00http://www.techsnuffle.com/2020/09/28/talespire-dev-log-230<p>With lights under control, I’ve turned my evil eye back to physics. Since I replaced the physics engine, gm-blocks and hide-volumes have not been working. This was expected, but it was time to fix them.</p>
<p>For sanity, I’m going to call Unity’s old physics engine ClassicPhysics, their new one DotsPhysics, and our wrapper BouncePhysics.</p>
<p>Hide volumes make use of the fact that, in ClassicPhysics, we could resize colliders on the fly. We needed something similar in our new system. First up was a different problem, however. When we convert Unity’s colliders to the form required by DotsPhysics, we remove the components. This means there isn’t an obvious place for the programmer to set the values. This meant BouncePhysics needed its own version of these components.</p>
<p>I knocked together a Sphere, Cylinder, Capsule, and Box collider components. I modified the BouncePhysicsBody to walk the hierarchy, converting the ClassicPhysics components to their BouncePhysics equivalent, and registering all these new colliders.</p>
<p>During this, I noticed some issues in how my code handled baking of scale when the collider is subject to multiple transformations. This sucked for a good few hours, but I was able to find code that showed how to do it properly, and I got it fixed.</p>
<p><img src="/assets/videos/bounceColliders.gif" alt="custom component adjustment" /></p>
<p>With that done, I was able to hook up the hide volumes and get them working again \o/</p>
<p>Next, gm-blocks. There I found an interesting bug when deselecting them. First, the gm-block would be destroyed, and as that happened, it would unregister itself from BouncePhysics. The issue arose because, in DotsPhysics, you reset and pass all the RigidBodys every frame. This means that although I had unregistered the object from BouncePhysics, the RigidBody was still in the DotsPhysics collision world. I can’t just remove that one body (the only option is Reset, which clears everything). Instead, I now set a flag inside the RigidBody to state that it is unregistered and then modified the collision queries to check for this flag when collecting results. This all takes place after the updates to motion have been calculated, so there is no risk of something bouncing off this unregistered body before the next frame, and by then, it will have been cleaned up.</p>
<p>Of course, this check (which is essentially a comparison of a ulong) is not free. I will probably have the system only use those new collectors on frames where something has been unregistered. Luckily in TaleSpire, we don’t have much churn in the physics objects.</p>
<p>During this, it made sense to clean up some of the physics code in general. Until now, I had made the class that holds the board data responsible for the physics ‘world’. This wasn’t ideal, and the physics related code was very out of place. I decided to centralize it in the BouncePhysics classes. This does mean that, for now, we only allow only one board to be using physics at a time (which could matter when changing boards), but the trade-off was that I could make the API much more like ClassicPhysics. The API design is important because, as soon as I merge this branch, Ree needs to be able to get up to speed quickly. The more familiar it is, the better.</p>
<p>In amongst this, there was the usual smattering of bugs, confusion, and other things that slow one down. But overall, it’s felt good to have some forward momentum.</p>
<p>Next, I really need to find out why dice in BouncePhysics are not behaving like they did in ClassicPhysics. It could be an issue in the conversion routines or, more likely, mistakes in the code than manages fixed-timestep and interpolation. I’ll set up some test scenes where I can compare everything side-by-side, and we’ll see what we can find out.</p>
<p>Alright, that’s all from me for now.
Seeya later</p>
TaleSpire Dev Log 2282020-09-25T17:00:21+00:00http://www.techsnuffle.com/2020/09/25/talespire-dev-log-228<p>Since our move away from GameObjects, we have had to make our own batching, culling, and animation systems. One that I had left for a while was lighting, so it was time to tackle that.</p>
<p>First off, the deeply annoying news. Unlike with the BatchRendererGroup, which gave us a way to avoid GameObjects and populate the data from multiple threads, there is no similar thing for lights. In fact, when I looked into Unity’s ECS, they were just spawning GameObjects for the lights and updating their transform each frame to be in line with the entities that supposedly contained them.</p>
<p>This is a pain, but it’s the only option, so we needed something similar.</p>
<p>As always, the fact that TaleSpire is a user-generated-content game means we have a bunch of complexity. We don’t know in advance how many lights are about to be spawned. Worse, because we are back using GameObjects, we know that spawning too many per frame will cause serious fps issues. This means we need a progressive spawning approach and pools for reusing light that that been destroyed.</p>
<p>Also, spaghet scripts are allowed to update light color, intensity, position, and rotation on a per-frame basis. Now position and rotation updates can be jobified if we put the light’s Transform in a TransformAccessArray. However, color and intensity can only be set from the main thread… joy >:[</p>
<p>On each change to a zone, we rebuild all the batches, including laying out the lights again. We have jobs to:</p>
<ul>
<li>find out what tiles and pros are using lights</li>
<li>collate the numbers of each kind of light</li>
<li>write the details for each instance of each kind of light (in parallel) into various collections.</li>
</ul>
<p>We push all of the current lights back into the pools for reuse and, over the course of multiple frames, spawn new lights or pull them from the pools.</p>
<p>Dynamic lights are updated by Spaghet scripts. If there is a change to the intensity or color, we push the index of the modified light into a NativeQueue. That queue is consumed on the main thread where we can apply the changes.</p>
<p>Getting all of this to play nice has taken me the last 3-4 days. There is still plenty of room for improving performance, but I need to profile using real-world scenes before making any more changes. Luckily we have fantastic community sites full of slabs I can use :D</p>
<p>Alright, the next stop is looking at little things that broke on the move to the new physics engine.</p>
<p>Seeya in the next log (which will hopefully take much less than a week this time :P)</p>
TaleSpire Dev Log 2272020-09-25T16:05:02+00:00http://www.techsnuffle.com/2020/09/25/talespire-dev-log-227<p>This one is interesting to write as it’s fairly downbeat. However, it is part and parcel of making stuff, so let’s do it.</p>
<p>As mentioned in previous posts, we have known for a while that TaleSpire would not scale to bigger boards using their GameObject abstraction. There was a lot of song and dance from Unity about their upcoming ECS, and while it looked pretty promising, we miscalculated the timeframe it would take Unity to bring it to life. That meant that this year, as we tested again, we saw we needed a custom solution. That’s what I’ve been working on.</p>
<p>Now I have tried hard to always be clear about when the issues we have run into have been Unity’s fault and when they have been ours. In general, the responsibilities have been ours as we are using packages marked experimental and not in general recommended for shipping. However, today’s principal issue is, I believe, not on us.</p>
<p>To move away from using GameObjects we have been building on the BatchRendererGroup. This object lets you specify what needs to be drawn and then provide the matrices to lay these instances out in world-space. The matrix array you fill is a NativeArray and, as such, is fully compatible with the job system[0]. This meant that we have been able to parallelize the population of these arrays, which has resulted in the massive improvements in the performance spawning tiles we have seen in previous dev-logs.</p>
<p>When you are using instanced rendering, you need to somehow get the per-instance data to your shaders. Traditionally in Unity, this is done using MaterialPropertyBlocks. You can associate a MaterialPropertyBlock with an object and, when it renders, that data is made available to your shader. When using BatchRendererGroup, this requirement seems to be filled by using GetBatchVectorArray (and co).</p>
<p>Now here is my portion of the fuckup. When I couldn’t get this to work in my early testing, I put it down to me not being too hot with Unity’s shaders, and that I would work it out later. Unlike the new stuff in Unity, the BatchRendererGroup is not marked as experimental as so I had the same confidence in the system that I do for other released parts of Unity. I put a few questions up on the forums and got back to work[1]. I rigged up a simple system where I looked up the data from ComputeBuffers using the instance-id as the index; this works fine without culling enabled[2].</p>
<p>The problem is that GetBatchVectorArray does not (as far as I can tell) work with the classic rendering system in Unity. This is not stated in the documentation, but in the docs for an experimental package called the ‘Hybrid Renderer,’ it mentions some things that can be construed to mean it only works with the new scriptable rendering pipelines.</p>
<p>UPDATE: While writing this log, I’ve seen that my bug report against this method has been accepted. I will keep you posted on how this goes.</p>
<p>This sucks. Worse, we know the old version won’t work; we are months into the rewrite with no way to go back. Frankly, it was rather distressing.</p>
<p>Now those of you who know a little about Unity might be wondering if we could just make something custom using DrawMeshInstanced or DrawMeshInstancedIndirect. The problem is that things drawn with those methods are not culled, and this is a problem as realtime lights/shadows require rendering the scene from multiple vantage points. Without culling, you end up rendering everything for each camera regardless of the direction it’s facing. This massively hurts performance. The BatchRendererGroup was excellent in this regard as it allowed us to implement the culling jobs ourselves.</p>
<p>One thing that the BatchRendererGroup does have is a flag that lets you specify that you only want to use it to render shadows. Another little detail is that TaleSpire’s shadows don’t need to respond to the tile drop-in/out animation we implement in the vertex shader. This gives us the possibility to keep using the BatchRendererGroup for shadows and keep the culling advantages.</p>
<p>However, that still leaves actually drawing the scene. The only thing I could think of was that I’d need to implement this myself. I’d use all the batching code we have written to layout data on the GPU, I’d write a GPU-frustum culling implementation that matched the one we use on the CPU side, and use DrawMeshInstancedIndirect to dispatch the draw calls.</p>
<p>Now I’ve never written this kind of thing before, but it felt like the only feasible option. Over a few days, I got this written and was finally able to cull our scenes again without horrible flickering.</p>
<p>Now the scary thing is that this was not part of the plan. It’s cost us a lot of time and how it scales is currently unknown. What I do know is now we have a whole pile of extra complexity to manage and improve. Not great.</p>
<p>However, there are some up-sides:</p>
<p>Currently, we have to use the same meshes for rendering, occlusion checks, and shadow rendering. I wanted to use different meshes for shadows and occlusion checks as this allows us to use lower-poly meshes (helping performance) and close tiny cracks the fog of war shouldn’t be able to see through. I was implementing that as part of the GPU-occlusion system. When I delayed that feature to after the Early Access release, we delayed the ‘separate shadow mesh’ feature too. With our new system, we can trivially use different meshes, so these improvements are now on the cards for Early Access.</p>
<p>As mentioned recently, most of Unity’s API has a max of 500 instances per batch when using non-uniform scaling. DrawMeshInstancedIndirect does not as all the data is already on the GPU. This means we have the potential to have much larger batch sizes. Currently, we batch on a per-zone basis, so we are a little limited, but it will not be hard to take zones that have not changed in a while and combine their batches. It’s something I’ll look into when I get back to improving performance.</p>
<p>Thirdly, we now have a bunch of tile/prop data on the GPU that we can easily write compute-shaders to operate on. This gives us more options for future developments.</p>
<p>So all in all, this has been pretty horrible. I’m very grateful that the community is so supportive and that the Eldritch Foundry update was so well received. That was a serious boost in a tough couple of weeks.</p>
<p>However, we are still very far from done. By my estimates, I was at least a month behind my personal goals before this escapade, so I’m not sure where we are now. Luckily we’ve been vague with deadlines, but we will need to sit down and look at the plan again.</p>
<p>What a year!</p>
<p>This dev log is long, so I’m gonna leave talking about lighting to the next post.</p>
<p>Seeya there</p>
<p>[0] This is important as, traditionally, most of Unity’ apis are only valid to call from the main thread.</p>
<p>[1] There is such a lot of work to do for Early Access that often I can’t afford to stress on issues that can be worked out another day. The important thing is to be making progress somewhere as often another day I’ll have had time for the solution to come to me.</p>
<p>[2] Culling means some instances are not being drawn and so the instance-id for a given tile can be different from frame to frame.</p>
TaleSpire Dev Log 2262020-09-16T13:27:14+00:00http://www.techsnuffle.com/2020/09/16/talespire-dev-log-226<p>A quick tip when dealing with <a href="https://docs.unity3d.com/ScriptReference/Rendering.BatchRendererGroup.html">BatchRendererGroup</a>. The instance limit per batch is 1023, but the UNITY_INSTANCED_ARRAY_SIZE limitation still applies[0], and so on desktops, this will likely cut your max batch size down to 500. This matters if you rely on instance-ids. I saw some very odd stuff when a single batch went over 511 instances; what was happening was that Unity was splitting the batch in two, one for the first 511 instances and one for the last one. This meant the instance-id for the 512th instance was 0 rather than 511, and this naturally resulted in incorrect behavior when using it to index into a buffer.</p>
<p>That was an afternoon of pain explained after a quick trip to the frame profiler, I was so concerned that my batching code was wrong that I didn’t consider the instance-ids not being what I expected. I can’t wait to know more of Unity’s internals as these issues are such a pain and seem avoidable.</p>
<p>Another node that I added last week is the random node. It can produce random floats, int, and vectors of each. It uses a simple stateless random function I’ve used in shaders before, which means it would be terrible for security, but is perfectly fine for visuals. The node has an option called ‘Update Mode’ that might be worth mentioning.</p>
<p><img src="/assets/images/randomNode.png" alt="random node" /></p>
<p>By default, if you give the node the same input (the seed), you are going to get the same result on the output. You will probably feed time into the input to get different results each frame. However, if you are using global time, then the output will be the same for all instances of this script running on all assets this frame. That might not be what you are going for! This is where the ‘Update Mode’ comes into play. You can set it to global, which gives the behavior described above, or you can pick one of two ‘local’ options, which both modify the seed in a way that will give different results. The ‘Local’ option means that it mixes in the tile’s unique id, so now the result will be different from other tiles running the same script. Note, though, that each asset within a tile can have a script, and maybe you don’t want those assets to get the same result from the same node. To handle this case, you have ‘Asset Local’, which also mixes the asset’s index into the seed along with the tile’s id.</p>
<p>Alright, back to work.
Seeya!</p>
<p>[0] Unity’s use of constant-buffers for its object-to/from-world matrices and constant-buffers have a max guaranteed size of 65k. See <a href="https://catlikecoding.com/unity/tutorials/rendering/part-19/">section 1.4 over here</a> for some interesting details.</p>
TaleSpire Dev Log 2252020-09-15T20:22:51+00:00http://www.techsnuffle.com/2020/09/15/talespire-dev-log-225<p>Hey folks. For the last few days, I’ve been looking at batching again. Unity has an interesting limitation in that the max batch size is 1023. This seems to be an artifact of some decision long ago, however, we need to make sure we don’t try and make batches bigger than this. Making our code respect this is a delicate operation as it’s very multi-threaded code and the access to the data are often not protected by Unity’s safety systems (for performance reasons). This means I’ve been taking it very slowly, it’s going well, and I hope to have it finished soon.</p>
<p>As that is quite a short update, here is something from last week.</p>
<p>There is a node in Spaghet for choosing a mesh from a list. We use this for the stop-motion style effect of the fire. It looks like this in the graph.</p>
<p><a href="multi-mesh-node.png">multi-mesh</a></p>
<p>While fixing bugs in the implementation, I had to fix things both on the TaleWeaver and TaleSpire sides, and the process got very tedious. At some point, I decided to re-prioritize work on the modding tools and hooked up Spaghet such that it works as a live previous in TaleWeaver. Here it is in action. Notice that every time we save, the changes are immediately applied.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/ybnJR2_FxR8" frameborder="0" allowfullscreen=""></iframe>
<p>This sped up the process a lot! I was able to find some mistakes in my animation packing code and fix those before getting back to batching.</p>
<p>Alright, that’s the lot for this update.</p>
<p>Peace.</p>
TaleSpire Dev Log 2242020-09-14T13:16:05+00:00http://www.techsnuffle.com/2020/09/14/talespire-dev-log-224<p>Hi everyone, last week was intense but productive. I’ve been down at Ree & Heckbo’s place, working on the engine. We had hoped to start integrating our branches, but my stuff still was not ready, so instead, we just kept working, making use of the collaboration advantages that being in the same room gives.</p>
<p>As there has been a bunch of stuff happening, I’m gonna spread it over a few posts.</p>
<p>Let’s start with something random: line-of-sight.</p>
<p>Line of sight checks came up in conversation in the community recently. The issue was that people have issues because one creature can block the line of sight to other creatures behind it. This is one of those things that sounds like it makes sense but just doesn’t feel right in-game. Any gaming using miniatures is always an approximation of the fantasy setting, and so sticking to it like it represented reality often doesn’t make sense.</p>
<p>Let’s take a concrete example. A giant is blocking the view to a goblin behind its leg. This sounds fine, but imagine the scene as it would play out in ‘real-life’. The giant would be towering above you, and as it walks, you catch glimpses of the goblin as it darts about trying to get a shot on your comrades. This fidelity is lost if we treat the table as ground truth.</p>
<p>Instead, it’s better to ignore iter-creature relationships when checking line-of-sight.</p>
<p>This poses a minor implementation issue that is worth exploring. Back in the alpha, we checked line-of-sight using collision rays. This worked, but it was inaccurate as you can only perform so many checks per frame. Back then, we checked less than 10 positions per creature, and both false positives and negatives were common. People rightly asked for a more accurate line of sight.</p>
<p>In response, we changed the approach to rendering the whole scene from the creature in question’s perspective. We rendered tiles and creatures into a cubemap using colors as IDs. We then used compute shaders to gather the results and report back what colors (and thus creatures) we visible. As you can see, this introduces the issue that any creature that is entirely obscured by another creature will be considered invisible.</p>
<p>To fix this limitation, we are going to need something new. The idea is to render the scene without creatures into the cubemap, then we will render all the creatures, but they will not write their depths (in-fact, we will probably discard all their fragments). This means that we should get a fragment shader call for every fragment in each creature that isn’t discarded by depth. We can then write the visibility info from the fragment shader into the result buffer, hopefully giving us the data we need.</p>
<p>This will also help fog-of-war, which uses the same visibility cubemap. (more on fog-of-war another day).</p>
<p>I’m gonna jump back into code now, more dev-logs coming when I take breaks!</p>
<p>Ciao</p>
TaleSpire Dev Log 2222020-09-07T02:02:20+00:00http://www.techsnuffle.com/2020/09/07/talespire-dev-log-222<p>Heya folks.</p>
<p>Ree has been working full steam on the flying mechanic, and it’s looking pretty damn nice. You should get your hands on it very-soon™ if all goes well.</p>
<p>Work on assets has been going well too. We have some good stuff we hope to show there soon-ish also.</p>
<p>Before the weekend, I tried to get more dice working. However, the D12 was spinning relentlessly, so I just put that down for a bit. I feel like we are past the big hurdles with that part of the physics integration, so it should be a case of just hunting down those issues.</p>
<p>I also started on porting the animated fire to the new system. This requires some relatively simple changes to the batcher but also means we need to recreate some scripts in spaghet. Here is the noodly goodness as it stands now.</p>
<p><img src="/assets/images/fireScriptGraph.png" alt="noodles" /></p>
<p>This will definitely be cleaner when we support collapsing subgraphs into function-nodes, but it’s not too bad even now. We are sampling a few curves in order to:</p>
<ul>
<li>pick the active mesh</li>
<li>rotate the object which holds the light</li>
<li>adjust the light’s intensity</li>
</ul>
<p>For the curious, this compiles down to a short sequence of operations.</p>
<p><img src="/assets/images/fireScriptOps.png" alt="ops" /></p>
<p>I still need to switch from local time to global time and add a random-number node to add some timing variation between each fire instance. I believe I still need to do a little work to handle some cross-frame state, but this hopefully won’t throw any spanners in the works.</p>
<p>Tomorrow I’ll pick up where I left off and try to get this lot running in TaleSpire. Except for lights, which are their own can of worms.</p>
<p>Seeya then.</p>
TaleSpire Dev Log 2212020-09-01T14:00:30+00:00http://www.techsnuffle.com/2020/09/01/talespire-dev-log-221<p>Hi everyone,</p>
<p>Integrating the new physics engine is going well. I wrote a component that converts old-style colliders and rigid-bodies into the new style.</p>
<p><img src="/assets/images/physConversion.png" alt="conversion" /></p>
<blockquote>
<p>Sorry about the weird colors here. Unity today decided that I last renewed my license tomorrow (yup), and so right now, it thinks I’m unlicensed and gave me the light editor theme[0]. The blue tint on the righthand-side lets me know when I’m in play mode, which is super helpful. It looks a lot less baby-blue when used with the dark theme :P</p>
</blockquote>
<p>I’ll pretty the UI up in time. For now, seeing everything is helpful.</p>
<p>It took me a while to work out where to get some mass related values right, so the dice rolled strangely, but now it’s a little better.</p>
<p><img src="/assets/videos/fixedTimestep.gif" alt="dice roll" /></p>
<p>Behind the scenes, a lot is going on, not least of which is that the physics is running with a fixed-timestep. This shouldn’t be a big deal as a fixed-timestep is mandatory for stable physics across framerates. However, this is not <a href="https://forum.unity.com/threads/fixed-timestep-features-coming-to-entities-dots-physics-packages.899687/"><em>fully</em> supported</a> in Unity’s ECS, so I was concerned that it would be difficult.</p>
<p>Luckily for us, it was not[1]. The physics engine is pretty great at giving you control, so I loop running the simulation for the required number of steps. If you have read about fixed-timestep before[2] you’ll know that you need to interpolate the results as the final step. The physics engine doesn’t have that support for that out of the box, so we added a job to collect that information and apply it as we write the transforms back to the GameObjects.</p>
<p>With that done, I need to replicate the parts of the old physics API that we use. If I can get dice rolls working again then I should have replaced most of what we need.</p>
<p>Have a good one folks,
Peace.</p>
<p>[0] yup I do know about the latest version making the dark theme free, but upgrading is risky and we are on a tight schedule right now.</p>
<p>[1] The interpolation is not hard, but the Unity folks have a lot more to support that we do, so we get to only handle the simpler case.</p>
<p>[2] The best known fixed-timestep explanation can be found <a href="https://gafferongames.com/post/fix_your_timestep/">over here</a>. It’s a good one.</p>
TaleSpire Dev Log 2202020-08-29T16:05:37+00:00http://www.techsnuffle.com/2020/08/29/talespire-dev-log-220<p>Phew, a few days away from the dev-log there. I got really caught up in some changes that were a real grind.</p>
<p>I’ve been working on the behavior of uncommitted changes. This work is taxing as you have a matrix of different states tiles might be in. Let’s take redo of delete for an example.</p>
<ul>
<li>You might be redoing an undo which has been committed</li>
<li>You might be redoing an undo which has not yet been committed but the delete action it undoes has been committed</li>
<li>You might be redoing an undo where neither the undo nor redo have been committed yet</li>
</ul>
<p>And that’s one action in isolation. What about the delete of assets which were made by undoing a separate delete. The matrix of possibilities applies to all the actions involved in that sentence too. In short, making sure you get a reasonable result takes a lot of concentration (and bizarre diagrams :D ).</p>
<p>When I finally got it running locally well, I connected up a second client and immediately disaster. The result was very wrong. I’ll admit I started panic sweating at that point. Last time this happened, we had to delay the Beta. Luckily this time, the system is much simpler, and I have an extensive test suite to lean on.</p>
<p>I dusted off the tests (I may have neglected them for a while) and started digging into issues. The great news was that, whilst the tests did find a couple of bugs in the new code, the underlying board code was still working fine.</p>
<p>This helped me relax and focused my attention back into the code that handles the batching of uncommitted tiles. After some time in the debugger, I finally spotted a case where tiles were not removed from the uncommitted state when committed. This amounted to a constant being 1 instead of -1. With that typo fixed, it started behaving rather well.</p>
<p>With that terrifying detour done, I needed to get back into physics. I have had to write a lot of the physics code fairly reactively rather than with a solid plan, and so I hit the point where I needed to clean things up. This sucked. It was a painful combination of being very tedious but also critical to get right. Some days are just like this.</p>
<p>These last weeks I’ve really been feeling the pain of not being able to just rely on the game-engine in the ways we usually do. Because of our bad estimate of when Unity’s new ECS would be ready, we are having to make a lot of systems ourselves. This is fine in theory (I love making stuff), but it’s meant losing at least a month of dev time that we did not expect. That’s always difficult, but our timeline was already tight. It’s pretty stressful.</p>
<p>Anyway, panic/rant aside, things are progressing. The thing on my plate now is hooking regular Unity GameObjects into our physics setup. We need something that takes current-gen Unity Physics components and makes equivalents in a format suitable for their new physics engine. We can then write some code to manage copying the values to and from the new physics engine to keep it all in sync.</p>
<p>If all that goes well then hopefully I’ll be rolling dice again soon!</p>
<p>Thanks for stopping by folks,</p>
<p>Seeya in the next dev-log</p>
TaleSpire Dev Log 2192020-08-27T02:06:31+00:00http://www.techsnuffle.com/2020/08/27/talespire-dev-log-219<p>Today has been spent working on the behavior of uncommitted action.</p>
<p>Whenever you add or delete tiles, we need to perform the action on the other players’ boards. To get the right result, the actions need to be applied in the correct order. This is done by sending all of these actions to a player, which was automatically designated as the ‘host’. They then relay the action to everyone, which gives us an order. We call an action which has been given an order a ‘committed action’.</p>
<p>Given that we can only apply the actions once they have been committed, we are in this annoying position where we need to wait for that roundtrip before we see a change from our keyboard/mouse click. That feels bad[0], so we needed to do something about it. By the time of the beta, we had made a system that could update the board’s visuals for our uncommitted changes so that the change looked immediate. It was complicated, but it was made much more so by the fact that we had to spread some changes over many frames due to Unity not being able to keep up with spawning without impacting frame-rate.</p>
<p>Now that we have taken control of batching, we can happily spawn thousands of tiles a frame with no issues, and this lets us drop the performance workarounds. I’m now rewriting the code that handles applying visual changes for uncommitted actions.</p>
<p>It’s going pretty well. I have uncommitted actions working for adding tiles and undo/redo of the same. I have made good progress on delete and, although it’s significantly more complicated, I’m pretty confident I’ll have it working tomorrow. If I can get this done in two days, that’ll be pretty great as the last version took months to get right. [1]</p>
<p>As soon as this is behaving, I will be switching back to physics. The first task is to make the ground a physics object again, and then I will be looking at dice. I hope to have some familiar dice rolling gifs by Monday :)</p>
<p>Hope this finds you well folks.</p>
<p>Peace</p>
<p>[0] It might seem like not a big deal, but we had complaints about the responsiveness of undo/redo in the alpha, and back then, that delay was mostly due to performing the action on key-up rather than key-down. Input latency really matters.</p>
<p>[1] Of course, it’s not an apples-to-apples comparison as the codebase is already structured in a way where this would be possible. However, it definitely feels much simpler.</p>
TaleSpire Dev Log 2182020-08-25T17:53:43+00:00http://www.techsnuffle.com/2020/08/25/talespire-dev-log-218<p>Heya folks,</p>
<p>Today I started out looking at the slab code again. The slab is the way we refer to a section fo tiles that have been copied. When you paste a slab string, that slab appears in your ‘hand’ in-game, and you can place it like a tile.</p>
<p>Recently we have been writing batching code, something that used to be handled by Unity. It got to the point where I needed something similar for the slabs, which I started yesterday. There are a couple of differences from the standard code batching code:</p>
<ul>
<li>We don’t want physics to be set up for the tiles/props</li>
<li>We don’t want scripts to start running</li>
<li>We need to update the position of every object in the slab every frame (as the player moves it)</li>
</ul>
<p>This mainly involved taking the existing code, taking a chainsaw to the parts we didn’t need, and stitching together the remains. It’s still very rough, but I was able to try pasting in the slabs from the community… which of course crashed the game :P It was obviously not the fault of the community content, it was that it was a much more complete test that I had run up until then and it found a few bugs. One was a case where we had assets in TaleWeaver with a <a href="https://docs.unity3d.com/2017.3/Documentation/Manual/class-MeshFilter.html">MeshFilter</a> with no mesh specified. Another was that I hadn’t written the code to handle missing assets.</p>
<p>I got the code to the point that I could paste a slab. However, it feels pretty rough as the ground plane in the game doesn’t exist in the new physics system yet. I’ll be back working on the physics code soon, and then I can take another crack at this.</p>
<p>The rest of today has been spent looking into building code. I need to re-implement the stuff that handles things that have been affected by changes that have not yet been committed (confirmed by the host). It’s pretty hard going, but I’m hopeful I can make some decent progress this week.</p>
<p>I’ll leave ya with a view from the cabin one of our friends were kind enough to invite us along to over the weekend,</p>
<p>Ciao</p>
<p><img src="/assets/images/trip0.jpg" alt="the hills" /></p>
TaleSpire Dev Log 2172020-08-25T03:40:31+00:00http://www.techsnuffle.com/2020/08/25/talespire-dev-log-217<p>After a lovely couple of days out of town, I’ve got back to business, and it’s been a busy one.</p>
<p>I spent the first half the day on dynamic colliders. For now, these are the colliders that are attached to tiles (and props) and are moved by scripts. This was a somewhat painful process as I ended up bouncing back and forth between TaleWeaver and TaleSpire as I needed to make some tweaks to the code that dealt with the object hierarchies.</p>
<p><img src="/assets/videos/dynamicCollider0.gif" alt="dynamic collider 0" /></p>
<p><img src="/assets/videos/dynamicCollider1.gif" alt="dynamic collider 1" /></p>
<p>With that behaving, I stubbed out the code for computing the motion data required by the physics engine. Finally, I tweaked the per-frame code so that the physics engine knows when it doesn’t need to be rebuilding all the internal data for static objects. This should, in normal cases, give decent performance.</p>
<p>For the rest of the day, I switched to slabs. I really wanted to be able to paste large objects from the community and see them appear immediately. Annoyingly they have managed to avoid working yet. The majority of the work is done. I’m just chasing down smaller bugs and mistakes, but it’s late, and I need to sleep.</p>
<p>I expect to write up a bit more on this once it’s working.</p>
<p>Seeya tomorrow.</p>
<p>p.s. I’ll show a pic from the weekend to those curious when I get them from my partner. I’m using a dumbphone currently, and photos are not it’s strong suit :D</p>
TaleSpire Dev Log 2162020-08-21T17:10:11+00:00http://www.techsnuffle.com/2020/08/21/talespire-dev-log-216<p><img src="/assets/videos/chest.gif" alt="chest" /></p>
<p>This looks like nothing. If you used TaleSpire, you’ve seen this a thousand times. For me however, this is great. The fact it looks the same means the following:</p>
<ul>
<li>Spaghet’s realtime scripts are being controlled correctly by the state-machine scripts</li>
<li>The custom animation system is working</li>
<li>The new batching system is working</li>
<li>All the work from the last week to replace the old physics engine has paid off. If you can pick a tile (which we are), then the raycasts are working.</li>
</ul>
<p>This has been a slog. Any time you have multiple days without being able to start the game, it’s mentally draining, and these last weeks have been tough. There are piles of issues still[0], but I think I <em>might</em> be through the worst part.</p>
<p>Have a great weekend everyone</p>
<p>[0] To name just one, mesh colliders aren’t properly oriented</p>
<p>p.s. I’ll be offline for the next few days as I’ll be off recharging in the country somewhere :)</p>
TaleSpire Dev Log 2152020-08-17T09:19:38+00:00http://www.techsnuffle.com/2020/08/17/talespire-dev-log-215<p>Progress feels slow but is steady.</p>
<p>The batcher now produces the data the physics engine needs for static colliders. Thanks to that, I’ve started writing some helper methods that mirror the parts of the older physics API we use. This should make the porting of the existing code noticeably easier. I’ll find out how that goes today as I’m planning to start porting that now.</p>
<p>You’ll notice I specifically mentioned static colliders above. I haven’t worried about the colliders attached to objects moved by scripts yet. The reason is that they are, for the most part, just a slightly more complicated version of the thing I’ve already done, whereas the board tools and dice are still unknown quantities.</p>
<p>I’m hoping the tool side goes smoothly today as I really want to start looking into things like dice and creature, which will have a hybrid setup in this system. By hybrid, in this case, we mean that we will be using Unity’s GameObjects, but we will write custom components to hook them into our physics setup [0].</p>
<p>Speaking of the physics setup, we’ve always had two sets of colliders we keep in different layers in Unity. We have the ‘explorer’ layer, whos colliders we use when placing tiles, and the ‘tile’ layer[1], which is the version used during play by dice/creatures etc. The split allows us (and modders) to provide a better building experience by tuning the explorer colliders. As I’ve been working with the new physics engine, I’ve started to think that it might be worth using two totally separate physics-worlds, rather than layers.</p>
<p>The reason is one of performance. When querying the colliders, we usually care about either the explorer layer, or the tile one and the physics engine has to skip the ones we don’t care about. It does this via a layer mask, so it’s a quick check, but not doing something is always faster than doing something. I’m also musing about if this allows us to do less work per frame by having different update frequencies for the two worlds. We will be trying to keep things fast by, per-frame, only giving the physics engine data for the zones[2], which contain the player’s cursor or dynamic objects like dice. However, there is a tradeoff between changing data (which requires more frequent rebuilds of internal data-structures[3]), and the number of things being tracked by the physics engine (which affects query times).</p>
<p>This is something that can be tuned over time, so I may just set up the code so that swapping and changing will be less painful in the future.</p>
<p>Alright, that’s all that I need to ramble about for now. Hope to see you in the next one.</p>
<p>Ciao.</p>
<p>[0] In contrast, all tiles/props are managed/batched by us. It’s a tradeoff between having Unity help us with lots of things and performance.</p>
<p>[1] Yeah, not great naming :P</p>
<p>[2] 16x16x16 regions of world space</p>
<p>[3] BVH https://en.wikipedia.org/wiki/Bounding_volume_hierarchy</p>
<p>p.s. oh yeah I’ve also started on tools to visualize colliders in our new system</p>
<p><img src="/assets/videos/cast.gif" alt="raycast" /></p>
TaleSpire Dev Log 2132020-08-14T08:45:48+00:00http://www.techsnuffle.com/2020/08/14/talespire-dev-log-213<p>Morning all,</p>
<p>Yesterday I got the new collider data loaded into TaleSpire and started writing the code that would lay out all the static colliders.</p>
<p>I originally planned to store the colliders per asset[0], but after writing a significant amount of code, I remembered that Unity.Physics’ colliders don’t support scaling. This means I need to store scald versions of the asset’s collider for each Tile/Prop[1] that uses it.</p>
<p>I realized this fairly late in the day, and it was a bit of a bummer to have to start rewriting then, so I stopped for the evening and relaxed instead.</p>
<p>Today I’ve got to detour into some other code, but I’ll get back to this asap.</p>
<p>We do learn from this however, that we won’t be able to animate the scale of the parts of assets that have colliders. I’m not sure if this means we’ll just disallow animating scale or only give warnings if you do it to something with scale. I’ll chat with Ree about this in our next daily.</p>
<p>Alright folks, hope you are doing well.</p>
<p>Seeya</p>
<p>[0] In TaleSpire, Tiles, Props, and Creatures are made of assets arranged in different ways. Assets themselves are made of objects, but you don’t address those sub-assets directly.</p>
<p>[1] Each kind of tile, not one for every instance of the tile.</p>
TaleSpire Dev Log 2122020-08-12T00:47:53+00:00http://www.techsnuffle.com/2020/08/12/talespire-dev-log-212<p>Today was pretty simple. I spent the day reading into the Unity.Physics package and trying to work out more of the nitty-gritty of how we will work with it. Tomorrow I’ll start implementing things.</p>
<p>The first thing to do will be to work out the serialization for colliders on assets. I’ll be slightly changing the data stored per object inside assets to avoid additional processing on the TaleSpire side. I’ll then be looking at the colliders that live on tiles and are used by the building tool.</p>
<p>It’ll then be over to Unity to look at loading this stuff and how to asynchronously create the mesh-colliders. I’ll need to ensure that the other systems aren’t notified that the asset is loaded until this processing is complete.</p>
<p>I’m sure I’ll run into some other little complications on the way, but that’s fine.</p>
<p>I expect this will take most of tomorrow, and so on Thursday, I’ll start updating the batcher to handle the physics data. Colliders attached to animated objects will likely require some attention to get right.</p>
<p>With all that in place, it will hopefully be time to write a little helper API and to start updating all the existing code to use this new system.</p>
<p>We’ll keep you posted on how this goes.</p>
<p>Seeya</p>
TaleSpire Dev Log 2112020-08-10T15:05:16+00:00http://www.techsnuffle.com/2020/08/10/talespire-dev-log-211<p>Finally, something decent to show again!</p>
<p>I won’t lie, the last couple of weeks of coding have been hard. I’ve been working on the new animation system, and, as you’ll see, this dragged in a LOT of other systems. While the final solutions are not overly complicated, finding out what they needed took a lot of tries, and the number of systems involved made it hard to keep in my head.</p>
<p>First, though, videos time!</p>
<p>The following clip shows 10k animated objects being spawned, all with their own scripts running independently, without dropping below 60fps. They are all in sync as I wanted to see lots of things happening, so I made time loop every 2 seconds, causing the animations to replay.</p>
<video controls="" src="/assets/videos/clams.mp4" type="video/mp4" width="620"></video>
<p>This might be a bit underwhelming to look at, but I’m so glad to see this. Let’s dive into why.</p>
<h3 id="scripting">Scripting</h3>
<p>For a long time, we knew we needed to be able to add behaviors to tiles and props. Without hackery, Unity doesn’t let you add more code at runtime, so we can’t just load c# code from asset-packs along with tiles and creatures. This got us looking into scripting languages.</p>
<p>We started looking at LUA. The C VMs are fast, but there are also c# VMs that have excellent integration with other c# datatypes.</p>
<blockquote>
<p>Side note:</p>
<p>A lot of the time, my job is looking at something simple and saying the words, “And then the user spawns 10000 of these” (let’s call this the ‘10k-case’). Making 10k of something in TS is very easy, just drag out a 100x100 region. This is why everything in user-generated-content games is more complicated than it seems.</p>
<p>My role is to look at this scaling problem try and limit the worst-case complexity. The players will always be able to make enough things that the game performs poorly, but we usually can find things to improve the situation.</p>
</blockquote>
<p>Looking at various scripting languages, my big concern is what happens when 10k are spawned as:</p>
<ul>
<li>Each needs an unknown and potentially unbounded amount of memory.</li>
<li>Each might be used in ways that aren’t thread-safe.</li>
<li>Everything object created needs managing by the GC.</li>
</ul>
<p>The last one is especially concerning. If you’ve ever shipped a game in a managed language (like c#), you’ve probably spent time minimizing memory allocations to stop the GC from impacting your frame rate. Without understanding the impact of a script, I was unsure we could make something that we knew would run well.</p>
<p>The next experiment was to use LUA like a scriptable composition system. We would provide performant ‘operators’ and modders would connect them up using a LUA script, which would be invoked less frequently (for state machines it would be on state change).</p>
<p>This was promising, but we are in this weird case where we have made LUA not like LUA anymore. So the people who like LUA and want to use it cant. And the people who hate LAU still have to use something they won’t like. And we still have the threading questions.</p>
<h3 id="spaghet">Spaghet</h3>
<p>This made me take a day to hack together a simple compiler. It produced a very simple byte-code that we would run on a little VM built on top of Unity’s Job System. By using the job system, the VM code is optimized by LLVM and we can run the scripts on many threads in a way that fits in well with our other systems.</p>
<p>Even simple tests showed promise. We split the world into two kinds of scripts:</p>
<ul>
<li>State-machine scripts</li>
<li>Realtime script</li>
</ul>
<p>State-machine scripts (currently) run only in response to user input, and they are fully synchronized across the network.</p>
<p>Realtime scripts can run every frame and are not synchronized (as there would be far too many messages). Their programming model is similar to shaders in that the tile’s visual state is recreated from scratch every frame from the input.</p>
<p>The trick now is that each state of a state-machine script can have an associated realtime script. This means that when in that state, the realtime script is run automatically.</p>
<p>For example, for a door, the state only changes (and is synchronized) when the user opens or closes it. However, by animating the door in the realtime scripts, you get the expected animation on all player’s machines at the right times.</p>
<h3 id="the-fall-of-the-gameobject">The fall of the GameObject</h3>
<p>Unity’s traditional approach to making this is all object-oriented. It’s compelling and, in the right hands, is an incredible canvas for creativity. Alas, when performance is critical object-oriented programming (OOP) is problematic. OOP makes it very hard to control where things stored are in memory, and this is essential when trying to batch jobs together to run quickly. CPUs can do unfathomable amounts of work per frame if they aren’t spending their whole time waiting on things to arrive from RAM.</p>
<p><img src="/assets/videos/cacheanim.gif" alt="relative speedasnimation" /></p>
<blockquote>
<p>Memory access latency from cpu caches v ram – Andreas Fredriksson</p>
</blockquote>
<p>On top of this, Unity’s GameObject’s are (relatively) slow to create, fine for most games, but terrible for our 10k-case.</p>
<p>We worked very hard trying to keep using GameObjects as they have such significant advantages for content creation and experimentation.
Arguably, this contributed to the delay of the Beta as working around the speed limitations had some very non-obvious interactions with our building systems. However, it was not to be, and even though the Beta shipped, we continuously see performance issues that we don’t have clear solutions too if we stick with the current approach.</p>
<p>Of course, we’ve known about this issue a long time, and the good news was that an answer was already in production. Unity’s ‘Entities’ system is their new approach to handling things in the game and promised impressive performance boosts. We were very much banking on that solving our issues.</p>
<p>This was a mistake on my part. Naturally, the Entities system is a complicated beast with its own tradeoffs, and Unity is taking its time to get it right. When we finally tested the Beta, it still had some concerning performance characteristics for our use case. That sucked.</p>
<p>The good news was that amongst all this new code, there were many high-performance tools that we could use to make our own system.
The problem is that, with GameObjects, we got so many things for free and now we need to handle them ourselves. Here are a few concerns:</p>
<ul>
<li>The new physics engine is excellent, but we use collision casts <em>everywhere</em> so most systems in the game are gonna need some modification</li>
<li>There is no culling or batching written for us. We need those.</li>
<li>There is no animation system at all</li>
<li>There is no way to specify lights expect using the classic approach</li>
<li>feel: This is a terrifying one. Game feel takes ages of tiny tweaks based on the creators’ intuition. If we lose that it’ll take a long time to recreate</li>
</ul>
<p>We knew we had to just go for it. As tiles and props are where the perf issues lie, if we can make a new system for them, we can afford to keep the other stuff as GameObjects.</p>
<h3 id="too-late-now-">Too late now >:)</h3>
<p>Anyway, there was nothing to do but to get started. I jumped over to TaleWeaver (our modding tool) and started work on the new data formats for everything we would need.</p>
<p>It was critical that we didn’t mess without our artist’s ability to work. Making games is hard, and everyone needs to be able to work as effectively as possible. Your smart little ideas and solutions must not tread on someone else’s ability to work. To me, this meant analyzing the GameObject representation in TaleWeaver, and writing out our own, TaleSpire specific, version of this data.</p>
<p>Luckily once again, new tools exist in Unity to help. The Entities package has a Blob data system that promised fast and job-safe data encoding un unmanaged memory. I made some tools that let it hook into Unity’s classic asset serialization system and got to work.</p>
<p>I extract animations, compile scripts, flatten transform hierarchies, and other simple stuff and pack the result into out blobs. Most of the slowdown here is split between learning new things in Unity, and working out what we need. We also made atlases from the tile icons.</p>
<p>I replaced Spaghet’s graph UI and iterated on the VM. Again, most of the time is just from working with new tools and trying to find the best ways to do the silly things we need.</p>
<p>On the TaleSpire side, I wrote a new asset database to accommodate the new formats we have made and hooked up enough of that to keep working[0]. I then started work on the batcher.</p>
<p>The batcher is a beast. Each frame we need to tell Unity what to draw and Unity, like any engine, needs that information in some format it can use. Luckily for us, most of our objects don’t move each frame, so if we can get it in the correct format when the board changes, we can keep reusing that data. From now on, we will use the term ‘statically-batchable’ to talk about all the things where we can use this technique. All of the objects where we do need to update the position/rotation/etc per frame we will dub ‘dynamically-batched’.</p>
<p>So I spent a day or so writing the static batcher. Each zone (16x16x16 unit region of space) in the game runs this independently and concurrently. We do not combine batches across zones. This means we make more draw-calls than we <em>could</em>, but this lets us enable and disable zones without recomputing the batches for any other zone.</p>
<p>Dynamically-batched content is harder. I had already decided that the animation system would be directly tied to the scripting system, that is, Spaghet will be animating the objects. That lets us simply say that any part of the tile that can be modified by a script during runtime is dynamically-batched, and the rest are static[1].</p>
<p>I spent some days getting the Spaghet wired up enough to continue. I then made jobs to run the realtime scripts, compute the transforms for the objects and write them into the batches. A fun thing to note is that the number of dynamically batched things is not changing frame to frame. This means that you can do the following:</p>
<ul>
<li>Size a batch with enough room for every dynamic & static instance you need</li>
<li>Write the static batch info leaving enough room at the end for the dynamic ones</li>
<li>Every frame write the updated dynamically-batched transform into the reserved space</li>
</ul>
<p>This is nice as you don’t need any extra allocations or draw calls. The downside is the complexity it adds to the system, but hey, we are here for the performance.</p>
<p>During all this, I kept finding small issues that meant I needed to change the data format to help the engine. This is the right thing to do, but switching back and forth between the projects was tough on the brain.</p>
<p>I’m gonna skip a pile of details for all of our sakes, but the result is that the thing we are now making is capable of getting us where we need to be. It’s not fast enough, or stable, but the tests are finally showing something promising.</p>
<blockquote>
<ul>
<li>Wrong thing to conclude: Unity is slow</li>
<li>Right thing to conclude: Unity <strong>is</strong> fast. But, for what we need, the default tools they have to drive are not.</li>
</ul>
</blockquote>
<p>Warning about performance numbers. We can’t make decent measurements of the game yet, I still need to write up a lot of new stuff before we can start getting metrics we can rely on. But what we can guarantee is that TaleSpire is getting faster.</p>
<p>I hope you enjoyed this. I also hope I managed to convey the reason I’ve been less mentally available recently. Things are going to stay ugly for me for most of this month, but I know we will have something cool in the end.</p>
<p>Have a great day.</p>
<p>Peace.</p>
<p>[0] Funnily enough, the old one is still in there now being used by the UI. I’ll drop it when I get to that part of the rewrite.</p>
<p>[1] there are obvious cases where we don’t have to recompute a tile’s transform every frame, for example, when an animation has finished playing. This is true and will be handled by Spaghet scripts knowing when they can go to ‘sleep’. I’ll cover this in another post.</p>
TaleSpire Dev Log 2102020-08-05T01:47:05+00:00http://www.techsnuffle.com/2020/08/05/talespire-dev-log-210<p>Hi folks, I skipped yesterday’s log as I was far too deep in code, and to be honest today has been much the same.</p>
<p>The TLDR is that I’m working on TaleWeaver and the modding tools. I’m doing this as we want to move to more directly managing the batching of what is drawn rather than leaving it all to Unity. Changing the way that works requires different data, a lot of which can be packed into the asset when it is exported from TaleWeaver. Changing this batching approach also means we need our own animation system, which is fine but related to the scripting system (Spaghet). Before you know it, we have major changes to scripting, animation, rendering, and more.</p>
<p>It’s been a heck of a lot to keep in my head at once. Every decision impacts how another system works. Complicating this, of course, is this should be at least reasonably performant, so you cant just use <insert your favorite design-pattern/paradigm/etc> and trust that it will come out rosy :P</p>
<p>All in all, I’ve just not had enough bandwidth to tackle anything else.</p>
<p>Ree has also been busy in the experiments around props. I’m already very excited to see what comes out of that.</p>
<p>Tomorrow I need to take some hours to look into some networking issues some users are having. Sorry for not getting to you sooner, you know who you are.</p>
<p>Warm regards from code-land.</p>
TaleSpire Dev Log 2092020-08-01T00:21:51+00:00http://www.techsnuffle.com/2020/08/01/talespire-dev-log-209<p>Phew, what a week. I’ve almost lost track of all the things that have been going on. Amongst it all, more code is getting written.</p>
<p>Ree has been able to get through enough other tasks that he’s got some time to start experiments with props behaviors. Props are one of the most significant missing pieces from the game right now, and whatever behavior we settle on, we are going to have to live with for a while, so it’s good to take our time with this and find something we like.</p>
<p>I’ve been back in TaleWeaver, looking at how board-assets work. Yesterday I wrote a tiny compiler for the state-machine scripts and added code for showing errors in the state-machine script graph.</p>
<p>Today I’ve written code to export the tile information in a new binary format. TaleWeaver now also makes atlases for the tile icons. This should allow TaleSpire to batch more draw calls in the UI, hopefully speeding that up a little bit.</p>
<p>The next stop for me is to take this new format, get it loading into TaleSpire, and rewrite the code that manages looking up tile information. I’ve also had great progress in the design of system that will run the realtime scripts, so everything is lining up to work well. Just got to keep hammering at it!</p>
<p>Hope this found you well.</p>
<p>Seeya</p>
TaleSpire Dev Log 2082020-07-28T01:08:09+00:00http://www.techsnuffle.com/2020/07/28/talespire-dev-log-208<p>Today, working with a total star from the community, we’ve been able to track down one cause of the “Stuck a main menu with a spinning hourglass bug”.</p>
<blockquote>
<p>TLDR: Check your proxy settings. You may find that manual proxy is enabled, but the address is blank. This exposes a bug in Mono (a thing Unity relies on), which breaks the code that finds the TaleSpire servers.</p>
<p>Here is what the setting looks like when it’s wrong. We think this might have been set by a Windows update, but we are not sure.</p>
<p>NOTE: This will not fix cases where the spinning hourglass is only sometimes a problem. I would love to look into any hourglass related issue you have though, so please reach out to me (@Baggers) on the TaleSpire discord.</p>
</blockquote>
<p><img src="/assets/images/proxyIssue0.png" alt="the problem" /></p>
<p>Now for the extended version of the story.</p>
<p>Many backers have never been able to play the TaleSpire beta due to a bizarre bug that means they get stuck waiting for TaleSpire to log in forever. Today I was once again struggling with this issue, so I decided to make a program to test the connection and make logs that would hopefully help us progress. I wanted the code to be as close to TaleSpire’s as possible, so I took the whole game and spent several hours ripping out pieces until I got down to just the essentials for the main menu. I added some logging and ended up with this:</p>
<p><img src="/assets/videos/TsConnectionTest.gif" alt="TsConnectionTest" /></p>
<p>With a new weapon in hand, I teamed up with a community member who had experienced the issue. For the next hour, they ran the test app and send me the logs. I would then modify the app and send them a new build. Together we finally got this trace.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Object reference not set to an instance of an object => at System.Net.AutoWebProxyScriptEngine.InitializeRegistryGlobalProxy () [0x0005b] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.AutoWebProxyScriptEngine.GetWebProxyData () [0x00007] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.WebProxy.UnsafeUpdateFromRegistry () [0x0001a] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.WebProxy..ctor (System.Boolean enableAutoproxy) [0x0000d] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.WebProxy.CreateDefaultProxy () [0x00012] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.Configuration.DefaultProxySectionInternal.GetSystemWebProxy () [0x00000] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.Configuration.DefaultProxySectionInternal.GetDefaultProxy_UsingOldMonoCode () [0x00036] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.Configuration.DefaultProxySectionInternal.GetSection () [0x00015] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.WebRequest.get_InternalDefaultWebProxy () [0x00022] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at System.Net.HttpWebRequest..ctor (System.Uri uri) [0x0008d] in <ae22a4e8f83c41d69684ae7f557133d9>:0
at (wrapper remoting-invoke-with-check) System.Net.HttpWebRequest..ctor(System.Uri)
at System.Net.Http.HttpClientHandler.CreateWebRequest (System.Net.Http.HttpRequestMessage request) [0x00006] in <7ebf3529ba0e4558a5fa1bc982aa8605>:0
at System.Net.Http.HttpClientHandler+<SendAsync>d__64.MoveNext () [0x0003e] in <7ebf3529ba0e4558a5fa1bc982aa8605>:0
</code></pre></div></div>
<p>A quick google took us here <a href="https://github.com/mono/mono/issues/10030">https://github.com/mono/mono/issues/10030</a>. A bug in Mono which can occur when a user has <code class="language-plaintext highlighter-rouge">ProxyEnable</code> in the registry, but no entry for <code class="language-plaintext highlighter-rouge">ProxyServer</code>. We ran the following in the command prompt to clarify the issue, but you can see it more easily by just going to Window’s Proxy settings (hehe the command line is always the first place I end up).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ProxyEnable
reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ProxyServer
</code></pre></div></div>
<p>Anyhoo, they then disabled the manual proxy setting, clicked ‘Save’ at the bottom of the settings page, and booted up TaleSpire. Instantly it worked as expected!</p>
<p>This is great, but I now need to find a workaround we can put in the code. I’m hoping that <a href="https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=netframework-4.7">HttpClient</a> will allow me to disable the proxy. Then I should be able to look up the keys and force disable the proxy if it’s the issue.</p>
<p>This is almost certainly not an issue in later versions of Unity, as they will probably be using a later version of Mono. This is a perfect candidate for the kind of issue that really makes you want to update the Unity version. However, that is always a bunch of work. We’ll see how this workaround goes.</p>
<p>That’s all I have for tonight. If all goes well I should be back on rendering code tomorrow.</p>
<p>Seeya</p>
TaleSpire Dev Log 2072020-07-26T00:52:34+00:00http://www.techsnuffle.com/2020/07/26/talespire-dev-log-207<p>Good evening all. For the last few days, I’ve been struggling with the graph UI and how I want to serialize the scripts. The TLDR is that I’m making progress again, but I’m about two days behind where I wanted to be.</p>
<p>One afternoon was spent trying to understand some odd behavior I was seeing when dragging assets onto nodes in the spaghet graph. I was able to make a decent reproduction of the issue, and the library author was incredibly quick at fixing it.</p>
<p>The graphs themselves are usually stored as <a href="https://docs.unity3d.com/Manual/class-ScriptableObject.html">ScriptableObjects</a> in Unity. This is ideal for the state-machine scripts, which I want to share between tiles, however, it did not fit my design for the realtime-scripts, which are closely tied to the assets the manipulate. I struggled with the serialization logic here for about a day, and I’m not satisfied with it yet. Regardless I need to make progress with TaleSpire, so I need to crack on.</p>
<p>Today I’ve started work on the UI, which lets you pick the script for a tile and then assign a realtime-script to each state. It’s looking something like this right now.</p>
<p><img src="/assets/images/twScriptSetup.png" alt="setting up behaviors" /></p>
<p>Basic, but enough to make progress.</p>
<p>Once I have this information serializing, I will finally start writing the new asset format. That will let me switch back to TaleSpire and rewrite the asset database. It should then be faster and compatible with the job system, which opens up some performance improvement possibilities in the future. However, we won’t be chasing those particular performance improvements, as we need to write the new animation system first.</p>
<p>Back with more real soon.</p>
TaleSpire Dev Log 2062020-07-22T23:50:29+00:00http://www.techsnuffle.com/2020/07/22/talespire-dev-log-206<p>Hey folks. I’ve now ported enough of my old Spaghet experiments to the new node graph that I can compile code again.</p>
<p>Here is a useless script:</p>
<p><img src="/assets/images/spaghetCompiled0.png" alt="spaghet0" /></p>
<p>On the left, we have some nodes (the node visuals are still WIP), and on the right Visual Studio’s debugger showing the result from the compiler. I’ve indicated the debug representation as it’s a bit easier to see the generated operations. The actual result is just a byte array, so I haven’t bothered to expand it (spoiler, it’s full of bytes).</p>
<p>I’ve also added some code to check for type errors and to report loops in the graph. This feels like the bare minimum requirements to be able to code in a sane way.</p>
<p><img src="/assets/images/spaghetLoopError.png" alt="spaghet1" /></p>
<p><img src="/assets/images/spaghetTypeErrors.png" alt="spaghet2" /></p>
<p>Tomorrow I’ll look into the output nodes. First, I’ll add ones for setting an asset’s transform and another for driving an animation. Both will let you drag the assets in questions straight from the Unity hierarchy so that it fits the standard Unity workflow.</p>
<p>That’s all for now. Seeya tomorrow.</p>
TaleSpire Dev Log 2052020-07-21T23:58:15+00:00http://www.techsnuffle.com/2020/07/21/talespire-dev-log-205<p>Good evening folks,</p>
<p>It’s been another decent couple of days convincing computers to do things.</p>
<p>Most of my time this week has been spent getting to grips with <a href="https://github.com/alelievr/NodeGraphProcessor">NodeProcessorGraph</a>, which so far I’m loving. The main developer has been super responsive, making me even happier about sticking with this library.</p>
<p>I didn’t write up yesterday as it was one of those days that you know you need to learn a system very quickly and so you just slam your brain against it all day until you realize why things are the way they are :) A necessary endeavor, but not one that leaves you much in the mood for writing.</p>
<p>Today, however, I’ve been knocking together very simple nodes for the state-machine scripts that tiles can use. The closest thing I have to a visual is this:</p>
<p><img src="/assets/videos/stateMachine0.gif" alt="oo nodes" /></p>
<p>This graph only expresses the state of the tile (or prop in future), and what actions transition between them. To do something visual, you will need the realtime-scripts.</p>
<p>A quick aside. In TaleSpire, tiles and props are things called <code class="language-plaintext highlighter-rouge">BoardAsset</code>s. <code class="language-plaintext highlighter-rouge">BoardAsset</code>s are collections of <code class="language-plaintext highlighter-rouge">Asset</code>s, which act as a single unit. The <code class="language-plaintext highlighter-rouge">BoardAsset</code> holds the information of each <code class="language-plaintext highlighter-rouge">Asset</code> required and it’s local position/rotation/etc within the unit (along with a bunch of other <code class="language-plaintext highlighter-rouge">BoardAsset</code> specific data). The cool thing is that a <code class="language-plaintext highlighter-rouge">BoardAsset</code> may be made of <code class="language-plaintext highlighter-rouge">Asset</code>s from totally different asset-packs. This means you will be able to remix the <code class="language-plaintext highlighter-rouge">Asset</code>s that people like us provide.</p>
<p>By separating the two, we can share the state machine among many tiles easily and then, per-tile, choose the visual behavior that occurs during each state by mapping states to the realtime-scripts which live on the <code class="language-plaintext highlighter-rouge">Asset</code>s.</p>
<p>Codesharing of realtime-scripts will be done through making functions. These will be sharable as strings so that we can all do the same kind of thing that we do with slabs today. That’s the idea at least :P I’ll be writing that bit soon.</p>
<p>Tomorrow will be focussed on the realtime-scripts. I want to see if I can get the compiler for those working again in the next few days. When that is done, I’ll see what the next logical step is. It’ll probably be writing the state-machine compiler, which will be pretty straightforward.</p>
<p>Ok, enough rambling for now.</p>
<p>Seeya tomorrow!</p>
TaleSpire Dev Log 2042020-07-20T00:11:50+00:00http://www.techsnuffle.com/2020/07/20/talespire-dev-log-204<p>On Friday, I got the code written to extract Unity’s animation data and pack it into the AssetBundle file in the way we wanted. This data will be utilized by the upcoming animation system to be used by tiles and props.</p>
<p>Before we get there, and as mentioned in the last dev-log, we need to update the boardAsset format that TaleSpire uses.</p>
<p>For the format rewrite, the first thing to review was what we were going to store. This quickly led me to look at the scripting system again. I tried to write a high level of some of the stuff I’m looking at, but it gets rather vague, so I’ll just talk about something concrete instead.</p>
<p>I am 80% sure I want tile/prop animation to be driven by the scripting system (Rather than tile/prop I’m just gonna say tile from now on for brevity). I already intended to have two kinds of scripts for tiles, state-machine scripts, and realtime-scripts.</p>
<p>The state-machine scripts are driven by user interaction and are synchronized across the network. Realtime scripts run every frame and are unsynchronized. I’ve imagined the realtime scripts to be like shaders in that they’ll run fast, in parallel, and don’t hold their own state frame-to-frame.</p>
<p>I’m looking into being able to specify a realtime script as the behavior for a given state-machine state. So if you are in a given state, the game runs the associated realtime script. The realtime script would set things such as the position/rotation/scale of parts of the tile, parameters of lights, and be able to drive animations.</p>
<p>The last item in that list of examples is the interesting one for today. If I can implement animation via the scripts, it would allow me to flesh out that part of the codebase now, and then look at custom animation code as a performance optimization for later.</p>
<p>Because of all this, I’ve started looking back into Unity’s support for node-graphs, as Spaghet is meant to be a visual scripting language. In the last 11 months, it seems like <a href="https://github.com/alelievr/NodeGraphProcessor">NodeGraphProcessor</a> has come a long way. I like a lot of what I see there, and so tomorrow I’m going to start digging into how to use it to implement the state-machine scripts.</p>
<p>Once again, not making hard claims about the delivery of specific features is paying off as, if required, I’ll be able to reorder a bunch of work without causing issues. I’m sometimes sorry that this means we can’t give you as many concrete dates as would otherwise be ideal; however, the ability to roll with the punches is really helpful.</p>
<p>Hope you folks have had a good weekend, more news as it happens :)</p>
<p>Seeya</p>
TaleSpire Dev Log 2032020-07-17T15:02:45+00:00http://www.techsnuffle.com/2020/07/17/talespire-dev-log-203<p>Phew, this one took a while to get out.</p>
<p>The art team is hard at work, prepping the next big asset patch. Should be another fun one!</p>
<p>On Wednesday, I had some boring life stuff to sort out, so I wasn’t working that day.</p>
<p>On Thursday, Jonny and I had a catchup meeting and roughed out a plan for the animation system for tiles & props. To get working on this, I first need to extract the animation data into a format we can work with from the job system. It took a while to realize that, if you load a prefab from a Unity <a href="https://docs.unity3d.com/Manual/AssetBundlesIntro.html">AssetBundle</a>, the full curve information is no longer available. This is ok, as in TaleWeaver we work with the original prefab files, but it was still a slow journey learning the ins and outs of this part of Unity.</p>
<p>With that worked out, I wrote the first pass on the extractor. This was looking ok, but the standard asset serializer only supports classes, and I want the data in a job-friendly format. Now, Unity does have a serialization scheme they use inside their ECS when serializing Entities to their Scene format. They call these <a href="https://docs.unity3d.com/Packages/com.unity.entities@0.2/api/Unity.Entities.BlobBuilder.html">BlobAssets</a>. These are fast, immutable, and pretty much ideal. However, they are not compatible with the AssetBundle serializer we mentioned early. To get around this, I have written an adapter that makes this possible. That took a long time, and I still would like to add a bit more data validation. However, I’m happy that I can now write the data in the way I want.</p>
<p>The next task is to rewrite the animation extraction code using this new store. I’ve been going slow here as I need to be able to uniquely a given resource (e.g. an animation clip) inside an asset, and I want to make sure that these are stable to the kinds of changes that are made when developing a tile or prop.</p>
<p>With that done, I’ll be writing a new system for packing the board asset data. I’ve looked at this several times when I was investigating CaptnProto and FlatBuffers, now I have a good workflow with BlobAssets I’m going to use that instead. The current JSON format will continue being used by TaleWeaver to allow the art team to iterate on that format as required; however, when we export an asset pack, we will use the new system instead.</p>
<p>And once all that is finally done, I can start writing the animation system :)</p>
<p>I have also been researching Unity’s new Physics engine again and am slowly designing how we are going to interface with that. It’s going well, but I wanted to get animations working first as some assets (like doors) move colliders using animations.</p>
<p>Alright, until next time.</p>
<p>Peace</p>
TaleSpire Dev Log 2022020-07-14T20:28:46+00:00http://www.techsnuffle.com/2020/07/14/talespire-dev-log-202<p>Allo folks,</p>
<p>Some good progress today. The new batcher seems to be laying out tiles correctly now, so I can kind of build again. I haven’t hooked into the new physics engine so I can only build on the build plane currently.</p>
<p>What was nice was I could do some quick performance tests to see what ballpark we are in. I was able to spawn 10000 tiles in one frame without dropping below 60fps which is very promising. There are things the code doesn’t have to deal with yet, such as animations and physics. However I’m feeling confident that we’ll be able to get a large speed up in tile spawning over what we have in the beta today.</p>
<p>Here is a very wip, potentially misleading clip :P</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/tPI9FeThDbc" frameborder="0" allowfullscreen=""></iframe>
<p>Next, I’ll be adding jobs to build the per-zone physics data. It won’t be the code that ships, but like this, it will give me some insights into how the final system might behave.</p>
<p>Seeya folks!</p>
TaleSpire Dev Log 2012020-07-12T21:07:32+00:00http://www.techsnuffle.com/2020/07/12/talespire-dev-log-201<p>Hey again folks.</p>
<p>Saturday’s work went well. I spent most of it reviewing the current code and double-checking the tile spawning implementation. We’ve put out a bunch of fixes since the beta release, and so I wanted to be sure the code still matched my design notes from earlier in the development. Luckily it did, and working through all the code paths gave me a much clearer idea of what needed doing first.</p>
<p>It looks like I can get something working without having to handle uncommitted changes[0] at the same time. This means the response time of build actions will be unacceptable (as it requires a round trip to the session host), but it will let me tackle the tooling and copy/paste before having to deal with that additional complexity.</p>
<p>That’s all for now.</p>
<p>Seeya</p>
<p>[0] In TaleSpire, in order to keep network traffic low enough to be manageable, we send ‘operations’ over the network. These are instructions telling the game what to change on each person’s copy of the board. For this to work, all operations must be applied in the correct order. However, games feel bad unless changes happen as soon as you request them, so we keep track of changes that have not been given their final order yet. These are what we call ‘uncommitted’ changes.</p>
TaleSpire Dev Log 2002020-07-10T21:57:52+00:00http://www.techsnuffle.com/2020/07/10/talespire-dev-log-200<p>Heya folks,</p>
<p>On Thursday, I continued work on the batching code. I got a first approximation written[0], but upon trying to hook it into the existing setup, I noticed that disabling the current system is not trivial. In fact, I had a choice between working a bunch for a test that wouldn’t teach me much[1] or just plowing into the real work of replacing the current system. I chose the latter.</p>
<p>This means that the goal I set for myself of “something rendering by the weekend” was not feasible, and I shouldn’t push myself unnecessarily. The task of replacing the system touches a lot of code and requires a different mental approach.</p>
<p>So instead of fretting Friday, I decided I’d take it off and just work Saturday instead. One of the blessings of working for yourself :)</p>
<p>Because of that, I have no news for today. I also expect that the updates from me for the next week will be variations of “Replacing part X of the old tile system”. However I will try to keep the posts coming in case anything interesting crops up.</p>
<p>Until then,
Bye!</p>
<blockquote>
<p>[0] Approximation as it just positions all the parts of the tile at the tile’s origin. So it’ll be a mess but will show if things are working. I did this as the math requires a focus to make sure I don’t screw it up and should be taken slowly.</p>
</blockquote>
<blockquote>
<p>[1] I have already made experiments to learn how the BatchRendererGroup behaves, and the process of writing the batching code had given me a good deal of the context I had hoped to get from the exercise.</p>
</blockquote>
TaleSpire Dev Log 1992020-07-09T01:20:28+00:00http://www.techsnuffle.com/2020/07/09/talespire-dev-log-199<p>It’s the summer holidays over here, so we’ll see a little less of Ree this week as he takes some well-earned breaks. However, you still have to put up with me :)</p>
<p>Today I’ve been looking at writing the batching for the rendering as we move away from Unity’s GameObjects. It’s slow going, but I should have something running before the weekend. This version won’t have GPU-culling, but it will let me progress on the rest of the systems that need rewriting. Once this is working, Ree and I can work out the bare essentials we need for the animation system the tiles will use.</p>
<p>I also had another brief look into Unity’s new physics engine, and I’m reasonably confident we’ll be able to drive that efficiently. More on that another time.</p>
<p>Back tomorrow with more very incremental news :)</p>
<p>Seeya</p>
TaleSpire Dev Log 1982020-07-07T23:40:29+00:00http://www.techsnuffle.com/2020/07/07/talespire-dev-log-198<p>Just signing off for the night, so not gonna write much.</p>
<p>I got the server patch tested and deployed. It will have no effect on the current TaleSpire release, but the next patch will start using it.</p>
<p>Right. Sleep is in order.
Bye!</p>
TaleSpire Dev Log 1972020-07-07T02:43:56+00:00http://www.techsnuffle.com/2020/07/07/talespire-dev-log-197<p>Hey folks, I hope your week has started well.</p>
<p>Ree has been busy taking care of business and working on rulers, and I’ve turned my attention to the backend.</p>
<p>I really want to get TS streaming zones rather than downloading the whole board, but before I do that, I want to look into any issues in the current system.</p>
<p>First orders of business are finishing the changes to session management and removing a bunch of old code. We have had quite a few updates to the server code since launch, and the old code has a mental cost when working on the system.</p>
<p>Most of this is relatively straightforward. However, it requires a lot of care and double-checking to make sure the updates never interrupt service.</p>
<p>I’ll probably push the server update tomorrow with is the safest part (adding new code). I can then update TaleSpire to use these new entry-points, and then soon we’ll be in a good position to remove the old ones.</p>
<p>Until next time.</p>
<p>Peace.</p>
TaleSpire Dev Log 1962020-07-02T19:42:43+00:00http://www.techsnuffle.com/2020/07/02/talespire-dev-log-196<p>Today we got another patch out, which felt good. Since then, work has continued of visuals for the rulers, and I’ve made a couple of tweaks I can show.</p>
<p>First, creatures have their own panel now. A little while back, we made placing creatures not transition to building mode, and this continues on that path of changes we want to the building experience.</p>
<p><img src="/assets/videos/creaturePanel0.gif" alt="creature panel wip" /></p>
<p>I’ve also patched the backend and thrown in test UI for units. This will let you set the default units for the campaign. In time we will likely add ways to add multiple units (for larger distances), and we’ve also thought about per board settings so your overworld could be measured in a more appropriate scale. That is all still up in the air, though, and the UI is just so we can make progress with the rulers.</p>
<p><img src="/assets/images/units0.png" alt="units wip" /></p>
<p>That’s all for now.</p>
<p>Peace.</p>
TaleSpire Dev Log 1952020-06-30T23:28:48+00:00http://www.techsnuffle.com/2020/06/30/talespire-dev-log-195<p>Hi all. Work has gone well today.</p>
<p>Ree’s been working across a few tasks include starting on the proper ruler UX.</p>
<p>I spent the first half the day working through the list of feature requests categorizing them. I’m hoping to put out a more organized list by the end of play tomorrow so that you can have a peek at it before the Dev Stream this Friday.</p>
<p>That work is super tedious, though, so I took a break and added support for customizing the names of the four stats used by all creatures. It looks like this now:</p>
<p><img src="/assets/videos/stats0.gif" alt="yay stats" /></p>
<p>This hasn’t yet been merged into master, but the backend has already been patched to support it. While I was doing that, I added the flag that indicates flying to all creatures. So we can now persist this info once we ship that feature.</p>
<p>That’s the lot for today.</p>
<p>Seeya next time</p>
TaleSpire Dev Log 1942020-06-25T00:05:40+00:00http://www.techsnuffle.com/2020/06/25/talespire-dev-log-194<blockquote>
<p>WARNING: The visuals for, and even the kinds of, rulers in this log are 100% temporary. The point is to work out a possible structure for the code and network sync. Drawing any data or conclusions from the clips presented in this update is meaningless.</p>
</blockquote>
<p>Heya folks, today Ree has been prototyping props, and I’m going to ramble below about what I’ve been noodling with.</p>
<p>After seeing a conversation in the discord about area-of-effect, I got a strong desire to hack on rulers again. My goal was to make a prototype of one possible mechanism which could give us fairly a simple way to define rulers.</p>
<p>I wanted the ability to support simple point-to-point or volume rulers, but also ones with some additional adjustable parameter (like a cone area-of-effect)</p>
<p>The approach is as follows:</p>
<ul>
<li>A ruler is a container which holds some different visuals for rulers (which I’ll call presentations in this update)</li>
<li>The ruler is a simple state machine with four states
<ul>
<li>Placing the first point</li>
<li>Placing the second point</li>
<li>Adjusting the ‘parameter’ (optional step)</li>
<li>Done</li>
</ul>
</li>
<li>The ruler might be visible to other players. In which case it handles the sync</li>
<li>A presentation does not have to support a parameter</li>
<li>Presentations can be cycled at any point before the ‘Done’ state</li>
</ul>
<p>The ‘parameter’ is one part that might seem a bit vague right now. We should talk about it, but first, have a peek at this clip.</p>
<p><img src="/assets/videos/ruler1.gif" alt="testing rulers kinds" /></p>
<p>Unlike the sphere and line rulers, the circle area-of-effect presentation had an additional step after placing the center and defining the radius. It allowed the user to show a slice of the circle. To make this, I needed the third point to be on a plane defined by the first and second points. From the volume selection tool, I had already seen that it was handy to define a plane to raycast against, so I decided to add that to the parameter. The optional parameter is either a point on the board (just like the first two), or it can define a plane to constrain the point. Simple stuff but might allow some useful tools.</p>
<p>So this prototype is done we can kick the tires and see if it’s in the right ballpark. It’s totally fine if we throw it all away, or gut it and do something similar. The main thing is that it’s something tactile in-game that we can play around with.</p>
<p>With this itch scratched, I’ve jumped branch to look at the session/chat panel prototype again.</p>
<p>That’s all for now.
Seeya</p>
<p>p.s. Here is a little clip showing sync working between two copies of TaleSpire.</p>
<p><img src="/assets/videos/ruler0.gif" alt="testing ruler sync" /></p>
TaleSpire Dev Log 1932020-06-21T16:31:32+00:00http://www.techsnuffle.com/2020/06/21/talespire-dev-log-193<p>Hi folks.</p>
<p>I’ve been poking a little at the occlusion culling, so I thought I’d show something I found handy.</p>
<p>We are not moving to Unity’s scriptable rendering pipeline yet. This means much of the work I’m doing right now is finding out how we should hook into Unity’s traditional rendering stack.</p>
<p>I really like <a href="https://docs.unity3d.com/ScriptReference/Rendering.BatchRendererGroup.html">BatchRendererGroup</a> as it lets us set up batches and not have to update them per frame[0]. We can then map the batch-id to a specific zone and kind of asset. This now means that any active camera (or one that is told to render) will render the things we have submitted in those batches.</p>
<p>We really want to use lower-poly meshes for the shadows and occlusion-culling, so what we should put in those batches is actually the low poly-meshes (which we are going to call occlusion-meshes from now on). However, we don’t want to render the mesh itself to the final scene, we just want it to cast shadows in the main view and populate a depth buffer for us to use later.</p>
<p>Now those used to Unity might think of layers. You can tag objects with a specific layer and then tell a camera to only render particular layers. The issue with this is that if you filter out an object, you also can’t see it’s shadows, so that won’t do. Next, we could see that when <a href="https://docs.unity3d.com/ScriptReference/Rendering.BatchRendererGroup.AddBatch.html">adding batches to the BatchRendererGroup</a> we can set the <code class="language-plaintext highlighter-rouge">ShadowCastingMode</code> to <code class="language-plaintext highlighter-rouge">ShadowsOnly</code>. Awesome, now we only get shadows in the main view… however, we then don’t get any depth information in the depth buffer. Damn. What ended up working was the following:</p>
<ul>
<li>Make a shader with <em>only</em> a <code class="language-plaintext highlighter-rouge">ShadowCaster</code> pass [1].</li>
<li>Make a material using this, and use that in <code class="language-plaintext highlighter-rouge">AddBatch</code>.</li>
<li>Render the scene from the same orientation using a camera with a replacement shader and with a depth RenderTexture as the target.</li>
</ul>
<p>What’s nice about this is that lights will now use our occlusion-meshes, but those meshes won’t show up in the final render.</p>
<p>With the depth buffer available, we now generate the depth-chain, as mentioned in a previous dev-log. We then have everything we need to get into the meat bit of the occlusion culling. Once we know what to draw, we will use <a href="https://docs.unity3d.com/ScriptReference/Graphics.DrawMeshInstancedIndirect.html">DrawMeshInstancedIndirect</a> to submit the high-poly meshes for rendering. At this point, we make sure to set <code class="language-plaintext highlighter-rouge">receiveShadows</code> to false and the <code class="language-plaintext highlighter-rouge">ShadowCastingMode</code> to <code class="language-plaintext highlighter-rouge">Off</code>. This lets us receive the shadows we computed earlier without asking Unity to compute more from these more detailed meshes.</p>
<p>Right now, all the tests I’m doing are just with randomly placed spheres, so screenshots are a little meaningless, however, once we get into the using real assets, I’ll start showing more.</p>
<p>Next, I need to work out a little mistake I’ve created in the depth-chain, refactor the tests to be a little easier to handle, and then get into the culling compute shader. Once I have this first test working, I’ll need to do a lot of planning to work out how we want to handle the data in the TaleSpire. The goal is to find a balance where we do as little per-frame work and as small per-frame GPU uploads as possible.</p>
<p>Until next time,
Peace.</p>
<p>[0] and handle culling ourselves which is handy too
[1] here is the shader we used in our recent tests</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// Very minimal
Shader "OccluderShadowCaster"
{
SubShader
{
Pass
{
Tags {"LightMode"="ShadowCaster"}
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#pragma multi_compile_shadowcaster
#pragma multi_compile_instancing
#include "UnityCG.cginc"
struct appdata
{
UNITY_VERTEX_INPUT_INSTANCE_ID
float4 vertex : POSITION;
float3 normal : NORMAL;
};
struct v2f {
V2F_SHADOW_CASTER;
};
v2f vert(appdata v)
{
v2f o;
UNITY_SETUP_INSTANCE_ID(v);
TRANSFER_SHADOW_CASTER_NORMALOFFSET(o)
return o;
}
float4 frag(v2f i) : SV_Target
{
SHADOW_CASTER_FRAGMENT(i)
}
ENDCG
}
}
}
</code></pre></div></div>
TaleSpire Dev Log 1922020-06-18T01:53:15+00:00http://www.techsnuffle.com/2020/06/18/talespire-dev-log-192<p>A small ‘still working’ update for the last couple of days. Ree has continued with terrain experiments and business work. I’ve been playing around with Unity’s <code class="language-plaintext highlighter-rouge">BatchRendererGroup</code> type to learn what we have in our toolbox as we start to rewrite how we handle the rendering of tiles and props.</p>
<p>More coming soon.</p>
<p>Ciao</p>
TaleSpire Dev Log 1912020-06-15T23:13:04+00:00http://www.techsnuffle.com/2020/06/15/talespire-dev-log-191<p>Hi folks.</p>
<p>Today, some of the things Ree and I have been working on don’t make for great writeups. He was on business tasks, which included working on our GDPR documentation, and I was finishing a feature we’ll be shipping in a few days if all goes well.</p>
<p>I’m not ready to say what it is as, although it’s tiny, we want to ship it alongside an upcoming asset update. You’ll just have to wait :p</p>
<p>After I finished off that feature, I started work on the in-game chat system. For messages to players within the same board, it’s easiest to use the RPC calls of the networking library we use. However, for campaign-wide messages, we will use our server to relay the message to all connections which are subscribed to that campaign. This is likely to get changed up depending on what we can do with WebRTC down the line.</p>
<p>I wrote the code to handle this on the backend, and while I was at it, I added some code to handle the upcoming bookmarks feature.</p>
<p>With that done, I started mocking up the UI for the chat panel. I’m no designer, so please excuse the ugliness, but a rough shape is forming at least.</p>
<p><img src="/assets/images/chat0.png" alt="uuuugly" /></p>
<p>The little boxes underneath the message text box are attachment slots. I think we will allow you to drop references to creatures, locations in the board, dice rolls, and other stuff and send them along with your message.</p>
<p>By the time this is ready, it will replace the session history, the messages you had there will now be in this log, which will support tag-based filtering.</p>
<p>You will be able to target your messages at any of the following:</p>
<ul>
<li>Everyone in the board</li>
<li>Everyone in the campaign</li>
<li>Everyone in a specific party</li>
<li>A specific player</li>
</ul>
<p>My current thinking is that you’ll usually use the drop-down to pick the target of the message, but if your message starts with <code class="language-plaintext highlighter-rouge">@<someone></code>, it will as if you switched the target to that player, sent your message, and switch the target back. This is handy in cases when you need to send something very quickly, like if you are co-ordinating a mutiny with another player.</p>
<p>In time we also will want to support language proficiency in the chat. This would mean you can read messages that arrive in languages your creature understands (e.g. elvish), but the text will be garbled if it doesn’t support that. Obviously, this needs more information to be attached to each creature, so this is more likely to happen closer to when we add rule-system support. It doesn’t feel like something that should be prioritized over other features, though.</p>
<p>Time to call it a night.</p>
<p>Seeya!</p>
TaleSpire Dev Log 1902020-06-14T17:08:38+00:00http://www.techsnuffle.com/2020/06/14/talespire-dev-log-190<p>This week has been dedicated to research and planning, and it’s been fantastic.</p>
<p>We have a lot of things that need to be done by Early Access. Between our plans, user requests, and polish, we definitely have our work cut out for us.</p>
<p>Luckily some features are also polish, probably none more so than performance. We’ve known for a long time that we would need to take some big steps to tackle performance issues. TaleSpire is a user-generated-content game, and these have some challenges which are not present in many games. These typically revolve around the fact that the scene can change dramatically at any time and so you cannot use techniques that require precalculating lots of things at build time.</p>
<p>There are tools which we’ve mentioned we wanted to look into, the big two being Unity’s new ECS and Gpu occlusion culling.</p>
<h3 id="ecs">ECS</h3>
<p>We started by performing some quick and dirty tests with entities by adding an entity conversion component to the tiles being spawned. We happily noted that in our first scene, the time spent on rendering noticeably dropped. There were some cases, however, where it didn’t seem so cut and dry, and so we did some digging. It certainly feels like there has been a significant performance regression in the hybrid-renderer since the ‘Mega City’ demo unity was showing off last year. Based on the code, this seems centered around material properties. This was a bit disheartening but did teach me about the <a href="https://docs.unity3d.com/ScriptReference/Rendering.BatchRendererGroup.html">BatchRenderGroup</a> class, which is likely going to become very important to us this year.</p>
<blockquote>
<p>Now, before folks get up in arms against Unity, this system is in preview, it’s not shipping quality and the version in unity 2020 apparently already has significant performance improvements. This is just how software development is. We are just inside the sausage factory here.</p>
</blockquote>
<p>Now it’s very true that, given the same engineers, general systems will often be slower than one made for a specific task. And from my reading, it seems that, in TaleSpire’s case, we can have opportunities to cache more heavily and get speedups based on assumptions we can make about our own data usage. We will dig into this over the coming months.</p>
<p>Another similar system is Unity’s new stateless physics system. From our tests, we see very promising speedups compared to the current system. Once again, I believe there are places where we can feed the system in a way that can limit the impact of some tasks they have to do when starting the physics step for that frame.</p>
<p>Now, changing the physics system means changing everything that uses the old system. That means board building tools, dice, and creatures, just to name three. And, let’s face it, those three are a lot of what TaleSpire is :D We are going to have to work very aggressively on this to get TS back into a playable state as quickly as possible, but it’s still going to take time.</p>
<p>If we can achieve both of these changes, we can eliminate the time it takes to spawn tiles. This means that we can remove the progressive-tile-load system, which is one of the most complicated parts of TaleSpire. Its whole job was to mitigate the fact that Unity’s previous object-oriented approach resulted in us not being able to spawn many objects per frame. This meant that what you saw, and the actual state of the board data are necessarily out of sync. We managed that, but, as mentioned, making that robust took a lot of work. I’d love to remove it.</p>
<p>So that’s all on the entities side. I’ve left out a lot of details and complexity, but I’m very optimistic that we can get see some big wins here.</p>
<h3 id="gpu-occlusion-culling">Gpu Occlusion Culling</h3>
<p>Occlusion culling is the act of not drawing things hidden from view (occluded) by other objects in the scene. It’s a very powerful concept but a reasonably complicated one, which often requires a lot of number crunching. Many games do this ‘offline,’ the level is analyzed by a program that precalculates what is visible from where. This would be done during builds of the levels and then would be used at runtime. As mentioned above, offline approaches are not ones TaleSpire can use, so what can we do? More recently, as GPUs have improved, people have worked out ways to achieve realtime versions of this on the GPU. It is very common in TaleSpire that people decorate the insides of builds, and none of those tiles usually need to be drawn when the camera is outside the building. In short, we might get substantial wins from occlusion culling.</p>
<p>I’d started implementing a GPU occlusion culler outside Unity before, but I’d lacked the Unity knowledge to know how to do it there without writing my own render pipeline. After seeing BatchRenderGroup, a bunch of stuff clicked, and now I’m pretty sure I know how to do this. As with my previous experiments, I’m basing my work on <a href="https://interplayoflight.wordpress.com/2017/11/15/experiments-in-gpu-based-occlusion-culling/">this blog post</a>, and a yesterday I wrote a MonoBehaviour that computes the hierarchical depth (Hi-Z) mip chain for a camera.</p>
<p>I’ve got a few things on my todo list for this week, but as soon as possible, I want to write a working test version of this so I can explore how far we can take this.</p>
<p>One side note from this is that we will be moving to use low poly meshes for all occluders. This means line-of-sight, fog-of-war, and shadows should all get a speed boost.</p>
<h3 id="games-systems">Games Systems</h3>
<p>As well as the research above, we spent a good deal of time exploring ideas for the following game systems.</p>
<h3 id="terrainwater">Terrain/Water</h3>
<p>We have an initial idea for a system that we hope will balance a decent game experience with compact data representation. No details on it today, but Ree is currently prototyping to see if the idea will feel good enough for a v1 in the Early Access.</p>
<h4 id="props-system">Props System</h4>
<p>The prop system has been near to releasing at least 3 times over the last 2 years. We sat down and started by discussing the implementation details (as it’s going to have to be made to work with all the changes mentioned above). However, as we talked, we grew increasingly concerned that the attachment-point based approach might not be good enough for handling the kinds of cases we had seen watching people build. We went back and forth on this for a bit and realized that we needed to prototype the tooling and see what it would end up feeling like. The last 20% of making something often takes 80% of the time, so it’s far better that we convince ourselves now than to build the wrong thing.</p>
<h4 id="locking-the-unity-version">Locking the Unity version</h4>
<p>Our modding system relies on Unity’s prefab format. Once people can make mods, we need to make sure that we don’t break mods by upgrading to a Unity version, which uses an updated prefab format. To us, this means we need to pick a Unity version and stick with it for a long time. Naturally, that means we can’t take advantage of the newer Unity feature or fixes, so we have to be sure that the version we have is something we can stick with. We’ll keep you posted on this too.</p>
<h3 id="wrapping-up">Wrapping Up</h3>
<p>I’d like to talk on stream a bit about this stuff and a selection of the feature-requests. We won’t have timelines for the requests, but we can at least address what we see going into TS and what won’t. At that time, I’ll release the list of feature requests. I’m not going to do it before as the notes need cleaning-up and that work takes a lot of time.</p>
<p>Alright, that’s it for now,</p>
<p>Seeya tomorrow!</p>
TaleSpire Dev Log 1892020-06-11T00:42:26+00:00http://www.techsnuffle.com/2020/06/11/talespire-dev-log-189<p>Hi all.</p>
<p>Today I finished collating all the entries in the #feature-requests channel on the TaleSpire discord. With that done, @Ree and I went through them to work out what we wanted to do for each. What was cool was that there weren’t many major surprises. Most were about systems that we still have plenty of plans around improving. I’ll probably need to organize a dev-stream at some point so we can go through some of them and talk about the plans for TaleSpire in general. However, I’m not scheduling that today.</p>
<p>The rest of this week will be planning and prototypes around other major systems we need. The big ones being terrain, water, and the rendering changes we need to get the performance we all crave :)</p>
<p>Hope this finds you well. Back with more tomorrow</p>
TaleSprie Dev Log 1882020-06-09T02:02:37+00:00http://www.techsnuffle.com/2020/06/09/talesprie-dev-log-188<p>Evening all.</p>
<p>Work goes well. @Ree has been looking back into the emote system work he did previously. Once that is up to scratch, we can get the data & sync side worked out, and then hopefully, it’s just testing and tweaks before we ship.</p>
<p>Today I started with a few hours of going through all the posts in the #feature-request channel in discord, making the big list of things to discuss internally. Starting on Wednesday, @Ree and I are taking some days to plan and hack on some experiments to set the direction for what we need to achieve over the coming year.</p>
<p>Inspired by an item on that list, I’ve been revisiting <a href="https://bouncyrock.com/news/articles/talespire-dev-log-119">my old NDI tests</a> to see how much work it would take to get this shipped experimentally. I can pick the source from NDI and get texture, but I’m currently fighting the UI side. Once that is doing what I want, I’ll need to handle some strangeness I saw when the call reconnects before looking at sync. I’m not yet sure how the sources are named on each client, so I’m not sure how to synchronize the feed information for each client. Regardless we’ll work it out or, if it takes too long, it’ll go back on the shelf for now.</p>
<p>That’s all for today. I’ll put out a bugfix tomorrow as today I was made aware of a bug that was introduced to line-of-sight in the last release.</p>
<p>Goodnight!</p>
TaleSpire Dev Log 1872020-06-02T17:19:15+00:00http://www.techsnuffle.com/2020/06/02/talespire-dev-log-187<p>A quick update tonight.</p>
<p>Work continues of the fog of war. I’ve been fixing bugs and implementing the code that handles cleaning up fog-of-war jobs when transitioning boards. I still think there are some memory bugs, but it’s all going in the right direction.</p>
<p>Ree’s work on the keyboard movement has gone well. We’ve found a couple of smalls bugs in the line-of-sight and creature scaling code, so we’ll get those fixed up asap.</p>
<p>Peace</p>
<p>p.s A quick video showing what the 30 unit radius of the fog-of-war update looks like</p>
<video controls="" src="/assets/videos/fow1.mp4" type="video/mp4" width="620"></video>
TaleSpire Dev Log 1862020-06-02T00:58:13+00:00http://www.techsnuffle.com/2020/06/02/talespire-dev-log-186<p>TLDR: The first prototype of 3D fog of war started working about an hour ago. Check it out:</p>
<video controls="" src="/assets/videos/fow0.mp4" type="video/mp4" width="620"></video>
<blockquote>
<p>BIG OL’ CAVEAT: This is not the final mesh or shader used on the fog. This just shows that the raw transforms <em>can</em> work.</p>
</blockquote>
<p>A lot of the leg work over the last days has been managing the pools of cubemaps, buffers and such used in the fog of war system so that we never allocate more often than we absolutely have to. We also have been making sure that no step blocks for longer than necessary and that we have something we can easily tune.</p>
<p>So here is the very rough rundown:</p>
<ul>
<li>when you place a creature that you control, the scene (for a 30 unit radius) is rendered into a cubemap holding creature ids and distances from the observer.</li>
<li>the ids are used by the line of sight system to accurately determine what creatures are visible to the creature you just placed.</li>
<li>the cubemap (and some other data) is kicked over to the fog-of-war system that works out every cell (a 1x1x1 unit volume) visible for the observer.</li>
<li>It packs this data into a buffer that is then applied to the zone’s fog-mask.</li>
<li>The updated fog mask is handed over to the mesher which generates a relatively low poly mesh for the fog and sends gives it to the zone to display</li>
</ul>
<p>Multiple of these can be done at once. The line-of-sight and fog-of-war updates that rely on processing the data in the cubemap are all on the GPU, and any other step that does any kind of data processing is done in jobs dispatched over multiple cores. I’ve not profiled this yet, but it’s feeling ok, and we have the tools to make this quick.</p>
<p>Right now, I’m just stoked this is starting to work. Tomorrow I’ll be bug-fixing, and after that, I’ll start work on the network sync for these updates. I’ll write a dev-log on that problem as it’s a fun one too.</p>
<p>We are also getting closer to the creature scaling update. We found a bug in the keyboard movement and we need to get fixed before we ship. We’ll keep you posted on that.</p>
<p>For now, I’m gonna go poke this system some more :)</p>
<p>Seeya!</p>
<p>p.s. Here is a pic of a fog mesh. You can see that the mesher does an ok job of cutting down the number of polys from the worst case.</p>
<p><img src="/assets/videos/fow0.png" alt="pew" /></p>
TaleSpire Dev Log 1852020-05-27T23:41:49+00:00http://www.techsnuffle.com/2020/05/27/talespire-dev-log-185<p>Heya folks,</p>
<p>Today has been spent testing and fixing the backend changes for creature scaling and hooking the creature scale feature into the data model.</p>
<p>The scaling is feeling great, Ree’s done great work, and I’m very excited to see this ship. There are still bugs to be fixed, so we’ll keep working on those.</p>
<p>The server side has now been patched, and so everything is ready to go.</p>
<p>The first thing I’ll be doing tomorrow is adding some code to detect mismatched versions of the client. This will stop a person with an older version of the client connecting to a session hosted by someone with a newer version (and vice versa). This is important as the board sync format can change and are not compatible between different versions of the client [0].</p>
<p>That’s the lot for today</p>
<p>Ciao</p>
<p>[0] Please note that this format is not the same as the board persistence format for which there are format upgrade methods written.</p>
TaleSpire Dev Log 1842020-05-26T23:01:53+00:00http://www.techsnuffle.com/2020/05/26/talespire-dev-log-184<p>Hi again :)</p>
<p>Today went reasonably well, although it was mostly getting my new laptop set up.</p>
<p>I finally tried out i3 and decided that, right now, it’s not for me, and I’ve gone back to using stumpwm. I’ll probably spend more time getting used to i3’s approach to splitting in the future. It’s stacks and workspaces did seem very cool although I should experiment with stumpwm’s groups feature first.</p>
<p>Getting my server dev environment set up was pretty painless (yay docker), although I currently have an issue where TaleSpire isn’t uploading boards to my local minio server. I’m not sure what the issue is, so I’ll need to do more testing tomorrow. I’ve prepared the patches for the DB and erlang server, so I now need to test TS with these changes. With that done, I’ll be able to push to production, and then we’ll be ready for creature scaling.</p>
<p>Ree’s been working hard on fixing all manner of issues relating to creature control at low (and high) fps. Getting that stable is a heck of a project, but it’s paying off.</p>
<p>To follow up on the log from earlier today, it does seem that the latest patch does fix the SSE4 issue, which is great! We’ll be able to close a bunch of tickets from github real soon :)</p>
<p>Have a good night folks,
Back with more tomorrow.</p>
TaleSpire Dev Log 1832020-05-26T12:59:08+00:00http://www.techsnuffle.com/2020/05/26/talespire-dev-log-183<p>Heya folks, time for another log.</p>
<p>We have a content update dropping soon, but also in there is a potential fix for those who have been unable to play due to their machine not supporting SSE4. This limitation was due to older versions of the Burst compiler not supporting earlier SIMD instructions. We had wanted to upgrade the Burst compiler package previously, but we found a bug in that package that blocked us. We (reported it to Unity not long back)[https://issuetracker.unity3d.com/issues/the-sizeof-of-struct-with-structlayout-returns-a-smaller-value-in-the-burst-compiled-code-compared-to-the-mono-compiled-code] and they have already shipped a fix! With the packages updated, we should now support fallback to SSE2 (which every x64 machine supports).</p>
<p>My laptop, after holding on for nine years, finally started breaking the other day. My new one has just arrived. That means I’m going to lose at least a day to swearing at windows for being such a massive, invasive, pain in the ass and setting up Linux so I can get server work done in a more agreeable environment.</p>
<p>I’ve also started on the server changes to support creature scale. These are going well, and I will begin integrating the changes on the client-side soon.</p>
<p>Alright, that’ll do for now. Back to the laptop for me.</p>
<p>Ciao</p>
TaleSpire Dev Log 1822020-05-22T15:49:54+00:00http://www.techsnuffle.com/2020/05/22/talespire-dev-log-182<p>A short one this time. Between two Norwegian national holidays and my own brain, I’ve been struggling with focus. Progress has slowly continued on the fog of war. I’ve written the compute shader, but I still need to do some plumbing, so I’ve nothing gif-worthy yet.</p>
<p>Ree’s work on the creature controller is wrapping up, so that will ship soon. There are a slew of fixes, and the work includes the bulk of the work for creature scaling. So once the first update has shipped (hopefully early next week), I’ll jump tasks to implement the sync and persistence for scaling.</p>
<p>The art team has also been working away, so you can expect a new asset update soon!</p>
<p>I’m going to spend the rest of today around the community, so if you are around, seeya there.</p>
TaleSpire Dev Log 1812020-05-20T02:17:14+00:00http://www.techsnuffle.com/2020/05/20/talespire-dev-log-181<p>Hi folks, another day of slow but steady progress on the fog of war. What I did today was hook up the code that queues up mesher jobs when the underlying fog data changes. I can now select a region, press a key, and the fog volume data for all the zones in the region is updated. The presentation for those zones then enqueues a job that reads the volume data and generates a minecraft-style mesh that will (in time) be used for fog. As, in this test, the selection tools work with cuboids, the fog only follows that shape, but we can see that the resulting mesh is reasonable (lower poly compared to the worst case), and I can see that the jobs are being run in parallel. So far, so good.</p>
<p><img src="assets/images/fogMesh0.png" alt="mesh" /></p>
<p>Now, this is done I can write the code that takes the cubemaps captured from the line-of-sight system that handles hiding creatures, and passes them to a new compute-shader which will make an update mask of what areas are visible. I’ll be doing this on the GPU as:</p>
<ul>
<li>The data I need is already on the GPU which means no having to shift it around</li>
<li>The approach I’ll be trying is trivial to parallelize without divergence and so should be very suited to running there.</li>
</ul>
<p>From the CPU side, we will then wait on a GPU-fence (which tells us when the compute-shader has finished), read the result from the compute-buffer, and apply it to the fog data in the zone. This change will then trigger the code we wrote today and we should see the new mesh which will hopefully have removed all parts that were in view. It will absolutely be rough, to begin with, but with the full round trip written, we can move onto playing with it and making it better.</p>
<p>I’ll go into more detail on the approach once I’ve got it working as it should be pretty simple.</p>
<p>Alright, seeya around.</p>
TaleSpire Dev Log 1802020-05-19T01:19:51+00:00http://www.techsnuffle.com/2020/05/19/talespire-dev-log-180<p>Hi folks, today I’ve been poking around with fog of war and working out how the updates are going to be spread over multiple frames. I’m going to keep working on this so I can select areas and apply fog to them. For now, the mesh generated will just be a big, grey, minecraft-looking mesh as we don’t need to worry about the visuals for a while yet.</p>
<p>Ree’s been working on business things today, and his work on the creature-controller has gone well too. There are still some things to work out with creature scaling over various framerates, but it’s getting closer.</p>
<p>Back soon with more!</p>
TaleSpire Dev Log 1792020-05-15T18:04:21+00:00http://www.techsnuffle.com/2020/05/15/talespire-dev-log-179<p>Nothing substantial today, I’m afraid. That last day or so, I’ve had something like writer’s block for coding, so I’ve moved my weekend a bit sooner.</p>
<p>I had started looking into fog-of-war, but I’ve got nothing new to report yet.</p>
<p>When I’m rested up, I’ll dig back into a few tile related bugs that have been reported this week, and then I’ll get back to the fog of war.</p>
<p>Have a good one folks.</p>
TaleSpire Dev Log 1782020-05-13T15:38:48+00:00http://www.techsnuffle.com/2020/05/13/talespire-dev-log-178<p>Heya folks,</p>
<p>After shipping the update yesterday, I reviewed all the usages of the data from the <code class="language-plaintext highlighter-rouge">boardAsset</code> files in TaleSpire. There were clear patterns in the usage that will allow us to pick better data layouts and divide the data that is needed in jobs from the data, which isn’t. This description is pretty vague, but basically, it’s good news :)</p>
<p>I’m going to play a little with non-burst-compiled jobs in Unity as they are going to be useful in the future when I need to do simple processing on non-blittable types, but don’t want to have to introduce a sync point.</p>
<p>I’ve also started looking back into a bug, which forced me to hold off from fog-of-war before the beta release. I had a case where rendering to a cubemap would result in the top and bottom faces being flipped. I didn’t report it at the time (which was dumb on my part), so I spent the first part of today reproducing the issue and submitting the issue to Unity for review.</p>
<p>You can find the repro that shows the issue <a href="https://github.com/Bouncyrock/potential-cubemap-descriptor-bug">here</a>. Luckily it seems this issue does not occur when using a different <code class="language-plaintext highlighter-rouge">RenderTexture</code> constructor as reported by this lovely person <a href="http://answers.unity.com/answers/1709964/view.html">here</a>. With this knowledge, I can dig back into this and see if I can make more progress on fog-of-war again.</p>
<p>I can also give an update that the last bug we reported to Unity has been successfully reproduced internally, so odds are good that we will see progress on that in the future.</p>
<p>That’s all for now. I’ll probably make a more detailed post on the fog of war implementation once I have some more news.</p>
<p>Peace</p>
TaleSpire Dev Log 1772020-05-12T02:23:10+00:00http://www.techsnuffle.com/2020/05/12/talespire-dev-log-177<p>I’ve been a little quiet today, but I’ve been around. I’ve mainly been testing and re-checking the code that handles board format upgrades. I didn’t push out an update today as, during testing, I saw some cases where GM-blocks showed up incorrectly. I’m pretty sure these are pre-existing bugs, but it’s better to delay the release and fix them, rather than take the risk.</p>
<p>Tomorrow I’ll carry on testing the board format upgrading and then put out the release. Initially, the release was to add fog control to atmospheres but it now also includes:</p>
<ul>
<li>A fix which means that when you return to boards, it (finally) remembers where your camera was and where it was facing</li>
<li>A fix for one instance of an error where, on rejoining a board that failed to load, you got a ‘failed to correctly assign internal id’ error.</li>
</ul>
<p>And of course the above gm-block fixes.</p>
<p>After this, I’m aiming to focus on updating the asset format used by TaleSpire. I’ve been talking about doing this for about a year, but other things were always more important. The big goals are to improve load times and to make the asset data accessible to the job system. This will be huge for me as there is far too much complexity and inefficiency in the asset code due to constantly having to repack data in structures amenable to the job system.</p>
<p>Of course, some community projects have started consuming some of this data and, even though use of that data is not supported, I’d rather not break those. What I think we’ll do is make a portion of the data available as a separate json file. TaleSpire itself won’t use that file, but it’ll be handy for others. I’ll keep you posted on these changes.</p>
<p>Have a great evening folks,</p>
<p>Peace.</p>
TaleSpire Dev Log 1752020-04-29T10:52:33+00:00http://www.techsnuffle.com/2020/04/29/talespire-dev-log-175<p>Yesterday didn’t bring too much new from me. I submitted the potential bug to Unity, pushed out our release, and then spent most of the day catching up with social media and discord posts.</p>
<p>Today I’m going to try staring at the ‘delete tiles coming back on reload/copy’ bug and see if I can make any headway on it. It’s a real nightmare of a bug as it’s hard to reproduce, and it seems to be about something that isn’t happening. It’s not throwing any apparent errors and doesn’t show up until way after the presumed event.
I’m not expecting to find it today, but maybe I can rule out some potential causes.</p>
<p>I hope you have a good day folks!</p>
<p>p.s. A note to people thinking of doing the indy game thing. Do not underestimate the amount of time it takes to write updates, handle support tickets, keep up to date with your community, and do the business stuff. It’s a privilege to have these ‘problems,’ but writing updates will take you longer than coding some features :p
The less of you there are, the more time you’ll each be spending. Shout-out to Ree & Dwarf who’ve handled all the support tickets and business-ness while doing all their other jobs too.</p>
TaleSpire Dev Log 1742020-04-28T00:23:03+00:00http://www.techsnuffle.com/2020/04/28/talespire-dev-log-174<p>Ah, it’s a been a day that reminds you that rarely in programming can you ‘just’ do something.</p>
<p>On Saturday, I wanted to push a patch that fixed a bunch of bugs, one of which was related to an SSE4 requirement in the current build.</p>
<p>The fix should have been simple, SSE2 support has been added in later versions of the Burst compiler, we can <em>just</em> upgrade the package and rebuild.</p>
<p>The package is still in preview, and so there can be bugs. However, sometimes that kind of tradeoff is worth it, especially given that we are still in Beta. If it passed our tests, I’d have been happy.</p>
<p>Alas, we quickly saw issues which only occurred in the Burst compiled jobs. Time to get testing :)</p>
<p>As expected, the commit that caused the issue was the one where we upgraded the packages. We then tested each version of Burst and Unity.Collections to track down the first version, which caused the error.</p>
<p>In the release notes for the earliest problematic version we saw that one change was:</p>
<blockquote>
<p>Improve codegen for structs with explicit layout and overlapping fields.</p>
</blockquote>
<p>This was a great candidate as we had a type that is essentially a c# union. You can write such structs in c# like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[StructLayout(LayoutKind.Explicit)]
public struct Test
{
[FieldOffset(0)] public int A;
[FieldOffset(4)] public Bar B;
[FieldOffset(4)] public Baz C;
}
</code></pre></div></div>
<p>I took our structs and removed parts until I had a simple case that showed the issue. I then moved it to a separate project, re-confirmed the packages and pulled other Unity versions to make sure it was still an issue.</p>
<p>I’ve made <a href="https://github.com/Bouncyrock/potential-burst-sizeof-bug">a repo for the test</a>, and tomorrow I’ll report this as a potential issue to Unity.</p>
<p>Now the take-away from this is not to shit on Unity. This is an under-development build of the package explicitly available for finding these kinds of issues. This is just development.</p>
<p>However, it’s a fun example of how ‘just download it’ is almost never really the case in any significant project.</p>
<p>Hope this finds you well, sorry I’ve not been around the community today, constantly recompiling does not make one that sociable :p</p>
TaleSpire Dev Log 1732020-04-25T02:29:01+00:00http://www.techsnuffle.com/2020/04/25/talespire-dev-log-173<p>Hi folks. Recently, all the dev updates have been patch release notes, so I thought I’d drop in to say hi.</p>
<p>Bug fixing has been intense, and we’ve covered a lot of ground. Over 100 tickets are closed on github, and we have no worries about being able to tackle plenty more. After seemingly fixing one of the significant connection issues the other day, I’ve felt pretty sluggish. Feels like my body telling me to take it easy, so I’ve tried to do that for the last two days.</p>
<p>This evening I started pulling exception logs to see what easy wins I could find there. Where possible, we push the first exception we find to the server, and that can be quite a goldmine. It went pretty well, and the next update is going to have a bunch of random fixes in it.</p>
<p>The more exceptions we can remove, the less spurious or random bugs we will see reported. Some times a bug we find is just the side-effect of some earlier thing breaking. Kind of like if your car flips off the road, it might have had something to do with the wheel that fell off just before :P</p>
<p>The fixes had stuff like:</p>
<ul>
<li>selections still trying to highlight tiles while switching boards</li>
<li>A state-machine for board sync hadn’t registered one of its failure states</li>
<li>UI trying to access stuff that had been disposed while transitioning to the main menu.</li>
</ul>
<p>Yeah, a real mix.</p>
<p>Ree has (amongst other things) been looking into a GPU instancing library and Unity’s new ECS. No conclusions reached yet, but we’ll let ya know once there is news there.</p>
<p>We’ve also switched to using some Unity packages, which are still in preview. This is a risk, as they are still finding bugs, but we’ve done it for a couple of reasons:</p>
<ol>
<li>It allows us to dig into the ECS stuff mentioned above</li>
<li>The preview version of their Burst compiler now seemingly supports fallback to SSE2 instead of requiring SSE4 (which has caused some users issues)</li>
</ol>
<p>The next patch will have these changes so we can see if that helps.</p>
<p>Alright, it’s rather late, so I’m going to get some sleep.</p>
<p>Peace</p>
TaleSpire Dev Log 1722020-04-15T23:44:44+00:00http://www.techsnuffle.com/2020/04/15/talespire-dev-log-172<p>Good-evening once again from the north, it’s been a good day.</p>
<p>I will be pushing a patch tomorrow that seems to fix the bug, which caused certain boards to break with <code class="language-plaintext highlighter-rouge">HardAssert</code> errors. I have also tested some of the broken boards, and 6 of the 7 I tested were able to load again.</p>
<p>Let’s start with caveats. This will not fix all broken boards, tomorrow’s patch only addresses this specific bug, and we don’t yet know if all boards which had the issues are recoverable. If your issue manifested as something other than lots of <code class="language-plaintext highlighter-rouge">HardAssert</code> errors in the logs, then this fix won’t recover your board. However, once this is out, I will start going after other bugs so we will see what can be fixed.</p>
<p>Ok, now the meat.</p>
<p>Many folks were stuck, not being able to get into their boards because of a bug which put this error in the logs:</p>
<blockquote>
<p>HardAssertFailure: HandleAddLayout: layout.AssetCount != _dequeuedAssetData.</p>
</blockquote>
<p>An assert is a check that must be true for the program to continue. In this case, the presentation for a zone (see yesterday’s dev-log for the details on what those things are) had been told it had to spawn a certain number of assets. However, there was not enough asset data in the queue to satisfy that request. Given that there was no sensible thing to do at that point, it threw an exception.</p>
<table>
<tbody>
<tr>
<td>Now, I should have written some code to catch that exception, tell you something had gone wrong, and given uploading the board. However, I forgot to add the check for that code path before the release. Sorry about that :</td>
</tr>
</tbody>
</table>
<p>The first part of the work was what I did yesterday. Removing an optimization that I was concerned could have been hiding the issue due to its complexity. This did work, and this morning, I was able to open a previously broken board.</p>
<p>However, that fix, whilst required, was only part of the issue. The root cause was still there.</p>
<p>TaleSpire has to store a lot of asset info. Each kind of asset has a GUID that identifies it. A GUID is a 16-byte value, which is pretty large and so we don’t want to store more than we have to. One simple thing to do is to group assets of the same kind together; then, we only need one GUID for all of them.</p>
<p>There are other values that can be treated similarly, and so we put these together in a struct called an AssetLayout. Each AssetLayout also has an ID (a uint in this case), which uniquely identifies it in the Zone.</p>
<p>An issue that was occurring was that, when a player deleted some assets that they did not place (or if they were from a freshly loaded board) and then pressed undo, the layouts that were restored were not given new unique Ids. This meant the Zone potentially contained multiple layouts with the same id.</p>
<p>This goes unnoticed at first, but on load of a board, we have to spawn all the assets. The information is sent off to the presentation, and that’s where we hit the issue of layouts not matching the expected numbers of assets, all because our booking had been messed up.</p>
<p>Luckily the undo portion was relatively easy to fix. But we still have broken boards. What we needed was for something to go and clean up the ids of the layouts.</p>
<p>Let’s take a slight diversion. Whenever a place places assets, we add a layout and the asset-data for whatever was just made. Over time, this will result in a lot of small layouts for the same kind of tile. We have to keep them separate at first due to how undo/redo works, but once they are out of the players undo/redo history (for instance, when we save the board), we have an opportunity to defragment this data.</p>
<p>I had not had time to implement this before release, but as luck would have it, this operation is exactly what we need to clean up the layout information in the broken boards.</p>
<p>So an afternoon of code later that is working. For now, I’m running this on load, rather than on save as this will hopefully fix some of the broken boards. However, once the patch has been out for a while, I’ll change it to happen on save instead.</p>
<p>So that’s that! I will review the patch tomorrow, test it with some more boards, and then I’ll get it out to you all via Steam.</p>
<p>Huge thanks to Skye, lollypopunk, Al McWhiggin, Shmuky, and Mercer for sharing their broken boards with me so I could do better testing.</p>
<p>Have a good one folks</p>
TaleSpire Dev Log 1712020-04-15T00:53:18+00:00http://www.techsnuffle.com/2020/04/15/talespire-dev-log-171<p>Today has gone slow and steady.</p>
<p>In the morning, we wrapped up the second beta patch and got that out. I spent a couple of hours around community conversations and then got down to work.</p>
<p>The goal is fixing the HardAssert errors that are plaguing people. To understand where those are coming from, we have to dive into some implementation details.</p>
<p>TaleSpire spawns a lot of objects, and we have to spawn them on short notice. We have no way to know at what point the player will just decide to drag out 10000 tiles and expect it to just work. Unity cannot spawn that many tiles in one frame[0], and so we have to spread the load over multiple frames.</p>
<p>The board is divided into 16x16x16 unit zones, and each operates largely independently of the others. This is important as boards have the potential to be huge and so we will want to only spawn the tiles in the area we are in. Also, because of multi-gm support, building could be happening in any other part of the board. This means TaleSpire must allow for zones to be in memory and fully editable without having to have spawned all the tiles in the zone. [1]</p>
<p>To that end, every zone may have a presentation. The presentation is the visual component of a zone and is what handles spawning tiles from the zone.</p>
<p>One issue with spreading an operation over many frames is that by the time it completes, another operation may have already arrived and have needed to edit the tiles you are spawning. Because of this, the presentation has a queue of changes to be applied. This means an operation makes its change to the data-model and then pushes the required change (and potentially new assets) into the presentation’s queue. We then currently give four milliseconds per frame [2] for the presentations to apply as many ops as they can.</p>
<p>Doing this, we give ourselves the ability to keep a reasonable frame rate. We don’t do a great job at this right now, but the pieces are in place for this to work [3].</p>
<p>Now let’s throw in networking. We cannot send all the tile data for each operation across the network; it’s not feasible. What we do instead (much like how RTS handle troop movements) is to send a description of what we need to be done (the operation) and let each client apply that change. As you can imagine, all clients have to agree on the order to apply these, or each will end up with a different end result. The details of that are beyond the scope of this post, but regardless we can talk about the issue that arises from this.</p>
<p>Agreeing on an order, and communicating the operations takes time, but games feel terrible unless actions are immediate. What we can (and do) do is to apply local changes immediately and then wait to see what order is agreed upon. If it turns out, we were already in the right order, then great! If not, we need to effectively revert our change, apply the changes that came before ours, and then reapply ours. Again making this fast in TaleSpire could be its own post, so I will give a TLDR by saying the way we do it involves applying the local changes eagerly to the presentation (and not the data) and then apply the change to the data-model when it arrives.</p>
<p>Finally, we can talk about the bugs :P</p>
<p>The HardAssert issues I’m currently working on are occurring when there is a mismatch on the presentation between the number of tiles the queued operation says needs to be spawned, and the number of tiles in the inbound queue. Clearly, there is something awry here, and so I am attacking one source of complexity in the code that enqueues operations for the presentation. The code in question was meant to act as an optimization in cases when a chunk of tiles is added and removed in the same frame (which can happen in undo/redo if you are super fast). However, given the rarity of that action, and the severity of the issues, I’m removing this in favor of something simpler for me to understand and debug.</p>
<p>Given all of the complexity I mentioned above (and plenty I left out), you can imagine that this has to be done rather carefully.</p>
<p>Today I managed to make the simplification, and so tomorrow I can start looking at the HardAssert errors themselves. I’ll then be talking to people in the community who have offered to let me use their boards as test cases, and we’ll see what we can learn from there.</p>
<p>Now that we have covered some of the background, I can also explain why we haven’t just added local save files, as many have suggested as a temporary fix.</p>
<p>When code in Unity throws an exception, it does not halt the whole program as you might expect. Instead, it travels up the stack to the calling MonoBehaviour and destroys that instead. Now let us imagine that that MonoBehaviour was handling something like applying the messages which have had their order decided. In that case, we can see what will happen, our local changes will be eagerly applied, but when you save and reload, lots of stuff is missing. In fact, everything that was changed after the MonoBehaviour died will be missing as all you were seeing was the presentation of the assets.</p>
<p>Of course, there should be something that freaks out if a message goes missing like that, but clearly, that is not happening. It’s very curious and very likely related (assuming this theory is correct)</p>
<p>We can also suppose that the cases where people see duplicate assets on reload could also be related. Imagine what happens if a delete message is lost. Locally we eagerly apply the change, but the delete message never arrives to change the data. So next time you reload, the tiles are back! In some cases, it’s more subtle as tiles made afterward are still there. Going into that would make this post even longer, so I’ll pass on that for now.</p>
<p>The takeaway from all of this is that all of it should be fixable, I’m not expecting fundamental issues here, just compounded bugs which result in a spectrum of exceedingly aggravating and destructive behavior.</p>
<p>As I progress on this, I’ll keep you abreast of what I’m finding.</p>
<p>Have a good one!</p>
<p>Peace.</p>
<p>[0] Unitys new DOTS systems change a lot about the performance of such operations. However, we could not move to it for the Beta as until very recently, their hybrid renderer did not support custom per-instance data, and that is critical to TaleSpire.</p>
<p>[1] We apply most operations in parallel using Unity’s job system, which means we can make very large data changes very quickly. There is also a lot we can do to improve this more, but that’s a subject for when things are more stable.</p>
<p>[2] This obviously should be adaptive to the current load, but it isn’t yet.</p>
<p>[3] This is a recurring theme of the rewrite from the alpha to the Beta. We have tried to put ourselves in a place where the goals we set for Early Access are achievable, even though certain features might not exist right now.</p>
TaleSpire Dev Log 1702020-04-13T18:48:03+00:00http://www.techsnuffle.com/2020/04/13/talespire-dev-log-170<p>Allo, bug fixing was gone reasonably well today.</p>
<p>This particular issue is not resolved but is mitigated for now.</p>
<p>Since the launch, we have been seeing a very large number of boards fail to sync because the backend refused to acknowledge that the people upload were GMs. The reason the backend thought this was the case was that their session no longer existed in the database.</p>
<p>Each time you connect, you are given a session, and it’s refreshed each time a keep-alive message is received from the game. The game sends keep-alive messages every 5 minutes. If a session is not updated for an hour, it is considered inactive and removed from the DB. It is also removed if TaleSpire signs out.</p>
<p>I guess we could sprinkle the word ‘should’ everywhere in the last paragraph as something clearly isn’t doing what it’s meant to.</p>
<p>The most obvious candidate was the code the removed the old sessions, and so I temporarily disabled it to observe the behavior. This didn’t seem to have an effect, and so I needed to investigate signout as a potential source of issues. However, whilst this is happening, we are obviously losing people’s boards, which is pretty heartbreaking, so we needed something sooner than waiting for data and hoping it illuminated the problem.</p>
<p>Each websocket connection made to the server spawns an Erlang process that is tied to that connection. When the connection dies, the process dies (and vice versa). We can store information along with this process, which allows us to tie information to your connection (the data remains server-side). We always authenticate you before creating the websocket connection, so the process almost represents the session we are interested in.</p>
<p>Ultimately I’ve wanted to move in this direction for a while but have not had time to. However, with things breaking, this afternoon became an exercise in how fast I can code carefully :D</p>
<p>It took about 6 hours to get the DB and server patches written and tested. We deployed at around 18:10 Norway time, and so far, it appears I’d only missed one thing. That was resolved quickly, though.</p>
<p>With that, all of the failures I’ve seen server-side about rights to save are gone. This does not mean all board persistence issues are fixed; it only means that this one cause is being handled. I still need to understand the session issue properly and keep an eye to see how things progress.</p>
<p>However, it does mean that this is no longer the highest priority. The next priority is now the ‘HardAssert’ failures that are corrupting board files. That is my task for tomorrow.</p>
<p>There is also likely to be another patch update either in a few hours or in the morning. We’ll keep ya posted</p>
<p>That’s the lot for now.</p>
<p>Peace</p>
TaleSpire Dev Log 1692020-04-12T13:34:47+00:00http://www.techsnuffle.com/2020/04/12/talespire-dev-log-169<p>Some people have noticed that TaleSpire will sometimes put a ♥ in the system clipboard and have been wondering why.</p>
<p>Our paste strings are a base64 encoded, compressed chunk of data. In order to use it in-game when need to decode and decompress it, which takes time.</p>
<p>Given that people want to paste a lot, we don’t want to have to go through that process every time. So we do the following:</p>
<p>If the system clipboard contains a string that isn’t ♥ we try to process it and paste it.</p>
<p>We store the processed version in an internal clipboard in the format we want</p>
<p>Now that we have done that, we want to use the internal version, but there is still a valid paste string in the system clipboard, which means we have to clear it or compare it to the last string we pasted. Those strings can be large, so it would be nice if we could have some shorter indicator.</p>
<p>We don’t want to clear the clipboard as we also want to give a nice message when a user tries to paste something that isn’t a valid paste string (we need to do this or paste can feel unresponsive, and we’ll get bug reports).</p>
<p>So instead, we put a little one character token into the system clipboard. Something that is rarely going to be there accidentally. I just used the Unicode heart as that felt nice at the time.</p>
<p>This is fast to check, and met the other requirements I had, letting us use the internal clipboard in most cases.</p>
<p>Ciao</p>
<p>p.s note the above is important even if you aren’t sharing pastes as when you copy we have to populate the clipboard with the string</p>
TaleSpire Dev Log 1682020-04-11T22:05:44+00:00http://www.techsnuffle.com/2020/04/11/talespire-dev-log-168<p>Heya folks,</p>
<p>Well, it’s the first dev log after the beta release and in a surprise to no-one except me I’m exhausted today :D
I had a good sleep, but clearly my body knows the release is done and fancies a bit more.</p>
<p>Today I did start looking at board sync issues however, as anything that breaks boards is a high priority.</p>
<p>To aid in the process as we hunt this down, I’ve added a small icon to request a board upload. It’s temporary as we want all persistence of boards to be totally transparent to players, however, obviously that is not the case right now. It also doubles as an indicator that shows you when the game knows there are changes that need to be persisted. It looks like this</p>
<video controls="" src="/assets/videos/upload-indicator.mp4" type="video/mp4" width="620"></video>
<p>After the usual 5 second delay, it will automatically upload, and then the icon will change back to its default state.</p>
<p>I also found a very ugly bug that could cause boards to become corrupted if the upload occurred while switching boards. What would happen is that on completion of an upload, the metadata for the current board (including its id) would be updated. However, uploads are async, so if you created a new board and if the upload took a while (multiple seconds), then it could have concluded when you were already in the new board. This would then overwrite the current board info with info from the previous board. This is daft behavior that is a hangover from the alpha.</p>
<p>I doubt it accounts for many of the issues seen, but it’s certainly one I’m glad we’ll be rid of soon.</p>
<p>I’ve implemented the change to TaleSpire, a related change to the backend, and tested on my local setup. I’m not going to put out a patch tonight as it’s 22:00 here and I have a glass of port that needs a drink and ‘breath of the wild’ that needs playing (also it’s very much tempting fate to push the last thing at night :p)</p>
<p>I’ll probably push this patch tomorrow.</p>
<p>The release was wild. I should probably write an update on just that soon. There were very last second (as in 15 minutes before release) server changes, which have resulted in DNS entries not having propagated in time (a glaring oversight on my part). Given the usual 48-72 hour propagation times, we are hoping to see a drop in the number of cases of the ‘stuck at login screen’ error by the end of Monday. After that, other cases will be a different bug, and we’ll attack that!</p>
<p>I’ll also be going through the github issues during the week. For now, my focus will be on the most board-breaking issues, but in time, we’ll get through them.</p>
<p>Thank you all for the amazing support, both in reporting issues but also just being so good about these bumps as we ride our way through them.</p>
<p>I hope you are all doing well and have been able to get far enough to see where we want to take this thing. Some of the creations we’ve seen you make already have been mind-blowing. This year is going to be great!</p>
<p>Have a great night folks,</p>
<p>Seeya with another update real soon.</p>
TaleSpire Dev Log 1672020-03-25T21:58:44+00:00http://www.techsnuffle.com/2020/03/25/talespire-dev-log-167<p>Quick one as I’m dead tired and am getting straight to bed after this.</p>
<p>No surprises here; we are working on the beta :p. Bug fixes everywhere. @Ree has been working on the GM blocks, UI, input system, cutscene mode, and more. I’ve been mainly working on fixing issues in creature sync, long-running server services that clean up old sessions & board snapshots, and other stuff that is a blur.</p>
<p>The production servers are up, and we are now testing internally on those. My focus is bug-fixing from now to the release.</p>
<p>Right after some sleep of course.</p>
<p>Goodnight all</p>
TaleSpire Dev Log 1662020-03-20T20:00:26+00:00http://www.techsnuffle.com/2020/03/20/talespire-dev-log-166<p>Hi again,</p>
<p>Yesterday I managed to get ‘unique creatures’ working in TaleSpire again. A unique creature is a creature which has has been marked as special by the GM, it is no longer tied to a single board, and it’s hp, stats, etc. are retained across the campaign.</p>
<p>We use <a href="https://doc.photonengine.com/en-us/pun/current/getting-started/pun-intro">Photon</a> for the realtime networking in TaleSpire. Anyone who is connected to the same board is connected via the same Photon ‘room’. When you are in the same room, you can send and receive RPCs, events, and synchronization messages.</p>
<p>However, if someone is in another room and summons a unique creature, we need to remove it from any board it was already in. We can’t use Photon for that, so instead, we use our backend. When you make tell the backend that a unique creature has entered a board, we broadcast a message using <a href="https://github.com/cabol/erlbus">erlbus</a> and then all listeners send a message down to TaleSpire itself. Naturally, you wouldn’t want every client receiving all messages, so we use topics. When you join a campaign, we subscribe your client to a topic named by the GUID for that campaign, and we do the same when you enter a board. This means it’s trivial to send messages to any client regardless of if they are in the same Photon room or not.</p>
<p>All of that appears to be behaving itself (although it could definitely feel better), and so today, I’ve been working on a selection of client-side bugs.</p>
<ul>
<li>Torch state not syncing</li>
<li>GMs couldn’t focus the cutscene camera of players</li>
<li>GMs couldn’t remove specific dice groups (clear all worked though)</li>
</ul>
<p>All of these are now behaving again. Next, I need to have a look at tile drop-in and some cases where things remain highlighted after the selection is removed.</p>
<p>I think tomorrow I’ll look at line-of-sight again and hook up the bit that actually hides and reveals creatures based on whether the creature you have selected can see them. Also, I’m 99% sure I’m sampling the cubemap incorrectly when collecting Ids, so I need to look at that and see what I’m screwing up.</p>
<p>Seeya tomorrow!</p>
TaleSpire Dev Log 1652020-03-19T10:40:57+00:00http://www.techsnuffle.com/2020/03/19/talespire-dev-log-165<p>Good morning folks,</p>
<p>Yesterday went pretty well. I was able to deploy the production DB, two servers, and show that they are communicating as expected.</p>
<p>One thing that took me a little bit to grok was how the two Erlang nodes were meant to find each other given only the node’s name. <a href="https://hackernoon.com/running-distributed-erlang-elixir-applications-on-docker-b211d95affbe">This article</a> talks about how epmdless is great, but how you also need something extra when the docker containers are on separate machines. We use <a href="https://github.com/scrapinghub/epmdless_dist">epmdless_dist</a> like the article shows, but the example has this code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>epmdless_dist:add_node(‘app1@host1.com’, 17012). % Add node into epmdless database
</code></pre></div></div>
<p>And I didn’t understand how erlang would know the address of my other server unless it was assuming that the <code class="language-plaintext highlighter-rouge">@host1.com</code> portion was the name for the server as well as the container. It turns out this confusion was valid, and there are a few variants of the <code class="language-plaintext highlighter-rouge">add_node</code> function. The one I needed was <code class="language-plaintext highlighter-rouge">add_node/3</code>. Here is the spec:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-spec add_node(Node, Host, Port) -> ok when
Node :: atom(),
Host :: inet:hostname() | inet:ip_address(),
Port :: inet:port_number()
</code></pre></div></div>
<p>After calling that, I could <code class="language-plaintext highlighter-rouge">net_adm:ping(‘app1@host1.com')</code> from the second server, and it correctly resolved the Erlang node. The two were then correctly connected, and <a href="https://github.com/cabol/erlbus">erlbus</a> messages started working across them transparently. Very cool :)</p>
<p>My job for today is to get ‘unique creature’ support back into TaleSpire. It got ripped out during the big engine refactoring as the code was so entwined with how we used to handle creatures. This time it should end up much more straightforward.</p>
<p>Until the next one,
Ciao</p>
TaleSpire Dev Log 1642020-03-18T01:51:20+00:00http://www.techsnuffle.com/2020/03/18/talespire-dev-log-164<p>Heya folks,</p>
<p>Time for another little update on what I’m poking at.</p>
<p>This week I’m focusing on server-side work, and the first thing I needed to look at was the new server setup. Previously the servers themselves were in a private subnet with the load-balancer being the only thing that was public-facing. With the move the websockets, I’m no longer using Amazon’s load-balancer, and so now the servers are public-facing again.</p>
<p>I needed a way for messages to be delivered between servers as players might be connected to separate instances in the same AZ. This should be made easy by <a href="https://erlang.org/doc/reference_manual/distributed.html">erlang’s distribution system</a>. However, we also want to make sure that we don’t accidentally connect servers, which shouldn’t be connected.</p>
<p>The first answer to this is to use erlang’s “cookie” setting. The cookie simply stops and two erlang nodes connecting if their cookies don’t match. Knowing that we set the cookie in the settings file before we push the build to aws. We also generate a random name for the node at that point. It’s a bit clumsy but will do the job for now.</p>
<p>One wrinkle is that our REPL is just another Erlang node, and so it’s cookie must match the server’s. To enable this, I wrote some elisp to look at the setting file of the node we are connecting to and extract the details we needed to initiate the connection. With this done, we keep the ability to connect and quickly query details or push changes if required.</p>
<p>I’ve also spent a little time fixing bugs on the client but nothing of significant interest right now.</p>
<p>The next step is to set up the production database, get two production servers set up, and check that the messages being sent across <a href="https://github.com/cabol/erlbus">erlbus</a> are being delivered to both nodes. I’ll also need to set up something simple for node discovery. Hopefully, that won’t take long.</p>
<p>Just a small Corona related update to wrap up. I’m definitely starting to come down with something, but I have no idea how much it’s going to interfere with development yet. Hopefully, it won’t, but I’ll keep you posted.</p>
<p>I hope you are doing well,</p>
<p>Back soon with more</p>
TaleSpire Dev Log 1632020-03-07T00:02:23+00:00http://www.techsnuffle.com/2020/03/07/talespire-dev-log-163<p>Hi again folks!</p>
<p>Work is going well. For the last two days, I’ve been focusing on handling network failures and testing board sync.</p>
<p>Sometimes you just lose connection for a moment; in that case, we want to get you back into the game as fast as possible. It’s quite likely that, since the client dropped out, nothing significant on the board has changed. In those cases, we can quickly confirm we are still up to date, make a couple of small tweaks and carry on as if nothing happened.</p>
<p>If there have been changes to the state of the board that aren’t trivially to reconstruct, we simply reload the board, which will bring your client back to the state it should be.</p>
<p>Another thing that can happen when someone leaves is host migration. When you have multiple people playing in TaleSpire, one client is the ‘host’. The host does the book-keeping that decides the order that all changes to the board happen in. When that client leaves that job moves to another client.</p>
<p>However, what if there were messages that were sent to the host before it dropped out. Did they make it? If they did, have they made it back to the clients yet? How do we know when to check? Then, if we are the sender of a lost message, we need to undo the change it made. Getting that nailed down was another of today’s tasks. I have a first version in, but it could do with more testing.</p>
<p>I also found some smaller bugs in board sync, which are now fixed, and I squashed a few other little things on the way.</p>
<p>Next, I’m finishing off the changes to board-sync to make it handle failures more gracefully, and I also need to have a look at pasting of slabs saved as strings as I think that is misbehaving right now.</p>
<p>Until next time,
Ciao!</p>
<p>p.s. I really was tempted to go into more detail about these fixes, but doing so would require a bunch more context, and I’d like to work on a couple more things before calling it a night.</p>
TaleSpire Dev Log 1622020-03-04T10:02:35+00:00http://www.techsnuffle.com/2020/03/04/talespire-dev-log-162<p>Howdy folks,</p>
<p>Alright, so yesterday I working on a few bugs in the code that handles the Unity GameObjects of the tiles (I’ll be referring to this as the ‘presentation’ as opposed the data-model which is separate).</p>
<p>I did a little work, so that undo/redo reuses portions of the <code class="language-plaintext highlighter-rouge">List</code> that holds the game objects. It wasn’t too tricky, but as the presentation adds/removes assets progressively over several frames, I had to take that into account.</p>
<p>I fixed a bug where rotated slabs weren’t be pasted correctly as it was considering the wrong zones for the paste. This was simply because, when considering what zones to paste to, I hadn’t rotated the bounds of the slab.</p>
<p>I fixed a few bugs relating to how, when we have updated the state of a script, we locate the asset so we can update the visuals. In the end, I’ve opted to have a per zone presentation hashmap. This is a bit annoying as I wanted to avoid yet maintaining another data structure if we could instead navigate to the asset another way (even if it was a bit less efficient); however, this will get us out the door.</p>
<p>Fixed a bug where progressive loading was skipping operations that it was meant to apply to the presentation. This was a dumb mistake of just spuriously advancing through the queue in two places. Probably a screw up during refactoring.</p>
<p>Added a bunch of asserts performing sanity checks on the data-model. These are only included in the debug build, so I get crashes nearer the cause of the issue, rather than always having to infer what screwed up.</p>
<p>There was also other stuff, but it gets too into the weeds to cover in a daily dev-log.</p>
<p>Today I’ll be going through a tonne of the code handling the network connections and board synchronization and trying to add sensible behaviors to the failure cases. For example, when we join and try to pull the board from another player:</p>
<ul>
<li>what if they aren’t ready to sync</li>
<li>what if they are already uploading the board</li>
<li>what if they start, but then that fails</li>
<li>what if they never receive the message to sync.</li>
<li>etc</li>
</ul>
<p>It needn’t be too fancy, but we need to have some logic dictating what to do, how many times to retry, and so on.</p>
<p>That’s all for now,
Seeya</p>
TaleSpire Dev Log 1612020-03-02T00:05:24+00:00http://www.techsnuffle.com/2020/03/02/talespire-dev-log-161<p>Things have been going really well.</p>
<p>The rewrites to fix the fundamental bugs are done, and I’ve been chasing down lots of regular bugs, mistake, typos, and other fuckups :)</p>
<p>It feels really nice to be back in a place where each issue is somewhat contained. A bunch are, naturally, related changes for the bug fix; however, as I’ve tracked each one down, it’s been more “Oh, of course” or “Why am I such a muppet”, rather than “Ah shit”. PROGRESS!</p>
<p>My plan for this next week is twofold. Firstly I’m going to spend some time finding and squashing the most obvious bugs, and then I’ll look at getting merged into master.</p>
<p>It’s looking likely that Ree and I are going to be able to meet up again soon. That’s going to be a bunch of fun as we’ll get to work on a lot of very user-facing stuff, and we’ll be inching closer to the build we’ll ship for the beta. Can’t wait!</p>
<p>Ok, that’s all for now.</p>
<p>Have a good one.</p>
TaleSpire Dev Log 1602020-02-29T14:33:09+00:00http://www.techsnuffle.com/2020/02/29/talespire-dev-log-160<p>I forgot to write this last night, so here is the update from yesterday.</p>
<p>I’ve rewritten copy and paste. I don’t have proper test coverage of these yet so I will need to look into that. First though I want work through the obvious bugs that show up through play. This should keep me busy for a day.</p>
<p>Seeya</p>
TaleSpire Dev Log 1592020-02-28T00:32:33+00:00http://www.techsnuffle.com/2020/02/28/talespire-dev-log-159<p>Today work progressed pretty well. I’ve updated the code that handles tile deletion and have moved over to fixing up tests. Copy and paste still need to be updated, but I’m more likely to find fundamental issues in the code I already have, so I definitely want to get that fixed up first.</p>
<p>I’m fixing up small mistakes from the refactor, and once that is done (and all current tests pass), I am going to start adding tests to try and find bugs when multiple builder’s operations interleave. Ideally, I want to use my automated testing code to find the bugs, but there are some limits here.</p>
<p>To perform automated tests, I made a very simplified implementation of the board, and it’s core operations, we call this the model. I then take the model, an instance of the actual board class, and a stream of random board operations. I apply the stream of operations to both the model and the board and then compare the state of each to ensure we got the same results.</p>
<p>The model, as mentioned, is very simplified. One simplification is that it doesn’t support multiple builders. This means that while it’s great for testing against one board, it’s not so useful for testing against two boards that are being manipulated by separate players.</p>
<p>We still can get some useful data, though. If we take a board and have two players add and delete things, it should be the same as if a single player performed those same actions in the same order. This means for those operations, we can use our model. However, undo/redo operate on a player’s own action history and so can’t be modeled the same way.</p>
<p>Regardless of the limitation, it’s still useful to use the automated tester, so I’m definitely going to implement the add/delete case mentioned above.</p>
<p>I should have that working tomorrow if all goes well.</p>
<p>Seeya then!</p>
TaleSpire Dev Log 1582020-02-26T17:11:37+00:00http://www.techsnuffle.com/2020/02/26/talespire-dev-log-158<p>Hi folks.</p>
<p>Today has gone well, I feel like the code that handles adding tiles and the undo/redo of ‘add tiles’ has taken shape well. Whilst I’m not stoked about increased complexity, I’m glad to see code shaping up in the right ways.</p>
<p>I’m going to start on the updated version of tile deletion tomorrow. It seems like it will be yield without too much issue, but I’m prepared for something to appear and confound me :)</p>
<p>I’m leaving paste for last. In theory, it shares a lot with the code that handles adding of assets, but previously there have been some tricky corner cases there. Time will tell.</p>
<p>This is one of those fixes that can’t be meaningfully tested until I update all the tile operations, so I’ll slog through it for now. I can’t wait to update the automated tests to scour this thing for bugs.</p>
<p>Seeya tomorrow</p>
TaleSpire Dev Log 1572020-02-25T23:52:41+00:00http://www.techsnuffle.com/2020/02/25/talespire-dev-log-157<p>A good time-adjusted evening to you all,</p>
<p>Today I’ve carried on with the bug-fixing. The day started out with more planning, as my initial plan didn’t cover some cases. I then spent the afternoon working on the code that handles adding assets. This seems to be going well so far.</p>
<p>Tomorrow I’ll continue with the ‘add assets’ code and then look at delete. I hope to finish that on Thursday and get onto copy/paste.</p>
<p>Ciao.</p>
TaleSpire Dev Log 1562020-02-24T19:31:11+00:00http://www.techsnuffle.com/2020/02/24/talespire-dev-log-156<p>It’s a nice quick update today.</p>
<p>I have set aside this week to work through the nasty bugs that delayed the release. I spent today wandering around my flat rambling about possible approaches, pausing to scribble furiously on a tablet, and then repeating those steps for 9 hours.</p>
<p>Between that and a few chats with Ree, I feel like I’ve got something that should work. Tomorrow I’ll start implementing and see if I’ve missed anything.</p>
<p>Seeya then!</p>
TaleSpire Dev log 1542020-02-22T12:40:24+00:00http://www.techsnuffle.com/2020/02/22/talespire-dev-log-154<p>Heya folks,</p>
<p>As planned, I started off Wednesday working on a bug, which meant that scripts were not getting unique ids. This manifested resulted in you clicking a door to open it, but instead, a chest in another room would open. Cute, but obviously broken :p</p>
<p>As I chased this down, I became more and more annoyed with how centralized all the state for these state-machines was. Originally I had started implementing the real-time scripts (ones that recompute their state every frame and arent synchronized over the network). There I want the data to be packed together so I could quickly iterate over large numbers of them. We didn’t need the real-time scripts to be ready for the beta launch, and so I had switched my focus to the state-machine scripts (used for doors, chests, etc). I reused a lot of code from the real-time scripts, and part of what we inherited was how their private data is stored. By storing this all together, syncing a zone also meant syncing this store. This worked but didn’t feel great. One result of this is that you end up having to download all the state for all the scripts in the whole board to be able to have the doors in one zone work properly.</p>
<p>As the delay has given us a little extra time, this became a great candidate for a refactor as it makes things better for the beta launch and also makes per-zone-sync (which is coming after beta launch) easier to implement.</p>
<p>If this has raised your eyebrows, that is fine. This is a slippery notion that a few of us were discussing on the discord the other day. It’s actually possible for a delay to make a second delay more likely as you now don’t <em>have to</em> ignore fundamental issues and just crank out something that works. You can suddenly address those issues, but in doing so, you will inevitably run into new edge cases, and bugs. Obviously, those new issues are now in your pile of things to do before the delayed release.</p>
<p>In this case, I’m hoping it’s worth it.</p>
<p>With the state for the scripts moved to the zones, I turned to creatures. We have two kinds of creatures in TaleSpire, normal and unique.</p>
<p>When a creature is made unique, it can be moved between boards in a campaign, and it has a little extra data associated with it. Traditionally uniques were the only creatures stored in the database; non-unqiues were saved into the board.</p>
<p>The issue with non-uniques being saved in the board is a question of how much needs to be synced. If their data is stored on the zone they are in, then simply rotating your character means you have to sync that whole zone (which may contain a thousand other tiles). If you decide to store creatures together outside of zones, then loading a single zone requires pulling all the data for all the creatures in the board, and there could be thousands of creatures in the whole board.</p>
<p>For now, we have instead moved non-unqiues to the database too. This lets us sync a single creature at a time and allows us to filter what we pull based on where they are in the world.</p>
<p>Being in the DB also gives us opportunities for tooling that could let GMs query info about creatures across the whole board without having to pull it all first.</p>
<p>Lastly, I finally spent the time to tweak the server and client, so that I can host the whole backend on my laptop and connect to it from Unity. This is awesome as I can iterate on both parts at the same time without having to push dev servers to AWS and wait on that. This cost me about a day but, in my opinion, it’s been totally worth it.</p>
<p>Right, now I’m going to go chill out for the weekend. Next week I’m planning to dedicate all my time to fix the bug in board sync that delayed the beta.</p>
<p>Seeya all on Monday!</p>
TaleSpire Dev Log 1532020-02-18T17:51:31+00:00http://www.techsnuffle.com/2020/02/18/talespire-dev-log-153<p>Since the end of last week, my focus has been looking into unknowns on the code side of the project. To that end, I’ve jumped over to the server-side of things.</p>
<p>The first task change was that, in the alpha, all the communication with our server (which only handle persistence) was over https. This means that polling was used in some places, and overall it was too slow. The simplest change to be able to push messages, given our current stack, was to move much of this communication to websockets. Once picking a <a href="https://github.com/sta/websocket-sharp">c# library</a> and getting basic communication set up, I needed to do something about all the request based code that was previously using https.</p>
<p>In order to make the changes on the c# side more iterative, I decided to write a simple request system that sent its messages over the websocket connection. When making the alpha, I had made a code generator in erlang that took a spec (written as an erlang data-structure) and generated erlang and c# code entry-points. This has made keeping both sides in line trivial and, now, it made it easy to update the code as I just had to update the code generator.</p>
<p>With that done, I spent some time fixing bugs from the refactor until I could log in, create boards, etc again. I then switched tasks again to certificates. We had used amazon’s certificate authority for production previously as they were free for AWS infrastructure, and our servers were behind one of amazon’s load-balancers. Now that we are using websockets, we would have to switch to their ‘network load balancer’, which I don’t have any experience with yet. For now, we’ve just picked up a good ol’ fashioned wildcard cert and will work with that. We spread load across servers in the new approach anyway, so using the load-balancer for that is no longer an issue (though there are other benefits). We can look back into their TCP level ‘network load balancer’ at a later date.
As an aside, we have previously used letsencrypt in staging, and while it did work well enough, I would need to make some changes to use it in production. This ends up being one of those cases where it’s cheaper to buy something simple that works than to use the free thing.</p>
<p>Also, on the server side, I’ve been reviewing our AMIs and docker images. One question I’m trying to get resolved right now can be found <a href="https://stackoverflow.com/questions/60296445/where-does-a-container-get-its-limits-no-file-and-similar-from-when-not-set-b">over here</a>. It’s a simple thing, but I have failed to find an official text stating the answer. If you have some expertise, it would be super useful to hear.</p>
<p>Yesterday I started fixing a bug, which caused us to assign identical script ids to tiles in different zones of the board when a drag spanned multiple zones. It’s simple enough but still requires some focus not to introduce a dumber bug in the process :p I also fixed a small bug that, for tiles only 0.5units, we spawned them too close together.</p>
<p>Today I’ll be finishing off the script Id issue and then possibly looking at rewriting the unique creature support.</p>
<p>Ciao!</p>
TaleSpire Dev Log 1512020-02-13T01:55:28+00:00http://www.techsnuffle.com/2020/02/13/talespire-dev-log-151<p>Today has been a tricky one. I found a significant bug in the building synchronization code that is definitely going to cost me a day or two. Given that I’m already behind on where I wanted to be this is a bit stressful. As I have an understanding of this problem but still have a good few unknowns on the server side I’m going to task switch for a few days and see if I can get a better overview there.</p>
<p>Not a fun update to write but here’s to the next few days regardless.</p>
TaleSpire Dev Log 1502020-02-10T18:16:38+00:00http://www.techsnuffle.com/2020/02/10/talespire-dev-log-150<p>Heya folks, yesterday I got the Spaghet that run our doors and chest hooked up and syncing correctly.</p>
<p>Spaghet is our little scripting language we made to deal with the question “if someone drags out a 30x30 slab of scripted tiles, how much resources are we using (both CPU & networking)”.</p>
<p>Scripts come in two kinds:</p>
<ul>
<li>state-machines: Their progress is driven by user interaction and is synchronized across the network.</li>
<li>realtime: Which run every frame and are unsynchronised</li>
</ul>
<p>Each script gets a small chunk of private storage (currently 32bytes) and runs on top of unity’s job system, which means we can run them concurrently across cores with no GC. We can also trivially pass the state of a script across the network and, with a little book-keeping, reapply it on the other side.</p>
<p>Currently, we are only using 1 script (the one that controls doors and chests), but we have a lot in place to be able to expand on this as we head through the beta. For now, I’m just happy to see it working.</p>
<p>Today after finishing some Spaghet work, I switched to looking again at copy-paste. Here I’m just trying to get it to the point that it is spawning the copied slab as expected without worrying about the feel of the tool. After that, we can look at this as more of a UX task.</p>
<p>Hmm, I think that’s all for now.
Back with more tomorrow</p>
TaleSpire Dev Log 1492020-02-08T12:52:03+00:00http://www.techsnuffle.com/2020/02/08/talespire-dev-log-149<p>Well, I’ve learned something about myself in this whole process. The closer I get to deadlines, the harder it is to convince myself to write updates when ‘the answers’ aren’t yet available. I’ve been struggling with a bunch of bugs and issues in the last couple of weeks and each time I see that I really should be writing an update I see the unsolved problems and think “If I can just get a bit further I’ll have something more concrete to talk about”. I guess dev logs getting sporadic around releases is just a thing that’s gonna happen for now. We shall see.</p>
<p>That said, hello again! A lot has been happening behind the scenes recently.</p>
<p>One big push on my side has been moving using ids rather than actual object references for creatures/players/etc across the whole project. So often in the future, the game will be receiving info on things that aren’t loaded on your local client, and it needs to be able to handle that. Luckily this only effects: building, sync, creatures, initiative mode, movement, gm request, and.. yeah well, everything. So, in short, I’ve been rewriting a significant portion of the codebase. It’s gone well, but it’s a lot of stuff to have in the air at once.</p>
<p>I’ve also finished implementing the sync of boards both to the server and between players. For now, it’s the same method as we used in the alpha. We are saving the whole board to a single file we load all at once. Naturally this wont work for big boards but I need to do some server work before I can add per-zone sync. We could ship the beta with this and then upgrade as we go. This may happen. The most important thing, for now, is that we get something that will get us out the door and is in the right shape to be moved to the new approach.</p>
<p>In fact, so much of the work feels different this time round in that we know what we need to make, but it’s a lot more cognitive weight :D. In the alpha, you make something, and then all the implications come pouring out; this time, you have to build and build with the main comfort being that, at least, you know it’s all needed.</p>
<p>Automated testing has been a lifesaver too. I recently added an option to the tests so that halfway through the test, it will serialize the board, deserialize it as a new board and hook it up like a networked client. It then continues the rest of the random actions and compares the two boards afterward. With these random tests, I’ve been able to find piles of dumb mistakes.</p>
<p>Other stuff that has been worked on in the last few weeks:</p>
<ul>
<li>
<p>Rewrote the undo/redo stack and the code that moved asset data between the active and inactive set. It had gotten far too complicated and was impeding progress. The new system is coarser but simple and fast.</p>
</li>
<li>
<p>Ree has been working on a control to allow you to move creatures between floors and across otherwise impassable obstacles easily.</p>
</li>
<li>
<p>Wrote new managers for tracking board, campaign, and network state. Some of the work on campaign state is getting it ready for the coming backend changes, and a pile of it is making sure that our connection to the realtime network stack isn’t so directly tied to whether we are considered in the board or not.</p>
</li>
<li>
<p>(Finally) hooked up single tile delete!</p>
</li>
<li>
<p>We want to support up to 16 clients building. Whilst writing the code that hands out those slots, I realized that we have a lot of what we need for spectator mode. We’ll have to do a bunch of work on the UX side for this but it’s more feasible now.</p>
</li>
<li>
<p>The atmosphere system is under heavy modification by Ree. He’s looking into how music is handled now. While we’ve been warned the first version will be pretty bare-bones, I’m personally excited for where this will go.</p>
</li>
<li>
<p>We have now been able to jobify lots of tasks that only update shader state. This includes tile drop-in a highlighting. This can be much faster now.</p>
</li>
<li>
<p>Moved the creature state to the new board representation. Attacks, stats, death, etc all go via the new system now.</p>
</li>
<li>
<p>Rewrote cutscenes sync</p>
</li>
<li>
<p>Disassembling class hierarchies for assets: Don’t do OOP kids :p More seriously, I had tangled the implementation of creature & tile board assets in a way that made progress harder. I have undone that now and, while there is seemingly more duplicate code, it’s way easier to manage and iterate on (and most of that code has subtleties that mean it shouldn’t be merged anyway)</p>
</li>
<li>
<p>Removed Lua</p>
</li>
<li>
<p>Rewrote how Spaghet scripts index their private state</p>
</li>
<li>
<p>Water prototypes! (see the bottom of this page)</p>
</li>
</ul>
<p>And waaay more. Lots of bugs are being worked on in the mix too.</p>
<p>It’s been a very productive time, but I’m still a good week behind where I wanted to be at this point. For context, the first ‘proper’ board load with the new system was 6 days ago, and the first networked play (with the new system) was yesterday. To be fair, it’s, of course, the case that for those to work tons of other stuff has to be working, so this isn’t <em>too</em> unreasonable. However it’s still a little tight. The best thing to do it work hard and keep you all in the loop.</p>
<p>Under the next one of these,
Seeya</p>
<p>p.s. Here’s a quick extra that we are showing off on the Kickstarter this week! More news on this as it happens</p>
TaleSpire Dev Log 1482020-01-18T13:05:45+00:00http://www.techsnuffle.com/2020/01/18/talespire-dev-log-148<p>I’ve been having a great time coding down at @Ree’s place, but it has meant I’ve neglected these updates. Let’s fix that.</p>
<p>We met up so we could collaborate on merging our feature branches together. Up until now, I’ve been focused on performance, and the data layer and @Ree focused on tooling. The goal isn’t to complete the merge, but instead to do any tasks that are easier when we are in the same room.</p>
<p>I’ve successfully dumped all of my code into the project, got it all building, test passing, etc. The managers for various systems are created now, but the building tools are still using the old code for now.</p>
<p>One massive source of complexity in the old version stemmed from our use of Photon, the 3rd party networking layer (which is great btw). By default, when a person leaves the game Photon automatically clears up all the networked objects they created[0]. This is no good for creatures (which were networked objects) as the other players still need to see them, so we disabled auto-delete of networked objects. This naturally meant we had to handle cleanup, which includes things like dice.</p>
<p>With the new system, we won’t ever have the whole board loaded at once, and this means some creatures won’t be loaded either. This also means we don’t want Photon to be helpful and make sure all networked objects are spawned as soon as someone joins the board. To deal with this, we removed the Photon network component from the creatures and gave a networked ‘handle’ to each player. This is attached to a creature when you select it and syncronizes the transform while the creature is held. On the other end, if the creature is loaded, it seems to behave exactly the same sa it did previously. If not, then no worries, once the creature is spawned, it will have the latest position will be sent anyway.</p>
<p>After a misguided prototype, I teamed up with @Ree to work out the handle api, and then everything came together very quickly.</p>
<p>This change actually lets us turn back on auto-delete in Photon and delete a bunch of cleanup code. Removing code is the best.</p>
<blockquote>
<p>NOTE: The handle approach may seem super obvious, but previously there had been reasons that it wasn’t a good fit. However, those reasons are out of date now thankfully :)</p>
</blockquote>
<p>On the subject of creatures, @Ree has been working on a new creature controller, which is much better suited for moving around these increasingly vertical scenes. It’s based on the excellent <a href="https://assetstore.unity.com/packages/tools/physics/kinematic-character-controller-99131">Kinematic Character Controller</a> asset for Unity. It’s an impressively stable base to build upon [1]. Our creature control still looks the same (we haven’t switched to slidy tank controls or anything :p), but this lets us handle moving through scenes more reliably than before. You still have the ability to lift the piece and throw it over walls when needed.</p>
<p>With the simplification of the networking code, I looked into upgrading to the latest version of Photon. I got partway through the migration before I hit an undocumented change. The <a href="https://doc.photonengine.com/en-us/pun/current/reference/serialization-in-photon#streambuffer_method">StreamBuffer</a> class used to be a subclass of <code class="language-plaintext highlighter-rouge">System.IO.Stream</code> and in the latest version, it isn’t. We have a bunch of code that relies on it being a stream, and so all that will need to be rewritten. This was already underway, but we need to keep the codebase working while all that gets finished and hooked up. For now, that means that I have to give up and do it later. Lost a few hours to this but ah well.</p>
<p>Another thing I have been doing during this merge is ripping out code. Old fog-of-war, line-of-sight, board format upgrading, all that and more got ripped out. There is no point fighting things if they aren’t required to keep tooling development progressing. It also is making it easier to work with or refactor code that used to have to interact with those removed systems.</p>
<p>There has been plenty more than this going on. @Ree is changing how the board grid works; I’ve been working on build speed and assemblies… and so on and so on.</p>
<p>In all this is going very well. We are plowing through tricky stuff and are ready to get back to working separately again, albeit this time with much more frequent merging as we approach Beta.</p>
<p>I’m taking Sunday as a rest day, but I’ll be back on Monday with more dev news.</p>
<p>Ciao.</p>
<p>[0] strictly, it’s the ones they own, which can be different if ownership has been transferred.
[1] also <em>amazingly</em> cheap for what it is.</p>
TaleSpire Dev Log 1472020-01-15T08:20:14+00:00http://www.techsnuffle.com/2020/01/15/talespire-dev-log-147<p>Hey again folks, another quick ‘kept of working’ post :)</p>
<p>Yesterday was spent working on Paste and Deletion of single tiles.</p>
<p>The latter is fun as we don’t keep unique ids for individual tiles as the memory usage adds up fast. This means we have to take some network-safe aspects to use to identify the tile. This is slower than looking up an id in a hashmap, of course, but given the frequency it happens the cost isn’t that egregious. The upside is not spending that memory, and not having to maintain whatever structures would have given us a direct lookup.</p>
<p>The above is why I’ve previously mentioned how late I left it to add the delete-single-tile feature to the new codebase, I wanted to be sure what data I would have available to work with. The less data we store per-tile, the better. So in the ideal case, we’d have very little to work with.</p>
<p>Whilst I’ve coded the hard, data-wrangling parts of paste, I’ve not finished the implementation as it requires hooking into the board-tool system in TaleSpire and @Ree has been writing on his branch. So that will get finished when we merge those branches.</p>
<p>Which leads us to right now. I’m on the train heading to @Ree’s so we can work on that merge. It’s gonna be a gnarly few days, but hopefully, we can get something working by Sunday.</p>
<p>One of the things that has let us work separately like this is that we have not worried about performance on the tooling branch and not worried about tooling on my data branch. We, of course, had to be communicating a lot to make sure neither of us were doing things that couldn’t be made fast/useful after the merge. So far, however, it’s been working pretty well. This means that once they are merged, I can move to start writing the optimized versions of the operations the tooling uses. And poor @Ree has to make my stuff feel nice :p</p>
<p>Alright, that’s the news for now</p>
<p>Ciao</p>
<p>p.s.</p>
<p>This didn’t fit above, but I wanted to mention it anyway. <code class="language-plaintext highlighter-rouge">UnmanagedMemoryStream</code> is cool!</p>
<p>It’s a c# type that takes a pointer and length in its constructor and gives you a Stream type that is compatible with tons of .net’s existing APIs. This has really come in handy when I’ve wanted to compress data from a NativeArray without unnecessary copying. Check it out!</p>
TaleSpire Dev Log 1462020-01-13T23:37:03+00:00http://www.techsnuffle.com/2020/01/13/talespire-dev-log-146<p>Hi again,</p>
<p>Today has mostly been spent on implementing ‘paste’. Although not finished, it is coming along well, and I think I’ve hit most of the surprises the implementation has for me.</p>
<p>I also think I’ve worked out some details to deleting single tiles that mean we can defer shifting data around until later. Amusingly (to me), I’ve left deleting of single tiles until now as it is, in some ways, more complex than deleting a selection of tiles. I’ll probably write a little more about that another day.</p>
<p>This is all rather vague, isn’t it? I’ve been sitting here thinking of more to say about it, but really it’s just been a day of coding and pondering.</p>
<p>Yup… that really is all for today.</p>
<p>Ciao!</p>
TaleSpire Dev Log 1452020-01-13T00:18:03+00:00http://www.techsnuffle.com/2020/01/13/talespire-dev-log-145<p>I didn’t write a log yesterday as I was really struggling with undo/redo histories, and it was just too painful!</p>
<p>In short, I was fighting with the behavior and then making sure the board was behaving the same as the simplified model we use for tests. I finally got that behaving this morning and so have turned my attention to copy/paste.</p>
<p>The short version is that I think I almost have Copy done, and so tomorrow morning, I’ll wrap that up and start implementing Paste. The main thing that takes time has been just working out how best to make it fast and then triple checking that it will behave correctly with our deterministic board update system.</p>
<p>The nice thing with how copy works is that you only need to send a fixed number of bytes of information across the wire for any size selection. As operations are applied in order, we send the selection and the point in the history the selection was made, and this results in the same tiles being selected on every client.</p>
<p>An annoying case, however, is pasting slabs of tiles from text strings (like you can find at <a href="https://talesbazaar.com/">TalesBazaar</a>). It’s a really powerful way to share content, but all the tile data does have to be sent to each client. It’ll be alright though, it’s just a case of making it as pain-free as possible :)</p>
<p>Ok, I’m getting super tired, and I’m not convinced that what I’m writing is coherent, so I’m gonna get some sleep.</p>
<p>Goodnight folks.</p>
TaleSpire Dev Log 1442020-01-10T23:13:49+00:00http://www.techsnuffle.com/2020/01/10/talespire-dev-log-144<p>Today I found and fixed a few nasty bugs, two of which were caught by the automated test generation mentioned yesterday.</p>
<p>I started off by writing the code to handle the undo/redo history. When that looked like it was working I set the automated test generator to make sequences of board changes between 80 and 100 operations long, this very quickly resulted in a series of operations that caused an exception.</p>
<p>The trouble was that it took 90 operations to trigger the bug which likely would be a huge pain to step through, so I made the worlds dumbest test minimizer.</p>
<p>When I get an error in an automated test I have it dump out the sequence of ops as code I can paste as a new test. This is what it gave me</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Test]
public void ManualMix3()
{
using (var asm = new BoardModelTestAssemblage())
{
var actions = new List<VTuple<OpName, int, int>>() {
VTuple.New(OpName.Add, 13, 0),
VTuple.New(OpName.Delete, 6, 17),
VTuple.New(OpName.Add, 56, 1),
VTuple.New(OpName.Add, 23, 4),
VTuple.New(OpName.Add, 27, 13),
VTuple.New(OpName.Delete, 3, 17),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 39, 13),
VTuple.New(OpName.Delete, 36, 7),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 57, 10),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 34, 15),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 11, 17),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 38, 6),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 0, 19),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Add, 33, 2),
VTuple.New(OpName.Delete, 45, 17),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Add, 5, 1),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 22, 6),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Delete, 38, 3),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 47, 3),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 3, 1),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 14, 12),
VTuple.New(OpName.Delete, 50, 18),
VTuple.New(OpName.Add, 13, 16),
VTuple.New(OpName.Delete, 45, 9),
VTuple.New(OpName.Add, 42, 12),
VTuple.New(OpName.Delete, 2, 8),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Delete, 34, 10),
VTuple.New(OpName.Add, 50, 18),
VTuple.New(OpName.Add, 48, 16),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Add, 13, 1),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Delete, 7, 12),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 11, 6),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 52, 7),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 31, 16),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 29, 1),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 55, 14),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 4, 18),
VTuple.New(OpName.Redo, 0, 0),
VTuple.New(OpName.Add, 26, 2),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Add, 47, 17),
VTuple.New(OpName.Delete, 25, 6),
VTuple.New(OpName.Delete, 40, 12),
VTuple.New(OpName.Add, 16, 4),
VTuple.New(OpName.Delete, 32, 19),
VTuple.New(OpName.Add, 0, 13),
VTuple.New(OpName.Add, 52, 5),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 55, 18),
VTuple.New(OpName.Delete, 1, 19),
VTuple.New(OpName.Add, 45, 11),
VTuple.New(OpName.Redo, 0, 0),
};
asm.ApplyActions(actions);
}
}
</code></pre></div></div>
<p>I wrote a little function that just, at random removed one operation from <code class="language-plaintext highlighter-rouge">actions</code> and ran the test again. If it still triggered the error it kept the new, shorter, list and repeated the process. If it didn’t trigger the error it tried a different operation. It was horrendously slow as it was doing it in dumbest, most brute force way imaginable but..It spat out this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>using (var asm = new BoardModelTestAssemblage())
{
var actions = new List<VTuple<OpName, int, int>>() {
VTuple.New(OpName.Add, 39, 13),
VTuple.New(OpName.Delete, 36, 7),
VTuple.New(OpName.Undo, 0, 0),
VTuple.New(OpName.Delete, 34, 15),
VTuple.New(OpName.Undo, 0, 0),
};
asm.ApplyActions(actions);
}
</code></pre></div></div>
<p>Much better :D</p>
<p>I then simplified the values to make my job even easier and was able to find the bug itself within 15 minutes. It wasn’t one thats exciting to read about, just some simple iteration mistake when undoing deletes of tiles, BUT it was super cool that we can do this.</p>
<p>If you find this fun look up things like quickcheck to see how the pros do it :p</p>
<p>With that and some other bugs fixed I am back to data wrangling.</p>
<p>So in all, today went well. I wish I had got further but that’s just how it goes.</p>
<p>Seeya tomorrow</p>
TaleSpire Dev Log 1432020-01-09T23:58:13+00:00http://www.techsnuffle.com/2020/01/09/talespire-dev-log-143<p>Today I got a useful portion of the automated testing working. It generates lists of changes (add/delete/undo/redo) and applies them to:</p>
<ol>
<li>An instance of the actual board data-structure</li>
<li>An instance of a simplified model</li>
</ol>
<p>There is also a second instance of the actual board data-structure connected to the first via a dummy network interconnect. This means that once we have applied a randomly generated list of changes, we can compare all 3 to see that all the results line up.</p>
<p>This isn’t a substitute for other tests as all could be wrong in the same way, but it’s already allowed me to improve undo-redo and track down a few small bugs.</p>
<p>I don’t have any automated test case minimization yet, but so far, it’s been relatively easy to minimize the cases that came up.</p>
<p>Next, I need to look at how data is moved to the ‘inactive set’.</p>
<p>Each client has an undo/redo history with a fixed length (currently 50). When the history gets longer than 50 the history event is no longer reachable. This means it cannot be undone. It also means that we don’t necessarily need to store it in the same way as we do for ‘active’ asset tile & prop data. Having a different data set, with a potentially different layout, can let us have optimizations that only make sense on that set.</p>
<p>To be moved to the inactive set, we need to know that the data isn’t going to be modified by an undo/redo still in the active portion of the history. This evening I’ve been implementing the code to handle this. It’s not done yet, but I’m hoping to wrap that up tomorrow morning. I’ll then start testing longer streams of operations to make sure they behave correctly.</p>
<p>I’ll then be back on the bug hunt. Before Christmas, I saw a case where undo/redo started getting me some very odd results (big chunks of tiles missing). I’m not sure if the bug was in the data itself or the code handling the progressive spawning/deletion of the game objects. I’m still hoping the former as that’s way easier to test, but I’m not sure quite how the bugs I’ve fixed the last two days would account for that. Ah well, we’ll see soon enough.</p>
<p>Thanks all for tonight,</p>
<p>Ciao</p>
TaleSpire Dev Log 1422020-01-08T21:14:58+00:00http://www.techsnuffle.com/2020/01/08/talespire-dev-log-142<p>Today is a short but sweet one.</p>
<p>Ree has been hunting down bugs in the new building tools and I’ve been working on the automated test suite which we are using to validate the building code[0].</p>
<p>That and of course the pre-orders are out, so we have been keeping an eye on things there.</p>
<p>Righto, I’m gonna get some sleep.</p>
<p>Ciao!</p>
<p>[0] I’ll probably put a little more detailed info out on that another day</p>
TaleSpire Dev Log 1402020-01-03T14:44:39+00:00http://www.techsnuffle.com/2020/01/03/talespire-dev-log-140<p>Today I’ve continued on server work:</p>
<p>I rewrote the sign-in code to promote the connection to a secure websocket if sign-in succeeds.</p>
<p>I’ve started wiring up erlbus so that the client will subscribe to the campaign and board and receive messages broadcast by other clients or the server.</p>
<p>One interesting note with that is that when messing with erlbus from the repl you might run into a badrpc error when trying out <code class="language-plaintext highlighter-rouge">ebus:messages</code>. This was happening as <code class="language-plaintext highlighter-rouge">ebus:messages</code> uses <code class="language-plaintext highlighter-rouge">rpc:multicall</code> behind the scenes and it wasn’t finding the function on one of the nodes. The node with the issue was the one I was connecting my remote shell from as, naturally, that doesn’t have the app compiled. The quick fix is just to use <code class="language-plaintext highlighter-rouge">-hidden</code> in your erl arguments as multicall contacts <code class="language-plaintext highlighter-rouge">[node() | nodes()]</code> by default and when you are hidden you wont show up in <code class="language-plaintext highlighter-rouge">nodes()</code> unless that behavior is specifically requested.</p>
<p>That’s the lot for today (other than the usual misc bug fixes). I’m flying back to Norway on Monday so most of the day will be a write-off due to travel. I may get a little done on Sunday but tomorrow I’ll be visiting some friends I haven’t seen in a few years (which is gonna be great!).</p>
<p>Ciao!</p>
TaleSpire Dev Log 1392020-01-02T17:38:36+00:00http://www.techsnuffle.com/2020/01/02/talespire-dev-log-139<p>Over the break, I’ve put down the front-end code to begin getting a handle on the changes that will be made to the backend. Most of this has been focused on reading, here are some of the things I’ve been dipping into:</p>
<h3 id="reading-things">Reading things</h3>
<ul>
<li>https://erlang.org/doc/man/gen_event.html some of the http handling code got a bit centralized a was slowing me down when making changes. I hadn’t used <code class="language-plaintext highlighter-rouge">gen_event</code>’s before as I found them a bit confusing. They are somewhat different than <code class="language-plaintext highlighter-rouge">gen_server</code> and <code class="language-plaintext highlighter-rouge">gen_fsm</code> as you don’t implement the manager but instead just the callbacks. It’s pretty neat once you grok it, though.</li>
<li>http://blog.differentpla.net/blog/2014/11/07/erlang-sup-event/ one part of understanding <code class="language-plaintext highlighter-rouge">gen_event</code> is knowing how to include it in your supervision trees. This covers those bits that mostly seem left out of other tutorials.</li>
<li>https://github.com/cabol/erlbus/ erlbus is an important piece of the changes I’m making, and so I’ve been getting familiar with it’s approach. It also contains one of the more lucid examples of using websockets in erlang, which was helpful in other tests.</li>
<li>https://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections An inspiring read. Little bits and bobs scattered throughout that were useful. Also, it’s just nice to see what certain pieces we use can potentially deliver (although we naturally aren’t doing this).</li>
<li>https://github.com/uwiger/gproc/blob/master/doc/erlang07-wiger.pdf This paper describes the evolution and technical realities that led to <a href="https://github.com/uwiger/gproc">gproc</a> which is something I intend to use in place of the standard global (across multiple erlang nodes) process registry.</li>
<li>http://erlang.org/doc/apps/stdlib/stdlib.pdf this naturally is WIP. Skimming the standard lib of any language is a great way to find out things you didn’t know were there, avoiding redundant work is nearly always a blessing</li>
<li>https://www.amazon.com/WebRTC-Cookbook-Andrii-Sergiienko/dp/1783284455 I’ve been skimming bits of this again as it’s likely to form part of the first implementation of the p2p voice & video chat</li>
<li>https://ninenines.eu/docs/ dear god ninenines is the best. Cowboy, gun, ranch. All fantastic quality, super robust and well documented. I have no idea how I’d function without their stuff.</li>
</ul>
<h3 id="making-things">Making things</h3>
<p>As we use <a href="https://www.photonengine.com/pun">Photon</a> for realtime networking in TaleSpire, we only need to focus on lower frequency events, such as persistence, for now. The alpha used a hacked together REST’ish api, which did the job but has all the expected issues (e.g. having to poll for changes). We are switching to using websockets for the Beta as it’s a relatively incremental step, will let us overcome some shortcomings, and still makes sense given the scale we will be at. It, too, will be a target for replacement in the future, but that is a subject for another day.</p>
<p>All of the server api the game uses is described by a data-structure that we then generate erlang & c# code from. I’ve updated the generator to create websocket handlers rather than the http/s ones. In doing so I’ve also been undoing some of the code that centralized some of the http management.</p>
<p>The last thing I did was update the code that polls to see which domain it is at before attempting to pick the right certificates and start serving over https.</p>
<p>The next step will be to rewrite the session handling code as it currently assumes a stateless connection.</p>
<p>Until next time,</p>
<p>Peace.</p>
TaleSpire Dev Log 1382019-12-06T09:09:52+00:00http://www.techsnuffle.com/2019/12/06/talespire-dev-log-138<p>I’m currently writing this on the train home,</p>
<p>I’ve been down working with Jonny for the last time (in person) this year, and it’s been a great few days. We made a bunch of progress in different places, so lets natter about that.</p>
<h3 id="tiles">Tiles</h3>
<p>First off, we were looking at tile placement and control. We may have found a nice change which could help with some tricky cases our alpha testers would get in when working with walls. My ultra vague language is due to it being so early in the prototype phase that we are not ready to talk about it yet. This is not the first time in the last 4 months when we have had a fix that play revealed to be worse than the initial problem. When we are more confident, we’ll do a full writeup (or maybe a short video) to go through the different things (Just placing tiles has surprising nuances to it :D).</p>
<h3 id="meatier-news">Meatier News</h3>
<p>As that is all arm waving and no substance, here is some real news: You will be able to place creature miniatures off-grid. This has been play-tested for a while internally now and, although it does introduce some challenges where it comes to UI/messaging, it does feel pretty cool. For those who have wrangled narrow corridors in the alpha, this should feel rather freeing :p</p>
<p>Do note that we are still keeping the tiles to the grid, however. Without this limitation, the worst-case complexity for various systems becomes totally unmanageable. Remember that doubling the length (or resolution) of the side of a cube increases the volume 8 times. So if you have anything that operates over the volume like say, fog of war or line of sight, your day would have gotten much worse (I know, I go on about this in almost every post).</p>
<h3 id="the-right-questions">The Right Questions</h3>
<p>I also think I know how cross-zone pathfinding will work now. I’ve been struggling a little with how to maintain a single graph as zones are loaded in and out. Jonny did me the simple favor of asking if it could be computed on demand. Building the nav-mesh from scratch each time a creature is picked up <em>sounds</em> like a lot of work, but it actually reduces the complexity a lot and opens up some opportunities for parallelization. We just need to keep the per-zone input data sorted in a way that helps this on-demand process as much as possible, and that bit is easier. More on this when I have tested it.</p>
<h3 id="monster-branch-hunter">Monster Branch Hunter</h3>
<p>We have also started the long process of getting everything merged into master. The part of this that required us to work closely was moving to Unity assemblies for the core project as suddenly their ‘magic folders’ stop working[0]. This means reorganizing the whole codebase, which wasn’t too bad[1]. However, this can make for seriously ugly merge conflicts and so it was best to stop development for a couple of hours, get the change merged in, and test on both our machines.</p>
<h3 id="testing">Testing</h3>
<p>I’ve also started working on another approach to ironing out bugs from the board code. It’s very much inspired by my uninformed scanning of property-based testing. We make a dirt-simple but accurate model of the code we wish to test, and then we generate random streams of operations that are applied to both the real version and the model, and we compare the results. The model gets to ignore all details of multithreading, being performant, etc. Its only job is to be understandable and give the behavior and results intended from the real version. In previous tests, I had already made a fake network interconnect so I could apply operations to one board and make sure the other board ended up with exactly the same result after sync so now I can wire things up like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> [Randomly generated board operations]
/ \
| |
v v
[Simple Model] [Real Board 0] <--fake network interconnect--> [Real Board 1]
</code></pre></div></div>
<p>We feed the model and ‘Real Board 0’ the same operations, but then compare the results for all three boards. This gives us a pretty decent amount of testing for almost no additional work.</p>
<p>I will stress that this isn’t property-based testing. I want to learn it, but I don’t have the time to take on a new technique right now. However, I know that even this limited approach can give good results, and I think I’ll be able to add some very basic test-case minimization too.</p>
<h3 id="christmas-approaches">Christmas Approaches</h3>
<p>And with that, it’s almost the end of the year. There is more to come, but I’m going to posting a bit less until January as I’ve managed to schedule everything to happen in one month, and apparently it’s arrived. For the next week, I’m on holiday, and then I’ll be heading to the UK for Christmas. During my time in the UK I’ll be working some evenings on server stuff, so expect a very significant shift in content then!</p>
<p>I hope this finds you well folks.</p>
<p>Peace.</p>
<p>[0] by default, Unity automatically makes assemblies that separate game and editor code based on special folder names. It’s handy but naturally goes away when you decide to handle assemblies yourself.</p>
<p>[1] There are a bunch of ways to layout your project, but we found it simplest to make 2 folders, ‘Runtime’ and ‘Editors’, and move all game code to the former and all the editor code to the latter. The directory structure on both sides is mirrored. We have a little helper menu for writing the boilerplate code for inspector editors, so we updated that to respect the new structure and added a right-click menu entry to jump to the editor folder if it existed.</p>
TaleSpire Dev Log 1372019-12-04T16:03:31+00:00http://www.techsnuffle.com/2019/12/04/talespire-dev-log-137<p>Hey! I best recap the last two days before even more get away from me.</p>
<h3 id="monday">Monday</h3>
<p>Monday was spent debugging some of my uses of unsafe code. Unity’s Job system has some great tools to detect race conditions, but they also let you disable it if you have a use case that demands it. This is a wonderful attitude, and I’m very grateful for that. However, boy can I corrupt some memory with this stuff.</p>
<p>The first thing was debugging a new NativeCollection I had made. This is a Stack with a fixed max size that can have multiple concurrent writers or readers (but not both at the same time). I use this to hand out and return large numbers of Ids to a limited resource, and to do so from concurrent jobs.</p>
<p>The id, in one case, is an index into a big array that stores state that is later pushed to the GPU for use in the shaders. Each visible tile has one entry in this array and gives that spot back when their presentation is destroyed. This can happen with large numbers of tiles across multiple zones simultaneously.</p>
<p>Because each tile gets a unique index into that array, we know that they will be the only thing accessing it, so we don’t want Unity to protect us from race conditions that would occur otherwise[0]. This means we add the <code class="language-plaintext highlighter-rouge">[NativeDisableContainerSafetyRestriction]</code> attribute.</p>
<p>That, of course, allows you to really make a mess, and sure enough, I did :p.</p>
<p>In this case, it wasn’t due to that attribute, however. I was using <a href="https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.UnsafeUtility.MemCpyReplicate.html">MemCpyReplicate</a> to set the default shader state across a portion of the array and I <em>may</em> have incremented the pointer to the strut rather than the pointer into the array. So I happily tried to copy invalid portions of memory into the array.</p>
<p>I was actually super lucky that this caused Unity itself to crash very consistently. These kinds of bugs are horrifying if they stay under the radar.</p>
<p>With that done, all tests passed again, and I got back to work.</p>
<h3 id="tuesday---wednesday-morning">Tuesday -> Wednesday morning</h3>
<p>Tuesday started with traveling to visit @Ree. It’s always great when we get to do this as certain kinds of tasks are so much easier.</p>
<p>In a related subject.. yesterday I was working on the line of sight shader :)</p>
<p>This starts with rendering the scene into a cubemap from the head of the creature you have just placed. Each thing rendered is colored based on an id (zero for all occluders and >0 for creatures or potentially other points of interest). We then run a compute shader to sample the cubemap and aggregate which ids are in there.</p>
<p>To start with, we want to store 32bit ids, which meant using a floating-point texture format. This seemed to work, but every time I read the value from the compute shader, the value was always clamped between 0 and 1.</p>
<table>
<tbody>
<tr>
<td>It’s my first time doing this in Unity, so I spent ages trying to find out what I’d done wrong until I gave up and ran <a href="https://renderdoc.org/">renderdoc</a> on it. I should remember to do this first as it turned out Unity was rendering the faces in non-float textures and then copying the data over :</td>
</tr>
</tbody>
</table>
<p><a href="https://docs.unity3d.com/ScriptReference/Camera.RenderToCubemap.html">RenderToCubemap</a> doesn’t seem to warn about this, but @Ree got me on the right path by saying we should try a RenderTexture instead. RenderToCubemap has an overload for this, and there is even this interesting line in the docstring for the overload that takes the cubemap directly</p>
<blockquote>
<p>If you want a realtime-updated cubemap, use RenderToCubemap variant that uses a RenderTexture with a cubemap dimension, see below.</p>
</blockquote>
<p>I’m guessing that this is a hint to the intermediate copies (and maybe a temporary FBO too?).</p>
<p>After moving over to that, I started getting values higher than one out of the texture YAY! The values are wrong, but frankly, I don’t care about it today. The pieces of the puzzle have all been started, and I can work on those from home. It’s best to use this time I have over here to touch on as many other things as possible.</p>
<h3 id="the-rest-of-today">The rest of today</h3>
<p>There have been more chats working out details of performance around tile hide and cutaway and much musing on how to merge out branches in a way that doesn’t slow us down.</p>
<p>That’s my next task, to update the main dev branch to use Unity’s <a href="https://docs.unity3d.com/Manual/ScriptCompilationAssemblyDefinitionFiles.html">assemblies</a> and begin moving in the new data model code.</p>
<p>We have to be very careful though as it’s soon Christmas and we really mustn’t block either us from being able to work as we won’t be available to help each other during that time.</p>
<p>That’s all for now.
Seeya!</p>
<p>[0] We aren’t, in this case, too concerned about <a href="https://en.wikipedia.org/wiki/False_sharing">false sharing</a> or other forms of contention that may hurt us. This will get more of a review later, however.</p>
TaleSpire Dev Log 1362019-12-01T21:30:57+00:00http://www.techsnuffle.com/2019/12/01/talespire-dev-log-136<p>Phew, so that was the meat of the week’s work. Let’s wrap up the loose ends.</p>
<p>This is probably skippable as it’s just a note of some other things that got changed.</p>
<h3 id="moving-to-unitymathematics">Moving to Unity.Mathematics</h3>
<p>Unity added a new <a href="https://github.com/Unity-Technologies/Unity.Mathematics">math library</a> recently and whilst having practical benefits, it also results in much nicer code to write and read so we have started using it for all new work.</p>
<p>I took an hour this week to move all the tile data code to use this except for Bounds as I want to make sure that the new behavior matches what I expect.</p>
<p>I also got to delete my Vector3Int32 class as Unity.Mathematics contains int3 and I renamed my Vector3Int16 to short3 and tweaked the API to match that of Unity.Mathematics so it all feels more consistent.</p>
<h3 id="realtime-changes">Realtime changes</h3>
<p>One future task that has been making me nervous is updating the realtime networked portion of TaleSpire.</p>
<p>I had a little breakthrough simplifying some of our code that got very spidery and complicated during the development of the alpha. I was going to write a little about that here but I’m gonna save it for another week as it would turn this into another long post.</p>
<h3 id="start-loading-assets-earlier">Start loading assets earlier</h3>
<blockquote>
<p>I probably should have mentioned this in a previous post but oh well.</p>
</blockquote>
<p>Unity loads <a href="https://docs.unity3d.com/Manual/AssetBundlesIntro.html">asset bundles</a> asynchronously. As we don’t want to try load every tile in TaleSpire at once, when you first place a tile we need to load it. This results in some frames passing before spawning continues.</p>
<p>This could be exacerbated by the fact that changes are queued as it won’t know a tile is needed until it’s time to spawn it.</p>
<p>For that reason, we make sure that as soon as the data representation has a new tile kind added, it notifies the asset loader so it can start loading the asset.</p>
<p>This will rarely matter but it may help when there is heavy load, and that’s the time you need the most help.</p>
<h3 id="merging-this-monster">Merging this monster</h3>
<p>I’ve also started kicking this code into a shape where it’s suitable for merging into master. Jonny and I divided up the work so that, technically, neither of us have tasks that block the other. However, we still need to end up with one game and my current monster branch goes against everything I like when developing.</p>
<p>Ah well, soon enough it’ll be in.</p>
<h3 id="actually-the-end-for-realz">Actually the end for realz</h3>
<p>Well, that’s the week. In that time I was also best man at the wedding of a dear friend so it’s been a doozy.</p>
<p>It’s probably time for a cup of tea.</p>
<p>Goodnight!</p>
TaleSpire Dev Log 1352019-12-01T20:23:16+00:00http://www.techsnuffle.com/2019/12/01/talespire-dev-log-135<p><em>Sound of trumpets</em> BEHOLD, Part 3 commences.</p>
<p>We have talked about updating the data and the presentation but we still have to animate the tiles.</p>
<p>In TaleSpire we want to be able to have animate tiles as they appear and disappear as it makes the game feel better. We have a lot of tiles to update so we have to take care of how we do this.</p>
<h3 id="a-simple-problem">A simple problem</h3>
<p>We wouldn’t want to have to update the positions of the GameObjects every frame so we animate the drop-in using the vertex shader. The animation curves are simply stored as ramps inside textures we can sample over time. This means we only have to update the per-instance data for the GameObjects when we want to change which animation they are running.</p>
<h3 id="a-secondary-problem">A secondary problem</h3>
<p>One annoying thing though is that, when changes to this state do come, they tend to come in large numbers. One example of this is during tile selection.</p>
<p>When players select tiles we raise them very slightly so show that they have been selected. This means we need to update the per-instance data for those tiles. Also if the tiles have props attached to them those props need to be raised too.</p>
<p>Unity’s interface to this is via <a href="https://docs.unity3d.com/ScriptReference/MaterialPropertyBlock.html">MaterialPropertyBlock</a>s and like so many things in Unity it can only be updated from the main thread. This kinda sucks as what we need to do is perfectly parallelizable</p>
<blockquote>
<p>If the tile bounds intersect the selection bounds then set the value</p>
</blockquote>
<p>This is extra annoying when we consider that selections bounds are 3D and so the number of tiles that may be involved goes up terrifyingly fast.</p>
<p>To resurrect our example from the previous articles a 30x30 tile slab contains 900 tiles. Due to TaleSpire’s smallest tile size being 1x0.25x1 units A 30x30x30unit region could (theoretically) contain over 80000 tiles.</p>
<p>Obvious that is insane and we will need to add some form of density limit, but the game will need to communicate this limit to the players in a way that feels fair.</p>
<p>Once again user-generated-content games bring their own flavor of crazy to the party :)</p>
<h3 id="the-secondary-problem-with-a-hat-on">The secondary problem with a hat on</h3>
<p>But we aren’t done, oh no we are not. Last time we talked about progressively applying changes to the presentation. Due to this, a selection can be made on tiles that are still spawning. According to the data layer, you selected perfectly valid tiles but the presentation hadn’t quite caught up. Maybe the tile spawned the frame after you let go of the selection and so as far as you percieved it was there when you let go.</p>
<p>Now there are different ways to resolve this but the way I want to go with for now is that the data is the source of truth, if you selected the right region you selected the tiles even if the presentation hadn’t quite got there yet.</p>
<p>I expect this to be an edge case that people won’t even notice, but it is better to have an answer for it.</p>
<h3 id="one-change">One change</h3>
<p>So let’s talk about one change that helps here. What we are going to do is move the tile’s animation state out of its per-instance data and into a separate <a href="https://docs.unity3d.com/ScriptReference/ComputeBuffer.html">buffer on the gpu</a> (let’s call this the tile-state-buffer). Then we will put the index into that buffer into the tile’s per-instance data instead.</p>
<p>This little change lets us solve all the above issues.</p>
<p>First off we will have a copy of the tile-state-buffer in local memory. This will be a <a href="https://docs.unity3d.com/Manual/JobSystemNativeContainer.html">native container</a> so that we can operate on it directly from Jobs. This means all those selection updates are done in parallel now.</p>
<p>Next, we just gained the option to push only part of the local buffer to the GPU per frame. This is super handy in cases where huge numbers of tiles are being changed per frame. There is a concern this delay to the start of the animation could feel slightly unresponsive, however, I’m expecting that as long as <em>something</em> starts happening on the same frame as the action it should feel ok. Especially as this means that lots of changes are being made.</p>
<p>This also plays nicely with our selection -v- progressive update issue. We can now make the changes to the tile-state-buffer independent of whether the presentation for the tile has finished loading yet. As soon as the GameObjects are spawned they will put their state index into their MaterialPropertyBlock and they will be up to date immediately.</p>
<h3 id="and-another-thing">And another thing</h3>
<p>One obvious this I skipped when presenting the selection problem is that updating all tiles in the selection is totally unnecessary. We only actually need to update the shader state for the tile that has become selected or unselected since the last frame.</p>
<p>From the following diagram, we can see that in 2d that means you are inside one of two rectangles. For 3D it’s 3 cuboids.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> frame 1 difference
+----------------+ +------------+---+
| | | |
frame 0 | | | 0 |
+------------+ | | +------------+---+
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | 2 |
| | | | | | |
| | | | | | |
| | | | | | |
+------------+ +----------------+ +------------+---+
</code></pre></div></div>
<p>Given that we can control how fast the panning in the game is this makes the number of tiles to update per frame much less. I left this until now though as selection makes a great example of the issue of tile counts in volumes and we have other operations for volumes that benefit from these changes too.</p>
<h3 id="more-more">More? More.</h3>
<p>Well, this got long. I think I’m gonna save the miscellaneous stuff for another post.</p>
<p>See you there!</p>
TaleSpire Dev Log 1342019-12-01T18:41:55+00:00http://www.techsnuffle.com/2019/12/01/talespire-dev-log-134<p>Alright, part 2!</p>
<p>So in the last post, we talked about updating the data representation of the board. This time let’s get into the bit the players see and interact with. The GameObjects.</p>
<p>In Unity’s currently supported approach, any <em>thing</em> in the world is a GameObject[0]. GameObjects do very little on their own but their properties and behavior come from Components.[1]</p>
<p>We don’t need to understand much more about this for now except that Unity naturally has to manage all these GameObjects and, as with much of the API, you can only interact with them from the main thread.[2]</p>
<h3 id="rendering">Rendering</h3>
<p>One of the nice things Unity does do for us is handling instancing so that it’s fast to draw many of the same kind of object. For example, if we have 1000 of the same tile in the scene then Unity can make one draw call to the GPU to draw them all.[3]</p>
<p>We aren’t going to talk about rendering any more in this post but we will revisit it in the next one as there are some issues we need to overcome.</p>
<h3 id="spawning">Spawning</h3>
<p>So we have a TileSpire-like game where we want to have the players be able to make tonnes of tiles, they should be able to drag out or paste big slabs and the delete individual tiles as they please.</p>
<p>If someone drags out a 30x30 slab, we need to spawn 900 tiles. Depending on the complexity of the tile Unity might not be able to handle instantiating that many new GameObjects in a single frame. This is serious though as when people are building we have building actions arriving from other players fairly frequently and we wouldn’t be able to apply the changes robustly until all the GameObjects are made.</p>
<p>The first change we could make is to keep the data for Tiles separate from their GameObjects. This also gives us opportunities to store that data in <a href="https://docs.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types">blittable</a> types and use Unity’s Job system to operate on them (like we talked about in the <a href="https://bouncyrock.com/news/articles/talespire-dev-log-133">last post</a>)</p>
<blockquote>
<p>I’m going to shift from talking explicitly about tile data and GameObjects to talking about the tile data and it’s ‘presentation’ unless the fact that GameObjects are involved is important</p>
</blockquote>
<p>With this, we have the opportunity to make the data changes immediately and apply them over a number of frames. This is the approach we are using and is a part of why we want to apply data changes as fast as possible, because there may be multiple arriving in a single frame.</p>
<h3 id="relationship-management">Relationship management</h3>
<p>One issue we have given ourselves is that, now that the data and presentation are separate, we need to keep then in sync. If someone deletes a tile we need to delete the presentation too and, naturally, the same goes for undo/redo/etc. We could keep an index from the data to the representation but this has implications:</p>
<ul>
<li>We need an extra integer of storage for every tile. 4 bytes is nothing on its own, but 100000 tiles is not a crazy number of tiles to expect to have to deal with so it does add up.</li>
<li>It may benefit us to have different layouts for the data and the presentation arrays. That could mean having to update the data->presentation index. One can then assume you’ll need to have a way to do this quickly.</li>
</ul>
<p>Now you may not care about either of those, which is valid too. But whichever way you pick there will likely be tradeoffs.</p>
<p>One simplification we can do is to note that tile data is added in chunks. If we keep presentations of tiles from the same chunk together then you only need indices per chunk, this can help in the approach you pick.</p>
<h3 id="avoiding-work">Avoiding work</h3>
<p>Other than being able to spread out work over multiple frames we get some trivial opportunities to avoid work too.</p>
<p>First off, if we can apply changes to data first and then generate the changes to the presentation afterward then we often get to skip work. Imagine an Add followed by a Delete arriving in the same frame, by looking at the data afterward we only need to spawn the tiles that remain, rather than first spawning them all and then applying the delete.</p>
<p>Also in the case where there is a small backlog you can get opportunities to skip entire actions. The best case is an Undo & Redo together (this can happen when people are playing around to see if they like a particular change). In that case, you can remove both actions from the queue as they cancel each other out.</p>
<h3 id="you-dont-always-need-to-present-anyway">You don’t always need to present anyway</h3>
<p>If you have players in two different parts of the board you may want the data for the zones in memory but not have to have a presentation of the Zone you are nowhere near. This is trivial here as the data is separate.</p>
<p>This makes jumping to that part of the board much less jarring than it would be as you are already to date with the latest version of that part of the board.</p>
<h3 id="applying-changes">Applying changes</h3>
<p>A Zone’s presentation now gets given a ‘budget’ of how much it can change this frame. It keeps popping changes from its queue and applies as much as it can until the budget runs out. The budget is represented by a floating-point number and we give separate costs to spawning tiles, reusing ones from the GameObject pool, deleting tiles, etc.</p>
<p>These numbers are made up by us but they are easily tweakable so we have a lot of freedom now to profile and find out what works. We’ll chat more about this in the future when we get more data.</p>
<h3 id="sticking-it-together">Sticking it together</h3>
<p>So we are doing a bunch of these things. We have a separate presentation for each Zone. They each have queues of changes to apply which, due to how tiles are managed in data, mainly boils down to adding and deleting chunks of GameObjects by their Id.</p>
<p>We use the code that applies the change to data to make simpler change instructions for the presentation so that it has much less work to do.</p>
<p>We don’t maintain any indexes between the data and presentation (beyond unique ids of chunks) so we don’t have to do any upkeep work, but we do end up scanning tiles in more cases that we would have to otherwise [4]</p>
<p>We don’t do all the work avoiding stuff yet but the hooks are in the code so we can add this easily when we want to. First off I need to iron out the bugs that remain.</p>
<h3 id="next-post">Next post</h3>
<p>Ok so that wraps up this post, in the next one, we need to talk a little about rendering and a few miscellaneous bits and bobs we’ve also done this past week.</p>
<p>Ciao</p>
<p>[0] Ignoring drawmeshinstanced and friends for this as it makes for an easier to parse, if slightly inaccurate, sentence.</p>
<p>[1] At least, only the Unity data, you can change the fields on your components as you like.</p>
<p>[2] I should also mention we are not using the new ECS yet as some features we need are only just arriving and we would still need to do a bunch of work to test everything we need will work in the new system. I’m 95% sure we’ll be moving to it within the next year though.</p>
<p>[3] Because of reason 1023 is the largest number of instances that can be made in one call in Unity. This is a limitation of their system and it’s a bit of a shame, but we will deal with this another time</p>
<p>[4] Tiles in the presentation are still currently in the same order as the data representation, so this does still leave us with ways of making applying deletes less costly. Currently, this is not an issue though so I’m leaving it for now</p>
TaleSpire Dev Log 1332019-12-01T16:41:10+00:00http://www.techsnuffle.com/2019/12/01/talespire-dev-log-133<p>Hi folks,</p>
<p>There have been no daily updates for the last six days as I’ve been in a sort of self-imposed crunch to hammer out some things that have been on my mind.</p>
<p>The first one was related to the performance of Add/Delete/Undo/Redo operations on tiles. The faster we do this the better and we have a bunch of opportunities to make this quick. The first was to move the core code of these features over to Unity’s <a href="https://docs.unity3d.com/2018.3/Documentation/Manual/JobSystemOverview.html">Job System</a>.</p>
<h3 id="why-we-really-have-to-care-about-performance">Why we really have to care about performance</h3>
<p>For newer arrivals to these writeups, performance may seem an odd concern for a game where the rate things are happening seems much lower than that of, for example, an FPS. For us, it’s not the frequency of actions that plague us but the volume of work that can be created by them, and the fact there is no general pattern to when they happen.</p>
<p>For example, dragging out a 3x3 square creates 9 tiles, but 30x30 is 900 tiles.</p>
<ul>
<li>Tiles are usually made of smaller assets so that’s 900*N object to spawn[0] and render</li>
<li>If any of those sub-objects have scripts then that’s 900 more scripts running per interactable sub-object</li>
<li>The first thing tiles do is drop into place, so you best be ready to animate 900 of those (we do this via shaders)</li>
<li>When people drag to select we slightly raise the tile to show it has been selected so you will need to be able to query many thousands of tile bounds quickly.</li>
<li>etc</li>
</ul>
<p>.. and 30x30 tiles isn’t many considering a board is meant to be 30000x30000 units wide and 10000 units high.</p>
<p>All of this isn’t to complain, we want the game to behave like this, but we do need to be clear to ourselves on what needs to be achieved within the 16ms of a frame[1]</p>
<h3 id="jobs">Jobs</h3>
<p>So back to the jobs. One of the first things I looked at was the collections we use to store the tile data. As of today, the data for tiles are split up somewhat like this:</p>
<ul>
<li>Board: contains N zones</li>
<li>Zone: contains up to 16 client-data-chunks, one for each client[2] who has built in a zone</li>
<li>ClientDataChunk</li>
<li>AssetData: Holds the data for the tiles themselves</li>
<li>AssetLayouts: Holes layout information of the data in AssetData</li>
</ul>
<p>As you can imagine we have lots of opportunities for performing operations in parallel here. Zones are a 16x16x16 unit chunk of the board and so if an add or delete crosses multiple zones all of those adds/deletes can potentially be done in parallel.</p>
<p>Also as each client’s data is separate within the zone we have opportunities here too. This has so far been less useful as an Add only affects the data of the client that performed it, and during deletes, we want to store the tiles that were deleted for undo/redo and so that would require collections that concurrently have arbitrary numbers of tiles written into them. Making a new collection type to cover this case was too much of an additional distraction and the common case is one or two people building so there is less parallelism to be gained here anyway. [3]</p>
<h3 id="a-diversion-into-better-collections">A diversion into better collections</h3>
<p>Here is a rough idea of how AssetData is laid out.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// (the data is actual flat but grouped here for clarity)
AssetData: [ [Tiles from Add 0] [Tiles from Add 1] [Tiles from Add 2] [Tiles from Add 3] ]
ActiveHead = 4
Length = 4
</code></pre></div></div>
<p>We tiles we consider active are all from the Add less than ActiveHead. So in the above case, all of them</p>
<p>One of the pain points from the alpha was Undo/Redo feeling unresponsive with large numbers of tiles. With this layout, the undo of an Add is just subtracting 1 from ActiveHead (we will talk about spawning the visuals for the tiles later). By not deleting the data yet Redo is just adding 1 to ActiveHead.</p>
<p>If you Undo and Add and then perform an add or delete action there is no way to Redo that add so we just overwrite its data with new tile data. Thus we rarely <em>need</em> to shrink the backing NativeList behind AssetData.</p>
<p>Now having the Head integer and Native collection separate is fine but as they need to be updated from jobs together it helps for them to be together in a native collection that properly uses Unity’s thread safety checks to make sure nothing funky is going on. To that end, I spent a little while making a collection with the terrible name NativeGreedyList.</p>
<p>It has a Capacity, and ActiveLength and a FullLength. The FullLength is the apparent length of the List. The ActiveLength is the ActiveHead integer from before. Capacity and FullLength are separate as you often want to allocate in larger increments than you immediately need so you have to resize less often.</p>
<p>Unlike NativeList we don’t every reduce the capacity unless explicitly required to (as the data past the ActiveLength is often still valuable to us)</p>
<h3 id="back-to-jobs-and-the-problem-with-multiple-people-building">Back to jobs and the problem with multiple people building</h3>
<p>Alright, so armed with new collections the work Jobifying everything continued. After a bunch of wrestling and learning, I got this into shape and so now the data updates are parallel. Yay!</p>
<p>But of course, there is more.</p>
<p>One thing that has been on my mind is ‘roll back’ and ‘reapply’ and that what I’m going to ramble about now.</p>
<p>When you have multiple people changing the board you need to make sure that each change is applied the same on every client to make sure they all end up with the same board[4]. When you only add stuff it’s easy but with Delete the result depends on the order the operations are applied.</p>
<p>To set the order each ‘action’ is sent to one client who is deemed the ‘host’ and they give it a history-id and send it on to the rest of the clients. Great, this gives everything a set order but it also means that you don’t see a change until it’s made that round trip to the host. That kind of delay is unacceptable (it feels awful) so we did the following:</p>
<ul>
<li>Send the action to the host</li>
<li>Apply the change locally immediately</li>
<li><em>time passes</em></li>
<li>An action arrives from the host
<ul>
<li>if the action is the change we already made, we don’t need to do anything.</li>
<li>if the action is from a different client we:
<ul>
<li>‘roll back’ our change</li>
<li>apply the action from the other client</li>
<li>‘reapply’ our change</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>This works but necessitates some form of ‘snapshot’ we can roll back to[5]. This has been a pain for a while and something I’ve never been happy with. However, during the week I was looking at the problem of spawning the Unity GameObjects for all these tiles (more on that in the next post) and I spotted a cool bunch of details.</p>
<blockquote>
<p>For the rest of the article we will use the term ‘uncommited’ to refer to actions/tiles we have made but that have not yet been confirmed by the host, and so would have to be rolled back</p>
</blockquote>
<ul>
<li>Each client has separate collections for their assets in a zone</li>
<li>Adds only affect the data for the sending client</li>
<li>Deletes can affect data for all clients</li>
<li>Deletes have not only a region in space but also a point-in-history (as a selection may be made before the person hits the delete key)</li>
<li>An uncommitted change, by definition, can only be on the client that made it</li>
</ul>
<p>So by making sure that delete only operates on tiles that have been committed then we don’t need to rollback for this case. We do have to reapply our deletes though as they may have added tiles one of our selections should have captured. When we reapply a delete we only do it to the data of the clients whose actions were committed before ours.</p>
<p>Undo & redo are also simple as they can only affect asset data for the client that performed the action[6]</p>
<p>With all this in place, we get to totally remove snapshots. It requires a bunch of tweaks to how changes coming from the network are applied.</p>
<p>This is one of those lovely times where data layout fundamentally change what has to be done to implement a feature. I find that stuff pretty cool.</p>
<h3 id="more">MORE!</h3>
<p>This post has got long enough but there is still plenty more to cover from this week so let’s get back to it in the next post.</p>
<p>Ciao</p>
<p>[0] We do use instancing but until Unity’s ECS supports per instanced data we are forced to stick with the classic GameObjects approach which means we do need to spawn separate GameObjects.</p>
<p>[1] or, of course, spread out across many frames</p>
<p>[2] A client, in this case, is a single running instance of the TaleSpire executable.</p>
<p>[3] This will be revisited in the future as that collection type would be very handy. I also didn’t use NativeQueue as ParallelWriter is only for use in IJobParallelFor jobs as far as I can tell.</p>
<p>[4] Previously we have talked about how we can’t afford to be sending all the tile data across the network so instead, we just send the building ‘actions’ the players perform.</p>
<p>[5] Using Undo/Redo wouldn’t work in this case due to how deleting tiles placed by other clients works[6]</p>
<p>[6] If you delete tiles from another client and then Undo that delete the tiles are not placed back in their collection. Otherwise, if they undo and then redo they restore those tiles, which feels a bit odd.</p>
TaleSpire Dev Log 1322019-11-24T01:23:54+00:00http://www.techsnuffle.com/2019/11/24/talespire-dev-log-132<p>The last two days touched a few different things.</p>
<p>First off I was working on the resource list for the Spaghet scripts. This holds references to Unity objects in the asset like Colliders and Animators, and also the configuration of radial menus and GM requests.</p>
<p>In implementing this I re-learned a terrible rule of Unity’s immediate mode UI. Don’t try and make the code ‘nice’ you’ll sink ages into chasing things that look like they would help only to hit some limitation of the system and have to throw it all away. DONT DO IT. I’m sure I was meant to have learned this lesson last time, but apparently, that wasn’t enough.</p>
<p>Once that was good-enough-for-now™ I started looking into reimplementing the doors/chests/etc using the new scripting system. For now, I’m not worried about improving the graph, as I want to switch to Unity’s node graph elements at some point, so I just wrote the ops by hand. In the process, I added a few more operations to the language: yield, goto, and an op to send a message to a resource from the resource list.</p>
<p>It’s not done yet but after the resource list thing, I was a bit tired of scripts and put it to the side for another day.</p>
<p>On Friday I started off with a planning session where I just try and work out any small details I can across the tasks on my todo list.</p>
<p>One that stood out as needing some work was pathfinding so I’ve started work on that. The goal for the beta is as follows.</p>
<blockquote>
<p>When you pick up a creature you should be able to see all the places you could walk to within a given range (determined by the creature)</p>
</blockquote>
<p>This means we are looking at a kind of flood-fill where we ‘walk’ something for a certain distance, but what do we walk?</p>
<p>Well, last year I walked an octree of the board itself (we did this for fog of war so the octree was available) and this was hard. This time I want a dedicated graph. I’ll write more about this once I’ve got it working.</p>
<p>And that’s the lot for now. It’s been a good week and there will be more of these logs next week.</p>
<p>Until then, peace.</p>
TaleSpire Dev Log 1302019-11-19T23:06:10+00:00http://www.techsnuffle.com/2019/11/19/talespire-dev-log-130<p>Today I carried on working on the scripting implementation.</p>
<p>Each script has a small section of memory that it will be able to store values to, my first job of the day was finishing of that data store. A few things were a little difficult to debug as VS2017’s debugger doesn’t show what data is on the other end of a pointer.</p>
<p>After finishing I found out that VS2019 does have this feature so I spent a couple of hours fighting to get that upgraded and playing with Unity. If you try this yourself remember:</p>
<ul>
<li>Don’t let VS install unity again, the version is old. Search for ‘Unity’ in the components section of the installer and find the tools “unity tools for visual studio” component</li>
<li>Open Unity’s package manager and make sure “Visual Studio Code Editor” is up to date</li>
<li>If VS2019 isn’t appearing in <code class="language-plaintext highlighter-rouge">Preferences -> External Tools -> External Script Editor</code> then click browse and find something similar to <code class="language-plaintext highlighter-rouge">C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\devenv.exe</code></li>
<li>If like me you only use VS for debugging and have a different editor for programming you will need to update omnisharp (or your equivalent) that and probably your editor integration too.</li>
<li>Restart everything :p Just for safety</li>
</ul>
<p>After all this, I hooked up the scripting system so that the board asset arrays now receive the references into Spaghet’s data store. All is working according to plan there.</p>
<p>The next thing I want to do is look at moving Spaghet into its own package so I can share that between TaleWeaver (the asset prep and modding tool) and TaleSpire itself.</p>
<p>That’s gonna take a little coordination with the rest of the team so during that I’m going to have a look at making the Unity component that lets you connect resources to your scripts. Here is the old (very unfinished) one that held the LUA scripts.</p>
<p><img src="/assets/images/scriptAttribs0.png" alt="script assets" /></p>
<p>The important part, for now, is the bit marked in red. You need to be able to add resource slots so you can drag in the animator or sound effect you are interested in. Then those resources can be referred to from the script.</p>
<p>Another potentially nicer way to handle it would be have the slots on the state machine nodes themselves.. hmm actually there might be a nice way to do that as the nodes are made with Unity’s imgui elements. I’ll look into that tomorrow.</p>
<p>Whatever approach I use for beta, I’ll then need to look for this element when the asset is first loaded in TaleSpire and submit the scripts to Spaghet.</p>
<p>Slowly slowly all these threads are weaving together.</p>
<p>More tomorrow.</p>
<p>Ciao</p>
TaleSpire Dev Log 1292019-11-19T01:07:45+00:00http://www.techsnuffle.com/2019/11/19/talespire-dev-log-129<p>Today I’ve continued working on scripting. My main goal has been to work out the most efficient way to handle the allocation and setup of the private data that each instance of a script is allowed to store data into.</p>
<p>There can be large numbers of tiles made in a single action and so making sure I’m no performing unnecessary copying of data has been important. On the flip side many creation events are of only a single tile so whatever we have needs to make sense at that scale too.</p>
<p>As I progressed with this I kept feeling that the <a href="https://docs.unity3d.com/Packages/com.unity.collections@0.0/api/Unity.Collections.html">native collections</a> I had available, whilst great, were not ideal for this task so I decided to look into how to write my own. I really wanted something chunked like NativeChunkedArray but with</p>
<ul>
<li>faster indexing</li>
<li>no deallocation of backing chunks unless explicitly requested</li>
<li>An api more focused on being backing store for the data rather than being focused on working on an element by element basis</li>
</ul>
<p>The <a href="https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.NativeContainerAttribute.html">official documentation</a> is both limited and also kind of out of date so I’d recommend starting with <a href="https://jacksondunstan.com/articles/4963">this fantastic article series</a> by Jackson Dunstan. In fact if you are considering doing any work with Unity’s new DOTS systems, trawl that site, it’s a goldmine.</p>
<p>Pulling apart his examples and getting to grips with Unity’s safety system took the rest of the evening, but it was definitely worth it. I’ve now got the base of a much more focused chunked store that I can ensure provides exactly what the scripting system needs.</p>
<p>That’s all for tonight.</p>
<p>Peace</p>
TaleSpire Dev Log 1282019-11-16T16:56:16+00:00http://www.techsnuffle.com/2019/11/16/talespire-dev-log-128<p>Update time! Progress has been very good.</p>
<p>@Ree has kept on hammering away at the building tools from the last update. He’s also been handling lots of behind-the-scenes organizational stuff which whilst not exciting to blog about is as critical to TaleSpire happening as anything else we do.</p>
<p>Jason has been chatting to loads of the backers who pledged at the ‘Help design a ___’ levels and sculpting is in full swing. Very exciting!</p>
<p>I’ve finally broken through the wall of fiddly details that was plaguing my board sync implementation and it’s looking pretty good now. This has freed me up to look at some other tasks that are on my todo list so I’m gonna ramble about that for a bit in this post.</p>
<p>First off I went back to Fog of War. Basically, the task is this:</p>
<ol>
<li>Take a 16x16x16 grid</li>
<li>In every cube write ‘true’ if there is meant to be fog there and ‘false’ if not</li>
<li>Now take this info and make a 3D mesh that contains all the cells marked ‘true’</li>
</ol>
<p>And I wanted to make sure we had step <code class="language-plaintext highlighter-rouge">3</code> worked out.</p>
<p>Now there are a couple of points to keep in mind:</p>
<ul>
<li>The mesh we need to make needs to conform fairly closely to the grid as we don’t want to see glimpses of anything we shouldn’t</li>
<li>We don’t need to make this look like fog, we just need a mesh we can start working with, we can do all kinds of polygonal and shader effects on top of a simple mesh that will look much more fog-like.</li>
</ul>
<p>As a great example check out this tweet from <a href="https://twitter.com/Ed_dV">@Ed_dV</a> !</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Inspired by Sea of Thieves and Sky, I've been playing with opaque polygonal clouds - sphere meshes that have their normals softened and distorted via command buffer. Still lots to work out, but progress is being made! No animation yet, just aiming for a fluffy render first. <a href="https://t.co/WVyk7X5p5b">pic.twitter.com/WVyk7X5p5b</a></p>— Edward del Villar (@Ed_dV) <a href="https://twitter.com/Ed_dV/status/1169153878841004033?ref_src=twsrc%5Etfw">September 4, 2019</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Those clouds are spheres, with lots of magic on top (if you watch the whole thing it shows the spheres without the magic).</p>
<p>Given those two points, I decided a good source of wisdom would be the kinds of algorithms that Minecraft-like games often use. Luckily for me <a href="https://0fps.net">Mikola Lysenko of 0fps.net</a> has written <a href="https://0fps.net/category/programming/voxels/page/1/">some amazing articles on these techniques</a> along with WebGL demos and source code, what a star! This gave me a huge boost and along with a few implementation tips from other sources I was able to put together a ‘Greedy Mesher’ (the name of one of the techniques) in Unity. Now, this next picture is <em>not</em> meant to be clouds, it was just a test to make sure it was working.</p>
<p><img src="/assets/images/jobifiedMesher0.png" alt="greedy mesher" /></p>
<p>For the coders out there, I also made this implementation using Unity’s job system which allows me to run the mesh generation jobs in parallel across all cores. It’s pretty speedy considering how little effort was required.</p>
<p>That is all on the subject of fog for now but we will be back with more in future.</p>
<hr />
<p>Next on my list of concerns was interactable Tiles and modding.</p>
<p>We have a bunch of interactable tiles in TaleSpire and this is only going to increase as we open up the ability for the community to make them.</p>
<p><img src="/assets/videos/sideTable0.gif" alt="sidetable" /></p>
<p>A real pain point when you make a user-driven content game like TaleSpire is that you have no way to know when or where someone will just suddenly throw tonnes of extra stuff for your game to do.</p>
<p>For example, take that side-table in the gif above. There is nothing to stop you dragging out a whole load of tables and the numbers get large FAST. Remember that if you drag out a square with 32 tiles down one side then that is over 2000 tiles that your game suddenly has to handle. They all need to appear on all the other player’s machines quickly and they all need to be interactable immediately.</p>
<p>That means all 2000 of them are running their own little scripts. How do you ensure that they don’t start crippling your game?</p>
<p>Now we naturally knew this was coming and previously we had been making small fast components that would then be wired up using a LUA script (that’s another programming language for those not into this stuff :] ).</p>
<p>It worked but I was still concerned about how it would scale.</p>
<p>There was another thing. Like I said we were using LUA in a weird way, as a way of setting stuff up. I started thinking that that might be the worst of all worlds for users as, for those who like LUA, there are lots of things they would be told not to do and for those who hate LUA, well they hate it :p</p>
<p>Were there other problems? You’re damn right there were!</p>
<p>As mentioned before, we need to keep all of these tiles in sync on everyone’s machines and so the idea of scripts just being able to fire off messages whenever they liked was a nightmare, it could really make the game unstable if done wrong.</p>
<p>Now, this sounds tautological but there are two kinds of state for a tile: State that matters to the story and state that doesn’t. For example, whether a fire is lit or not matters, the exact positions of every smoke particle does not. As long as it roughly conforms all is well.</p>
<p>So it would be good if we can separate these kinds of scripts out from each other so we have a shot at being able to enforce sensible behavior over the network.</p>
<p>So after pages and pages of notes and a lot of coffee, I decided it would almost be worth making our own compiler that suited our needs but we needed</p>
<p>Now, this bit gets technical so feel free to skip to the row of asterisks!</p>
<p>+++++++++++++++++++++++++++++++++++++++++++++++++++</p>
<p>I wanted something that</p>
<ul>
<li>has a simple execution model.</li>
<li>was focused on only the tasks we need.</li>
<li>run on multiple cores</li>
<li>used no GC</li>
<li>was reasonably fast</li>
</ul>
<p>Here’s the approach I went for:</p>
<p>0.
I decided to start with only 4 data types: int, float, int3, and float3</p>
<p>1.
I use a node graph as the code representation. I’m currently using the excellent <a href="https://github.com/Siccity/xNode">XNode</a> but if/when Unity’s new graph UI stabilizes I will probably move to that.</p>
<p>I added code to detect cycles and compute the correct types at each node</p>
<p>2.
Walk the graph and generate a tree of <a href="https://en.wikipedia.org/wiki/Intermediate_representation">IR</a> nodes. This is mainly for convenience but in the conversion, we also pick correctly typed IR nodes in a few specific places.</p>
<p>3.
Walk the IR nodes to compute the layout for the stack</p>
<p>4.
Walk it again and generate a bytecode that represents the program</p>
<p>5.
Do a little cleanup pass and kick the result out as a <code class="language-plaintext highlighter-rouge">NativeArray<byte></code></p>
<p>6.
Write a <a href="https://docs.unity3d.com/2018.3/Documentation/Manual/JobSystem.html">Job</a> that takes the instruction array, an array of private state and a few other bits and run our little bytecode in parallel.</p>
<p>7.
Marvel at the power of coffee</p>
<hr />
<p>It took 18 hours to get the first version up and running and the results were promising.</p>
<p>Here is a picture of some nonsense ‘code’</p>
<p><img src="/assets/images/spaghet0.png" alt="behaviour" /></p>
<p>The graph above (whilst being gibberish) can be run for 10000 tiles in around 2ms, which is way slower than it will be but it’s acceptable for a first pass. It told me the approach was worth more work so I took Friday to flesh it out a bit.</p>
<p>Oh, and the language is called Spaghet!</p>
<p>With that we have the kind of script that can be used for the ‘not story critical stuff’ but we still need the other scripts too.</p>
<p>To that end, I’m now working on how to script the <a href="https://en.wikipedia.org/wiki/Finite-state_machine">state machines</a> that will run the important stuff. Here is a little picture of the prototype I’m currently working on:</p>
<p><img src="/assets/images/spaghet2.png" alt="statemachine" /></p>
<p>Naturally, the look of the graph and the nodes available will be improved too.</p>
<p>Soooo yup, it’s been a good, busy week. I’m going to be working on this and all the code that holds it together all this week. I’m hoping to have the major plumbing done by Wednesday though.</p>
Cutting Tiles Take 22019-10-06T19:44:47+00:00http://www.techsnuffle.com/2019/10/06/cutting-tiles-take-2<p>This weekend Ree and I also took another look at the tile cutaway effect. This one is super important not just to reveal the part of the building you are looking into, but also to cut down walls that would get in the way of controlling your creatures.</p>
<blockquote>
<p><strong>Please note:</strong> All the following screenshots are not from TaleSpire or even Unity, they are from some test code I was using to play with cutaway effects. It’ll look much nicer in TaleSpire :p</p>
</blockquote>
<p>We had prototyped this effect before. Here is a sphere we have chopped the top of using the technique.</p>
<p><img src="/assets/images/cutaway0.png" alt="c0" /></p>
<p>The basic approach was that we discard all the fragments that are above the cutaway and then texture the backfaces as if they were positioned at the flat cutaway plane.</p>
<p>However, we saw an issue when the meshes of the models were intersecting (we also saw this in some of the assets on the Unity store for drawing cutaways)</p>
<p><img src="/assets/images/cutaway1.png" alt="c1" /></p>
<p>So what can we do? Well if we look at this with the power of MS Paint this is what we have.</p>
<p><img src="/assets/images/cutaway2.png" alt="c2" /></p>
<p>So let’s draw some lines from an imaginary camera point to a few points across the cutaway spheres. Then let’s make a score for each line if the line crosses a backface you add 1 point, for a frontface you subtract 1.</p>
<p><img src="/assets/images/cutaway3.png" alt="c3" /></p>
<p>Neat, notice how if the score is above zero then it means you are seeing through the cutaway. Also note that because it’s addition and subtraction, we can do it in any order.</p>
<p>We can perform this check for every pixel really easily. Let’s render all the backfaces first and additively blend all the fragments. Then let’s render all the front faces into the same buffer subtractively blending.</p>
<p>Then we have an image which is the mask for the cutaway area.</p>
<p><img src="/assets/images/cutaway4.png" alt="c4" /></p>
<p>At this point, we have effectively solved the original problem. We can now render the tiles (throwing away the stuff above the cutaway) and then render the cutaway using the mask we have made.</p>
<blockquote>
<p>Note: you could definitely do some nice stuff with stencil buffers to speed all this up.</p>
</blockquote>
<p>And there it is, a simple cutaway handling intersections.</p>
<p><img src="/assets/videos/cutaway0.gif" alt="c5" /></p>
<p>Now this may be the version we use in TaleSpire or it may not, we still need to work on it some more and see how it performs once it’s in. But either way, it’s neat :)</p>
<p>Alrighty, that’s all for now.</p>
<p>Peace</p>
TaleSpire Dev Log 1272019-09-17T08:24:47+00:00http://www.techsnuffle.com/2019/09/17/talespire-dev-log-127<p>Recently I’ve been spending most of my time working on undo/redo support and general data model work inside TaleSpire. Because of that, I thought I’d take a post to ramble a little about undo/redo and how a ‘simple’ feature can grow when you have a few constraints.</p>
<p>This is not the exact way we are doing things in TaleSpire, but it’s an exploration of an idea I went through whilst looking at the feature.</p>
<p>In this post, I’m only going to talk about placing/deleting tiles rather than creatures to keep things simpler.</p>
<h3 id="chapter-1---building-basics">Chapter 1 - Building basics</h3>
<p>So lets first have a look at a little building:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/06kWQrEoAis" frameborder="0" allowfullscreen=""></iframe>
<p>Players can do a few things:</p>
<ul>
<li>They can place a single tile</li>
<li>They can drag to place multiple tiles</li>
<li>They can delete a single tile</li>
<li>They can select and then delete multiple tiles</li>
<li>They can copy and paste slabs of tiles</li>
</ul>
<p>We are going to refer to placing tiles as an <code class="language-plaintext highlighter-rouge">add</code> operation and deleting tiles as a <code class="language-plaintext highlighter-rouge">delete</code> operation</p>
<p>There is no moving of tiles in TaleSpire, anything that looks like a move is actually a <code class="language-plaintext highlighter-rouge">delete</code> operation followed by an <code class="language-plaintext highlighter-rouge">add</code> operation. So cut and paste work like this. However, for now, we will ignore copy/paste and just look at the fundamental operations.</p>
<p>What can we say we know about <code class="language-plaintext highlighter-rouge">add</code> operations:</p>
<ol>
<li>The building tools should ensure that the new tile/s do not intersect any existing geometry</li>
<li>The building tools should ensure that no two tiles in a drag <code class="language-plaintext highlighter-rouge">add</code> operation intersect.</li>
</ol>
<p>For now, we will just leave these points here but we will refer back to them later. One thing of note though is that the ‘does not intersect’ property is something we want to try and maintain. Due to undo/redo it’s pretty easy for people to violate this[0] but the rest of the system should try to maintain this as much as possible.</p>
<p>All right, so our building tools let the player <code class="language-plaintext highlighter-rouge">add</code> and <code class="language-plaintext highlighter-rouge">delete</code> tiles, but we have to maintain this state in memory. Our world (the board in TaleSpire) can be far too big to fit in memory all at once so we divide the world up into <code class="language-plaintext highlighter-rouge">zone</code>s which we can load in and out on demand. Because of this, we will want <code class="language-plaintext highlighter-rouge">zone</code>s to be as independent as possible and so each <code class="language-plaintext highlighter-rouge">zone</code> is going to maintain it’s own tile data.</p>
<p>An <code class="language-plaintext highlighter-rouge">add</code> or <code class="language-plaintext highlighter-rouge">delete</code> of a single tile seems like a simple place to start, it should only affect one <code class="language-plaintext highlighter-rouge">zone</code> right? Well, it depends on the size of the tile. Let’s say our <code class="language-plaintext highlighter-rouge">zone</code>s are 16x16x16 units in size and that the minimum size of a tile is <code class="language-plaintext highlighter-rouge">1x0.25x1</code> (where the 0.25 is the vertical component), well then if a single tile is 40x1x40 then it could be overlapping nine different <code class="language-plaintext highlighter-rouge">zone</code>s. Even a humble 2x2x2 tile can overlap eight <code class="language-plaintext highlighter-rouge">zone</code>s if it is in the corner. For the sake of sanity let us say that a tile can be no bigger than half the size of the zone (8x8x8), this way it can at most overlap eight zones, just like the 2x2x2 tile. We will also leave this issue here for now as we will have enough to think about, however, keep in mind how many little things we are ignoring for the sake of the readability of the article, all of these will need to be resolved before your game ships :p</p>
<p>Back to <code class="language-plaintext highlighter-rouge">add</code>ing tiles! When tiles are placed we need to store data about them, things like the kind of tile it is, the position, rotation, etc. We will just talk about a tile’s data as if it’s a single <em>thing</em> for simplicity[1].</p>
<p>First <code class="language-plaintext highlighter-rouge">player0</code> drags out 3 tiles.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tile-data = [add(* * *)]
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">tile-data</code> is some linear collection holding the tile data and <code class="language-plaintext highlighter-rouge">add</code> is an object holding three tiles represented by asterisks (<code class="language-plaintext highlighter-rouge">*</code>) in this case.</p>
<p>Ok so they <code class="language-plaintext highlighter-rouge">add</code> 2 more tiles</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tile-data = [add(* * *), add(* *)]
</code></pre></div></div>
<p>Neat. Now let’s look at <code class="language-plaintext highlighter-rouge">undo</code>. Well, that’s pretty simple right now as we have been storing our tiles in the order they happened, let’s just drop that most recent add.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tile-data = [add(* * *)]
</code></pre></div></div>
<p>Hmm, but now we need to support <code class="language-plaintext highlighter-rouge">redo</code>, it’s a shame we just threw away the data on the <code class="language-plaintext highlighter-rouge">undo</code>. What if we didn’t, what if we had a little store of things that have been ‘undone’. That means after the <code class="language-plaintext highlighter-rouge">undo</code> the data would look like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tile-data = [add(* * *)]
undone-data = [add(* *)]
</code></pre></div></div>
<p>and now redo is simple too. We pop the data off of the <code class="language-plaintext highlighter-rouge">undone-data</code> stack and push it back onto <code class="language-plaintext highlighter-rouge">tile-data</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tile-data = [add(* * *), add(* *)]
undone-data = []
</code></pre></div></div>
<p>So far so groovy. But now let’s think about deleting tiles. Well, that doesn’t map to our data layout as neatly. Our tile data is organized in chunks that are ordered by when they happened in history, not by where they are in the world. Our <code class="language-plaintext highlighter-rouge">zone</code>s do offer a limited amount of subdivision of the world so currently, this means we need to visit each zone the selection intersects and then scanning through all the tiles in those zones to see which are in the selection. [2]</p>
<p>So about those zones.. how many are going to be involved? Let’s look at this 2d version.</p>
<p><img src="/assets/images/selectZones.png" alt="2d selection example" /></p>
<p>Notice how, as the size increases the number of <code class="language-plaintext highlighter-rouge">zone</code>s we need to consider is going up by the <code class="language-plaintext highlighter-rouge">width x height</code> and it’s more painful in 3D as it’s increasing by <code class="language-plaintext highlighter-rouge">width x height x depth</code>, so roughly cubing with respect to edge length. This is going to be quite a lot of zones to check. But how many tiles can a zone hold? Well a zone is 16x16x16 but a tile can be as small as 1x0.25x1 then that is 16x64x16 tiles 16384tiles…ugh this is getting to be a lot of work.</p>
<p>But a ray of hope appears, notice the hatched areas in the diagram above. Those are zones that are completely enclosed by the selection. Naturally, this means a <code class="language-plaintext highlighter-rouge">delete</code> operation will delete all the tiles within so we have an opportunity for a big optimization. For any zone that is completely enclosed we just remove all tiles, and we only compare the selection with the tiles themselves for zones that are only partially intersected by the selection. In general, this means the outer ‘surface’ of <code class="language-plaintext highlighter-rouge">zone</code>s. The number of zones on the surface of the selection grows roughly in the order of (x^3 - max(x-2, 0)^3) which is much more manageable.</p>
<p>The situation has improved then but we still need to apply the operation to all those zones, and quickly. Luckily you have been watching the recent developments in your game engine of choice and have learned that they have added a neat <a href="https://docs.unity3d.com/Manual/JobSystem.html">job system</a> which can spread work over multiple cores and has a specialized compiler which can optimize the heck out of your code. “Yay” you say, but one limitation all your data need to be <a href="https://docs.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types">Blittable types</a> or arrays of such so our previous ‘list of chunks of tiles’ approach isn’t going to work. Furthermore, it would be neat if the tile data was packed together in memory so you could write that jobs that traverse the whole set of tiles rather than jumping around memory to separate chunks. This becomes important as you watch your players and notice that, whilst they do drag out large chunks of tiles, they also place a lot of tiles one at a time.</p>
<p>Ok so let’s look at our old data layout</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tile-data = [add(* * *), add(* *)]
undone-data = []
</code></pre></div></div>
<p>Now let’s make this a single array, and instead of undo removing one and adding to another we with have an index which we will call the <code class="language-plaintext highlighter-rouge">active-head</code>. It will point to the most recent active chunk of tiles</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [add(* * *), add(* *)]
</code></pre></div></div>
<p>Our <code class="language-plaintext highlighter-rouge">active-head</code> is represented by the <code class="language-plaintext highlighter-rouge">v</code> above <code class="language-plaintext highlighter-rouge">tile-data</code> in code it’ll just be an integer.</p>
<p>With that change undo looks like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [add(* * *), add(* *)]
</code></pre></div></div>
<p>and redo does this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [add(* * *), add(* *)]
</code></pre></div></div>
<p>Cool, but now to remove those extra objects, let’s flatten the array and move the layout information to a separate array</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [* * * * *]
v
layout = [3 2]
</code></pre></div></div>
<p>So far so good. Now our layout tells us how many tiles are in each chunk and <code class="language-plaintext highlighter-rouge">tile-data</code> stores all the tiles packed together.</p>
<p>Undo now looks like this…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [* * * * *]
v
layout = [3 2]
</code></pre></div></div>
<p>and redo is still the reverse. This means undo and redo are still as simple as changing an integer from the data-model side[3]</p>
<p>Delete is different though, delete is based on a region on the board so we need to check each tile to see if it intersects with that region and, if it does, then we remove it from the <code class="language-plaintext highlighter-rouge">tile-data</code> array. Undo naturally requires us to put that data back. Now from a data angle that is annoying as it could leave ‘holes’ in our array as in the following diagram (deleted tiles represented by <code class="language-plaintext highlighter-rouge">x</code>)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [x * x x *]
v
layout = [3 2]
</code></pre></div></div>
<p>This is no good as now we have to handle this. We could have each tile have a flag that says if it’s alive or dead. We would then need to check that any time we scan over this array. This is ok but, unless we undo the delete, we still have memory being wasted as it’s occupied by data for tile that been deleted. So maybe we reuse the gaps? Sounds great, but we can’t fill in the gaps with tiles from newer operations as otherwise, we lose the property that tile-data is in ‘history order’ and thus lose the ability for undo/redo to be represented by the <code class="language-plaintext highlighter-rouge">active-head</code>.</p>
<p>So if we don’t want holes but can’t fill them then another option is to compact the array as we delete, so whenever tile data is deleted the tile data to the right is shifted left to fill in the gaps. This requires more copying during the delete but let’s assume we profiled it and it proved to be the best option[4]. This is what we will assume for the rest of the article.</p>
<p>Let’s now look at the above <code class="language-plaintext highlighter-rouge">delete</code> operation with compaction.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [* *]
v
layout = [1 1]
</code></pre></div></div>
<p>Neat! but we should note we have complicated <code class="language-plaintext highlighter-rouge">undo</code> for ourselves. If we undo a delete We will have to ‘make room’ in tile-data for the old tiles to be copied back into. There is one blessing though, we only care that chunks are in history order, not the tiles within the chunks. That means we don’t need to make room in the same places as we had the tiles originally…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [_ * _ _ *]
v
layout = [3 2]
</code></pre></div></div>
<p>Rather, we can make room at the end of each chunk…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [* _ _ * _]
v
layout = [3 2]
</code></pre></div></div>
<p>And still keep all the undo/redo benefits we had previously with the active-head approach.</p>
<p>This also simplifies redo of a delete, can you spot how?</p>
<p>Originally we had to scan through the tiles to find the ones to delete. Redo requires us to delete those same tiles again but we know exactly where they are now, they are at the end of each chunk.</p>
<h3 id="chapter-2---multiple-players-and-a-quandary">Chapter 2 - Multiple players and a quandary</h3>
<p>Now things get ‘fun’. We can have up to 16 players on a board at a single time and all of them could be given the permissions to build. That means we need to keep that all in sync on all machines but all make sure everything feels instant.</p>
<p>Let’s get one thing out of the way quickly, order matters. By this I mean the obvious quality that an <code class="language-plaintext highlighter-rouge">add</code> followed by a <code class="language-plaintext highlighter-rouge">delete</code> can give a different result from a <code class="language-plaintext highlighter-rouge">delete</code> followed by an <code class="language-plaintext highlighter-rouge">add</code>, from this we know we need to apply changes in a fixed order. For us, that means when we build we will send a request to a ‘host’ which will decide what gets applied in what order and tells the other player’s games (called a <code class="language-plaintext highlighter-rouge">client</code> from now on). Note that we also need to see a change the moment it happens so the order will be:</p>
<ul>
<li>Player tries to make a change</li>
<li>The local client applies the change immediately and sends a change request to the host</li>
<li>the local client listens for change acknowledgments from the host</li>
<li>if the acknowledgement is for the thing we just built then we are done</li>
<li>if the acknowledgement is for another client’s change then we undo our change, apply their change and then reapply our change</li>
</ul>
<p>Now that is very briefly covered we will assume that things are applied in the same order on each client and ignore that aspect of the problem for the rest of the article.</p>
<p>To keep everything in sync we need to be able to send information about what is being done by each player. We already know that our selections could be very large so we don’t want to have to send information about every specific tile that was modified, so instead we will want to send just what happened e.g. ‘player 0 added 30 wall tiles in this area’. This is ok for us as we are already assuming a fixes order of operations so we can assume that every player has the board in the same state when they apply the change.</p>
<p>Multiple players building also gets interesting due to each player has their own undo/redo history. This is very important as shared histories get <em>very</em> annoying <em>very</em> quickly. However, this messes with our little data layout from before. In the following layout I have changed out the asterisks for a designation that tells us who added the tile, this is only for our benefit in this article, it needed to be encoded in the actual data like this.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
tile-data = [p0 p0 p1 p1 p0 p0]
v
layout = [2 2 2]
</code></pre></div></div>
<p>So three chunks of tiles have been added. First <code class="language-plaintext highlighter-rouge">player0</code> (<code class="language-plaintext highlighter-rouge">p0</code>) added 2 tiles, then <code class="language-plaintext highlighter-rouge">player1</code> (<code class="language-plaintext highlighter-rouge">p1</code>) added 2 and then <code class="language-plaintext highlighter-rouge">p1</code> added 2 more.</p>
<p>Now let’s say <code class="language-plaintext highlighter-rouge">p1</code> hits undo, that means the middle two tiles need to be undone but we can’t do this just by shifting the active head.</p>
<p>Here we face two choices, either we drop the <code class="language-plaintext highlighter-rouge">active-head</code> approach or keep it and find a way around the above issue. Neither way is wrong and both require changes to how the data will be stored. For the sake of the article, I am just going to continue with the active head approach so we can see how this has to be modified to keep its nice properties in the face of multiple players.</p>
<h3 id="chapter-3---multiple-players-with-the-active-head-approach">Chapter 3 - Multiple Players with the active-head approach</h3>
<p>OK, so let’s say we like the way we can just change a single integer to represent undo/redo using the active-head approach. What would be needed to keep this working with multiple players?</p>
<p>Well each player could have their own tile and layout data</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
p0-data = [p0 p0 p0 p0]
v
p0-layout = [2 2]
v
p1-data = [p0 p0]
v
p1-layout = [2]
</code></pre></div></div>
<p>This lets us regain the quality that undo for <code class="language-plaintext highlighter-rouge">p1</code> only affects <code class="language-plaintext highlighter-rouge">p1</code>’s data.</p>
<p>From now on let us say that if a tile is in <code class="language-plaintext highlighter-rouge">p0-data</code> then <code class="language-plaintext highlighter-rouge">player0</code> ‘owns’ that tile and if the tile is in <code class="language-plaintext highlighter-rouge">p1-data</code> then <code class="language-plaintext highlighter-rouge">player1</code> ‘owns’ that tile. With that little bit of terminology, we can look at delete again.</p>
<p>Delete is based on a selection of tiles on the board, it does care about who ‘owns’ the tile behind the scenes; if the tile is in the selection, it will be deleted. That means that when we scan the data for tiles to delete we need to scan each player’s tile array.</p>
<p>Remember how undo of a delete needed to restore tiles to where they came from? Well, what do we do for tiles that we delete from other players? It turns out that adding them back to the other player’s chunks (and thus their history) results in behavior that is fairly confusing to play as the result of undo or redo depends to an extent on the current state of another player’s undo or redo. How robust a game feels depends a lot on to what degree a player’s expectations of game behavior match reality and so it’s sometimes better to have a simpler behavior even if a more correct but complicated option is available. I’m going to claim this is such a case without trying to prove it here :)</p>
<p>The result of this is that we don’t want to try to push undone deletes back into the data arrays of other players. So where do we put them? How about the end of our data section? This is a bad idea as is breaks the active-head approach again. So for us to continue using this approach we now need an additional store per player for undone deletes.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> v
p0-data = [p0 p0 p0 p0]
v
p0-layout = [2 2]
p0-deletes = []
v
p1-data = [p0 p0]
v
p1-layout = [2]
p1-deletes
= []
</code></pre></div></div>
<p>The addition of more per player data probably feels annoying (it did to me) but one upside is that this array can use the same active-head tricks that we have used for the player’s data array but with a slight difference, the tiles exist in-game only if they are to the right of the active-head. In this case:</p>
<ul>
<li>a delete of tiles owned by other players pushes them onto the player’s <code class="language-plaintext highlighter-rouge">deletes</code> array and advances the active-head</li>
<li>an undo of the delete shifts the active-head to the left, the things to the right are now active in-game</li>
<li>a redo of the delete shifts the active-head right again</li>
</ul>
<h3 id="conclusion">Conclusion</h3>
<p>This might feel like it’s all getting a bit out of hand. That is a fair conclusion to make! This is why I said at the beginning this writeup is just an exercise in following an idea through to where it leads rather than making a call on the best approach. I hope you have some thoughts on this and also a feeling for where this kind of feature can get complicated.</p>
<p>I have left out a bunch of things like “how do we maintain the mapping between the tile data my game engine’s game objects when our data can only contain Blittable types?” and “How long can undo/redo histories be?”. They are fun in themselves as I do recommend considering them if you decide to try and come up with your own schemes.</p>
<h3 id="notes">Notes</h3>
<p>[0] For example imagine <code class="language-plaintext highlighter-rouge">player0</code> placing a block of tiles, then <code class="language-plaintext highlighter-rouge">player0</code> deletes those tiles, then <code class="language-plaintext highlighter-rouge">player1</code> places some tiles in that (now empty) spot. Finally <code class="language-plaintext highlighter-rouge">player0</code> undoes their delete and suddenly <code class="language-plaintext highlighter-rouge">player1</code>’s tiles intersect with <code class="language-plaintext highlighter-rouge">player0</code>’s.</p>
<p>[1] This means we are ignoring both the fields within and also the various possible layouts for those fields.</p>
<p>[2] We could of course store the ‘bounds’ (a 3d region) of the tiles organized in a way that makes spatial queries easier, and then have those contain indexes back to the tiles they refer to. This will mean sorting/maintaining this structure as tiles are added/deleted, so it’s not a free lunch but you will want to look at this if you are building this yourself.</p>
<p>[3] you still need to handle the relationship between the data and actual <code class="language-plaintext highlighter-rouge">GameObject</code>s or entities (depending on the system you are using)</p>
<p>[4] remember that building only happens as fast as humans can click, whereas some things might be happening every frame (60 times a second). It can prove better to prioritize the more common case over the less frequent one even if it does result in a slight drop in performance for the less common case.</p>
TaleSpire Dev Log 1262019-09-03T10:31:48+00:00http://www.techsnuffle.com/2019/09/03/talespire-dev-log-126<p>Current dev work is focused on the new building systems.</p>
<p>My goal is to have the basics of the data model worked out locally by the end of the week. Most of this right now is focused on how best to handle undo/redo so that it’s fast but also simple to synchronize. I had previously designed had some schemes for this but some tools we’d like to use are <a href="https://issuetracker.unity3d.com/issues/dots-ecs-hybrid-renderer-materialpropertyblock-instanced-data-does-not-get-correctly-updated">currently still in preview</a> and some would require significant engineering to work around (In this case driving rendering ourselves would require a lot of work on the lighting system) and we don’t have time for that.</p>
<p>@Ree has been getting back into prototyping the bit that players interact with, the tools and game feel of building. Progress is being made in both areas but nothing worth showing yet.</p>
<p>Thanks for dropping by! More updates soon.</p>
TaleSpire Dev Log 1252019-08-28T18:43:26+00:00http://www.techsnuffle.com/2019/08/28/talespire-dev-log-125<p>Whyyyyy, WHY did I say on discord that the next alpha update would be ‘soon’? It angered the demo-gods and we’ve been dealing with an asset-loading problem ever since.</p>
<p>That issue has now been fixed and I feel like we know when that update will be landing. However I am not evening hinting at when that could be given what wrath that might incur.</p>
<p>We have also been moving our assets out of Unity Collab. We have been using Collab for a while but there are aspects to it that make it painful for teams and there has been no sign that it will improve. For now we are using Git LFS which is what we have been using for the game itself for over a year. Git LFS, once set up, it’s been pretty good to us so far.</p>
<p>The money from the Kickstarter has landed and @Ree has been busy working on the business side of things. We are looking forward to both being able to say we are officially hired by Bouncyrock and also to give more little bits of info regarding what happens to money after you make your pledge.</p>
<p>I (Baggers) have just got back from a long weekend with family in the west of Norway. The last year and a half has been a lot of work so it’s really nice to be seeing folks again.</p>
<p>More news as it happens!</p>
TaleSpire Dev Log 1242019-08-15T23:40:30+00:00http://www.techsnuffle.com/2019/08/15/talespire-dev-log-124<p>It’s been a nice day today, both @Ree and I got some coding done. Here are a few highlights:</p>
<ul>
<li>
<p>Dice feel regressed a while back and Ree got that fixed back up again.</p>
</li>
<li>
<p>He also updated the implementation of the radial menus which we hope has fixed the issues with some stats not showing. I tested this but we had never been able to get a reliable reproduction of the issue so we will see how this fix holds up in the wild.</p>
</li>
<li>
<p>I found a <em>interesting</em> default behavior in the networking library we used which was resulting in huge hangs when updating certain player information. Once found the fix was trivial.</p>
</li>
<li>
<p>We have enabled Unity’s incremental garbage collector to smooth over another GC spike we were seeing. It’s very much a band-aid but it’s a nice one for the alpha before we get to addressing the core issues behind the allocations.</p>
</li>
</ul>
<p>More news and an update to the alpha in the near future.</p>
<p>Peace</p>
TaleSpire Dev Log 1232019-08-13T22:56:52+00:00http://www.techsnuffle.com/2019/08/13/talespire-dev-log-123<p>Day two of the dev planning and again things went well. The main job today was going back through the task lists and working out which ones would be dependent on which. This is important so we don’t get blocked by each other and that, for the cases where the is a dependency, there will be work the blocked person can be doing.</p>
<p>We wont be putting out these orders as they will change and we’d like to keep things flexible.</p>
<p>Following on from yesterdays work I also got the backend running fully without and internet connection so I’m hoping that with a little more work I can point Unity at my laptop and run the whole TaleSpire stack locally. This will be a big win for iteration time when developing the backend.</p>
<p>Lastly and most prettily @Dwarf has made an awesome looking Kobold which we will be seeing in game shots of in the near future.</p>
<p><img src="/assets/images/kobold.gif" alt="yay" /></p>
<p>That’s all for today,</p>
<p>Peace</p>
TaleSpire Dev Log 1222019-08-12T18:35:16+00:00http://www.techsnuffle.com/2019/08/12/talespire-dev-log-122<p>The end of last week saw me playing with a cool toy so I thought I’d write about it.</p>
<p>First though, I’ve carried on with my digging into various erlang bits and bobs.</p>
<p>With a bit of prodding I was able to get <a href="https://github.com/oltarasenko/epmdless">epmdless</a> working with my erlang/docker setup. This is great as <a href="https://learnyousomeerlang.com/distribunomicon">distributed erlang</a> is one of the powerful features that many erlang libraries lean on and previously this was hampered by docker requiring explicit port mappings. If you are interested in epmdless in conjunction with docker you can check out <a href="https://www.erlang-solutions.com/blog/running-distributed-erlang-elixir-applications-on-docker.html">this article here</a>.</p>
<p>I also spent some quality time getting to know <a href="https://github.com/uwiger/gproc">gproc</a>, <a href="https://github.com/cabol/erlbus">erlbus</a>, websockets a little better and I’m much happier now with where I’d use them in my projects. I’ve not got much more concrete to say on these but having a good understanding of common tools is helping with the design of the next iteration of the backend (more news on this in future too).</p>
<p>Lastly as fun bit (for me). For a time I was really enjoying that I was able to use docker to test the TaleSpire backend locally. I had webservers, db servers etc on their own little network and I could get a reasonable idea of how things should run. However at one point we started using amazon’s S3 for storage and that put a spanner in the works as I couldn’t test that locally. I could make a alternative approach for local dev but that’s more code to maintain and that could fail in ways that confuse my testing. Luckily there is a project called <a href="https://docs.min.io/">minio</a> which runs in a docker container and presents the same api as S3. I’ve already modified the image to include my preferred tweaks and have tested making presigned urls which work <strong>GREAT</strong>.</p>
<p>So now I get to go remove some ‘local dev’ hacks and I get a simpler, more realistic, fully local, dev environment. LOVE IT.</p>
<p>Alright, that’s all the fun nerdage for now, back to planning :)</p>
<p>p.s.
One thing I got stuck on initially with epmdless was that I was using <code class="language-plaintext highlighter-rouge">rebar3 shell</code> in the entrypoint script for my container. This will not work with epmdless as the epmd module vm args would need to be passed to shell and that would then freak out as it wouldnt know where to find the epmdless module. So instead try having the entrypoint script make a debug release of your erlang app and then start that in foreground mode. Then your <code class="language-plaintext highlighter-rouge">config/vm.args</code> will be used and everything will work fine.</p>
<p>Also, remember to expose the remsh port for epmdless so you can use the remote shell. I used this for my erlang repl settings in emacs. Once you have the official examples running most of this should be fairly self explanatory.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setq inferior-erlang-machine-options
'("-env" "EPMDLESS_DIST_PORT" "18999"
"-env" "EPMDLESS_REMSH_PORT" "18000"
"-name" "emacs@vanguard0.com"
"-setcookie" "cookie"
"-remsh" "myapp@myapp.com"
"-proto_dist" "epmdless_proto"
"-epmd_module" "epmdless_client"
"-pa" "/app/_build/default/lib/epmdless/ebin/"))
</code></pre></div></div>
TaleSpire Dev Log 1212019-07-29T15:31:15+00:00http://www.techsnuffle.com/2019/07/29/talespire-dev-log-121<p>The end of last week looked at lot like this:</p>
<p><img src="/assets/images/undo-redo.png" alt="undo/redo lisp test" /></p>
<p>This was because I want sketching out ideas for the undo/redo system and I wanted a nice simple model for doing this. This is 4x4 map where tiles are represented by letters.</p>
<p>These tests allowed me to find small mistakes in my assumptions and to fix those in a much simpler (and less distracting) environment than doing the work in Unity. It was made it trivial to test out how I want to resolve the conflict between needed to apply changes immediately and needing to apply changes in the same order on all clients to get the right result.</p>
<p>I was using lisp for this as it’s where I’m most comfortable and also because the ease of recompiling smaller chunks of code just makes for nice fast iteration times.</p>
<p>Since then I’ve been looking at fog of war but that has mainly been prep for the planning sessions that will take place when Ree is well. So nothing to show there is it mainly amounts to lots of pages of my notebook being covered in scribbles :)</p>
<p>For now, that’s the update.</p>
<p>Keep an eye out for more news coming soon!</p>
TaleSpire Dev Log 1202019-07-26T08:48:45+00:00http://www.techsnuffle.com/2019/07/26/talespire-dev-log-120<p>Heya folks, behind the scenes I’ve been working on the next version of the undo/redo system along with new data layouts to make batching (for rendering) faster.</p>
<p>Undo/redo is one of the areas we’ve had serious lag before when working with large slabs of tiles. When looking at the code it was understandable why, however, there are interesting complexities to the system when you have multiple people adding and deleting tiles concurrently.</p>
<p>For the last two days, I’ve been working on a new design for the system. When doing we also have to make sure that the data layout we choose for this doesn’t make it slow to perform frequent tasks like rendering.</p>
<p>This is one of the bits I love with this job, sitting down with a pen and paper and just wrestling with a design.</p>
<p>I think I have something now but I’m not going to put it here yet as it needs some further validation. What I’ll do next is code a small prototype so I can see the properties of the system more clearly.</p>
<p>More details on that in the coming days :)</p>
<p>Peace.</p>
TaleSpire Dev Log 1192019-07-22T22:52:15+00:00http://www.techsnuffle.com/2019/07/22/talespire-dev-log-119<p>Hello folks,</p>
<p>Research continues and, for now, that isn’t giving me much to show so I took an hour out to explore a requested feature: NDI Support.</p>
<p><a href="https://ndicentral.com/">NDI</a> is a standard that lets applications deliver video streams via a local area network. For us, we are interested in being able to take these video feeds and use them in TaleSpire. This originally came up as a request from the community as it is apparently a popular way for streamers to integrate video (such as their player’s skype chat) into their streams.</p>
<p>It sounded cool and luckily where is already a <a href="https://github.com/keijiro/KlakNDI">Unity package available</a> for working with NDI. It worked like a charm so HUGE props to keijiro for that great work.</p>
<p>This was a very limited test but it is encouraging. We would need to do a bunch more work to be comfortable shipping this and we wouldnt want to mess with the roadmap we have promised the Kickstarter backers. All that said this was a fun test and one I’m excited to revisit if it makes sense later on.</p>
<p>That’s all for now</p>
<p>Peace!</p>
<p><img src="/assets/videos/ndi0.gif" alt="test" /></p>
TaleSpire Dev Log 1182019-07-16T00:07:04+00:00http://www.techsnuffle.com/2019/07/16/talespire-dev-log-118<p>Another short dev log while we are still in the Kickstarter campaign.</p>
<p>Recently I’ve carried on studying. It looks like the hybrid renderer doesn’t handle per-instance data yet (which is mad) so we won’t be able to use that yet. This means we will probably want to avoid Unity’s ECS for now and just work with something similar but custom until that stack matures. This is no real issue as although we will have to do a little bit of tooling work when we need insight we do have experience in that.</p>
<p>Ree has always been the lead dev when working with graphics and shaders in Unity as he has many years of experience beyond me. I (Baggers) come with a decent amount from the GL side though so I’ve spent some time getting more used to hlsl and tooling we’ve been using. It’s fun, a lot of things are set up for you of course so I’m not sure (without the scriptable rendering pipeline) how one really packs data or does very custom stuff, but there’s plenty of time to learn that and the team has bags of experience I can lean on. This is really just making sure we are cross skilled enough that we don’t bottleneck each other too much (although when it comes to content creation I’m hopeless :D)</p>
<p>That’s all for today,</p>
<p>Peace</p>
TaleSpire Dev Log 1172019-07-09T02:50:48+00:00http://www.techsnuffle.com/2019/07/09/talespire-dev-log-117<p>A quick update for today.</p>
<p>The day was mostly spent reading up on the new <a href="https://docs.unity3d.com/Packages/com.unity.physics@0.0/manual/index.html">Unity.Physics package</a> to see if it will be viable for use in TaleSpire. It is a preview package and that does come with a level of risk, however we only use the currently physics system for dice and for casting rays against for tools. This means our requirements are very simple and even at it’s current state the package has what we would need, assuming of course that it works as documented.</p>
<p>It certainly has some surprising aspects such as how it’s not <a href="https://forum.unity.com/threads/issues-with-physics-velocity-in-ecs.692953/">framerate independent by default</a>, but handling that seems to be easy enough.</p>
<p>Peace</p>
TaleSpire Dev Log 1162019-07-03T23:33:28+00:00http://www.techsnuffle.com/2019/07/03/talespire-dev-log-116<p>Heya folks, yesterday and today I have spent getting familiar with Unity’s Job Systems and new ECS.</p>
<p>I’ve written toy ECS’ before and am currently working on a little optimizing compiler for querying table data so a lot of things were very familiar. The bulk of the video and prose content for Unity’s new systems is focused on the fact that the new thing is ‘different, but don’t worry it’s fast and not that hard’ etc, etc. As I’m already sold on the premise this is nice but of limited here. Beyond that, you better be comfortable reading other people’s code :p</p>
<p>The best resource so far has been the <a href="https://github.com/Unity-Technologies/EntityComponentSystemSamples">ECS Samples</a>. It’s a small of samples evolving a piece of code from a <code class="language-plaintext highlighter-rouge">foreach</code> on the main thread, through different flavors of jobified tasks. One of the biggest advantages of the code is just seeing what is still in use. If you glanced at the <a href="https://github.com/Unity-Technologies/EntityComponentSystemSamples/blob/master/Documentation%7E/cheatsheet.md">cheatsheet</a> you’d be missing certain things (like sync points).</p>
<p>The reference docs are passable but there are plenty of things where the description of a method is the name. When I hit walls in C# I often read Mono’s source to get an idea of what to expect, but naturally many of the things in the new ECS are either proprietary or are in c++ code I can’t reach easily so those missing doc-strings become a pain point.</p>
<p>Given that, in TaleSpire, we load most assets dynamically I’d like to avoid having to use the Entity conversion routines on load of each asset, I’m fine with running them in TaleSpire and serializing <em>something</em> useful, but I’m not sure how we prefabs play with ECS entities at this stage. I’d be very surprised if Unity’s mega-city demo was using conversion routines so I’m sure there is still more to understand here. Also the lack of documentation around the <code class="language-plaintext highlighter-rouge">Hybrid.Renderer</code> is a pain in the proverbials, I’d really like to look into zone culling in TaleSpire to potentially improve rendering performance but information seems scarce here. I’m sure it’s all in the mega-city demo but that doesn’t feel like the best introductory material to be using.</p>
<p>Regardless, between the samples and <a href="https://docs.unity3d.com/Packages/com.unity.entities@0.0/manual/ecs_entities.html">these docs</a>, I’m making decent enough progress. We’ll definitely be using this in TaleSpire and I’ll have this under my belt well before the campaign ends.</p>
<p>Peace</p>
TaleSpire Dev Log 1152019-07-01T21:40:22+00:00http://www.techsnuffle.com/2019/07/01/talespire-dev-log-115<p>Let’s get back into the swing of these!</p>
<p>During the Kickstarter I’m using any free time I have to study and make little systems to get familiar with systems we may end up using</p>
<p>Today I’ve mainly been getting familiar with <a href="https://docs.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types">blittable types</a>. The reason for this is that Unity’s <a href="https://unity.com/dots">DOTS</a> primarily operates on them.</p>
<p>I got interested in how easy it would be to de/serialize such data, given these types can have precise memory layouts specified for them. The that end I was making little <code class="language-plaintext highlighter-rouge">BinaryWriter</code>s and readers that had Write methods that could take any blittable type or array of blittables and handle them correctly.</p>
<p>I’m going to make a dump of some useful links as most of the stuff I was doing was simple enough, but was a good exercise for familiarity. It was the first time I’ve significant use of pointers in c# and it was a very simple and unsurprising experience.</p>
<p>The first handy thing was that the <a href="https://devblogs.microsoft.com/premier-developer/dissecting-new-generics-constraints-in-c-7-3/">unmanaged constraint</a> is available. Whilst unmanaged types are not the same as blittable types there is a useful overlap. Using this allows us to have signatures like <code class="language-plaintext highlighter-rouge">unsafe void Foo<T>(T value) where T : unmanaged</code>. Also note that <code class="language-plaintext highlighter-rouge">sizeof</code> is defined to work on unmanaged types. It was very easy to write generic methods doing useful stuff with c#, I’m please with it so far.</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/fixed-statement">Scoped GC pin</a></li>
<li><a href="http://msdn.microsoft.com/en-us/library/s69bkh17.aspx">Heap allocate</a>, check out all of Marshal, it’s pretty handy</li>
<li><a href="https://aakinshin.net/posts/blittable/">more info on blittables</a></li>
<li><a href="https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.marshal.ptrtostructure?view=netframework-4.8#System_Runtime_InteropServices_Marshal_PtrToStructure_System_IntPtr_System_Object_">create a struct from data referenced by pointer</a></li>
<li>You can explicitly cast typed pointers to <code class="language-plaintext highlighter-rouge">IntPtr</code></li>
<li><a href="https://docs.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.unsafe?view=dotnet-plat-ext-2.1">Unsafe</a> seems like a handy class until you see that it’s <a href="https://github.com/mono/mono/blob/master/mcs/class/corlib/System.Runtime.CompilerServices/Unsafe.cs">not implemented in mono</a></li>
<li>But then you find <a href="https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.UnsafeUtility.html">UnsafeUtility</a> in Unity itself and it’s way better. <code class="language-plaintext highlighter-rouge">memcpy</code>, <code class="language-plaintext highlighter-rouge">malloc</code> etc are all there</li>
</ul>
<p>The big disappointment of the day is how quickly one ends up having to reinvent the wheel as most Streams, Buffers, Writers, etc, never expose their buffer or provide overloads taking <code class="language-plaintext highlighter-rouge">IntPtr</code>. The more you poke to work around this the more it feels like you should just make your own. It’s not ideal. Saying that, I have spent most of today in dotnet code rather than Unity’s so I do still need to see what is there.</p>
<p>Until tomorrow,
Ciao.</p>
TaleSpire Dev Log - 18 hours until the Kickstarter2019-06-23T14:14:45+00:00http://www.techsnuffle.com/2019/06/23/talespire-dev-log--18-hours-until-the-kickstarter<p>So here we are! The clock ticks and, baring any disaster, in the next 18 hours the Kickstarter will go live.</p>
<p>We are currently beavering away shining edges and triple checking wording.</p>
<p>It’s an odd feeling to be this close to finding out the world makes of this but it’s a privilege to be here.</p>
<p>Thank you to everyone that showed us that they were interested in this.
Thank you to the alpha testers who delved deep and proved the limitations of the systems
And thank you to you for reading these little posts.</p>
<p>The new day is upon us,
The gates are soon to open,
Let us see what is on the other side</p>
<p>Warmest Regards,</p>
<p>Dwarf, Baggers & Ree
(The rather sleepy Bouncyrock crew)</p>
TaleSpire Dev Log 1132019-06-11T23:42:16+00:00http://www.techsnuffle.com/2019/06/11/talespire-dev-log-113<p>Not really a dev log as right now there’s not a whole lot of dev going on. I’ve been just sitting writing an rewriting bits of the kickstarter page. It’s an odd feeling as I flit between thinking ‘ah your thinking about it too hard’ and thinking ‘man this kickstarter decides whether I have to refresh my CV soon’. Heavy.</p>
<p>It’s odd writing this when there isn’t much to report. Assets for the trailer have been made, the trailer itself is going well. Really I’m writing as it feels weird going a few days without reporting the latest to you.</p>
<p>We’ll be letting you know the kickstarter date one week before it happens.. which is soon.</p>
<p>Peace folks</p>
TaleSpire Dev Log 1122019-06-07T01:51:05+00:00http://www.techsnuffle.com/2019/06/07/talespire-dev-log-112<p>Heya all, time for a quick behind the scenes again.</p>
<p>Those who have been keeping count since the dev stream will know that we are only a couple of weeks to go before we should be doing our kickstarter. We really need to focus on this so we will be pretty much stopping updates to the game for the next couple of weeks. I’m going to get the ‘duplicate board’ feature shipped this weekend and then I hope to fit in some bugfixes, but I think this is what we need to do to make sure we get everything done.</p>
<p>Other than that I finished v1 of the ‘duplicate board’ feature mentioned above. It’s a little rough around the edges but nothing not acceptable for an alpha. We just need to make an icon for that now and it’s ready to ship.</p>
<p>Not much more to say other than thank-you once again for caring so much about our little game.</p>
<p>We’ll do our best.</p>
TaleSpire Dev Log 1112019-06-06T11:40:33+00:00http://www.techsnuffle.com/2019/06/06/talespire-dev-log-111<p>I was too tired to writeup last night’s dev log so here it is a little late.</p>
<p>I really want to get a user visible feature out this week so I implemented ‘duplicate board’. This will let you click on an icon in the board panel and duplicate the chosen board. This will be handy for a lot of cases but I love the idea of making a town, copying it and then destroying the town for the players to visit again <em>after the calamity</em>.</p>
<p>The hard parts had been researched or implemented when I was looking into the S3 part of cleaning up old board files. All that was left to do was write the database queries and expose it to the game itself.</p>
<p>With the core feature now working I just need to do cleanup and handle the fact that it currently doesnt copy the board description (an easy fix).</p>
<p>Back later today with more news</p>
TaleSpire Dev Log 1102019-06-04T00:11:26+00:00http://www.techsnuffle.com/2019/06/04/talespire-dev-log-110<p>Good-evening folks,</p>
<p>Today I spent most of my time working on the code that deletes old boards from the backend storage.</p>
<p>Every time you load a board TaleSpire takes a new snapshot of the board that you then modify during that session. We keep hold of these snapshots so that we can at some point give you the ability to roll back to one of these backups in case something drastic happens.</p>
<p>Of course this means we are slowly accruing more and more board snapshots so they have to be cleared up. The goal was to keep only the 10 most recent snapshots. Any older than that will be deleted.</p>
<p>Now even though a snapshot is old it would still be good to be able to recover it for a short window of time so we have set up ‘buckets’ on S3 which auto-delete files that have been in there more than 14days. This means the task is to identify the old board snapshots and move them to these buckets.</p>
<p>As this operation needs to touch every row in the ‘board files’ table in the database it could be rather costly so we are using postgresql’s <a href="https://www.postgresql.org/docs/9.3/rules-materializedviews.html">materialized views</a> to help with this. This lets us chose when to update the view at the cost of it being out of date after that. This is fine for us as the worst that happens is that we don’t delete a few snapshots as soon as we could, it will never result in us deleting something we shouldn’t.</p>
<p>After that it was a case of setting up the erlang process that would take care of this at regular intervals. I got the hard part done, namely identifying and moving the files, but for whatever reason the event isn’t firing automatically. This will be some simple mistake on my part so I’m not worried about fixing that relatively soon.</p>
<p>This also means I’ve tested the bulk of the code I would need to implement ‘duplicate board’ so if all goes well I’ll get that out this weekend.</p>
<p>Until next time,</p>
<p>Peace.</p>
TaleSpire Dev Log 1092019-06-01T18:39:09+00:00http://www.techsnuffle.com/2019/06/01/talespire-dev-log-109<p>Heya folks, @Baggers here again,</p>
<p>I didn’t write a log last night as what I thought was a quick nap on the sofa turned into the kind of sedation usually reserved for surgery.</p>
<p>On Friday I wrapped up the experiments I was doing with resource loading in Unity. I was able to load the icons and index of assets at startup which significantly dropped the pause at first campaign load. The branch i was doing the experiments on is rather a mess so I’ll need to clean that up but it looks promising.</p>
<p>I also took the morning to take on some contracting work. I’ve done these very rarely since starting on TaleSpire but it keeps bread on the table. Here’s to the Kickstarter making that unnecessary :)</p>
<p>Behind the scenes @Ree has been doing great work on the campaign trailer and the assets and tech surrounding that.</p>
<p>Right, back to the weekend!</p>
<p>Peace.</p>
TaleSpire Dev Log 1082019-05-30T01:46:15+00:00http://www.techsnuffle.com/2019/05/30/talespire-dev-log-108<p>Alright today I was mainly experimenting with asset loading. We currently have a big pause when we load assets and so naturally we need to fix that. Some of it is just moving things earlier, spreading things over frames etc, but we need to try this stuff out to see what works.</p>
<p>There’s not really a lot more to tell for now. Research is cool but make for pretty generic dev logs ‘TODAY I DUN STUFF’.. good job me.</p>
<p>Tomorrow is a national holiday but we’ll be back with more on Friday.</p>
<p>Peace!</p>
TaleSpire Dev Log 1072019-05-29T00:12:40+00:00http://www.techsnuffle.com/2019/05/29/talespire-dev-log-107<p>Heya folks tiny log today as I don’t have much of note to report, lots of poking around with Unity and reading up on a bunch of things we may need networking wise in future.</p>
<p>The good thing is that the dungeon tile set is progressing well. It’s been delayed by wanting to add more texture and variation to it but hopefully we get something out tomorrow.</p>
<p>I best run for now.</p>
<p>Peace.</p>
TaleSpire Dev Log 106 - Back to the back2019-05-27T20:06:37+00:00http://www.techsnuffle.com/2019/05/27/talespire-dev-log-106--back-to-the-back<p>Today started off with looking into how we are going to handle copying and sharing boards. We store all the board files on S3 so the first part of the job was to look into how to copy/delete etc files on s3 from erlang.</p>
<p>I’ve been using this nice little library called <a href="https://github.com/chef/mini_s3">mini_s3</a> so far. It has done all the hard work but there are a few places where certain options have not been handled. For example the ec2 instance gets temporary permission to s3 from it’s IAM role and so when we make requests we need to pass along the <code class="language-plaintext highlighter-rouge">x-amz-security-token</code>. The functions in <code class="language-plaintext highlighter-rouge">mini_s3</code> don’t handle that yet.</p>
<p>I have <a href="https://github.com/cbaggers/mini_s3/commits/master">forked the library</a> and have added support for the amz token to <code class="language-plaintext highlighter-rouge">s3_url</code> and <code class="language-plaintext highlighter-rouge">list_objects</code>. I’ll be adding support to more functions soon but <code class="language-plaintext highlighter-rouge">list_objects</code> served as a good test.</p>
<p>For the rest of the day I was working on some other experiments that aren’t worth rambling about.</p>
<p>Seeya in the next dev log</p>
TaleSpire Dev Log 105 - Revenge of the return of the reawakening of the dev log2019-05-21T21:53:21+00:00http://www.techsnuffle.com/2019/05/21/talespire-dev-log-105--revenge-of-the-return-of-the-reawakening-of-the-dev-log<p>Hello! we return to our intrepid dev log with the least dev’y dev log of them all.</p>
<p>Today I’ve re-read all of the #feature-request channel and broken it down into 150 (suspiciously round but that was the actual number) separate requests.</p>
<p>Ree and I then filtered them into piles of things we will be doing, things that will be addressed in some other form and things that don’t fit TaleSpire.</p>
<p>That list was then pounded with hammers until merged with our internal feature list and a Kickstarter feature list was born! This is one of the main things we’ll be talking about in the dev stream we are hosting tomorrow. You can find the event for that <a href="https://www.twitch.tv/events/DS_alf9EQkuaNKCj3cv5MA">here</a></p>
<p>I’ve then continuing to prepare for the stream. It’s nice to see this taking shape and I’m looking forward to sharing all this stuff with you tomorrow.</p>
<p>Until then</p>
<p>Peace</p>
TaleSpire Dev Log 1042019-05-06T22:56:35+00:00http://www.techsnuffle.com/2019/05/06/talespire-dev-log-104<p>Allo everyone,</p>
<p>Today we made progress in a couple of areas.</p>
<p>I focused on changing how we prepare assets for TaleSpire. This is done in a set of tool in unity we call TaleWeaver. The new prop system allows you to clip props to other tiles at certain ‘attachment points’ in TaleWeaver you can now lay out these attachment points and set what the default prop for that point is. This means we can have pretty pre-made tiles that you can disconnect things from, which lets us hit that balance between providing player control and lets people just throw things together.</p>
<p>We now need to add attachment points to all tiles and, since that is quite a task, I’m looking at automating a bunch of it. That will be my task tomorrow.</p>
<p>@Ree has been working on camera control systems for the kickstarter trailer and added an adjustable radial-menu item that can be used for controlling stats. We’ve just hooked it up for HP and it seems to work well so we will provide HP and 4 other stats to start with. These stats are of course sync’d during the session and persist across the whole campaign if you make a creature unique.</p>
<p>The goal for midweek is for you to get copy/paste and the stats stuff so it’s gonna be a fun one!</p>
<p>Seeya tomorrow!</p>
TaleSpire Dev Log 1032019-05-03T20:08:36+00:00http://www.techsnuffle.com/2019/05/03/talespire-dev-log-103<p>Heya folks,</p>
<p>Programming has over the years reminded me again and again that problems with trivial descriptions often hide catacombs of complexity, and that any time you use the word <strong>just</strong> to yourself you are probably in trouble.</p>
<p>‘I <strong>just</strong> need to do this <code class="language-plaintext highlighter-rouge"><insert simple thing here></code>’ is a ancient spell than summons bugs and leads you into sleepless nights of crunch. SO.. when yesterday I was commenting on how copy paste would be difficult I thought I was speaking sensibly and from experience, and thus the gods of code decided that I would get smacked in the face with my own ignorance yet again.</p>
<p>This a long winded way of saying. This works</p>
<p><img src="/assets/videos/copyWip2.gif" alt="I know nothing" /></p>
<p>Which is nice, if mildly bewildering.</p>
<p>There are of course issues and general lack of polish. One side effect of this feature is that it really makes it easy to run into other bugs in the alpha. It’s now very easy to make very large, very tile heavy boards. Very easy to duplicate areas which already have issues with fog of war and amongst other things our selection tools still aren’t great with large piles of assets.</p>
<p>We’ll may have this in your hands next week so be gentle :D</p>
<p>With this working it made it possible to save out simple data that described prefabs. This will make it possible to save small sets of tiles to use as building blocks in game. In future we will have ways of sharing these but our first order of business to bring something similar to our current tiles. For example consider this tile:</p>
<p><img src="exampleTile0.png" alt="tavern wall" /></p>
<p>It’s wonderful to have tiles like this but maybe you want to remove the bellows and add a lamp there, these changes could let that happen (in time). Again there are lots of details to get this to work well and it will mean updating a lot of existing assets, but we are going in the right direction.</p>
<p>Until next time.</p>
<p>Peace!</p>
TaleSpire Dev Log 1022019-05-03T09:31:14+00:00http://www.techsnuffle.com/2019/05/03/talespire-dev-log-102<p>Heya folks, Baggers here.</p>
<p>It’s been a few days since the last dev log so here’s a quick update.</p>
<p>I’ve been working on the new prop system and have got it into our master branch. At this point building/creature-dodging/syncing all technically works but it feels bad right now. As I’ve been head-butting props for a little while I decided to take a day off from that and I’ve started looking at copying selections of tiles. Here’s a little gif of the start:</p>
<p><img src="/assets/videos/copyWip0.gif" alt="copy" /></p>
<p>Now as usual just because it looks like a significant part of the feature is done does not mean that it is, this was 10 minutes of work, the hard part is still to come :) However it’s under way and once we can get multi-tile placement working it opens up a lot of potential features, for example sharing small chunks of boards.</p>
<p>I also wanted to take a moment to talk about what my priorities are on the code side right now. The goal is to keep giving you folks nice small things before the kickstarter. A lot of the more fundamental fixes require work that would take over a month to complete and it doesn’t feel awesome to be stopping updates in the lead up to the event that really decided whether we get to keep making this for you or not (especially as without funding we won’t be able to finish the big fixes anyway).</p>
<p>The bug reporting you have been doing is invaluable and we are so grateful so many of you have written up such excellent reports on github. The big stuff will be fixed but first we need give a bit of the ol’ razzle dazzle :D</p>
<p>As ever, we are around most days on the discord so do stop by and poke us if there is anything on your minds,</p>
<p>Peace.</p>
TaleSpire Dev Log 1012019-04-26T00:34:49+00:00http://www.techsnuffle.com/2019/04/26/talespire-dev-log-101<p>Today there isn’t much to report as I’ve carried on with the prop system. We will be able to add this without breaking anything that currently exists.</p>
<p>We will probably deprecate all the tiles under the prop section and then provide all the same assets using the new system. This means existing behaviour is maintained but anything new will have the new behavior.</p>
<p>As usual work still rumbles on and we’ll be sharing more as it happens</p>
<p>Peace</p>
TaleSpire Dev Log 1002019-04-25T00:10:54+00:00http://www.techsnuffle.com/2019/04/25/talespire-dev-log-100<p>Heya folks, it’s the 100th log so whilst I haven’t got any special dev news I’d just like to thank all of you for checking out these little updates and TaleSpire as a whole. I could easily wax lyrical on how much your feedback means and how it has helped us work out what TaleSpire needs to be, but I’ll spare you that for now. Regardless, Thank-You!</p>
<p>Now back to the snooker,</p>
<p>Today I’ve been working on the prop system. I’ve pushed the fact we are prepping large system refactoring out of my mind and am hacking in a version of prop support for the alpha.</p>
<p>The first version will go something like this:</p>
<ul>
<li>Tiles will stay pretty much how they are, snapping to the tile grid and rotating in 90 degree increments</li>
<li>Props will be a special kind of asset which can be snapped onto attachment points on tiles</li>
<li>Tiles will have many attachment points. These (for this version) will be authored in TaleWeaver when making the tile asset.</li>
<li>Props will not obstruct line of sight or effect fog of war</li>
<li>Placing a creature in the same space as a prop will cause the prop to temporarily ‘dodge’ out of the way (springing back afterwards)</li>
<li>Props will be rotatable in finer steps (currently things 22 degree steps so 16 per revolution)</li>
<li>Props rotate around the normal to the surface the attachment is on (really it’s that they rotate around a vector defined by the attachment but the common case is normal of surface)</li>
<li>Props will become children of tiles in the scene so will show and hide with whichever one they are attached to</li>
</ul>
<p>The code I’m writing for this is real ugly but I’m just gritting my teeth and plowing on as getting a playable version will allow play testing which is way more valuable than holding of until after all the other things I want to change.</p>
<p>As this is a slightly larger task we aren’t putting out a midweek update so we’ll have to see what we have for the weekend.</p>
<p>Seeya around</p>
TaleSpire Dev Log 992019-04-18T23:05:21+00:00http://www.techsnuffle.com/2019/04/18/talespire-dev-log-99<p>Good evening folks.</p>
<p>Not much to report today I was mainly working on a redesign to how tiles, props, pathfinding and zones work, so pretty fundamental. This stuff has been rolling around my head for a month or so but today a bunch of ideas arrived pretty fully formed which was nice. I’m not going to go into detail yet as I’ve still got to work on how I want to identify tiles across the network and how to handle multiple people placing tiles in the same place in the new approach.</p>
<p>Tomorrow I have a day off but I’ll be back around over the weekend (hopefully with the next little update) and I’ll keep working on these ideas alongside the regular feature work. We aren’t rushing this fundamental stuff as it’s more important to me that the alpha is a picture of our intent before the kickstarter, rather than trying to nail down final versions of various internal systems.</p>
<p>Ciao</p>
TaleSpire Dev Log 982019-04-16T18:51:21+00:00http://www.techsnuffle.com/2019/04/16/talespire-dev-log-98<p>Today has been fun.</p>
<p>I fixed a bug that appeared when editing the lineup in initiative mode. Each creature added would recreate the whole list which, whilst it technically worked, it looked bad. This was a simple ‘reused the elements which were there previously’ fix. So that was nice.</p>
<p>One of our lovely gang in discord then pointed out that when you first spawn dice they have a nice spin which is an effective randomizer, but when you pick up rolled dice they don’t spin. Now then you throw we do add some random angular momentum so the result should still be random, however it didn’t feel great so I’ve added the spin on pickup too.</p>
<p>Lastly I’ve done most of the work to add a way to control the volume of sound effects in the same fashion as music and ambiance. I’m still hunting down a bug where this new volume is not applied correctly but it’s nearly done.</p>
<p>My goal for tomorrow is to get an update out so if I don’t get the effect volume stuff done in time it’s no big deal, we’ll probably have another update at the weekend anyway.</p>
<p>Peace folks.</p>
<p><img src="/assets/videos/pickupSpin0.gif" alt="spin" /></p>
TaleSpire Dev Log 972019-04-15T23:02:45+00:00http://www.techsnuffle.com/2019/04/15/talespire-dev-log-97<p>Good [time adjusted] evening to you all,</p>
<p>Today I’ve been keeping busy with bug fixes and some small workarounds. These changes will be out in the next update (hopefully wednesday or thurday)</p>
<ul>
<li>We now show the name of the selected creature along with the player name in the GM UI</li>
<li>We clear the history panel when switching boards</li>
<li>Fixed a case where, if you controlled many creatures and clicked on their image in player mode it might focus the wrong creature</li>
<li>Fixed an ugly exception that would occur if you disconnected after having used the turn based mode</li>
<li>Fixed a case where the first GM joined after other non-gm players and the gm would be put in cutscene mode</li>
<li>Tweaked the number of tiles updating per frame. We still get ugliness in large, multi-floor boards but this makes it a tiny bit more playable whilst we find a better solution.</li>
</ul>
<p>We also pushed out the next 900 invites, so thats 1900 in two days which feels nice.</p>
<p>Now to get fixing so we can get another batch out!</p>
<p>Peace</p>
<p><img src="/assets/videos/nameBadge0.gif" alt="creature name" /></p>
<p>p.s. I’m not going to get much done tomorrow as the machine I use for streaming and testing has died so I need to sort that out first.</p>
TaleSpire Weekday Update2019-04-11T23:35:09+00:00http://www.techsnuffle.com/2019/04/11/talespire-weekday-update<p>After the big asset drop on the weekend it’s time for a small asset update, we added a little side-table that you can find in props.</p>
<p><img src="/assets/videos/sideTable0.gif" alt="side-table" /></p>
<p>In the world of new features you can now make a board in a campaign the default, this means that whenever someone joins the campaign they arrive there first.</p>
<p><img src="/assets/videos/defaultBoard.gif" alt="default board" /></p>
<p>We also fixed some bugs:</p>
<ul>
<li>One that stopped atmospheres syncing correctly</li>
<li>One which stopped the invite box working correctly in the main menu</li>
<li>And one where a creature’s radial menu would stay visible after the creature was deleted (making it very easy to get exceptions)</li>
</ul>
<p>We are currently tracking another annoying one where creatures which should be visible and in line of sight are not showing. If we can fix this tomorrow then we will push out an update just for that then.</p>
<p>Thanks for following our little spire.</p>
TaleSpire Dev Log 962019-04-11T00:12:49+00:00http://www.techsnuffle.com/2019/04/11/talespire-dev-log-96<p>Heya folks. Today I spent a good chunk of the day continuing through the errors dumped yesterday. I’m about a third of the way through and have already got a good list of bugs we didn’t have previous reported on github. Most of the these are bugs that would have occurred without the game dying so will have manifested in annoying ways.</p>
<p>Yesterday I mentioned that one bug we wanted to squash was the one causing atmospheres not to sync properly. This has no been fully worked out and I’ll be finishing off the fix in the morning. Hopefully the rest fall just as easily.</p>
<p>Ciao</p>
TaleSpire Dev Log 952019-04-10T00:05:32+00:00http://www.techsnuffle.com/2019/04/10/talespire-dev-log-95<p>Only a quick one tonight as it got late whilst I wasn’t looking and I should get some sleep :)</p>
<p>Today I pushed the server changes to support default boards to production. That went smoothly and the feature is working well internally, we should be able to get that to you this Thursday (if all goes well).</p>
<p>I then exported all the errors that have been automatically reported to our server and I’ve started going through them. I’ve already found solutions to a couple of bugs but there naturally are many more. One particular annoying one that has been reported is that sync of atmosphere settings can stop working.. that sucks and so I’m hoping to see <em>something</em> that might give some clues to what happened there.</p>
<p>Until tomorrow.</p>
<p>Peace</p>
TaleSpire Dev Log 942019-04-09T00:15:02+00:00http://www.techsnuffle.com/2019/04/09/talespire-dev-log-94<p>Yay it’s good to be back to dev logs again. Today I prototyped the default board feature. This lets you pick the board that players automatically join when they join the campaign. It doesn’t make for an exciting gif but here it is anyway! WARNING: CODER GRAPHICS, NOT FINAL LOOK :p</p>
<p><img src="/assets/videos/defaultBoard0.gif" alt="default board" /></p>
<p>I’ve also been doing a bit of a survey of the game as it stands today and looking at what the data in our system looks like. We are going to need to make things faster whilst also making boards <strong>much</strong> bigger so now is a good time to be seeing what patterns have appeared and what could be changed to be able to leverage your PC more effectively. This is all pretty arm wavy but I expect I’ll do a brain dump on this sometime this week as many features we are (slowly) working on like large boards, new floor system and the new fog of war all depend on what we decide here.</p>
<p>Tomorrow I’ll also be fixing up the ‘percentage’ dice and trying to hunt down what has been causing the atmosphere sync issues folks have been seeing.</p>
<p>Until then,</p>
<p>Peace.</p>
TaleSpire Dev Log 882019-03-30T00:33:38+00:00http://www.techsnuffle.com/2019/03/30/talespire-dev-log-88<p>Heya folks, thanks for stopping by again.</p>
<p>Today was fairly productive:</p>
<ul>
<li>Fixed a bug where lit torches from hidden floors would be visible on load</li>
<li>Fixed bug where picking an asset from the panel which was a member of a group would instead pick the first asset from the group</li>
<li>Added a community requested feature where holding tab shows the names of the creatures</li>
</ul>
<p>The ui layer currently showing the creatures names is going to be greatly expanded on in future for GM tools. Being able to reveal GM only assets, chests containing pre-prepared assets, GM notes etc. That’s gonna be fun.</p>
<p>@Ree has been working on a variety of things including the animation overhaul mentioned recently. A bunch of features will be possible with that for example more emotes, spell effects and death/ko states. Gah, cant wait.. it’s gonna be awesome when we all get to be on this full time.</p>
<p>There’s other stuff in the works of course, keep and eye out for another alpha update soon.</p>
<p>Peace.</p>
<p><img src="/assets/videos/names.gif" alt="names" /></p>
<p>p.s. Almost forgot. We now have a site to check if you have been sent an invite key. Check it out here: http://keys.talespire.com/</p>
TaleSpire Dev Log 872019-03-27T23:04:15+00:00http://www.techsnuffle.com/2019/03/27/talespire-dev-log-87<p>Today was an interesting mix.</p>
<p>I got around the staging server issue from yesterday by using a different subdomain..yeah I’m not happy with it either. I’m guessing that the certs weren’t cached and so every time I pushed a new build it was asking letsencrypt for new certs.. they have a limit of how many they will give out in a period of time so maaaaybe that’s it. However I was sure I had seen that before and it would error rather than giving something invalid. So I don’t know.</p>
<p>The correct thing to do would be to inspect the certs I received and see what was up with them. However this week we are really pressed for time so for now I’ll settle for dodging the issue rather than fixing it.</p>
<p>Because of that I was able to test the new ‘Check if I should have received an invite’ site. It works so, after a few tweaks to make it mobile friendly, we called it done for now. We will make it available to you with the next wave of invites.</p>
<p>Speaking of those we have been way behind on that, the content updates we were wanting to complete first have taken longer to complete as we have started prepping material for the upcoming kickstarter. This is also why we haven’t had a mini update this week (booooo :[)</p>
<p>After the above stuff I started doing some work on dice, there are some ownership and syncing tweaks I want to make.</p>
<p>I also added basic per-server request stats to our front end server code. This just means we get per minute info on what requests are made to the server in a format that should be easy to graph in cloudwatch. Super low tech approach but if it works for now it will be a boon.</p>
<p>That’s all for today. Back with more tomorrow.</p>
<p>p.s Oh I’m away from this Sunday to next Wednesday as I’m off to a programming conference. There will be a small gap in these dev logs but they’ll be back as soon as I am :)</p>
TaleSpire Dev Log 862019-03-27T00:42:28+00:00http://www.techsnuffle.com/2019/03/27/talespire-dev-log-86<p>Evening all. One of those annoying logs today as it doesn’t have a conclusion.</p>
<p>I finished up the site which allows you to check if you have been sent and invite to the alpha. I then pushed a new staging server for final testing and saw ssl issues, but not good log as to what the issue was. I assumed I had just messed something up in my recent changes so tried a commit from the other day (which was previously working).. same issue. I tried a bunch of things but I’m just getting the same bloody unhelpful error.</p>
<p>I’ve got some other things to work on tomorrow so I’ll focus on this on Thursday.</p>
<p>Aggravating to not have more data to share but thats the way for today.</p>
<p>Peace</p>
TaleSpire Dev Log 852019-03-26T01:06:23+00:00http://www.techsnuffle.com/2019/03/26/talespire-dev-log-85<p>Tiny update tonight as I didnt realize how late it had gotten here.</p>
<p>The main thing of interest today was that I set up the backend code for checking if you should have recieved an invite code. This is gonna be helpful in the next wave of invites.</p>
<p>I just need to make the webpage for this now.</p>
<p>Goodnight folks</p>
TaleSpire Dev Log 842019-03-21T22:21:42+00:00http://www.techsnuffle.com/2019/03/21/talespire-dev-log-84<p>Turns out that the issues with the server images was simple after all.</p>
<p>Back in january I tweaked the images to include the amazon agent so we could get memory usage info in cloudwatch. When I did that I must have left some files from an old build around. Later on, when the newer code was pulled and extracted, it didnt replace what was already there and we ended up running this weird frankenbuild when we had old code and a new db schema.</p>
<p>Dumb but fine. I then fixed up the server side code for storing the default board. I’ll push that to live tomorrow and then soon we’ll work on ingame UI for this. I expect we’ll add a marker to the entries in the board panel.</p>
<p>Ok, time to sign off.
Peace</p>
TaleSpire Dev Log 832019-03-20T23:45:37+00:00http://www.techsnuffle.com/2019/03/20/talespire-dev-log-83<p>We’ve had a lot of good days of progress so it was about time for an annoying one.</p>
<p>Today I was looking to push some server side changes to allow GMs to set a default board for a campaign, this would be the one that you join automatically when launching the campaign. However when I pushed the code to a new staging instance I was getting and instance with the update to date database schema but the code from back in January.</p>
<p>It seems that earlier in the year I must have messed up our staging server machine image. This means that tomorrow is going to have to be looking into that and fixing that up. The initial creation of those images was a pain in the ass and I expect this to be no exception.</p>
<p>Ah well, that’s just how it goes. It’ll be fixed soon enough.</p>
<p>Let’s see if I get it tomorrow,</p>
<p>Goodnight</p>
TaleSpire Dev Log 822019-03-18T22:42:06+00:00http://www.techsnuffle.com/2019/03/18/talespire-dev-log-82<p>Hey again, We had planned to get a content update out over the weekend but alas that did not happen as we hadn’t finished updating all the current assets. This has thus been pushed to later this week. For the midweek update I’m hoping to push the tag search (which is now working great) and the GM only dice.</p>
<p>‘GM Only’ dice rolls are special in that only the GMs can see the symbols on the faces and the roll notifications. Here is a clip of making a gm only roll and then switching between GM and player mode.</p>
<p><img src="/assets/videos/gmOnlyDice0.gif" alt="gm only roll" /></p>
<p>We are hoping to making the UI for the GM only roll and then we’ll get this update asap.</p>
<p>Thanks for checking out these little updates!</p>
<p>Peace.</p>
<p>p.s. Here is the version of the tag search we will be shipping</p>
<p><img src="/assets/videos/tagSearch0.gif" alt="tag buttons" /></p>
TaleSpire Dev Log 812019-03-14T23:24:50+00:00http://www.techsnuffle.com/2019/03/14/talespire-dev-log-81<p>Today has been focused on tags. I spent the morning adding tags to our assets and then the afternoon playing with the filtering code.</p>
<p>Here is how it is looking:</p>
<p><img src="/assets/videos/tagSuggest2.gif" alt="filter" /></p>
<p>The UI elements we use are due a rewrite thanks to Unity’s nested prefab feature that landed relatively recently. As such I’ve been trying to add this in a way we can move easily when it comes to it. I’m getting happier with that and now I just need to make the tags that have been entered appear as the little blue buttons next to the input field.</p>
<p>That’s all for now. Seeya tomorrow.</p>
TaleSpire Midweek Release2019-03-13T15:47:00+00:00http://www.techsnuffle.com/2019/03/13/talespire-midweek-release<p><img src="/assets/videos/lightRange.gif" alt="light range" /></p>
<p>It’s been 3 days and it feels like it’s time for another update. This time we have two new features.</p>
<p>The first is that creatures now have torches to help them seem in dark places. Right click and you’ll see the option in the radial menu.</p>
<p>The second is that the selection tool and dragging to place tiles now both show the dimensions of the selection. This can help a lot when trying to make walls line up when building big rooms.</p>
<p>The fires of development are still hot so we will be back soon with more goodness.</p>
<p>Fare thee well</p>
TaleSpire Dev Log 802019-03-12T16:42:54+00:00http://www.techsnuffle.com/2019/03/12/talespire-dev-log-80<p>Welcome back,</p>
<p>Today I started looking at the tag filter for the asset panel. It’s one of those cases where the current way the UI is set up is not great but it’s better for us to work around it and get the feature out than it is to delay the feature for the rewrite. I’ve been poking around with the different components to see what the easiest way to get this in will be.</p>
<p>It’s not the most exciting thing but here are some gifs anyway:</p>
<p>This one is from early in the day just checking that we can give tag suggestions. At this point the potential tags were hardcoded for testing.</p>
<p><img src="\assets\videos\tagSuggest0.gif" alt="hardcoded tag search" /></p>
<p>Here was from a little bit later testing that we could use tags from the asset database. At this point the only two assets to have tags were the two doors and the only tag was ‘door’ which is why you don’t see other suggestions.</p>
<p><img src="\assets\videos\tagSuggest1.gif" alt="filtering" /></p>
<p>The next things on the todo list are:</p>
<ul>
<li>Add some UI to TaleWeaver (our asset tool) to allow adding tags</li>
<li>Go through all our current assets adding tags</li>
<li>Make the blue tag buttons in the search bar work.</li>
<li>Get the code cleaned up to the point that it will be trivial to replace when we refactor the UI code.</li>
</ul>
<p>When those are done this will be in a good enough to get into the alpha. I expect that will be on the weekend update.</p>
<p>Peace.</p>
TaleSpire Dev Log 792019-03-11T22:35:31+00:00http://www.techsnuffle.com/2019/03/11/talespire-dev-log-79<p>Today has been fun. Hacking together small features feels super rewarding after being down in the weeds for a while.</p>
<p>Today I got size indicators working for multi-select and dragging out tiles. The multi select version needs a little work as sometimes the rounding produces slightly unintuitive values, but that should be a quick fix so you should have this in your hands in the midweek update</p>
<video controls="" src="/assets/videos/sizeIndicator0.mp4" type="video/mp4" width="620"></video>
<p>Next I started work on another community requested feature, torches for creatures.</p>
<video controls="" src="/assets/videos/creatureTorch0.mp4" type="video/mp4" width="620"></video>
<p>This one still needs a few tweaks so I’m expecting this will be in the weekend update.</p>
<p>@Ree has been hard at work refactoring how we handle animations for creatures so you will be seeing some new stuff there soon. The first thing we expect to bring is creature death animations. This will also leave the ‘corpse’ in the scene so you can revive it if you like.</p>
<p>The reason for the revamp of the animation stuff is that it allows us to more easily add other <em>things</em> into the timelines, like sound & visual effects (think sexier magic attacks!) so it’s opening up some exciting stuff.</p>
<p>Seeya tomorrow</p>
TaleSpire Dev Log 782019-03-10T21:17:44+00:00http://www.techsnuffle.com/2019/03/10/talespire-dev-log-78<p>Weekend update!</p>
<p>This time we have two prototype features for your testing pleasure.</p>
<p>The first is that you are now able to click on GM requests and history entries to focus the thing being talked about. For example if a request comes in that a specific user would like to open a door, you can click the request to jump to the door they are referring to. If you want to know which tile an entry in the history is referring to simply click it to go there.</p>
<p>The second feature is the ability to move creatures with the keyboard. It works reasonably well on single floors but not across floors yet. Naturally this will mature over time. Currently to use this hold down shift and use WSAD.</p>
<p>Alrighty, expect the next update in a few days.</p>
<p><img src="/assets/images/thump.gif" alt="thump thump thump" /></p>
TaleSpire Dev Log 772019-03-07T00:20:01+00:00http://www.techsnuffle.com/2019/03/07/talespire-dev-log-77<iframe width="560" height="315" src="https://www.youtube.com/embed/hdNcnuiZIUE" frameborder="0" allowfullscreen=""></iframe>
<p>Today I was looking into a couple of features that have been on the wishlist for a while, deleting campaigns and focusing on history events. The video above shows the WIP of the latter.</p>
<p>That bit is nearly done but I need to tweak how I’m handling doors/props etc. Shouldn’t take much longer though.</p>
<p>The delete campaign feature is done except for an annoying UI issue but that just down to my infamiliarity with the system, I’ll bug @Ree about it soon and we’ll have it done in a jiffy.</p>
<p>Hopefully this will mean a new update by the latest Monday.</p>
<p>Time to sign off.</p>
<p>Peace!</p>
TaleSpire Dev Log 762019-03-05T23:37:01+00:00http://www.techsnuffle.com/2019/03/05/talespire-dev-log-76<p>Update time again folks! Fire up steam to grab the latest fixes</p>
<p>We are looking to up the pace of the updates over the coming weeks so keep an eye out for new prototypes and features to play with.</p>
<p>This little bugfix update contains:</p>
<ul>
<li>A fix to sync the ‘initiative mode’ lineup when a player joins</li>
<li>Ensure additional GMs end up in the correct game mode when joining when the game is in progress</li>
<li>A fix so that deleting the creature whose turn it is, correctly sets the next creature as active</li>
<li>A fix to the undo shortcut so it doesn’t also select the next asset in the group when building.</li>
<li>GMs no longer jump straight into the ‘edit lineup’ tool when switching to initiative mode.</li>
</ul>
<p>Back soon with more!</p>
TaleSpire Dev Log 752019-03-05T00:19:05+00:00http://www.techsnuffle.com/2019/03/05/talespire-dev-log-75<p>Hi again,</p>
<p>Today I finished testing the fixes for initiative mode (known in game as ‘combat mode’ right now). I want to get a new update out very soon so we can get these fixes in your hands.</p>
<p>I then fixed a few UI issues that were arising when additional GMs were joining the board during a battle. One was simply the new GM’s UI was in the wrong mode. Another was that, upon entering battle mode they were immediately kicked to the UI for editing the lineup.. this was confusing more than anything else.</p>
<p>We’ve also had another little planning meeting and I’m hoping we can give you lots of reasons to be checking these updates over the next month a bit leading up to the kickstarter :)</p>
<p>Back tomorrow with more bits and bobs.</p>
<p>Peace.</p>
<blockquote>
<p>This dev log can also be found at https://bouncyrock.com/news/articles/talespire-dev-log-75</p>
</blockquote>
TaleSpire Dev Log 742019-03-02T00:53:08+00:00http://www.techsnuffle.com/2019/03/02/talespire-dev-log-74<p>It’s late but I don’t want to skip this today so here we go.</p>
<p>Today I’ve mainly been focusing on bugs relating to the order of creatures in the battle-mode (also called initiative-mode) lineup. One such issue can be found <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/88">here</a>.</p>
<p>The main one from today was fixing the behavior when the currently active creature is deleted as control passed not being passed to the next in the list. I fixed that but I have seen a number of issues and I’ll need to test those pretty thoroughly before I’m happy to push this out. This code feels like a place where it is very easy to introduce regressions.</p>
<p>I’m hoping I’ll get the majority of the related issues cleaned up on Monday so we can push another update asap.</p>
<p>We actually have a bunch of asset changes too so we’d like to get those out soon too. These changes flesh out the various asset collections to match the tiles available in the test tile set.</p>
<p>Time to sleep, gotta get up early to watch the SpaceX RD1 launch tomorrow morning :)</p>
<p>Peace.</p>
TaleSpire Dev Log 732019-02-25T18:10:31+00:00http://www.techsnuffle.com/2019/02/25/talespire-dev-log-73<p>Allo again, back with more of today’s messing around.</p>
<p>In the morning as planned I fixed the case where the <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/68">Attack prompt remains if figurine is removed before confirming request </a>. During the testing I found that there was a case where you could get the following.</p>
<ul>
<li>Gm0, Gm1 and Player0 join</li>
<li>Gm0 adds Creature0 & Creature1</li>
<li>Player0 tries to attack Creature1 with Creature0 and the UI appears to indicate that the attack is waiting on approval</li>
<li>On Gm0 & Gm1’s clients the UI appears to approve of decline the attack</li>
<li>Gm0 approves the attack</li>
<li>The UI disappears on Gm0 & Player0’s clients</li>
<li>The UI is erroneously still visible on GM1’s client</li>
</ul>
<p>This was relatively easy to fix but testing took a while as after each attempt the build has to be copied to a couple of machines to test all 3 connections.</p>
<p>After that I added some platform specific code for controlling the hardware cursor so we can start tackling <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/53">this issue</a>.</p>
<p>Okidokey that’s all for today. Tomorrow I start with a bug where deleting a creature can cause subtle issues with the turn based line-up ui and.. well then I’ll grab another from the pile.</p>
<p>Peace</p>
TaleSpire Dev Log 722019-02-24T19:00:19+00:00http://www.techsnuffle.com/2019/02/24/talespire-dev-log-72<p>A quick sunday update for you.</p>
<p>Today I’ve been looking at <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/68">this issue</a> and looking into how best to tackle it. I’ve come up with a fix I’ll be coding and testing tomorrow morning but the issue was interesting as it gave me a change to look at the GM Request code with fresh eyes. Once again I’m seeing places where an effort to keep things extensible for an uncertain future has really just led to a more complicated system than neccessary (or rather one that has needed patching to line up with the actual behaviour we wanted later). I’ll definitely be looking at making this dumber in future.</p>
<p>This post is super vague :D but explaining the details (in this case) doesnt bring with it any cool takeaways.</p>
<p>Seeya tomorrow!</p>
TaleSpire Dev Log 712019-02-21T22:06:06+00:00http://www.techsnuffle.com/2019/02/21/talespire-dev-log-71<p>Progress has been unexciting but continuous. I’ve pushed the changes that reveal corners of boards correctly in player mode. It does reveal a little too much but this will be fixed in future when the resolution of the ‘fog of war’ datastructure is increased.</p>
<p>Today I added the class for prop assets and refactored the building/serializing/etc code to handle this new asset kind. This being seperate now lets us at unique logic for these assets. We have experimented with having props temporarily ‘dodge’ out of the way of creatures and also be able to be places and rotated with more fine grain control than of tiles. Those experiments can now we ironed out and made part of the shipping game!</p>
<p>@Ree, whilst ill, has put a bunch of time into the changes we need to be able to more reliably get the invite emails passed filters. More on this soon.</p>
<p>I’m not going to be around on friday but will be working on Sunday instead so hopefully I’ll have another post for you then.</p>
<p>Peace.</p>
TaleSpire Dev Log 702019-02-20T00:43:51+00:00http://www.techsnuffle.com/2019/02/20/talespire-dev-log-70<p>Evening folks,</p>
<p>I’ve been messing around with fog of war and whilst it’s not there yet there has been progress.</p>
<p>The task right now if fixing <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/50">this issue</a>. The short version of the problem is that we have a data-structure that describes where has be uncovered (we call this a creature’s ‘memory’) and we have to sample it to see if a tile is visible. We can’t just sample in the center as maybe a tile is multiple units wide and only the corner has been revealed. We also can’t just sample at the center of each sub-tile as we get situations like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ||
||
____|| _____
__A__||__B__|
</code></pre></div></div>
<p>Where <code class="language-plaintext highlighter-rouge">A</code> is a wall and floor tile and <code class="language-plaintext highlighter-rouge">B</code> is just a floor tile. If <code class="language-plaintext highlighter-rouge">B</code> has been revealed then we have to see the wall (otherwise users get confused as to if it’s the end of the board or if it’s a hidden tile). This means sampling slightly outside of the bounding box of the tile. However this still can cause issues, such as the one linked above.</p>
<p>I played around with a few more schemes but today swapped it out with a mechanism that does a breadth first search of all the nodes within a bounds in the creature’s ‘memory’. This has a more expensive worst case than the other approaches, but the tests were seeming to show that to take enough samples for a good result we would be paying a pretty high cost anyway. The creature memory is stored in a quadtree so in many cases the search can succeed early as we don’t have to search the full depth to find a populated cell.</p>
<p>I haven’t got anything more today so I’ll bail now and give ya more news soon.</p>
<p>Peace</p>
TaleSpire Dev Daily 692019-02-14T23:17:05+00:00http://www.techsnuffle.com/2019/02/14/talespire-dev-daily-69<p>It’s that time again, today was spent:</p>
<ul>
<li>Finishing off the fix for <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/66">If GM permissions are rescinded while any GM-only UI is visible, they stay on-screen</a></li>
<li>Case where game spuriously entered cutscene mode when GM rights were removed from a player</li>
<li>Initiative mode wasn’t updating the edit UI for remote GMs when local GM changed the line-up</li>
<li>Reordering of lineup was not handled correctly in the edit UI</li>
<li>Fix <code class="language-plaintext highlighter-rouge">UIList</code>’s <code class="language-plaintext highlighter-rouge">Clear</code> method which was causing order issues when new elements were added.</li>
</ul>
<p>Tomorrow I think I’m going to look at that fog of war issue causing corners not to appear. The first part is visualizing the points that are sampled for fog of war info and then we’ll see where the bug takes us.</p>
<p>Peace.</p>
TaleSpire Dev Log 682019-02-13T23:55:51+00:00http://www.techsnuffle.com/2019/02/13/talespire-dev-log-68<p>Today I fixed the issue that all GM shared one flashlight. I also started looking into <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/66">this issue</a> where removing GM permission from a player doesn’t close some of the GM UI they have open. I think I’m on the right track with that one so hopefully I’ll have that one done in the first few hours tomorrow.</p>
<p>Alright that’s the most interesting stuff.</p>
<p>Peace</p>
TaleSpire Dev Log 672019-02-12T23:21:28+00:00http://www.techsnuffle.com/2019/02/12/talespire-dev-log-67<p>Evening all.</p>
<p>Yesterday we pushed out a little update. It was significant for us as it’s the first update after upgrading the version of Unity we were using. That’s always tense as it can be hard to ensure that everything is still working the way it was before. The update also included a proper fix for the bug from <a href="http://techsnuffle.com/2019/01/29/talespire-dev-log-61">log 61</a> along with a few new nice to haves like shortcuts for camera rotation etc.</p>
<p>That bug fix was going to go out last friday but when we tested the release we saw that synchronization was broken between the clients for this like doors, chests etc. That was super weird and after a LOT of head scratching we found out that it was due to not having reprocessed a bunch of assets after the Unity upgrade. That was a bummer that we lost a day or two to it but good as rereading all the code helped me find and fix another unrelated issue.</p>
<p>Today I’ve mostly been looking at tickets reported by the community. We’ve not done a great job getting back to the awesome people who have filed tickets and that bums me out. Even though most get seen as they arrive the very least we should be doing is letting you know whether we can reproduce the issue. I’m gonna try to make sure I leave some time every day for that so I don’t fall behind again.</p>
<p>Once again thankyou all so much for the super thorough reports that have been coming in, they truly are invaluable.</p>
<p>Now that the last big tickets are dealt with I have a small gap before the UI for the new features is ready so I will get back on tickets again. Expect movement on those real soon!</p>
<p>Time for some sleep.</p>
<p>Peace</p>
TaleSpire Dev Log 662019-02-08T08:25:53+00:00http://www.techsnuffle.com/2019/02/08/talespire-dev-log-66<p>Heya folks. A quick one today. With both some life business and serialization fixes out of the way it is finally time to get back to tickets. People have been amazing at filing them so the fact I haven’t been triaging these for a week sucks. I’m gonna get through reproducing as many as I can today so I can then focus on fixing next week.</p>
<p>@Dwarf has been killing it making tonnes of progress updating all of the tiles to the new sizes and fleshing out the sets to include everything in the test tile set we made available in the last update. These will all count as new sets as, because of dimensions changes we can’t just swap then out with the originals without breaking your existing boards. Of course that will be coming with the floor changes but until then we want to keep everything working.</p>
<p>@Ree has made a pile of progress on props. As they will become their own class of object they can now have their own behaviors. This includes being able to be placed in way more place, orienting to the surface they are placed on and automatically ‘dodging’ out of creatures way if the creature steps in the same tile as some props. He’s also started on the floor UI as by nailing that we can then start on the deeper engine changes.</p>
<p>All go but mostly behind the scenes. We can’t wait to show you more though so thanks for joining us on this journey.</p>
<p>Peace</p>
<p>p.s One thing we haven’t made time for yet is tweaking the emails so they are less likely to end up in spam. We will be doing this for the next invite push though. When we do that we will re-email all codes to all of you who are already meant to have access.</p>
TaleSpire Dev Log 652019-02-07T00:01:16+00:00http://www.techsnuffle.com/2019/02/07/talespire-dev-log-65<p>Good-evening folks!</p>
<p>Tonight I dont have a writeup but I have a video of a simple shader effect prototype. It is just a simple vertical cutaway we can use on tiles for when your creatures are in tight coridoors. It’s going to be evolved a bunch to make it more aethetically pleasing but this was just so I could prove to myself that the core idea would work.</p>
<p>Only the first 50 minutes of the video are about making the effect so you don’t need to hang around for the whole thing. Also this is naturally a technical video so if you arent into hairy muppets rambling whilst typing lots of stuff then this may not be for you :)</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/9irTK9sqifo?start=462" frameborder="0" allowfullscreen=""></iframe>
TaleSpire Dev Log 642019-02-05T22:43:30+00:00http://www.techsnuffle.com/2019/02/05/talespire-dev-log-64<p>Today I finished testing the fixes for serializing board-assets with state (doors, chests etc). There will be one side-effect when this goes out and that is all door/chests etc will revert to their default state.</p>
<p>This side effect is a trade-off. We are fundamentally changing how that code worked and so to handle this change we would need to do a of work and testing to port the current state over. This would be the <strong>only</strong> choice once we have released but whilst we are at this point in the alpha we want to focus on getting more content out there and that is blocked until this fix it out.</p>
<p>So yup that’s all for today. I’ve lost some work time due to sorting out some health stuff and tomorrow will be most taken up by that too. However in the evening I will be prototyping the start of the tile cutaway shader in my lisp stream so that’ll be fun.</p>
<p>Until tomorrow,
Peace :)</p>
TaleSpire Dev Log 632019-02-01T00:07:12+00:00http://www.techsnuffle.com/2019/02/01/talespire-dev-log-63<blockquote>
<p>WARNING: We want to be open in these technical blogs so you are reading unfinished, unproven ideas. This should not be taken as a promise of anything.</p>
</blockquote>
<p>Today @Ree and I have been white-boarding a replacement for the floor system. We need to do this as the current system has a number of issues including (but not limited to):</p>
<ul>
<li>Tiles are revealed by floor which means large features need to be built on multiple floors or built in a huge stack from one floor which intersects the others</li>
<li>building raised platforms that belong to one floor requires stacking and then deleting tiles as tiles always try to drop to the floor.</li>
<li>hair loss</li>
<li>Can’t delete or position floors as the floor is infinitely wide. If you have a town and delete a floor it deletes that floor for every building in the town.</li>
</ul>
<p>There are also a bunch of other things that fall out of this regarding how fog of war works, which is another area where the current version of the game has issues</p>
<p>We are making progress but it’s a very extensive change. The goal is to let you create floor plains that are separate from each other. You will be able to adjust their height from the previous floor (in fixed increments) and they will grow as you place more tiles. The footprint is maintained down the whole stack of floors.</p>
<p>This would let you have buildings where you can set floor height, delete floors, etc. Building raised platforms is easy and you just build the plain and build on top of it.</p>
<p>Tile reveal and fog of war is also not ideal in open areas like towns right now. You build lovely multi story buildings and don’t see them until you switch floor. Also, as we reveal what your character can walk to, players don’t see roofs often either. We are musing on inverting the logic of fog of war, where you place a origin for the fog (let’s call it a fog-machine for now :D) and then scroll (or something) to adjust how far the fog flows. this gives the GM control over what is shown and the same (or similar) method we have now is used to uncover it. Each ‘body’ of fog is separate so it should hug walls in a better way improving on the current tile reveal.. we also need some more shader magic to improve the look too.</p>
<p>So yup, massive changes could be afoot. This is ALL just thoughts and musings right now. We are going to start prototyping the gameplay side of floor building tonight and I’m musing on ways to maintain some of the nicer qualities of the fog of war whilst getting some of the new stuff.</p>
<p>I expect I’ll do a stream soon to talk about the problem and the design space even if we have nothing new to show. We’ll keep you posted with things as they happen.</p>
<p>Peace!</p>
TaleSpire Dev Log 622019-01-31T14:35:00+00:00http://www.techsnuffle.com/2019/01/31/talespire-dev-log-62<p>A slightly unusual one today. I needed to prototype the approach I want to use for TaleSpire’s line of sight and I decided to try doing it in a couple of hours on my lisp stream. Here is the result:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/f6QtA_tiKm8" frameborder="0" allowfullscreen=""></iframe>
<p>We got it working but right at the very end of the stream so there isnt a sexy payoff. We’ll revisit this to make sure it’s working as expected and then reimplement it in Unity.</p>
<p>WARNING: The stream contains a lot of parens :D It’s also built on prototypes from other episodes so sorry that we dotn go into detail on how most of it works.</p>
TaleSpire Dev Log 612019-01-29T22:04:06+00:00http://www.techsnuffle.com/2019/01/29/talespire-dev-log-61<p>I’ve changed the name until I get my arse back in gear and start doing these daily again.</p>
<p>The weekend was rather eventful. We had a collection of bugs that acted together to corrupt peoples boards. The good news is that a while back we started making checkpoint saves of boards on each session so we are able to revert to those un-borked version.</p>
<table>
<tbody>
<tr>
<td>The principle bug was a mistake is some of the serialization code that made an assumption that the number of Assets in a Tile didn’t change. This was a very dumb assumption and it was really just luck that this hadn’t bitten us sooner. This was compounded by the fact that, on failing to load, the game didn’t recognize this and jump back to the empty board. No, instead it decided the board was out of date and saved the now broken board back to the server :</td>
</tr>
</tbody>
</table>
<p>Needless to say that changed my Sunday plans. Thanks to some invaluable help from those affected (extra props to Blappo here) we were able to replicate the issue and get a patch out a few hours later. I’m currently working on a proper fix for this but it needs to be done carefully so we don’t slip in any more problems.</p>
<p>Also in a couple of days I’ll be meeting up with @Ree so we can start work on the next revision of the floor system in TaleSpire. No spoilers today but news (and pics) will come soon.</p>
<p>Alright, time to wrap up,</p>
<p>Seeya</p>
TaleSpire Dev Daily 602019-01-23T13:04:13+00:00http://www.techsnuffle.com/2019/01/23/talespire-dev-daily-60<p>Hey again, I’ve not written for a few days as it would have been a ‘still working on the same thing’ post.</p>
<p>I’m done with the issues with unique creatures for now. As ever it doesn’t feel like it’s in its final form but it’ will certainly address a few of the issues:</p>
<ul>
<li>When a player joins a board their permissions over unique creatures were not always maintained</li>
<li>Uniques spawned into unsynced board disappeared on first sync</li>
<li>Only first GM gets the unique creature panel populated</li>
<li>Renaming unique creatures doesn’t update the UI promptly</li>
</ul>
<p>This stuff should be in the next update and we hope to get that out soon. That (if all goes well) should coincide with the next batch of invites. The backend has been fine with the first batch and with the bugs we have fixed recently it makes sense to get more folks in.</p>
<p>I’m going to pick a few simpler tickets now so I can get through a bunch over the coming days. Along with that I need to do some work on the client so we can support ‘props’ that are separate from tiles. More news on that soon as it should be a fun one :)</p>
<p>Ok folks, until next time.</p>
TaleSpire Dev Daily 592019-01-17T23:19:49+00:00http://www.techsnuffle.com/2019/01/17/talespire-dev-daily-59<p>Hey again, it’s been a satisfying day. I spent the morning tracing through the logic that handles creature ownership and some users have seen a bug where people lose control of unique creatures seemingly randomly. By doing that I found one mistake that was causing 2 bugs:</p>
<ul>
<li>a unique creature deleted and respawned from the unique creature panel has incorrect ownership</li>
<li>making a creature unique gives the GM permissions to the unique on the server even if they didn’t have that previously have that client side.</li>
</ul>
<p>Luckily the bugs came from human error rather than something fundamentally wrong with the design of the system, so that was nice. I also managed to consistently reproduce another bug:</p>
<ul>
<li>If a player joins a room that already contains a unique creature they should have permission to, it doesnt give them that permission.</li>
</ul>
<p>I expect this is a mistake is how we setup the synchronized version of the board.</p>
<p>I think tomorrow I should be able to get that bug fixed with plenty of time to spare. With the rest of the I want to go after a more fleeting and thus trickier bug. I have seen cases where I have dragged in a unique and all seems to be fine, but then the unique disappears. It almost looks like twhat happens when a unique is summoned to another board so I’ll probably start debugging there and see what comes up.</p>
<p>I’d be quiet happy if I can just get a reliable way to reproduce the bug tomorrow. I’ll then be in a good place to fix it Monday, or over the weekend if I fancy.</p>
<p>After the month or so of crunch I am really trying to get back in the habit of doing things other than TaleSpire in my free time so I don’t burn out. I’ve been playing ff9 again after a 2 month break and I really want to start looking back into my personal projects again. Maybe next Wednesday I’ll restart the lisp streams.</p>
<p>Right, time to get some rest.</p>
<p>Peace.</p>
TaleSpire Dev Daily 582019-01-16T23:35:06+00:00http://www.techsnuffle.com/2019/01/16/talespire-dev-daily-58<p>Evening all, only a quick one tonight as I’m super tired. However I can let you know that we just had a pretty exciting planning session where we went through a bunch of designs of ways to improve the building system. From walls to thin corridors and from floors to props, the next iteration of the building system could be very interesting.</p>
<p>Not everything will be solved in the way that seems most obvious but when we have finished some prototype and have something to back up our ramblings we’ll put out a video (maybe a stream) where we talk about the design issue, our findings and where we are taking it.</p>
<p>Probably a month out but we’ll certainly keep you posted. Also don’t worry this wont delay the next batch of people admitted to the alpha, we’ll be letting more of you in before the new system lands.</p>
<p>Until tomorrow,</p>
<p>Ciao</p>
TaleSpire Dev Daily 572019-01-16T00:36:24+00:00http://www.techsnuffle.com/2019/01/16/talespire-dev-daily-57<p>Quick rundown of client-side from today:</p>
<ul>
<li>Fix a bug that occurred in battle mode when clicking next with an empty creature list</li>
<li>Made the <code class="language-plaintext highlighter-rouge">?</code> button hide the help if already open</li>
<li>Made the atmosphere music loop with a configurable delay between each loop.</li>
<li>Fixed drag-select so that it would work properly when you let go over a UI element</li>
<li>Atmosphere block are now drag-selectable</li>
<li>Try to ensure that steam definitely knows that the game has shutdown [0]</li>
<li>Don’t sync board if you are leaving due to the board being deleted.</li>
</ul>
<p>I also started on the following</p>
<ul>
<li>Can’t stand on <code class="language-plaintext highlighter-rouge">dirt (Tilled)</code></li>
<li>Unique creature panel is not updated on creature rename.</li>
</ul>
<p>The dirt one I though would be a doddle and it was missing some required information, however after adding that it still doesn’t work. It’ll be something stupid I’m overlooking but that’s where it is now</p>
<p>The unique creature ticket is the start of a few issues I’ll be working on in this area. For some reason the ui is not updating when the rename is applied and I have to close the panel and open it again.</p>
<p>After that I’ve got to look into some issues with creature ownership. Those are annoying as I don’t know what triggers them so it’s gonna be a lot of logging, testing and code review. We’ve already seen this mess with people’s streams so it’s a high priority one for me right now</p>
<p>I also took a couple of hours to look at some work I need to do on the backend. One of the pressing tickets is that, currently, there is no limit to the number of older versions of your boards we store. We obviously need this as storing more that we need increases our costs and thus limits how far each sale of TS can take us. It’s not a hard task but having some time to muse over the best way to structure the queries was nice. The second backend task was to design a basic stats collecting module so we can start getting better info on how many people are active at different times of day and, on average, what kinds of requests they make. This is important so we can make some better guesses at how load etc will increase as we move from hundreds to thousands of users and beyond.</p>
<p>One last thing, when I went to bed last night I realized that the line of sight technique is pretty simple if we limit the number of creatures involved. It goes roughly like this (for this example we will use 256 as the max number of creatures):</p>
<ul>
<li>make an ssbo which contains an array of 256 bools</li>
<li>find all creatures in range and give each an id between 0 & 256</li>
<li>clear the ssbo so all entries are false</li>
<li>render the scene from the selected creatures position, in the 6 cardinal directions, into a cubemap</li>
<li>render the occluders as black and creatures with their red color component based on their id. e.g. <code class="language-plaintext highlighter-rouge">vec4(id/255)</code></li>
<li>draw a quad and use the fragment stage to read a single texel from each face. Write <code class="language-plaintext highlighter-rouge">true</code> into the array using the red component of the texel as the index into the ssbo array e.g. <code class="language-plaintext highlighter-rouge">result[(int)(sample.r * 256)] = true</code></li>
<li>pull the data back to the cpu side.</li>
</ul>
<p>The slots that are true state that the respective creature is in line of sight. We can use a 512x512 texture to trade off accuracy for speed but given that we only need this once per time something moves it should be no problem to have something a little higher res.</p>
<p>I haven’t thought of a way to do it without an ssbo (or writable image) yet though.</p>
<p>I might make a prototype of this in lisp as a stream next week. Sounds pretty achievable in 2 hours.</p>
<p>Goodnight folks, back tomorrow with more.</p>
<p>[0] We can the Steamworks shutdown call in an OnDestroy which should have been triggered on all normal terminations. However sometimes it still didnt work, so we’ve added another call to it from a method that is explicitly called on shutdown.
!</p>
TaleSpire Dev Daily 562019-01-15T00:59:34+00:00http://www.techsnuffle.com/2019/01/15/talespire-dev-daily-56<p>Hey again folks, these dailies had to be put to the side during the crunch to get the alpha out of the door, but now a few days have passed I’m ready to get them started again.</p>
<p>Today has been a slow day as I guess my sleep schedule is still somewhat recovering, however I worked on a couple of bugs.</p>
<p>The first was <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/47">‘Ambient lighting stacks rather than replaces prior atmospheric lighting’</a>. This turned out to have a simple mistake which took me a while to find as I was sure it was an error in the LUT post-process. Instead it was that some atmospheres don’t specify new values for the lighting and, instead of the defaults being used, the previous value just remained.</p>
<p>The next were <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/51">‘Line of sight is behaving oddly’</a> & <a href="https://github.com/Bouncyrock/TaleSpire-Alpha-Public-Issue-Tracker/issues/61">‘Line of sight indicators show for creatures on all floors’</a>. These ones have a number of components to it that make it fiddly so currently I’m working on some tweaks (hacks?) to what we have so that the result is more acceptable whilst we come up with a better approach to replace it in future.</p>
<p>Previous, in order to see if one creature could see another we just had a simple sphere cast between the two.</p>
<p><img src="/assets/images/los0.png" alt="0" /></p>
<p>Issues included not taking creature eye position into account and that if the ray clipped some scenery, as in the case above, the creature would disappear even though most of it was in view.</p>
<p>To mitigate this I’m now taking multiple raycasts between an approximation of the head positions of the two creatures. We take one ray from the center of the head, one from slightly to the left and again from slightly to the right. On the destination creature we aim a bit wider in order to take the model size into account.</p>
<p>We also cast 1 ray from the source creature’s head to close to the base of the target creature. This lets us catch cases where a creature can part of another creature on a lower floor.</p>
<p><img src="/assets/images/los1.png" alt="1" /></p>
<p>As mentioned before, these are really tweaks to patch over issues with the current approach. One thing we’ve mused over is capturing a low res version of the scene into a cubemap (as in some lightprobe techniques) but naturally instead of rendering the colors of the scene we render IDs of creatures into the result. we can then see exactly what the creature would be able to see (more or less) and could accumulate the IDs into an SSBO (or similar buffer) to be used the following frame for showing/hiding the creatures in view.</p>
<p>The only non-trivial[0] part is the accumulation pass but it sounds like a perfect task for a simple compute shader.</p>
<p>I’ll write more about the new approach when I have done some more work there.</p>
<p>I’m gonna push my line of sight tweaks tonight so the rest of the team can have a play with them. Hopefully they’ll be out in the next update.</p>
<p>In non technical news it’s amazing to have the alpha out there. It’s mind blowing how quickly you see things you didn’t think of (someone playing chess in it for example :D ) and it’s intimidating how quickly the community content overpowers your ability to both keep up to date, and get work done. Some folks have already been amazing at filing really good bug reports in our issue tracker and guiding the new people as they first settle in, so it bodes very well for this as it scales.</p>
<p>Another thing I’ve not personally found a good avenue for is feature requests. Any game, tool, whatever.. feels a certain way due as much to what it doesn’t do as what it does. Setting good limits and excelling in your space makes for a better experience than piling it all on. Some features are non-brainers and we will definitely want to get them in, however there is larger set of perfectly reasonable features that just aren’t a good fit for this project. I guess this is part of the reason I see TS to be part of the roleplaying ecosystem rather than something that’s trying to replace anything. I’d sooner see 5 focused competitors than one bloated tool without focus.</p>
<p>The issue then is handling these tickets. If you’ve seen sites like the now dead Ubuntu Brainstorm you’ll know the issue. Things that are accepted get fixed but the things that don’t accumulate and can become little foci of disappointment. You can prune these but that too has to be done with care as otherwise you either get a lot of duplicate requests or you could piss off the great folks that took the time to request in the first place. I have no answer to this and I may just be over-thinking it anyway. I expect we’ll just put up a trello (or something similar) and just see how we go.</p>
<p>Tomorrow I’m kicking off with some work on ‘atmosphere music not looping’, some small UI bugs and then I’ll dive back into the big pool of bugs and see which can be swatted next.</p>
<p>Until then,</p>
<p>Peace.</p>
<p>p.s. After going to be I quickly realized it’s easy to do the line of sight stuff just using fragment shaders. Trade off is fixed max number of creatures (e.g. 256) but that seems ttoally fine given the current numbers of creatures we are dealing with.</p>
<p>[0] non-trivial as I haven’t done much compute work and none in Unity so far.</p>
TaleSpire Dev Daily 552018-12-17T23:26:33+00:00http://www.techsnuffle.com/2018/12/17/talespire-dev-daily-55<p>A fairly productive day today.</p>
<ul>
<li>Cleaned up client & backend code for Steam signup/signin. Feels good now</li>
<li>return validated alias from backend on signin</li>
<li>tweaks to how creature’s memories of boards are handled for serializing</li>
<li>refactored how and when unique creatures leave board and when they are name non-unique</li>
<li>tweaks to summon creature to allow specifying the location it will be summoned to.</li>
</ul>
<p>I also had been working on the fog of war to fix a bug where doors & walls flush against a zone boundary would not be shown if the character with on in the adjacent zone. I was pretty happy with how it was behaving earlier but I must have had some spurious data as now I’m seeing some very odd cases where tiles aren’t appearing. Beats me how that happened but I’ll need to fix that up in the morning.</p>
<p>I’m also off to Jonny’s tomorrow so we can work a bit closer of a few bugs & features. Remote working is great but occasionally it really helps to jump on the train and yell at the same monitor :)</p>
<p>Peace</p>
TaleSpire Dev Daily 542018-12-16T22:48:41+00:00http://www.techsnuffle.com/2018/12/16/talespire-dev-daily-54<p>It’s a quick post as I’m still working for an hour or so.</p>
<p>Save & sync on exit is working fine now. I made a few changes on the backend to support user aliases that are separate from signin_names (for those in future who will log in without using something like steam). Aliases are allowed to clash with the aliases of others (like profile name in steam).</p>
<p>I’m close with the steam signin but there is something I’m missing as I’m getting an error message back from steam on the server side. I’ll get it soon but it may be tomorrow as I’m running out of hours for today :(</p>
<p>C’est la vie.</p>
<p>Back to work.</p>
<p><code class="language-plaintext highlighter-rouge">[EDIT 23:07]:</code> It works! well, it crashes past the point where Steam has accepted the token, so yay. Login will be done real damn soon :)</p>
TaleSpire Dev Daily 532018-12-16T01:41:28+00:00http://www.techsnuffle.com/2018/12/16/talespire-dev-daily-53<p>I’ve enjoyed today.</p>
<p>We (the TaleSpire team) had a quick meeting to look at what was feasible in the days remaining. We are still going to try the closed alpha out on the 31st so it’s all hands (all 6 hands :D) in order to make that. It’s gonna be tight but it could work. Minimal viable product is the name of the game.</p>
<p>Campaign history is in a very basic working state and so I jumped back to looking at unique creatures. The UI had a few requirements I hadn’t met yet so I did a bunch of work getting that in shape. I finished that about half an hour ago. In short the list of data describing the unique creatures in the campaign can be used to spawn the previews we use when placing creatures. When the preview is placed we spawn the actual creature object at that spot passing in the ‘unique creature info’ this let’s us populate all the correct values (name, hp, etc) and tell the backend that the creature has moved boards.</p>
<p>UI is getting tighter as Jonny is hammering at it night an day. We are not going to have the final look by the 31st but we will be iterating on it pretty hard (along with everything else) in the new year.</p>
<p>Tomorrow my primary two tickets are getting the Steam signup/login finalized for the alpha and fix the ‘sync on quit’ feature. If neither of those cause any drama then I’ll probably look into a rather cute fog of war issue. Currently there is a case where if the obstruction object is on the boundary of a zone it may not show, which is terribly confusing when it’s a locked door :p</p>
<p>Goodnight folks!</p>
TaleSpire Dev Daily 522018-12-13T23:49:47+00:00http://www.techsnuffle.com/2018/12/13/talespire-dev-daily-52<p>Today has gone well. Not a whole bunch of stuff to tell but the invite panel is fine now and ui work is continuing steadily.</p>
<p>I found some bugs that require a tiny bit of context to explain.</p>
<p>Every tile in the board has an ID the game can use to address it. These IDs are how the network code specifies a tile when it wants to say something like ‘delete that tile’. Naturally IDs need to keep the same on every client and no two tiles should have the same id.</p>
<p>The bugs resulted in duplicate IDs. What was happening was that we synchronized the IDs when sending the board and we weren’t including the ones from things that had been deleted and were in the undo/redo history. This could result in the following sitation:</p>
<ul>
<li>Player 1 places 10 tiles (ids 0 - 10)</li>
<li>Player 1 leaves</li>
<li>Player 2 deletes some of the tiles (ids 5 - 10)</li>
<li>Player 1 returns and Player 2 sends Player 1 the board (sending ids 0 - 4)</li>
<li>Player 1 places 3 tiles (starting at 5 as it thinks that is free)</li>
<li>Player 2 hits undo (restoring tiles 5-10)</li>
<li>tiles 5-8 now have duplicates.</li>
</ul>
<p>We could either issue new IDs on undo/redo or just make sure we keep better track of ids in the undo/redo history. The latter felt like less of a band-aid so I refactored the code to handle that. It took a while to test enough to feel comfortable that it was behaving properly again but it seems ok (comforting no?).</p>
<p>This delayed some things but is always worth it as it’s such an ugly bug when it appears as it normal manifests differently on the different player’s clients.</p>
<p>Tomorrow I’m back on session history which in theory should be simple. I think after that I’ll need to tackle one of the remaining tickets which could require significant re-engineer in the worst case (most of the tickets have fairly understood scope).</p>
<p>Until next time!</p>
TaleSpire Dev Daily 512018-12-12T23:52:24+00:00http://www.techsnuffle.com/2018/12/12/talespire-dev-daily-51<p>Today we landed the new UI, we are now hooking it up and squashing bugs.</p>
<p>The tickets are rather specific but here are a few as the daily so far is a tad short :)</p>
<ul>
<li>Start hooking up the campaign invite panel</li>
<li>Allow a lone Gm who is in player mode to perform actions that would normally require Gm approval without the request.</li>
<li>Add hooks for providing strings to the session history panel (only the lua state-machine scripts are using it right now)</li>
<li>Handle more cases where board loading can be interrupted</li>
<li>Tweaks to the radial menu to make it feel nicer (such an insightful daily)</li>
<li>Change under what circumstance we focus the camera on a creature (For some reason I had it set up to focus on the first time you selected any creature)</li>
<li>More things that are so specific to the internals that it would be meaningless</li>
</ul>
<p>Ok. That’s enough that I’m not entirely wasting people’s time by posting this.</p>
<p>Tomorrow we’ll be working on more of the same. I also need to look at the code to react to a player’s rights (e.g. if they can gm) being changed during the session.</p>
<p>Peace</p>
TaleSpire Dev Daily 502018-12-12T00:29:24+00:00http://www.techsnuffle.com/2018/12/12/talespire-dev-daily-50<p>I hadn’t expected to be doing a fairly hefty refactor of the board loading today but that’s where I’ve found myself. I’m trying to nail down the behavior that occurs when the user you are syncing the board from leaves during that sync, or if the sync fails, or.. well you get the idea, we are looking at failure cases and how they behave.</p>
<p>The downside is having to retest a bunch of basic functionality but naturally the upside is that we are fixing issues the old one had.</p>
<p>The new UI hasn’t landed yet so nothing new to report there yet.</p>
<p>The only other important thing I’ve been working on today is adding a retry callback to our http request so that, in event of failure, you can choose if the request is retried or not. This was a pretty quick change as a bunch of the csharp code that wraps the calls to the backend is generated from a description on the erlang side. This has made it easy to make frequent changes to the backend api and know that the frontend code matches it exactly. More on that another time though.</p>
<p>Goodnight folks. Tomorrow is just gonna be more of this robustness work.</p>
<p>p.s. Reentrancy in state machines is a pain</p>
TaleSpire Dev Daily 492018-12-10T23:52:19+00:00http://www.techsnuffle.com/2018/12/10/talespire-dev-daily-49<p>Hey again, back after a few days without dailies.</p>
<p>Development is still going ok. Jonny has been working hard on getting the new UI to the point where we can merge it into master, looks like we are close there now. Once that lands we can hook things up and write the features that were waiting on that.</p>
<p>I mainly been working on query retries and managing the connection to the backend. I’ve merged in the branch with support for Steam login so I can get back to that soon. Board checkpointing took a little longer than planned but is working nicely now. At the start of each play session a new file for the board is created on the backend so we keep a history of those boards. For now we will just let these pile up but we’ll have to look into how many we want to keep around in future.</p>
<p>Another boring but necessary feature from today was adding limits to the number of campaign invite codes a person can make per minute and per day.</p>
<p>Tomorrow I want to look at having queries check with the connection manager to see if they should bother retrying on failure. I also need to look at the error codes from the backend queries so we can give meaning messages or take specific actions. After that it might be making the version of the steam login that we will use in the alpha.</p>
<p>Along with this we are squashing bugs but also finding more so the deadline is still looking very tight. I mean the worst thing that happens is the alpha is in January, but man.. I’d really love to get it out this year.</p>
<p>Ah well, gotta keep hammering.</p>
<p>Seeya tomorrow</p>
Response to 'Reasons why lisp games suffer'2018-12-07T08:22:54+00:00http://www.techsnuffle.com/2018/12/07/reasons-why-lisp-games-suffer-corrections<p>I haven’t been on <code class="language-plaintext highlighter-rouge">r/lisp</code> for a week or so due to being rather low on free time as we try and get an <a href="http://talespire.com">alpha of our game</a> out of the door. When I did pop back I saw <a href="https://terminal625.blogspot.com/2018/12/5-reasons-why-lisp-games-suffer-and.html">‘5 Reasons why Lisp Games Suffer and proposed solution’</a> which I thought would be interesting, but what I found was bizarre and misinforming.</p>
<p>This article is, in my opinion, wrong from end to end so I’d like to go through it a bit. I realize this may come across as shitting on the authors intent which isn’t my goal however being named explicitly in the article I feel like I can respond in some fashion. Also it’ll sound like I’m shitting on lisp, which I don’t feel I am. I adore working in this language but it is also just a piece of technology, just a tool, and we should be realistic about our tools when it comes to a given context.</p>
<h3 id="the-article">The article</h3>
<p>Let’s start with the first line</p>
<blockquote>
<p>Lisp games, like the Lisp AI and Lisp Machine hype in the 1980’s, has proven yet again to fail to deliver.</p>
</blockquote>
<p>This implies some massive lisp game hype akin to that of the lisp machines in the ai boom..so where is this hype? I’ve been around the lisp game community for a while now and none of the people who have been most active in making things are spouting anything like that line suggests. If there are people exposing lisp as some way to fix the game industry then they probably havent been making games in lisp. A lot goes into a game and decent programming languages are one part.</p>
<blockquote>
<p>Let’s say I’m a lisp noob, or even an average gamer. I think, hey, what’s the state of Lisp in games?</p>
</blockquote>
<p>Ultimately gamers don’t give a shit about your tech choices. If it runs well and is fun it could be TCL and MUMPS for all most people will care. Yes there are higher profile articles diving into the tech of games and these get a lot of followers but this doesn’t mean the majority are peering into the internals and judging games based on this.</p>
<p><strong>sidenote:</strong> <em>for those interested in this stuff I highly recommend <a href="http://www.adriancourreges.com/blog/2015/11/02/gta-v-graphics-study/">Adrian Courrèges</a> articles</em></p>
<blockquote>
<p>The Poverty of #Lispgames TLDR: Lisp games has many channels, but not enough content</p>
</blockquote>
<p>I’m not sure what ‘not enough’ means in this case, not enough for what? for who? If it’s not enough to satisfy the fallacious first line of the article then there is no doubt that this will never be satisfied.</p>
<p>Setting up channels for sharing content is a cheap endeavor compared to making the content. People are having fun, they enjoy lisp and they want to share stuff and big platforms make that easy, I’m not sure that holding off making those channels until the community reaches some assumed level of success makes any sense. Unless your goals and assumptions are rather out of sync with what those making these channels seems to be.</p>
<blockquote>
<p>But where are the games?</p>
</blockquote>
<p>SURPRISE! games are hard and just having a nice language doesn’t make that go away.</p>
<p>Also a lot of us enjoy working on problems, and games afford us a lot of interesting limitations which allow us to stretch our minds by finding solutions that would fit in this language we love.</p>
<p>CEPL is a prime example of this. CEPL doesn’t need to exist at all, it’s purely my attempt to make a high level api with which I could <em>play</em> with GL and yet which doesn’t hide any of GL’s features. This is tricky, fun problem as I feel that many apis I have worked with provide simplicity by hiding things and I don’t want that.</p>
<p>If my drive was to make a specific game I would shut up and make that game… BUT, and this is an important bit, if your goal is to make a specific game then you should be using the best tools available to make that game, and that likely is not going to include lisp.</p>
<p>That might sound odd given my background but please bear with me, I’ll rant a bit more about this below. Let’s wade through the rest of this thing first.</p>
<blockquote>
<p>Outnumbered by Blub</p>
</blockquote>
<p>This section is just a bunch of BS based on false assumptions.</p>
<p>Lisp is a niche language, and making games is a niche within it. Golly gosh I wonder why there isnt much being made.</p>
<p>Even within that sub-niche there is a significant percentage of people who just want to make tools, not games, so there goes a few more potential games.</p>
<p>Even the use of the term ‘blub’ pisses me off here. When we talk about making a game we have left the realm of being academic about a language and have entered the world of making a product. That product has to run on real devices used by real people. Those devices have actual hardware that have actual characteristic and so a language’s ability to leverage those devices is what dictates how good it can be for a given task. When you speak of non lisps are ‘blub’ you disregard the actual characteristic of those languages and how they fit to the task.</p>
<p>There are many folks in the game industry who don’t like c++, but it’s still the number one language used for 2 core reasons (there are more but for my narrative 2 will suffice):</p>
<ol>
<li>
<p>Because it remains one of the few major languages to give the kind of control over data and instruction layout that is <em>mandatory</em> if you want to ship a high end game on all platforms.</p>
</li>
<li>
<p>Momentum. There are a lot of existing libraries written in C++ and C++ is difficult to interface with efficiently from other languages. And yes I know about <code class="language-plaintext highlighter-rouge">extern "C"</code> but try taking the <a href="https://github.com/bulletphysics/bullet3">bullet physics library</a> or even <a href="https://www.autodesk.com/developer-network/platform-technologies/fbx-sdk-2019-0">the fbx sdk</a> sdk and expose those with a sane C api. It’s a monster of a task and just not worth it for studios with the kinds of time and money budgets they have.</p>
</li>
</ol>
<blockquote>
<p>The Sorry Tale of Those Who Tread the Great Lisp Path</p>
</blockquote>
<p>The entitlement in this section is strong.</p>
<p>I don’t know how someone could call out a game creator by saying ‘So much for sticking to common lisp’ with a straight face. They don’t owe you, or lisp, anything. Lisp is a tool.</p>
<p>Same for warweasel’s work on Clinch. Clinch had crazily large ambitions and it didn’t work out. That happens, and it’s fine. Now he’s got a specific game he wants to work on and so has picked a tool to make that feasible, that is a good move.</p>
<p>Then there is just random bitching that they don’t like Shinmera’s video style. Not sure what that has to do with anything, however as a regular viewer I liked them. The world has room for long and shortform videos.</p>
<p>As someone who <a href="https://www.youtube.com/user/CBaggers/videos">does a little of that myself</a> I can tell you that my 5 minutes videos can often take longer to make than my 2 hour videos due to the number of takes and tweaks required to make sure the content hits all of the beats it needs to in such a short time slot. Even putting that aside, somehow equating this to a reason that lisp games are failing is a seriously warped outlook.</p>
<p>Next let’s take a quick detour before I get to the bit that really raised my eyebrows</p>
<blockquote>
<p>Difficult to install, difficult to run, difficult to mod, difficult to use</p>
</blockquote>
<p>Nothing here is in any way limited to games in lisp. It only applies to the developers so doesn’t affect potential popularity of the resulting game, and the whole ‘modding’ aspect of it doesn’t make any sense. I’ll spare you a diatribe on this unless it becomes necessary.</p>
<p>And Finally!</p>
<blockquote>
<p>Is Baggers leaving?</p>
</blockquote>
<p>Fuck off.</p>
<p>Why not just contact me if that was something so noteworthy you think it belongs in here. Although I’m pretty slow to reply to emails I do get to them. I’m also always logged into the lisp discord channel and am DM’able on twitter. In short there are a pile of ways to reach me and lack of even an attempt to do that is lazy at best.</p>
<p>I’d mind less if this was an off the cuff twitter comment, but this is an article purporting to shed light on some issue and provide some answers. Even the minimum amount of work would have answered your question. I even explained I wasn’t leaving in many of my streams (not that I expect people to be watching them, but if you make claims a minimum of due diligence wouldn’t hurt).</p>
<p>This whole article reeks of this kind of laziness, it points the finger at all the wrong things whilst missing everything that does stop me using lisp for serious games today.</p>
<p>For what it’s worth, I’m currently working my ass off to try and get a game out the door. I’ve saved up enough money that I could quit my job and try this but that stash wont last for long. So fuck yeah I’m going to pour myself into that work, fuck yeah I’m gonna work my hardest to see this thing succeed and fuck you for calling me out just because I haven’t given you significant free work in 4 months.</p>
<h3 id="actual-thoughts-on-lisps-applicability-for-games">Actual thoughts on lisp’s applicability for games</h3>
<p>Phew, with that out of the way I’ll try and speak a little to the original question. Why not lisp?</p>
<p>I am not going to make any bigger claims on the community or industry as a whole so this is just where I’m at right now. I’ll also touch a little on why I’m not using lisp for my current games.</p>
<h4 id="1-performance-matters">1. Performance matters</h4>
<p>You need to hit your target fps, whether that is 30 or 60 but it’s usually the latter. A game that runs too slow or that inconsistent performance doesn’t just stop being fun, it stops being a game. Of course many games don’t push the machine that hard, but we are talking about games as a whole and a vast number do.</p>
<p>We also need to run on mobile devices which don’t have the grunt your desktop gaming rig does.</p>
<p>Performance (beyond the efficiency gained from appropriate algorithms) is going to be dictated from your ability to control what is being processed and when. Maybe I can’t afford a GC in the middle of my frame, maybe to get the entity count I need I really have to maintain data locality.</p>
<p>ANSI CL doesn’t have many provisions to allow for this kind of control. Certain implementations do have some and you can do a lot with CFFI, but at what point do the restrictions you impose on yourself making another language a better candidate?</p>
<p>There is a lot of room for lisp to do good things in this space and I’m looking forward to working on some things in this area but for those who just want write a game today.. maybe it’s not for them.</p>
<h4 id="2-games-of-any-real-size-are-rarely-single-person-endeavors">2. Games of any real size are rarely single person endeavors</h4>
<p>Making a game from scratch is a lot of work. Even with libraries and engines (see point 2) you are likely to be collaborating with people. Given a limited budget this means you all need to be working at your peak, so trade-offs will be made to optimize for the team as a whole.</p>
<p>Yeah I’d love to code in lisp every day. But (even putting aside lisp not running everywhere I need it) i’d be throwing away 13 years of experience of another member of my team. There are 3 of us and so I just kneecapped a third of my team for what?</p>
<h4 id="3-libraries-and-engines">3. Libraries and Engines</h4>
<p>This is really a continuation of the last point but when you use an engine or library you lean on the work of hundreds or thousands of other developers. Have a peek at <a href="https://www.unrealengine.com/en-US/blog/unreal-engine-4-21-released">this changelog</a> for ONE sub-version of Unreal, then think about the tens or hundreds of man-years on work that has gone into that one version.</p>
<p>We have less of these libraries in lisp and so a lisp game developer today is going to need to do far more work than one using Unity (for example) due to being forced to create more from scratch.</p>
<h4 id="4-high-level-api-design-is-a-fickle-beast">4. High level api design is a fickle beast</h4>
<p>In lisp it’s very easy to make high level apis and DSL over systems. However how relevant a particular solution is to a given person varies wildly. It’s not just aesthetic either, the features of the language we choose have performance implications and not every game can afford to make the choices you can afford for yours. Generic methods are an example here, a great feature but everything has a cost.</p>
<p>In my opinion the approach of providing low level bindings over C libraries separately from the high level api is a way better way of fostering code reuse in lisp (borodust is a shining example of this).</p>
<h4 id="5-portability-matters">5. Portability matters</h4>
<p>I need to get code on mobile platforms & consoles. Lisp doesn’t help me here. I could shell out a shit tonne of money for lispworks or dabble with the relatively unknown mocl. But that only gets me mobile and it’s still only the language, I need to port all the tools and many of my C dependencies probably don’t work on the PS4.</p>
<p>The alternative is I can use Unity or Unreal for free and use code that was tuned for games on those platforms today.</p>
<h4 id="6-content-is-king">6. Content is king</h4>
<p>The content pipeline is staggeringly absent from the provoking article. The ability for you to consume models, video, images etc from the swathe of authoring tools out there is a serious challenge and having <a href="http://www.assimp.org/">assimp</a> and <a href="https://www.libsdl.org/projects/SDL_image/">sdl_image</a> is not nearly enough.</p>
<p>Many of the libraries you will need are C++ and not super amenable to being wrapped. Even for the ones that are that work still needs to be done and be maintained.</p>
<p>Remember that you are a member of a team and telling your artist to just use blender and obj is not an option.</p>
<p>With unity I can drag in FBX files (in the latest format) and I get what I expect. And the representation of the data in memory is being handled based on how I’m handling and rendering those objects.</p>
<h3 id="something-a-little-more-upbeat">Something a little more upbeat</h3>
<p>CL is a great language that still has a lot to give. I’m gonna be using it for a long time including for games. It lives in the place of being fantastic and performant for scripting and also having the control to be truly viable for higher performance code. The implantation specific extensions to the language open up a ton of possibilities for squeezing more out of the machine and the abilities that macros afford us means there is a lot of meaningful stuff we can do trivially that require complex and unwieldy systems in other languages.</p>
<p>For the right project and with the right support, CL can be a great tool in your journey to make a game. But it makes perfect sense that it isn’t that for everyone right now. And it’s ok if it doesn’t ever become that.</p>
<p>I’ll be back to lisping early next year and I’d love to natter more about these things but this post was too long a thousand words ago.</p>
<h3 id="to-the-original-author">To the original author</h3>
<p>I’d heartily recommend making a non-trivial game in lisp. Like something that was mainstream 10 years ago and try and remake it. The problems you run into will tell you so much and your fixes (if you choose to share them) will help the community in measurable ways.</p>
<p>Personally I’d recommend halflife 2. It’s 14 years old now but you should be able to find rips of the assets which removes that whole part of the equation.</p>
<h3 id="enough-lets-wrap-this-up">Enough, lets wrap this up</h3>
<p>To close I’d like to say the following:</p>
<p>If you’d like to make things, I’d advise trying to gain an understanding of yourself and what you need. What is driving you, what do you feel incomplete without. If you <em>need</em> a specific game to exist then ignore the pseudo-religious posturings of various technology advocates and just focus on making it a reality. Use the best tools for the job and get it out there. I’ll be rooting for you.</p>
TaleSpire Dev Daily 482018-12-07T00:03:31+00:00http://www.techsnuffle.com/2018/12/07/talespire-dev-daily-48<p>A quick one today. I got TaleSpire up and running again with the latest backend changes. Next we want to store multiple versions of each of your boards (we may expose this in future so you can restore an earlier version). All the code required is pretty much there but I just need to work out the strategy we want to use for when we make a new checkpoint and then plumb what we have together to do that.</p>
<p>Then it’s back to refactoring the queries for handling retries etc.</p>
<p>Time for sleep. Ciao</p>
TaleSpire Dev Daily 472018-12-06T00:03:13+00:00http://www.techsnuffle.com/2018/12/06/talespire-dev-daily-47<p>Some serious yak shaving has been taking place today.</p>
<p>The goal was to add the ‘manager’ that would keep track of the connection to the backend and handle cases where we lose connection. One task was to be able to inform a request if it should retry, but hmm.. that means I should really refactor the queries a little.</p>
<p>With my new goal I dug into the queries a bit and saw that I should really take care of the changes to board I needed to make first. We want to be able to keep multiple versions of the board on the server so we have a backup history of sorts. This requires some work on the database.</p>
<p>With my new goal I started writing the board changes and saw that a bunch of this code will need revisiting anyway as we expose some guids we use as keys and that it’s a good plan. So I made the basic changes and then switched task.</p>
<p>With my new goal..well it’s a monster. The guids were used in a tonne of places and so about 70% of the queries in the backend needed to be refactored. This has taken me the rest of the day but, as it’s work we needed to do for the release anyway, I’m not worried.</p>
<p>I’m not at the point where I have pushed a new staging build and am testing TaleSpire (spoiler there are still bugs). There are a good few hours of work before this is working properly again and I’m not sure how much I will get done tomorrow as I need to visit the doctor. We shall see anyway.</p>
<p>Until next time,</p>
<p>Peace</p>
TaleSpire Dev Daily 462018-12-05T00:18:09+00:00http://www.techsnuffle.com/2018/12/05/talespire-dev-daily-46<p>Evening all! Today has seen a bunch of bug fixes which are rather specific but I’ll mention a few.</p>
<ul>
<li>Changing floor rapidly on board load could cause assets not to load</li>
<li>Deleting the board you are in now takes you to the campaigns default board (currently a new empty board)</li>
<li>Fix for case some requested data arrives for a board asset which has been deleted</li>
<li>Grabbing a board asset whilst board loading caused error as creature landed in invalid area</li>
<li>This is half the fix, the other half is a UI fix to make sure you can’t pick things up before the zone is fully loaded</li>
<li>Add inspector button for checking for duplicate network ids in board assets</li>
<li>Fix case which resulted in duplicate network ids after reloading board</li>
<li>Add compression to network messages for building/deleting multiple tiles</li>
<li>batch multi-tile messages to avoid making single message too large</li>
<li>not as robust as I’d like but good enough for alpha</li>
<li>Fix case where other player deleting assets we have selected caused our selected region to grow</li>
<li>A bunch of code cleanups</li>
</ul>
<p>I also hit an annoying aspect of mono. In c# you are able to set the <a href="https://docs.microsoft.com/en-us/dotnet/api/system.net.security.remotecertificatevalidationcallback?view=netframework-4.7.2">validator</a> used when a HttpWebRequest needs to check if a http request is valid. One of the things that is meant to be passed to that callback is the certificate chain. I use this to verify the intermediates on top of our certificate being checked. On MS c# this works perfectly, but in the mono version unity is using it isn’t. Super aggravating but not much we can do right now, hopefully it will improve as the .net version support in Unity gets better.</p>
<p>The standard certificate validation is fine of course so no worries on the security side. I just wanted the option to do a bit extra.</p>
<p>Anyhoo that’s all for today. Tomorrow I am looking at managing the connection to the backend and the handling/retrying/etc of failed requests.</p>
<p>Peace.</p>
TaleSpire Dev Daily 452018-12-03T23:47:07+00:00http://www.techsnuffle.com/2018/12/03/talespire-dev-daily-45<p>Ah today has been fun again. A few things that were done were:</p>
<ul>
<li>A few more little fixes to serialization of unique characters,</li>
<li>If you delete a character that is unique, remove it’s ‘uniqueness’ server side</li>
<li>Added text to help debug fog of war (this was great, more below)</li>
<li>fix two bugs in fog of war search when the search crossed zones</li>
<li>make sure that when the octree node is larger than a tile that the fog of war search takes that into account in calculating distance walked (just fixed up some lazyness when coding the feature)</li>
<li>Fix case where creatures you don’t control seem to load in an incorrect position when you join and existing play session on a board</li>
<li>Fix dumb bug where non GMs weren’t loading unique creatures properly if they were the first player to reach a board</li>
<li>Fix bug in quadtree deserialization</li>
<li>Serialize fog of war for unique creatures. Sync it to players on join and store it in board so you can leave and rejoin board much later and it is remembered.</li>
<li>Add code to recognise and upgrade boards stored in an old format</li>
<li>other misc small fixes</li>
</ul>
<p>Of these the most fun was the adding some additional info to the debug view of the fog of war so I’ll jsut rabbit on a bit about that. This really is one of the places where Unity shines, its so easy to add additional UI to the editor, not just in the inspectors but in the scene view too. I was struggling working out why the fog of war was behaving strangely so I decided to show a number at every ‘step’ of the search to show how far the character could still walk (how many steps remaining). It took a few tweaks as the fog of war search is done on a different thread and UI elements have to be drawn from the UI thread, also I wanted the text to remain visible for a number of seconds. Anyhoo the result was that every time you placed a creature the scene view looked like this:</p>
<p><img src="/assets/images/fillDebug.png" alt="debug0" /></p>
<p><em>the red circle is where the creature is</em></p>
<p>When I did this in the morning I could immediately see that the numbers were wrong when the search crossed a boundary. Turns out that was a simple case of not initializing an int in one of the constructors for the search. Also there was a case where the number went from 0 to over 4 billion, that’s an easy one, unsigned integer wrap around when subtracting 1 from 0. Each is a simple issue but it just became obvious as soon as I could see a tiny bit more info. Yay tools!</p>
<p>Out of curiosity I added a little slider to the creature’s inspector to control how far the search would walk, this was great to fine tune and find a default value that felt nice.</p>
<p><img src="/assets/images/fillDebug2.png" alt="debug1" /></p>
<p>The other stuff doesnt make for nice pictures but is all obviously needed for the alpha.</p>
<p>Here’s to tomorrow, hopefully it’s as productive.</p>
<p>Peace</p>
TaleSpire Dev Daily 442018-12-02T01:18:54+00:00http://www.techsnuffle.com/2018/12/02/talespire-dev-daily-44<p>Hey again! today has been spent taking places where we would make multiple calls to the backend and merging them into single calls. Makes stuff a bit faster, simpler and more robust.</p>
<p>There’s really not all that much to say about that. It’s a bit fiddly as it is relatively easy to introduce subtle bugs and I’ve been chasing down a few of those (tomorrow morning will start with squashing one).</p>
<p>Other than finishing that work off I’ll be getting things in a place to merge into the master branch. I’m not as far along as I’d like but hopefully I’ll pick up pace again.</p>
<p>G’dnight all</p>
TaleSpire Dev Daily 432018-12-01T00:21:06+00:00http://www.techsnuffle.com/2018/12/01/talespire-dev-daily-43<p>Work rolls onwards. Unique creatures seem to be working ok. I still want to persist who controls them across sessions so I’ve added the server-side code for that, now I just need to hook that up client-side.</p>
<p>A GM can now easily move all the players from one board to another, although we still need to add the UI for this.</p>
<p>This weekend I’m going to focus on getting the branch I’ve been working on merged into master. To do this I add some dev features back in. If I have time I need to add the code to handle cases where the connection gets flakey.</p>
<p>Hopefully we will have some news on the updated UI in a week or so.</p>
<p>Watch this space!</p>
TaleSpire Dev Daily 422018-11-28T23:07:00+00:00http://www.techsnuffle.com/2018/11/28/talespire-dev-daily-42<p>Hi all, sorry for the lack of daily yesterday. I got a bit burnt out so I took a short break to get my head back on straight. Today I wrapped up some refactoring of code handling losing network connection and similar cases. Tomorrow I’m doing some work on the server side of the unique creature code. I’m hoping that’s something I can get done in the morning and then looking code to move your party between board smoothly (that should be easy too).</p>
<p>Hopefully I’ll have more details tomorrow.</p>
<p>Thanks for stopping by!</p>
TaleSpire Dev Daily 412018-11-26T23:14:12+00:00http://www.techsnuffle.com/2018/11/26/talespire-dev-daily-41<p>Well, to balance out the successful days, there is today. Today I had a issue when some of the rush to get features written had resulted in some code that was harder to deal with that it needed to be. ‘No worries’ I think I’ll just refactor XYZ and this task should be easy… 12 hours later and I can barely remember what the issue was that motivated the refactor. I have essentially spent a whole day for a questionable gain when time is what I have least of right now.</p>
<p>So yeah, that’s not a great day. At this point its probably worth just pushing through and finishing up the cleanups. Hopefully the changes are worth it, but if I were to do it again I’d probably have just shut my face and kept hacking until the alpha was out.</p>
<p>Let’s see if I’ve learned a lesson.</p>
<p>Until next time</p>
<p>Peace</p>
TaleSpire Dev Daily 402018-11-23T20:12:59+00:00http://www.techsnuffle.com/2018/11/23/talespire-dev-daily-40<iframe width="560" height="315" src="https://www.youtube.com/embed/TjR_sWK0sLw" frameborder="0"></iframe>
<p>As promised the video of hammering the networked undo/redo. It’s pretty stable in the video but I do have one bug in that system that I’m going to fix tomorrow (looks like a simple mistake in when updating the history stack, nothing too worrying).</p>
<p>Today has been focused on asset deleting and making sure that all the subsystems that have to hold references to board assets (like tiles) release them if the asset is deleted. An couple of hours was spent on refactoring the multi-select tool in my ongoing (entirely wholesome) love affair with state machines, its a good deal more robust now and the code to release delete assets is much faster. That latter part is important as any time you have a selection someone else might delete some or all of the tiles you have selected.</p>
<p>There is still a lot more I can get out of this speed-wise but it’s not wise to spend more time there given that it’s not yet an issue and there is so much to get done for the alpha.</p>
<p>On that subject, work continues. I’m up to 128 tickets on github :p but having gone through the codebase I feel the number of engine related tickets are not going to grow violently more before we release. Lots for the UI though :)</p>
<p>Right. I’m off to chill and watch ‘an american werewolf in london’, such a classic!</p>
<p>Peace.</p>
TaleSpire Dev Daily 392018-11-22T23:26:36+00:00http://www.techsnuffle.com/2018/11/22/talespire-dev-daily-39<p>Happily it’s all still going well. I started the day switching the board syncing from http to https, a small task but of course it’s very important. The only reason it hadn’t been done already what that things were being built quickly to test the mechanism. I’ll ramble on about my future plans regarding security another day.</p>
<p>Thanks to some recent changes to the board-asset code we now know the place the pieces will land the moment they are released by the player. This lets us start the fog of war update a few frames earlier which makes things feel snappier.</p>
<p>Also thanks to <a href="https://twitter.com/jonnyree">@jonnyree</a> adding drag-select I added deletion of multiple tiles. It feels so good to crazily mash ctrl-z and ctrl-y on one client and see that the result is totally stable on the other. I know of a few places where things could go wrong though so I am fixed them up tomorrow.</p>
<p>I’ll post a video of the undo/redo tomorrow too, even though it’s doing what it has to do I’m still stoked to see it doing it so well :)</p>
<p>Alright, time to hit the hay.</p>
<p>Ciao</p>
TaleSpire Dev Daily 382018-11-21T17:38:38+00:00http://www.techsnuffle.com/2018/11/21/talespire-dev-daily-38<p>Today went pretty well. I mainly focused on adding invite codes for campaigns. This simply gives you a shortcode that your friends can paste into TaleSpire to join your otherwise hidden campaign.</p>
<p>We will add other ways of doing this post alpha but to start with this will be how it works. It makes it quite easy to drop the code into your discord chat and have everyone join pretty quickly. It also avoids that thing skype has where they allow everyone to query all other users by name when adding a contact.. gross.</p>
<p>We are pretty adamant about not being like social platforms which try to push events in your face in order to drive ‘engagement’, so whatever we add to this in future will need to maintain the goal of keeping TS polite.</p>
<p>Other that the above as miscellaneous fixes, I also went through the code base filing tickets for every todo I had so now I hope I have a slightly more complete view of what I need to do for the alpha release.</p>
<p>Until tomorrow, peace.</p>
TaleSpire Dev Daily 372018-11-20T16:33:08+00:00http://www.techsnuffle.com/2018/11/20/talespire-dev-daily-37<p>I started out the day with 65 tickets, I closed 8, so now I have 66 tickets. Yay programming :D</p>
<p>Just to be clear, this is just due to filing thing as I came upon them rather than breaking more things. So yup, it’s been a good day.</p>
<p>I’m not sure how interesting tickets like these are to anyone, but it’s stuff like:</p>
<ul>
<li>adding ‘delete board’</li>
<li>session keep-alive</li>
<li>fixes for silly ux things (focusing on the piece every time you pick it up)</li>
<li>staying in ‘cutscene mode’ (to stop players moving pieces) until a Gm has joined the session</li>
</ul>
<p>.. and even more specific stuff.</p>
<p>It’s all stuff that obviously needed doing they are all low risk tickets so they got left until more of the major system had been worked out.</p>
<p>Right time to go see some mates,</p>
<p>Seeya all!</p>
TaleSpire Dev Daily 362018-11-19T21:15:57+00:00http://www.techsnuffle.com/2018/11/19/talespire-dev-daily-36<p>After a couple of days of quiet I’m back. Saturday I was just tired and Sunday I wasn’t working due to a headache (probably just a bit overloaded).</p>
<p>Today has gone well. I’ve just spent it working through small issues. Changes to information passed back at login, syncing on exit, fixes to tile visibility after a build action, and more stuff.</p>
<p>Tomorrow I’m either going to hammer on with more tickets from the pile (there are plenty) or I’ll look at the synchronizing of the fog of war information. The serializing of that fog of war info is super simple but, as multiple players can ‘own’ the same creature, I need to have a think about who is in charge of calculating that stuff. Probably fine to just pick something dirt simple like ‘the player with the lowest player id’. We can always revisit this.</p>
<p>Alrighty, time for a drink.</p>
TaleSpire Dev Daily 352018-11-17T00:01:13+00:00http://www.techsnuffle.com/2018/11/17/talespire-dev-daily-35<p>Allo again,</p>
<p>Today I was working on the cutscene system and it was much more involved than expected. In short the reason was that it touched on a whole bunch of logic and UI code that (as it was a prototype) had never been written expecting multiple GMs.</p>
<p>Damn I just realized that’s pretty much all there is to say about it without having to rambling on about pointless minutia..bummer :D</p>
<p>Guess that’s it then! Tomorrow I’ll be traveling in the morning but then will probably look into refactoring the GM request UI. Should be some fun edge cases in that.</p>
<p>Seeya</p>
TaleSpire Dev Daily 342018-11-16T00:27:02+00:00http://www.techsnuffle.com/2018/11/16/talespire-dev-daily-34<p>It’s a little past midnight but who cares, it’s been a great day.</p>
<p>Jonny has got the basics of the new placement code working. It’s now much easier to move pieces under overhangs regardless of if they are on the same floor or not. He also got drag-select in so I will hook that up in a few days so we can delete/move multiple tiles.</p>
<p>I’ve been working on the game state and player handling code. It’s now much more reliable and a fair bit cleaner. This work makes sure that if ‘host’ (the client tasked with syncing to the server) leaves that the job is passed to another client.</p>
<p>I’ve also started work on supporting having the same user logged into multiple clients simultaneously. In the future this could allow us to provide additional tools that hook into the current play session.</p>
<p>It’s hard to stress how much nitty gritty stuff has been, if not implemented, at least properly speced out for the release. I’m stoked for the next 2 weeks of progress.</p>
<p>G’night folks.</p>
TaleSpire Dev Daily 332018-11-14T23:45:06+00:00http://www.techsnuffle.com/2018/11/14/talespire-dev-daily-33<p>Hi again folks,</p>
<p>I’m down at <a href="https://twitter.com/jonnyree">@jonnyree’s</a> place right now for a couple of days hacking on TaleSpire and it’s been going great. We have been going through the features that will be in the alpha and making decisions of behaviors that, up to now, were a bit vague. That has unblocked a bunch of tickets so they are all on my todo list for the next couple of weeks (not that I have been short of work).</p>
<p>Currently I’m fixing some bugs by setting up some simple state machines for managing the current game mode and client state. This just makes some things more explicit and makes the transitions between the states very explicit.</p>
<p>Tomorrow I’ll switch from this onto the rewrite of the placement code. This is the bit that handles the behavior of picking up and moving pieces. Naturally it’s critical to nail the feeling here as it’s its your primary interaction with the game. It seems like you could just attach the piece like a pendulum and let physics drive the experience but nope :) Lot’s a pair programming will be on the cards.</p>
<p>Alright, that’s the lot for tonight.</p>
<p>Peace</p>
TaleSpire Dev Daily 322018-11-12T23:20:26+00:00http://www.techsnuffle.com/2018/11/12/talespire-dev-daily-32<p>Good evening! Today I got the multiplayer undo/redo code merged in, fixed some bugs related to NetworkIDs for objects and started planning the code to handle host migration.</p>
<p>Tomorrow’s task will be to finish of the aforementioned plan and get as much of it implemented as possible. The goal there is just making sure the game knows what to do when players with specific roles leave or join and how to migrate responsibilities as it happens.</p>
<p>On Wednesday I’m back visiting <a href="https://twitter.com/jonnyree">@jonnyree</a> where we have a of tonne planning to do (especially around gameplay behaviors for the alpha) and we need to make a solid foundation for the new piece placement code. This is one area where the feel is super important so it’s best for us to pair up on that.</p>
<p>As every we’ll be keeping you posted as this progresses.</p>
<p>Seeya!</p>
TaleSpire Dev Daily 312018-11-11T23:30:48+00:00http://www.techsnuffle.com/2018/11/11/talespire-dev-daily-31<p>Today was spent chasing down smaller issues and, for features that need UI work, writing down behaviors which I don’t think we have defined well enough.</p>
<p>As an example for the latter we haven’t defined whether a player who does not have permission to GM can join a game without a GM. It might make sense as maybe the GM has said they’ll be there in 5 mins. It could be cool to let the player look around and re-familiarize themselves with the board, however there are also certain tasks that a player needs to have certain permissions to perform and that player may not have them. To be more specific some server calls require someone with GM rights.</p>
<p>The first task will be to define what behavior we think feels best and doesnt pose any risk to the integrity of the play session. Then we need to work out what technically needs doing and whether it will be feasible (along with everything else) to get done for the alpha.</p>
<p>Please don’t get to hung up on the example above though, it’s just and example and there are plenty more tickets to review :)</p>
<p>Today I also worked a little more on sync, making sure host migration worked properly. There are still some details to take care of there but it requires some of that planning mentioned above. I also am finally writing the undo/redo code for multiple simultaneous GMs that will be in the alpha. This is resulting in some changes to the serialization code which should help us down the line.</p>
<p>As the alpha rolls onward we will need to be able to make changes to the file format that the boards are stored in. To support this I’ve started stubbing out the format upgrade code. This will just handle the case where a newer version of the game is opening an older version of a board file. Simple but necessary stuff.</p>
<p>All in all it’s been an ok day. I’ll kick of tomorrow by finishing the undo/redo stuff and then dive into the next pile of tickets.</p>
<p>Until then,</p>
<p>Peace</p>
TaleSpire Dev Daily 302018-11-10T00:12:55+00:00http://www.techsnuffle.com/2018/11/10/talespire-dev-daily-30<p>Evening all!</p>
<p>Today was spent testing the ‘unique creature’ code across multiple clients. This went well and found a good few bugs. I rewrote the board sync as that was on the todo list for a while and started looking at other aspects on sync which need to be better.</p>
<p>The first one on my mind is making sure that each client knows whether their version of the board is more recent than the version on the server, it’s a simple problem but has a few subtleties to it. It’s not good enough to know that the server version is different as it is possible that you are building a board on your own (for a future session) and the power goes out. We would like to use the cached version of the level on restart and know that it can be pushed over the server version. A simple version id can handle this but I’d still like to muse a little longer on the potential downsides of this.</p>
<p>Another thing I need to consider is how scriptable assets handled with regard to sync, we obviously want to be timely about pushing the state of things like locked doors & chests to the server but with modding it will be possible to create assets who change state near constantly, how frequently do we really want to sync this? I’m not gonna spend too much time of this now as modding wont be exposed in the alpha and next year I’ll be adding support for huge boards which is going to require changes to sync anyway.</p>
<p>There are bunch of other little details but this will do for now,</p>
<p>Until tomorrow,</p>
<p>Peace</p>
TaleSpire Dev Daily 292018-11-08T23:06:52+00:00http://www.techsnuffle.com/2018/11/08/talespire-dev-daily-29<p>Evening folks, unique creatures seem to be syncing well now and I’m on a bit of a hunt to clean up a bunch of warnings that have been piling up for a while. This touches on one rather annoying aspect of Unity where, if a script has become disconnected from a prefab (due to a rename or something, it can’t tell you what the file used to be called so it’s damn hard to find out how to fix it. Bites when a bunch of stuff in unity is pretty great.</p>
<p>Anyhoo tomorrow I’ll introducing a second client and then will be doing more testing and fixes.</p>
<p>Thanks for stopping by!</p>
TaleSpire Dev Daily 282018-11-07T18:40:56+00:00http://www.techsnuffle.com/2018/11/07/talespire-dev-daily-28<p>Today has been spent in a test and fix loop. Every now an again I find something that I had missed or put off is actually required and then I have to push some changes server side. I’m fairly happy with that process so I’ll probably write a post on it sometime.</p>
<p>Tomorrow I’ll be carrying on with the same as today. I got some error to do with Photon IDs when spawning the unique creatures so I’ll start with that. Then it’ll be hooking up the calls to update server side info on the unique creatures.</p>
<p>Seeya tomorrow</p>
TaleSpire Dev Daily 272018-11-06T17:32:11+00:00http://www.techsnuffle.com/2018/11/06/talespire-dev-daily-27<p>Evening all, another small update. I’ve continued work on the syncing of unique creatures which are creatures that can move between boards and who’s stats, hp, inventory, etc are stored on the campaign (player controlled characters are the main example).</p>
<p>Tomorrow will be spent testing what I have so far and trying to find where it can break down, hopefully Thursday will then be spent cleaning it up. As such it’s going to be a couple more days with short dailys as there is very little to tell. It’s just small iterations and plenty of thinking :)</p>
<p>Until next time,
Peace</p>
TaleSpire Dev Daily 262018-11-05T20:55:06+00:00http://www.techsnuffle.com/2018/11/05/talespire-dev-daily-26<p>Hi again, today has been spent hammering away at more sync code. Campaigns and boards seem to be playing alright and we ‘unique creature’ stuff is starting to take shape too.</p>
<p>Hmm it’s a bit of a short daily but the rest of the work has mainly been tweaks and fixes, so yup that’s all for now.</p>
TaleSpire Dev Daily 252018-11-01T21:50:42+00:00http://www.techsnuffle.com/2018/11/01/talespire-dev-daily-25<p>Well today was a major pain in the ass. It started ok, I was exposing the recent changes to the api but it all went south when I got to syncing board data. Up until I had been lazy and had been saving the boards in the database as a binary blob, not good practice I know but it let us get moving quickly. Now of course we need a better path and so it was time to do something, there were a couple of choices that seemed obvious.</p>
<h3 id="sync-board-zones-to-database">Sync board zones to database</h3>
<p>This one is still keeping binary data in the db but we sync individual zones. This is cool as we can then pull data on demand which gets us in the direction of large board support. The issue though is that we don’t need that feature for the alpha and it’s a bit of an unknown what extra code we will need on the client side.</p>
<p>I don’t imagine it would be hard, but that never stopped problems turning out to be a nightmare before :)</p>
<p>Seriously though, given how tight it’s going to be to get the alpha out as it is we don’t need more unknowns.</p>
<h3 id="sync-the-whole-board-to-s3">Sync the whole board to S3</h3>
<p>This one was the clear winner from the client side, we use the same code we have but upload to amazon s3 instead. However the unknown here was on the server side. I’m still kind of a noob with s3 and the erlang library ecosystem often puts you in a place where you need to cobble some things together.</p>
<p>This time I was lucky in that the folks who make Chef have a library called <a href="https://github.com/chef/mini_s3">mini_s3</a>. However, as documentation is often thin of the ground, I needed to read parts of Chef to understand how to use it.. and this is where the noob thing caused issues.</p>
<p>Almost the biggest issue when trying to learn from code you don’t understand is that you have very poor metrics for separating things, it’s hard to derive what is intentional and what is a hack and terminology doesn’t yet give you the usual clues to help separate concepts. I went on a wild goose chase as I thought that Chef’s OrgID & Checksums were intrinsic to the s3 code so I was reading far back into the codebase trying to derive how they were getting the values so I could work out to do for our code.</p>
<p>Eventually, after a lot of shouting, I stumbled on my mistake and got things working. It’s painful as it was all for a single function, one that generates presigned urls to allow upload or download to a specific file on s3 for a limited period of time.</p>
<p>With that done I got back to what I though would get finished this morning, finishing exposing the new api functions and pushing the a version to staging.</p>
<p>That brings us to now. I need to do a little more testing (which I’ll do now) and then I’ll get back to hook this up to the engine.</p>
<p>Here’s to tomorrow</p>
TaleSpire Dev Daily 242018-10-31T17:24:21+00:00http://www.techsnuffle.com/2018/10/31/talespire-dev-daily-24<p>Hey folks, pushing this boulder up hill is slow going but it’s getting there.</p>
<p>Today I continued work on the ‘joining campaigns and boards’ code. Mainly I was looking at how we can used locally cached versions of the boards to make sure that when you log in you see stuff immediately. We of course ping the server to get the latest info on things like the campaigns you are a member of, but those requests can take a few seconds and the most common case is starting a new campaign or continuing and existing campaign.</p>
<p>Whilst working on the database side of this I was touching code I would have to revisit immediately when I got to ‘unique creatures’ (see yesterdays post for details on them) so I decided that I would write the tables and queries for those things today.</p>
<p>And that about brings us up to now, tomorrow will start with exposing this to the web front-end and then its back into the engine to consume this new stuff.</p>
<p>Seeya tomorrow!</p>
TaleSpire Dev Daily 232018-10-30T20:47:47+00:00http://www.techsnuffle.com/2018/10/30/talespire-dev-daily-23<p>Motivation is a fickle thing, after cranking through a bunch of quick tasks I end up craving something more meaty and meaningful to slam my head against, but give that month and I’m just craving the quick reward cycles of small tasks again.</p>
<p>Today was a switch back to the latter, I’ve been neck deep in the search stuff for so long the deadline went from feeling insane to hopeless. Getting that merged in late last night let me move onto refactoring how joining campaigns and boards is handled and it’s felt great.</p>
<p>A lot of the work has been tracing down what code owns what and massaging it into a new shape. <a href="https://www.photonengine.com/en-US/PUN">Photon’s PUN</a> library (the networking lib we use) has a concept of a ‘room’ where every client in the room can communicate. This maps nicely to our boards and this is what will allow us to have a single campaign with different party members in different boards concurrently (each with their own GMs if you have the people for it).</p>
<p>I have also stubbed out the code for the local and server autosave and started looking into how we handle ‘unique creatures’.</p>
<p>Internally we talk about any controllable thing being a creature, whether it’s a humanoid hero or a sewer rat. Most creatures below to the board and are stored with it, however a creature can be made unique which now means their data lives on the campaign and they can be moved between boards. Every player’s character is naturally a unique creature, but a GM can take any creature and make them unique. This can be cool for story reasons for example, maybe due to the player’s actions the aforementioned rat touches a forbidden emerald and becomes imbued with terrible knowledge, purpose and dark magic, the GM may want to have that rat be a reoccurring part of the story so making it unique makes that much easier to manage.</p>
<p>Ah I cant wait for all that to be working, tomorrow will be finishing off this board refactor and then I’m getting that unique stuff added. That will require changing serialization of both boards and campaigns but it shouldn’t be that tricky.</p>
<p>Until tomorrow,</p>
<p>Peace</p>
TaleSpire Dev Daily 222018-10-30T00:12:53+00:00http://www.techsnuffle.com/2018/10/30/talespire-dev-daily-22<p>Heya folks, last time was a <a href="https://www.youtube.com/watch?v=n0-aj_sTWQ0">video</a> where we looked at some development surrounding the fog of war implementation. I’ve been hammering away on bugs, both the ones that showed up on the stream and plenty of others.</p>
<p>Things are starting to get a bit more solid and I’ll be merging that work into our master branch tonight so the others can start testing and working with it too.</p>
<p>I’m not super happy with performance in a few cases so I’ll be going back to that but there is a lot of other stuff to do first.</p>
<p>To finish off tonight I need to do a little work on the asset building tool and then tomorrow will probably be when I start looking at refactoring a bunch of the code around joining campaigns and other network related stuff.</p>
<p>Seeya tomorrow</p>
<p><img src="assets/images/fogOfWarClimbing.png" alt="climbing search" /></p>
TaleSpire Dev Daily 202018-10-26T00:05:40+00:00http://www.techsnuffle.com/2018/10/26/talespire-dev-daily-20<p>Hi all, I’m going to delay doodles until the implementation is finished but we are changing how the fog of war works..again :p</p>
<p>The last version had some cases we hadn’t considered and so I’m hammering out a new version now. In fact I’m doing it now and don’t want to lose my focus so I’m gonna get back to that now. Sorry this is short but didnt want to leave you without an update.</p>
<p>Peace</p>
TaleSpire Dev Daily 192018-10-25T00:37:42+00:00http://www.techsnuffle.com/2018/10/25/talespire-dev-daily-19<p>Hmm today is a weird one to write up. On one had I made some progress on little tasks which are too specific to be worth expanding and on the other I found some rather annoying consequences of the fog of war implementation.</p>
<p>It all basically boils down to the current constraints result in some unintuitive or slightly sub-par experience and fixing them is fiddly. I want to do a big ol’ breakdown on this but I want to leave it until at least tomorrow as @jonnyree and I are going to have a planning meeting on this tomorrow.</p>
<p>This may result in a relatively large change so, given how little time we have left this year, we really need to nail down something that can work for the alpha and then stick with it.</p>
<p>This doesn’t stop us rewriting it during the alpha but that would make mean less user testing of each implementation which means less data and time to work with.</p>
<p>Trade-offs are.. well, a thing.</p>
<p>On that very insubstantial note,</p>
<p>Ciao</p>
TaleSpire Dev Daily 182018-10-23T17:11:57+00:00http://www.techsnuffle.com/2018/10/23/talespire-dev-daily-18<p>A tiny log today as I’m heading out to see some friends now.</p>
<p>Today I took a little break from engine work to look at server side. The main take has just been tweaking our the DB to store the data we want for campaigns and characters and supporting granular permissions for who can do what. Those controls may not all be exposed but it’s not hurting to have it there.</p>
<p>When the UI for the creating/editting campaigns lands I’ll expose this from the web server and hook it up.</p>
<p>Until tomorrow.</p>
<p>Peace</p>
TaleSpire Dev Daily 172018-10-22T23:56:46+00:00http://www.techsnuffle.com/2018/10/22/talespire-dev-daily-17<p>Hi again, today has been a day of churn. We have known for a couple of weeks that the way floors have worked in the prototype was unsatisfactory. For context you used to be able to delete floors and insert floors below others, which sounds great and felt great to.. on small boards. However when you start making a whole village and want to add a floor to a single building then the problems quickly become apparent, you add a floor to the entire village!</p>
<p>So instead of that we will be replacing it with the ability to select a region and:</p>
<ul>
<li>delete it</li>
<li>delete it and drop the tiles from floors above down</li>
<li>push tiles up to the next floor (which pushes the ones above up etc)</li>
</ul>
<p>This will give the level of control we had before but without the caveats. Of course this requires a little more work from the user, but it feels like a necessary inconvenience.</p>
<p>Also, in order to be able to support large board in the future we needed to get away from anything that operated on entire floors so chunking up the board has been on the todo list for a while (we already had started this for the fog of war code). This is where the churn came in as, as you can imagine, floors were pretty central in the codebase so changing that out has touched a tonne of code. Getting things to behave again after those changes took most of today but it’s creeping toward feeling solid again.</p>
<p>I’m not 100% sure what I’m working on tomorrow. I still have a lot of tidy-up work to do, I need to look into how to add ‘hill climbing’ to the octree fill (more on that another day) and I need to make some server-side changes very soon.</p>
<p>Guess I’ll see how I feel in the morning.</p>
<p>For now have a wee video showing the fog of war responding to the door opening. I have to switch between DM and player here as attempting to open the door will try sending a message to the DM.. who is on the same machine and the UI code gets a bit confused :)</p>
<p>Seeya</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/54uEaKz0aU8" frameborder="0"></iframe>
TaleSpire Dev Daily 162018-10-19T23:56:03+00:00http://www.techsnuffle.com/2018/10/19/talespire-dev-daily-16<p>Almost forgot this evening’s one so it’s gonna have to be quick. Today I worked on the import of the occlusion data from TaleWeaver I’ve got a couple of small issues left where Unity’s behavior mas left me a touch confused but no doubt I’m just being a dummy.</p>
<p>I have noticed a shortening of these dailys over the last few days but fear not, they’ll be getting a bit longer as things get exciting again :)</p>
<p>Peace</p>
TaleSpire Dev Daily 152018-10-18T23:35:57+00:00http://www.techsnuffle.com/2018/10/18/talespire-dev-daily-15<p>Good-evening folks! Today was spent getting used to the profile, finding out ‘deep-profile’ is useless, finding out that some parts are pretty fragile and then making a few speedups.</p>
<p>Mainly it has just be reducing the amount of info that needs to be written into the octrees for each zone. This is kind of avoiding the real issues though so we will need to get back to actually reducing the time things take, but that is for another day. The main thing to tackle was rewriting the code that find which parts of a tile are solid and which aren’t. This was previously hacked into TaleSpire itself and happened on load, but this is daft as it never changes so I have moved this to the tile modding tools.</p>
<p>I’m now going through all the existing assets and adding extra info to all of the floors and reexporting the assets with their new collision info. Tomorrow will basically be finishing that off and using this new data in TaleSpire, which should tale performance back to where it was. I will also serialize the octrees for file/load and syncing floors as it could be a pretty quick win.</p>
<p>Seeya</p>
TaleSpire Dev Daily 142018-10-17T17:24:50+00:00http://www.techsnuffle.com/2018/10/17/talespire-dev-daily-14<p>Today I got a bunch of thing fixed so the new fog of war is starting to behave itself. There is still a bunch to do so my 7 day estimate at the beginning of last week is looking dumber by the day, however it’s still a good day.</p>
<p>I can load boards again, walk my character around and see that the systems are communicating properly. The next things on the todo are (in no particular order):</p>
<ul>
<li>Make tile deletion work with the fog of war</li>
<li>Improve performance of adding tiles (see below)</li>
<li>Only add occluders (a special kind of tile) to the fog of war zones</li>
<li>Only update tile visibility info for zones that are on screen or nearby</li>
<li>serialize the fog of war (maybe)</li>
</ul>
<p>Right now, when loading boards there is an expected but noticeable lag. I know it’s related to the fog of war system but I need to know which parts are causing the issue. I’m definitely doing that before serializing the fog of war data as that would just mask any issues.</p>
<p>Right, gotta go get ready for a stream</p>
<p>Seeya</p>
TaleSpire Dev Daily 132018-10-16T23:10:44+00:00http://www.techsnuffle.com/2018/10/16/talespire-dev-daily-13<p>Another short update for today. I got the fog of war logic mostly hooked up but in a very dissatisfying ugly way. The job for tomorrow is to try and wrangle it into a more sensible form and then start testing and tweaking the fill.</p>
<p>The tiles are being revealed now but as we are replacing some parts of the projects others are pretty broken. An example side effect is that right now you can walk through hidden walls. This ends up being rather demoralizing as the harder you work the more broken things get. Of course the logic part of the brain knows we have done this shit plenty of times before and in a few days it’ll be good again, just gotta keep grinding :)</p>
<p>Until next time</p>
<p>Ciao</p>
TaleSpire Dev Daily 122018-10-15T22:19:23+00:00http://www.techsnuffle.com/2018/10/15/talespire-dev-daily-12<p>Today has been spent replacing the old fog of war with the new code. This code has paths from and through many other systems so it feels like I’ve been pulling on threads to see what moves and then unweaving them.. flashbacks to running network cables in an old job :p</p>
<p>Like yesterday I don’t have solid stuff to show but it feels like it’s getting close. Tomorrow will start with testing followed by hooking the level loading into the fog of war stuff. This either means having each loaded asset write add themselves to the zone’s octree or (much more likely) we just serialize the octrees.</p>
<p>I also need to look into which zones are updated per frame. For a huge map you don’t want some zone a mile away updating things that don’t matter.</p>
<p>Thanks for stopping by, seeya next time</p>
TaleSpire Dev Daily 112018-10-14T22:17:38+00:00http://www.techsnuffle.com/2018/10/14/talespire-dev-daily-11<p>Today I needed to take a little time to be a human so less coding was done.</p>
<p>Before that I did get flood fill working across subsections of the board (zones). The next task is storing the results for the given character and being able to switch between different character’s perception.</p>
<p>Those successes only appear as a couple of messages in the log right now, so instead here is a visualization of the octrees of different zones as the character moves through them</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/D_JEwPD-PBg" frameborder="0"></iframe>
<p>Peace</p>
TaleSpire Dev Daily 102018-10-13T23:04:56+00:00http://www.techsnuffle.com/2018/10/13/talespire-dev-daily-10<p>I got less done than I’d hoped today but that’s alright. We had to plan out some finer details of how the fog of war should behave under different conditions and required a few changes in how we are storing places the character has already revealed.</p>
<p>I’ve hacked out the data structure for this and am writing on the worker that will run the flood fills.</p>
<p>Not really much more to say right now,</p>
<p>ciao</p>
TaleSpire Dev Daily 92018-10-12T20:27:07+00:00http://www.techsnuffle.com/2018/10/12/talespire-dev-daily-9<p>I like these days. These days I get to give good news. And today’s news is that the constrained flood fill works!</p>
<p>Let’s start with the pictures. In the next picture the white areas are all the fog of war we want to reveal if the character was at the red crosses. We can see that it hasn’t flowed into the blocked off room (even though the search does work vertically) as we have told it not to search above the eyeline of the character.</p>
<p><img src="/assets/images/floodFill/simple.png" alt="simple fill" /></p>
<p>We can see this more clearly for this scene</p>
<p><img src="/assets/images/floodFill/setup.png" alt="ledge setup" /></p>
<p>If we search from the green cross we get</p>
<p><img src="/assets/images/floodFill/fromFloor.png" alt="from green" /></p>
<p>And from the red cross we get</p>
<p><img src="/assets/images/floodFill/fromLedge.png" alt="from red" /></p>
<p>This satisfies the requirements we had so hooray!</p>
<p>My (albeit brief) googlings didn’t give me a guide to how to do flood fill on an octree so here is my approach. We are going to do it on quadtrees as that makes the diagrams simpler, but the extensions to 3d in this case are trivial.</p>
<p>We start with our quadtree. The green colored areas are solid and the white are empty. The red cross is the starting point</p>
<p><img src="/assets/images/floodFill/step0.png" alt="step0" /></p>
<p>Our first step is to walk down to the deepest node containing our point. We will call the recursive function that does this <code class="language-plaintext highlighter-rouge">WalkToPoint</code>. We take with us:</p>
<ul>
<li>a list called <code class="language-plaintext highlighter-rouge">faces</code>. We will explain faces soon but just know we can add things to this list and they will be processed later.</li>
<li>a list called <code class="language-plaintext highlighter-rouge">spaces</code>. We add the empty nodes we reached to this list</li>
</ul>
<p>When we get to the deepest node containing the point we check if it’s solid. If it is then game over the flood goes nowhere. If not then we add this node to the <code class="language-plaintext highlighter-rouge">spaces</code> list and then:</p>
<ul>
<li>starting at the center of the node pick 4 positions, one up, down, left and right from the center. The distance to the point must be the width of this node.</li>
<li>add a <code class="language-plaintext highlighter-rouge">Face</code> to the <code class="language-plaintext highlighter-rouge">faces</code> list. The face contains the position, the depth of the current node (3 in our case) and the direction from that new position to this node.</li>
</ul>
<p><img src="/assets/images/floodFill/step1.png" alt="step1" /></p>
<p>So for the position we picked to the right we make a face this this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
Position: Vector2(),
Depth: 3,
From: Left
}
</code></pre></div></div>
<p>We add all 4 new <code class="language-plaintext highlighter-rouge">faces</code> to the <code class="language-plaintext highlighter-rouge">faces</code> list and return;</p>
<p>When we get back to the function that started the first step we go into a loop doing the following:</p>
<ul>
<li>take a <code class="language-plaintext highlighter-rouge">Face</code> out of the <code class="language-plaintext highlighter-rouge">faces</code> list</li>
<li>call <code class="language-plaintext highlighter-rouge">WalkToFace</code> on that <code class="language-plaintext highlighter-rouge">Face</code></li>
<li>keep doing this until the <code class="language-plaintext highlighter-rouge">faces</code> list is empty.</li>
<li>the <code class="language-plaintext highlighter-rouge">spaces</code> list now holds all the reachable empty nodes</li>
</ul>
<p>Clearly <code class="language-plaintext highlighter-rouge">WalkToFace</code> is doing the bulk of the work so lets look at how it behaves for 3 of the 4 faces we enqueued in the first function. We are only looking at 3 as they cover this different behaviours our code has to handle</p>
<p><img src="/assets/images/floodFill/step2.png" alt="step2" />
<img src="/assets/images/floodFill/step3.png" alt="step3" /></p>
<p>For the position on the right we start by walking towards the <code class="language-plaintext highlighter-rouge">Position</code> in the <code class="language-plaintext highlighter-rouge">Face</code> just like we did for the first point, however we are keeping track of what depth we are at and as soon as our depth matches (or exceeds) the depth in the <code class="language-plaintext highlighter-rouge">Face</code> we change our behaviour. In this case we have reached the deepest node already so we do the same as we did for the point, namely enqueue the 4 <code class="language-plaintext highlighter-rouge">Face</code>s, add the node to the <code class="language-plaintext highlighter-rouge">spaces</code> list and return;</p>
<p><img src="/assets/images/floodFill/step4.png" alt="step3" /></p>
<p>The point above the first is a little different as before we hit the depth in the <code class="language-plaintext highlighter-rouge">Face</code> we reach the deepest node. No worries, just enqueue the 4 faces like the last 2 times and return.
Also note that in this case we have points outside the quadtree, what you do with those is up to you. In TaleSpire we have divided the board up into zones and each has it’s own octree so we pass those external points off the respective zone for processing.</p>
<p><img src="/assets/images/floodFill/step5.png" alt="step4" /></p>
<p>In this last case we walk to the depth specified in the <code class="language-plaintext highlighter-rouge">Face</code>, now we are here we still have more child nodes. This is where the <code class="language-plaintext highlighter-rouge">From</code> information in the <code class="language-plaintext highlighter-rouge">Face</code> is important. Remember that we said that once our depth >= the <code class="language-plaintext highlighter-rouge">Face</code>’s depth we change the behaviour. Now we stop picking nodes by the <code class="language-plaintext highlighter-rouge">Position</code> in <code class="language-plaintext highlighter-rouge">Face</code> and instead recurse into each of the child nodes on the side stated in the <code class="language-plaintext highlighter-rouge">From</code> field in <code class="language-plaintext highlighter-rouge">Face</code>. Doing this gets us to the 2 nodes marked with blue lines in the pic above.</p>
<p><img src="/assets/images/floodFill/step6.png" alt="step5" /></p>
<p>If that isn’t clear here is a different example. Here we are coming from the right so all the rightmost nodes (marked in red) are reached.</p>
<p><img src="/assets/images/floodFill/extra.png" alt="step6" /></p>
<p>Anyhoo if we keep following the above steps we will flood to all the nodes we can reach from the initial point. Note the you mustn’t process any node that has already been added to <code class="language-plaintext highlighter-rouge">spaces</code>, but other than that we are good.</p>
<p><img src="/assets/images/floodFill/done.png" alt="done" /></p>
<p>Doing this in 3D simply means handling 2 more directions.</p>
<p>There are lots of little places the speed this up, for example:</p>
<ul>
<li>track when all child nodes have been added to <code class="language-plaintext highlighter-rouge">spaces</code> or are solid. Then mark the parent node as processed and never process again.</li>
<li>you don’t need to return all the way to the initiating function to process <code class="language-plaintext highlighter-rouge">WalkToFace</code> calls. Keep track of the position in the <code class="language-plaintext highlighter-rouge">spaces</code> when you started processing a given node. Some of entries in the list after that point will be the children trying to reach other, neighboring, child nodes.</li>
<li>Calculate the max number of <code class="language-plaintext highlighter-rouge">spaces</code> that could we enqueued in a single walk and preallocate the <code class="language-plaintext highlighter-rouge">spaces</code> list so that it doesn’t have to grow when things are added. Less frequent allocations are almost always a plus.</li>
</ul>
<p>But that’s enough for now. Saturday I’m going to get cross zone flooding working and Sunday will be spent hooking this up to actually show and hide objects. I also need to rewrite how I capture the collision info and handle deletions. I really wanted to get this done this week so the next week can be dedicated to refactoring how characters get placed and handle computing max walk distances, but I can tell already this will need a couple more solid days of work. Ah well.</p>
<p>Peace folks</p>
TaleSpire Dev Daily 82018-10-11T23:03:12+00:00http://www.techsnuffle.com/2018/10/11/talespire-dev-daily-8<p>Todays daily is short.</p>
<p>FAILURE!</p>
<p>So far today I have failed to make a fill implementation I’m happy with. I know the exact approach I’m going for an have made some progress on the steps after extracting the reachable region, but the damn fill has been eluding me.</p>
<p>In the last few minutes I’ve started another way which, whilst will require a bit more bouncing around the tree, is a much simpler implementation.</p>
<p>Hopefully tomorrow it will be working and I can yak about that. Until then it’s just a day with a bunch of failures. No worries of course, failures small and large are pretty much a constant when coding new things so its just a sign of doing things :)</p>
<p>Goodnight</p>
TaleSpire Dev Daily 72018-10-10T23:04:50+00:00http://www.techsnuffle.com/2018/10/10/talespire-dev-daily-7<p>Here we are again. This time I don’t have much to show as I spent most of the day at the whiteboard. So instead I’ll talk about the problem I’ve been mulling over.</p>
<p>When a character piece (henceforth just called a ‘character’) is placed on a tile we want to show every tile it could possibly reach, regardless of distance. We dont want to show things behind walls or locked doors. Also we need to do this quickly.</p>
<p>Actually let’s pause to clarify something. It is going to be tempting to redefine the problem given the issues that we are about to look at. You may not even like the idea of the behavior and want to change it for that reason. Please, for now, trust that we have tested this in game and it feels nice, so for now this is the problem we are trying to solve.</p>
<p>Alright, back to the snooker..</p>
<p>Before the game had floors it was effectively 2d, the approach then was to <a href="https://en.wikipedia.org/wiki/Flood_fill">flood fill</a>, performing raycast to see if there was an occluder (a wall) that would stop the progression. It might be rather wasteful but it worked well enough for a demo and you can get away with a lot of raycasts per frame when your game is as simple (in terms of how much stuff is going on) as ours.</p>
<p>However now we add floors and everything changes, now we have a third dimension to reckon with and everything gets much more expensive. One other interesting detail is although we have separate floors we also want the users to be able to make features that reach through to other floors. For example imagine a grand hall with a obsidian pyramid in the center, we want to show the whole pyramid from the ground floor even though it is several floors high as this gives the moment gravitas that would be lost if it you only saw a tenth of it.</p>
<p>One basic thing we did in the 2d version and will still do in the 3d version is divide the world up into zones. A zone is a 3d region of a certain number of tiles in the X and Z dimensions and some height (lets say 10 floors worth) from a given floor.</p>
<p>With this known we specify the subproblem as: Find every reachable place in the zone and which zones we can access from this zone.</p>
<p>Also it’s worth noting that the player will be moving the character frequently so the result should be cachable or very cheap to recompute.</p>
<p>In my mind we need to know what is solid and what is empty space. Let’s say the zone was 81x81 tiles in size and just as high as it is wide. The tiles are 1 Unity unit in size so that’s 81x81x81 units, assuming we only need to know if a unit cube is solid or not then a dumb solution would require us to store 531441 booleans (true or false) so say whether each part of the zone is solid or empty. Clearly for large boards this is untenable.</p>
<p>One common and reliable way to store 3d spatial data more efficiently is using an <a href="https://en.wikipedia.org/wiki/Octree">octree</a>. It recursively subdivides space into 8 equal subspaces down to the required resolution. So (for a 2d example), rather than having the 2d array on the left with 36 slots we have the approach on the right with 10 slots as we don’t subdivide if there is no new information inside.</p>
<p><img src="/assets/images/d7_0.jpg" alt="quadtree" /></p>
<blockquote>
<p>yup it’s not a totally accurate drawing but I hope ya get the idea</p>
</blockquote>
<p>Each node in the quadtree has, either a marker saying it’s solid/empty or has 4 child nodes.</p>
<p><img src="/assets/images/d7_1.jpg" alt="quadtrees are trees" /></p>
<p>For octrees it’s the same deal but with 8 nodes and 3 dimensions.</p>
<p>So, now we have our data-structure we can fill it with information about what is solid and what isn’t. Then we could so something rather similar to the old flood fill, only this time we start from the character position, find what region in the octree they are in and walk to the neighboring regions.</p>
<p>Most of the time we move somewhere it’s going to be one of the places we have already searched so we can cache the result of the previous walk. We can also cache the octree itself.</p>
<p>So far so good, but there are issues:</p>
<p><img src="/assets/images/d7_2.jpg" alt="woops" /></p>
<p>a. Our 3d search can search over walls we need to limit that but that will limit their
b. Not having to search to work out what parts of the quadtree/octree are accessible would be cool. They have that built in recursive structure so can we leverage that?
c. Tiles are place on the floor itself or on other tiles, can we propagate information between them on creation that could help us? could we serialize it when saving the level so we dont have to recompute on load?</p>
<p><code class="language-plaintext highlighter-rouge">b</code> gets rather tantalizing. If we could asign each node in the tree an integer id, we could then take the <code class="language-plaintext highlighter-rouge">min</code> of the neighbours and it would resolve to all nodes having the same id if they are reachable. The resolve takes time though, could we find a way to do this in 1 (or some fixed low number) of passes? Is it worth it?</p>
<p><img src="/assets/images/d7_3.jpg" alt="auto-tag" /></p>
<p>Unrelated but important; if you want to be fast then <a href="https://en.wikipedia.org/wiki/Locality_of_reference">cache locality</a> matters. Could we store the data in a way that helps our intended access patterns (searching)? One example could be using a <a href="">Z-order curve</a> to linearize the data and using something like a skiplist to keep the data sparse.. again would this actually help? It all depends on our data and how we access it.</p>
<p>I’ve been asking a lot of questions as it frames the issue nicely, but I do have some WIP answers but explaining them will balloon this post even more so for now I’ll just say I hope to have something I can show by the end of Sunday. I’d also like to stress that these problems arent novel, there is a lot of literature and most games will be dealing with much more interesting cases than this.. still it’s fun to think about.</p>
<p>Until tomorrow,
Ciao</p>
TaleSpire Dev Daily 62018-10-09T18:55:50+00:00http://www.techsnuffle.com/2018/10/09/talespire-dev-daily-6<p>It was a slow day today. I carried on working on the fog-of-war system which is what works out where in a board a given character could get to. It’s being rewritten as the version we have demoed so far did not handle multiple floors.</p>
<p>The behavior needed is that, when a character is placed, everywhere that is accessible in the board should be visible. To do this we are splitting up the (occupied parts of the) board into zones and in each zone we have a structure that describes where is solid. This makes it cheaper to search the zone when ascertaining what you can see, as we don’t need to hammer Unity’s collision system all the time.</p>
<p>This fiddly thing is we want it to be fast and so I’m trying to balance cache locality of data with wanting a pretty sparse data-structure. I had wanted to make use of Unity’s new job system and there you need decent size chunks of ‘flat’ data. For that reason I had wanted to avoid octrees and look at simpler bucketing instead. My current approach is pretty crappy but I really need to get <em>something</em> working so I can start measuring.</p>
<p>The good part is that I added the Zones class and got the low res collision info from yesterday written into the Zone. Tomorrow I will try and not think about how bad performance will be and just write the search.. or if I cant then I will take the search back to the whiteboard :)</p>
<p>Until then, seeya</p>
TaleSpire Dev Daily 52018-10-08T20:10:41+00:00http://www.techsnuffle.com/2018/10/08/talespire-dev-daily-5<p>Hey all,</p>
<p>The planning weekend went really well. We whiteboarded out the new visibility system and the code that handles where is walkable. The will no doubt be changes to the design but we can see how that goes this week. We also ran through the core user journey and planned out the UI, data requires and systems that still need making. Whilst I dont have specifics for you right now I am even more confident that you folks will have some form of early access in your hands this year.</p>
<p>I started the steam login integration. Thanks to <a href="https://steamworks.github.io/">steamworks.net</a> the unity side was easy and the server side was just basically a get request so no issues there.</p>
<p>I do want to clarify that Steam will not be the only way to log in. We don’t want to force people into a specific platform so we will have a few options when the release comes out. It may be what we use for the alpha though, we shall see.</p>
<p>Right, I’m tired so I’m going to stop coding now and chill.</p>
<p>Peace</p>
<p>p.s On the left is a test of some generated lower res information to be used in the visibility system.</p>
<p><img src="/assets/images/vis.png" alt="stuff and things" /></p>
TaleSpire Dev Daily 42018-10-05T21:36:54+00:00http://www.techsnuffle.com/2018/10/05/talespire-dev-daily-4<p>Today I put aside the undo/redo work and focused on fixing little bugs instead. As I’m off to <a href="https://twitter.com/jonnyree?lang=en">@jonnyree’s</a> place this weekend I really wanted to get the recent stuff as stable as possible so we can merge it to master before I dive into another chunk of the project.</p>
<p>That mainly involved fixing some bugs around network IDs, a dumb mistake in serialization, and a bunch of little cleanups.</p>
<p>Hmm, there’s not really much else to say. Ah well, there are plenty of days like this.</p>
<p>Seeya folks, back on Monday with the new plans</p>
<p>Peace</p>
TaleSpire Dev Daily 32018-10-04T17:31:45+00:00http://www.techsnuffle.com/2018/10/04/talespire-dev-daily-3<p>Allo again,</p>
<p>Today wasn’t the most satisfying. I wanted to prototype the undo/redo scheme I had doodled out but some of how floors was implemented was making it difficult. Even though we are redesigning the floor system this weekend it was more work to work around the issues than just tweak them so, after a couple of hours reaching that realization, I spent a good chunk of the day <a href="https://en.wiktionary.org/wiki/yak_shaving">yak shaving</a>.</p>
<p>With that somewhat out of the way I had a go at implementing the scheme. I already has working local undo/redo, so the task was making it work with multiple people editing simultaneously.</p>
<p>One example issue (which I mentioned yesterday) can be summarized like this:</p>
<blockquote>
<p>There are two GMs, <code class="language-plaintext highlighter-rouge">A</code> & <code class="language-plaintext highlighter-rouge">B</code> and each have their own undo/redo history</p>
<ul>
<li>GM <code class="language-plaintext highlighter-rouge">A</code> places a floor-tile</li>
<li>GM <code class="language-plaintext highlighter-rouge">A</code> places a crate</li>
<li>GM <code class="language-plaintext highlighter-rouge">B</code> deletes the crate</li>
<li>GM <code class="language-plaintext highlighter-rouge">A</code> presses undo</li>
</ul>
</blockquote>
<p>One possibility is then when <code class="language-plaintext highlighter-rouge">B</code> deletes the crate, the history event for placing the crate in <code class="language-plaintext highlighter-rouge">A</code>s history could be removed. This way <code class="language-plaintext highlighter-rouge">A</code> can’t attempt to undo or redo something that has gone.</p>
<p>I’m a bit adverse to deleting stuff from the history however as you lose data. Instead I wanted to try putting a flag on the history event that inhibits it. When <code class="language-plaintext highlighter-rouge">A</code> hits ‘undo’ it will skip over the inhibited event and do the one before it. The nice side effect is that if <code class="language-plaintext highlighter-rouge">B</code> was to undo the delete, we can uninhibit the history event in <code class="language-plaintext highlighter-rouge">A</code>s history again, allowing undo and redo to proceed as before.</p>
<p>The idea has edge cases though and, whilst I have seen a few of them, I’m not going to bog this post down with them yet. I’ll do some more experimentation and then report back to you all :).</p>
<p>As for tomorrow I think I’m going to leave this on a branch and get back to a few other bugs that are more likely to be an issue when working this weekend. There is one rather nasty one regarding level loading an reuse of supposedly unique ids :)</p>
<p>Until then,</p>
<p>Peace</p>
TaleSpire Dev Daily 22018-10-03T17:04:39+00:00http://www.techsnuffle.com/2018/10/03/talespire-dev-daily-2<p>Today I refactored how we spawn and sync multiple tiles of the same kind which is used when dragging out tiles when in build mode. The new implementation uses less memory to synchronize but more importantly has a single entry in the undo/redo history so you can undo a slab of tiles in one go. In the process of doing this I also simplified one way of sending messages which means there is less boilerplate code to write (less code less bugs).</p>
<p>The last third of the day was mostly spent doodling on the whiteboard to work out a nice way of handling how edits made by multi game masters can cause conflicts in the undo/redo history.</p>
<p>A simple is case is as follows.</p>
<p>There are two GMs, <code class="language-plaintext highlighter-rouge">A</code> & <code class="language-plaintext highlighter-rouge">B</code> and each have their own undo/redo history[0]</p>
<ul>
<li>GM <code class="language-plaintext highlighter-rouge">A</code> places a floor-tile</li>
<li>GM <code class="language-plaintext highlighter-rouge">A</code> places a crate</li>
<li>GM <code class="language-plaintext highlighter-rouge">B</code> deletes the crate</li>
<li>GM <code class="language-plaintext highlighter-rouge">A</code> presses undo</li>
</ul>
<p>What should happen?</p>
<p>If <code class="language-plaintext highlighter-rouge">B</code> hadn’t delete the crate then <code class="language-plaintext highlighter-rouge">A</code>’s undo would have removed the crate. But the crate is already gone.</p>
<p>If we do nothing but move back in the history then <code class="language-plaintext highlighter-rouge">A</code> will be confused. A successful input without an output makes the user feel like either they did something wrong or the system is broken.</p>
<p>It feels like the sane thing is that if nothing can be done we skip to the previous entry. But then what do we do with redos?</p>
<p>Also if GM <code class="language-plaintext highlighter-rouge">A</code> dragged out 20 tiles and <code class="language-plaintext highlighter-rouge">B</code> deletes just one, what is the correct behavior. I think I’d expect the rest of the tiles to be removed. So then does redo restore them all?</p>
<p>I think I have a solution for this but I only got it partially finished today. I’ll get back into this tomorrow and see how it feels.</p>
<p>If that doesn’t feel nice then I am considering briefly showing a ‘ghost’ of the deleted object so you at least get the indication that <em>something</em> happened.</p>
<p>Right, time to get some food and get ready for the lisp stream in a few hours.</p>
<p>Ciao.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/uBbsZV9aTeo" frameborder="0"></iframe>
<p>[0] Separate undo/redo is pretty important to avoid people undoing each other’s work, this rapidly escalates to bloodshed.</p>
<hr />
<blockquote>
<p>Boring Caveats</p>
<p>These are just my daily notes as I work on TaleSpire. It’s so cool that people are interested in the game and it’s way more fun to share this stuff that just keep it away, however nothing said here should be taken as any kind of promise that a thing will exist in a given release or work in the way stated. Stuff changes constantly and I’m wrong about at least 1000 things a day so whatever I’m stoked about today may well be tomorrows nightmare.</p>
<p>So yeah, that’s that. Back to the code :)</p>
</blockquote>
TaleSpire Dev Daily 12018-10-02T17:02:29+00:00http://www.techsnuffle.com/2018/10/02/talespire-dev-daily-1<p>Alright, another day down.</p>
<p>Today was primarily spent adding the undo/redo system which is looking pretty good now. A few bugs to iron out but nothing that looks terrifying.</p>
<p>As you’d probably imagine it’s a simple list of history actions and an index of where in the history you currently are. Each event has <code class="language-plaintext highlighter-rouge">Undo</code> and <code class="language-plaintext highlighter-rouge">Redo</code> methods and some serialized state for the thing that was being added/removed/etc. This a dirt simple approach and can get pretty unwieldy for systems with many more kinds of actions but for our little thing it’s just fine.</p>
<p>During making this I noticed that the board synchronization code was being too conservative about when it could send the board so now it gets sent much sooner after loading from disk. The basic run down goes something like this:</p>
<p>Tiles load their assets (visuals and scripts) asynchronously so it can be a relatively long time (a few frames) after the tile is created before the Lua scripts inside are running. Because of this, when deserializing we cache the state intended for this asset until the Lua scripts are fully set up.</p>
<p>When loading levels we obviously want to send the full state to the other players so we waited for all the assets to be full loaded before sending the level to other players. This was pointless as a give tiles can only be in 3 states regarding initialization.</p>
<ul>
<li>it’s fully set up and running</li>
<li>it’s been initialized, has some cached state, and is still loading</li>
<li>it’s been initialized, has no cached state, and is still loading (like when you first place it)</li>
</ul>
<p>In all of these cases we can serialize as, even if it hasn’t finished loading, we have the pending state or we know it’s going to have default state.</p>
<p>We also could just sync the level data from the local file and append the additional setup data (character network ids etc), and we may go that way, however for now the runtime cost of load-then-sync is so low currently that it’s not worth it.</p>
<p>Tomorrow I will be refactoring dragging out tiles. Simple stuff but it needs some attention so it plays nicer with sync and undo.</p>
<p>Seeya tomorrow!</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/S7jK9wxwq9A" frameborder="0"></iframe>
<hr />
<blockquote>
<p>Boring Caveats</p>
<p>These are just my daily notes as I work on TaleSpire. It’s so cool that people are interested in the game and it’s way more fun to share this stuff that just keep it away, however nothing said here should be taken as any kind of promise that a thing will exist in a given release or work in the way stated. Stuff changes constantly and I’m wrong about at least 1000 things a day so whatever I’m stoked about today may well be tomorrows nightmare.</p>
<p>So yeah, that’s that. Back to the code :)</p>
</blockquote>
TaleSpire Dev Thang 02018-10-01T22:16:53+00:00http://www.techsnuffle.com/2018/10/01/talespire-dev-thang-0<p>Evening all! I really need to start doing my daily signoffs somewhere people can see.. so here we are!</p>
<p>Most of my work these days is rewriting systems from the prototype you may have seen on the streams. As you can imagine there are lots of things that were hacked in to see how they would feel but were never fleshed out. Today I was working on the sync of state from scriptable tiles. Currently only the state-machine script we are using for the doors and chests is using this change but it’s now easy for us to expand on.</p>
<p>I also finally let my partner try out the building system (see it’s not just you folks who haven’t been let in :D) and have a big ol’ list of things to change. Many are things we knew about but it’s always good to get fresh eyes on it.</p>
<p>This weekend I’m off to <a href="https://twitter.com/jonnyree?lang=en">@jonnyree’s</a> place so we can plan out the rewrite of the floor and fog-of-war systems so expect news on that next week.</p>
<p>Tomorrow I think I’m going to look at basic undo-redo and see what feels nice when you have multiple people editting the board at the same time.</p>
<p>That’s it for now.</p>
<p>Peace</p>
<p>p.s</p>
<p><img src="/assets/images/ghosthouse.png" alt="ghost house!" /></p>
Notes to Self Part N: SSL Checker2018-09-17T12:57:24+00:00http://www.techsnuffle.com/2018/09/17/notes-to-self-part-n-ssl-checker<p>I will get back to blogging at some point, I kinda miss it.</p>
<p>However today I just need to note that this guy is super handy for checking that a site’s certificates are set up properly.</p>
<p>https://www.sslshopper.com/ssl-checker.html</p>
<p>At least it’s super useful for a noob like me.</p>
Windows emacs setup2018-07-05T12:40:04+00:00http://www.techsnuffle.com/2018/07/05/windows-emacs-setup<p>My mate was looking at using emacs on windows and so I wrote a little ‘how to’. I then added my own opinions of stuff to start with so it got bigger. Made sense to just dump it here:</p>
<h2 id="note">Note</h2>
<p>I’m probably out of date! I just saw you can install emacs via pacman in msys which might handle a bunch of the ‘path’ & environment variable shit for you. I’m going to look into this and then update this guide.</p>
<h3 id="install">Install</h3>
<ul>
<li>download this chap <a href="http://ftp.gnu.org/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip">http://ftp.gnu.org/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip</a></li>
<li>There is no install process so just extract it to c:\ so you have a folder like c:\emacs-25-1</li>
<li>open the c:\emacs-<em>*-</em>\bin folder</li>
<li>copy the path to that directory</li>
<li>add it to the PATH environment variable</li>
<li>make a folder called ‘home’ folder in c:\ (or if you have one for msys skip this step). I do this as twice I’ve had windows shit up after updates and tell me my account isnt mine anymore and I hate fighting that shit</li>
<li>Add a environemnt varable called HOME and set it to your new (or mysys) ‘home’ directory</li>
<li>Basic setup is done but there are some things you will want.</li>
</ul>
<h3 id="package-manager">Package manager</h3>
<p>Dont download packages yourself, keeping them current is a pain, emacs comes with a package manager but the official package source is a little behind the times so we will add one that is much more used by the community ‘melpa’</p>
<p>See this video for how to use and install it: <a href="https://www.youtube.com/watch?v=Cf6tRBPbWKs">https://www.youtube.com/watch?v=Cf6tRBPbWKs</a></p>
<p>Short version is:</p>
<ul>
<li>hit ‘M-x’ [0] which will open the minibuffer at the bottom of emacs, type ‘customize’ and hit return.</li>
<li>Type ‘package’ in the search bar and hit return.</li>
<li>Scoot your cursor down to the arrow next to ‘package archives’ and hit return to open that subtree</li>
<li>hit return on the ‘INS’ button and add ‘melpa’ as the archive name and ‘http://melpa.org/packages/’ as the url</li>
<li>‘C-x C-s’ to save (which means) hold down control and press ‘x’ then ‘s’</li>
</ul>
<p>This will have edited your .emacs file in your ‘home’ directory, this is nice as lots of packages use the customize system and its often friendlier than searching docs for the right thing to edit.</p>
<h3 id="magit-reticently-optional">Magit (reticently optional)</h3>
<ul>
<li>restart emacs (rarely neccessary but I want to be sure evertyhing is fresh) and hit ‘M-x’ and type ‘list-packages’ and hit return</li>
<li>give it a second to pull the package list and then install ‘magit’ (see that video for a guide of how to do that)</li>
<li>This is the nicest damn git client, I’d install emacs for this even if I wasnt using it as a text editor.</li>
</ul>
<h3 id="biased-baggers-emacs-file-additions-optional">Biased Baggers .emacs file additions (optional)</h3>
<p>Open your .emacs file and paste the following at the top of the file</p>
<p>It looks like a lot just its just shit I’ve slowly accrued whilst using emacs.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; so package manager is always good to go
(package-initialize)
;; utf8 is a good default
(setenv "LANG" "en_US.UTF-8")
;; when you start looking for files its nice to start in home
(setq default-directory "~/")
;; Using msys?
;; to make sure we can use git and stuff like that from emacs
(setenv "PATH"
(concat
;; Change this with your path to MSYS bin directory
"C:\\msys64\\usr\\bin;"
(getenv "PATH")))
;; Fuck that bell
(setq ring-bell-function #'ignore)
;; Turn off the menu bars, embrace the keys :p
(tool-bar-mode -1)
(menu-bar-mode -1)
;; Dont need emacs welcome in your face at every start
(setq inhibit-splash-screen t)
;; typing out 'yes' and 'no' sucks, use 'y' and 'n'
(fset `yes-or-no-p `y-or-n-p)
;; Highlight matching paren
(show-paren-mode t)
;; This might be out of date now, need to ask kristian
(setq column-number-mode t)
;; You can now used meta+arrow-keys to move between split windows
(windmove-default-keybindings)
;; Kill that bloody insert key
(global-set-key [insert] 'ignore)
;; Stop shift mouse click opening the font window
(global-set-key [(shift down-mouse-1)] 'ignore)
(global-set-key [(control down-mouse-1)] 'ignore)
;; C-c C-g now runs 'git status' in magit. Super handy
(global-set-key (kbd "\C-c \C-g") `magit-status)
;; Jump to matching paren. A touch hacky but I nabbed it from somewhere and has worked well enough for my stuff
;; probably something better out there though (this is language independent though).
;; Move to one bracket and hit 'Control )' to jump to the other bracket
(defun goto-match-paren (arg)
"Go to the matching if on (){}[], similar to vi style of %"
(interactive "p")
;; first, check for "outside of bracket" positions expected by forward-sexp, etc.
(cond ((looking-at "[\[\(\{]") (forward-sexp))
((looking-back "[\]\)\}]" 1) (backward-sexp))
;; now, try to succeed from inside of a bracket
((looking-at "[\]\)\}]") (forward-char) (backward-sexp))
((looking-back "[\[\(\{]" 1) (backward-char) (forward-sexp))
(t nil)))
(global-set-key (kbd "C-c )") `goto-match-paren)
</code></pre></div></div>
<h3 id="ssh-agent-hack-only-needed-if-you-have-the-issue-i-did">ssh-agent hack (only needed if you have the issue I did)</h3>
<p>Windows is a pita with some of this stuff. I have emacs on machine using the bash provided with git rather than a dedicate msys or whatever install and I got confused some ssh-agent issues. So I just made a windows shortcut that runs emacs from git bash and that helped. I get propted for my ssh-agent password on launch (which I do once a day) and then im free to work.</p>
<p>The shortcut just pointed to: <code class="language-plaintext highlighter-rouge">"C:\Program Files\Git\git-bash.exe" -c "emacs-25.2"</code></p>
<h3 id="control-key-optional">Control Key (optional)</h3>
<p>The control key is in the wrong place and it’s not good for the hand to keep having to reach for it. Let’s make capslock an extra control key.</p>
<p>make a file called <code class="language-plaintext highlighter-rouge">control.reg</code> and paste this in it</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,1d,00,3a,00,00,00,00,00
</code></pre></div></div>
<p>Save it, run it and restart windows for it to take effect.</p>
<h3 id="done-optional-p">Done (optional :p)</h3>
<p>Save and restart emacs again. Hopefully there are no errors (bug me if there are)</p>
<p>Remember that emacs is your editor, nothing is too stupid if it makes your experience better. For example I kept mistyping certain key combos so I bound the things I kept hitting instead to the same functions, it’s small but it makes me faster.</p>
<p>Thats all for now, seeya!</p>
<p>[0] (M stands for meta and is the alt key [the naming comes from the old <a href="https://pbs.twimg.com/media/C942BKSU0AE7p0q.jpg">lisp machine keyboards iirc</a>)</p>
Cleaning up a photo of a whiteboard2018-04-23T15:20:47+00:00http://www.techsnuffle.com/2018/04/23/cleaning-up-a-photo-of-a-whiteboard<p>I have small bad handwriting so some of the automatic methods out there really didnt work for me.</p>
<p>This is an excellent start:</p>
<p>https://gist.github.com/adsr303/74b044534914cc9b3345920e8b8234f4</p>
<p>In case that gist is gone:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Duplicate the base image layer.
Set the new layer's mode to "Divide".
Apply a Gaussian blur with a radius of about 50 (Filters/Blur/Gaussian Blur...)
</code></pre></div></div>
<blockquote>
<p>Note: Alternatively just perform a ‘difference of gaussians’ on the image setting the first to radius 70 and the second to radius 1</p>
</blockquote>
<p>I used a Gaussian blue with radius 70 in both directions. <code class="language-plaintext highlighter-rouge">Mode</code> is a dropdown near the top of the layers panel.</p>
<p>Next pick a background color and paint over the world offending background features; stuff like magnets and erasers.</p>
<p>Then flood fill with that color to normalize as much as possible.</p>
<p>Then flood fill with white. This should be pretty effective now.</p>
<p>Open up the hue/saturation (some name like that) tool up the saturation as the colors will be a bit washed out.</p>
<p>Finally open up brightness & contrast and drop the brightness a tiny bit.</p>
<p>That’s it. Much nicer to stare at and maintains all the features.</p>
<p><img src="assets/images/wboard.png" alt="dif" /></p>
slime-enable-concurrent-hints2018-04-17T19:20:10+00:00http://www.techsnuffle.com/2018/04/17/slimeenableconcurrenthints<p>This is just me dumping something I keep forgetting. When you are live coding you often block the repl, you can call swanks <code class="language-plaintext highlighter-rouge">handle-requests</code> function to keep the repl which works but the minibuffer likely wont be showing function signatures (or other hints) any more. To enable this add this to your <code class="language-plaintext highlighter-rouge">.emacs</code> file and call using <code class="language-plaintext highlighter-rouge">M-x slime-enable-concurrent-hints</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun slime-enable-concurrent-hints ()
(interactive)
(setf slime-inhibit-pipelining nil))
</code></pre></div></div>
Fuck yeah progress!2018-04-03T22:28:14+00:00http://www.techsnuffle.com/2018/04/03/fuck-yeah-progress<p>I’ve been on holiday for a week and it’s been great for productivity; the march to get stuff out my head before my new job goes rather well.</p>
<h2 id="more-docs">More docs</h2>
<p>First stop <a href="https://github.com/cbaggers/rtg-math">rtg-math</a>. This is my math library which for the longest time has not been documented I fixed that this week so now we have a <a href="http://techsnuffle.com/rtg-math/rtg-math-reference.html">bootload of reference docs</a>. Once again <a href="https://shinmera.github.io/staple/">staple</a> is hackable enough that I could get it to produce signatures with all the type info too. Blech work but I can not think about that for a while. One note is that the ‘region’ portion of the api is undocumented as that part is still WIP.</p>
<h2 id="sdfs">SDFs</h2>
<p>Next I wanted to get some signed distance functions into <a href="https://github.com/cbaggers/nineveh">Nineveh</a> (which is my library of useful gpu functions). SDFs are just a cool way to draw simple things and so with the help of code from shadertoy (props to iq, thoizer & Maarten to name a few) I made a nice little api for working with these functions.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/MPKPAzRx1ZM" frameborder="0"></iframe>
<p>One lovely thing was in the function that handles the shadows, it needs to sample the distance function at various points and rather than hardcoding it we are able to pass it in as a function. As a result we get things like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g point-light ((fn (function (:vec2) :float))
(p :vec2)
(light-position :vec2)
(light-color :vec4)
(light-range :float)
(source-radius :float))
(shaped-light fn
#'(length :vec2)
p
light-position
light-color
light-range
source-radius))
</code></pre></div></div>
<p>Here we have the point-light function, it takes a function from <code class="language-plaintext highlighter-rouge">vec2</code> to <code class="language-plaintext highlighter-rouge">float</code> and some properties and then calls <code class="language-plaintext highlighter-rouge">shaped-light</code> passing in not just this but also a function that describes the distance from the light source (allowing for some funky shaped lights, though this is WIP).</p>
<p>This made me super happy as using first class functions in shaders as a means of composition had been a goal and seeing more validation of this was really fun.</p>
<h2 id="particle-graphs">Particle Graphs</h2>
<p>One thing with these functions is when you want to visualize them it’s a tiny bit tricky as for each point you have a distance, which makes 3 dimensions. We can plot the distance as a color but it requires remapping negative values and the human eye isnt as good as differentiating color as position.</p>
<p>To help with this I made a little particle graph system.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/mXjyJgPblrY" frameborder="0"></iframe>
<p><code class="language-plaintext highlighter-rouge">define-pgraph</code> is simply a macro that generates a pipelines that uses instancing to plot particles, however it was enough to make visualizing some stuff super simple. However as they are currently just additive particles it is difficult to judge which are in front sometimes. I will probably parameterize that in future.</p>
<h2 id="live-recompiling-lambda-pipelines">Live recompiling lambda-pipelines</h2>
<p>In making this I realized that there was a way to make CEPL’s lambda pipelines[0] be able to respond to recompilation of the functions they depend on without any visible api change for the users. This was clearly to tempting to pass up so I got that working.</p>
<p>The approach is as follows (a bit simplified but this is the gist):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((state (make-state-struct :pipeline pline ;; [1]
:p-args p-args))) ;; [2]
(flet ((recompile-func ()
(setf (state-pipeline state) ;; [4]
(lambda (&rest args)
(let ((new-pipeline
(recompile-pipeline ;; [5]
(state-p-args state)))) ;; [6]
(setf (state-pipeline state) ;; [7]
new-pipeline)
(apply new-pipeline args))))));; [8]
(setf (state-recompiler state) #'recompile-func)
(values
(lambda (context stream &rest args) ;; [3]
(apply (state-pipeline state) context stream args))
state)))
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">[1]</code> first we are going the take or pipeline function (<code class="language-plaintext highlighter-rouge">pline</code>) and box it inside a struct along with the arguments <code class="language-plaintext highlighter-rouge">[2]</code> (<code class="language-plaintext highlighter-rouge">p-args</code>) used to make the pipeline. If we hop down to the final lambda we see that this <code class="language-plaintext highlighter-rouge">state</code> object is lexically captured and the boxed function unboxed and called each time the lambda is called. Cool so we have boxed a lambda, however we need to be able to replace that inner pipeline whenever we want.</p>
<p>To do this we have the <code class="language-plaintext highlighter-rouge">recompile-func</code>, ostensibly we call this to recompile the inner pipeline function, however there is a catch: the recompile function could be called from any thread. Threads are not the friend of GL so we instead <code class="language-plaintext highlighter-rouge">recompile-func</code> actually replaces the boxed pipeline <code class="language-plaintext highlighter-rouge">[4]</code> with the actual lambda that will perform the recompile. As it is only valid for the pipeline to be called from the correct thread we can safely assume this. So next time the pipeline is called it’s actually the recompile lambda that will be called; this finally does the recompile <code class="language-plaintext highlighter-rouge">[5]</code> (using the original args we cached at the start <code class="language-plaintext highlighter-rouge">[6]</code>) and then replaces itself in the state object with the new pipeline function <code class="language-plaintext highlighter-rouge">[7]</code>. As the user was also expecting rendering to happen we call the new pipeline as well <code class="language-plaintext highlighter-rouge">[8]</code>.</p>
<p>There are trade-offs of course, this indirection will cost some cycles and as we don’t know how the signature of the inner pipeline function will change we can’t type it as strictly as we would want to otherwise. <em>However</em> we can opt out of this behavior by passing <code class="language-plaintext highlighter-rouge">:static</code> to the <code class="language-plaintext highlighter-rouge">pipeline-g</code> function when we make a lambda pipeline, this way we get the speed (and no recompilation) when we need it (e.g. when we ship a game) and consistency and livecoding when we want it (e.g. during development).</p>
<p>Needless to say I’m pretty happy with this trade.</p>
<h2 id="vari-describe">vari-describe</h2>
<p><code class="language-plaintext highlighter-rouge">vari-describe</code> is a handy function which returns the official glsl docs if available. This was handy but I have a growing library of gpu functions that naturally have no glsl docs.. what to do? The answer of course is to make sure docstring defined inside these are stored by varjo per overload so they can be presented. In the event of no docs being present we can at least return the signatures of the gpu function which, as long as the parameter names were good, can be quite helpful in itself.</p>
<p>This took a few rounds of bodging but works now. A nice thing also is that if you add the following to your <code class="language-plaintext highlighter-rouge">.emacs</code> file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun slime-vari-describe-symbol (symbol-name)
"Describe the symbol at point."
(interactive (list (slime-read-symbol-name "Describe symbol: ")))
(when (not symbol-name)
(error "No symbol given"))
(let ((pkg (slime-current-package)))
(slime-eval-describe
`(vari.cl::vari-describe ,symbol-name nil ,pkg))))
(define-key lisp-mode-map (kbd "C-c C-v C-v") 'slime-vari-describe-symbol)
</code></pre></div></div>
<p>You can hold control and hit <code class="language-plaintext highlighter-rouge">C V V</code> and you get a buffer with the docs for the function your cursor was over.</p>
<p>This all together made the coding experience significantly nicer so I’m pretty stoked about that.</p>
<h2 id="tiling-viewport-manager">Tiling Viewport Manager</h2>
<p>This is a project I tried a couple of years back and shelved as I got confused. In emacs I have a tonne of files/programs open in their respective buffers and the emacs window can be split into frames[9] in which the buffers are docked; this way of working is great and I wanted the same in GL.</p>
<p>The idea is to have a standalone library that handles layouting of frames in the window into which ‘targets’ (our version of buffers) are docked. These targets can either just be a CEPL <a href="http://techsnuffle.com/cepl/api.html?#CEPL.VIEWPORTS%3AVIEWPORT">viewport</a> or could be a <a href="http://techsnuffle.com/cepl/api.html?#CEPL.SAMPLERS%3ASAMPLER">sampler</a> or <a href="http://techsnuffle.com/cepl/api.html?#CEPL.FBOS%3AFBO">fbo</a>.</p>
<p>As the target is itself just a class you can subclass it and provide your own implementation. One thing I was testing out was making a simple color-picker using this. Here’s a wip, but the system cursor is missing from this, the point being ‘picked’ is at the center of the arc.</p>
<p><img src="/assets/images/picker.png" alt="yay" /></p>
<h2 id="back-to-work">Back to work</h2>
<p>Back to work now so gonna lisping will slow down a bit. In general I’m just happy with this trend, it’s going in the right direction and it feels like this could get even more enjoyable as I build up <a href="https://github.com/cbaggers/nineveh">Nineveh</a> and keep hammering out bugs as people find them.</p>
<p>Peace all, thanks for wading through this!</p>
<p>[0] pipelines in CEPL are usually defined at the top level using <code class="language-plaintext highlighter-rouge">defpipeline-g</code>, lambda pipelines give us <code class="language-plaintext highlighter-rouge">pipeline-g</code> which uses Common Lisp’s <code class="language-plaintext highlighter-rouge">compile</code> function to generate a custom lambda that dispatches the GL draw.
[9] in fact emacs has these terms the other way around but the way I stated is usually eaiser for those more familiar with other tools</p>
Lisping Furiously2018-03-25T19:23:09+00:00http://www.techsnuffle.com/2018/03/25/lisping-furiously<p>This week has been pretty productive.</p>
<p>It started with looking into a user issue where compile times were revoltingly slow. It turns out they were generated very deeply nested chains of <code class="language-plaintext highlighter-rouge">if</code>s and my code that was handling indenting was outstandingly bad, almost textbook example of how to make slow code. Anyway after getting that fixed up I spent a while scraping back milliseconds here and there.. there’s a lot of bad code in that compiler.</p>
<p>Speaking of that, I knock Varjo a bunch, and I’d never make it like this again, but as a vessel for learning it’s been amazing, most issues I hear about in compilers now I can at least hook somewhere in my head. It’s always ‘oh so <em>that’s</em> how real people do this’, very cool stuff. Also, like php, varjo is still unreasonably still providing me with piles of value; what it does is still what I wanted something to do.</p>
<p>After that I was looking into user extensible sequences and ended up hitting a wall in implementing something like <a href="http://www.doc.gold.ac.uk/~mas01cr/papers/ilc2007/sequences-20070301.pdf">extensible-sequences</a> from sbcl. I really want <code class="language-plaintext highlighter-rouge">map</code> & <code class="language-plaintext highlighter-rouge">reduce</code> in Vari, but in static languages this seems to mean some kind of iterator.</p>
<p>It would really help to have something like interfaces but I can’t stand the idea of not being able to say how another user’s type satisfies the interface, so I started looking into adding traits :)</p>
<p>I was able to hack in the basics and implement a very shoddy <code class="language-plaintext highlighter-rouge">map</code> and could get stuff like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TESTS> (glsl-code
(compile-vert () :410 nil
(let* ((a (vector 1.0 2.0 3.0 4.0))
(b (mapseq #'(sin :float) a)))
(vec4 (aref b 0)))))
"// vertex-stage
#version 410
void main()
{
float[4] A = float[4](1.0f, 2.0f, 3.0f, 4.0f);
float[4] RESULT = float[4](0.0f, 0.0f, 0.0f, 0.0f);
int LIMIT = A.length();
for (int STATE = 0; (STATE < LIMIT); STATE = (STATE++))
{
float ELEM = A[STATE];
RESULT[STATE] = sin(ELEM);
}
float[4] B = RESULT;
vec4 g_GEXPR0_767 = vec4(B[0]);
gl_Position = g_GEXPR0_767;
return;
}
</code></pre></div></div>
<p>but the way array types are handled in Varjo right now is pretty hard-coded when I had it making separate functions for the <code class="language-plaintext highlighter-rouge">for</code> loop each function was specific to the size of the array. So 1 function for <code class="language-plaintext highlighter-rouge">int[4]</code>, 1 function for <code class="language-plaintext highlighter-rouge">int[100]</code> etc.. not great.</p>
<p>So I put that down for a little bit and, after working on a bunch of issues resulting in unnecessary (but valid) code in the GLSL, I started looking at documentation.</p>
<p>This one is hard. People want docs, but they also don’t want the api to break. However the project is beta and documenting things reveals bugs & mistakes. So then you have to either document something you hate and know you will change tomorrow, or change it and document once.. For a project that is purely being kept alive by my own level of satisfaction it has to be the second; so I got coding again.</p>
<p>Luckily the generation of the reference docs was made much easier due to <a href="">Staple</a> which has a fantastically extensible api. The fact I was able to hack it into doing what I wanted (with some albeit truly dreadful code) was a dream.</p>
<p>So now we have these:</p>
<ul>
<li><a href="http://techsnuffle.com/varjo/vari-reference.html">Reference Docs for the Vari language</a></li>
<li><a href="http://techsnuffle.com/varjo/varjo-reference.html">Reference Docs for the Varjo compiler</a></li>
<li><a href="https://github.com/cbaggers/varjo/blob/master/docs/user-guide.md">The start of a user guide for Varjo</a></li>
</ul>
<p>Bloody exhausting.</p>
<p>The Vari docs are a mix of GLSL wiki text with Vari overload information & handwritten doc strings for the stuff from the Common Lisp <a href="http://techsnuffle.com/varjo/vari-reference.html#COMMON-LISP">portion of the api</a>.</p>
<p>That’s all for now. On holiday for a week and I’m going to try get as much lisp stuff out of my head as possible. I want my lisp projects to be in a place where I’m focusing on fixes & enhancements, rather than features for when I start my new job.</p>
<p>Peace</p>
It's been a long time2018-03-12T12:45:12+00:00http://www.techsnuffle.com/2018/03/12/its-been-a-long-time<blockquote>
<p>.. how have you been?</p>
<ul>
<li>glados</li>
</ul>
</blockquote>
<p>I fell out of the habit of writing these over christmas so it’s reall time for me to start this up again.</p>
<p>Currently I am reading <strong>__</strong>__ in preparation for my new job but I have also had some time for lisp.</p>
<p>On the lisp side I have been giving a bit more time to Varjo as there is a lot of completeness & stability things I need to work on over there. I’ve hacked a few extra sanity checks for qualifiers, however the checks are scattered around and it feels sucky so I really need to do a cleanup pass.</p>
<p>In good news I’ve also made a sweep through the CL spec looking for things that are worth adding to Vari. This resulted in <a href="https://github.com/cbaggers/varjo/issues/168#issuecomment-371074348">this list</a> and I’ve been able to add whole bunch of things. This really helps Vari feel closer to Common Lisp.</p>
<p>One interesting issue though is around clashes in the spec; lets take <code class="language-plaintext highlighter-rouge">round</code> for example. In CL <code class="language-plaintext highlighter-rouge">round</code> rounds to the nearest even, in GLSL it rounds in an implementation defined direction. GLSL does provide <code class="language-plaintext highlighter-rouge">roundEven</code> which more closely matches CL’s behavior. So the conundrum is, do we defined Vari’s <code class="language-plaintext highlighter-rouge">round</code> using <code class="language-plaintext highlighter-rouge">roundEven</code> or <code class="language-plaintext highlighter-rouge">round</code>. One way we may trip up CL programmers, the other way we trip up GLSL programmers, and also anyone porting code from GLSL to Vari without being aware of those rules. We can of course provide a <code class="language-plaintext highlighter-rouge">fast-round</code> function, but this doesnt neccessarily help with the issue of discoverability or ‘least surprise’.</p>
<p>I’m leaning towards keeping the GLSL meaning however, simply as we are already not writing true CL and our dialect makes sacrifices in the name of performance already. Maybe this is ok.</p>
<p>That’s all I’ve got for now, seeya all next week.</p>
<p>p.s. No stream this week as <a href="https://www.twitch.tv/ferrisstreamsstuff">ferris</a> is giving a <a href="https://www.meetup.com/Rust-Oslo/events/247872474/">rust talk</a> here in Oslo.</p>
<p>p.p.s I also feel the ‘I haven’t been making lil-bits-of-lisp videos in ages’ guilt so I need to get back to that soon.</p>
Raining Features2017-11-27T09:08:01+00:00http://www.techsnuffle.com/2017/11/27/raining-features<p>Every now and again people ask about if CEPL will support compute and I’ve always said that it would happen for years. The reason is that, because compute is not part of the GL’s draw pipeline, I thought there would be a tonne of compute specific changes that would need to be made. Turns out I was wrong.</p>
<p>Wednesday I was triaging some tickets in the CEPL repo and saw the one for compute. I was about to ignore it when I thought that it would be nice to read through the GL wiki to see just how horrendous a job it would be. 5 minutes later I’m rather disconcerted as it looked easy.. temptingly easy.</p>
<p>‘Luckily’ however I really want SSBOs for writing out of compute and <em>that</em> will be hard .. <code class="language-plaintext highlighter-rouge">looks at gl wiki again</code> .. Shit.</p>
<p>Ok so SSBOs turned out to be a much smaller feature than expected as well. So I gave myself 24 hours, from late Friday to late Saturday to implement as many new features (sanely) as I could.</p>
<p>Here are the results:</p>
<h3 id="sync-objects">Sync Objects</h3>
<p>GL only has one of these, the fence. CEPL now has support for this too.</p>
<p>You make a fence with <code class="language-plaintext highlighter-rouge">make-gpu-fence</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf some-fence (make-gpu-fence))
</code></pre></div></div>
<p>and then you can wait on the fence</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(wait-on-gpu-fence some-fence)
</code></pre></div></div>
<p>optionally with a timeout</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(wait-on-gpu-fence some-fence 10000)
</code></pre></div></div>
<p>also optionally flushing</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(wait-on-gpu-fence some-fence 10000 t)
</code></pre></div></div>
<p>Or you can simply check if the fence has signalled</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gpu-fence-signalled-p some-fence)
</code></pre></div></div>
<h3 id="query-objects">Query Objects</h3>
<p>GL has a range of queries you can use, we have exposed them as structs you can create as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(make-timestamp-query)
(make-samples-passed-query)
(make-any-samples-passed-query)
(make-any-samples-passed-conservative-query)
(make-primitives-generated-query)
(make-transform-feedback-primitives-written-query)
(make-time-elapsed-query)
</code></pre></div></div>
<p>To begin querying into the object you need to make the query active. This is done with <code class="language-plaintext highlighter-rouge">with-gpu-query-bound</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-gpu-query-bound (some-query)
..)
</code></pre></div></div>
<p>After the scope of <code class="language-plaintext highlighter-rouge">with-gpu-query-bound</code> the message to stop querying is in the gpu’s queue, however the results are not available immediately. To check if the results are ready you can use <code class="language-plaintext highlighter-rouge">gpu-query-result-available-p</code> or you can use some of the options to <code class="language-plaintext highlighter-rouge">pull-gpu-query-result</code>, let’s look at that function now.</p>
<p>To get the results to lisp we use <code class="language-plaintext highlighter-rouge">pull-gpu-query-result</code>. When called with just a query object it will block until the results are ready:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(pull-gpu-query-result some-query)
</code></pre></div></div>
<p>We can also say not to wait and CEPL will try to pull the results immediately, if they are not ready it will return nil as the second return value</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(pull-gpu-query-result some-query nil) ;; the nil here means don't wait
</code></pre></div></div>
<h3 id="compute">Compute</h3>
<p>To use compute you simply make a gpu function which takes no non-uniform arguments and always returns <code class="language-plaintext highlighter-rouge">(values)</code> (a void function in C nomenclature) and then make a gpu pipeline that only uses that function.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defstruct-g bah
(data (:int 100)))
(defun-g yay-compute (&uniform (woop bah :ssbo))
(declare (local-size :x 1 :y 1 :z 1))
(setf (aref (bah-data woop) (int (x gl-work-group-id)))
(int (x gl-work-group-id)))
(values))
(defpipeline-g test-compute ()
:compute yay-compute)
</code></pre></div></div>
<p>You can the <code class="language-plaintext highlighter-rouge">map-g</code> over this like any other pipeline..</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(map-g #'test-compute (make-compute-space 10)
:woop *ssbo*)
</code></pre></div></div>
<p>..with one little difference. Instead of taking a stream of vertices we now take a compute space. This specify the number of ‘groups’ that will be working on the problem. The value has up to 3 dimensions so <code class="language-plaintext highlighter-rouge">(make-compute-space 10 10 10)</code> is valid.</p>
<p>We also soften the requirements around gpu-function names for the compute stage. Usually you have to specify the full name of a gpu function due to possible overloading e.g. <code class="language-plaintext highlighter-rouge">(saturate :vec3)</code> however as compute shaders can only take uniforms, and we don’t offer overloading based on uniforms, there can only be one with a given name. Because of this we allow <code class="language-plaintext highlighter-rouge">yay-compute</code> instead of <code class="language-plaintext highlighter-rouge">(yay-compute)</code>.</p>
<h3 id="ssbos">SSBOs</h3>
<p>The eagle eyed of you will have noticed the <code class="language-plaintext highlighter-rouge">:ssbo</code> qualifier in the <code class="language-plaintext highlighter-rouge">woop</code> uniform argument. SSBOs give you storage you can write into from a compute shader. Their api is almost identical to that of UBOs so I copied-pasted that code in CEPL and got SSBOs working. This code will most likely be unified again once I have fixed some details with binding however for now we have something that works.</p>
<p>This means we can take our struct definition from before:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defstruct-g bah
(data (:int 100)))
</code></pre></div></div>
<p>and make a gpu-array</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf *data* (make-gpu-array nil :dimensions 1 :element-type 'bah))
</code></pre></div></div>
<p>and then make an SSBO from that</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf *ssbo* (make-ssbo *data*))
</code></pre></div></div>
<p>And that’s it, ready to pass to our compute shader.</p>
<h3 id="phew">.Phew.</h3>
<p>Yeah. So all of that was awesome, I’m really glad to have a feature land that I wasnt expecting to add for a couple more years. Of course there are bugs, the most immediately obvious is that when I tried the example above I was getting odd gaps in the data in my SSBO</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TEST> (pull-g *data*)
(((0 0 0 0 1 0 0 0 2 0 0 0 3 0 0 0 4 0 0 0 5 0 0 0 6
0 0 0 7 0 0 0 8 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)))
</code></pre></div></div>
<p>The reason for this that I didnt understand layout in GL properly. This also means CEPL doesnt handle it properly and that I have a bug :) (this bug)[https://github.com/cbaggers/cepl/issues/193].</p>
<p>Fixing this is interesting as it means that, unless we force you to make a different type for each layout e.g.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defstruct-g bah140 (:layout :std140)
(data (:int 100)))
(defstruct-g bah430 (:layout :std140)
(data (:int 100)))
</code></pre></div></div>
<p>..which feels ugly, we would need to support multiple layouts for each type. Which means the accessor functions in lisp would need to switch on this fact dynamically. That sounds slow to me when trying to process a load of foreign data quickly.</p>
<p>I also have a nagging feeling that the current way we marshal struct elements from c-arrays is not ideal.</p>
<p>This things together make me think I will be making some very breaking changes to CEPL’s data marshaling for the start of 2018.</p>
<p>This stuff needs to be done so it’s better we rip the band-aid off whilst we have very few known users.</p>
<p>News on that as it progresses.</p>
<h3 id="yay">Yay</h3>
<p>All in all though, a great 24 hours. I’m currently learning how to extract std140 layout info from varjo types so that will likely be what I work on next Saturday.</p>
<p>Peace</p>
Little bits of progress2017-11-20T09:49:00+00:00http://www.techsnuffle.com/2017/11/20/little-bits-of-progress<p>Hi again! This last week has gone pretty well. Shared contexts have landed in CEPL master, the only host that supports them right now is SDL2 although I want to make a PR to CEPL.GLFW as it should be easy to support there also. Glop is proving a little harder as we really need to update the OSX support, I started poking at it but it’s gonna take a while and I’ve got a bunch of stuff on my plate right now.</p>
<p>I started looking at <a href="https://www.khronos.org/opengl/wiki/Vertex_Rendering#Multi-Draw">multi-draw</a> & <a href="https://www.khronos.org/opengl/wiki/Vertex_Rendering#Indirect_rendering">indirect rendering</a> in GL as I think these are the features I want to implemented next. However I immediately ran into existing CEPL bugs in defstruct so I think the next few weeks are going to be spent cleaning that up and fixing the struct related issues from github.</p>
<p>AAAAges back I promised beginner tutorials for common lisp and I totally failed to deliver. I had been hoping the Atom support would get good enough that we could use that in the videos. Alas that project seems to have slowed down recently[0] and my guilt finally reached the point that I had to put out <em>something</em>. To that end I have started making a <a href="https://www.youtube.com/playlist?list=PL2VAYZE_4wRJi_vgpjsH75kMhN4KsuzR_">whole bunch of little videos</a> on random bits of common lisp. Although it doesnt achieve what I’d really like to do with proper tutorials, I hope it will help enough people on their journey into the language.</p>
<p>That’s all for now,
Peace.</p>
<p>[0] although I’m still praying it get’s finished</p>
Transform Feedback2017-11-13T09:10:35+00:00http://www.techsnuffle.com/2017/11/13/transform-feedback<p>Ah it feels good to have some meat for this week’s writeup. In short transform feedback has landed in CEPL and will ship in the next[0] quicklisp release.</p>
<h3 id="what-is-it">What is it?</h3>
<p>Transform feedback is a feature that allows you to write data out from one of the vertex stages into a VBO as well as passing it on to the next stage. It also opens up the possibility of not having a fragment shader at all and just using your vertex shader like the function is a gpu based <code class="language-plaintext highlighter-rouge">map</code> function. For a good example of it’s use check out <a href="http://prideout.net/blog/?tag=opengl-transform-feedback">this great tutorial by the little grasshopper</a>.</p>
<h3 id="how-is-it-exposed-in-cepl">How is it exposed in CEPL?</h3>
<p>In your gpu-function you simply add an additional qualifier to one or more of your outputs, like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g mtri-vert ((position :vec4) &uniform (pos :vec2))
(values (:feedback (+ position (v! pos 0 0)))
(v! 0.1 0 1)))
</code></pre></div></div>
<p>Here we see that the gl_position from this stage will be captured, now for the cpu side. First we make a gpu-array to write the data into.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf *feedback-vec4*
(make-gpu-array nil :element-type :vec4 :dimensions 100))
</code></pre></div></div>
<p>And then we make a transform feedback stream and attach our array. (transform feedback streams can have multiple arrays attached as we will see soon)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf *tfs*
(make-transform-feedback-stream *feedback-vec4*))
</code></pre></div></div>
<p>And finally we can use it. Assuming the gpu function above was used as the vertex stage in a pipeline called <code class="language-plaintext highlighter-rouge">some-pipeline</code> then the code will look like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-transform-feedback (*tfs*)
(map-g #'prog-1 *vertex-stream* :pos (v! -0.1 0)))
</code></pre></div></div>
<p>And that’s it! now the first result from <code class="language-plaintext highlighter-rouge">mtri-vert</code> will be written into the gpu-array in <code class="language-plaintext highlighter-rouge">*feedback-vec4*</code> and you can pull back the values like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>`(pull-g *feedback-vec4*)`
</code></pre></div></div>
<p>If you add the feedback modifier to multiple outputs then they will all be interleaved into the gpu-array. However you might want to write them into seperate arrays, this can be done by providing a ‘group number’ to the <code class="language-plaintext highlighter-rouge">:feedback</code> qualifier.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g mtri-vert ((position :vec4) &uniform (pos :vec2))
(values ((:feedback 1) (+ position (v! pos 0 0)))
((:feedback 0) (v! 0.1 0 1))))
</code></pre></div></div>
<p>making another gpu-array</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf *feedback-vec3*
(make-gpu-array nil :element-type :vec3 :dimensions 10))
</code></pre></div></div>
<p>and binding both arrays to a transform feedback stream</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf *tfs*
(make-transform-feedback-stream *feedback-vec3*
*feedback-vec4*))
</code></pre></div></div>
<p>You can also use the same pipeline multiple times within the scope of <code class="language-plaintext highlighter-rouge">with-transform-feedback</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-transform-feedback (*tfs*)
(map-g #'prog-1 *vertex-stream* :pos (v! -0.1 0))
(map-g #'prog-1 *vertex-stream* :pos (v! 0.3 0.28)))
</code></pre></div></div>
<p>CEPL is pretty good at catching and explaining cases where GL will throw an error such as: not enough vbos (gpu-arrays) bound for the number of feedback targets -or- 2 different pipelines called within the scope of <code class="language-plaintext highlighter-rouge">with-transform-feedback</code></p>
<h3 id="more-stuff">More stuff</h3>
<p>During this I ran into some aggravating issues relating to transform feedback and recompilation of pipelines, it was annoying to the point that I rewrote a lot of the code behind the <code class="language-plaintext highlighter-rouge">defpipeline-g</code> macro. The short version of this is that the code emitted is no longer a top-level closure and also that CEPL now has ways of avoiding recompilation when it can be proved that the gpu-functions in use havent changed.</p>
<p>I also found out that in some cases <code class="language-plaintext highlighter-rouge">defvar</code> with type declarations is faster than the captured values from a top level closure, even when they are typed. See <a href="https://github.com/cbaggers/cepl/blob/master/core/protocode/var-speed-test.lisp">here</a> for a test you can run on your machine to see if you get the same kind of results.[1]</p>
<h3 id="shipping-it">Shipping it</h3>
<p>Like I said this code is in the branch to be picked up by the next quicklisp release. This feature will certainly have it’s corner cases and bugs but I’m happy to see this out and to have one less missing feature from CEPL.</p>
<p>Future work on transform feedback includes using the <a href="https://www.khronos.org/opengl/wiki/Transform_Feedback#Feedback_objects">transform feedback objects</a> introduced in GLv4 to allow for more advanced interleaving options and also nesting of the <code class="language-plaintext highlighter-rouge">with-transform-feedback</code> forms.</p>
<p>Next on the list for this month is shared contexts. More on that next week!</p>
<p>Ciao</p>
<p>[0] late november or early december
[1] testing on my mac has given different results, use <code class="language-plaintext highlighter-rouge">defvar</code> and packing data in a struct is faster than multiple <code class="language-plaintext highlighter-rouge">defvar</code>s but slower than the top level closure. Luckily it’s still on the order of microseconds a frame (assuming 5000 calls per frame) but measurable. It’s interesting to see as packed in <code class="language-plaintext highlighter-rouge">defvar</code> was faster on my linux desktop <code class="language-plaintext highlighter-rouge">¯\_(ツ)_/¯</code> I’ll show more data when I have it.</p>
Multiple Contexts are ALIVE..kinda2017-11-07T08:56:13+00:00http://www.techsnuffle.com/2017/11/07/multiple-contexts-are-alivekinda<p>Allo again!</p>
<p>Its now November which means it’s <a href="https://nanowrimo.org/">NanoWrimo</a> time again, each year I like to participate in spirit by picking a couple of features for projects I’m working on and hammer them out. This is well timed as I haven’t had much time for adding new things to CEPL recently.</p>
<p>The features I want to have by the end of the month are decent multi-context support and single stage pipelines.</p>
<h3 id="multi-context">Multi-Context</h3>
<p>This stuff we have already talked about but I have bit the bullet and got coding at last. I have support for non-shared contexts now but not shared ones yet, the various hosts have mildly-annoyingly different approaches to abstracting this so I’m working on finding the sweet spot right now.</p>
<p>Regarding the defpipeline threading issues from last week I did in the end opt for a small array of program-ids per pipeline indexed by the cepl-context id. I can make this fast enough and the cost is constant so that feels like the right call for now. CEPL & Varjo were never written with thread safety in mind so it’s going to be a while before I can do a real review and work out what the approach should be, for now its a very ‘throw some mutexes in and hope’ situation, but it’s fine…we’ll get there :p</p>
<p>One side note is that all lambda-pipelines in CEPL are tied to their thread so don’t need the indirection mentioned above :)</p>
<h3 id="single-stage-pipelines">Single Stage Pipelines</h3>
<p>This is going to be fun. Up until now if you wanted to render a fullscreen quad you needed to:</p>
<ul>
<li>make a gpu-array holding the quad data</li>
<li>make a stream for that gpu-array</li>
<li>make a pipeline with:</li>
<li>a vertex shader to put the points in clip space</li>
<li>a fragment shader</li>
</ul>
<p>The annoying thing is that the fragment shader was the only bit you really cared about. Luckily it turns out there is a way to do this with geometry shaders and no gpu-array and it should be portable for all GL versions CEPL supports. So I’m going to prototype this out on the stream on wednesday and, assuming it works, I’ll make this into a CEPL feature soon.</p>
<p>That covers making pipelines with only a fragment shader of course but what about with only a vertex stage? Well <a href="https://www.khronos.org/opengl/wiki/Transform_Feedback">transform feedback</a> buffers are something I’ve wanted for a while and so I’m going to look into supporting those. This is cool as you can then use the vertex stage for data processing kinda like a big pmap. This could be handy when you data is already in a gpu-array.</p>
<h3 id="future-things">Future things</h3>
<p>With transform feedback support a few possibilities open up. The first is that with some dark magic we could run a number of variants of the pipeline using transform feedback to ‘log’ values from the shaders, this gives us opportunities for debugging that weren’t there before.</p>
<p>Another tempting idea (which is also easier) is to allow users to call a gpu-function directly. This will</p>
<ul>
<li>make a temporary pipeline</li>
<li>make a temporary gpu-array and stream with just the arguments given (1 element)</li>
<li>run the code on the gpu capturing the result with a temporary transform-feedback buffer or fbo</li>
<li>convert the values to lisp values</li>
<li>dispose of are the temporaries</li>
</ul>
<p>The effect is to allow people to run gpu code from the repl for the purposes of rapid prototyping. It obviously is useless in production because of all the overhead but being able to iterate in the repl with stuff like this could really be great.</p>
<h3 id="thatll-do-pig">That’ll do pig</h3>
<p>Right, time to go.</p>
<p>Seeya soon</p>
Mutliple Contexts2017-10-18T11:41:55+00:00http://www.techsnuffle.com/2017/10/18/mutliple-contexts<p>This last weekend I put a little time into multi-context support in CEPL.</p>
<p>CEPL has it’s own context (<code class="language-plaintext highlighter-rouge">cepl-context</code>) class that holds both the gl-context handle[0] and also state that is cached to improve performance. <code class="language-plaintext highlighter-rouge">cepl-context</code>s are passed implicitly[1] down the stack and are tied to a single thread.</p>
<p>Most of the work was just finding simple errors in my code an shoring them up, but I did find one tricky case and that was in pipelines. So a pipeline is usually defined in a top level declaration like so:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defpipeline my-pipeline ()
:vertex some-gpu-function
:fragment some-other-gpu-function)
</code></pre></div></div>
<p>And this generates all the bootstrapping to compile the gpu functions, get the GL program-id, etc. However that program-id is a GL resource and belongs to a single GL context. As it is right now it’ll be the context that calls this pipeline first..ew.</p>
<p>So how to tackle this? We could create one program-id per context, however this means either looking up the program-id based on the context per call in a pipeline local cache..or looking up the program-id in a context local cache based on the pipeline. Neither is great, as extra lookups per call are something we should be avoiding.</p>
<p>Another option is to have shared GL contexts. This is nice anyway as it means we can share textures/buffers/etc between threads which I think is a nice default behavior. However even with this solution there are still issues with pipelines.</p>
<p>The state of a gl program object is naturally shared between the two threads too, that state includes which uniforms are bound, so if two threads try to use the same pipeline with different uniforms then we are in a fun data-racey land again.</p>
<p>This seem to lead back to the ‘gl program per gl context’ thing again. I’ll ponder this some more but I think it’s the only real option.</p>
<p>Happy to hear suggestions too,</p>
<p>I think that’s all for now</p>
<p>Peace</p>
<p>[0] in the future I expect I will allow multiple GL contexts per CEPL context
[1] or explicitly if you prefer</p>
Small things2017-10-11T09:02:48+00:00http://www.techsnuffle.com/2017/10/11/small-things<p>This last week hasn’t seen much exciting code so there isn’t too much to write up.</p>
<p>I’m still dreaming up some way to wrangle data in my games in a way that maximizes performance whilst keeping live redefinition in tact, however this isn’t even fully formed in my head yet so there is no code to show or even speak of. However I’ve been increasingly interested in relational databases recently. The fact that you only define the layout of your table data and queries, and that the system just works out what other passes as intermediate data-structures it needs to work best is pretty sweet. You can get a free book on <a href="https://www.red-gate.com/library/inside-the-sql-server-query-optimizer">mssql query optimizer here</a>.</p>
<p>CppCon is also out, here are a few good talks I’ve been watching so far:</p>
<ul>
<li><a href="https://youtu.be/NH1Tta7purM">Carl Cook “When a Microsecond Is an Eternity: High Performance Trading Systems in C++”</a></li>
<li><a href="https://youtu.be/bSkpMdDe4g4">Matt Godbolt “What Has My Compiler Done for Me Lately? Unbolting the Compiler’s Lid”</a></li>
<li><a href="https://youtu.be/YM8Xy6oKVQg">P. McKenney, M. Michael & M. Wong “Is Parallel Programming still hard?”</a> (spoiler..yes)</li>
<li><a href="https://youtu.be/86seb-iZCnI">Olivier Giroux “Designing (New) C++ Hardware”</a></li>
</ul>
<p>I’ve also just had a <a href="http://gchandbook.org/">book on Garbage Collection</a> delivered. YAY! It’s another one of those amazing computer systems where you get to directly impact people, but without having the deal with horrible human factors (like unicode & dates & BLEEEEGHH). I’m pretty stoked to work through this book.</p>
<p>Other than this researchy stuff I’ve still been streaming. Last week we played with a physics engine and tonight we are going to implement chromatic aberration :) I’m pretty happy with where the streaming has been going, the nerve wracking part of the process these days is finding things I can do in the two hours rather than the stream itself.</p>
<p>That’ll do for now, seeya next week</p>
Not much this time2017-10-02T09:12:21+00:00http://www.techsnuffle.com/2017/10/02/not-much-this-time<p>My lack of focus over the weekend was disappointing so I haven’t got much to report. The one thing I did get done however was to finish adding types to <a href="https://github.com/cbaggers/issac">my WIP lisp bindings</a> for the <a href="http://newtondynamics.com/">newton-dynamics physics engine</a>. This was motivated by the fact that although I had got the basics working a while back, I had seen some overhead from the lisp code; that should be minimized now.</p>
<p>I think I might try using the physics bindings on this week’s stream. Could be fun.</p>
<p>Other than that I’ve been reading and procrastinating. <a href="https://www.red-gate.com/library/inside-the-sql-server-query-optimizer">This book</a> is now in my ‘to read’ list, I have no desire to make a proper database but I’m super interested in how their query planner/optimizers work.</p>
<p>That’s all for now, seeya!</p>
Sketch2017-09-25T08:58:26+00:00http://www.techsnuffle.com/2017/09/25/sketch<p>This weekend I put a bit of time into <a href="https://github.com/vydd/sketch">Sketch</a> which I, to my shame, have not worked on in a while. Sketch is a lovely project by Vydd which looks to sit in a similar place to processing, but in the lisp world.</p>
<p>A while back I was approach to look into porting it to CEPL so we could have the shader development process of CEPL in Sketch. We started by monkey-patching CEPL in which provided a fantastic test case for performance and resulted in some big refactoring and wins back in July.</p>
<p>Sketch was previously built on the excellent <a href="https://github.com/lispgames/sdl2kit">sdl2kit</a> but there aren’t enough hooks in the projects to have them work together yet so I’m currently replacing the bootstrapping. I stripped down a bunch of code and have a test which shows things are rendering so that’s a start. However CEPL’s support for multiple contexts is untested so this project is really gonna force me to implement that well which is AWESOME. Incidentally sketch was the project that forced me to add CEPL’s multi window support (which will also get more robust as I port this).</p>
<p>Other than that I’m busy with other projects and ideas that may become stuff in the future, I’ve got so much to learn :) This last week has seen me binging on xerox parc related research talks (mainly smalltalk stuff) which has been building up a nice healthy level of dissatisfaction. I have proto-ideas rocking around with big ol’ gaps in their narratives, so I’m just pushing a load of chunks of software dna into my head in the hope of some aberrant collision will result in some useful mental genesis will occur. TLDR feed brain hope to shit ideas.</p>
<p>That’ll do for this post.</p>
<p>Seeya!</p>
The long path to shader debugging2017-09-18T09:32:42+00:00http://www.techsnuffle.com/2017/09/18/the-long-path-to-shader-debugging<p>Writing shaders (in lisp or otherwise) is fun, however debugging them is not. Where on the CPU we get exceptions or error codes, on the gpu we get silence and undefined behavior. I really felt this when trying (and failing) to implement procedural terrain generation on the livestream. I tried to add additional outputs so that I could inspect the values but it was very easy to make a mistake and change the behavior of the shader..or worse to forget it was there and waste time debugging a side effect from the instrumentation. I need a more reliable way to get values back to the CPU. Luckily CEPL has some great places we can hide this logic.</p>
<p>Quick recap, in CEPL we define GPU functions and then compose them into a pipeline using <code class="language-plaintext highlighter-rouge">defpipeline-g</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defpipeline-g some-pipeline ()
(vertex-stage :vec4)
(fragment-stage :vec2))
</code></pre></div></div>
<p>This is a macro that generates a function called <code class="language-plaintext highlighter-rouge">some-pipeline</code> that does all the wrangler to make the gl draw call. You then use it by using <code class="language-plaintext highlighter-rouge">map-g</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(map-g #'some-pipeline vertex-data)
</code></pre></div></div>
<p>This is another macro that expands into some plumbing and (ultimately) a call to the <code class="language-plaintext highlighter-rouge">some-pipeline</code> function.</p>
<p>Putting aside other details what we have here is 2 places we can inject code, one in the function body and one at the function call-site. This gives us tonnes of leverage.</p>
<p>My goal is to take some gpu-function like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g qkern ((tc :vec2) &uniform (tex :sampler-2d) (offset :vec2))
(+ (* (texture tex (- tc offset)) 0.3125)
(* (texture tex tc) 0.375)
(* (texture tex (+ tc offset)) 0.3125)))
</code></pre></div></div>
<p>And add calls to some function we will call <code class="language-plaintext highlighter-rouge">peek</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g qkern ((tc :vec2) &uniform (tex :sampler-2d) (offset :vec2))
(+ (peek (* (texture tex (peek (- tc offset))) 0.3125))
(* (texture tex tc) 0.375)
(* (texture tex (+ tc offset)) 0.3125)))
</code></pre></div></div>
<p>Peek will capture the value at that point and make it available for inspection from the CPU side of your program.</p>
<p>The way we can do it is to:</p>
<ul>
<li>compile the shader normally (we need to do this anyway)</li>
<li>inspect the AST for calls to peek and the types of the argument</li>
<li>create a new version of the shader with peek replaced with the instrumenting code</li>
</ul>
<p>For example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g qkern ((tc :vec2) &uniform (tex :sampler-2d) (offset :vec2))
(let (((dbg-0 :vec2))
((dbg-1 :vec4)))
(+ (setf dbg-1 (* (texture tex (setf dbg-0 (- tc offset))) 0.3125))
(* (texture tex tc) 0.375)
(* (texture tex (+ tc offset)) 0.3125))
(values dbg-0 dbg-1)))
</code></pre></div></div>
<p>This code will work mostly the same way except that it will be returning the captured values instead of the original one. I say ‘mostly’ as now the code that doesnt contribute to the captured values is essentially dead code and it is likely that the GLSL compiler will strip chunks of it.</p>
<p>So now we have an augmented shader stage as well as the original, <code class="language-plaintext highlighter-rouge">defpipeline-g</code> can generate, compile and store these and on each <code class="language-plaintext highlighter-rouge">map-g</code> it can make 2 draw calls. First the debug one capturing the results using <a href="https://www.khronos.org/opengl/wiki/Transform_Feedback">transform-feedback</a> (for the vertex stages) and FBOs for the fragment stage. Because <code class="language-plaintext highlighter-rouge">map-g</code> is also a macro we use it to implicitly pass the thread-local ‘CEPL Context’ object to the pipeline function. This lets us write debug values into a ‘scratch’ buffer stored on the context making the whole process transparent.</p>
<p>With this data available we can then come up with nice ways to visualize it. Just dumping it to the REPL will usually be a bad move as a single <code class="language-plaintext highlighter-rouge">peek</code> in a fragment shader is going to result in a value for every fragment, which (at best) means 2073600 values for a 1920x1080 render target.</p>
<p>There are a lot of details to work out to get this feature to work well[0], however it could be a real boost in getting real data[1] back from these pipelines and can work on all GL versions CEPL supports.</p>
<p>Seeya next week,
Peace.</p>
<p><code class="language-plaintext highlighter-rouge">[0]:</code> transform feedback only works from the last implemented vertex stage, so if you have vertex, tessellation & geom stages, only geom can write to the transform feedback buffer.
<code class="language-plaintext highlighter-rouge">[1]:</code> Another option was to compile the lisp like shader language to regular lisp. However implementing the GLSL standard library exactly is hard and it’s impossible to capture all the gpu/manufacturer specific quirks.</p>
More small steps2017-09-11T09:28:06+00:00http://www.techsnuffle.com/2017/09/11/more-small-steps<p>Over the weekend I got a little lisping done and was working on something that has been rolling around my head for a couple of years.</p>
<p>During the standardization process of lisp as well as agreeing on what would go in, there were also things cut. Some of those things have become de facto standards as all the implementations ship them, however some seem rather fundamental.</p>
<p>One of the more fundamental ones that didn’t make it was the idea of introspectable (and extensible) <a href="http://clhs.lisp.se/Body/03_aad.htm">environment objects</a>.</p>
<p>The high level view goes something like this: An environment object is a set of lexical bindings, having access to this (and any metadata about those bindings) would allow you to do more semantic analysis of the code. Given that any macro is allowed access to the environment object when evaluated this would allow a macro to expand differently depending on the data in the environment.</p>
<p>For example let’s say that we use the environment to store static type information in the environment; we could then potentially optimize certain function calls within that scope using this information (like using static dispatch on a generic function call).</p>
<p>Now a while back stassats kindly shared <a href="http://paste.lisp.org/display/310079">a snippet</a> which allows you to <em>essentially</em> define a variable who’s value is available during macro expansion. Over the weekend I’ve been playing with this to provide a way to allow the following.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defmacro your-macro-here (&environment env x y)
(with-ext-env (env)
(augment-environment env :foo 10)
`(+ x y)))
</code></pre></div></div>
<p>So you can wrap the code in your macro in <code class="language-plaintext highlighter-rouge">with-ext-env</code> and this lets you get access to a user-extensible environment object. We would then provide functions (like <code class="language-plaintext highlighter-rouge">augment-environment</code>) to modify the environment, in the above code to store the value <code class="language-plaintext highlighter-rouge">10</code> against the key <code class="language-plaintext highlighter-rouge">:foo</code> however we could use this for type info.</p>
<p>The downsides are that we don’t get all the data that was potentially available in the <a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html">proposed feature</a>. I’d really like to have access to all the standard declarations made as well as our additional ones.</p>
<p>Luckily it’s possible to make a new CL package with modified versions of <code class="language-plaintext highlighter-rouge">let</code>, <code class="language-plaintext highlighter-rouge">labels</code>, etc and in those to capture the metadata and make it available to our extensible environment.</p>
<p>With this we may be able to make something that convincingly does the job of a extensible macro environment. I have made a prototype of the meat (the passing of the environment itself) and so next is it wrap the other things into a package and then see if it is useful.</p>
<p>Other than this I’ve been poking around a little with Unity. It’s fun to see how it’s done in the big leagues and to see where ones approach aligns and diverges from a larger player’s philosophy.</p>
<p>That’s all for now,</p>
<p>Seeya!</p>
Rolling forward2017-09-08T11:24:57+00:00http://www.techsnuffle.com/2017/09/08/rolling-forward<p>Once again I don’t have much to report but things are going ok.</p>
<p>The streams are still going and still going well, this week was using a cute approach from http://nullprogram.com/blog/2014/06/01/ to make voronoi diagrams. You can see that stream <a href="https://youtu.be/82o5NeyZtvw">here</a></p>
<p>I also revisited my bindings to the <a href="http://newtondynamics.com/">Newton Dynamics</a> physics engine. Last time I had tested it seem they were abysmally slow. Luckily a quick review showed me that the ‘max fps’ for the simulation was set too low and that to something sensible made a world of difference. Some profiling also revealed that the bindings have a non-trivial amount of overhead which I believe I can remove by declaring the types and turning up the performance optimizations.</p>
<p>That’s all for now
Ciao</p>
Back from wherever2017-08-30T11:18:37+00:00http://www.techsnuffle.com/2017/08/30/back-from-wherever<p>2 days back I started getting into coding in the evenings again. For a while my brain has not been into it but it looks like I might be back at last.</p>
<p>So far I havent done much but tonight I’ll be doing another stream where I will try to implement <a href="https://en.wikipedia.org/wiki/Boids">boids</a>.</p>
<p>That’s all for now, except that if you are an old fan of Command & Conquer you can get the whole game here for free http://nyerguds.arsaneus-design.com/cnc95upd/cc95p106/ . It looks sketchy but it’s legit and DAMN it’s fun :D</p>
<p>Peace all</p>
Scrabbling back onto the wagon2017-08-10T09:07:31+00:00http://www.techsnuffle.com/2017/08/10/scrabbling-back-onto-the-wagon<p>I fucked up for a few weeks by not doing these. Sorry about that.</p>
<p>Update will be easy as I have been in ‘absorb phase’ for a few weeks and so I’ve not been coding much in my free time.</p>
<p>Aside from watching a few films and playing a few games my intake has mainly been around the ‘data oriented design’ space. I’ve been rewatching these:</p>
<p><a href="https://youtu.be/WDIkqP4JbkE">code::dive conference 2014 - Scott Meyers: Cpu Caches and Why You Care</a>
<a href="https://youtu.be/fHNmRkzxHWs">CppCon 2014: Chandler Carruth - Efficiency with Algorithms, Performance with Data Structures</a>
<a href="https://youtu.be/rX0ItVEVjHc">CppCon 2014: Mike Acton - Data-Oriented Design and C++</a></p>
<p>And then</p>
<p><a href="https://youtu.be/cX_GhJ6BuWI">code::dive 2016 conference – Chandler Carruth – Making C++ easier, faster and safer (part 1)</a>
<a href="https://youtu.be/E36BUcTWo50">code::dive 2016 conference – Chandler Carruth –Making C++ easier, faster, safer (part 2)</a>
<a href="https://youtu.be/s4wnuiCwTGU">On “simple” Optimizations - Chandler Carruth - Secret Lightning Talks - Meeting C++ 2016</a>
<a href="https://youtu.be/eR34r7HOU14">2013 Keynote: Chandler Carruth: Optimizing the Emergent Structures of C++</a>
<a href="https://youtu.be/RT46MpK39rQ">Things that Matter - Scott Meyers | DConf2017</a></p>
<p>I’ve also been reading the wonderful <a href="http://futuretech.blinkenlights.nl/misc/cpumemory.pdf">What every programmer should know about memory</a> which is poor in name but outstanding in content. It’s really for programmers who need to get everything out of the machine and folks who need to understand why they aren’t. I’m not finished yet but its already been very helpful.</p>
<p>From that my interest in assembly programming was growing again. The primary reason is that lisp has a disassemble function that lets you see what your function got compiled to. Here is a destructive ‘multiply a vector3 by a float’ function:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CL-USER> (disassemble #'v3-n:*s)
; disassembly for RTG-MATH.VECTOR3.NON-CONSING:*S
; Size: 51 bytes. Origin: #x22966965
; 65: F30F105201 MOVSS XMM2, [RDX+1] ; no-arg-parsing entry point
; 6A: F30F59D1 MULSS XMM2, XMM1
; 6E: F30F115201 MOVSS [RDX+1], XMM2
; 73: F30F105205 MOVSS XMM2, [RDX+5]
; 78: F30F59D1 MULSS XMM2, XMM1
; 7C: F30F115205 MOVSS [RDX+5], XMM2
; 81: F30F105209 MOVSS XMM2, [RDX+9]
; 86: F30F59CA MULSS XMM1, XMM2
; 8A: F30F114A09 MOVSS [RDX+9], XMM1
; 8F: 488BE5 MOV RSP, RBP
; 92: F8 CLC
; 93: 5D POP RBP
; 94: C3 RET
; 95: 0F0B10 BREAK 16 ; Invalid argument count trap
</code></pre></div></div>
<p>Cool, but naturally not much good if I can’t read it. So that is my driving factor: Understanding this ↑↑↑↑.</p>
<p>I have been repeatedly advised to start with something simpler that x64 asm, but i just cant find any motivation, when it comes down to it I dont want to program for the c64 any more, and arm/mips holds no appeal until I have a need for it. So against better judgement I picked up the <a href="https://support.amd.com/TechDocs/24592.pdf">AMD64 Architecture Programmer’s Manual Volume 1</a> and started reading.. and it’s nice. I quickly run into things I dont get of course but then it’s off to youtube again where <a href="https://www.youtube.com/playlist?list=PLetF-YjXm-sCH6FrTz4AQhfH6INDQvQSn">kupula’s series of x86_64 Linux Assembly tutorials</a> gave me a nice soft intro.</p>
<p>This is a clearly the start of a <em>looooong</em> road, but I’m in no rush, and everything I learn is directly helping me grok that disassembly above.</p>
<table>
<tbody>
<tr>
<td>I think that’s it. I’m still streaming and still struggling with the procedural erosion stuff..turns out there are more bugs in the paper than expected :</td>
</tr>
</tbody>
</table>
<p>But that’s for another day</p>
<p>Seeya!</p>
Post Holiday2017-07-18T10:28:25+00:00http://www.techsnuffle.com/2017/07/18/post-holiday<p>I haven’t written for a while as I had a two week holiday and decided to take a break from most things except streaming. It was lovely and I got so much done that this can only really be a recap:</p>
<h3 id="performance">Performance</h3>
<p>I made a profiler for CEPL that is plenty fast enough to use in real-time projects. I then used some macro ‘magic’ to instrument every function in CEPL, this let me target the things that were slowest in CEPL. I could happily fill a post on just this, but I will spare you that for now. Needless to say, CEPL is faster now.</p>
<p>I am contributing to a project called <a href="https://github.com/vydd/sketch">sketch</a> by porting it to CEPL and I was really annoyed at how it was doing 90fps on one stress test and the CEPL branch was doing 20. After the perfomance work the CEPL branch was at 36fps but still sucking compared to the original branch. In a fit of confusion I commented out the rendering code and it only went up to 40fps..at which point I realized that, on master, I had been recording fps from a function that was called 3 times a frame :D so the 90fps was actually 30!</p>
<p>The result was pretty cool though as the embarrassment had probably pushed be to do more than I would have done otherwise.</p>
<h3 id="gl-coverage">GL Coverage</h3>
<p>I have added CEPL support for:</p>
<ul>
<li>scissor & scissor arrays</li>
<li>stencil buffers, textures & fbo bindings</li>
<li>Color & Stencil write masks</li>
<li>Multisample</li>
</ul>
<p>and also fixed a pile of small bugs.</p>
<h3 id="streaming">Streaming</h3>
<p>Streaming is going well although removing the ‘fail privately’ from programming is something I’m still getting used to. (I LOVE programming for letting me fail so much without it all being on record)</p>
<p>This wednesday I’m going to try and do a bit of procedural terrain erosion which should be cool!</p>
<h3 id="fbx">FBX</h3>
<p>FBX is a proprietary format for 3D scenes which is pretty much an industry standard. It is annoyingly proprietary and the binary format for the latest version is not known. There is a zero-cost (but not open source) library from autodesk for working with fbx files but it’s a very c++ library so using it from the <a href="https://en.wikipedia.org/wiki/Foreign_function_interface">FFI</a> of most languages is not an option.</p>
<p>Because this is a pita I’ve decided to make the simplest solution I can think of, use the proprietary library to dump the fbx file to a bunch of flat binary files with known layout. This will make it trivial to work with from any language which can read bytes from a file. I’m specifically avoiding a ‘smarter’ solution as there seem to be a bunch of projects out there in various states of neglect and so I am making something that, when autodesk inevitably make a new version of the format, I can change without much thought.</p>
<h3 id="the-effect-on-my-life">The effect on my life</h3>
<p>It’s odd to be in this place, 5 years of on/off work on CEPL and this thing is starting to look somewhat complete. There is still a bunch of bugfixes and newer GL features to add but the original vision is pretty much there. This seems to have freed up the mind to start turning inwards again and I’ve been in a bit of a slump for the last week. These slumps are something I have become a bit more aware of over the last year and so have become a little better at noticing.. Still, as this is about struggles as well as successes and as the rest of this post is listing progress it seemed right to mention this.</p>
<p>Peace folks</p>
input given2017-06-23T09:36:19+00:00http://www.techsnuffle.com/2017/06/23/input-given<p>I’m late writing this week but there has been progress.</p>
<p>So after the PBR fail I knew I had to put some energy in different places for a week as I needed content for the stream. I knew dealing with input was coming soon and my input system was ugly, so that was an obvious candidate.</p>
<p>This is probably the 4th attempt I’ve made of making an event system. The originals were to varying degrees attempts on the callback/reactive input systems but they suffered a few key problems:</p>
<ul>
<li>Callbacks can be hard to reason about when there are enough of them</li>
<li>Subscribing to something means that the provider now holds a reference to the subscribee, which means it cant be freed if you forget to detach it (this is an issue as we experiment in the repl a lot so throwing things away should be easy)</li>
<li>Input propagation can end up driving the per frame execution (more on this in a minute)</li>
<li>In the versions where I made immutable events the allocation costs per frame could be very high[0]</li>
</ul>
<p>The third one was particularly tricky, and it only became apparent to me when I had some specific use cases:</p>
<p>The first one was with mouse position events. The question is, which event was the last one for that frame. Let’s say I’m positioning something based on mouse position and that causes a bunch of work in other parts of the game. If I receive 10 move events in a frame I don’t want to do that work 10 times so there is going to be some kind of caching and I need to know when I have received the last event. This is a bit artificial and there are plenty of strategies around it but caching is something that ends up appearing a lot in the event propagating approach.</p>
<p>Next was when I had made a fairly strict entity component system and was trying it out in a little game. In this system you processed the components in batches, so you would process all of the <code class="language-plaintext highlighter-rouge">location</code> components and then all of the <code class="language-plaintext highlighter-rouge">renderable</code> components, etc. This posed a problem as events needed to be delivered to components but events were pumped at the beginning of the frame and components were processed later. I didn’t want to break the model so again I fell back to caching.</p>
<p>I needed something different and so the latest version, <a href="https://github.com/cbaggers/skitter">skitter</a>, is much simpler.</p>
<p>First some design goals:</p>
<ul>
<li>The api is pull based[1]</li>
<li>There must only be allocations happening when new input-sources (like mice, gamepads) are added or when new controls (like buttons, scroll wheels) are added to input sources.</li>
<li>skitter shouldn’t need to know about any specific input api, it’s only dealing in its own state.</li>
<li>It should be possible to both redefine controls live OR tell the system to make them static and optimize the crap out of them.</li>
</ul>
<p>In the system you now define a control like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; name ↓↓ lisp data type ↓↓ ↓↓ initial value
(define-control wheel (:static t) single-float 0.0)
</code></pre></div></div>
<p>This is like defining a type in the input system, you give it a name, tell it the kind of data it holds and it’s initial value.</p>
<p>the <code class="language-plaintext highlighter-rouge">:static t</code> bit means this will be statically typed and will not be able to be changed live</p>
<p>this one is slightly different:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(define-control relative2 (:static t) vec2 (v! 0 0) :decays t)
</code></pre></div></div>
<p>Here we have told it to ‘decay’. This means that each frame the value returns to the initial value. This is because you don’t get an event from (all/any?) systems to tell you that something has stopped happening.</p>
<p>We can now make an input source</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(define-input-source mouse ()
(pos position2)
(move relative2)
(wheel wheel2)
(button boolean-state *))
</code></pre></div></div>
<p>Here a mouse has an absolute position, a relative movement, a possibly 2d scroll wheel and buttons (we dont know how many so we say <code class="language-plaintext highlighter-rouge">*</code>)</p>
<p>We can then call <code class="language-plaintext highlighter-rouge">(mouse-pos mouse-obj)</code> to get the position.</p>
<p>You can then write code that takes the events from your systems (<code class="language-plaintext highlighter-rouge">sdl2</code>, <code class="language-plaintext highlighter-rouge">qt</code>, <code class="language-plaintext highlighter-rouge">glfw</code>, etc) and puts them into the source by using functions like <code class="language-plaintext highlighter-rouge">(skitter:set-mouse-pos mouse-obj timestamp new-position)</code></p>
<p>Bam cross system event management. It’s dirt simple to use and the hairy logic is all handled by a few macros internally.</p>
<p>BUt we have missed one thing that callbacks were good for, events over time. Reactive approaches have this awesome way of composing streams of events into new events and, whilst I can’t have that, I do want something. The caching in our system poses a problem as we now <em>only</em> have the latest event. So rather that bringing the events to the processing code we will bring the processing to the event (or something :D).</p>
<p>What we do is add a <code class="language-plaintext highlighter-rouge">logicial-control</code> to our input-source. Maybe we want double-click for our mouse, we will make a kind of control with it’s own internal state that sets itself to true when it has seen two events within a certain timeframe:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(define-logical-control (double-click :type boolean :decays t)
((last-press 0 integer))
((button boolean-state)
(if (< (- timestamp last-press) 10)
(progn
(setf last-press 0)
(fire t))
(setf last-press timestamp))))
</code></pre></div></div>
<p>This is made up & untested code, however how it should work is that we have new kind of control called double-click, it’s state is boolean. It has a private field called last-press which holds the time since the last press. It depends on the boolean-state of another control and internally we give this thing the name ‘button’. The <code class="language-plaintext highlighter-rouge">if</code> is going to get evaluated every time one of the dependent controls (<code class="language-plaintext highlighter-rouge">button</code> in this case) is updated. <code class="language-plaintext highlighter-rouge">fire</code> is the function you call to in a logical-control to update your own state.</p>
<p>What is nice with this is that you can then define attack combos as little state-machines, put them in logical controls and attach them to the controller object. You can now see the state of Ryu’s Shoryuken move in the same way you would check if a button is being held down.</p>
<p>The logical-control is still very much a WIP but I’m happy with the direction it is going.</p>
<p>That’s all for input for now, I have been doing other stuff this week like making a tablet app which lets you use sliders and pads to send data to lisp [2] and of course more streaming[3] but it’s time for me to go do other stuff.</p>
<p>This weekend I am going to have a go at making a little RPG engine with all this stuff. Should be fun!</p>
<p>[0] you could of course cache event objects, but then you need <em>n</em> of each type of event object (because we may dispatch on type) and we can’t make them truly immutable.</p>
<p>[1] I made a concession that you can have callbacks as they are used internally and it hopefully means people won’t try and hack their own in on top of the system.</p>
<p>[2]
<img src="https://everyweeks.com/xI8fR7QIcw4ACYyRWfnwcF1anUF436rN-tapp.jpg" alt="tablet app" /></p>
<p>[3]</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/CdDVjRVOifM" frameborder="0"></iframe>
Swing and a miss2017-06-14T11:17:50+00:00http://www.techsnuffle.com/2017/06/14/swing-and-a-miss<p>So this last weekend I worked on PBR again as there was a wonderful <a href="https://learnopengl.com/#!PBR/IBL/Specular-IBL">new tutorial out</a>. The good news is that it cleared up a lot of points of confusion for me. The bad news is my version is still incorrect :(</p>
<p>I have been through every damn line of glsl to make sure that the PBR implementation itself matched the tutorial..which leaves the major possibility that it was something else all along; that some part of the deferred pass is incorrect.</p>
<p>It would explain a lot but also be crazy annoying.</p>
<p>The other possibility of course is that my implementation of PBR doesn’t match the tutorial but Im having an increasingly hard time believing that.</p>
<p>Other than that, streaming is going well and I am doing another one tonight, being forced to learn something well enough to explain it is good stuff.</p>
<p>Anyhoo, that’s this week, hopefully next time it will be better news :D</p>
<p><img src="https://everyweeks.com/4pm938SHSMVKdkJNeQhpROPzbzEG6B6o-pbyarr.png" alt="ugh" /></p>
Ughhh2017-06-06T10:25:39+00:00http://www.techsnuffle.com/2017/06/06/ughhh<p>I’ve been slogging through some really boring stuff these last few days. Boring but necessary.</p>
<p>People have been asking for a stream on using <a href="https://github.com/cbaggers/varjo">Varjo</a> and so that’s what I’m doing tomorrow. However I’ve been more and more bothered by the fact I would do this stream and, within a month, it would be obsolete as knew I wanted to change how things were structured. This really meant I had to bite the bullet and get it done.</p>
<p>It’s done now but these changes affected every project I work on so naturally testing took some time. The good news is that all of these changes are in the release branches for the various projects and so will ship with the next <a href="https://www.quicklisp.org/beta/">quicklisp</a> cycle.</p>
<p>Next up was documentation for CEPL. I’ve been shipping a bunch of new goodness recently so plenty of things either needed documenting or tweaking. Also the documentation generator I use now supports markdown so I went through <em>everything</em> reformatting and editing things.. that was terrifically boring..I finished that around 4am on Sunday. You can find the result <a href="http://techsnuffle.com/cepl/api.html">here</a></p>
<p>Then I was informed by the lovely quicklisp folks that a couple of my libraries werent building..damn. I hadnt tested those in the rush of everything else. That took an hour to fix but it’s good now.</p>
<p>I also wrote the documentation for my <a href="https://github.com/cbaggers/cl-soil">cl-soil</a> wrapper library (MORE BORING!) which can be found <a href="http://techsnuffle.com/cl-soil/api.html">here</a></p>
<p>Making good product is hard man, I had a 3 day weekend and feel tired and a bit grumpy (oh poor me <code class="language-plaintext highlighter-rouge"><tiny violin></code> :p) but at least tomorrows stream will make sense.</p>
<p>…</p>
<p>Except.</p>
<p>I may have worked out how to have <code class="language-plaintext highlighter-rouge">&rest</code> arguments (varargs in other languages) in <a href="https://github.com/cbaggers/varjo#name-origins">Vari</a> so I may try and hack that in tonight..which means more testing :D I’m a glutton for punishment.</p>
<p>There were also some other fixes this week but not interesting enough to be worth writing about now.</p>
<p>Until next week, Ciao.</p>
2 days late again2017-06-01T11:12:49+00:00http://www.techsnuffle.com/2017/06/01/2-days-late-again<p>But there is news at least!</p>
<h3 id="varjo">Varjo</h3>
<p>I changed varjo’s <code class="language-plaintext highlighter-rouge">if</code> form so that if certain conditions are right it will emit a ternary operator instead of a full <code class="language-plaintext highlighter-rouge">if</code> statement in the glsl</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(glsl-code
(translate
(make-stage :vertex '((a :int)) nil '(:450)
'((let ((x (if (< a 10)
a
(- a))))
(v! 1 2 3 4))))))
"// vertex-stage
#version 450
layout(location = 0) in int A;
void main()
{
int X = (A < 10) ? A : (- A);
gl_Position = vec4(float(1),float(2),float(3),float(4));
return;
}
"
</code></pre></div></div>
<p>Aside from this I fixed a few bugs and made little tweaks such as; if you use <code class="language-plaintext highlighter-rouge">-></code> in a name in lisp it becomes <code class="language-plaintext highlighter-rouge">_to_</code> in glsl. A tiny touch but it makes the generated code nicer to read.</p>
<h3 id="nineveh">Nineveh</h3>
<p>This last week I added color space conversion gpu-functions and also functions for generating mesh data for common primitives (sphere, box, cone, etc)</p>
<h3 id="streaming">Streaming</h3>
<p>Here are my last 2 streams. I’m getting more used to this each time and it’s lovely seeing a few regulars in the chat, this week was especially lovely as there were lots of questions. Not much else to say on this other than I’m gonna try and keep up the ‘stream every Wednesday’ goal.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/i66wYiG_5bk" frameborder="0"></iframe>
<iframe width="560" height="315" src="https://www.youtube.com/embed/de_lwAnD9HE" frameborder="0"></iframe>
<p>Ciao</p>
Streaming!2017-05-24T18:53:35+00:00http://www.techsnuffle.com/2017/05/24/streaming<p>This’ll be quick post, both because I’m a day late posting this and also because I’m streaming in an hour.</p>
<p>Staying on that subject, I streamed a thing! I’m pretty stoked as it seemed to go well (despite nerves) and the response has been great. You can find the recording below.</p>
<p>Other than that I’ve been doing some research for future streams, fixing bugs in Varjo and adding some features to various libraries. I have a day off tomorrow so I’m thinking of hammering on with color conversion functions and also with mesh generation.</p>
<p>I’ve also been putting some time into porting <a href="https://github.com/vydd/sketch">sketch</a> to CEPL but I’m still struggling with performance. The original had minimal state changes (1 program, 1 vao) and used buffer streaming to push tonnes of verts every frame.. however it makes >4000 draw calls a frame. This use case runs into some rather slow parts of CEPL so I’ll need to do work there. More on this next week</p>
<p>Right, time to go.</p>
<p>Ciao</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/82o5NeyZtvw" frameborder="0"></iframe>
It's been a good week2017-05-16T22:42:09+00:00http://www.techsnuffle.com/2017/05/16/its-been-a-good-week<p><a href="https://github.com/quicklisp/quicklisp-projects/issues/1292">Some</a> <a href="https://github.com/quicklisp/quicklisp-projects/issues/1291">other</a> <a href="https://github.com/quicklisp/quicklisp-projects/issues/1290">projects</a> have been accepted into the lisp package manager.</p>
<p>I put out 2 new videos:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/soqNpHL3HEM" frameborder="0"></iframe>
<iframe width="560" height="315" src="https://www.youtube.com/embed/WvYNiEMWO0Y" frameborder="0"></iframe>
<p>I have added graphing functions to <a href="https://github.com/cbaggers/nineveh">Nineveh</a> my ‘standard library’ of gpu functions for CEPL</p>
<p><img src="http://techsnuffle.com/assets/images/graphing0.png" alt="graphs" /></p>
<p>And I’ll be streaming <a href="https://www.twitch.tv/baggers___">here</a> this Wednesday 18:00 UTC. The plan is to learn some graphic programming stuff by using lisp to take things apart on the stream. This week I’m going to start on the road to understanding noise functions by taking apart some gpu hashing functions. I’m a bit nervous but if it goes well I’m gonna try doing these weekly.</p>
<p>That’s all for now. Seeya.</p>
The stocking of Nineveh2017-05-09T09:17:59+00:00http://www.techsnuffle.com/2017/05/09/the-stocking-of-nineveh<p>What a week. I’m a bit exhausted right now as I’ve been pushing a bit, but the results are fairly pleasing to me.</p>
<h3 id="glsl-quality">GLSL Quality</h3>
<p>First off I did a bunch of work in making the generated GLSL a bit nice to read. The most obvious issue was that, when I returned multiple values I would created a bunch of temporary variables. The code was all technically correct but it didn’t read well. So now like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g foo ((x :float))
(let ((sq (* x x)))
(values (v2! sq)
(v3! (* sq x)))))
</code></pre></div></div>
<p>can compile to glsl like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vec2 FOO(float X, out vec3 return_1)
{
float SQ = (X * X);
vec2 g_G1075 = vec2(SQ);
return_1 = vec3((SQ * X));
return g_G1075;
}
</code></pre></div></div>
<p>Which is fairly readable. 1 temporary variable is still used, but that is to maintain the order of evaluation that was expected.</p>
<p><code class="language-plaintext highlighter-rouge">return</code> is one of areas of the compiler with the most magic as we generate different code based on the context the code is being compiled in. It could be a return from a regular function (like above), or it could be from a stage in which case we need to turn it into <code class="language-plaintext highlighter-rouge">out</code> interface block assignments etc. This meant it took a bunch of testing to get it correct.</p>
<h3 id="compiler-cleanup">Compiler cleanup</h3>
<p>For a while I’ve been trying to clean up the code around how multiple return values are handled. I had a few false starts here (which ended in force-pushing away a day’s worth of work) but it’s done now. They are still plenty of scars in the codebase but the healing can at least begin :p</p>
<h3 id="bugs-in-release">Bugs in release</h3>
<p>I then found a bug & a regression in the code I had prepped for the next release (into the common lisp package manager). As the releases are not made on a fixed cadence I wasn’t sure how long I had so I dived into that. It turned out one of those cases where even good type checking wouldn’t have helped. I had removed some code that looked redundant but was actually resolving the aliased name of a type.</p>
<h3 id="compile-in-every-direction">Compile in every direction</h3>
<p>With that done I had sat down for some well earned farting around. Last week I had knocked up a simple <a href="https://www.shadertoy.com/">shadertoy</a> like thing[0] so I went back to working through that.</p>
<p>I got to this part where it was recommending going and implementing various curve functions from folks like <a href="http://www.iquilezles.org/">IQ</a> & <a href="http://www.flong.com/">Golan Levin</a>. So dutifully I started.. but I was getting this nagging feeling that I should have more noise functions. So after porting some functions from the chaps above I found [this article] (https://briansharpe.wordpress.com/2011/10/01/gpu-texture-free-noise/) on texture-free noise.</p>
<p>Needless to say I got hooked on that, and then finding out out he had a whole <a href="https://briansharpe.wordpress.com/2011/10/01/gpu-texture-free-noise/">library</a> I decided that I needed it, all of it, in lisp :)</p>
<p>So that was the weekend. I polished up my naive GLSL -> Lisp translator and got cracking porting. Having this much code pouring in was a really good test-case for the compiler and so I got to clean up a few more bugs in the process.</p>
<p>One interesting addition to the compiler was to allow floating point numbers as strings in the lisp code. It looks like this: <code class="language-plaintext highlighter-rouge">(+ (v4! 1.5) (v4! "1.4142135623730950488016887242097"))</code> the reason for this is that some floating point numbers used in shaders are used for their exact bit-pattern rather than strictly for the value itself. This optional string syntax avoid the risk of the host lisp using a different floating point representation a messing up the value in some way.</p>
<p>The first pass of the import is done. Next is cleanup and documentation but that is less critical. My standard library <a href="https://github.com/cbaggers/nineveh/tree/master/noise">Nineveh</a> is looking slightly better stocked now :)</p>
<h3 id="validation">Validation</h3>
<p>One thing that felt great was noticing that, in the naive GLSL -> Lisp translator and got cracking porting. Having this much code pouring in was a really GLSL noise library, there were cases where it would be nice to switch out the hashing function used in a noise function. In the GLSL it was done with comments but I realized this was a <em>PERFECT</em> place to use first class functions. So here it is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g ctoy-quad-frag ((uv :vec2)
&uniform (now :float)
(mouse :vec2)
(mouse-norm :vec2)
(mouse-buttons :vec2)
(screen-res :vec2))
;; here is the hash function we pass in ↓↓↓↓↓↓
(v3! (+ 0.4 (* (perlin-noise #'sgim-qpp-hash-2-per-corner
(* uv 5))
0.5))))
</code></pre></div></div>
<p>In that code we call <code class="language-plaintext highlighter-rouge">perlin-noise</code> passing in the hashing function we would like it to use. And here is the generated GLSL.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// vertex-stage
#version 450
uniform float NOW;
uniform vec2 MOUSE;
uniform vec2 MOUSE_NORM;
uniform vec2 MOUSE_BUTTONS;
uniform vec2 SCREEN_RES;
vec3 CTOY_QUAD_FRAG(vec2 UV);
float PERLIN_NOISE(vec2 P);
vec2 PERLIN_QUINTIC(vec2 X2);
vec4 SGIM_QPP_HASH_2_PER_CORNER(vec2 GRID_CELL, out vec4 return_1);
vec4 QPP_RESOLVE(vec4 X1);
vec4 QPP_PERMUTE(vec4 X0);
vec4 QPP_COORD_PREPARE(vec4 X);
vec4 QPP_COORD_PREPARE(vec4 X)
{
return (X - (floor((X * (1.0f / 289.0f))) * 289.0f));
}
vec4 QPP_PERMUTE(vec4 X0)
{
return (fract((X0 * (((34.0f / 289.0f) * X0) + vec4((1.0f / 289.0f))))) * 289.0f);
}
vec4 QPP_RESOLVE(vec4 X1)
{
return fract((X1 * (7.0f / 288.0f)));
}
vec4 SGIM_QPP_HASH_2_PER_CORNER(vec2 GRID_CELL, out vec4 return_1)
{
vec4 HASH_0;
vec4 HASH_1;
vec4 HASH_COORD = QPP_COORD_PREPARE(vec4(GRID_CELL.xy,(GRID_CELL.xy + vec2(1.0f))));
HASH_0 = QPP_PERMUTE((QPP_PERMUTE(HASH_COORD.xzxz) + HASH_COORD.yyww));
HASH_1 = QPP_RESOLVE(QPP_PERMUTE(HASH_0));
HASH_0 = QPP_RESOLVE(HASH_0);
vec4 g_G5388 = HASH_0;
return_1 = HASH_1;
return g_G5388;
}
vec2 PERLIN_QUINTIC(vec2 X2)
{
return (X2 * (X2 * (X2 * ((X2 * ((X2 * 6.0f) - vec2(15.0f))) + vec2(10.0f)))));
}
float PERLIN_NOISE(vec2 P)
{
vec2 PI = floor(P);
vec4 PF_PFMIN1 = (P.xyxy - vec4(PI,(PI + vec2(1.0f))));
vec4 MVB_0;
vec4 MVB_1;
//
// Here is that hashing function in the GLSL
// ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
MVB_0 = SGIM_QPP_HASH_2_PER_CORNER(PI,MVB_1);
vec4 GRAD_X = (MVB_0 - vec4(0.49999));
vec4 GRAD_Y = (MVB_1 - vec4(0.49999));
vec4 GRAD_RESULTS = (inversesqrt(((GRAD_X * GRAD_X) + (GRAD_Y * GRAD_Y))) * ((GRAD_X * PF_PFMIN1.xzxz) + (GRAD_Y * PF_PFMIN1.yyww)));
GRAD_RESULTS *= vec4(1.4142135623730950488016887242097);
vec2 BLEND = PERLIN_QUINTIC(PF_PFMIN1.xy);
vec4 BLEND2 = vec4(BLEND,(vec2(1.0f) - BLEND));
return dot(GRAD_RESULTS,(BLEND2.zxzx * BLEND2.wwyy));
}
vec3 CTOY_QUAD_FRAG(vec2 UV)
{
vec4(MOUSE_NORM.x,MOUSE_BUTTONS.x,MOUSE_NORM.y,float(1));
return vec3((0.4f + (PERLIN_NOISE((UV * float(5))) * 0.5f)));
}
void main()
{
CTOY_QUAD_FRAG(<dummy v-vec2>);
gl_Position = vec4(float(1),float(2),float(3),float(4));
return;
}
</code></pre></div></div>
<p>I’m stoked at how natural the function call looks in the resulting code. Also it’s extra func that the function passed in has 2 return values and everything just worked :)</p>
<p>For completeness here are a couple of the other functions involved</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g perlin-noise ((hash-func (function (:vec2) (:vec4 :vec4)))
(p :vec2))
;; looks much better than revised noise in 2D, and with an efficent hash
;; function runs at about the same speed.
;;
;; requires 2 random numbers per point.
(let* ((pi (floor p))
(pf-pfmin1 (- (s~ p :xyxy) (v! pi (+ pi (v2! 1.0))))))
(multiple-value-bind (hash-x hash-y) (funcall hash-func pi)
(let* ((grad-x (- hash-x (v4! "0.49999")))
(grad-y (- hash-y (v4! "0.49999")))
(grad-results
(* (inversesqrt (+ (* grad-x grad-x) (* grad-y grad-y)))
(+ (* grad-x (s~ pf-pfmin1 :xzxz))
(* grad-y (s~ pf-pfmin1 :yyww))))))
(multf grad-results (v4! "1.4142135623730950488016887242097"))
(let* ((blend (perlin-quintic (s~ pf-pfmin1 :xy)))
(blend2 (v! blend (- (v2! 1.0) blend))))
(dot grad-results (* (s~ blend2 :zxzx) (s~ blend2 :wwyy))))))))
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g sgim-qpp-hash-2-per-corner ((grid-cell :vec2))
(let (((hash-0 :vec4)) ((hash-1 :vec4)))
(let* ((hash-coord
(qpp-coord-prepare
(v! (s~ grid-cell :xy) (+ (s~ grid-cell :xy) (v2! 1.0))))))
(setf hash-0
(qpp-permute
(+ (qpp-permute (s~ hash-coord :xzxz)) (s~ hash-coord :yyww))))
(setf hash-1 (qpp-resolve (qpp-permute hash-0)))
(setf hash-0 (qpp-resolve hash-0)))
(values hash-0 hash-1)))
</code></pre></div></div>
<p>All credit of implementation goes to Brian Sharpe for this excellent noise library</p>
<h3 id="peace">Peace</h3>
<p>OK I’ll stop now and spare you more details. I’m just so happy to see this behaving.</p>
<p>Have a great week.</p>
<p>Ciao</p>
<p>[0] <a href="https://github.com/cbaggers/fraggle/blob/master/main.lisp">https://github.com/cbaggers/fraggle/blob/master/main.lisp</a></p>
Trials and Tessellations2017-05-02T08:42:17+00:00http://www.techsnuffle.com/2017/05/02/trials-and-tessellations<p>Tessellation works! I was <a href="https://github.com/cbaggers/cepl.examples/blob/master/examples/tessellation.lisp#L37">able to replicate</a> the fantastic tutorial from <a href="http://prideout.net/blog/?p=48">The Little Grasshopper</a> and finally get a test of a pipeline with all 5 programmable stages.</p>
<p>This took a little longer than expected as my GPU complained on the following shader</p>
<blockquote>
<p>Error compiling tess-control-shader:
0(6) : error C5227: Storage block <em>FROM_VERTEX_STAGE</em> requires an instance for this profile
0(13) : error C5227: Storage block <em>FROM_TESSELLATION_CONTROL_STAGE</em> requires an instance for this profile</p>
</blockquote>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#version 450
in _FROM_VERTEX_STAGE_
{
in vec3[3] _VERTEX_STAGE_OUT_1;
};
layout (vertices = 3) out;
out _FROM_TESSELLATION_CONTROL_STAGE_
{
out vec3[] _TESSELLATION_CONTROL_STAGE_OUT_0;
};
void main() {
vec3 g_G1327 = _VERTEX_STAGE_OUT_1[gl_InvocationID];
vec3 g_GEXPR01328 = g_G1327;
_TESSELLATION_CONTROL_STAGE_OUT_0[gl_InvocationID] = g_GEXPR01328;
float TESS_LEVEL_INNER = 5.0f;
float TESS_LEVEL_OUTER = 5.0f;
if ((gl_InvocationID == 0))
{
gl_TessLevelInner[0] = TESS_LEVEL_INNER;
gl_TessLevelOuter[0] = TESS_LEVEL_OUTER;
gl_TessLevelOuter[1] = TESS_LEVEL_OUTER;
gl_TessLevelOuter[2] = TESS_LEVEL_OUTER;
}
}
</code></pre></div></div>
<p>As far as I can tell, the above code is valid according to the spec, but my driver wasn’t having any of it[0]. This meant I had to do some moderate code re-jiggling. You can’t just add an <code class="language-plaintext highlighter-rouge">instance</code> name to the blocks as then you get:</p>
<blockquote>
<p>Error Linking Program
0(13) : error C7544: OpenGL requires tessellation control outputs to be arrays</p>
</blockquote>
<p>Grrr. This makes total sense according to the spec though, so grr goes back to error <code class="language-plaintext highlighter-rouge">C5227</code>. FUCK YOU <code class="language-plaintext highlighter-rouge">C5227</code>.</p>
<p>Anyway that got fixed and this happened</p>
<p><img src="/assets/images/tess.png" alt="yay" /></p>
<p>With that done I made sure the same worked using CEPL’s <a href="https://github.com/cbaggers/cepl.examples/blob/master/examples/tessellation-inline-glsl.lisp#L38">inline glsl stages</a> and tested a bunch.</p>
<p>This left me at an odd point. CEPL has a lot less gaps in the API that it used to[1], it feels kind of ready for me to use.</p>
<p>So I sat down with <a href="https://thebookofshaders.com/">The Book of Shaders</a> and got started. I simply can’t give that project enough praise, it’s wonderfully written and remains interesting whilst also pushing you just enough to really digest the details of what they want you to learn. I made a little <a href="https://www.shadertoy.com/">shadertoy</a> substitute[2] and started doing the exercises. I have to say I’m pretty proud with how fun it all felt, it’s been a lot of work to get something that is lispy but doesn’t feel hampered by being the non-native language, so those hours of play were very vindicating.</p>
<p>Within a very short time <a href="https://thebookofshaders.com/">The Book of Shaders</a> is getting you to amass a collection of helper functions for your shaders. I already have a (very wip) project called <a href="https://github.com/cbaggers/nineveh">Nineveh</a> for this purpose, so it’s time to start working on that again! I want a kind of ‘standard library’ for CEPL, somewhere you can find a pile of implementations of common functions (e.g. noise, color conversion, etc) to either use to at least use as a starting point.</p>
<p>There are often cases where libraries like this fall short as you want a slight variation on a shader. Say for example you like their <a href="http://www.iquilezles.org/www/articles/warp/warp.htm">FBM</a> function but want one more octave of noise. I’m hoping macros can help a little here as we can provide something that generates the variant you need.</p>
<p>This is likely naive but it’ll be fun to try.</p>
<p>Right, that’s my lot for now. Seeya!</p>
<hr />
<p>[0] I was meant to be using the compatibility profile too so I’m not sure what was up</p>
<p>[1] See <a href="http://techsnuffle.com/2017/04/26/checking-in">last weeks post</a> for what remains</p>
<p>[2] There’s about 50 lines not shown here which handles input & screen events.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(in-package :ctoy)
(defun-g ctoy-quad-vert ((pos :vec2))
(values (v! pos 0 1)
(* (+ pos (v2! 1f0)) 0.5)))
(defun-g ctoy-quad-frag ((uv :vec2)
&uniform (now :float)
(mouse :vec2) (mouse-norm :vec2) (mouse-buttons :vec2)
(screen-res :vec2))
(v! (/ (s~ gl-frag-coord :xy) screen-res) 0 1))
(def-g-> draw-ctoy ()
:vertex (ctoy-quad-vert :vec2)
:fragment (ctoy-quad-frag :vec2))
(defun step-ctoy ()
(as-frame
(map-g #'draw-ctoy (get-quad-stream-v2)
:now (* (now) 0.001)
:mouse (get-mouse)
:mouse-norm (get-mouse-norm)
:mouse-buttons (get-mouse-buttons)
:screen-res (viewport-resolution (current-viewport)))))
(def-simple-main-loop ctoy
(step-ctoy))
</code></pre></div></div>
Checking in2017-04-26T10:55:51+00:00http://www.techsnuffle.com/2017/04/26/checking-in<p>So this week I don’t have much technical to write about, the main things I’ve been doing are</p>
<ul>
<li>Bug-fixing/refactoring</li>
<li>Getting everything into master and then finally</li>
<li>Getting the next release ready.</li>
</ul>
<p>This month’s release will bring a huge slew of changes into the compiler, starting with all the geometry shader work I’ve been doing. I’ve also landed the new backwards compatible host API for CEPL which brings multiple window support among other things.</p>
<p>In the next 2 weeks I want to have finished my latest cleanup of Varjo and added support for tessellation stages.</p>
<p>After this I’ve reached an interesting place with regards to CEPL as I think the major features will be done. I’ve still got a few things I want support for before I call it v 1.0:</p>
<ul>
<li>Scissor</li>
<li>Stenciling</li>
<li>Instanced Arrays</li>
<li>Multiple Contexts</li>
</ul>
<p>And each of those seem like fairly contained pieces of work. Even though there are things in GL I haven’t considered supporting yet (like transform feedback) I don’t think I’ve ever felt this close to CEPL being ‘ready’.. It’s an odd feeling.</p>
<p>After that I’d like to focus of stability, performance and on <a href="https://github.com/cbaggers/nineveh">Nineveh</a> which is going to be my ‘standard library’ of GPU functions and graphics related stuff.</p>
<p>Until next week,</p>
<p>Peace</p>
Geometry Shader Shenanigans2017-04-18T11:59:50+00:00http://www.techsnuffle.com/2017/04/18/geometry-shader-shenanigans<p>When I gave that talk a few weeks back I said, rather naively in hindsight, that geometry & tessellation shaders should work.. man was I wrong there. It turned out to be rather fiddly to find a balance that felt lispy, worked with my current analogies and worked across all GLSL version (or at least failed gracefully).</p>
<p>Lets start at the beginning.</p>
<h3 id="passing-values-between-glsl-stages">Passing values between GLSL stages</h3>
<p>To pass values from stage to stage in GLSL starts simply, you declare something as <code class="language-plaintext highlighter-rouge">out</code> in one stage and <code class="language-plaintext highlighter-rouge">in</code> in the next e.g:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out vec4 foo; // in the vertex shader
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>in vec4 foo; // in the fragment shader
</code></pre></div></div>
<p>Nice and simple. It also works for arrays of values.</p>
<p>Next we add a geometry (or tessellation) shader. Now we are working with primtives (or patches) so the <code class="language-plaintext highlighter-rouge">out</code>s from the last stage become an array of <code class="language-plaintext highlighter-rouge">in</code>s in the geometry stage.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out vec4 foo; // in the vertex shader
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>in vec4[2] foo; // in the geometry shader
</code></pre></div></div>
<p>The length of the array is dictate by the size of the primitive, so <code class="language-plaintext highlighter-rouge">lines</code> are length 2, <code class="language-plaintext highlighter-rouge">triangles</code> are length 3, etc.</p>
<p>But how about if the <code class="language-plaintext highlighter-rouge">out</code> was an array?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out vec4[10] foo; // in the vertex shader
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>in vec4[2][10] foo; // in the geometry shader
</code></pre></div></div>
<p>Simple right? I thought so and I updated my compiler to work with this. I kept having this nagging feeling though, something about interface blocks. Turns out I should have listened to the feeling sooner. Support for arrays of arrays only arrive in GLSL in v4.3 and before that you needed to use an interface block:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out VertexOuts
{
vec4[10] foo;
}
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out VertexOuts
{
vec4[10] foo;
} gs_in[2];
</code></pre></div></div>
<p>Which seems ok, but it has subtleties to it that make this hard to abstract for all GLSL versions. Lets have a look at the lisp.</p>
<p>First a vertex shader:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g test-vert ((position :vec4) (uv :vec2))
(values position normal))
</code></pre></div></div>
<p>the outputs from the above are a <code class="language-plaintext highlighter-rouge">vec4</code> which is used as the <code class="language-plaintext highlighter-rouge">gl-position</code> and a <code class="language-plaintext highlighter-rouge">vec2</code> which is passed to the next stage. Here is a valid fragment shader to go with this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g test-frag ((uv :vec2) &uniform (tex :sampler-2d))
(texture tex uv))
</code></pre></div></div>
<p>So far, so simple. Next let’s look at a geometry shader that would match that vertex shader’s outputs. We will assume we are rendering triangles.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g test-geom ((uv (:vec2 3)))
...)
</code></pre></div></div>
<p>Makes sense right? We just array the inputs. However we know that our interface block is going to cause problems. In this case <code class="language-plaintext highlighter-rouge">uv</code> isn’t really <code class="language-plaintext highlighter-rouge">vec2[3]</code> it’s <code class="language-plaintext highlighter-rouge">in blockName { vec2 }gs_in[3]</code>. We have a mismatch between the abstraction and reality. However it’s a useful lie, it takes the GLSL behavior and makes things more consistent and thus easy to understand, so if we can keep it I’d prefer to.</p>
<p>The answer (at least for now) is to make an ‘ephemeral’ type, this is a type that can’t exists in GLSL for real, but one that our compiler can handle.</p>
<p>So this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g test-geom ((uv (:vec2 3)))
(let ((a uv)
(index 1))
(aref uv index)
..))
</code></pre></div></div>
<p>becomes something like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>..
in IN_BLOCK
{
vec2 uv;
} gs_in[3]
void main()
{
int index = 1;
gs_in[index].uv;
..
}
</code></pre></div></div>
<p>Notice that <code class="language-plaintext highlighter-rouge">int index = 1</code> ended up in the GLSL but there is no <code class="language-plaintext highlighter-rouge">a = <something></code>, that is because of that ephemeral type. The reason that it has to be ephemeral is that until we use <code class="language-plaintext highlighter-rouge">aref</code> we can’t access the <code class="language-plaintext highlighter-rouge">uv</code> slot inside the block. Also you cant write <code class="language-plaintext highlighter-rouge">IN_BLOCK tmp = gs_in[1]</code> or <code class="language-plaintext highlighter-rouge">IN_BLOCK[3] tmp = gs_in</code> as whilst interface blocks may look like structs, but they dont’ behave like them and those two example as simply illegal in GLSL. It’s a damn shame really as otherwise this would be much easier!</p>
<p>One other option we could have gone for instead of this ephemeral array business we could invent a ‘primitive’ type. So the geom shader could be:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g test-geom ((uv triangle))
(let ((a uv)
(index 1))
(aref uv index)
..))
</code></pre></div></div>
<p>But then <code class="language-plaintext highlighter-rouge">test-geom</code> is no longer a stand alone function, we would need to wait until <code class="language-plaintext highlighter-rouge">test-geom</code> was used in a pipeline to know what slots <code class="language-plaintext highlighter-rouge">triangle</code> contained. One of the beauties of CEPL is that all gpu functions are simply functions until they are used in a pipeline, they can be used as stages or just be called from other gpu functions. If we use <code class="language-plaintext highlighter-rouge">triangle</code> then we can’t compile this gpu function ahead of time, which would suck as that feature currently lets you find and fix bugs faster. On top of this <code class="language-plaintext highlighter-rouge">triangle</code> would also have to be ephemeral as there is no <code class="language-plaintext highlighter-rouge">triangle</code> type in GLSL</p>
<p>The same also goes for requiring the user to define the interface between vertex & geom shaders as structs. You still need the ephemeral & you require the user to repeat themselves.</p>
<p>This is one of those cases where every option is kinda sucky but the first one described feels the least sucky, and that is the one I have gone for.</p>
<h3 id="implement-it">Implement it</h3>
<p>The above took a few days of experiments and pain to get ironed out, and then it needed to be implemented. As usual working in areas of the compiler that haven’t been touched for a while uncovered bugs and general weaknesses that needed fixes.</p>
<p>Another thing that was playing a lot on my mind was that one of the features CEPL has is support for is defining a stage as raw GLSL and using it in a pipeline, so whatever I made needed to not break that. Having users is awesome and I really want to make sure I don’t fuck up their workflow if I can help it. For example one person is already using the GLSL stage support in CEPL to use Tessellation shaders, which is something I’ve never used! The fact that they can do this makes me happy and eases the pain whilst I slowly find nice lispy ways of doing the same.</p>
<p>Oh, I almost forgot..</p>
<h3 id="primitives">Primitives!</h3>
<p>As we are in the world of Geometry & Tessellation we need to talk about primitives. One of the things I love about Varjo is that it can pass information from stage to stage so the user doesn’t have to write the same stuff multiple times, we should be able to do the same with primitives. I wanted to be able to take the OpenGL draw-mode and pass it through the compilation. This allows Varjo to not only write some of the GLSL automatically (e.g. <code class="language-plaintext highlighter-rouge">layout(triangles) in;</code> in the Geometry stage) but also to know the length of the array’d interface block going into the geometry stage. Having little details like this checked is worth my time as then Varjo can give much nicer and more targeted errors that GL can.</p>
<p>I’ll skip the details on this for now as this post is already getting long.</p>
<h3 id="where-are-we-now">Where are we now?</h3>
<p>We are here</p>
<p><img src="https://cloud.githubusercontent.com/assets/1579326/25100902/4d6c89f8-23b2-11e7-971b-50aafd92e89e.png" alt="norms" /></p>
<p>I finally got these damn things working. Here we are using a geometry shader to draw the normals of the sphere.</p>
<p>There is still a bunch of stuff to do and there are some aspects that really pushed the limits of how ‘lispy’ stuff could be made. I’ll save that for another post. I’ll simply say thanks for reading and leave you with the pipeline for drawing the lines in the above.</p>
<p>Ciao</p>
<hr />
<p>This lisp code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g normals-vert ((vert g-pnt) &uniform (model->clip :mat4))
(values (* model->clip (v! (pos vert) 1))
(s~ (* model->clip (v! (norm vert) 0)) :xyz)))
(defun-g normals-geom ((normals (:vec3 3)))
(declare (varjo:output-primitive :kind :line-strip :max-vertices 6))
(labels ((gen-line ((index :int))
(let ((magnitude 0.2))
(setf gl-position (gl-position (aref gl-in index)))
(emit-vertex)
(setf gl-position
(+ (gl-position (aref gl-in index))
(* (v! (aref normals index) 0f0)
magnitude)))
(emit-vertex)
(end-primitive)
(values))))
(gen-line 0)
(gen-line 1)
(gen-line 2)
(values)))
(defun-g normals-frag ()
(v! 1 1 0 1))
(def-g-> draw-normals ()
:vertex (normals-vert g-pnt)
:geometry (normals-geom (:vec3 3))
:fragment (normals-frag))
</code></pre></div></div>
<p>makes this glsl:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>("#version 450
in vec3 fk_vert_position;
in vec3 fk_vert_normal;
in vec2 fk_vert_texture;
out _FROM_VERTEX_
{
out vec3 _VERTEX_OUT_1;
};
uniform mat4 MODEL_62CLIP;
void main() {
vec3 return1;
vec4 g_G1550 = (MODEL_62CLIP * vec4(fk_vert_position,float(1)));
return1 = (MODEL_62CLIP * vec4(fk_vert_normal,float(0))).xyz;
gl_Position = g_G1550;
vec3 g_G1552 = return1;
_VERTEX_OUT_1 = g_G1552;
return;
}
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#version 450
layout (triangles) in;
in _FROM_VERTEX_
{
in vec3 _VERTEX_OUT_1;
} inputs[3];
layout (line_strip, max_vertices = 6) out;
void GEN_LINE(int INDEX);
void GEN_LINE(int INDEX) {
float MAGNITUDE = 0.2f;
gl_Position = gl_in[INDEX].gl_Position;
EmitVertex();
gl_Position = (gl_in[INDEX].gl_Position + (vec4(inputs[INDEX]._VERTEX_OUT_1,0.0f) * MAGNITUDE));
EmitVertex();
EndPrimitive();
}
void main() {
GEN_LINE(0);
GEN_LINE(1);
GEN_LINE(2);
}
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#version 450
layout(location = 0) out vec4 _FRAGMENT_OUT_0;
void main() {
vec4 g_G1553 = vec4(float(1),float(1),float(0),float(1));
_FRAGMENT_OUT_0 = g_G1553;
return;
}
</code></pre></div></div>
Lisping Elsewhere2017-04-07T12:53:13+00:00http://www.techsnuffle.com/2017/04/07/lisping-elsewhere<p>From Sunday to Wednesday I was away on Brussels at the European Lisp Symposium. It’s an opportunity to see good talks but more importantly to me it’s a catch up with some folks I haven’t seen in a year and nerd out.</p>
<p>Aside from that I’ve been cracking away at getting <a href="https://github.com/vydd/sketch">sketch</a> working using CEPL under the hood. It’s rendering using lisp shaders now and live recompile works great. There has been a slight performance drop as <code class="language-plaintext highlighter-rouge">sketch</code> used <a href="http://onrendering.blogspot.no/2011/10/buffer-object-streaming-in-opengl.html">buffer object streaming</a> to upload the vertices and I’m not doing that yet. In fact CEPL doesn’t expose <code class="language-plaintext highlighter-rouge">glMapBufferRange</code> at all so first order of business is fixing that. After that I want to make a <code class="language-plaintext highlighter-rouge">vertex-ring</code> it’s going to be an object that encapsulates the buffer object streaming pattern and will make using it seamless.</p>
<p>I have also been working on fixes to <a href="https://github.com/cbaggers/varjo">Varjo</a> that will finally get Geometry shaders working. The task for the last few days has been rewriting how Varjo handles <code class="language-plaintext highlighter-rouge">return</code> & making Varjo use <a href="https://www.khronos.org/opengl/wiki/Interface_Block_(GLSL)">interface blocks</a> for passing data between stages. The reason for working on <code class="language-plaintext highlighter-rouge">return</code> (apart from it being buggy) was that in geometry shaders your primary task is to ‘emit’ extra geometry rather than ‘returning’ transformed data via the <code class="language-plaintext highlighter-rouge">out</code> vars.</p>
<p>I’m hoping that this weekend I can get those two things coded and tested. After that I need to look at the Geometry shaders themselves, I know there are some ugly details around <code class="language-plaintext highlighter-rouge">in</code>s & <code class="language-plaintext highlighter-rouge">out</code>s so wish me luck.</p>
Tired but happy2017-03-28T09:52:45+00:00http://www.techsnuffle.com/2017/03/28/tired-but-happy<p>I’m tired right now but also really happy with the reception the video got.</p>
<p>I’ve had a number of PRs to CEPL & Varjo with fixes for various things. Of special note is a chap called <code class="language-plaintext highlighter-rouge">djeis97</code> who has been given me a pile of help with issues in my geometry shader generation. He’s been using the inline glsl features of CEPL too which is awesome.</p>
<p>In order for geometry shaders to work properly I need to do the following:</p>
<ul>
<li>Improve CEPL’s handling of arrays</li>
<li>Generate interface blocks between shader stages</li>
<li>Fix some hacks around <code class="language-plaintext highlighter-rouge">return</code> & <code class="language-plaintext highlighter-rouge">out</code> values</li>
<li>Add some helpers for emitting vertices.</li>
</ul>
<p>I’ve made some progress on the arrays in the last week and will be starting on interface blocks asap.</p>
<p>I’ve also had one chap who works on a project called <a href="https://github.com/vydd/sketch">sketch</a> decide that he has had enough with dealing with opengl directly and wants to build his project on top of CEPL. This is <em>exactly</em> the kind of project I’ve wanted to see built with CEPL, stuff that has a clear set of trade-offs and makes making stuff super simple. His current approach supports multiple windows however, and CEPL did not, so I have been getting that in place. This has meant (even though Im not implementing yet) looking at multiple GL contexts and threading.</p>
<p>CEPL has no business with your threads and expects you to use them wisely, <code class="language-plaintext highlighter-rouge">sketch</code> however was built on top of <a href="https://github.com/lispgames/sdl2kit">sdl2kit</a> which handles threading for you. This mismatch with CEPL is a little problematic and, as I don’t yet have any libraries for helping with the threading situation, I’m going to make a shim to make sdl2kit drive CEPL. It’ll be butt ugly but will hopefully be enough for now.</p>
<p>It’s the European Lisp Symposium in Brussels next week so it would be awesome if I had something new to give a lightning talk on..or even just to show of with :)</p>
<p>At some point I need sleep too.</p>
<p>Ciao!</p>
Traveling Again2017-03-23T11:29:50+00:00http://www.techsnuffle.com/2017/03/23/traveling-again<p>Turns out I booked a bunch of small trips last year and they all are around now. Sorry for the late update.</p>
<p>I got back to coding yesterday and have been working on the compiler. Since the talk I have some great support from <a href="https://github.com/djeis97">djeis97</a> and <a href="https://github.com/jobez">jobez</a> on issues in CEPL & Varjo. One big thing I realized was more broken that expected was Geometry Shaders, so that is current top priority.</p>
<p>The prerequisites for fixing this are better array support in Varjo & support for interface blocks, so whilst traveling I was reading the GLSL spec again. Last night I added checks to make sure you are not trying to access an uninitialized variable and then started on adding lisp’s <code class="language-plaintext highlighter-rouge">make-array</code> function.</p>
<p>As I currently don’t track <code class="language-plaintext highlighter-rouge">const</code>‘ness in Varjo I can’t validate a bunch of the array rules that the GLSL requires. However those are errors that will be caught by the driver’s own glsl compiler so I’m not worrying about those too much yet.</p>
<p>That’s all for now
Peace</p>
Tweaking2017-02-26T23:48:26+00:00http://www.techsnuffle.com/2017/02/26/tweaking<p>As I wrote so late last week I haven’t got too much to add here, however I have made progress on <code class="language-plaintext highlighter-rouge">tweak</code>.</p>
<p>The goal is to be able to inspect & change data easily while the program is running. You wrap a form in <code class="language-plaintext highlighter-rouge">(tweak ..)</code> and you get a small window (rendered using gl via nuklear) which allows you too mess with the value.</p>
<p>For example given this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defparameter *some-val* 10)
</code></pre></div></div>
<p>You can write this in the main body code</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(tweak *some-color*)
</code></pre></div></div>
<p>And you get this:</p>
<p>And if you scrub the value it immediately applies to the form you are tweaking. You can define different tweak ui’s for different lisp types so extending will be easy. I still need to add a couple of helpers for that though.</p>
<p>Other than that I made a little companion library to SDL-TTF which allows people to render text straight to CEPL textures. So this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-font (font "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf")
(text-to-tex "Yo!" font (v! 1 0 0 1)))
</code></pre></div></div>
<p>Gives you a texture with the text.</p>
<p>That’s the lot I’m afraid. Time for sleep.</p>
Late again2017-02-23T17:59:01+00:00http://www.techsnuffle.com/2017/02/23/late-again<p>I have traveled to see family this week so sorry for the late writeup.</p>
<p>This week went pretty well. I went through the <a href="https://github.com/tuccio/IBLGGX">PBR example</a> I had been using line by line making notes and then tried to match my implementation to theirs.</p>
<p>I had made some progress but still the colors looked washed out, something must have been putting the color values way out of wack. Finally I saw it, I had been adding a color I should have been multiplying. A facepalm and a tweak later color started looking like this</p>
<p><img src="https://raw.githubusercontent.com/cbaggers/lark/no-ecs/fuckin_finally.png" alt="much better" /></p>
<p>I’m still not convinced that the example’s approach for creating the final color is correct, however this is plenty good enough for now.</p>
<p>With that done I revisited my UI code (which uses nuklear) and found that it was broken…shits.</p>
<p>A bunch of git archaeology told me the issue was related to my GL context caching. Double shits</p>
<p>..and then that the crash occurred because I had made CEPL’s handling of the context more robust. At this point a drink was in order, but after that, fixing!</p>
<p>The result was that I hadn’t been unbinding VAOs correctly when rendering and so then subsequent buffer unbindings had been captured in the VAO. This sucks but was a 1 line fix and CEPL is much better for it.</p>
<p>Theres other stuff going on buts they are not worth yakking about right now.</p>
<p>Seeya</p>
Grind2017-02-13T11:04:37+00:00http://www.techsnuffle.com/2017/02/13/grind<p>As I was so late writing last week I don’t have a lot that’s new, but I need to keep the rhythm of writing these.</p>
<p>I have been focused on getting my maths library up to scratch. I did a bunch of work on getting the non-consing api cleaned up and merged. This involved fixing a whole host of optimization notes from the compiler. The compiler I use is pretty good at explaining where it is forced to ‘waste’ speed and why. The biggest single change was changing the return type of the vector <code class="language-plaintext highlighter-rouge">length</code> & <code class="language-plaintext highlighter-rouge">distance</code> functions to show that it could only return positive floats. That was important as the square root of a negative number is a complex number, so by indicating the functions could only be greater than 0 the compiler could ignore the complex number cases.</p>
<p>I also found a branch where I had started adding support for lines, rays, line-segments & axis aligned bounding boxes in ℝ3. I fixed up the line-segments & AABBs and got that merged too.</p>
<p>Finally I started the non-consing (destructive) version of the matrix apis. This is also on master.</p>
<p>Tonight I want to test the non-consing api against the consing version to make sure I haven’t introduced any bugs. I’ll then redefine the consing version using the non-consing functions to do the work. At that point I’m probably ready to stop scratching this itch and focus on graphics stuff again.</p>
<p>Peace</p>
Boring Necessities2017-02-09T21:47:47+00:00http://www.techsnuffle.com/2017/02/09/boring-necessities<p>This week (and a bit) has felt like a bit of a blur as I’ve mainly been working on boring stuff that just needed doing. I’ll be honest, I can’t remember what I did so I’ll just go spelunking through my commits and see what I find.</p>
<h1 id="lisp-games-cross-test">lisp-games-cross-test</h1>
<p>I started yet another project :p. This came out of me feeling the uncertainty you get when you have written a good chunk of that stack you use (and don’t have enough tests). It’s stupid to have written this thing which is meant to be nice for creating things and then be stymied by and lingering sense that something, somewhere must be breaking.</p>
<p>One thing I really wanted to be sure about was that my maths library <a href="https://github.com/cbaggers/rtg-math">rtg-math</a> was solid.</p>
<p>As I didn’t want to rely on my own (very limited) knowledge I decided to make a basic fuzz testing setup where I compare my library against other math libraries written in Common Lisp.</p>
<p>I built on top of the excellent <a href="https://common-lisp.net/project/fiveam/">fiveam</a> testing library and made a fairly nice system for defining comparative tests across several libraries. It takes into account that a library’s features will not exactly overlap with another. I’m happy with it so far but it’s not ready for github yet.</p>
<p>Anyhoo I set it to work and found that a lot of stuff is working just fine. One function (the matrix to quaternion function) was broken. At first I stupidly blamed the c++ I based the function on until, after many re-reads, I noticed that the <code class="language-plaintext highlighter-rouge">[]</code> operator had been overloaded to index from <code class="language-plaintext highlighter-rouge">x</code> and ignore the <code class="language-plaintext highlighter-rouge">w</code> component altogether. This meant I had a bunch of ‘off by one’ indexing errors in that function :facepalm:.</p>
<p>However I did find a typo in the book’s code which has now been <a href="https://github.com/jvanverth/essentialmath/issues/57">fixed</a>! So that’s a nice side effect of this work.</p>
<p>I then got massively mistake about the api of another library I was comparing to (sorry axion). The process of being gently schooled by psilord on the <code class="language-plaintext highlighter-rouge">#lispgames</code> irc was incredibly helpful but did send me into frequent ‘What if my entire api is wrong’ panics.</p>
<p>Luckily he provided me with some <a href="https://fgiesen.wordpress.com/2012/02/12/row-major-vs-column-major-row-vectors-vs-column-vectors/">excellent</a> <a href="https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/geometry/row-major-vs-column-major-vector">links</a> and I’m slowly getting a better grasp of the terminology.</p>
<p>Luckily, as my library is heavily based on the c++ code from the text book I learned from, my library is fine and then only problems were (and to some extent remain) with my brain :D</p>
<h1 id="rtg-math">RTG-Math</h1>
<p>Whilst working on the above I also got a friendly jab that my vector math was too slow. It turns out the ‘jab-er’ in question was comparing his destructive (mutable) api with my non-destructive one. This is kind of understandable as I didn’t have a destructive api to compare against.. but of course I fixed that <code class="language-plaintext highlighter-rouge">>:)</code></p>
<p>Below is the (very basic) test I did to check how it performs:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; First we make a list of 1000 random vector3s
;; ↓↓
RTG-MATH> (let ((to-add (loop :for i :below 1000 :collect
(v! (- (random 2f0) 1f0)
(- (random 2f0) 1f0)
(- (random 2f0) 1f0))))
(res (v! 0 0 0)))
;;
;; Here we make the list circular
;; ↓↓
(setf (cdr (last to-add)) to-add)
;;
;; Then we iterate 100 million times adding the elements of
;; the circular list into the variable 'res'
;; ↓↓
(time
(loop :for i :below 100000000 :for e :in to-add :do
(v3-n:+ res e)))
res)
Evaluation took:
0.748 seconds of real time
0.748000 seconds of total run time (0.748000 user, 0.000000 system)
100.00% CPU
2,297,194,923 processor cycles
0 bytes consed
#(-1248991.9 2382024.5 358581.84)
</code></pre></div></div>
<p>Not too shabby! <code class="language-plaintext highlighter-rouge">bytes consed</code> in lisp means ‘memory allocated’ so we see this only mutating the
vectors already created.</p>
<p>The test is pretty rough and ready however the list is long enough that it wouldn’t just fit in a cache line, and the random calculates are outside the loop as they the slowest part of my original test.</p>
<p>In the process of making these destructive versions of the API I have been going through the majority of the codebase adding type declarations to everything I could. I still have some optimization work and cleanup to do but it’s mostly giving the compiler the information it needs to do it’s job.</p>
<p>Optional typing is fucking great</p>
<h1 id="nineveh">Nineveh</h1>
<p><a href="https://github.com/cbaggers/nineveh">Nineveh</a> is intended to be a ‘standard library’ of gpu functions and helpers for CEPL. It has code that doesn’t belong in a GL abstraction but is really useful to have. Currently it’s not in quicklisp (the lisp package manager) as it’s still too small & too in flux however progress is being made.</p>
<p>The main addition this week was a <code class="language-plaintext highlighter-rouge">draw-tex</code> function which will take any texture or sampler and draw it to the screen. Simple stuff but it has a bunch of options for drawing in certain portions of the screen (top-left, bottom-right etc) scaling, rotation, drawing cubemaps in a nice ‘cross’ layout and maintaining aspect ratios while doing this.</p>
<p>Also <a href="https://www.twitch.tv/ferrisstreamsstuff">Ferris</a> helped me find some (really nasty) bugs in one of the functions in the library which was ace.</p>
<h1 id="pbr">PBR</h1>
<p>I spent yesterday evening at Ferris’ place where I got a bunch of help with some artifacts I was seeing in my code. It helps so much to have someone who knows what certain things should look like. When it comes to graphics coding I really lack that.</p>
<p>In the process of this I found the previous mentioned bug in nineveh, a bug in my compiler (<code class="language-plaintext highlighter-rouge">let</code> wasn’t throwing an error for duplicate var names) and added a way to support <code class="language-plaintext highlighter-rouge">uint</code> literals.</p>
<p>This has made me much more confident in the code, and with where the rest of artifacts I’m seeing are probably coming from. HOPEFULLY will have some new screenshots soon.</p>
<h1 id="sleep">Sleep</h1>
<p>I haven’t been getting enough, so I’m going to bed.</p>
<p>Seeya!</p>
Captain Merge'in2017-01-31T23:07:59+00:00http://www.techsnuffle.com/2017/01/31/captain-mergein<p>It’s done, finally merged. Hundreds of commits, 15 issues closed, a monster is dead.</p>
<p>The biggest thing I’m noticing right now is how slow the evening goes when you aren’t crunching :D</p>
<p>I was looking at last week’s post to see what I was up to and I realize that although I’ve been feeling like I was going really slow I’ve had 52 commits in the last week which includes the following:</p>
<h3 id="added-a-simple-api-for-users-to-be-able-to-inspect-the-compilation-environment-in-their-macros">Added a simple api for user’s to be able to inspect the compilation environment in their macros</h3>
<p>Common Lisp allows you to access the environment in the macros, however the interface to interact with that object didnt make it into the standard. It was, however, documented so I can still read it [0]. I decided not to implement that api yet though and instead I just made the functions I felt made sense. It will be easy to revisit this when I feel the drive to.</p>
<h3 id="added-declare">Added ‘declare’</h3>
<p>Lisp allows you to declare information about things in your code, whether it is whether a function should be inline or the type of a value, it’s handled through <code class="language-plaintext highlighter-rouge">declare</code></p>
<p>Once again, although it didn’t make it into the standard, extending the kinds of things that can be declared has been thought about and documented. For now I just hooked it into my compile-time-metadata system. A little reading reveals that the proposed system is a rough superset of what I offer so at some point I will implement that and redefine by metadata using it.</p>
<h3 id="added-deftype">Added ‘deftype’</h3>
<p>This is one place I left the lisp spec entirely. <code class="language-plaintext highlighter-rouge">deftype</code> in lisp is mad, types are seen as sets of objects (makes sense) and they don’t restrict you in how you define that. For example I can say:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; a function that returns true if given an array where
;; all it's dimensions are of equal length
;;
(defun equidimensional (a)
(or (< (array-rank a) 2)
(apply #'= (array-dimensions a))))
;; the type of all square arrays
;;
(deftype square-matrix (&optional type size)
`(and (array ,type (,size ,size))
(satisfies equidimensional)))
</code></pre></div></div>
<p>This is super expressive but totally impossible to check at compile time as you can have arbitrary function calls in there. The compilers do a great job on using the flexibility in some cases though, which I won’t yak about now.</p>
<p>In GLSL things are very different, we don’t want to do anything at runtime unless we can help it. We can’t make any types except structs and I already have a nice mechanism for that.</p>
<p>However there are so things that may be interesting.</p>
<ul>
<li>
<p>Making a ‘special kind of’ some exisiting type. For example a <code class="language-plaintext highlighter-rouge">rotation</code> type that is essentially a <code class="language-plaintext highlighter-rouge">vec3</code>. In the GLSL it will be a <code class="language-plaintext highlighter-rouge">vec3</code> however by making the type checker see it as a different type we could make it impossible to pass a position as the rotation argument to a <code class="language-plaintext highlighter-rouge">rotate</code> function by accident. I call these <code class="language-plaintext highlighter-rouge">shadow</code> types (I probably need to rethink this name :p).</p>
</li>
<li>
<p>Making a type that only exists at compile-time for the purpose of holding metadata. This ‘ephemeral’ type never appears in the resulting GLSL and is an advanced feature but can be useful. I use these to represent vector spaces in my code and then use macros to inject code to create the matrices that transform between the spaces.</p>
</li>
</ul>
<p>To make a shadow type I write <code class="language-plaintext highlighter-rouge">(deftype rotation () :vec3)</code></p>
<p>To make an ephemeral type I write <code class="language-plaintext highlighter-rouge">(deftype space () ())</code></p>
<p>In the case of shadow functions I also allow you to easily ‘copy’ functions that are defined for the type you are shadowing. This means you don’t have to redefine addition, subtraction etc for your <code class="language-plaintext highlighter-rouge">rotation</code> type, you just copy it from <code class="language-plaintext highlighter-rouge">vec3</code></p>
<h3 id="mooore-types">Mooore Types</h3>
<p>One problem with how I used to write lisp is that too often I would use lists when I should have used classes. This has bitten me in the arse one to many times and so I refactored a bunch of stuff to have their own types.</p>
<p>This will help with future refactoring and also as I start working out the ‘final’ public api for my compiler.</p>
<h3 id="rewrite-space-feature-in-cepl">Rewrite Space Feature in CEPL</h3>
<p><code class="language-plaintext highlighter-rouge">Spaces</code> are a feature I have spoken of before but I will quickly recap. They are a fairly simple idea, you define a scope with a vector space and inside that scope all vectors have to be in that space. If you try and use a vector from another space it will be automatically converted to the correct space before anything else happens. This lowers the chances of messing up your vector math and makes the code more self-documenting.</p>
<p>Previously this was implemented by:</p>
<ul>
<li>compiling the code once</li>
<li>analysing the AST for places that needed conversions</li>
<li>patching the code with the needed conversions</li>
<li>recompiling and using the new GLSL</li>
</ul>
<p>This felt super hacky as all of the process was specific to my compiler. I have now rewritten it using macros, there is much less that will feel alien to a lisper[1] and it all happens in one pass now.</p>
<h3 id="change-the-default-minify-filter-for-samplers-in-cepl">Change the default minify-filter for samplers in CEPL</h3>
<p>I thought I had a bug in the textures or samplers in CEPL. I was using the <code class="language-plaintext highlighter-rouge">textureLod</code> function in my shaders but no matter what mipmap level I chose nothing would change.</p>
<p>I spent hours looking at the code around mipmaps and finally found out that nothing was wrong, but I had misunderstood how things were meant to work. I had imagined that <code class="language-plaintext highlighter-rouge">textureLod</code> allowed you to query the colors from a texture’s mipmaps regardless of the filter settings, whereas actually you need to <code class="language-plaintext highlighter-rouge">minify-filter</code> to either linear-mipmap-nearest or linear-mipmap-linear before ANY querying using <code class="language-plaintext highlighter-rouge">textureLod</code> is possible.</p>
<p>CEPL defaulted to just <code class="language-plaintext highlighter-rouge">linear</code> as it felt the simplest and I didnt want to confuse beginners (irony goes here) but instead I created a behaviour that was surprising and required knowledge to understand. This interests me and I’ll muse over this in future when designing apis.</p>
<h3 id="bugfixes-everywhere">Bugfixes EVERYWHERE</h3>
<p>Just whole bunch of other stuff. Not much to note here other than how quickly trying to make nice errors seems to balloon code.</p>
<p>For example if you are processing 10 things, and one fails, it is easy to detect this in the <code class="language-plaintext highlighter-rouge">process_thing</code> function and throw an error. However this means that the user may fix the one thing retry and hit the same error on the next thing. Really we want to aggregate the errors and present an exception saying ‘these 15 things failed for these reasons’. This would give the user more chance of seeing the patterns that are causing problems. This is especially true with syntax error in your code. You don’t want them one at a time, you want them on aggregate in order to see what you are doing wrong.</p>
<p>This also means that your error messages should be able to adjust depending on the number of things that went wrong. For example when 15 things break the ‘these 15 things failed for these reasons’ message is cool, but if only 1 thing broke we want ‘this thing failed for this reason’. These subtle language differences can make a huge difference to how the user feels about the tool they are using. Pluralizing words when there is only one thing sounds dumb and just adds to the annoyance of the tool not working.</p>
<p>I don’t have ideas for a clean solution to this yet, but one day I want something for this.</p>
<h3 id="sleep">Sleep</h3>
<p>I’m damn tired now. I’m off to bed.</p>
<p>Peace</p>
<p>[0] https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html
[1] and once I have the stuff from [1] it will be even better</p>
Inside the machine2017-01-23T10:42:23+00:00http://www.techsnuffle.com/2017/01/23/inside-the-machine<p>I’ve been back in the compiler this week and it’s just been constant refactoring :p</p>
<p>I kicked of the week with tweaks & bugfixes to the first-class function code. I then started on adding some compile-time metadata to Varjo.</p>
<p>The idea is super basic: We want to associate some data with the values flowing through the shader.</p>
<p>We already have <code class="language-plaintext highlighter-rouge">flow-id</code>s, which represent the values in the shader, so we could just add a hashtable that maps these ids to the data. Simple, and it seems to work!</p>
<p>At the very least it will work enough to make some progress on the idea anyhoo. You are able to add, but never remove metadata. Metadata can also be combined, for example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((x (v! 1 0 0 0)) ;; ← lets assume that both these have metadata
(y (v! 0 1 0 0))) ;; ← with a key called 'foo'
(if something
x
y))
</code></pre></div></div>
<p>What should be the metadata for the result of the if? Clearly this depends on what the metadata is for. So in the case of conditionals like <code class="language-plaintext highlighter-rouge">if</code> or <code class="language-plaintext highlighter-rouge">switch</code> we call a method called <code class="language-plaintext highlighter-rouge">combine-metadata</code> defined for that metadata type, and it is it’s job to decide the metadata for the new result.</p>
<p>This mechanism is very simple but should be enough to get some useful results.</p>
<p>So with this in place I could dive back into re-implementing my <code class="language-plaintext highlighter-rouge">space</code> analysis feature! Progress.. excitement… Ah fuck it, I really need symbol-macros.</p>
<p>Symbol macros are pretty simple:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((x some-object))
(symbol-macrolet ((foo (slot-value x 'foo))) ;; You specify a symbol 'foo' and what it will expand into
(print foo) ;; the compiler then replace every instance of 'foo' in your code with the expansion
(setf foo 20) ;; before it compiles to code
(print foo)))
</code></pre></div></div>
<p>So what actually ends up getting compiled is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((x some-object))
(print (slot-value x 'foo))
(setf (slot-value x 'foo) 20)
(print (slot-value x 'foo)))
</code></pre></div></div>
<p>So that should be simple to implement.</p>
<p><em>SHOULD BE</em></p>
<p>Alas when I made the macroexpander[0] for Varjo I was super lazy. I didn’t implement any lexically-scope macros and just expanded everything in a prepass. This really hurt as I suddenly needed to go change how all this was written and mix the expansion into the compile pass.</p>
<p>This refactor took all of Sunday and it was one of those ones where you are changing so much fundamental stuff that you can’t compile for an hour or so at a time. As someone who compiles many times a minute this just left me feeling very unsettled. However when it was done the little test suite I have made me feel fairly confident with the result.</p>
<p>Another nice thing about all this is that my macros will now be able to take the compilation-environment as an argument just like in real common lisp. This means I can provide some of the environment introspection features that are described in <a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node227.html">CLTL2</a> but sadly didn’t make it into the standard. This together with metadata make the macros extremely powerful.</p>
<h3 id="next-week">Next week</h3>
<p>The first order of business is compiler-macros, which should be easy enough. Then I want to add support for <code class="language-plaintext highlighter-rouge">&rest</code> (vararg) arguments to the compiler. Then I will get back intothe <code class="language-plaintext highlighter-rouge">spaces</code> feature and see how far I can get now.</p>
<p>Peace</p>
<p>[0] back in 2013, holy shit!</p>
Back in the create phase2017-01-17T16:47:48+00:00http://www.techsnuffle.com/2017/01/17/back-in-the-create-phase<p>The other week I was nattering about how I feel I oscillate slowly between needing to create and needed to consume, these last 2 weeks I have definitely been back in the ‘create’ mode. I’m hammering away at the compiler and the new stuff is starting to feel nice. I won’t bother with deep explanations as it’s wouldn’t work in this format, however I have been doing the following:</p>
<ul>
<li>
<p>Refactored up how the compiler compiles functions. I certainly can’t say this code is clean yet, but it is at least not lazy. I used to just splice the function code in as a local function and let the code de-duplicator take care of things, this worked well enough for a while but I’ve been fighting it recently so I’m happy to sort this out.</p>
</li>
<li>
<p>Tests! These have been worth their weight in gold this week and I’ve been tidying them up and adding a few more as I debug things.</p>
</li>
<li>
<p>Moved the flow-id to the type: In short I have an ID that is used to work out where values (especially arguments) travel inside functions. When talking to a chap at work about this he mentioned ‘dependent types’ and this stuck with me. Given that this is metadata about the things flowing around, it is the kind of info that could live on a type. I have thus moved this to a field inside the type object and in doing so cleaned up a TONNE of things. This felt great.</p>
</li>
<li>
<p>Started cleaning some of the code used to represent functions from the glsl spec and realized a bunch of it was redundant. Deleting code is heavenly.</p>
</li>
<li>
<p>Deleted more old code related to types (from when I tried to encode the glsl spec’s stupid gentypes directly )</p>
</li>
<li>
<p>fixed assorted bugs</p>
</li>
<li>
<p>found more bugs</p>
</li>
<li>
<p>Started work on supporting the <code class="language-plaintext highlighter-rouge">void</code> type. Funnily enough I have not supported <code class="language-plaintext highlighter-rouge">void</code> so far in my compiler (except for the main function). The reason was that all code in lisp is an expression, so everything has a return (even if it’s <code class="language-plaintext highlighter-rouge">nil</code>) to that end I worked to make sure this felt right in Varjo. However this has always left a few places that felt janky so I’m making void work now. This is pretty easy.</p>
</li>
</ul>
<p>I’m basically going to be hammering on more of the same this week. I want to get this working and into master asap. One downside of the above is that in CEPL I had to introduce a hack to keep my <code class="language-plaintext highlighter-rouge">spaces</code> feature working. However I realized the other day how to replace all the crazy AST transforms I do in CEPL with a much simpler mechanism that is more general and will live in Varjo. More on this soon.</p>
<p>That’s the lot.</p>
<p>Seeya</p>
Balls!2017-01-10T11:51:47+00:00http://www.techsnuffle.com/2017/01/10/balls<p>It’s been a good week for progress.</p>
<h3 id="gl-state-cache">GL State Cache</h3>
<p>The first big thing I achieved was to add GL state caching to CEPL. This has fixed some bugs, avoided some unnecessary state changes and generally made prototyping a bit easier.</p>
<p>For those not familiar that with OpenGL I’ll try explain the whats and whys now. When using OpenGL you are basically working with a crazy vending machine for graphics, you shove what you want draw in one hole, and then flick a bunch of switches and hit go. The switches control ‘how’ it is going to draw. One you have ‘hit go’ it will start rumbling and very soon will have a result. Whilst it is working you can prepare (and start) the next job, setting things up and flicking switches, GL keeps a queue of what it has to go.</p>
<p>The ‘vending machine’ in the above analogy is more formally called the ‘GL state machine’. The state machine has some ‘interesting’ aspects such as:</p>
<ul>
<li>
<p>setting some properties on the state machine cause it to have to finish what it was doing before they can be applied, this blocks all queued jobs until that is done. This is really bad for performance</p>
</li>
<li>
<p>querying the properties of the machine is slow. Which means that although changing things can be fast, looking at what you are changing isnt :p</p>
</li>
</ul>
<p>So we want to avoid changing things on the state machine when we can help it, but we can’t afford look at the machine to see if we can avoid changing things.. the solution here is a cache. In the cache we record what we think/know the state of GL is, then when we want to change things we can look at the cache to see if we already have the correct value, and if so, we can avoid requesting the change.</p>
<p>It took a surprisingly long time to get a solution I was happy with in CEPL as I was trying to find a balance between speed and usability. A cache like this can be super useful when debugging as you can look at it and get check your assumptions with the actual state of GL, however working with the raw GL object IDs is (generally) faster. This is also an area where I found the API shaped by the fact I’m in a dynamic programming language, it’s taken some time to find something I’m happy(ish) with.</p>
<h3 id="ibl-0">IBL [0]</h3>
<p>After that I went back to trying to implement PBR rendering. I had previously (with help) managed to get point lights working, however I really want to get image based lighting (IBL) in the mix. Now I’ve tried this a few times and have really struggled, partly because the resources available never have a complete example, they are almost always by people with enough rendering experience that they can assume some things to be obvious.. it is almost never obvious to me :p.</p>
<p>This time I went back and attacked <a href="http://www.codinglabs.net/article_physically_based_rendering.aspx">this tutorial</a>. Long story short: I don’t believe that the way he is convolving his env maps in the provided code is the way he is doing it in his renders. I made sure that my compiler was producing the EXACT same code as he had and I got completely useless results.</p>
<p>In a fit of desperation I went back onto github searching for any projects that have something working. Many use frameworks like <code class="language-plaintext highlighter-rouge">3js</code> so while they are awesome, they we of limited help. However I did find the following:</p>
<ul>
<li>
<p><a href="https://github.com/Corralx/BRDFExplorer/tree/master/shader">A 3js project</a> where they have a tonne of the different BRDFs implemented with consistent signatures. This will be a great resource for future learning.</p>
</li>
<li>
<p><a href="https://github.com/tuccio/IBLGGX">A C++ project</a> which has (what looks like) a full implementation of PBR with IBL, using pretty raw DirectX and DXUT (a DirectX equivalent of GLUT). This thing seems to be a perfect resource for me, it’s not an engine, just a project by some person looking to understand a single implementation of this. I’m stoked to be able to read this.</p>
</li>
</ul>
<p>I immediately cloned that C++ project and started reading. They are using the same papers as I have been trying to learn from, however they clearly understood enough to fill in the gaps and (more importantly) pick a selection of approaches that worked together.</p>
<p>I rewrote my env map convolve shader based on what they had AND IT WORKED! The image below shows the same balls taking light from env maps made for two slightly different roughness values. It’s not much but the fact that you can see that the right version has a slightly blurrier ‘reflection’.</p>
<p><img src="/assets/images/ConvoleEnvMapTwoDifferentRoughnesses.jpeg" /></p>
<p>I’m now confident that I will be able to get the rest working. More on this in the coming weeks.</p>
<h3 id="other-stuff">Other Stuff</h3>
<p>Around this work a bunch of little things got done including:</p>
<p>In CEPL</p>
<ul>
<li>Fix a bunch of GL state bugs using the new cache feature</li>
<li>Start migrating ugly global CEPL state to the cache</li>
<li>Fix mipmap size bug in texture code</li>
</ul>
<p>In Varjo</p>
<ul>
<li>Generate easier to read GLSL for symbols like π, φ, θ, etc</li>
<li>Identify and write tests for a bug in function deduplication (not yet fixed). Basically the way I handle global user defined functions is crappy & I need to fix it.</li>
</ul>
<p>In Nineveh (A library for useful GPU/Rendering functions)</p>
<ul>
<li>Add a function that takes a cubemap texture and makes 1 FBO for each mipmap level where the attachments of the fbo are the cube faces at that mipmap level.</li>
</ul>
<h3 id="thats-all-folks">That’s all folks</h3>
<p>That’s this week done, I really hope I have something visual to show off next week.</p>
<p>Until then, Peace.</p>
<p>[0] Sorry but I won’t try to explain this section simply yet as I don’t understand it well enough yet.</p>
I'm late!2017-01-04T11:18:14+00:00http://www.techsnuffle.com/2017/01/04/im-late<p>Woops, I’m late posting this week. I haven’t got too much to show however there has been some progress.</p>
<p>The big piece of work was on the compiler. Let’s say we have this bit of code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((fn #'sin))
(funcall fn 10))
</code></pre></div></div>
<p>Which says:</p>
<blockquote>
<p>line 1: Let there be a variable named fn which is bound to the function ‘sin’
line 2: Take the value bound to ‘fn’ and call it as a function with the argument ‘10’</p>
</blockquote>
<p>Now as of last week this wouldn’t work in my compiler. The reason was that functions in GLSL can have overloads, for example <code class="language-plaintext highlighter-rouge">pow</code> can take a <code class="language-plaintext highlighter-rouge">float</code>,<code class="language-plaintext highlighter-rouge">vex2</code>,<code class="language-plaintext highlighter-rouge">vec3</code> or <code class="language-plaintext highlighter-rouge">vec4</code>. So when I said <code class="language-plaintext highlighter-rouge">#'sin</code> (get the function named <code class="language-plaintext highlighter-rouge">sin</code>) which <code class="language-plaintext highlighter-rouge">sin</code> overload was I talking about?</p>
<p>This meant I had to write this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((fn #'(sin :float)))
(funcall fn 10))
</code></pre></div></div>
<p>Where <code class="language-plaintext highlighter-rouge">#'(sin :float)</code> reads as ‘the function named sin that takes one argument of type float’.</p>
<p>This worked but felt clumsy so my goal was to make the first code example work.</p>
<p>The way I went about it was to say:</p>
<blockquote>
<p>When we compile a form like #’xyz where there are not argument types specified, work out all the functions named ‘xyz’ that can be called from that position in the code. Pass this set of functions around with the variable binding information in the compiler an then when the user writes a <code class="language-plaintext highlighter-rouge">funcall</code> look at the types of the arguments and work out which overload to use.</p>
</blockquote>
<p>The nice thing was that the code to pick the most appropriate function for some args from a set of functions already existed, naturally as we need to do this for regular function calls. So that code was generalized a bit and recycled.</p>
<p>The rest took some time as I kept bumping into details that made this more complicated than I had hoped, mostly because my compiler was (and is) written on the fly without any real technical knowledge. I’ve certainly learned stuff that I want to apply to the compiler, but some of these things require large changes to the codebase and I’m not really ready to get into that yet. I have some time constraints as I want to give a talk on this in the near future so I really just need it to work well enough for that.</p>
<p>Other than this I speed-watched a couple of intro unity courses and started reading their docs. The courses were ok’ish I got what I needed from them but I’m hoping that wasnt meant to be good code style as it felt very messy. Time will tell.</p>
<p>That’s all for today,
Ciao</p>
Baked2016-12-28T12:51:51+00:00http://www.techsnuffle.com/2016/12/28/baked<p>This week I got first class functions into CEPL. It’s not release quality yet but it seems to work which is ace.</p>
<p>What I can now do is define a pipeline that takes a function as a uniform. It compiles this to typecheck it but GLSL doesn’t support first-class functions so the pipeline is marked as partial.</p>
<p>So if I have a gpu function I’m going to use as my fragment shader in some particle system:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g update-particle-velocities ((tex-coord :vec2)
&uniform (positions :sampler-2d)
(velocities :sampler-2d)
(animator (function (:vec4 :vec4) :vec4)))
(let* ((position (texture positions tex-coord))
(velocity (texture velocities tex-coord)))
;; calc the new velocity
(funcall animator positions velocities)))
</code></pre></div></div>
<p>I can make a partial pipeline:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(def-g-> update-velocities ()
:vertex (full-screen-quad :vec4)
:fragment (update-particle-velocities :vec2))
</code></pre></div></div>
<p>If I try and run this CEPL will complain that this is partial. Now let’s make a dumb function that slowly pulls all particles to the origin:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g origin-sink ((pos :vec4) (vel :vec4))
(let ((dif (- pos)))
(+ velocity (* (normalize dif) (sqrt (length dif)) 0.001))))
</code></pre></div></div>
<p>And now we can complete the pipeline:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defvar new-pipeline (bake 'update-velocities :animator '(origin-sink :vec4 :vec4)))
</code></pre></div></div>
<p>I’m not happy with the syntax yet but the effect is to create a new pipeline with the uniform <code class="language-plaintext highlighter-rouge">animator</code> permanently set to our <code class="language-plaintext highlighter-rouge">origin-sink</code> function.</p>
<p>We can then map over that new pipeline like any other:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(map-g new-pipeline quad-verts :positions pos-sampler :velocities vel-sampler)
</code></pre></div></div>
<p>The advantage of this is that I can make a bunch of different ‘animator’ functions and make pipelines that use them just with <code class="language-plaintext highlighter-rouge">bake</code>. It still feels odd to me though so It’ll take some time to get used to and cleaned up.</p>
<p>One thing that is interesting me recently is that in livecoding it almost encourages more ‘static’ code. Composition is great for programming but to a degree makes things opaque to the experimenter. If you call a function that returns a function you are sitting in the repl with this function with very little insight into what it is/does. You have to maintain that mapping yourself. It <em>may</em> have something lexically captured, it may not.. I’m not sure where I’m going with this, expect that maybe it would be cool to be able to get the code that made the lambda while at the repl so you can get some context.</p>
<p>Anyhoo, this next week I need to work on the compiler a bit, clean up some of the stuff above & do spend some time studying.</p>
<p>Seeya</p>
This and That2016-12-20T11:23:30+00:00http://www.techsnuffle.com/2016/12/20/this-and-that<p>I seem to oscillate slowly between needing to create and needed to consume media. This week I’ve either jumped back to consume or I’m procrastinating.</p>
<p>Either way I’ve not been super productive, let’s start with the consumption:</p>
<h3 id="nom-nom-knowledge">Nom nom knowledge</h3>
<ul>
<li>
<p>Listened to <a href="https://www.youtube.com/playlist?list=PLSRtWqmC79Ka5YqNJeHP6gbzLTVtWrqh_">Iceberg Slim’s biography</a> - On stage one time Dave Chappele was talking about how the media industry worked and likened it to pimping. He said this book pretty much explained the system. It’s damn depressing.</p>
</li>
<li>Watched some TED talks. Filtering out all the TEDX shit really makes the site more valuable</li>
<li><a href="http://www.ted.com/talks/craig_venter_unveils_synthetic_life#t-848430">craig venter unveils synthetic life</a> - It’s now been 5 years since the first man-made lifeform was booted-up. Fucking incredible work. Everything this guy works on is worth your time</li>
<li><a href="http://www.ted.com/talks/uri_hasson_this_is_your_brain_on_communication">your brain on communication</a> - FMRIs taken on people being read stories. Very cool way of setting up the tests to reveal the layered nature of brain processing</li>
<li><a href="http://www.ted.com/talks/andrew_pelling_this_scientist_makes_ears_out_of_apples#t-262467">scientist_makes_ears_out_of_apples</a> - less on the bleeding edge but very cool. Seems that stripping living things down totheir cellulose structure is possible in a home lab. I’d like to mess with this some day.</li>
<li><a href="http://www.ted.com/talks/trevor_aaronson_how_this_fbi_strategy_is_actually_creating_us_based_terrorists">how this fbi strategy is actually creating us based terrorists</a> - Fucking infuriating but relieving to see how much info is available through the court proceedings</li>
<li><a href="http://www.ted.com/talks/laura_schulz_the_surprisingly_logical_minds_of_babies#t-1202930">The surprisingly logic minds of babies</a> - I loved this one. Babies can infer based on probabilities. Very cool to see how these tests are constructed and good food for thought when musing on how brains work.</li>
<li><a href="http://www.ted.com/talks/bj_miller_what_really_matters_at_the_end_of_life#t-925839">What really matters at the end of life</a> - Watch this. Is about designing for dying. Very moving, very important, I wish I could say all my family could leave on the terms he describes. Also badass is the fact that the talk is given by a cyborg and that isnt the point of the talk. I love this part of our future.</li>
<li><a href="http://www.ted.com/talks/david_sengeh_the_sore_problem_of_prosthetic_limbs">The sore problem of prosthetic limbs</a> - How to make comfortable sockets for prosthetic limbs. I’m fairly sure this can be done without the MRI step, which would make it much cheaper and more portable. I’ll have to look into this at some point</li>
<li><a href="http://www.ted.com/talks/riccardo_sabatini_how_to_read_the_genome_and_build_a_human_being#t-802703">How to read the genome and build a human being</a> - Really cool to see how machine analysis of the geneome can let us predict some characteristics with high accuracy (height to within 5cm for example). A good intro talk</li>
<li>
<p><a href="http://www.ted.com/talks/adam_savage_my_love_letter_to_cosplay#t-545334">Adam Savage’s love letter to cosplay</a> - Passion and community, I loved this talk.</p>
</li>
<li>I’ve also been listening to a book called <a href="https://en.wikipedia.org/wiki/The_Information:_A_History,_a_Theory,_a_Flood">‘The Information: A History, a Theory, a Flood’</a> again. It’s an outstanding walk through our history of understand what information is. From african drums, to morse code, to computers, to genes. This book rules. Read/Listen to it. I’ve been repeating 2 chapters this last week trying to bake them into my brain. The first was on entropy (and how information & entropy are the same), and the second is on genes and how information flows from and through us.</li>
</ul>
<h3 id="making-stuffs">Making stuffs</h3>
<p>The first order of business was to look at PBR. Previously I had got deferred point lights to work, however I failed hard at the IBL step. Luckily I rediscovered <a href="http://www.codinglabs.net/article_physically_based_rendering.aspx">this tutorial</a> as understood how it fit into what I was doing. Last time I had tried to stick to one paper as (in my ignorance) each approach felt very different to me.</p>
<p>I wrote the shader in lisp and immediately ran into a few bugs in my compiler. This sucked and the fixes I made werent satisfying. The problems were dumb side-effects of how i was doing my flow analysis, I’m pretty sure now that I want to get my <code class="language-plaintext highlighter-rouge">compiler-time-values</code> feature finished and then rewrite the code that uses the flow analyzer to use that instead.</p>
<p>I then ran into a few rendering issues. The first turned out to be a bug in my implementation of a function that samples from a cross texture (commonly used for environment maps). The next 2 I haven’t fixed yet:</p>
<ul>
<li>the env map filtering pass is super slow</li>
<li>the resulting env map has horrifying banding</li>
</ul>
<p>I checked the generated glsl and it looks fine. I’m struggling to work out how I’m screwing this up. I guess it could be that I have a bug in how I’m binding/unbinding textures and this is causing a flush..that could account for the speed…and maybe the graphical fuckups? I don’t know man.</p>
<p>Despite that it feels good to be back in that codebase. One thing that really stood out though was how much first-class functions could make the codebase cleaner and more flexible. I had started that feature partially for fun but more and more it seems it’s going to be very useful.</p>
<p>Given that I spent last night digging into that branch of my compiler. I decided that even without support for closures it would still be a good feature. So I did the following:</p>
<ul>
<li>Throw exception on attempts to pass a closure as a value. I’ve tried to make the error message friendly so people get what is happening</li>
<li>fixed a couple of glsl generation bugs around passing functions as objects</li>
</ul>
<p>I then spent a little time looking into how to generalize my compile-time-value feature, this will mean I can not only pass around functions but values user defined types. I’m going to use this for vector spaces. I realized that this doesn’t currently have enough power to cover all the things I could do with flow analysis, this is a bummer but at this point I had drunk too much wine to come up with good ideas so I called it a night.</p>
<p>Next week I need to crack the new version of the spaces feature, get that merge in and get back into the PBR.</p>
<p>.. Oh and Christmas :p</p>
Depending on Types2016-12-12T20:38:21+00:00http://www.techsnuffle.com/2016/12/12/depending-on-types<p>Working on the compiler in the previous weeks got me in a mode where I was thinking about types again.</p>
<p>I’m pretty set on the idea of making static type checking for lisp, however I’m interested in not only being able to validate my code, but also to play with random types systems and maybe make my own. I need an approach to doing this kind of compile-time programming.</p>
<p>Macros give a super simple api, they are just a function that gets called when the code is being compiled. Say we make a macro that multiply some numbers at compile time (a bad usage but that doesnt matter for the example)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defmacro multiply-at-compile-time (&rest numbers)
(assert (every #'numberp numbers)) ;; check all the arguments were number literals
(apply #'+ numbers)) ;; multiply the numbers
</code></pre></div></div>
<p>And now we have a function that calculates the numbers of seconds in a given number of days:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun seconds-in-days (number-of-days)
(* number-of-days (multiply-at-compile-time 60 60 24)))
</code></pre></div></div>
<p>When we compile this the macros need to be expanded, so for each form the compiler will look and see if is a list where the first element is the name of a macro. If it is then it calls the <code class="language-plaintext highlighter-rouge">macro-function</code> with the code in the argument positions. So it’ll go something like this:</p>
<ul>
<li>Looks at: <code class="language-plaintext highlighter-rouge">(defun seconds-in-days etc etc)</code></li>
<li>is <code class="language-plaintext highlighter-rouge">defun</code> the name of a macro? Yes, call macro-function <code class="language-plaintext highlighter-rouge">defun</code> and replace <code class="language-plaintext highlighter-rouge">(defun seconds-in-days etc etc)</code> with the result</li>
<li>the code is now:</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> (setf (function seconds-in-days) ;; NOTE: Not the actual expansion from my compiler, just an example
(lambda (number-of-days)
(* number-of-days (multiply-at-compile-time 60 60 24))))
</code></pre></div></div>
<ul>
<li>is <code class="language-plaintext highlighter-rouge">setf</code> the name of a macro? (let’s say no for the sake of this example) No? ok continue</li>
<li>Looks at: <code class="language-plaintext highlighter-rouge">(function seconds-in-days)</code></li>
<li>is <code class="language-plaintext highlighter-rouge">function</code> the name of a macro? No? ok continue</li>
<li>this repeats until we reach <code class="language-plaintext highlighter-rouge">(multiply-at-compile-time 60 60 24)</code></li>
<li>is <code class="language-plaintext highlighter-rouge">multiply-at-compile-time</code> the name of a macro? YES, call the macro-function <code class="language-plaintext highlighter-rouge">multiply-at-compile-time</code> with the arguments <code class="language-plaintext highlighter-rouge">60</code>, <code class="language-plaintext highlighter-rouge">60</code> and <code class="language-plaintext highlighter-rouge">24</code> and replace <code class="language-plaintext highlighter-rouge">(multiply-at-compile-time 60 60 24)</code> with the result.</li>
</ul>
<p>The final code is now:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf (function seconds-in-days)
(lambda (number-of-days)
(* number-of-days 86400)))
</code></pre></div></div>
<p>Technically we have avoided multiplying <code class="language-plaintext highlighter-rouge">60</code>, <code class="language-plaintext highlighter-rouge">60</code> and <code class="language-plaintext highlighter-rouge">24</code>.. once again, this of course this is a TERRIBLE use of macros as these kinds of optimizations are things that all decent compilers do anyway.</p>
<p>The point here though is that we implement a function and then there is a mechanism in the language that knows how to use it. This makes it endlessly flexible.</p>
<p>So if I’m going to make a mechanism for hooking in types then I want a similar interface, implement something and have it be called.</p>
<p>Now I know nothing about how ‘real’ type-systems are put together so I bugged a dude at work who knows this stuff well. Olle, is ace, he’s currently writing his own dependently-typed language for low level programming and so clearly knows his shit. When I mentioned I wanted this to be fairly general he recommended I look at ‘bidirectional type checking’.</p>
<p>A quick google later I had a pile of pdfs, but one clearly stood out at being a best place for me to start and that is <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">this one</a>. It’s a very gentle intro to the subject and with a few visits to youtube & wikipedia I was able to get through it.</p>
<p>One take away from it is that we can drive the process with a couple of functions <code class="language-plaintext highlighter-rouge">infer-type</code> & <code class="language-plaintext highlighter-rouge">check-type</code>. <code class="language-plaintext highlighter-rouge">infer-type</code> takes a form (code) and an <code class="language-plaintext highlighter-rouge">environment</code> (explained below) and works out what the type of the form is. <code class="language-plaintext highlighter-rouge">check-type</code> takes a form, an environment an expected type, it then infers the type of the form and ensures it is ‘equal’ to the required type. The environment mentioned above is the object that stores the mapping between names and types, so if you define an <code class="language-plaintext highlighter-rouge">int</code> variable named <code class="language-plaintext highlighter-rouge">jam</code> then that relationship is stored in the <code class="language-plaintext highlighter-rouge">environment</code></p>
<p>Unless I’ve massively misunderstood this paper this sounds like the start of a pretty simple api. Let’s fill in the gaps.</p>
<p>First we need to be able to define our own checker. Lets make up some syntax for that:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defchecker simple-checker
:fact-type simple-type)
</code></pre></div></div>
<p>It will define a class for the environment and a class for our types. We could then define a couple of methods:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defmethod infer (code (env simple-checker))
...)
(defmethod check (code (env simple-checker))
...)
</code></pre></div></div>
<p>I specialize the methods on the type of the environment, who’s name matches the name of the checker we defined.</p>
<p>Let’s now say we want to compile this code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let* ((x 10)
(y 20))
(+ x y))
</code></pre></div></div>
<p>First, we expand the code (I have made a expander that get’s it in a form that is useful for my checking)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((x 10))
(let ((y 20))
(funcall #'+ x y)))
</code></pre></div></div>
<p>My system the walks the code trying to find out facts (types) about the code and replacing each form with the form <code class="language-plaintext highlighter-rouge">(the <some-fact> <the form>)</code>. The system knows how to handle some of the fundamental lisp forms like <code class="language-plaintext highlighter-rouge">let</code>, <code class="language-plaintext highlighter-rouge">funcall</code>, <code class="language-plaintext highlighter-rouge">if</code>, etc but for the rest the <code class="language-plaintext highlighter-rouge">infer</code> & <code class="language-plaintext highlighter-rouge">check</code> methods are going to be called. For example <code class="language-plaintext highlighter-rouge">infer</code> is going to be called for <code class="language-plaintext highlighter-rouge">10</code> and <code class="language-plaintext highlighter-rouge">20</code> but the system will handle adding the bindings from <code class="language-plaintext highlighter-rouge">x</code> & the result from <code class="language-plaintext highlighter-rouge">infer</code> to the <code class="language-plaintext highlighter-rouge">environment</code>.</p>
<p>The result could look like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(the #<simple-type int>
(let ((x (the #<simple-type int> 10)))
(the #<simple-type int>
(let ((y (the #<simple-type int> 20.0)))
(the #<simple-type int>
(funcall (the #<simple-type func-int-int> #'+)
(the #<simple-type int> x)
(the #<simple-type int> y)))))))
</code></pre></div></div>
<p>where each of these <code class="language-plaintext highlighter-rouge">#<simple-type int></code> is a instance of our <code class="language-plaintext highlighter-rouge">simple-type</code> class holding some info of the type it represents.</p>
<p>This type annotated code will then be the input to a function that turns it into valid common lisp code. The simplest version of this would simply remove the <code class="language-plaintext highlighter-rouge">(the #<fact> ..)</code> stuff from the code but a more interesting version would convert the type objects into lisp type names. So something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(the (signed-byte 32)
(let ((x (the (signed-byte 32) 10)))
(the (signed-byte 32)
(let ((y (the (signed-byte 32) 20.0)))
(the (signed-byte 32)
(+ (the (signed-byte 32) x) (the (signed-byte 32) y)))))))
</code></pre></div></div>
<p>This is valid lisp, if you tell lisp to optimize this code it will produce very tight machine code. <code class="language-plaintext highlighter-rouge">[0]</code></p>
<p>Usually lisp doesnt need anywhere near this number of type annotations to optimize code but having more doesn’t hurt :p</p>
<p>The result of this could be that we have a dynamic language where we can take chunks and statically type it, with checkers of our own devising, gaining all the benefits in checking and optimization that we would from any other static language.</p>
<p>.. That is of course if it works.. I could be talking out my arse here! :D</p>
<p>I need to do some more reading before diving back into this and I really should do some work on my PBR stuff again. So I will leave this project until next year. I’m just happy to have finally made a start on it.</p>
<p>Seeya</p>
<p>[0] Yet again, this is a trivial example but the idea extends to complex code. The advantage of having a readable post vastly outweighs being technically accurate</p>
Hello progress my old friend2016-12-06T15:52:25+00:00http://www.techsnuffle.com/2016/12/06/hello-progress-my-old-friend<p>Ah this week was so much better, my brain and I were on the same team.</p>
<p>I made good progress in my compiler with first class functions. The way I implemented it is roughly as follows:</p>
<p>I make a class to represent compile-time values</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defclass compile-time-value (v-type)
(ctv))
</code></pre></div></div>
<p>It inherits from <code class="language-plaintext highlighter-rouge">v-type</code> as that is the class of my compiler’s types.</p>
<p>It has one slot called <code class="language-plaintext highlighter-rouge">ctv</code> that is going to store what the compile things the actual value is during compilation.</p>
<p>IIRC this associating of a value with a type is called ‘dependent types’. However I’m going to avoid that name as I don’t know nearly enough about that stuff to associate myself with it. I’m just going to call this compile-time-values or <code class="language-plaintext highlighter-rouge">ctv</code>s.</p>
<p>Next we need a type for functions.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defclass function-spec (compile-time-values)
(arg-spec
return-spec))
</code></pre></div></div>
<p>Here we make a type that has a list of types for the arguments (<code class="language-plaintext highlighter-rouge">arg-spec</code>) and a list of types for the returns (<code class="language-plaintext highlighter-rouge">return-spec</code>). Return is a list as lisp supports multiple return values. Being a <code class="language-plaintext highlighter-rouge">ctv</code> the compiler can now associate values with this type.</p>
<p>Note we don’t have a name here as this is just the type of a function, not any particular one. In my compiler I have a class called <code class="language-plaintext highlighter-rouge">v-function</code> that describes a particular function. So there is a <code class="language-plaintext highlighter-rouge">v-function</code> for <code class="language-plaintext highlighter-rouge">sin</code> for example.</p>
<p>In lisp to get a function object we use the <code class="language-plaintext highlighter-rouge">#'</code> syntax. So <code class="language-plaintext highlighter-rouge">#'print</code> will give you the function named <code class="language-plaintext highlighter-rouge">print</code>. <code class="language-plaintext highlighter-rouge">#'thing</code> expands to <code class="language-plaintext highlighter-rouge">(function thing)</code> so in my compiler I defined a ‘special form’ called <code class="language-plaintext highlighter-rouge">function</code> that does the following:</p>
<ol>
<li>look up the <code class="language-plaintext highlighter-rouge">v-function</code> object for that name</li>
<li>make an instance of <code class="language-plaintext highlighter-rouge">function-spec</code> with the result of step <code class="language-plaintext highlighter-rouge">1</code> as the <code class="language-plaintext highlighter-rouge">ctv</code></li>
<li>use the result of step <code class="language-plaintext highlighter-rouge">2</code> as the type of this form.</li>
</ol>
<p>Nice! this means the specific function is now associated with this type and will be propagated around.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((our-sin #'sin))
(funcall our-sin 10))
</code></pre></div></div>
<p>Later our compiler will get to that <code class="language-plaintext highlighter-rouge">funcall</code> expression. It will look at the type of <code class="language-plaintext highlighter-rouge">our-sin</code> and see the <code class="language-plaintext highlighter-rouge">ctv</code> associated with it. It will then transform <code class="language-plaintext highlighter-rouge">(funcall our-sin 10)</code> to <code class="language-plaintext highlighter-rouge">(sin 10)</code> and compile that instead.</p>
<h3 id="functions-that-take-compile-time-values-as-arguments">Functions that take compile time values as arguments</h3>
<p>We do a very simple hack when it comes to this. If we have something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; this takes a func from int to int and call it with the provided int
(defun some-func-caller ((some-func (function (:int) :int))
(some-val :int))
(funcall some-func some-val))
</code></pre></div></div>
<p>And we call it in the following code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((our-local-func ((x :int))
(* x 2)))
(let ((our-val 20))
(some-func-caller #'our-local-func our-val)))
</code></pre></div></div>
<p>Then the compiler will swap out the <code class="language-plaintext highlighter-rouge">(some-func-caller #'our-local-func our-val)</code> call with a local version of the function with the compile time argument hardcoded</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((our-local-func ((x :int))
(* x 2)))
(let ((our-val 20))
(let ((some-func #'our-local-func))
(labels ((some-func-caller ((some-val :int))
(funcall some-func some-val)))
(some-func-caller our-val)))))
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">some-func</code> var is in scope for the local function <code class="language-plaintext highlighter-rouge">some-func-caller</code> so the transform we mentioned earlier will just work. The rest is just a local function transform and the compiler already knew how to do that.</p>
<p>Things get more complicated with closures and I havent finished that. I can now pass closures to functions but I cannot return them from functions yet. I know how I could do it but it feels hacky and so I’m waiting for more inspiration before I try that part again.</p>
<h3 id="primed-for-types">Primed for types</h3>
<p>With all this compiler work my brain was obviously in the right place to start things about static typing in general. Being able to define your own type-system for lisp is something I have wanted for ages, but as support for this isn’t built into the spec I’ve been trying to work out what the ‘best approach™’ is.</p>
<p>quick notes for those interested. Lisp has an expressive type system and a bunch of features to make serious optimizations possible. However it doesnt have something to let me define my own type system and use it to check my lisp code.</p>
<p>The problem boils down to macroexpansion. If you want to typecheck your code you want to expand all those macros so you are just working with vars, functions & special-forms (dont worry about these). However there isn’t a ‘macroexpand-all’ function in the spec[0]. There is a function for macroexpanding a macro form once, however this does not take care of things like the fact that you can define local, lexically scoped macros. This means there is an ‘expansion environment’ that is passed down during the expansion process and manipulating this is not covered by the spec.</p>
<p>There is however a tiny piece of fucking voodoo code that was written by one of the lisp compiler guys. It allows you to define a locally scope variable that is available at compile time within the scope of the macro. With this i can create and object that will act as my ‘expansion environment’ and let me have what I need.</p>
<p>Anyhoo, the other day I case up with a scheme for defining blocks of code that will be statically checked and how I will do macroexpansion. It’s not perfect, but it’s predicable and that will do for me.</p>
<p>I am going to make a library who’s current working title is <code class="language-plaintext highlighter-rouge">checkmate</code>. It will provide these static-blocks and within those you will be able to emit <code class="language-plaintext highlighter-rouge">fact</code>s about the expressions in your code. For function calls it will then call a <code class="language-plaintext highlighter-rouge">check-facts</code> method with the arguments for the function and all the <code class="language-plaintext highlighter-rouge">fact</code>s it has on them. You can overload this method and provide your own checking logic.</p>
<p>The <code class="language-plaintext highlighter-rouge">fact</code>s are just object of type <code class="language-plaintext highlighter-rouge">fact</code> and can contain anything you like. And because you implement <code class="language-plaintext highlighter-rouge">check-facts</code> you can define any logic you like there.</p>
<p>This should give me a library which makes it easier to define a type system. I can subclass <code class="language-plaintext highlighter-rouge">fact</code> and call that <code class="language-plaintext highlighter-rouge">type</code> and inside <code class="language-plaintext highlighter-rouge">check-facts</code> I implement whatever type checking logic I like.</p>
<p>A while back I ported an implementation of the <a href="https://en.wikipedia.org/wiki/Hindley%E2%80%93Milner_type_system">Hidley (Damas) Milner checking algorithm</a> to lisp so my goal is to make something where I plug this in and get classic ML style type checking on lisp code.</p>
<p>Wish me luck!</p>
<h3 id="next">Next?</h3>
<p>I’m not sure, my next year is going to contain a lot of study so I hope I can keep on top of these projects as well. The last few weeks have certainly reminded me to trust my instincts on what I should be working on, and it’s good to feel ‘out of the woods’ on that issue.</p>
<p>Peace</p>
<p>[0] Although a bunch of implementation do provide one. I made a <a href="https://github.com/cbaggers/trivial-macroexpand-all">library to wrap those</a> so technically I do have something that could work</p>
Let it lie2016-11-28T13:37:22+00:00http://www.techsnuffle.com/2016/11/28/let-it-lie<p>I missed a week (SSSHHAAAME) because I didn’t get much of note done. I got mapping over tables working, along with destructuring of flat record data but the output -v- how much time I was sitting in front of the machine simply didnt add up.</p>
<p>I’ve decided to stop fighting my brain and just put the project down for a bit. This sucks as it means I fail my November goal but I just have to accept that I either need to force myself to do something my brain isn’t enjoying (which isn’t the best way to wind down after work) or do something else. At the very least I get to confirm things I have been learning about how I learn/work, so that is some kind of positive I can scrape from this.</p>
<p>With this accepted I started looking at first-class functions in my shader compiler (Varjo).</p>
<p>It’s been a month since I touched this, so I spent a little time re-familiarizing myself with the problem and then I got to work.</p>
<p>First order of business was to get <code class="language-plaintext highlighter-rouge">funcall</code> working with variables of function type that are in scope. Something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((x #'foo))
(let ((y x))
(funcall y 10)))
</code></pre></div></div>
<p>I got the logic working for the above and then I spent a few hours making some parts of my compiler more strict about types. Some areas were just too forgiving about whether you had to provide a type object or let you pass a type signature instead. This made some other code more complicated that it needed to be. This was a relic from a much older version of the compiler.</p>
<p>I then spent some time thinking about how to handle passing functions to functions. I can use my flow analyzer and multiple passes but I don’t want to use that hammer if things can be easier.</p>
<p>For example let’s take this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((foo ((f (function (:int) :int))) ;; take a func from :int -> :int and call it with 10
(funcall f 10))
(bar ((x :int)) ;; some func from :int -> :int
(* x 2)))
;; do it
(funcall #'foo #'bar))
</code></pre></div></div>
<p>I can replace this the <code class="language-plaintext highlighter-rouge">(funcall #'foo #'bar)</code> with this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((foo ()
(let ((f #'bar))
(funcall f 10))))
(funcall #'foo))
</code></pre></div></div>
<p>which will get turned into</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((foo ()
(bar 10)))
(funcall #'foo))
</code></pre></div></div>
<p>This means I generate a new local function for each call-site of a function that takes functions. The compiler will any remove duplicate definitions.</p>
<p>At this point it’s worth pointing out one of the design goals of this feature. Predictability. This code is valid lisp:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun pick-func ()
(if (< some-var 10)
#'func-a
#'func-b))
(defun do-stuff ()
(funcall (pick-func) 10))
</code></pre></div></div>
<p>But at runtime we can’t pass functions around, so the best we could do for the above is to return an <code class="language-plaintext highlighter-rouge">int</code> and <code class="language-plaintext highlighter-rouge">switch</code> based on that.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>int pick_func() {
return (some_var < 10) ? 0 : 1;
}
void do_stuff () {
switch (pick_func()) {
case 0:
func_a(10);
break;
case 1:
func_b(10);
break;
}
}
</code></pre></div></div>
<p>This would work but this pattern can be slow if used too much. For now Varjo instead chooses to disallow this and make you implement it yourself. This means there are less cases where you are guessing what the compiler is up to if your code is slow. The compiler will be able to generate very precise error messages to explain what is happening in those cases.</p>
<p>That’s all for now. I’ve also got a bunch of ideas for this that are still very nebulous, I’ll write more as they become concrete.</p>
<p>Ciao</p>
Trudge2016-11-15T16:58:16+00:00http://www.techsnuffle.com/2016/11/15/trudge<p>This week I have been kind of working on the data-structure thing I mentioned last week.</p>
<p>The reason that it is ‘kind of’ is that I am having a big problem focusing on actually coding it. Over the course of the weekend I procrastinated my way into watching 3 movies rather than coding.</p>
<p>It is an odd one as I love the idea of the project, I want it to exist and (at least in the abstract) I am really interested in the implementation. However, for whatever reason, I am just struggling to stay focused when coding the damn thing. I haven’t pinned down what the issue is, but if I do I’ll report it here.</p>
<p>OK so what did I do?</p>
<ul>
<li>A bunch of small benchmarks to prove premise to myself</li>
<li>Defined the base types</li>
<li>Worked out how the live redefining of types will work.</li>
<li>Started work on record types (which will be the types of the columns of the tables)</li>
</ul>
<p>The third one took the most time as I want both the old and new type to exist simultaneously, this will allow me to create the type and then sweep through the existing tables to try and upgrade them to the new type. If this fails for some (or the users halts it for some reason) then we can still keep working with both the old and new types. I had to prove to myself that I could do this in a way that wouldn’t just pollute the namespaces with crazy numbers of types & functions.</p>
<p>Another great side effect of this is that we can compile the types and functions with optimization set to high, this gives us the most accurate picture of how our code will behave when we ship it. We can do this as only the <code class="language-plaintext highlighter-rouge">tables</code> implementation calls these functions or uses these types directly, so there is almost no place that this will cause the user issues (unless they go out of their way to do that).</p>
<p>Sadly that’s all for this week. Let’s hope next week goes a little better</p>
My plan for november2016-11-07T21:34:06+00:00http://www.techsnuffle.com/2016/11/07/my-plan-for-november<p>I’ve been writing up what I’ve been up to every week on a site called <a href="https://everyweeks.com">everyweeks.com</a>. It was started by some friends and I’ve been trying to keep myself to it. However it has meant I’ve been bad as posting here.</p>
<p>Before now I’ve felt it rude to just dump a link here each week. I thought I should be writing fresh content for each site. But fuck it, if the choice is a weekly link to an article or a dead blog..I choose the link.</p>
<p>So here is <a href="https://everyweeks.com/entry/5820e2d0e25a9c9c7573b9d6">this weeks one</a>. It’s a plan of something I want to make this month.</p>
Much reading but less code2016-08-08T12:44:09+00:00http://www.techsnuffle.com/2016/08/08/much-reading-but-less-code<blockquote>
<p>TLDR: If you add powerful features to your language you must be sure to balance them with equally powerful introspection</p>
</blockquote>
<p>I haven’t got much done this week, I have some folks coming from the uk soon so cleaning up and getting ready for that. I have decided to take on more projects though :D</p>
<p>Both revolve around shipping code which was the theme last week as well. At first I was looking at some basic system calls and it seemed that, if there wasn’t a good library for this already then I should make one. Turns out that was mostly me not knowing what I’m doing. I hadn’t appreciated that the posix api specifies itself based on certain types and that those types can be different size/layouts/etc on different platforms, which means that we can just define lisp equivalents without knowing those details. To get around this we need to use cffi’s groveler but this requires you to have a C compiler set up and ready to go. This, in my opinion, sucks as the user now has to think about more than the language they are trying to work in. Also because all libraries are delivered though the package manager you tend not to know about the C dependencies until you get an error during build which makes for pretty poor user experience.</p>
<p>To mitigate this what we can do is cache the output of the C program along with info on what platforms it is valid for. That second part is a little more fiddly as the specification that is given to the <code class="language-plaintext highlighter-rouge">groveler</code> is slightly different for different platforms and those difference are generally expressed with read-time conditionals. If people used reader conditionals in the specification then the cache is only valid if the result of the reader-conditionals matches. The problem is that the code that doesn’t match the condition is never even read so there is nothing to introspect.</p>
<p>One solution would be tell people not to use reader-conditionals and use some other way of specifying the features required, we would make this a standard and we would have to educate people on how to use it.</p>
<p>But <code class="language-plaintext highlighter-rouge">#+</code> <code class="language-plaintext highlighter-rouge">#-</code> are just reader macros so we could replace them with our own implementation which would work exactly the same except that it would also record the conditions and the results of the conditions.</p>
<p>This turned out to be REALLY easy!</p>
<p>The mapping between a character pattern like <code class="language-plaintext highlighter-rouge">#+</code> and the function it calls is kept in an object called a <code class="language-plaintext highlighter-rouge">readtable</code>. We don’t want to screw up the global readtable so we need our own copy.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> (let ((*readtable* (copy-readtable)))
...)
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">*readtable*</code> is the global variable where the readtable lives, so now an use of the lisp function <code class="language-plaintext highlighter-rouge">read</code> inside the scope of the <code class="language-plaintext highlighter-rouge">let</code> will be using our <code class="language-plaintext highlighter-rouge">readtable</code> (this is effect is thread local).</p>
<p>Next we replace the <code class="language-plaintext highlighter-rouge">#+</code> & <code class="language-plaintext highlighter-rouge">#-</code> reader macros with our own function <code class="language-plaintext highlighter-rouge">my-new-reader-cond-func</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> (set-dispatch-macro-character #\# #\+ my-new-reader-cond-func *readtable*)
(set-dispatch-macro-character #\# #\- my-new-reader-cond-func *readtable*)
</code></pre></div></div>
<p>And that’s it! <code class="language-plaintext highlighter-rouge">my-new-reader-cond-func</code> is actually a closure over an object that the conditions/results are cached into but that’s just boring details.</p>
<p>The point is we can now introspect the reader conditions and know for certain what features were required in the spec file, and we do this without having to add any new practices for library writers.</p>
<p>This the reason for the TLDR at the top:</p>
<blockquote>
<p>If you add powerful features to your language you must be sure to balance them with equally powerful introspection</p>
</blockquote>
<p>Or at least trust programmers with some of the internals so they can get things done. You can totally shoot your foot off with these features, but that ship sailed with macros anyway.</p>
<p>I wrapped all this up in a library you can find here: https://github.com/cbaggers/with-cached-reader-conditionals</p>
<h3 id="other-stuff">Other Stuff</h3>
<p>Aside from this I:</p>
<ul>
<li>
<p>Pushed a whole bunch of code from master to release for <code class="language-plaintext highlighter-rouge">CEPL</code>, <code class="language-plaintext highlighter-rouge">Varjo</code>, <code class="language-plaintext highlighter-rouge">rtg-math</code> & <code class="language-plaintext highlighter-rouge">CEPL.SDL2</code></p>
</li>
<li>
<p>Requested that that new <code class="language-plaintext highlighter-rouge">with-cached-reader-conditionals</code> library and two others (for loading images) are added to quicklisp (the lisp package manager) So more code shipping! yay!</p>
</li>
</ul>
<h3 id="enough-for-now">Enough for now</h3>
<p>Like I said, next week I’ll have people over so I won’t get much done, however my next goals are:</p>
<ul>
<li>
<p>Add groveler caching to the FFI</p>
</li>
<li>
<p>Add functionality to the build system to copy dynamic libraries to the build directory of a compiled lisp program. This will need to handle the odd stuff OSX does with frameworks</p>
</li>
</ul>
<p>Seeya folks, have a great week</p>
Stuck in port2016-08-01T12:08:40+00:00http://www.techsnuffle.com/2016/08/01/stuck-in-port<p>I have been looking at shipping cl code this week, some parts of this are cool, some are depressingly still issues.</p>
<p><code class="language-plaintext highlighter-rouge">save-and-die</code> is essentially pretty easy but as I use C libraries in most my projects it makes things interesting as when the image loads and looks for those shared-objects again the last place it found them. That’s ok when I’m running the project on the same machine but not when the binary is on someone else’s.</p>
<p>The fix for this is to, just before calling <code class="language-plaintext highlighter-rouge">save-and-die</code>, close all the shared-objects, and then make sure I reload them again when the binary starts. This is cool as I can use #’list-foreign-libraires and make local copies of those shared-object files and load them instead. Should be easy right?..Of course not.</p>
<p>OSX has this concept called <a href="https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html">run-path dependent libraries</a>, feel free to go read the details but the long and the short is that we can’t just copy the files as they have information baked in that means it will look for dependent libraries outside of our tidy little folder.</p>
<p>Luckily the most excellent <a href="https://github.com/Shinmera">shinmera</a> has, in the course of wrapping QT run into everything I am and plenty more and dealth with it ages ago. He has given me some tips on how to use <a href="http://thecourtsofchaos.com/2013/09/16/how-to-copy-and-relink-binaries-on-osx/">otool and install_name_tool</a>. Also from talking to Luis on the CFFI team we agree that at least of the unload/reloading stuff could be in that library.</p>
<p>To experiment with this stuff I started writing <a href="https://github.com/cbaggers/shipshape">shipshape</a> which let’s me write <code class="language-plaintext highlighter-rouge">(ship-it :project-name)</code> and it will:</p>
<ul>
<li>compile my code to a binary</li>
<li>pull in whatever media I specify in a ‘manifest’</li>
<li>copy over all the C libraries the project needed and do the magic to make them work.</li>
</ul>
<p>This library works on linux but not OSX for the reasons listed a couple of paragraphs up. However it is my playpen for this topic.</p>
<p>In the process of doing this I had a facepalm moment when I realized I hadn’t looked beyond <code class="language-plaintext highlighter-rouge">trivial-dump-core</code> to see what was already available for handling the dump in a cross platform way. Of course asdf has robust code for this. I felt so dumb that I know so little about ASDF that I resolved to thoroughly read up on it. Fare’s writeups are awesome and I had a great time working through them. The biggest lesson I took away was how massively underspec’d <code class="language-plaintext highlighter-rouge">pathnames</code> are and, whilst uiop does it’s best, there are still paths that cant be represented as <code class="language-plaintext highlighter-rouge">pathnames</code>. Half an hour after telling myself I wouldnt try and make a path library I caught my brain churning on the problem so I started an experiment to make to most minimal reliable system I can, it’s not nearly ready to show yet but the development is happening <a href="https://github.com/cbaggers/pathology">here</a>. The goal is simply to be able to define and create kinds of path based on some very strict limitations, this will no be a general solution for any path, but should be fine for <code class="language-plaintext highlighter-rouge">unix</code> and <code class="language-plaintext highlighter-rouge">ntfs</code> paths. The library will not include anything about the OS, so then I will look at either hooking into something that exists or making a very minimal wrapper over a few functions from libc & kernel32. I wouldnt mind using iolib but the build process assumes a gcc setup which I simply cant assume for my users (or even all my machines).</p>
<p>One thing that I am feeling really good about is that twice in the last week I have sat down to design APIs and it has been totally fluid. I was able to visualize the entire feature before starting and during the implementation very little changed. This is a big personal win for me as I don’t often feel this in any language.</p>
<p>Wow, that was way longer than I expected. I’m done now. Seeya!</p>
A bunch o stuff2016-07-26T13:10:32+00:00http://www.techsnuffle.com/2016/07/26/a-bunch-o-stuff<p>Ok so it’s been a couple of weeks since I wrote anything, sorry about that. 2 weeks back I was at <a href="www.solskogen.no">Solskogen</a> which was awesome but I was exhausted afterwards and never wrote anything up, let’s see if I can summarize a few things.</p>
<h3 id="pbr">PBR</h3>
<p>I’m still struggling with the image based lighting portion of physically based rendering. Ferris came round for one evening and we went through the code with a comb looking for issues. This worked really well and we identified a few things that I had misunderstandings about.</p>
<p>The biggest one (for me) was a terminology issue, namely if an image is in <code class="language-plaintext highlighter-rouge">SRGB</code> then it is non linear, but an image in an <code class="language-plaintext highlighter-rouge">SRGB texture</code> is linear when you sample it. I was just seeing SRGB and thinking ‘linear’ so I was making poor assumptions.</p>
<p>It was hugely helpful to have someone who has a real knowledge and feel for graphics look at what I was doing.</p>
<p>I have managed to track down some issues in the point lighting though so that is looking a little better now. See the image at the top for an example, the fact that the size of the reflection is different on those different materials is what I am happiest to see.</p>
<h3 id="c-dependencies">C Dependencies</h3>
<p>Every language needs to be able to interface with C sooner or later. For lisp it was no different and the libraries for working with shared objects are <strong>fantastic</strong>. However the standard practice in the lisp world (as far as I can see) is not to package the c-libraries with the lisp code but instead to rely on them being on the user’s system.</p>
<p>This is alright on something like linux but the package manager situation leaves a lot to be desired on other platforms. I have been struggling with Brew and MacPorts shipping out of date or plain broken versions of libraries. Of course I could work on packaging these myself but it wouldnt solve the Windows case.</p>
<p>Even on linux sometimes we find that the cadance of the release cycles is too slow for certain libraries the <a href="http://www.assimp.org/">asset importing library</a> I use is out of date in the debian stable repos.</p>
<p>So I need to package C libraries with my lisp libraries, that ok, I can do that. But the next issues is with users shipping their code.</p>
<h3 id="shipping-lisp-with-c-libraries">Shipping Lisp with C Libraries</h3>
<p>Even if we package C libraries with lisp libraries this only fixes things for the developer’s machine. When the dump a binary of their program and give it to a user, that user will need those C libraries too.</p>
<p>The problem is that ‘making a binary’ in lisp generally means ‘saving the image’. This is like taking a snapshot of your program as it was in a particular moment in time..</p>
<p>Every value</p>
<p>Every variable</p>
<p>Every function</p>
<p>..is saved in one big file. When you run that file the program comes back to life and carries on running (some small details here but meh).</p>
<p>The problem is that one of the values that is saved is <em>the location of the c libraries</em> :p</p>
<p>So this is what I am working on right now, a way to take a lisp project, find all the c-libraries it uses, copy them to a directory local to the project and then do some magic to make sure that, when the binary is run on another person’s machine, that it looks in the local directory for the C dependencies.</p>
<p>I think I have 90% of the mechanism worked out last night. Tonight I try and make the bugger work.</p>
<h3 id="misc">Misc</h3>
<ul>
<li>Fixed bugs running CEPL on OSX</li>
<li>Added implementation specific features to lisp FFI library</li>
<li>Add some more functions to <code class="language-plaintext highlighter-rouge">nineveh</code> (which will eventually be a library of helpful gpu functions)</li>
<li>Fix loading of cross cubemap hdr textures</li>
<li>add code to unbind samplers after calling pipelines (is this really neccesary?)</li>
<li>fix id dedup bug in CEPL sampler object abstraction</li>
<li>added support for Glop to CEPL (a lisp library that can create a GL context and windows on various platforms, basically a lisp alternative to what I’m using SDL for)</li>
<li>GL version support for user written gpu functions</li>
<li>super clean syntax for making a fbo with color 6 attachment bound to the images in a cube texture.</li>
<li>support bitwise operators in my gpu functions</li>
<li>Emit code to support GLSL v4.5 implicit casting rules for all GLSL versions</li>
</ul>
<p>And a pile of other bugfixes in various projects</p>
Misc bits2016-07-10T20:39:56+00:00http://www.techsnuffle.com/2016/07/10/misc-bits<p>A few things have been happening</p>
<h3 id="geometry">Geometry</h3>
<p>Fixed a few bugs in CEPL so a user could start making geometry shaders, AND HE DID :) He is using my inline glsl code as well which was nice to see.
Cross platform stuff</p>
<p>Fixed bug in CEPL which resulted from changes to the SDL2 wrapper library I use. The guys making it are awesome but have slightly different philosophy on threading. They want to make it transparent as possible, I want it explicit. Their way is better for starting out or games where you are never going to have to worry about that performance overhead, but I can risk that in CEPL.</p>
<p>The problem that arose was that I was calling their initialization function and was getting additional threads created. To solve this I just called the lower level binding functions myself to initialize sdl2. As ever though huge props to those guys, they are making working with sdl2 great.</p>
<p>I also got a pull request fixing a windows10 issue which I’m super stoked about.</p>
<p>With this CEPL is working on OSX and Windows again</p>
<h3 id="pbr-one-day">PBR one day..</h3>
<p>Crept ever closer to pbr rendering. I am going too damn slowly here. I ran into some issues with my vector-space feature. Currently spaces must be uploaded as uniforms and I don’t have a way to create a new space in the shader code. I also don’t have a way to pass a space from one stage to another. This was an issue as for tangent-space normals you want to make tangent space in the vertex shader and pass it to the fragment shader.</p>
<p>To get past this issue more quickly I added support for the get-transform function on the gpu. The get-transform function takes 2 spaces and returns the matrix4 that transform between them.</p>
<p>This just required modifying the compiler pass that handled space transformation so didnt need much extra code.</p>
<h3 id="filmic-tonemapping">Filmic ToneMapping</h3>
<p>A mate put me onto this awesome page on ‘Filmic Tonemapping Operators’ and I obviously want to support HDR in my projects so I have converted these samples to lisp code. I just noticed that I havent pushed this library online yet, but I will soon.</p>
<h3 id="the-mother-of-all-dumb-fbo-mistakes">The mother of all dumb FBO mistakes</h3>
<p>I have been smacking my head against an issue for days and it turned out to be a user level mistake (I was that user :p).</p>
<p>The setup was a very basic deffered setup, so the first pass was packing the gbuffer, the second shading using that gbuffer. But whilst the first pass appeared to be working when drawing to the screen it was failing when drawing to an FBO, the textures were full of garbage that could only have been random gpu data and only one patch seemed to be getting written into.</p>
<p>Now as I havent done that enough testing on the multi render target code I assumed that it must be broken. Some hours of digging later it wasnt looking hopeful.</p>
<p>I tested on my (older) laptop..and it seemed better! There was still some corruption but less and more of the model showing…weird.</p>
<p>This was also the first time working with half-float textures as a render target, so I assumed I had some mistakes there. More hours later no joy either.</p>
<p>Next I had been fairly sure viewports were involved in this bug somehow (given that some of the image looked correct) but try as I might I could not find the damn bug. I tripple checked the size of all the color textures.. and the formats and the binding unbinding in the abstrations.</p>
<p>Nothing. Nada. Zip</p>
<p>Out of desperation I eventually made an fbo and let CEPL set all the defaults except size…AND IT WORKED…what the fuck?!</p>
<p>I looked back at my code that initialized the FBO and finally saw it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> (make-fbo `(0 :dimensions ,dim :element-type :rgb16f)
`(1 :dimensions ,dim :element-type :rgb16f)
`(2 :dimensions ,dim :element-type :rgb8)
`(3 :dimensions ,dim :element-type :rgb8)
:d))
</code></pre></div></div>
<p>That <code class="language-plaintext highlighter-rouge">:d</code> in there is telling CEPL to make a depth attachment, and to use some sensible defaults. However it also is going to pick a size, which as a default will be the size of the current viewport <code class="language-plaintext highlighter-rouge">*smashes face into table*</code></p>
<p>According to the GL spec:</p>
<blockquote>
<p>If the attachment sizes are not all identical, rendering will
be limited to the largest area that can fit in all of the
attachments (an intersection of rectangles having a lower left
of (0 0) and an upper right of (width height) for each attachment).</p>
</blockquote>
<p>Which explains everything I was seeing.</p>
<p>As it is more usual to make attachments the same size I now require a flag to be set if you want attachments with different sizes along with a big ol’ explanation of this issue in the error message you see if you don’t set the flag.</p>
<h3 id="well">Well..</h3>
<p>With that madness out of the way I fancy a drink. Seeya all later!</p>
Mainlining Videos2016-06-27T10:42:40+00:00http://www.techsnuffle.com/2016/06/27/mainlining-videos<p>It has been a weird weekend. I had to go to hospital for 24 hours and wasnt in a state to be making stuff but I did end up with a lot of time to consume stuff, so I thought I’d list down what I’ve been watching:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=lrrQaa_-hzU">CppCon 2014 Lightning Talks - Ken Smith C Hardware Register Access</a> This was ok I guess it is mainly a way of dressing up register calls so their sytax mirrors their behaviour a bit more. After having worked with macros for so long this just feels kinda sensible and nothing new. Still was worth a peek</li>
<li>Pragmatic Haskell For Beginners - Part 1 (cant find a link for this) - I watched a little of this and it looks like it will be great but I want to watch more fundamentals first and then come back to this.</li>
<li><a href="https://www.youtube.com/watch?v=ZHqFrNyLlpA">JAI: Data-Oriented Demo SOA, composition</a> - Have watched this before but rewatched it to internalize more of his approach. I really am considering implementing something like this for lisp but want to see how many place I can bridge lisp and foreign types in the design. I highly recommend watching his talk on <a href="https://www.youtube.com/watch?v=ciGQCP6HgqI">implicit context</a> as I think the custom allocator scheme plays really well with the data-oriented features (and is something I want to take ideas from too)</li>
<li><a href="https://www.youtube.com/watch?v=5cNyrkjJ5KY">Java byte-code in practice</a> - started watching this one but didnt watch all the way through as not relevent to me right now. I looked at this stuff while I was considering alternate ways to do on-the-fly language bindings generation, but I don’t need this now (I wrote <a href="https://medium.com/@fusetools/a-sane-way-of-mixing-languages-in-fuse-660b351c2f96">a piece our new approach a while back</a>)</li>
<li>Relational Programming in miniKanren by William Byrd <a href="https://www.youtube.com/watch?v=zHov3fKYqBA">Part 1</a> <a href="https://www.youtube.com/watch?v=nFE2E91VDAk">Part 2</a> - This has been on my watch list for ages, a 3 hour intro to mini-kanren. It was ace (if a bit slow moving). Nice to see what the language can and cant do. I’m very interested in using something like this as the logic system in my future projects.</li>
<li><a href="https://www.youtube.com/watch?v=G_eYTctGZw8">Production Prolog</a> - Second time watching this and highly recommended. After looking at mini-kanren I wanted to get a super highlevel feel on prolog again so watched this as a quick refresher of how people use it.</li>
<li><a href="https://www.youtube.com/watch?v=ejFJIAsvdEg">Practical Dependently Typed Racket</a> Wanted to get a feel for what these guys are up to. Was nice to see what battles they are choosing to fight and to get a feel for how you can have a minimal DTS and it still be useful</li>
<li><a href="https://www.youtube.com/watch?v=Iq9DzN6mvYA">Jake Vanderplas - Statistics for Hackers - PyCon 2016</a> - As it says. I feel i’m pitiful when it comes to maths knowledge and I’m very interested in how to leverage what I’m good at to make use of the tools statisticians have. Very simple examples of 3 techniques you can use to get good answers regarding the significance of results.</li>
<li>John Rauser keynote Statistics Without the Agonizing Pain - The above talk was based on this one and it shows, however the above guy had more time and cover more stuff.</li>
<li><a href="https://www.youtube.com/watch?v=Ux0YnVEaI6A">Superoptimizing LLVM</a> - Great talk on how one project is going about finding places in LLVM that could be optimized. Whilst it focuses on LLVM the speaker is open about how this would work for any compiler. Nice to hear how limited their scope was for their first version and how useful it still was. Very good speaker.</li>
<li><a href="https://www.youtube.com/watch?v=SQ7qKKQrSBY">Director Roundtable With Quentin Tarantino, Ridley Scott and More</a> I watched this in one of the gaps when I was letting my brain cool down. Nothing revalutionary here, just nice to hear these guys speak.</li>
<li><a href="https://www.youtube.com/watch?v=GdqC2bVLesQ">Measure for Measure: Quantum Physics and Reality</a> - Another one that has been on my list for a while. A nice approachable chat about some differing approaches to the wave collapse issue in quantum phsyics.</li>
<li><a href="https://www.youtube.com/user/ThoughtSpaceZero/videos">Introduction to Topology</a> This one I gave the most time. I worked through the first 20 videos of this tutorial series and they are FANTASTIC. The reason for looking into this is that I have some theories of the potential of automatic data transformation in the area of generating programs for rendering arbitrary datasets. I had spent an evening dreaming up what roughly I would need and then hada google to see if any math exists in this field. The reason for doing that is that you then know that smart people have proved whether you are wasting your time. The closest things I could find were based in topology (of various forms) so I think I need to understand this stuff.
I’ve been making <a href="https://github.com/cbaggers/topology-notes">some notes</a> so I’m linking them here but don’t bother reading them as they are really only useful to me.</li>
</ul>
<p>That’s more than enough for now, I’m ready to start coding again :p</p>
<p>Peace.</p>
<p>p.s. I also watched ‘The Revenant’ and it’s great. Do watch that film.</p>
Reading on the road2016-06-21T09:02:56+00:00http://www.techsnuffle.com/2016/06/21/reading-on-the-road<p>Hey,</p>
<p>I don’t have anything to show this week as I have been travelling for the last few days. However this has given me loads of time to read so I’ve had my face in PBR articles almost constantly.</p>
<p>I started off with this one <a href="http://www.frostbite.com/wp-content/uploads/2014/11/course_notes_moving_frostbite_to_pbr.pdf">‘moving frostbite to pbr’</a> but whilst it is awesome I found it really hard to understand without knowing more of the fundamentals.</p>
<p>I had already looked at this <a href="allegorithmic's comprehensive PBR guide">https://www.allegorithmic.com/pbr-guide</a> which was a fantastic intro the subject.</p>
<p>After this I flitted between a few more articles but got stuck quite often, the issue I had was finding articles that bridged the gap between theory and practical. The real breakthrough for me was reading these two posts from codinglabs.net:</p>
<ul>
<li><a href="http://www.codinglabs.net/article_physically_based_rendering.aspx">Physically Based Rendering</a></li>
<li><a href="http://www.codinglabs.net/article_physically_based_rendering_cook_torrance.aspx">Physically Based Rendering - Cook–Torrance</a></li>
</ul>
<p>After these two I hada much better feel of what was going on and then was able to get much further in <a href="http://blog.selfshadow.com/publications/s2013-shading-course/karis/s2013_pbs_epic_notes_v2.pdf">this article from Epic on the Unreal Engine’s use of PBR</a></p>
<p>Now <a href="http://blog.selfshadow.com/publications/s2013-shading-course/hoffman/s2013_pbs_physics_math_notes.pdf">this one</a> I probably should have read sooner, but it was still felt good to go through this again with what I had gained from the Epic paper.</p>
<p>And finally I got back to <a href="http://www.frostbite.com/wp-content/uploads/2014/11/course_notes_moving_frostbite_to_pbr.pdf">the frostbite paper</a> which is outstanding but took a while to absorb. I know I’m going to be looking at this one a lot over the coming months.</p>
<p>That’s all from me, seeya folks.</p>
GLSL Spec2016-05-31T10:01:27+00:00http://www.techsnuffle.com/2016/05/31/glsl-spec<p>Man I totally forgot to blog about this here.</p>
<p>I got really annoyed with the glsl spec. It’s full of definitions like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>genType clamp(genType x,
genType minVal,
genType maxVal);
genType clamp(genType x,
float minVal,
float maxVal);
genDType clamp(genDType x,
genDType minVal,
genDType maxVal);
</code></pre></div></div>
<p>Once you know that:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">genType</code> = float, vec2, vec3, vec4</li>
<li><code class="language-plaintext highlighter-rouge">genDType</code> = double, dvec2, dvec3, dvec4</li>
</ul>
<p>It is easy to read as a human, but inaccurate as a machine. The reason is that we see that when we call <code class="language-plaintext highlighter-rouge">clamp</code> with two <code class="language-plaintext highlighter-rouge">float</code>s we get a <code class="language-plaintext highlighter-rouge">float</code>, and when given 2 <code class="language-plaintext highlighter-rouge">vec2</code>s we will get a <code class="language-plaintext highlighter-rouge">vec2</code>. But when trivially parsed it looks like clamp returns some generic type. This is false. Say we have some function <code class="language-plaintext highlighter-rouge">foo(vec2)</code>, in glsl this is legal:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>foo(clamp(vec2(1,2), vec2(2,3));
</code></pre></div></div>
<p>Because the return type of clamp is concrete, it’s only the spec has compressed this information for ease of reading.</p>
<p>This may seem like a really tedious rant, but to me it’s super important as it make it more complicated to use this data, and I havent even started on the fact that the spec is only available as inconsistantly formatted html man pages or PDF.</p>
<p>What I wanted was a really specification for the functions and variables in glsl. Every single overload, specified in the types of the language.</p>
<p>The result of this need is the <a href="https://github.com/cbaggers/glsl-spec">glsl-spec project</a> which has exactly what I wanted. Every function, every variable, specified using GLSL types, with GL version info, and available as s-expressions & json.</p>
<p>Let’s go make more things</p>
<p>Peace</p>
Compiler usability & overloading2016-05-31T10:00:27+00:00http://www.techsnuffle.com/2016/05/31/compiler-usability--overloading-<p>This last week has been fairly chill.</p>
<p>Whilst I was at the conference I had a couple of folks who do graphics research take an interest in CEPL and so I decided I should put a little time into making the compiler a little easier to use. The result is a function called v-compile that takes the code for a stage as a list and returns the compiled result. Using it looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VARJO> (v-compile '((a :float)) :330
:vertex '(((pos :vec3))
(values (v! pos 1) a))
:geometry '(((hmm (:float *)))
1.0)
:fragment '(((hmm :float))
(labels ((fun ((x :float))
(* x x)))
(v! 1.0 1.0 hmm (fun a)))))
</code></pre></div></div>
<p>The first two arguments are the uniforms <code class="language-plaintext highlighter-rouge">((a :float))</code> (in this case one float called <code class="language-plaintext highlighter-rouge">a</code>) and the glsl version you are using (<code class="language-plaintext highlighter-rouge">330</code>)</p>
<p>You specify the stage as a keyword and then provide a list. The first element of the list is the list of arguments to that stage. e.g. <code class="language-plaintext highlighter-rouge">((pos :vec3))</code> the rest of the list is the code for that stage e.g. <code class="language-plaintext highlighter-rouge">(values (v! pos 1) a)</code></p>
<p>I also took all of the work I did expanding the glsl spec and use it in the compiler now. At compile time my compiler reads the glsl-spec and populates itself with all the function and variable definitions. This also means that varjo now works for all version of glsl YAY!</p>
<p>I also added <strong>very</strong> tentative support for geometry and tesselation stages. I didnt have time to learn the spec well enough to make my compiler check the interface properly, but instead it just does very basic checks and gives a warnign that you should be careful.</p>
<p>Finally I made it easy to add new functions to the compiler and made CEPL support gpu-function overloading. So now the following works.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g my-sqrt ((a :int))
(sqrt a))
(defun-g my-sqrt ((a :vec2))
(v! (sqrt (x a)) (sqrt (y a))))
</code></pre></div></div>
<p>Seeya folks</p>
Woops, wrong about the samplers2016-04-02T15:58:21+00:00http://www.techsnuffle.com/2016/04/02/woops-wrong-about-the-samplers<p>hehe turns out in my refactoring hubris I had forgotten how my samplers worked. THey are actually used like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-sampling ((*tex* *sam*))
(map-g #'prog-1 *stream* :tex *tex*))
</code></pre></div></div>
<p>Which is more reasonable :) I still want to change it though</p>
So close to docs!2016-03-31T11:14:19+00:00http://www.techsnuffle.com/2016/03/31/so-close-to-docs<p>I got so close :D</p>
<p>I’ve been working on <a href="../cepl/api.html">these docs for CEPL</a> over the easter break, and it’s been great, got so much of the API cleaned up in the process.</p>
<p>When you own an entire codebase, writing documentation is probably the best way to get total overview of your api again. Also writing docs isnt fun, but writing docs for an api you dont like, which you wrote and have full control over is horrifying… so you fix it and then the codebase is better.</p>
<p>So aye, docs are good an I was within 2 macros of finishing the documentation. Specifically I was documenting <code class="language-plaintext highlighter-rouge">defpipeline</code> which was hard.. which annoyed me as, if I cant explain it, it must be to complicated for users. So I broke out the whiteboard and started picking the things about it that I liked. I came up with:</p>
<ul>
<li>automatic uniform arguments (you dont have to add the arguments to the signature as the compiler knows them)</li>
<li>very fast blending parameters (It does some preprocessing on the params to make sending them to gl fast)</li>
<li>local fbos (you can define fbos that belong to the pipeline)</li>
<li>all the with-fbo-bound stuff is written for you.</li>
</ul>
<p>After a lot of mumbling to myself I found that, with the exception of ‘automatic uniform arguments’, all of these advantages could be has without needing <code class="language-plaintext highlighter-rouge">defpipeline</code> to support composing other pipelines.</p>
<p>Local FBOs was basically a closure, but the problem was that the gl context wouldnt be available when the closure vars were initialized. To fix this I will make <em>all</em> CEPL gpu types be created in an uninitialized state, capturing their arguments in a callback that will be run as soon as the context is available. As a side effect this should mean that gpu-types can now be used from defvar defparameter etc with no issues, which will be lovely.</p>
<p>with-fbo-bound will still exist but we will add two things. First we will have <code class="language-plaintext highlighter-rouge">map-g-into</code> which is like <code class="language-plaintext highlighter-rouge">map-g</code> except it takes a render-target (not an fbo, more on that in a minute). <code class="language-plaintext highlighter-rouge">map-g-into</code> will just bind the target and do a normal map-g. This make the code’s intent clearer that it was in the old <code class="language-plaintext highlighter-rouge">defpipeline</code> syntax.</p>
<p>fast blending parameters was interesting. To cache the blending details in a foreign array (which allowed fast upload) we need somewhere to keep the cache. You also want to be able to swap out blending params easily on an fbo’s attachments. This resulted in the idea of adding <code class="language-plaintext highlighter-rouge">render-target</code> This is a simple structure that holds an fbo a list of attachments to render into and the blending params for the target and (optionally) attachments. Blending param will be remove from the fbos as it lives on the <code class="language-plaintext highlighter-rouge">render-target</code> instead. This means you can have multiple <code class="language-plaintext highlighter-rouge">render-target</code>s that are backed by the same fbo but set to render into different attachments with different blending params. <strong>Much</strong> better and we can keep the speed as we can cache the foreign-array on the <code class="language-plaintext highlighter-rouge">render-target</code>.</p>
<p>All this stuff got me thinking and I realised that <code class="language-plaintext highlighter-rouge">with-sampler</code> is terrible. Here is how it’s used:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-sampler sampler-obj
(map-g #'pipeline stream :tex some-texture))
</code></pre></div></div>
<p>Now with two textures:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-sampler sampler-obj
(map-g #'pipeline stream :tex some-texture :tex2 some-other-tex))
</code></pre></div></div>
<p>The sampler applies to both! How are you meant to sample different textures differently? I was so fucking dumb for not seeing this when I made it. Ok so the <code class="language-plaintext highlighter-rouge">with-</code> thing is a no go. That means we should have samplers that (like render-targets) reference the texture and hold the sampling params. There will be a little trickery behind the scenes so that seperate samplers with the same attributes actually share the <code class="language-plaintext highlighter-rouge">gl sampler object</code> but itherwise will be fairly straightforward.</p>
<p>The advantage is that suddenly the api very behaviorally consistant.</p>
<p>You make gpu-arrays to hold data and then gpu-streams to pass the data to pipelines
You make textures to hold data and then samplers to pass the data to pipelines
You make fbos to take data and then render-targets to pass data from pipelines</p>
<p>This feels like progress. Time to code it up…and fix all the docs</p>
Getting CEPL into Quicklisp2016-03-29T15:14:54+00:00http://www.techsnuffle.com/2016/03/29/getting-cepl-into-quicklisp<p>Sorry I havent been posting for ages. There has been plenty of progress but I have been lax writing it up here.</p>
<p>The biggest news up front is that CEPL is now in quicklisp, this has been a long time coming but now it’s much now easier to get CEPL set up and running.</p>
<p>To do this means cleaning up all the cruft and experiments that were in the CEPL repo. The result is that like GL CEPL has a host that provides the context and window, the event system is it’s own project, as does the camera, and much else is cleaned and trimmed.</p>
<p>CEPL can now potentially have more hosts than just SDL2 though today it remains the only one. I really want to get the GLOP host written so we can have one less C dependancy on windows and linux.</p>
<p>More news coming :)</p>
OH hell yes!2016-01-27T01:05:56+00:00http://www.techsnuffle.com/2016/01/27/oh-hell-yes<p><img src="/assets/images/s8.png" /></p>
<p>The code one the left is the render pipeline for the image on the right. All can be editted live. I love this.</p>
<p>Also using the new <code class="language-plaintext highlighter-rouge">spaces</code> feature. More details coming soon!</p>
<p>p.s. It’s only super basic shading but that fact the features are working means that I can focus on the fun stuff!</p>
Hammering out the api2016-01-11T18:58:19+00:00http://www.techsnuffle.com/2016/01/11/hammering-out-the-api<h3 id="where-we-left-off">Where we left off</h3>
<p>As I said the other day I have #’get-transform working, albeit inefficiently, and after that it was time to think about how these features should be used. Of course we have <code class="language-plaintext highlighter-rouge">in</code> blocks, but there were questions (that I mentioned last time) like:</p>
<ul>
<li>What happens if you try to pass a type like position or space (that don’t exist at runtime) from one shader stage to another?</li>
<li>Can these kind of types be used in gpu-structs?</li>
<li>Should stages have an implicit space?</li>
<li>If you pass up a position as a uniform, where does it’s space come from?</li>
</ul>
<p>Questions like this have been taking a lot of time to sort out. It’s one of those funny things that, when the language doesn’t limit you and your goal is ‘the best’ programming experience, then any of hundreds of crazy solutions are possible so choosing becomes harder.</p>
<h3 id="maybe-this">Maybe this</h3>
<p>One possibility was to make space a first class concept in cepl, so all gpu-arrays and such would have a space. That would mean that I could have the terrain vertex data in a gpu-array in world space. e.g.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> (make-gpu-array terrain-data :type 'terrain-vert :space world-space)
</code></pre></div></div>
<p>and then the vertex shader could technically look like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> (defun-g terrain-vertex-shader ((vert terrain-vert))
(values (pos vert)
(normal vert)
(uv vert)))
</code></pre></div></div>
<p>Now! because the vertex shader implicitly takes place in clip space and cepl knows that the vert data is in world-space then it could add the world->clip space transforms for you.</p>
<p>This sounds kinda awesome, and with cepl I can totally make this work… but I decided against it. Why? Well because it bakes a certain use-case into a data-structure, here is a simple example where it breaks down.</p>
<p>I have the data for a bush, I will instance this a thousand times across a landscape, each tree will have a different model->world transform. At this point what is the space associated with the bush data doing? It’s not needed yet it’s ‘attached’ to the data.</p>
<p>An hour of wrangling around that idea made it seem like this idea just added complexity so, for now at least, it’s out.</p>
<p>A few more ideas ending up in a similar place. Namely:</p>
<blockquote>
<p>cepl, in some places, is reaching the limit of how much it can help without becoming an engine.</p>
</blockquote>
<p>this is an interesting feeling in itself.</p>
<p>So after all that I decided the healthiest thing was to leave that for a bit and focus on the cpu side of the space feature.</p>
<h3 id="goblins-of-the-brain">Goblins of the brain</h3>
<p>One thing that was odd was that I had this nagging feeling that I had <em>something</em> wrong with my concept of how spaces should be organized so I’ve been staring at books & tutorials again trying to find the something I had missed. Quite suddenly the following popped into my head:</p>
<blockquote>
<p>I have assumed that all spaces have one parent. I have conflated hierarchical and non-hierarchical relationships</p>
</blockquote>
<p>And now I need to try and explain what I mean by that :)</p>
<p>The whole point of making a space graph rather than a scene graph was to avoid the idea that game entities would be part of the graph, which I believe is a massive code-smell.</p>
<p>I want only spaces to be in the graph, so far so good.</p>
<p>I define my spaces with some transforms relative to a parent space. Seems OK. But is it? I realized I ran into some issues when looking at the eye-space to clip-space relationship.</p>
<p>There isn’t a clear parent. Changing the position of the eye (game camera) doesn’t transform clip-space in the way moving a table affects the objects on the table. Also defining eye-space as a child of clip space feels weird.</p>
<p>What I had was what I’m calling a non-hierarchical relationship. There <strong>is</strong> a valid transform between them but it isn’t defined in terms of a parent-child relationship. Examples of hierarchical things like arms and hands, where moving the arm moves the hand.</p>
<p>So what I want now is:</p>
<ul>
<li>lots of space-trees, which are hierarchical. They are defined in a parent-child way and are great for character limbs, foliage branches etc.</li>
<li>non-hierarchical relationships between the root nodes of those space-trees.</li>
</ul>
<p>So is this special? Nope :) It more my journey than something totally new. The first part of the definition from Wikipedia says</p>
<blockquote>
<p>A tree node (in the overall tree structure of the scene graph) may have many children but often only a single parent</p>
</blockquote>
<p>but later clarifies:</p>
<blockquote>
<p>It also happens that in some scene graphs, a node can have a relation to any node including itself, or at least an extension that refers to another node.</p>
</blockquote>
<p>OpenSceneGraph has multiple parents…which to me sounds like a bit of a terminology mistake, what does a spatial hierarchy with many parents mean?</p>
<p>Also having graph pointers sounds scary, they would have to be quite strict in what they can do..and if they have strict behavior then surely there can be a better term.</p>
<p>Some other engines avoid the issue somewhat by keeping the scene graph to the hierarchical stuff and let the subject of clip-space and such be a concern of shaders. My brief look a Unity seems to suggest this approach.</p>
<p>Whilst that last one avoids confusion at first it feels like you kind of only defer the issue to another part of the code-base. I would hope we could come up with some analogy in code that extends across these domains. This may not be as easy at first but should be simpler in the end.</p>
<h3 id="and-now">And Now</h3>
<p>Since that realization I’ve been sketching out my plan for how to make this. It isnt a massive overhaul which is nice and there seem to be some obvious places to start optimizing later on.</p>
<p>For now I’m just going to get something slow working and play with it to see what happens, but I have to say I feel like a mental blockage has been free’d and I’m pretty optimistic about what comes next.</p>
Space progress2016-01-09T11:48:04+00:00http://www.techsnuffle.com/2016/01/09/space-progress<p>Having got something working on the shader generation side I have gone back to the cpu side of the story.</p>
<p>When the shader compiler sees a position changing space like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(in s
(in w
(p! (v! 1 1 1 1)))
0)
</code></pre></div></div>
<p>it turns it into glsl something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>W-TO-S * vec4(1,1,1,1)
</code></pre></div></div>
<p>Where W-TO-S is a matrix4 which is to be uploaded. On the cpu side cepl adds something like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((w-to-s (spaces:get-transform w s)))
(cepl-gl::uniform-matrix-4ft w-to-s-uniform-location w-to-s))
</code></pre></div></div>
<p>Up until now the <code class="language-plaintext highlighter-rouge">get-transform</code> function has just returned the identity matrix, so it was time to flesh that out.</p>
<p>I now have a implementation that looks like it works but is incredibly inefficient. It will do the job for now, and as cepl is already uploading the result, the feature should be working now.</p>
<p>This leaves the question of what good code looks like when using this feature. I have a few things in my mind already that need answering:</p>
<ul>
<li>What happens if you try to pass a ephemeral type like position or space from one shader stage to another?</li>
<li>Should stages have an implicit space (I think so)</li>
<li>Can ephemeral types be used in gpu-structs (I think this should be made to work)</li>
<li>If you pass up a position as a uniform, where does it’s space come from?</li>
<li>If you made a gpu-struct slot a position, should the space come from the stage?</li>
<li>If there are implicit spaces in shaders then we need implicit uniform upload. Do we just use the implicit-uniforms feature for this? How does that interact with the compiler’s space-transform pass.</li>
</ul>
<p>This should keep me busy :)</p>
I dun a artikale2016-01-08T13:49:18+00:00http://www.techsnuffle.com/2016/01/08/i-dun-a-artikale<p>I work on <a href="https://www.fusetools.com/">Fuse</a> as my day job (which is awesome) and have to deal a lot with interop with java/objc (which are not :D).</p>
<p>In the course of making, remaking, refining and smashing-my-head-against bindings I came to some conclusion about bindings. This article is about the work Olle (another awesome fuse employee) and myself did and what the first result looks like.</p>
<p><a href="https://medium.com/@fusetools/a-sane-way-of-mixing-languages-in-fuse-660b351c2f96?utm_source=slack&utm_medium=slack&utm_campaign=foreigncodeannouncement">https://medium.com/@fusetools/a-sane-way-of-mixing-languages-in-fuse-660b351c2f96?utm_source=slack&utm_medium=slack&utm_campaign=foreigncodeannouncement</a></p>
Pro-crastinating2016-01-04T23:10:18+00:00http://www.techsnuffle.com/2016/01/04/procrastinating<ul>
<li>Found some bug</li>
<li>looked at behind the scenes code</li>
<li>it looked complex</li>
<li>procrastinated by writing doc strings for bunch of functions</li>
<li>now understand the process, know where to start fixing the issue.</li>
<li>regret doing this less often</li>
</ul>
Booya! Space checking lives!2016-01-03T17:47:20+00:00http://www.techsnuffle.com/2016/01/03/booya-space-checking-lives<p>Oh progress feels so good!</p>
<p>Ok, now given some code like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g vert ((vert pos-col) &uniform (s space-g) (w space-g))
(in s
(in w
(p! (v! 1 1 1 1)))
0)
(values (v! (pos vert) 1.0)
(col vert)))
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">in</code> portion does nothing useful, it purly exists so that there is a <code class="language-plaintext highlighter-rouge">position</code> (the <code class="language-plaintext highlighter-rouge">p!</code>) which is created inside the scope of the space <code class="language-plaintext highlighter-rouge">w</code>. That <code class="language-plaintext highlighter-rouge">position</code> is returned into the scope of space <code class="language-plaintext highlighter-rouge">s</code>. To ensure this is valid the compile needs to multiple the position by some kind of <code class="language-plaintext highlighter-rouge">w space to s space</code> matrix.</p>
<p>Lets look at the compile output (it’s been cleaned up for readability)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#version 330
layout(location = 0) in vec3 fk_vert_position;
layout(location = 1) in vec4 fk_vert_color;
out vec4 OUT_34V;
uniform mat4 W-TO-S;
void main() {
(W-TO-S * vec4(1,1,1,1));
0;
gl_Position = vec4(fk_vert_position,1.0f);
OUT_34V = fk_vert_color;
}
</code></pre></div></div>
<p>Here we can see that there is no mention of <code class="language-plaintext highlighter-rouge">w</code>, <code class="language-plaintext highlighter-rouge">s</code>, <code class="language-plaintext highlighter-rouge">position</code>s or <code class="language-plaintext highlighter-rouge">spaces</code>. The compiler has created a uniform to hold the <code class="language-plaintext highlighter-rouge">w space to s space</code> transform and multiplied it with the vec4.</p>
<p>On the cpu side we can find this code</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
(let ((w-to-s (spaces:get-transform w s)))
(when w-to-s
(let ((cepl-gl::val w-to-s))
(when (>= w-to-s-uniform-location 0)
(cepl-gl::uniform-matrix-4ft w-to-s-uniform-location cepl-gl::val)))))
...
</code></pre></div></div>
<p>Here we see that the original spaces <code class="language-plaintext highlighter-rouge">w</code> & <code class="language-plaintext highlighter-rouge">s</code> are passed to the <code class="language-plaintext highlighter-rouge">get-transform</code> function which returns the matrix4 which is uploaded.</p>
<p>As you would imagine there are still plenty of little bugs to clean up, but those feel like nothing compared to how much work this has taken. I’m pretty stoked right now.</p>
<p>I’m looking forward to posting a proper example of how you would use this in regular shaders soon.</p>
<p>That’s enough for this week</p>
<p>Ciao!</p>
Spaces still in progress2015-12-09T23:48:23+00:00http://www.techsnuffle.com/2015/12/09/spaces-still-in-progress<p>Bloody hell that month went fast. Well the obvious is that the spaces feature is not done. I lost a week an a half to business, but it was in palo alto so I got to visit the computer history museum which was badass.</p>
<p>I’m also stoked at how much I did get done in that time. Varjo progress is marching onwards and the parts of cepl I found hairy are slowly becoming less so.</p>
<p>The big changes are happening in varjo purely because varjo is outgrowing the simple model that has served it up to now. Right now I’m making sure that a varjo compile pass not only gives you the result (and metadata) but also includes the ast. Varjo is currently single pass (and that was fine) but cepl wants to be able to use varjo in a multipass fashion. It needs the ast so it can create the varjo code for the second pass (where, in this case, spaces are turned into matrices).</p>
<p>It is likely that, in time, varjo will just become properly multipass. But I’m going to do this in steps as I don’t want to be working on features for features sake, I want the space feature to drive this.</p>
<p>More news as it comes</p>
Flow analyzer complete enough2015-11-23T10:12:42+00:00http://www.techsnuffle.com/2015/11/23/flow-analyzer-complete-enough<p>So this week I got the flow analyzer working WOO! There are a few cases it’s missing but the hard stuff (iteration) is done and there’s enough that I can start looking at how to use it</p>
<p>So a refresh:</p>
<ul>
<li>On the cpu side I have a tree of transforms representing different spaces (like a lightweight scenegraph)</li>
<li>I want to use these in a sane way on the gpu</li>
<li>I cant afford any runtime logic becuase then we are pissing performance up a wall.</li>
<li>Matrices describe transforms between spaces. We are interested in the spaces yet we have no explicit way of handling them.</li>
<li>If we could use spaces in our shaders and at compile time turn them into matrices (that are then uploaded as uniforms) then things would be much clearer.</li>
<li>To be able to resolve this at compile time we need a flow analyzer…which we now have, yay!</li>
</ul>
<p>So the next question was how to use this.</p>
<p><code class="language-plaintext highlighter-rouge">position</code> is a point (vector) with a <code class="language-plaintext highlighter-rouge">space</code>. Position is a compile time construct which will be compiled to a regular vector</p>
<p><code class="language-plaintext highlighter-rouge">space</code> will be the name of the type. Spaces will be passed up as uniforms</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g test ((vert :vec4) &uniform (world space))
...)
</code></pre></div></div>
<p>I was imagining a <code class="language-plaintext highlighter-rouge">transform</code> function that let you transform a position from one space to another:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(transform pos world-space clip-space)
</code></pre></div></div>
<p>This is kinda dumb though as the position knows what space it is in. So maybe</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(transform pos clip-space)
</code></pre></div></div>
<p>Hmm, might be ok. What would be sweet though is a scope where everything inside is guaranteed to be in a certain space</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(in world-space
(some code)
(more code))
</code></pre></div></div>
<p>wait though, this makes <code class="language-plaintext highlighter-rouge">transform</code> redundant as</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(in clip-space pos)
</code></pre></div></div>
<p>Would transform the pos to clip-space, and it would return as it’s the last thing in the scope. Sweet!</p>
<p><code class="language-plaintext highlighter-rouge">in</code> is also nicer to write than transform so I think we have a winner.</p>
<p>Note on making positions: in cepl/varjo vectors are made with the <code class="language-plaintext highlighter-rouge">v!</code> function. As in <code class="language-plaintext highlighter-rouge">(v! 0 1 2)</code> returns a vector3. Position will have a similar interface but it will expect a space, either as an argument or taken from the current <code class="language-plaintext highlighter-rouge">in</code> scope. So <code class="language-plaintext highlighter-rouge">(p! 0 1 2 world-space)</code> gives you a position3 in world space. <code class="language-plaintext highlighter-rouge">(in world-space (p! 0 1 2))</code> does the same (I may kill the first version so <code class="language-plaintext highlighter-rouge">in</code> is the only way to do it)</p>
<p>So assuming this worked how does cepl use it?</p>
<p>My current thinking is that cepl will iterate over the function calls looking for calls involving spaces and positions. It then works out the matrices needed for each calls, de-duplicates the result and adds them as uniforms. Intermediate calculations need to be added to the source code (which I havent worked out the details for yet) and then the new lisp shader code (now without any spaces or positions) is recompiled. The result of this second pass the the shader uploaded to the gpu.</p>
<p>I’m off to america this week so I have no idea if I will get any of this done. We shall see :)</p>
The flow analyzer LIVES (a bit)2015-11-17T01:38:55+00:00http://www.techsnuffle.com/2015/11/17/the-flow-analyzer-lives-a-bit<p>Had a couple of days off coding but now I’m back. Yet again Ι found limits in the my compiler’s architecture, this time in how I handled environments.</p>
<p>So before what Ι had was roughly as follows:</p>
<p>I have a tree of code that needs compiling. The compiler walks down the tree and compiles each form. The environment is simply an object that holds all the variables and functions that exist at that point in the program. To implement variables scope I simply copy the environment as it was at the start of the scope, add the new variable/s and then throw it away at the end. Because the stuff outside the scope only see’s the original environment it couldnt access variables from the inner scope.</p>
<p>This has worked great for ages but now by flow analyzer wants the answer to the question “what scope was this variable made in?” and that data has been thrown away.</p>
<p>So now I’ve switched to a tree of environments, and they are now all immutable. This was a damn big change so Ι took the opportunity to start using a testing framework so that I could build basic tests while getting everything working again. This went really well so now Ι can show the following.</p>
<p>Let’s take this code. It’s useless but you can see that we make two vectors (that’s the <code class="language-plaintext highlighter-rouge">v!</code> forms) each starting with x</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defshader test ()
(let ((x 1))
(v! x 0)
(progn
(let ((z 3))
x))
(v! x 0 0 0)))
</code></pre></div></div>
<p>Now, if I let the flow analyzer do it’s work and inspect the data for the <code class="language-plaintext highlighter-rouge">v!</code> calls, I get something like.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(V! 514 515)
(V! 514 518 519 520)
</code></pre></div></div>
<p>The numbers are called <code class="language-plaintext highlighter-rouge">flow-id</code>s, they are used to track the flow of a value through a program. If two are the same then they are the same value (even if they get bound to different variables or passed through functions).</p>
<p>So above we can see that <code class="language-plaintext highlighter-rouge">514</code> appears as the id of the first arg for both <code class="language-plaintext highlighter-rouge">v!</code> forms, that means it’s not just the same variable (<code class="language-plaintext highlighter-rouge">x</code>) but the same value too.</p>
<p>Now in the following example we set <code class="language-plaintext highlighter-rouge">x</code> to be a different value (it’s in a progn and a let so I could see if it would track the flow of execution properly).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defshader test ()
(let ((x 1))
(v! x 0)
(progn
(let ((z 3))
(setq x z)))
(v! x 0 0 0)))
</code></pre></div></div>
<p>The flow analysis of the <code class="language-plaintext highlighter-rouge">v!</code> calls in this program look like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(V! 506 507)
(V! 509 510 511 522)
</code></pre></div></div>
<p>This time the <code class="language-plaintext highlighter-rouge">flow-id</code>s of the first argument are different. This shows that, even though they both take <code class="language-plaintext highlighter-rouge">x</code>, the system knows the value has been changed.</p>
<p>With these fixes Ι can finally handle conditions as the trees of environments are trivial to diff, so Ι will be able to combine the flow-ids from the two execution paths.</p>
<p>Thanks for reading!</p>
(setf mayhem damnit)2015-11-10T20:54:27+00:00http://www.techsnuffle.com/2015/11/10/setf-mayhem-damnit<p>When I was making varjo I hacked in support for setf very quickly. It gave the rough feeling of <a href="http://www.lispworks.com/documentation/HyperSpec/Body/05_aa.htm">generalized references</a> but even at the time I was thinking that this hack would come back to kick me in the arse..Turns out yesterday it took a run up first..</p>
<p>So I had to take a detour from my detour writing the flow analyzer to rewrite setf. Luckily that wasnt too bad and the code makes way more sense now. So tonight I’m getting back to the flow analyzer and looking at <code class="language-plaintext highlighter-rouge">if</code> expressions.</p>
<p>These are a bit fiddly as you (usually) cant tell ahead of time which result is going to be returned. So from the point of the flow analyzer we will say they both do. So if:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((a 2)
(b 3))
(if (something-something-stuff)
a
b))
</code></pre></div></div>
<p>Then we have to assume both <code class="language-plaintext highlighter-rouge">a</code> & <code class="language-plaintext highlighter-rouge">b</code> return and continue accordingly.</p>
<p>But wait, what if there are side effects in there?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((c 2))
(if (something-something-stuff)
(setf c 10)
(setf c 20))
c)
</code></pre></div></div>
<p>Then we have to be aware that the setf is in a condition when we update the flow information.</p>
<p>It’s probably something people who know this kind of thing think is trivial but the way I’ve implemented this so far wasnt done in a way that took this into account. Still exciting though and will be cool when Ι crack it.</p>
<p>Back with more soon</p>
Stumpwm invert colors of one window2015-11-09T20:47:25+00:00http://www.techsnuffle.com/2015/11/09/stumpwm-invert-colors-of-one-window<p>One of the last things I was missing on my new stumpwm setup was being able to invert the colors of a single window. Turns out this is not hard to do.</p>
<ol>
<li>Make a script called <code class="language-plaintext highlighter-rouge">inv.sh</code> containing the following</li>
</ol>
<p>#! /bin/bash</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if [ "$(pidof compton)" ];
then
pkill compton
else
xdotool getactivewindow | xargs -I {} compton --backend glx --invert-color-include 'client={}'
fi
exit
</code></pre></div></div>
<ol>
<li>
<p><code class="language-plaintext highlighter-rouge">chmod +x inv.sh</code></p>
</li>
<li>
<p>Add the following to your .stumpwmrc</p>
<p>(define-key <em>root-map</em> (kbd “C-i”) “exec /home/baggers/Code/shell/inv.sh”)</p>
</li>
</ol>
<p>Boom, done. When you click <code class="language-plaintext highlighter-rouge"><your-prefix> C-i</code> the currently focused windom will be inverted. Running it again disables the effect.</p>
<p>This is pretty hacky and I’d like to make something more integrated with stumpwm so Ι can have multiple inverted windows at a time. That will be easy enough when it comes to it, but for now this does the job.</p>
nanowrimo & flow analysis2015-11-09T00:20:13+00:00http://www.techsnuffle.com/2015/11/09/nanowrimo--flow-analysis<p>So more progress today. It’s too late to explain this well but Ιm gonna try dashing it out anyway.</p>
<p>So rendering involves matrices everywhere, a matrix describes a transform, and we often define spaces in terms of a transform and a parent space. The odd thing however is that the spaces involved arent stored with the transform so the context has to be maintained in the programmers head.</p>
<p>This isnt entirely accurate as on the cpu side we may have a matrix stack or a scene graph that does maintain this. But we dont have that context on the gpu (and dont want pay the overhead of it anyway).</p>
<p>The question of whether this could be done in a nice way was what prompted by blogpost yesterday.</p>
<p>I started by assuming Ι could have spaces on the gpu but that Ι wanted to be able to upload on the transforms that were needed. This meant knowing which ‘spaces’ were used in which functions as <code class="language-plaintext highlighter-rouge">(transform space-x space-y vec4)</code> needs a different matrix from <code class="language-plaintext highlighter-rouge">(transform space-y space-x vec4)</code>. This very quickly made me realize that I need some kind of flow analysis in varjo; I need to know where all the values can flow through the system.</p>
<p>Today I got the basics of this working and am going to finish it off over the next few days. I will then use this as a way to statically check the validity of transforms in shaders, which should make more of the shaders intention more transparent. Finally as these primitives will only exist at compile time, we get no performance cost at runtime (as by then its just matrices) woo!</p>
<p>right time for bed,
Seeya</p>
make-load-form2015-11-09T00:06:19+00:00http://www.techsnuffle.com/2015/11/09/makeloadform<p>This post is more of reminder to myself than anything else.
Just now Ι made a little macro that not only emitted regular code (symbols and lists) but also had an instance of a class in there. This caused an error Ι hadnt seen before:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>; caught ERROR:
; don't know how to dump #<FLOW-IDENTIFIER :ids (-2905)> (default MAKE-LOAD-FORM method called).
</code></pre></div></div>
<p>A quick google found <a href="http://coding.derkeiler.com/Archive/Lisp/comp.lang.lisp/2011-03/msg00539.html">this</a> this the excellent answer by Mario:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The problem you are having is due to the fact your the macro not only
creates code, but that this code has actual live objects embedded into
them in this case. That's the def-collection thing. That is not
necessarily a problem, but in this case it is because sbcl does not know
how to save such an object in a fasl. You can either figure out how to
make an appropriate load form, or do something that avoids the use of
the actual live object.
</code></pre></div></div>
<p>Intriguing, so ‘load forms’ but be a concept in lisp. Let’s have a look:</p>
<p><a href="http://clhs.lisp.se/Body/f_mk_ld_.htm#make-load-form">CLHS: make-load-form</a></p>
<p>Cool, so it seems that the mechanism for definign how literals are serialized and deserialized by the compiler (sorry if those terms are inaccurate for this lisp feature) and it also seems this can apply to structures and standard objects if you implement make-load-form:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>structure, standard-object: Objects of type structure-object and
standard-object may appear in compiled constants if there is an
appropriate make-load-form method defined for that type.
The file compiler calls make-load-form on any object that is
referenced as a literal object if the object is a generalized instance
of standard-object, structure-object, condition, or any of a (possibly
empty) implementation-dependent set of other classes. The file
compiler only calls make-load-form once for any given object within a
single file.
</code></pre></div></div>
<p>I still feel amazed at how well common lisp turned out, I get lovely suprises like this all the time as I get deeper into the language.</p>
<p>Anyway I need to read deeper into this, looks like fun</p>
Controlling the compiler is AWESOME2015-11-07T20:06:42+00:00http://www.techsnuffle.com/2015/11/07/controlling-the-compiler-is-awesome<p>I really should have gotten used to this idea, with the amount of macro stuff Ι’ve been doing with lisp, but being able to mess around with compiler is just fun.</p>
<p>In the process of my nanowrimo project on making ‘spaces’ for cepl Ι’ve been looking into how these will be handled on the gpu. I want to be able to reason about these more explicitly than just transforming some vector with some matrix, but at runtime Ι need to keep things as simple and with as little indirection as possible.</p>
<p>Whilst on the cpu side I’m living in a (mostly) dynamically typed language, on the gpu I’m fully static. This opens up some possibilities for using/abusing my type system to check these things for me. I am able to add types that exist purely so the typechecker can validate what I’m doing and that will just compile down to a regular glsl vector4.</p>
<p>So one example I could try is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((pos (p! some-vec world-space)))
(transform pos view-space))
</code></pre></div></div>
<p>Where p! is a function that returns a value with type ‘position-world-space’. I can then use the metadata produced by the compiler to work out what matrices (for the various spaces) need to be uploaded. I can also ensure you don’t do something potentially meaningless (like adding two vectors in different spaces) without first indication that that was what you intended.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((pos-1 (p! some-vec world-space))
(pos-2 (p! some-vec clip-space)))
(+ pos-1 pos-2))
</code></pre></div></div>
<p>This is not very interesting though as it’s not normally the mistake you make. Something more interesting would be when you are in a fragment shader (which is implicitly screen space) and you want to do some post-proc lighting based in clip-space. You could use a simple block like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(in-space clip-space
..your code..)
</code></pre></div></div>
<p>And you could be sure that any ‘position’ that was used inside would be transformed to the correct space prior to the calculations being carried out. Any time the spaces match it’s a no-op so no conversion code is emitted.</p>
<p>I’m thinking that a chunk to this can be implemented with the compiler the way it is, but for the bits that can’t it means I have to add and expose some new compiler features, which is cool.</p>
<p>So that’s why I’m excited by compilers today :D</p>
<p>Back to work!</p>
<p>p.s. Of course Ι am missing out a bunch of details especially that at least the names of the spaces would have to be statically defined. But ho hum, it will make sense soon :)</p>
Fantastic series on Clojure persistent datastructure internals2015-11-07T16:19:59+00:00http://www.techsnuffle.com/2015/11/07/fantastic-series-on-clojure-persistent-datastructure-internals<p>I’ve just had the joy of stumbling onto this series by <a href="http://hypirion.com/about-me">Jean Niklas L’orange</a> about clojure persistent datastructure. I had gained a basic understanding of the theory of these before now but this series of posts break it all down really well, including the optimizations that make this datastructure (more) practical.</p>
<p><a href="http://hypirion.com/musings/understanding-persistent-vector-pt-1">Understanding Persistent Vector - Part 1</a>
<a href="http://hypirion.com/musings/understanding-persistent-vector-pt-2">Understanding Persistent Vector - Part 2</a>
<a href="http://hypirion.com/musings/understanding-persistent-vector-pt-3">Understanding Persistent Vector - Part 3</a>
<a href="http://hypirion.com/musings/understanding-clojure-transients">Understanding Clojure Transients</a></p>
<p>Not only is fantastically written but Ι believe you could work through this series and write a decent persistent vector implementation by the end of it.</p>
<p>Top notch!</p>
NanoWrimo-22015-11-03T21:48:29+00:00http://www.techsnuffle.com/2015/11/03/nanowrimo2<p>This is just a quick check in to log some progress.</p>
<p>Tonight I’ve been poking around in the current event system, mainly it was cleanup but Ι also added the ability to have weak & strong subscriptions to a event node. The weak ones were neccesary to allow GC to cleanup things that were created at the REPL and are now orphaned. However for well written code that guarentees nodes are unsubscribed, the overhead of weakrefs is undesirable.</p>
<p>Currently nodes can also have two functions applied to them, a filter (which can stop an event propagating) and a body (which is just run with the event as an argument). These were added in a hurry so I need to go back and turn these into something sensible. I guess I will just combine them and have the result of the body be an event or nil, events get propagated, nil doesn’t.</p>
<p>We shall see.</p>
NanoWrimo1 - This month's project2015-11-02T21:25:04+00:00http://www.techsnuffle.com/2015/11/02/nanowrimo1--this-months-project<p>Right, in this post I want to go over what I’m going to be making this month, but first a summary of day 0.</p>
<p>Rewriting <a href="https://www.shadertoy.com/view/ltlXDr">backscatter</a> in cepl turned out to be a <em>really</em> good way to find bugs in the compiler, it was depressingly good at finding them actually. The changes have made the struct handling in varjo much better and made the function dependency code a lot better in cepl.</p>
<p>Whilst the compiler is now producing good glsl, the pipeline crashes hard as soon as you use it :p. Oddly enough commenting out a couple of the uniforms being passed to the pipeline stops it crashing, so I’m assuming I have either screwed up the uniform uploading, or it didn’t work before in examples like this.</p>
<p>Either way it is not the primary focus for this month so I am forcing myself to leave it as it is for now and concentrate on nanowrimo.</p>
<p>On that note let’s talk about that :)</p>
<p>The project for this month is to add a repl friendly, gc’able, optimizable way of managing hierarchies of transforms in cepl. If this sounds like a scene-graph then that’s no surprise, but I’m consciously trying to avoid putting game objects in the graph directly. By this Ι mean Ι don’t want to have some ‘game-object’ type that you have to subclass and that then goes in the graph.</p>
<p>What I would prefer is some means of defining spaces and then having objects subscribe to them. This decouples objects from the means of organizing them.</p>
<p>In this system it is normally wise to have some kind of caching system. This pays off when there are multiple spaces that share a parent. For example if you are calculating the position of each finger on a hand you dont need to recompute the position of the hand each time. Another property that melds well with this is that in many cases you can compute the transforms lazily. You dont need to calculate a parent transform unless the child needs it (or it is needed itself).</p>
<p>Slam this all together and you get a kind of event system. One that caches the latest event rather than propagating them and can be resolved by pulling.</p>
<p>So far so good, but there is one was case I’m interested in. It may be necessary to walk many of the spaces. To that end it would be good to have a flush event that would push all the pending events through the system.</p>
<p>The finale: Input events are just events which always flush. This means it may be possible to merge these event systems to produce a unified interface. One which would allow subscribing to any event. This is basically some odd rx hybrid (with less state safety). Care will have to be taken around the boundaries between different style nodes
but more safe systems can be built on top of this (or it can be ignored and just used to feed some higher level system)</p>
<p>This may end up not working out but it’s the experiment for the month and we shall see how it goes :)</p>
<p>Time to go watch a bunch of talks on streams/frp/etc to steal ideas. Wooo!</p>
More dead bugs2015-11-01T22:23:12+00:00http://www.techsnuffle.com/2015/11/01/more-dead-bugs<p>Got him! turns out there was a bug in the code that merged compilation environements. One of those things telling me Ι need to have better tests in this code.</p>
<p>Anyhoo, that’s done. back to business.</p>
Compiler bugs2015-11-01T18:54:27+00:00http://www.techsnuffle.com/2015/11/01/compiler-bugs-332432423<p>Given this gpu struct:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defstruct-g material ()
(additive-color :vec3 :accessor additive-color)
(diffuse-color :vec3 :accessor diffuse-color)
(specular-color :vec3 :accessor specular-color)
(specular-exponent :float :accessor specular-exponent)
(background-ammount :float :accessor background-ammount))
</code></pre></div></div>
<p>Only the call the <code class="language-plaintext highlighter-rouge">#'yy</code> breaks</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(varjo::defshader marg ((y material))
(labels ((xx () (additive-color y))
(yy ((x material)) (additive-color x)))
(additive-color y) ;; this works
(xx) ;; so does this
(yy y)) ;; but this doesnt
(v! 0 0 0 0))
</code></pre></div></div>
<p>Wat?!</p>
<p>In real news I’m catching a good few bugs in varjo today. Can’t wait to squash this one though :p</p>
NanoWrimo-02015-11-01T13:55:50+00:00http://www.techsnuffle.com/2015/11/01/nanowrimo0<p>First day of November, first day of my first <a href="http://nanowrimo.org/">NanoWriMo</a>. Traditionally this is a challenge to write a 50000 word novel in a month but I’m going to be coding along with this instead. As word/line counts are pointless in code I’m going to try do some significant work on cepl.</p>
<p>I’m going to aim to make a space-graph (analogous to a scenegraph) I’ll be writing this up as I go so Ι won’t burden this post with it.</p>
<p>Today however Ι am bug hunting by taking a rather nice shader called <a href="https://www.shadertoy.com/view/ltlXDr">backscatter</a> from shadertoy and remaking it in cepl. It was written by a good friend who tells me it’s too slow, which may be fair but it is also clearly written and pretty enough to be motivating.</p>
<p>writing it has already let me find a few bugs, including one in Varjo’s structs so I am off to fix that now :)</p>
Events - Rewritten again2015-10-26T10:39:18+00:00http://www.techsnuffle.com/2015/10/26/events--rewritten-again<p>Phew, now <a href="http://www.fusetools.com">Fuse</a> has shipped the beta I’m not crunching so much at work so I’m back working on cepl again. (Check out Fuse btw, it’s early days but theres some kickass shit in there, I’ll be doing some posts on it in future)</p>
<p>I’ve not a nice branch full of changes to merge in soon in which I’ve been working on two main things:</p>
<ol>
<li>
<p>Better error messages
This took a lot more work than just tweaking the strings. I’ve gone though varjo (the lisp->glsl compiler) trying to find places where I can make suggestions of possible correct alternatives. The obvious example for this is variable names so if you type the wrong name varjo will make suggestions based on names that are significantly similar (I’m just using jaro-winkler-distance for this).
In cepl it also involved going through the more hairy code, trying to seperate it into more sensible functions and also making sure I give more specific errors especially around the code that handles converting between texture-formats & lisp types.</p>
</li>
<li>
<p>New event system
So I changed this yet again :p. The reasons for doing this have mainly been around limiting scope. I dont want unnecesarily high level abstractions in cepl core, I want ‘low enough’ abstractions that are easy to build on.</p>
</li>
</ol>
<p>So this time the main change is that event listeners are remvoed in favour of event-nodes.
Event nodes can subscribe to other event nodes to recieve their events. Any event node can also be subscribed to. Each node holds a list of weak refs to the node that are subscribed to it, and a list of strong refs to the nodes that it subscribes to. In this way we have a event graph that the garbage collector can clean up if you don’t unsubscribe explicitly.</p>
<p>“Why add this weak ref complexity” would be a fair question to ask, and the reason is making a balance between good practice and pragmatism. Common Lisp allows awesome levels experimentation with the repl and incremental compiling. It is inevitable that, whilst explicitly disconnecting event-nodes is the correct thing to do, managing that is a PITA when you are just trying to play with ideas. To this end we make sure we can GC away parts of the event graph that are no longer needed, letting you focus on the important stuff. This kind of thing is what I mean by ‘low level enough’, I’m not trying to bake in some glitch free reactive programming framework (though those are awesome), but I still want to make the experience good and lispy.</p>
<p>Less importantly, with the new changes events are now structs with read only fields.</p>
<p>The driving force for the above changes has been that I now have event sources other than SDL2. I have made a simple android app that sends touch events over sockets to cepl to allow more direct interaction with whatever I’m making. It’s early days but the events from the cepl remote-control are now flowing through the event system just like mouse & keyboard events so I’m pretty stoked :)</p>
<p>This physical hardware business makes recording youtube videos more tricky so I’m gonna go buy some extra cameras today so I can try show this stuff off.</p>
<p>Hopefully I’ll be back soon with a new video for you.
Ciao</p>
Excellent CRISPR Introduction2015-07-10T12:18:16+00:00http://www.techsnuffle.com/2015/07/10/excellent-crispr-introduction<p>Not sure how long until we will be programming biology but it doesnt hurt to start knowing about the technology which will make it robust. CRISPR is an interesting topic in how it works and also with it’s reliability (relative to previous methods).</p>
<p>Andrew Gibiansky has writting an excellent post on CRISPR which was very easy to understand, <a href="http://andrew.gibiansky.com/blog/genetics/crispr/">check it out here</a></p>
Sometimes...2015-07-06T12:23:48+00:00http://www.techsnuffle.com/2015/07/06/sometimes<p>…the most terrifying thing code can do is run whilst you looks at some part of it so terrible that you cant understand how it even started. It’s like cutting someone in half and they walk off and try and carry on their day.</p>
Better Error Messages2015-07-06T11:11:57+00:00http://www.techsnuffle.com/2015/07/06/better-error-messages<p>Been a bit quiet recently but yesterday I spent the day working on conditionals in cepl.</p>
<p>There are simply too many cases where the error is cryptic, or fails way later in the process that it should have.</p>
<p>Defpipeline should now through more sensible errors now and varjo will try and suggest types if you use one it doesn’t know. e.g.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VARJO> (defshader test ((x sampler2d))
(texture x (v! 0 0)))
</code></pre></div></div>
<p>gives</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Varjo: Varjo: Could not find the correct type for type-spec SAMPLER2D
Perhaps you meant one of these types?:
:usampler-2d
:sampler-3d
:sampler-2d-shadow
:sampler-2d-rect
:sampler-2d-ms
:sampler-2d-array
:sampler-2d
v-sampler-2d
:sampler-1d
:isampler-2d
:sampler
[Condition of type UNKNOWN-TYPE-SPEC]
Restarts:
0: [RETRY] Retry SLIME REPL evaluation request.
1: [*ABORT] Return to SLIME's top level.
2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1005078003}>)
</code></pre></div></div>
<p>I have finished cleaning up type errors for textures and will be focusing on c-arrays & fbos after this.</p>
<p>That should hopefully stop the cases where Ι assume that cepl is buggy rather than checking my own typos :D</p>
Tweak paredit so it doesnt put spaces before parens if they follow certain chars2015-06-26T16:47:58+00:00http://www.techsnuffle.com/2015/06/26/tweak-paredit-so-it-doesnt-put-spaces-before-parens-if-they-follow-certain-chars<p>I quickly hacked this together to fix fixes a frustration where trying to write <code class="language-plaintext highlighter-rouge">#a(1 2 3)</code> gets ‘corrected’ by paredit to <code class="language-plaintext highlighter-rouge">#a (1 2 3)</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defvar paredit-dont-add-space-after
'((?s ?#) (?a ?#) (?λ) (?- ?#) (?+ ?#) (?c ?#) (?o ?#) (?p ?#) (?r ?#) (?s ?#) (?x ?#)))
(add-hook 'paredit-mode-hook
(lambda ()
(setq paredit-space-for-delimiter-predicates
(list (lambda (endp delimiter)
(if (not endp)
(save-excursion
(let ((1-back (char-before))
(2-back (char-before (- (point) 1))))
(null
(cl-loop for (1b 2b) in paredit-dont-add-space-after thereis
(and 1-back (char-equal 1b 1-back)
(if 2b
(when 2-back
(char-equal 2b 2-back))
t))))))
t))))))
</code></pre></div></div>
Machine Readable GLSL Spec2015-06-22T08:27:04+00:00http://www.techsnuffle.com/2015/06/22/machine-readable-glsl-spec<p>TLDR: <a href="https://github.com/cbaggers/gl-to-sexp/blob/master/spec/glsl">Look Here</a></p>
<p>There was one thing during making my glsl compiler that was terrible; One thing worse than any of the bugs I have hammered out so far.</p>
<p>The GLSL Spec</p>
<p>The spec is available either as pdf or html. It is designed for humans to read but I needed my program to know about every function in glsl. To that end I spent days processing the contents of the pdf, and some other open code, in order to get a list of all the functions with their return & argument types.</p>
<p>Since then I have wanted to update all the functions with information on which versions they are available from, but this has seemed a daunting undertaking.</p>
<p>Just the other day however, some people have put together a new gl documentation site <a href="http://docs.gl">http://docs.gl</a>, whilst the information itself still is geared to be human readable it is at least better laid out and stored in a github repo.</p>
<p>I took a few hours over the weekend to knock up a script that would extract the basic info I need from the glsl spec. Potentially I could do the opengl one too, but for now that is not urgent. You can find the results <a href="https://github.com/cbaggers/gl-to-sexp/blob/master/spec/glsl">here</a>.</p>
<p>The data is one big s-expression which contains all the glsl functions and variables.</p>
<p>#Layout</p>
<p>each element looks something like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(((("EmitStreamVertex" "void") (("stream" "int"))))
(("EmitStreamVertex" NIL NIL NIL NIL NIL NIL :400 :410 :420 :430
:440 :450)))
</code></pre></div></div>
<p>Each item has two elements:</p>
<ul>
<li>A list of function definitions</li>
<li>A list of information on which version are supported</li>
</ul>
<h3 id="function-definition">Function definition</h3>
<p>Each function definitions is laid out as follows:</p>
<ul>
<li>The first element is a pair of the function name as a string and the return type as a string.</li>
<li>The rest of the list are pairs of the argument name as a string and the argument type as a string</li>
</ul>
<h3 id="version-information">Version Information</h3>
<p>Each element in the version info list is laid out as follows:</p>
<ul>
<li>The first element is the name of the function and in some cases enough arg info to differentiate it from other incarnations of the same function. This is one string which sucks, so I need to parse this so we can apply the version info directly.</li>
<li>The rest of the elements are the versions supported. It is either <code class="language-plaintext highlighter-rouge">nil</code> or a keyword specifying the version. The <code class="language-plaintext highlighter-rouge">nil</code>s are becuase the orginal tables were rows of ticks of dashes to indicate versions supported. I turned the ticks into the version keyword and the dashes into <code class="language-plaintext highlighter-rouge">nil</code>. Really I need to remove the <code class="language-plaintext highlighter-rouge">nil</code>s and I will do that very soon.</li>
</ul>
<h2 id="glsl-variables">GLSL Variables</h2>
<p>In the event that the function definition list is <code class="language-plaintext highlighter-rouge">nil</code> then the entry is a variable. You will find it’s name as the first element in the version table along with the versions that support it.</p>
<p>Right I hope this helps. This will be in a state of flux for a while as I also want to extract all documentation for each entry in glsl and I’m sure there will be tweaks to be made.</p>
<p>I Hope it helps someone else.</p>
New videos coming soon!2015-06-12T09:40:59+00:00http://www.techsnuffle.com/2015/06/12/new-videos-coming-soon<p>Last week was a good week for making videos, and cepl in general. With the big merge, primitive osx support and sampler objects it has been feeling like cepl is progressing very quickly all of a sudden.</p>
<p>I have had to throw away a couple of videos I was going to release however, as they are now very out of date. I had hoped to get full blend-mode support in last night but as I failed at that it will likely be merged in in the next 3 days. The next videos in the pipeline will be on sampler-objects and blend-modes, I will the re-record my “Intro to CEPL for people who know GL”, as this was one of the casualties of the last week’s progress.</p>
<p>Thanks to anyone who is reading this, Ciao</p>
Sampler Objects2015-06-11T11:09:24+00:00http://www.techsnuffle.com/2015/06/11/sampler-objects<p>Textures in opengl have parameters that affect how they are sampled. They dictate what happens when they are magnified or minified, what happens if you sample outside the texture etc.</p>
<p>Sampler objects allow you to override the sampling params in a texture. One sampler object can be used on multiple textures in a single draw call.</p>
<p>As of last night these are now available in cepl if <code class="language-plaintext highlighter-rouge">(>= (version-float *gl-context*) 3.3))</code>.</p>
<p>You can use them like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defvar sampler (make-sampler :wrap #(:repeat :clamp-to-edge :repeat)))
(with-sampling ((tex sampler))
(map-g #'some-pipeline some-stream :tx tex))
</code></pre></div></div>
<p>With sampling overrides a texture’s sampling params with the sampler object for the duration of the scope. The texture then reverts to it’s own sampling parameters.</p>
<p>The functions that are used on samplers to change their parameters also work on textures so:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf (lod-bias some-texture) 0.5)
(setf (lod-bias some-sampler) 0.5)
</code></pre></div></div>
<p>are both valid.</p>
<p>Next up are blend-modes. I have had to hold off making new videos for a few days as they are going out of date so fast as we keep getting new features.</p>
OSX SUPPORT WOOO!2015-06-05T08:58:14+00:00http://www.techsnuffle.com/2015/06/05/osx-support-wooo<p>IT FINALLY FUCKING WORKS!</p>
<p><img src="/assets/images/osx.png" /></p>
<p>Video coming saturday explaining in detail how to get set up on osx but the tldr is:</p>
<ul>
<li>run osx-sbcl-launch.sh from the terminal</li>
<li>slime-connect</li>
<li>(in-package :cepl)</li>
</ul>
<p>Back soon!</p>
Progress2015-03-10T02:22:53+00:00http://www.techsnuffle.com/2015/03/10/progress<p>It’s buggy as hell right now but I’ve been working on a new way (for cepl) to write and compose shaders.</p>
<p>You now write gpu functions and compose them using defpipeline, the gpu functions can be used as stages or as regular functions and varjo will compile them correctly for each task. We also have lost explicit return in favour of implicit tail returns, just like in regular common lisp.</p>
<p>A quick code sample</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun-g v ((vert g-pc))
(values (v! (cgl:pos vert) 1.0)
(:smooth (cgl:col vert))))
(defun-g f ((color :vec4))
color)
(defpipeline prog-1 (g-> v f))
</code></pre></div></div>
<p>Compiles to</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// vertex shader
#version 330
layout(location = 0) in vec3 fk_vert_position;
layout(location = 1) in vec4 fk_vert_color;
smooth out vec4 out_86v;
void main() {
vec4 return1;
vec4 v_tmp_83v_84v = vec4(fk_vert_position,1.0f);
return1 = fk_vert_color;
gl_Position = v_tmp_83v_84v;
out_86v = return1;
}
// fragment shader
#version 330
in vec4 out_86v;
layout(location = 0) out vec4 output_color_87v;
void main() {
output_color_87v = out_86v;
}
</code></pre></div></div>
An out of control reddit post2015-02-24T09:32:48+00:00http://www.techsnuffle.com/2015/02/24/an-out-of-control-reddit-post<p>I tried to write a short reply to someone on extending a language with macros…I failed…it got long</p>
<p>This was the thread: http://www.reddit.com/r/lisp/comments/2wy18r/is_this_what_people_mean_when_they_say_lisp/</p>
<hr />
<p>[note: this assumes you’re a lisp newbie, sorry if this is patronising]
Is it what ‘THEY’ mean when they say Lisp allows you to invent your own language. No, not really. But it is nice to be able to use symbols normally deemed off limits right? But syntax is part of language so what you are doing is in the sphere of language. Let’s have a look at a macro:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defmacro fn^ (name args &body body)
`(defun ,name ,args
,@body))
</code></pre></div></div>
<p>Ok so first let’s see what it does and then how it does it.</p>
<p>A macro is a function that runs at a different time, weird right?! It the above kind is run <em>before</em> your program is compiled. the arguments will be source code and the return value will be new code to go in place of the old code. So this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(fn^ some-func (x) (* x 10))
</code></pre></div></div>
<p>-becomes-</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defun some-func (x) (* x 10))
</code></pre></div></div>
<p>See what happened there? The macro was give the following arguments:
name -> some-func
args -> (x)
body -> ((* x 10))</p>
<p>if not you may want to read up on ‘backquote syntax’, it is (aproximately) a very tidy way of splicing lists and data together.</p>
<p>Ok so now we have functions and this kind of macro (yes there are other kinds) we can image that we redefine every common contruct in common lisp.</p>
<p>lets say: let becomes val<-, make-hash-table becomes hsh^,</p>
<p>(f^ fnc (%)
(val<- ((x (hsh^)) (y :dflt))
(<> % x)))</p>
<p>Well it’s clearly still lisp but it’s very unfamilar (and ugly! :D). While this may affect language design it doesnt feel like a new language. Let’s do something else with these macros</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defmacro val<- (&rest form)
(let* ((body (last form))
(bindings (butlast form))
(b (loop for i below (1- (length bindings)) by 2 collect
(list (elt bindings i) (elt bindings (1+ i))))))
`(let ,b
,@body)))
</code></pre></div></div>
<p>so now</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(val<- ((x (hsh^)) (y :dflt))
(<> % x))
</code></pre></div></div>
<p>could be writen as</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(val<- x (hsh^) y :dflt
(<> % x))
</code></pre></div></div>
<p>OK! so now this is different. This feels more like changing the language as we messing with how the code behaves.
The issue is…now we are messing with language we not only have the normal problems a programmer deals with but also with those of a language designer. These are issues of feel and experience as well as functionality (it’s very like api design but with much deeper implications), the answers are often subjective and based on what the language is designed to solve. A big one for me is ‘Does the programmer have to think more to acheive something using this’.</p>
<p>In my own project I use macros to make defining glsl shaders feel like defining regular lisp functions.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defshader some-name ((x :int))
(+ x 10))
</code></pre></div></div>
<p>And translate the lisp code into glsl for the programmer.</p>
<p>We have only scratched the surface of what we can do in this little blogpost. There are also other kinds of macros, like reader macros for example. Reader macros are function that run before the macros we have seen already. They don’t recieve the code as lists of symbols, they recieve the actual characters from the code you wrote. They can allow for cool and often crazy stuff, all the of literal syntax you see around lisp is made usign reader macros: ‘ ` , #() . etc are all made possible using reader macros, and you can extend that in any way you can concieve.</p>
<p>I hope this was of some use. Sorry it turned into a whole post rather than a comment!</p>
<p>p.s.
Also check out:
http://enthusiasm.cozy.org/archives/2013/07/optima <- macros give CL pattern matching
https://gist.github.com/chaitanyagupta/9324402 <- Info on reader macros
Look symbol-macrolet <- this type of macro swaps out a symbol for another whole form</p>
Cepl Video Update2015-02-04T23:18:58+00:00http://www.techsnuffle.com/2015/02/04/cepl-video-update<p>Just stuck a new video on youtube showing a bit of multipass rendering, normal mapping and refraction.</p>
<iframe width="420" height="315" src="https://www.youtube.com/embed/yqCQ7etGnhM" frameborder="0"> </iframe>
What a fucking suprise2015-01-16T17:10:56+00:00http://www.techsnuffle.com/2015/01/16/what-a-fucking-suprise<p><a href="http://www.belfasttelegraph.co.uk/news/local-national/northern-ireland/bbc-uses-ripa-terrorism-laws-to-catch-tv-licence-fee-dodgers-in-northern-ireland-30911647.html">BBC uses RIPA terrorism laws to catch TV licence fee dodgers in Northern Ireland</a></p>
RE: Your Content Feed Is Broken2015-01-16T09:33:18+00:00http://www.techsnuffle.com/2015/01/16/re-your-content-feed-is-broken<p><a href="https://medium.com/@tyrale/your-content-feed-is-broken-f8c6576077c2">https://medium.com/@tyrale/your-content-feed-is-broken-f8c6576077c2</a></p>
<p>“Our Point of Acknowledgment is the moment when we (the user) understand what we are viewing and can then recall that this information has been consumed. It is a continuous moving variable that helps us track where we are in time.”</p>
<p>Time tracking doesn’t require acknowledging every point though, just enough discrete points that we get a sense of moving forward, coupled with the critical assurance that the points will not rearrange themselves.</p>
<p>Movie and book analogies are broken, both experiences are designed knowing the content volume is fixed (designed is a very generous term if you are dealing with dvd/blueray UIs .. or most devices for hard media for that matter)</p>
<p>Interesting concept but naive implementation just optimizes for a different view habit, the author acknowledges this by saying:
“Each feed will need to choose an appropriate location to place new users.” - this is a non-trivial problem, deciding a user’s content priority is hard, if you are picking one and dictating then we are just setting things up so that another well intentioned blog post like this will come up saying ‘wouldn’t it be great to have our media presented like X’</p>
<p>“whining as loud as the rest of you” - Dismissing issues as whining is a massive fallacy. User experience is not just design, it isn’t just entry points, it covers the gamut of ways the user and the medium interact; experience is not a one way phenomenon.</p>
<p>How you treat people’s data is part of experience as if you fuck up they experience it. Whether it is losing credit card details or leaking browsing habits, the experience lives beyond a page view if you choose to retain data.</p>
<p>The fundamental we are dealing with, once again is, ‘how do I track my position in a world changing to rapidly to comprehend alone?’.</p>
<p>This question remains open and is hard, I believe these problems are solvable with going down the road of companies/websites/etc saying ‘well with just one more bit of data we can make your life better, wait…no just two more bits’.</p>
Eval into file2015-01-08T00:28:51+00:00http://www.techsnuffle.com/2015/01/08/eval-into-file<p>Just wrote a handy function for working with slime.</p>
<ul>
<li>It takes the top level form at that point e.g. <code class="language-plaintext highlighter-rouge">(make-instance 'test)</code></li>
<li>Wraps it in a defparameter with a new var name <code class="language-plaintext highlighter-rouge">(defparameter <iv-0> (make-instance 'test))</code></li>
<li>evals it using slime and injects the var name (in this case <code class="language-plaintext highlighter-rouge"><iv-0></code>) into the file just after the toplevel form</li>
</ul>
<p>This is useful when sketching out ideas in a file and you want to compile something but also capture the result in a global for messing with later</p>
<p>running it 3 times would give you something like</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(make-instance 'test)
<iv-2> <iv-1> <iv-0>
</code></pre></div></div>
<p>which makes it very easy to wrap a list around and throw it where you see fit.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defvar eval-into-file-count 0)
(defun slime-eval-into-file ()
"Evaluate the current toplevel form.
store the result in a new global and insert the
var into the code"
(interactive)
(let* ((form (slime-defun-at-point))
(var-name (concat "<iv-" (number-to-string eval-into-file-count) ">"))
(form-with-var (concat "(defparameter " var-name form ")")))
(setq eval-into-file-count (+ eval-into-file-count 1))
(end-of-defun)
(slime-eval-async `(swank:eval-and-grab-output ,form-with-var)
(lambda (result)
(cl-destructuring-bind (output value) result
(push-mark)
(insert value " "))))))
</code></pre></div></div>
<p>Ciao</p>
Hither and thither2014-12-04T23:38:30+00:00http://www.techsnuffle.com/2014/12/04/hither-and-thither<p>Ah one thing I need to nail before geometry shaders is better support for FBOs as I remembered I need a way to draw to mutiple textures from one shader.</p>
To out or not to out2014-12-03T23:49:36+00:00http://www.techsnuffle.com/2014/12/03/to-out-or-not-to-out<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defpipeline prog-1 ((vert vert-data))
(:vertex (setf gl-position (pos vert))
(out (the-color :smooth) (col vert)))
(:fragment (let ((lerp-value (/ (y gl-frag-coord) 500.0)))
(out outputColor (mix the-color
(v! 0.2 0.2 0.2 1.0)
lerp-value)))))
</code></pre></div></div>
<p>Currently setting the out vars of a shader stage is done with the out special form. The above code compiles to</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// vertex shader
#version 330
layout(location = 0) in vec4 fk_vert_position;
layout(location = 1) in vec4 fk_vert_colour;
smooth out vec4 the_color;
void main() {
gl_Position = fk_vert_position;
the_color = fk_vert_colour;
}
// fragment shader
#version 330
smooth in vec4 the_color;
out vec4 outputcolor;
void main() {
float lerp_value_4v = (gl_FragCoord.y / 500.0f);
outputcolor = mix(the_color,vec4(0.2f,0.2f,0.2f,1.0f),lerp_value_4v);
}
</code></pre></div></div>
<p>Notice how the out form essentially compiles to a global var and a setf.</p>
<p>In the quest for lispyness, should this stay as it is? There isn’t anything wrong with setf per se, however it is worth investigating other forms to see if anything surprises us…and by us I mean me :)</p>
<p>Here is a version with a version of values that allows naming</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defpipeline prog-1 ((vert vert-data))
(:vertex (values (gl-position (pos vert))
((the-color :smooth) (col vert))))
(:fragment
(values (outputColor
(let ((lerp-value (/ (y gl-frag-coord) 500.0)))
(mix the-color
(v! 0.2 0.2 0.2 1.0)
lerp-value))))))
</code></pre></div></div>
<p>Eh..the vertex shader is ok but this introduces an extra level of indentation that looks ugly</p>
<p>Ok so how about setf style? Setf allows a name-value style to set multiple variables</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(setf a 1
b 2)
</code></pre></div></div>
<p>So for the shaders that is…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defpipeline prog-1 ((vert vert-data))
(:vertex (values gl-position (pos vert)
(the-color :smooth) (col vert)))
(:fragment
(values
outputColor
(let ((lerp-value (/ (y gl-frag-coord) 500.0)))
(mix the-color
(v! 0.2 0.2 0.2 1.0)
lerp-value)))))
</code></pre></div></div>
<p>.. nope, doesn’t feel better aesthetically and now makes identifying the val and name forms more difficult.</p>
<p>Declare style?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defpipeline prog-1 ((vert vert-data))
(:vertex
(declare (out gl-position (the-color :smooth)))
(values (pos vert) (col vert)))
(:fragment
(declare (out outputColor))
(values (let ((lerp-value (/ (y gl-frag-coord) 500.0)))
(mix the-color
(v! 0.2 0.2 0.2 1.0)
lerp-value)))))
</code></pre></div></div>
<p>..Interesting, yesterday I was looking at declare for types, so if that was the case then using this would feel natural.</p>
<p>Lets see what happens if the types are in declare too</p>
<p>(defpipeline prog-1 (vert)
(:vertex
(declare (vert-data vert) (out gl-position (the-color :smooth)))
(values (pos vert) (col vert)))
(:fragment
(declare (out outputColor))
(values (let ((lerp-value (/ (y gl-frag-coord) 500.0)))
(mix the-color
(v! 0.2 0.2 0.2 1.0)
lerp-value)))))</p>
<p>Haha, not much as almost everything here was already being done with inference.</p>
<p>There is still an issue though that is best summed up in Erik Naggum’s quote on c++. For my purposes the fact that this is about c++ is irrelevant.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>C++ is philosophically and cognitively unsound as it forces a violation
of all known epistemological processes on the programmer. as a language,
it requires you to specify in great detail what you do not know in order
to obtain the experience necessary to learn it. -- Erik Naggum
</code></pre></div></div>
<p>Declaring types in advanced forces the programmer to make an assertion about the nature of the code not yet written.
This violates the concept of live coding as we are coding, not because we have the solution, but because we are looking for one.</p>
<p>Now obviously with coding in the restricted environment of the GPU types are currently a necessity so the best we can do is minimize the cognitive damage. Other than having inference everywhere we possibly can, the other way is to delay specifying the type until it is actually needed, in most cases with the arg definition.</p>
<p>Hmm so no progress on this but I’m glad I’ve been able to empty my head a little.</p>
<p>Goodnight</p>
Sane defaults for Uniforms2014-12-03T08:24:49+00:00http://www.techsnuffle.com/2014/12/03/sane-defaults-for-uniforms<p>OK so at the end of the last post I started musing about having sensible defaults for uniforms values. The logic behind this follows:</p>
<p>So we have a shader ‘vert’</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defvshader vert ((position :vec4) &uniform (i :int) (loop :float))
(let ((pos (v! (* (s~ position :xyz) 0.3) 1.0)))
(setf gl-position (+ pos (calc-offset (+ (float i)) loop)) )))
</code></pre></div></div>
<p>And now I want to add a new uniform: (thing :int)</p>
<p>So do I add the new argument to vert, compile and it freaks out as it has no value? Or do I add it to the calls to vert first and watch it freak out as the shader doesn’t take this?</p>
<p>Both modes suck, and it obviously makes sense to modify the shader first, but on first compile have some default value that won’t cause the shader to panic.</p>
<p>This is easy for numbers, vectors and matrices as all have a concept of an identity, but what about structs?</p>
<p>Well all gl-structs ultimately have to be composed of types glsl understands, therefore we could have the defglstruct method generate an identity value we can use.</p>
<p>What about textures? Do we create a default unit texture for each sampler type? Could do. And we don’t have to worry about ‘index out of bounds’ as by default indexes just wrap for textures. (this is not true for texelFetch, the result is undefined, but shouldnt crash so will but enough time to provide real values)</p>
<p>according to here: http://stackoverflow.com/questions/6354208/glgentextures-is-there-a-limit-to-the-number-of-textures</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"The only limit of glGenTextures is given by the bit width of the texture name (GLint), which is 32 bit;"
</code></pre></div></div>
<p>Which means one (1x1..n) texture for each of the 18 sampler types shouldn’t be an issue.</p>
<p>Now what about uniform buffers? I guess I will want something similar to gl-structs, but as I haven’t implemented UBOs yet I won’t worry about it until I am.</p>
<p>Right, back soon!</p>
<p>Ciao</p>
Soon there will be a new Stage2014-12-02T23:50:22+00:00http://www.techsnuffle.com/2014/12/02/soon-there-will-be-a-new-stage<p>After having merged in all the nice changes to foreign arrays I have decided to start looking at what is probably the biggest hole in the glsl compiler right now, Geometry Shaders.</p>
<p>I’m beavering away on some refactoring and code cleanup to make this tractable, which is going rather well. There should be no major changes to the compiler needed as I added all the geometry shader specific functions ages ago. I think it will mainly be glsl ‘out’ variable type transformations. I will also need to be able to specify the geometry type (tris, points, lines) in the context as that logic will be needed for proving correctness in the geometry stage.</p>
<p>I am not too fussed about adding Tessellation shaders urgently, mainly because I haven’t learned how to use them yet.</p>
<p>I am also musing over the idea of supporting the use of ‘declare’ to specify shader argument types and the context. So that the following</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defvshader test ((verts pnc) &uniform (i :int) (origin :vec3)
(cam-to-world :mat4) &context :version 330)
...shader code here)
</code></pre></div></div>
<p>Would instead look like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defvshader test (verts &uniform i origin cam-to-world)
(declare (pnc verts) (:int i) (:vec3 origin)
(:mat4 cam-to-world) (context :version 330))
...shader code here)
</code></pre></div></div>
<p>Which is more lispy…but as types are mandatory I’m not sure how I feel about this. Does this also mean I specify ‘out’ variables here?</p>
<p>** NOTE: At the start of this post I knew what I was going to blog about, but now I am off on a mental wander. So sorry for the round about nature of the rest of the post :) **</p>
<p>It does make the main signature clearer, especially where array types are concerned.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(i (:int 3))
</code></pre></div></div>
<p>The above is the current signature for an array of 3 ints, get a few of these in the argument list and it gets messy.</p>
<p>But this also means I will have to do the same for labels…</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((test ((i :int) (y (:int 3)))
(+ i (aref y 0))))
(test 1 some-int-array))
</code></pre></div></div>
<p>..becomes..</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((test (i y)
(declare (:int i) ((:int 3) y))
(+ i (aref y 0))))
(test 1 some-int-array))
</code></pre></div></div>
<p>Hmm, not too bad, the first example is a jumble… how about ‘let’ forms?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((x 1)
((y :int) 2)
((z (:int 3)) some-array))
(some-func x y z))
</code></pre></div></div>
<p>Notice how the types are optional as they can be inferred, so in declare style</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((x 1)
(y 2)
(z some-array))
(declare (:int y) ((:int 3) some-array))
(some-func x y z))
</code></pre></div></div>
<p>Ok now in this case it is a BIG improvement to the actual readability of the declarations themselves.</p>
<p>The big worry for me though is giving you two places to look for one piece of information when reading the code. What you want to know is “What is x” and you have to look at for it’s declaration (so you can know it’s scope) and the declare (to know it’s type).</p>
<p>The other possibility is to be super un-common-lispy and use some special reader syntax that is ONLY valid in bindings.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(labels ((test ([i :int] [y (:int 3)])
(+ i (aref y 0))))
(test 1 some-int-array))
(let ((x 1)
([y :int] 2)
([z (:int 3)] some-array))
(some-func x y z))
</code></pre></div></div>
<p>Ugh…even writing it down feels wrong. Nah, scrap this idea. I think the declare style is growing on me but I will need to give this some time. I’d appreciate any ideas on this one.</p>
<p>Ciao</p>
<p>p.s. But wait, one other advantage of declare form, it allows us to have default values for uniforms args. Hmm, this quite possibly means more state changes, as every unspecified uniform must take the defaut value, whereas currently uniforms are memoized. Ok so bad idea. I should probably have the concept of sane defaults though. I should move this to a new post :)</p>
It's Just Pretty2014-11-25T00:21:00+00:00http://www.techsnuffle.com/2014/11/25/its-just-pretty<p>First some code!:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> (setf c (make-c-array #(1 2 3 4)))
#<C-ARRAY :element-type :UBYTE :dimensions (4)>
> (setf g (make-gpu-array #(1 2 3 4)))
#<GPU-ARRAY :element-type :UBYTE :dimensions (4) :backed-by :BUFFER>
> (setf g (make-texture #(1 2 3 4)))
#<GL-TEXTURE-1D (4)>
> (texref *)
#<GPU-ARRAY :element-type :R8 :dimensions (4) :backed-by :TEXTURE>
</code></pre></div></div>
<p>I have tweaked the make-*-array and make-*-texture commands so that, if you only pass in a lisp data structure, cepl will try and find a the smallest gl type that can be used across the whole data-set.</p>
<p>For example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(make-c-array #(1 2 3 4)) ;; <- element type will be :ubyte
(make-c-array #(1 -2 3 4)) ;; <- element type will be :byte
(make-c-array #(1 -2 200 4)) ;; <- element type will be :int
(make-c-array #(1 -2 200 4.0)) ;; <- element type will be :float
(make-c-array #(1 -2 200 4.0d0)) ;; <- element type will be :double
</code></pre></div></div>
<p>This is one of those nice cases where you get to make repl exploration more fun with no cost to speed. The reason for this is no high level code will be using this inference as it would be stunningly stupid to pay that cost for no reason. All file loading (which is the most common case) will have the type name anyway so it can just be passed in as usual.</p>
<p>Little details, but pleasant ones</p>
<p>p.s. This is in a branch for now as I am hunting down some issues I have introduced while modifying the indexing functions</p>
Install wget on osx without MacPorts2014-09-23T08:02:56+00:00http://www.techsnuffle.com/2014/09/23/install-wget-on-osx-without-macports<p>http://osxdaily.com/2012/05/22/install-wget-mac-os-x/</p>
<p>Go check it out, very useful.</p>
Delays in Pushing Code2014-07-28T09:55:06+00:00http://www.techsnuffle.com/2014/07/28/delays-in-pushing-code<p>I have been hoilding off pushing the new features to Cepl recently as I have one bug in defglstruct that really is confusing me.</p>
<p>At first it seems very simple, it is complaining that certain functions (A) dont exist with compiling functions (B). Now (A) do exist by the time (B) is called so it doesnt cause any immediate problems, but the compiler wanrings are very annoying.</p>
<p>The weird bit is that (A) do exist at that time. For example if I expand the defglstruct macro and then run the resulting code it works with no warnings.</p>
<p>There is something I’m just missing but it is the reason I havent merged into master yet.</p>
<p>I may give this some more attention tonight.</p>
<p>Ciao</p>
Buffer Backed Textures2014-07-28T09:26:48+00:00http://www.techsnuffle.com/2014/07/28/buffer-backed-textures<p>Last night I was musing over rendering graphs into textures and I was interested in how to write just a couple of pixels to the texture without reuploading the image. Now I can use gl’s ‘copy sub image’ functions but looking through my copy I saw I hadn’t implemented buffer backed textures; this gave me a great excuse to finally get it done!</p>
<p>In cepl buffer backed arrays would more accurately be named array-backed-textures as we already have the gpu-array abstraction over buffers. This works in my favour though as currently the abstraction for textures in cepl is as follows:</p>
<ul>
<li>A texture is a structure that holds a number of images. You use texref (like aref) to get at one of the images.</li>
<li>Images are just arrays so they are exposed as texture backed gpu-arrays.</li>
</ul>
<p>So calling texref on a regular texture gives you a texture-backed gpu-array and calling texref on a buffer-backed texture give you a buffer-backed gpu-array.</p>
<p>So to get change one pixel of a texture to the color red we can do this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(with-gpu-array-as-c-array ((texref texture) :tempname arr)
(setf (aref-c arr 10 10) (v! 1 0 0)))
</code></pre></div></div>
<p>I have been coding a lot of c++, java and c# recently and so coming back to lisp and doing things like the above just makes me so damn happy! I love this language.</p>
Varjo and Multiple Value Return2014-07-24T15:46:00+00:00http://www.techsnuffle.com/2014/07/24/varjo-and-multiple-value-return<p>Hi folks, having recently got married (yay!) and got a flat (another yay!) I have been getting back into lisp coding again.</p>
<p>The latest addition to the Varjo compiler is support for ‘values’ and ‘multiple-value-bind’. Now as glsl doesn’t have support for multiple value returns we have to use something else. The mechanism I have now works something like this:</p>
<ul>
<li>
<p>If you have a ‘values’ form within a multiple-value-bind form the multiple-value-bind form turns into a ‘let’ and the ‘values’ becomes ‘setf’s to the variables in the let.</p>
</li>
<li>
<p>If you have a values form and it isnt within a multiple-value-bind then one of the following happens
** If the values form can propagate to a ‘return’ form then the function is given out-vars and they are set from the place where the values form was originally.
** If it cant reach a return then it collapses to a progn.</p>
</li>
</ul>
<p>If a varjo function with out-vars is used within a multiple-value-bind then the multiple-value-bind becomes a let statement and the first var is set by the function return and the rest are passed as out-vars to the function.</p>
<p>If the function is used outside of a multiple-value-bind then the form is surrounded in a ‘let’ as the outvars still need to be set.</p>
<p>This results is something like the following.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VARJO> (defshader test ()
(labels ((thing ((x :int)) (return (values x (* x 2)))))
(multiple-value-bind (x y) (thing 1)
(+ x y)))))
;; Gets compiled to
#version 330
int thing_26v(int x_23v, out int return1);
int thing_26v(int x_23v, out int return1) {
return0 = x_23v;
return1 = (x_23v * 2);
return return0;
}
void main() {
int a_27v0;
int a_27v1;
a_27v0 = thing_26v(1,a_27v1);
(a_27v0 + a_27v1);
}
</code></pre></div></div>
<p>I still have a few bugs to iron out but it is very nearly there.
One step closer to lisp!</p>
Eclipse project overlaps error2014-05-05T14:02:42+00:00http://www.techsnuffle.com/2014/05/05/eclipse-project-overlaps-error<p>I tried adding an existing android project today and got this error from Eclipse.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>“<name of project> overlaps the location of another project..”
</code></pre></div></div>
<p>It was reported as a project desciption problem. Turns out eclipse doesnt like importing projects from the direcotry it has designated as the workspace. Move the project outside that folder (or change the workspace directory) and try again.</p>
<p>What a pain in the arse!</p>
Android Ndk Build Error Multiple Targets2014-05-05T13:57:56+00:00http://www.techsnuffle.com/2014/05/05/android-ndk-build-error-multiple-targets<p>Had this error today:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* multiple target patterns. Stop.
</code></pre></div></div>
<p>(Turns out I needed to delete the obj file from the android project)[http://stackoverflow.com/questions/10293659/getting-multiple-target-patterns-stop-error-when-attempting-to-build-for-and]</p>
<p>This post is mainly to help me remember this but hopefully it helps someone else too.</p>
Cepl on OSX2014-04-15T17:29:22+00:00http://www.techsnuffle.com/2014/04/15/cepl-on-osx<p>This has been a massive pain in the arse and still isn’t working in the way I would like it but there is some progress so it is worth testing.</p>
<p>First off, while I love sbcl, I cannot get sdl2 to create a fully working window. I get a window with no border or one which does not receive input events.</p>
<p>So instead I tried clozure (ccl) and have had some success.</p>
<p>First cepl:repl now wraps it’s contents in an sdl2:in-main-thread form. So you can continue to use this as usual.</p>
<p>Next you need to wrap the code that contains your main loop in an ‘in-main-thread’ form. In the case of the provided examples you can just write (sdl2:in-main-thread () (run-demo)).</p>
<p>This gives you an active window that is decorated and accepts events which is wonderful. The only problem so far is that #’update-swank doesn’t work and so you cant use slime’s repl while the demo is running. This is a deal breaker for me, as that ruins my workflow so I need as solution to consider using osx for any cepl development.</p>
<p>Hopefully a solution to this will emerge soon. I then really need to go clean up the code surrounding this fix.</p>
<p>Here’s hoping</p>
<p>[edit]
Seems the repl in the inferior-lisp buffer (or shell) is active and usable. However the repl buffer doesnt yet. Can anyone with more experience with ccl help me out here?</p>
Fps Counter with Temporal Functions2014-04-14T00:00:46+00:00http://www.techsnuffle.com/2014/04/14/fps-counter-with-temporal-functions<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((count 0))
(tdefun fps ()
(incf count)
((each (seconds 1)) (print count) (setf count 0))))
</code></pre></div></div>
<p>It’s simple but it makes me happy simply because it is written in term of time.</p>
<p>p.s. Oh yeah, as you may have guessed from this, temporal functions work now!</p>
Followup on the Temporal Functions2014-04-12T04:04:04+00:00http://www.techsnuffle.com/2014/04/12/followup-on-the-temporal-functions<p>Just thought I’d warn anyome that cares that the current implementation is terrible!</p>
<p>It was good enough to prove a point to myself but I am currently cleaning up the code and fixing the bugs. The result is quite nice but it is bending my mind and it is time for a sleep.</p>
<p>Ciao</p>
Cepl Foreign Data2014-04-11T10:21:27+00:00http://www.techsnuffle.com/2014/04/11/cepl-foreign-data<p>Up until now cepl has been putting it’s own wrapper around <a href="https://github.com/rpav/cl-autowrap">cffi</a> data in order to make it easier to manage and play with in lisp.</p>
<p>I wanted to start supporting more complicated foreign data and so was just about to start writing when I remembered cl-autowrap. If you are a lisp person and you haven’t checked autowrap out yet then please do, it is a fantastic wrapper generator which given this..</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(c-include "file.h")
</code></pre></div></div>
<p>..Will write an entire lisp wrapper around the foreign library. It also provides hooks (via <a href="http://common-lisp.net/project/cffi/">trivial-garbage</a> ) to allow the CL’s garbage collector to manage your foreign data.</p>
<p>A little digging around in the back-end (with some great help from rpav, cheers man!) of autowrap has shown me that there is more than enough data back there for me to be able to write something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(make-available-for-gpu 'some-type)
</code></pre></div></div>
<p>And then have the type fully integrated into my glsl compiler.</p>
<p>I do need to augment the c-autowrap slightly as I want to be able to recieve cepl’s own c-array objects instead of autowrap’s. But this is certainly do-able and I’ll have a good look at it next week.</p>
<p>Righto folks, I need coffee</p>
<p>Seeya</p>
Cepl in Time2014-04-04T18:23:46+00:00http://www.techsnuffle.com/2014/04/04/cepl-in-time<p>This post is about time and my ongoing experiments of trying to find a way of representing it in cepl. If you aren’t interested in the journey but are interested in the solution then jump down to the ‘So what’s the result?’ section.</p>
<p>###Why Bother?</p>
<p>It would be very valid to raise the fact that for real-time applications there are some very well-know and suitable ways of handling time (e.g. <a href="http://gafferongames.com/game-physics/fix-your-timestep/">fixed timestep</a> ) so why wouldn’t we use these techniques?</p>
<p>Often when coding interactively you just want to run something in the repl in pure isolation. These kind of experiments can give rise to new techniques and allow you to toy with idea more easily as you don’t have to worry about any other part of your program. Lisp support this style of programming by having axiomatized some of the core concepts of the language[1] and I wanted that for time as well. I didn’t want to have to take some time delta from another part of my program in order to use it in some function or concept I was developing, I want to focus purely on the task at hand.</p>
<p>###What should time look like?</p>
<p>So after I had decided I wanted some kind of primitive for time in cepl I needed to decide what this would look like. As usual I stole from lisp. Cons cells encapsulate the most basic nature of a list: It is some ‘thing’ followed by some other ‘thing’. So what is time?</p>
<p>Well first off I decided not to look at it from a physics point of view, as this would be unnecessarily complicated, and stick to how time seems to be. Well time is continuous, it doesn’t seem to ‘arrive’ in lumps… but computers by their nature are incremental. So if time is flowing and we treat it like a stream of water, we want to capture time in buckets and then use the captured time to drive some function. OK, that’s weird but time-buffers are kind of what fixed-time-steps implement so lets have ‘time-buffers’ as a concept.</p>
<p>Next…where does the time come from? Well in computing we are used to having streams of data coming from some data source so let’s have a time-source. It also makes sense that we might want to speed up and slow down time so let’s have a time-source be able to be based on another time source with a modifier. We could use this to make a time source that is running 10 times slower than real time. Now if we allow passing of time sources to our time buffers we have a side effect free way of implementing slow motion…even if this only temporary it could be useful for observing some effect slowly to see it is behaving correctly.</p>
<p>When we speak about time we often use some condition to qualify when we are talking about. For example:</p>
<ul>
<li>BEFORE tomorrow do x</li>
<li>BETWEEN 10am and 2pm do x</li>
<li>AFTER 10 minutes from now do x</li>
</ul>
<p>These predicates should take a time an return t or nil.</p>
<p>The ‘AFTER’ example is interesting as it includes some syntax for talking about time relative to another point in time. So we will need some kind of ‘from-now’ function.</p>
<p>###The weird bit…</p>
<p>OK so this next part of the process was not reasoned about or based on logic. I was writing down the above and trying to work out other time predicates when I heard a voice in my head say <strong><em>‘LEXICAL TIME’</em></strong>. This was an odd thing to hear as I had no idea what my brain was talking about..so I stopped for a bit the more I thought about it the more I started to like the idea, we have a temporal-function so you define functionality that implicitly understood time.</p>
<p>I thought it could be some macro that wrapped up the elements we had already created and let the user write code knowing that time was being taken care of.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;;this was the first pseudo-code of a temporal-lambda
(tlambda (x) (within 0 5000) (print x))
</code></pre></div></div>
<p>###The problems</p>
<p>As usual ideas burst into flame on contact with reality. Here are the major points that arose.</p>
<p>####Expiry</p>
<p>The first problem I noticed was in a tlambda that uses the ‘before’ predicate e.g:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(tlambda () ((before (from-now (seconds 10))) (print "hi")))
</code></pre></div></div>
<p>Notice that after 10 seconds from now this function will never run again. It essentially has expired. Now I could return some kind of value but this sucks as I was these temporal functions to be identical to regular functions. A friend then recommended using conditions, which turned out to be perfect. So we now have an ‘expired’ condition that can be caught by time aware code.</p>
<p>But when we look at this again we notice that really expired isnt limited to time…it is part of a more general set of conditional-functions. Which crudely stated as a macro could be.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defmacro defcfun (name args condition &body body)
`(defun ,name args
(if ,condition
(progn ,@body)
(signal-expired))))
</code></pre></div></div>
<p>So temporal functions are a type of conditional function. I’m not sure how useful this is…but it was interesting to me.</p>
<p>####Composition and Syntax</p>
<p>The next big issue was in composition and what the resulting code looked like. I was starting to get something working with functions and closures but I started running into a problem as I wanted a predicate that returned t once every x seconds/milliseconds.. something like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(tlambda () ((each (second 2)) (print "hi"))
</code></pre></div></div>
<p>But what we actually need is to create a closure with a stepper in the closed vars.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((stepper (make-stepper (seconds 2))))
(tlambda () (when (funcall stepper) (print "Hi")))
</code></pre></div></div>
<p>Given that tlambdas were meant to implicitly understand time this seemed ugly.</p>
<p>This was compounded by the next problem…</p>
<p>####Time overflow</p>
<p>Let us define an imaginary tlambda with a nice syntax:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(tlambda ()
(then ((before (from-now (seconds 5))) (print "hi 1"))
((before (from-now (seconds 5))) (print "hi 2"))
((before (from-now (seconds 5))) (print "hi 3"))))
</code></pre></div></div>
<p>What this is meant to do is:</p>
<ul>
<li>run the first before clause until it expires and then</li>
<li>run the next clause until expires and then</li>
<li>run the third clause until it expires and the signal that the whole tlambda has expired.</li>
</ul>
<p>Now let’s imagine that we create this tlambda but wait 12 seconds before calling it. The first time we call it should now recognize that the first two clauses have already expired and jump straight to 2 seconds into the third clause.</p>
<p>At this point I could not think of a way of doing this that didnt invlove a DSL. This became even more apparent when I thought about optimizing as I can’t inline the stepper closures and inside the tlambda there would be A LOT of signals being fired and caught.</p>
<p>I was quite annoyed that my knowledge of lisp was potentially limiting me here but at least I could have a go at making a macro version which could be more optimized (if perhaps not as functional as I would originally have liked)</p>
<p>####So what’s the result?</p>
<p>Well the result is a very nice DSL which creates lambdas that know time and can mix temporal syntax with default lisp syntax very easily.</p>
<p>Here are a few examples:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; when you use default lisp code, the result is a regualr lambda
(tlambda () (print "Hi"))
;; inline lambdas are fine too, again this is just a regular
;; lambda
(tlambda ()
(let ((text "hello"))
((lambda (x) (print x)) text)))
;; This function will for the first 10 seconds print "hi jim" each
;; time it is called. After 10 seconds it will print "jim" each
;; time it is called.
;; Temporal forms are conditional, the predicate must be true for
;; the rest of the form to be evaluated. For this reason we borrow
;; the syntax from the cond macro.
;; Also notice that unlike cond every line is evaluated. So the
;; (print "jim") line always works. This also means this function
;; never actually expires. regular lisp forms can be seen as
;; eternal.
(tlambda ()
((before (from-now (seconds 10))) (princ "hi "))
(print "jim"))
;; time syntax must be either at the root of the tlambda or inside
;; one of the special forms defined for time functions.
;; the following is not valid, as the temporal condition form is
;; within a standard lisp form
(tlambda ()
(print ((before (from-now (seconds 10))) "hi")))
;; tlambda introduces 2 new special forms: then & repeat
;; they each run each containing form until it expires and
;; only then move to the next form. The difference is that
;; once all the forms have expired 'then' will signal it has
;; expired, whereas repeat will start from the first form again
(tlambda () (then ((before (from-now (seconds 10))) (print "hi"))
((once) (print "allo")))
(tlambda () (repeat ((before (from-now (seconds 10))) (print "hi"))
((once) (print "allo")))
;; progn is also modified so that it can contain the temporal
;; condition forms
(tlambda ()
(progn ((before (from-now (seconds 2))) (print "hi"))))
</code></pre></div></div>
<p>####Expansion</p>
<p>Here is an example of what is generated by tlambda (I have tidied it up a little and changed gensym names for ease of reading but functionally it is the same):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; This:
(tlambda ()
(then ((before (from-now (seconds 3))) (print "hi"))
((before (from-now (seconds 4))) (print "bye"))))
;; expands too:
(let ((counter 0)
(deadline-1 (from-now (seconds 3)))
(deadline-2 (from-now (seconds 4))))
(lambda ()
(if (= counter 2)
(signal-expired)
(when (and (< counter 2))
(if (afterp deadline-1)
(progn
(when (= counter 0)
(incf counter)
(setf deadline-2 (- (from-now (seconds 4))
(- (funcall *default-time-source*)
(afterp deadline-1)))))
(if (afterp deadline-2)
(when (= counter 1) (incf counter))
(when (beforep deadline-2) (print "bye"))))
(when (beforep deadline-1)
(print "hi")))))))
</code></pre></div></div>
<p>####What is missing?</p>
<ul>
<li>Tests and Bugfixing: There are still bugs in this and need to squash them</li>
<li>Generate cleaner code: Currently it works well but could be nicer for human reading</li>
<li>Generate type declarations: For speed optimization</li>
</ul>
<p>Well that is all for now. Thanks for reading. The interesting thing now will be playing with this and seeing if the abstraction is actually useful!</p>
<p>Ciao</p>
<p><em>1.</em> The simplest example of this is lists in lisp. Rather than just being given a list type, with it’s implementation hidden away, we get the cons cell. The cons cell is the smallest abstraction of a linked list and with it we can build lists, tree and other graphs and then use the techniques we have for navigating cons cells (car and cdr (and friends)) to managed all of these different data structures. Also see <a href="http://lib.store.yahoo.net/lib/paulgraham/jmc.ps">Paul Graham’s ‘The Roots of Lisp’</a>. It’s a short article, but it hows how with 7 basic primitives you can write an interpreter for an entire lisp language.</p>
Varjo Gets Raw Glsl Functions2014-03-20T20:11:31+00:00http://www.techsnuffle.com/2014/03/20/varjo-gets-raw-glsl-functions<p>Yesteday I added the ability for varjo to use blocks of raw glsl code which was a feature I have had a couple of requests for.</p>
<p>The raw blocks are presented as functions and you have to specify the arguments and return type using varjo syntax. This allows varjo to perform the type checks that make this raw function safe to use in your shaders.</p>
<p>The body however is a string and as such I am not doing any checks to ensure the code is valid. If you lie to the compiler things will break :)</p>
<p>This is how a function can be created and used in shaders. Notice that like all external functions it will be automatically included if used in any shader.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VARJO> (v-def-raw-glsl-func raw-test ((a v-int) (b v-float))
v-float
"return a * b;")
VARJO> (defshader test-shader ()
(raw-test 1 2.4))
#<VARJO-COMPILE-RESULT {25EA8929}>
VARJO> (glsl-code *)
"#version 330
float raw_test_1v(int a, float b);
float raw_test_1v(int a, float b) {
return a * b;
}
void main() {
raw_test_1v(1,2.4f);
}
"
</code></pre></div></div>
<p>One thing I would also like to add is some extra syntax to allow calling out to shader functions written in lisp. This should be pretty easy but I have a HUGE list of other things to implement first so that will have to wait for now.</p>
<p>Seeya folks</p>
Back From Antartica2014-03-18T22:17:32+00:00http://www.techsnuffle.com/2014/03/18/back-from-antartica<p>Heya folks, I’ve just got back from sailing to Antartica with the JST. This was simply incredible in way I just can’t work out how to put into text yet, but will soon.</p>
<p>Also I have lots of things I want to add,fix & tweak for my lisp projects so watch this space as, work permitting, I should be putting out a fair bit of stuff this year.</p>
<p>Hope you are all well,
Ciao</p>
<p>Here is the a quick preview of what we have been up to!</p>
<p><img src="/assets/images/jst1.jpg" /></p>
FUCK YEAH2013-11-12T23:58:00+00:00http://www.techsnuffle.com/2013/11/12/fuck-yeah<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#<CODE {100752B7A3}>
Class: #<STANDARD-CLASS CODE>
Group slots by inheritance [ ]
Sort slots alphabetically [X]
All Slots:
[ ] CURRENT-LINE = "mix(1.0f,2.0f,0.5f)"
[ ] INVARIANT = NIL
[ ] OUT-VARS = NIL
[ ] RETURNS = NIL
[ ] TO-BLOCK = NIL
[ ] TO-TOP = NIL
[ ] TYPE = #<V-FLOAT {100751FFE3}>
[set value] [make unbound]
</code></pre></div></div>
<p>slime-inspector never looked so good!</p>
Getting Close2013-11-12T22:27:00+00:00http://www.techsnuffle.com/2013/11/12/getting-close<p>Been a crazy week of headbuttng a macbookpro until a cocoa application worked followed by a fantastic demoparty (thanks everyone at kindergarten for a phenominal weekend!).</p>
<p>I’m finally back in lisp code (phew!) and I’m around 2/3rds of the way through a rewrite of the Varjo lisp->glsl compiler.</p>
<p>The reason for the rewrite now is that someone asked for a tour through the code explaining how it worked…after 5 minutes of reading I decided it could only be dark-magic holding it together so I have tried to rewrite it to take into account a few of the things I’ve learnt.</p>
<p>Given that I wrote it when I really didnt know anything about glsl I had a lot to change! I have also taken a few of the additions from cepl I had been hacking on and made them core features of the compiler. The big one in this category is you can define shader functions that are only included in the glsl source if they are used.
This means you can make a nice ‘standard library’ of functions for your shaders and know they won’t be included unless they are used in a given shader.</p>
<p>There are also macros (regular for now much compiler soon) and generic functions.</p>
<p>I’m hopefully finishing the last of the argument validating this evening and then I think the last big things are reimplementing the special functions and rewriting the string generation. Hopefully I will be done in a week or two.</p>
<p>Right, enough procrastinating, back to work!</p>
CEPL meets SDL22013-10-26T12:58:00+00:00http://www.techsnuffle.com/2013/10/26/cepl-meets-sdl2<p>Hey folks,
Another cool week or so as cepl appeared in the lisp subreddit and the videos got 980 views in one day, which for me was pretty exciting!</p>
<p>One of the things that came up very quickly was some issues on osx which I originally thought was down to immutable textures. So I added mutable texture storage and the problem still persisted.</p>
<p>Thanks to the fantastic people reporting bugs and in the irc channels I was informed that on some systems when you create a opengl context it picks a lower profile unless you specifically request a higher one. These request functions are only available in sdl2 so I have very hastily ported to sdl2.</p>
<p>This is obviously still a project in the alpha stage of development (is there an earlier term as I’m still working out what this project will be?) so there are plenty of bugs. If you do find them and have a chance please email me or make a bug report in github. This is a part time project but I want to get it in a state suitable for real experimentation with graphics so I will get to the bugs, just slowly sometimes :D</p>
<p>Righto, thanks for stopping by.</p>
<p>Oh almost forgot I have added a very crude compile chaining system so now when you compile a shader stage or sfun it will compile any stage or pipeline that uses it. Woo! I have doen this is a really hacky way but it proves the behaviour is nice so I will build from there.</p>
defsfun2013-10-19T18:49:00+00:00http://www.techsnuffle.com/2013/10/19/defsfun<p>One of the recent changes to CEPL has been to add the defsfun macro. This allows defining a shader function that doesnt belong to particular stage, if you use it in a shader it justs gets added to the code.</p>
<p>Here is an example of one:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(defsfun sphere ((p :vec3) (r :float))
(return (- (length (* rot p)) r)))
</code></pre></div></div>
<p>Now the goal of Varjo was to make shader code look and feel like lisp so the following thought should have been obvious to me from the start but it wasnt. The thought is that if this is so nearly lisp then I could implement lisp versions of a few functions and have a lisp version of a shader that I can run locally for easy testing.</p>
<p>Again, this should have been obvious but I’m quite excited about the posibilities. As I have alreay wrapped the datatypes I can just pass the local version a c-array rather than a gpu array and get it to work.</p>
<p>Obviously the local versions would be too slow to use for rendering but would be great for testing and development in the repl. Also as the shader functions require types w can translate these into declares that will help with the efficiency.</p>
<p>This is a fair way down my list of things to do but after I have nailed time and space abstractions down a bit I will revisit this.</p>
Phew! Lots Happening2013-10-19T17:27:00+00:00http://www.techsnuffle.com/2013/10/19/phew-lots-happening<p>It has been a crazy few months. I got offered, and accepted, a job at a startup in Norway so about a month back I moved and set up in Oslo.</p>
<p>I’d finally settled in and suddenly CEPL turned up on reddit and people were checking it out, which was great…except I had been making loads of breaking changes in master so it was a bloody mess!</p>
<p>Things are really coming along well though so I need to make some videos showing how the memory handling works. I think I’ll do that tomorrow.</p>
<p>The most recent change I have been making is a rewrite of CEPL’s front-end to the Varjo shader compiler. Previously, when you defined a shader pipeline in CEPL you have to write the shaders inside the defpipeline form. Now they can be defined as separate entities and then the defpipeline stitches them together. Also you can define individual functions outside of the shader stages and they will get mixed in only if they are used in a stage. Along with shader macros I now have some nice tools to experiment with different ways of extending glsl.</p>
<p>I am going to rewrite my ray marcher example to use these features to see what effect it has on the ease of writing and reasoning about the shader pipeline.</p>
<p>Here is a video of the compiler stuff:</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/2Z4GfOUWEuA" frameborder="0"> </iframe>
Ironman Armor Build2013-06-08T10:00:05+00:00http://www.techsnuffle.com/2013/06/08/ironman-suit<p>This is a photo dump from a build I just completed for a friend. I will be back to add detail to this soon. Ciao!</p>
<p><img src="/assets/images/ironman/im00.jpg" />
<img src="/assets/images/ironman/im01.jpg" />
<img src="/assets/images/ironman/im02.jpg" />
<img src="/assets/images/ironman/im03.jpg" />
<img src="/assets/images/ironman/im04.jpg" />
<img src="/assets/images/ironman/im05.jpg" />
<img src="/assets/images/ironman/im06.jpg" />
<img src="/assets/images/ironman/im07.jpg" />
<img src="/assets/images/ironman/im08.jpg" />
<img src="/assets/images/ironman/im09.jpg" />
<img src="/assets/images/ironman/im10.jpg" />
<img src="/assets/images/ironman/im11.jpg" />
<img src="/assets/images/ironman/im12.jpg" />
<img src="/assets/images/ironman/im13.jpg" />
<img src="/assets/images/ironman/im14.jpg" />
<img src="/assets/images/ironman/im15.jpg" />
<img src="/assets/images/ironman/im16.jpg" />
<img src="/assets/images/ironman/im17.jpg" />
<img src="/assets/images/ironman/im18.jpg" />
<img src="/assets/images/ironman/im19.jpg" />
<img src="/assets/images/ironman/im20.jpg" />
<img src="/assets/images/ironman/im21.jpg" />
<img src="/assets/images/ironman/im22.jpg" />
<img src="/assets/images/ironman/im23.jpg" />
<img src="/assets/images/ironman/im24.jpg" />
<img src="/assets/images/ironman/im25.jpg" />
<img src="/assets/images/ironman/im26.jpg" />
<img src="/assets/images/ironman/im27.jpg" />
<img src="/assets/images/ironman/im28.jpg" />
<img src="/assets/images/ironman/im29.jpg" />
<img src="/assets/images/ironman/im30.jpg" />
<img src="/assets/images/ironman/im31.jpg" />
<img src="/assets/images/ironman/im32.jpg" /></p>
(+ 'ECL 'QT)2012-10-28T21:28:41+00:00http://www.techsnuffle.com/2012/10/28/-ecl-qt<h3 id="eql">EQL!</h3>
<p><a href="http://www.password-taxi.at/EQL">EQL</a> does look like the absolute answer to my app making prayers. It is a set of ECL bindings for QT which should mean I’m able to both use the power of real-time development, the excellent GUI designing tools of <a href="http://qt.digia.com/">QT</a> and still have small executables thanks to ECL.</p>
<p>The only issue is I can’t install it :b !</p>
<p>Yeah I’m having problems compiling it just now, though I have written to the author to see if he can give any advice. Whatever I find out I will post here.</p>
<p>It is no understatement to say I’m very excited! To be favourable on mobile devices an app really should be as small in size and memory footprint as possible, and this might just let me do both in one fell swoop!</p>
<p>Here’s hoping!</p>
Common Lisp on the Nexus 72012-10-28T16:45:38+00:00http://www.techsnuffle.com/2012/10/28/common-lisp-on-the-nexus-7<p>Right now I’ve got Ubuntu on this Nexus, I’m going to want to code something for it. It’s already set up for C and python dev but not a drop of Lisp in sight (<em>gasp</em>)!</p>
<p>In this post I’m going to get two versions up and running on the Nexus. <a href="http://ecls.sourceforge.net/">ECL</a> and <a href="http://ccl.clozure.com/">Clozure</a>.</p>
<p>To make this easier for yourself make sure you have installed ssh so you can do all of this from a real keyboard!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install ssh
</code></pre></div></div>
<h3 id="ecl">ECL</h3>
<p>ECL is a great fit for this platform as it compiles down to C code so the executables are small and the tight integration with the FFI will make this very useful. Also it is super easy to install as it is in the repos.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install ecl
</code></pre></div></div>
<p>and your done! Obviously your going to want to have more than just the repl so further down in this post we will look at installing Emacs and Slime.</p>
<h3 id="clozure">Clozure</h3>
<p>Next is Clozure. Now I’m usually working in SBCL when I’m on a regular machine (mainly as that’s what my mate set me up with when I started learning!) but Clozure (I believe) has had an Arm port for a while longer and also SBCL’s install process for Arm was a bit hairy for a beginner like me.</p>
<p>Setting up Clozure involves a little bit more work than ECL, but not by much.</p>
<p>Right so once you are ssh’d into yout tablet, you need to pull the latest version of Clozure from the repository.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> svn co http://svn.clozure.com/publicsvn/openmcl/trunk/linuxarm/ccl
Note: I'm using the build from trunk which is not guarenteed
stable, it may be worth trying the release version instead
but as I have not tested this yet I can't show the steps here.
If you do try it, please let me know how it went!
</code></pre></div></div>
<p>Once that has finished we need to make a quick change to the config as, as default, Clozure for Arm7 is set up for soft floating point. We need hard. So:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> cd ccl/lisp-kernel/linuxarm/
make clean
</code></pre></div></div>
<p>Now you need to open up the ‘float_abi.mk’ file in your favourite editor and uncomment the hard float line and comment out the soft float one. After the change the file should look like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> # This file should define FLOAT_ABI as one of "softfp" or "hard".
# If you change this, do 'make clean' to remove any object files
# compiled for the other ABI.
#FLOAT_ABI = softfp
FLOAT_ABI = hard
</code></pre></div></div>
<p>We are almost done! now run the following:
make
cd ../..
armcl –not-init</p>
<p>At this point you should get the clozure repl!</p>
<p>Now run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(rebuild-ccl :full t)
</code></pre></div></div>
<p>And clozure will be rebuilt. This will take a few minutes so keep yourself busy with a little victory dance!</p>
<h3 id="editor">Editor</h3>
<p>Wonderful, so now I want an editor to play around with. Unfortunately I am not versed in Vim so I can’t provide and info for you Vim fans but I do use Emacs so here goes!:</p>
<p>Well the first order of buisness is to get Emacs. Now it may seem overkill to have this on your device, you may prefer just to install swank and use your desktop but for me I’d like to ssh in an use Emacs from the command line.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sudo apt-get install Emacs24 emacs-goodies-el slime
</code></pre></div></div>
<p>Now we just need to add the following to the .emacs file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(eval-after-load "slime"
'(progn
(setq slime-lisp-implementations
'((ccl ("/home/ubuntu/.ccl/armcl"))
(ecl ("/usr/bin/ecl"))))
(slime-setup '(
slime-asdf
slime-editing-commands
slime-fancy-inspector
slime-fontifying-fu
slime-fuzzy
slime-indentation
slime-mdot-fu
slime-package-fu
slime-references
slime-repl
slime-sbcl-exts
slime-scratch
slime-xref-browser
))
(slime-autodoc-mode)
(setq slime-complete-symbol*-fancy t)
(setq slime-complete-symbol-function
'slime-fuzzy-complete-symbol)))
(require 'slime)
</code></pre></div></div>
<h3 id="done">Done!</h3>
<p>Well thats all done! I’ll try and keep this updated with any progress I make on the tablet. I’m very excited to see what happens next.
Ciao</p>
Ubuntu and Nexus72012-10-28T10:56:05+00:00http://www.techsnuffle.com/2012/10/28/ubuntu-and-nexus<iframe width="560" height="315" src="http://www.youtube.com/embed/vUCqi_gyIDs" frameborder="0"> </iframe>
<p>Given the above blurry video of Ubuntu on the Nexus 7 and knowing that in it’s current state, it will be bug ridden, unoptimised and dangerous to ones sanity, why would someone go shell out £170 for one?</p>
<p>Well I just have so let me explain my logic!
I use android, I like it, I have no hatred of Apple or Google but while I have moved more of my computing life into software that is open, I have moved more of my personal life into services which are not.</p>
<p>This has been growing on my mind more and more lately and while I’m not ready to completely pull myself from these services I do know that I must start looking. There have been plenty of <a href="http://talkweb.eu/openweb/1842">news stories</a> recently which should serve as warning to where these services are heading and the kind of problems which will increasingly arise.</p>
<p>My phone is my biggest and most continuous link to proprietary services, namely google mail, search, maps, etc. By following along with the development of Ubuntu on mobile devices I hope I can lay the groundwork for my own move to an open source based mobile life.</p>
<p>Again I don’t believe that Google are using my data for malevolent purposes…But I do believe that as the market place changes companies try and leverage what they have in new ways to make money. I don’t want to be that future administration’s leverage.</p>
<p>Another, very important reason is that it will be really cool! I want to see Ubuntu become a viable alternative to what exists already, and look forward to all advances and experience that will move all around the Linux ecosystem from having a solid Debian based OS being transformed into a solid mobile platform.</p>
<p>Bring it on!</p>
<p>p.s. I also want to port CEPL to OpenGl ES and get lisping on this thing!</p>
DSLs In Common Lisp2012-10-22T22:18:21+00:00http://www.techsnuffle.com/2012/10/22/dsls-in-common-lisp<iframe width="560" height="315" src="http://www.youtube.com/embed/5FlHq_iiDW0?list=UUBByjlfugSdMHPlf73YbVBA&hl=en_GB" frameborder="0"> </iframe>
<p>I have just started looking for resources on writing domain specific languages in common lisp and stumbled upon this <strong>excellent</strong> video by Rainer Joswig on the subject. He gives a quick run through and his explanations (as always with Rainer) are clear and illuminating.</p>
<p>I’m also buzzing at the idea of eventually getting my brain into a place where I move as naturally through a problem as Rainer shows in this video. All to often I feel the apprehension caused by not remembering just to let go and just start writing…I guess this affects all disciplines in some form or another but I am beginning to appreciate how differently I can code in Lisp to how I did in Python for example.</p>
<p>How much is down to the language and how much is just me developing as a programmer remains to be seen but it certainly feels different.</p>
<p>Enjoy the video.</p>
A Lighter Shade of Monkey2012-10-22T20:43:39+00:00http://www.techsnuffle.com/2012/10/22/a-lighter-shade-of-monkey<iframe width="420" height="315" src="http://www.youtube.com/embed/YsQZ20ZnnNA" frameborder="0"> </iframe>
<p>BWUHHAHA I’m back! I have spent the last while working on code that really doesnt make for good
videos so I have been quiet but I have something new to show.</p>
<p>This video shows a really simple directional light acting on the monkey head model.</p>
<p>What I’m really pleased with is that I didn’t need to be rigorous which checking that the code would work before I ran it. I was able to run it an mess around with things until they worked. A good example was with the ordering of the matrix multiplications. If you have done some of this stuff yourself you’ll know that, unlike multiplying regular numbers the order you multiply matrices is very important to the outcome so while:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 2 x 4 = 4 x 2
</code></pre></div></div>
<p>..It is not the same when using matrices. I frankly was confused with the orders when I first ran the demo and it showed! The monkey head was all over the place and the light wasn’t working. What was wonderful was I was able just to keep swapping things around until I got it right. I caused a lot of bugs in the process but, as you are still able to compile parts of your code while the debugger is waiting, I was able to undo my mess and carry on running the demo as if I wasn’t incompetent!</p>
<p>All in all I’m pretty chuffed with all of this and have plenty more to show too. I will defintely have to write a long post about all the work I have been doing on the opengl wrapper part of CEPL, it makes the code much easier to write, and read for that matter!</p>
<p>Well that’s all for another day,</p>
<p>Take care.</p>
It's Time...2012-08-23T22:06:18+00:00http://www.techsnuffle.com/2012/08/23/its-time<p>This weeks major distraction from blogging here has been time. I haven’t lacked it but I have been playing with it.</p>
<p>I really didn’t want to work on time just yet, I wanted to add input control to cepl so that I could let people actually <strong>play</strong> with what I have been doing. Oddly though my brain wouldn’t let me, it was one of those times in creativity that you get overruled by the voice in your head and no matter how hard you try to start working on something your brain drags you back to what <em>it</em> wants to be working on.</p>
<p>So I gave in and dropped working on input and had a look at time instead.</p>
<p>Time is intrinsically woven into games, and how you handle it has major impacts on how your game code is structured. As I’m working in a very non-standard language I feel very free to go looking for non-standard ways of doing things.</p>
<p>It’s been a beautiful feeling as I have really felt like I’m exploring ideas rather than just implementing them. I go on gut instinct and the mere action of writing the idea out in code reals the pitfalls as well as the advantages to the idea.</p>
<p>While I was noodling around with one particular function a wee voice popped into my head and said <em>“Lexical Time”</em>… which made no sense but sounded cool, but before I got any further it said <em>“temporal lambda”</em> which did sound cool…and also made sense.</p>
<p>The idea that came unbidden with the voice is that you could define functions which only evaluate their body when they meet a certain time condition (or temporal condition so we can sound fancy!). One example of such a condition would be <strong>BEFORE</strong>.</p>
<p>I must stress that I have no idea how useful these things may be, only that they sounded interesting. So with a little help from a great friend, I created temporal lambdas.</p>
<p>The first thing I am using them for is actually for the input control that my brain wouldn’t let me look at before (It clearly had a plan!). Key combos are very common in games for example you may have to press “A B A B <- ->” within 2 second to perform a special attack. With temporal lambdas, is really easy as it takes care of the ‘within 2 seconds for me”.</p>
<p>The key combo checker could be created like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(let ((key_cache nil))
(ct:tlambda (make-time-cache) (make-withinp 0 2000) (key)
(setf key_cache (cons key key_cache))
(equal key_cache '(:left :right :b :a :b :a))))
</code></pre></div></div>
<p>So the programmer only needs to focus on the code to accumulate the keys and check if they are valid, and the temporal lambda makes sure that it will only be valid within 2 seconds (2000 milliseconds)</p>
<p>I’ll hopefully post some videos soon, once I have some control code written, so you can see what I’m yapping about.</p>
<p>I’m also just excited to see where temporal lambdas will be useful, again the idea came out of the ether so I don’t have a real plan for them yet. However if/when I do I will post up whatever I find.</p>
<p>Goodnight!</p>
Android and Open Source2012-08-23T22:05:00+00:00http://www.techsnuffle.com/2012/08/23/android-and-open-source<p>I had an annoying realisation the other day that, while my computers run almost entirely libre software, the majority of the apps on my phone were not open at all.</p>
<p>Given the awesome fact that a huge percentage of mobile devices are now running an open source OS the fact that the apps aren’t is a bloody joke.</p>
<p>Luckily some great people have put together an ‘App Store’ which only contains libre software, its called (FDroid)[http://f-droid.org/]</p>
<p>There are some great wee apps and as the apps are made to be free there are no adverts either.</p>
<p>Enjoy!</p>
Time to Grind2012-08-17T11:20:39+00:00http://www.techsnuffle.com/2012/08/17/time-to-grind<p>Missed a day of posting here again but progress is being made.</p>
<p>I seem to be falling into nice rhythm with developing cepl. First I work through some tutorial to add functionality to cepl, then I spend a few days trying to find abstractions for what I’ve just done.</p>
<p>Right now I’m in the later of those two. I’m getting annoyed with having to remember what type of vector I am using when I run vector functions against it so I’m currently working on generic function which take care of calling the correct function for me so I don’t have to think about it. I’ve also got tired of writing:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(make-vector3 1.0 2.0 3.0)
</code></pre></div></div>
<p>..every time I need a new vector and having to make sure the numbers are floats. To that end I have made a macro v! so that</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(v! 1 2 3)
</code></pre></div></div>
<p>..works exactly the same and the long version except for also handling type conversions.</p>
Just me and my Monkey2012-08-15T00:00:00+00:00http://www.techsnuffle.com/2012/08/15/just-me-and-my-monkey<p>It works! Turns out it wa a very simple fix as the .obj files start counting from 1 wheras I expected counting from 0. One quick tweak later and we have this:</p>
<p><img src="/assets/images/fixed_vertices.png" /></p>
<p>Love it!</p>
Animated Sorting Algorithms2012-08-15T00:00:00+00:00http://www.techsnuffle.com/2012/08/15/animated-sorting-algorithms<p>Sorting algorithms..there’s a whole bundle of them each with advantages and disadvantages over the others. If you want to understand some of these method and have a fairly visual mind <a href="http://www.sorting-algorithms.com/?">this site</a> will probably be right up your alley.</p>
<p>Use that site and play around with different initial conditions and see which method works best. This can really be helpful along some terser text as a way of seeing what the author is on about.</p>
<p>I’m on call this week for work so I best go try grab some shuteye before someone calls.</p>
<p>Ciao</p>
Slowly Slowly Glitchy Monkey2012-08-13T00:00:00+00:00http://www.techsnuffle.com/2012/08/13/slowly-slowly-glitchy-monkey<p>Well I have tried loading an .obj model in cepl for the first time…and well have a look..</p>
<p><img src="/assets/images/vertex_issues.png" /></p>
<p>Not so good!..Its meant to look more like the grey version on the lower right. Seems like I’ve still got some issues with indexing or something, but at least its getting vaguely recognisable!</p>
<p>More to come.</p>
SSH Taking ages to connect?2012-08-12T00:00:00+00:00http://www.techsnuffle.com/2012/08/12/ssh-taking-ages-to-connect<p>I had an interesting little issue the other day where connecting to some servers was taking aroudn 15 seconds before showing a login prompt. Everything after that was speedy so I knew it wasnt going to be a connection time issue.</p>
<p>I ran the following to get verbose connection info:
ssh -v servername</p>
<p>and saw this error:
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: An invalid name was supplied
Cannot determine realm for numeric host address</p>
<p>It turns out that ssh is trying to use Kerberos (GSS-API). To turn this off we can simply edit our ssh config file.</p>
<p># open up ~/.ssh/config
# add the following line
GSSAPIAuthentication no</p>
<p>And that fixes it!</p>
Python and Emacs2012-08-12T00:00:00+00:00http://www.techsnuffle.com/2012/08/12/python-and-emacs<p>When I first started getting into using Emacs for all my editing I had major problems trying to get a decent Python setup working. I spent hours messing around on line getting nothing but a headache, it almost made me drop Emacs entirely. However after finding a couple of clearer tutorials I was able to bodge something together, but if I’m honest Its never felt good enough. <a href="https://github.com/gabrielelanaro/emacs-for-python#readme">Luckily it seems that some lovely soul has put together this</a>!</p>
<p>In the author’s words, the goal is <em>“collecting and customising the perfect environment for python development, using the most beautiful emacs customisation to obtain a really modern and exciting (yet stable) way to edit text files.”</em></p>
<p>Next time I do some Python hacking (i.e. when I can tear myself away from Lisp!) I’ll try this out and let you know how it went.</p>
<p>Hopefully this can help some other Python loving Emacs noobs get started with a really slick python environment.</p>
<hr />
<p>In other news I’ve written a .obj file parser so I can read some 3D models into cepl (my lisp opengl experiment). I should have something to show you by the end of this week.</p>
<p>Goodnight!</p>
Entity Systems in Games2012-08-11T00:00:00+00:00http://www.techsnuffle.com/2012/08/11/entity-systems-in-games<p>As I have been making good headway on studying OpenGL I have been starting to have a look into more of the things I will need to make real games using my tools.</p>
<p>One of the obvious first steps is working out how I will manage all the different types of ‘things’ (lights, boxes, triggers, explosions, hats..and err..BIG HATS) in the games and also how to defining and extending their functionality as easy and powerful as possible.</p>
<p>It doesnt take much googling to see that object-orientated programming is not actually the way to go. This seems strange at first as OO can seem a natural fit for defining types of object. I mean you can have something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>character
|
|__human
| |___soldier
| |___civilian
|
|__alien
|____alien queen
|____alien soldier
</code></pre></div></div>
<p>But it turns out that in real life, things change (who knew?!) and, as new type of object are need in the game, functionality that use to be specific to one class of game object has to be moved up the tree.</p>
<p>For example, lets look at out pretend object hierarchy above. When the game designers first added the ‘alien solider’ they wanted to let it be able to call re-enforcements, this is fine but the ‘human->solider” also has this functionality. As good coders we want to reduce code duplication but where do we put the ‘call re-enforcements’ functionality? The only shared class is the ‘character’ class so we would have to put it there…but now this means that the ‘civilian’ and the ‘alien queen’ also inherit ‘call re-enforcements’ even though they will never do this in-game.</p>
<p>It turns out that this kind of thing happens a lot, and you start getting classes (like ‘character’ here) which become dumping grounds for functionality. These classes are often called Blobs.</p>
<p>Ok so if this isn’t what we want then what other options do we have?</p>
<p>Well the one that is interesting me is called an Entity System. It detatches the functionality from the data which allows for composition of game objects in a way that is much more flexible than multiple inheritance …well rather than me doing a poor job replicating the articles I have read I will just link them below, hopefully they will help you as much as they have me.</p>
<ul>
<li><a href="http://t-machine.org/index.php/2007/09/03/entity-systems-are-the-future-of-mmog-development-part-1/">Entity Systems are the future of MMOG development</a> This is a series of posts on the Entity Sytems, give yourself some time to read throught these.</li>
<li><a href="http://t-machine.org/index.php/2007/09/03/entity-systems-are-the-future-of-mmog-development-part-1/">Evolve Your Hierarchy - Refactoring Game Entities with Components</a> This has a narrower focus but good explanations, it was the first article I read on the subject.</li>
<li><a href="http://gamedev.stackexchange.com/questions/31473/role-of-systems-in-entity-systems-architecture">Role of systems in entity systems architecture</a> If the above have thrown you a little this is a brilliant breakdown of entity systems.</li>
</ul>
<p>Right I’m very tired now, Goodnight Everyone!</p>
Ch-ch-ch-ch-modifications2012-08-07T20:05:00+00:00http://www.techsnuffle.com/2012/08/07/chchchchmodifications<p>Well I’ve been very quiet recently and that’s mainly down the my mild addiction to playing with Lisp. I been beavering through <a href="http://letoverlambda.com/">Let Over Lambda</a> which ia fantastic book on the crazy stuff in common lisp (code that writes code etc) which has been bending my mind plenty.</p>
<p>I’ve also got sick of how the opengl interface works and have gone on something of a spring clean. I’ve made some tools on top of the exisiting common lisp opengl library in order to make it a nicer to work with.</p>
<p>Its definitely not ‘lispy’ enough yet but pushing data into buffers, making vaos and managing programs and their uniforms is much easier now. I’ve got it all <a href="http://github.com/cbaggers/cepl">up on github</a> though I wouldn’t recommend playing with it yet as I’m damn sure there will be some HEAVY changes before I’m happy with it.</p>
<p>Here is a quick video of me playing around with a small chunk of code which handles the rotation of the objects in the scene. Notice how errors don’t crash the program and I can simply recompile the offending function and carry on. Also note that the repl still works so I can mess around with the camera position from there.</p>
<iframe width="640" height="480" src="http://www.youtube.com/embed/qcahUrvytqs" frameborder="0"> </iframe>
<p>I’m now bored of these little 3D whatchamagigs now so I need to write an importer .obj files so I can get something interesting up on the screen.</p>
<p>Ciao</p>
Today's Blob of Inspiration2012-08-07T00:00:00+00:00http://www.techsnuffle.com/2012/08/07/todays-blob-of-inspiration<p>Its’ wonderful just to see some of the people involved and their passion. Watch this after you have had your fill of the landing itself.</p>
<iframe width="640" height="480" src="http://www.youtube.com/embed/uWr1DazvDVI" frameborder="0"> </iframe>
<p>It really feels like the start of something huge again, the SpaceX docking had me tearing up and this just caps it off for me, Well done to everyone involved, I cant wait to watch the next few years of this.</p>
Jekyll and YouTube Videos2012-06-21T00:00:00+00:00http://www.techsnuffle.com/2012/06/21/jekyll-and-youtube-videos<p>If you are having problems with embedding youtube videos in Jekyll blogs and getting errors then check out <a href="http://www.whatwherewhy.me/blog/2012/01/17/youtube-iframe-code-gotcha-in-maruku/">this link</a>.</p>
<p>It turns out that the markdown processor doesn’t like the ‘allowfullscreen’ part of the embed code.</p>
<p>Delete it and the video embeds just fine!</p>
Its moves, IT LIVES!2012-06-20T00:00:00+00:00http://www.techsnuffle.com/2012/06/20/its-moves-it-lives<iframe width="600" height="450" src="http://www.youtube.com/embed/XLkUI89fgRI" frameborder="0"> </iframe>
<p>Here is the first little video showing a 3d object translating and scaling while I edit the code and inject it into the running program. This really makes tweaking lots of fun as even the errors can be corrected without having to close the program and recompile.</p>
Slowly Getting There2012-06-17T00:00:00+00:00http://www.techsnuffle.com/2012/06/17/slowly-getting-there<p>Learning OpenGL in lisp is pretty cool, its my first lisp project, first time using opengl and first time learning the workings of a foriegn function interface. All this makes for hairy debugging as its always nice to learn from some familiar terortory so you have a rough idea, when something goes wrong, whether it’s you cocking up or some other bit of code.</p>
<p>This time however,no such luck.. Each problem has the potential to be me missunderstanding the tutorial..or OpenGL…or the C++ code I’m translating from…or the lispy abstractions around the c library provided by cl-opengl!
But there is progress… I’m up to tutorial 5 in the awesome modern OpenGL tutorials at <a href="http://arcsynthesis.org/gltut/">ArcSynthesis</a>.</p>
<p><img src="/assets/images/arc-tut-5.jpeg" /></p>
<p>It doesnt have a depth buffer yet so the smaller object is actually further away than the apparently larger one, but that in the next tutorial!</p>
Rosetta Code and Studying2012-06-10T00:00:00+00:00http://www.techsnuffle.com/2012/06/10/rosetta-code-and-studying<p>This is really a ‘note to self’ but if you are learning a language and have experience with other, then the <a href="http://rosettacode.org">Rosetta Code</a> really help when you hit that problem when you yell “THIS IS SO EASY IN -insert language here-!”.</p>
<p>While trying out Common Lisp and I keep coming up against little problems that I would happily tap out fluently if I was coding in Python but which in Lisp leave me scratching my head. Rosetta Code has helped a few times now and yet I keep forgettign about it…Hopefully now I’ve wirtten this it may stick!</p>
Lisp, OpenGL and Uninterrupted development2012-06-10T00:00:00+00:00http://www.techsnuffle.com/2012/06/10/lisp-opengl-and-uninterupted-developement<p>As with anything I want to learn, it seems I need a project to drive me. The project is often WAY above my ability but it makes for a very motivating goal to keep in mind while trudging through the more tedious/mind-braking parts of whatever I’m studying.</p>
<p>The project I have in mind with Lisp is a live game coding environment. I listened to an awesome talk on <a href="http://twit.tv/show/floss-weekly">FLOSS Weekly</a> a little while back on <a href="https://github.com/overtone/overtone">Overtone</a>, a live music coding environment, written in <a href="http://clojure.org">Clojure</a>. If you are struggling to see how this would work have a look at <a href="http://vimeo.com/22798433">this video</a>.</p>
<p>The real feeling I get here is that, just like in music, coding can have composing and jamming. What I mean is that programming (to me at least) can often feel like I’m composing, I am primarily solitary, noodling around with little ideas and bits of code until I get a good idea of where I’m going and then I need to plan and write so I can eventually present what I have made.</p>
<p>Jamming has the result visible from the start, and because what is happening is immediately viewable (audible) other can chip in and shape the result, even if they are not playing themselves.</p>
<p>I want to do this with code, specifically visual programming and games.</p>
<p>These ideas obviously aren’t new, they have been said plenty of times before, and there have been strides in the direction. Every time the speed of iterating in a language is improved, we are one step closer to this goal.
Recently a fantastic guy called <a href="http://worrydream.com/">Bret Victor</a> gave <a href="http://www.youtube.com/watch?v=PUv66718DII">this amazing talk</a> where he explained that “creators need an immediate connection with what they’re creating” and demonstrated it beautifully with (amongst other wicked demos) a 2d platformer where the changes he made in the code were immediately reflected in the game. This seems cool enough but the result of this immediate feedback is that he could ‘jam’ with the code, he could make changes with worrying about crashing out of the demo and just explore the ‘what ifs’ of what he was writing.</p>
<p>Now while Lisp doesnt implement this at all it does have one feature that is very nice, you can compile the code for a single function and add it to the running program. This also means you can redefine the code you are running while it’s running which is a great first step.</p>
<p>Just a minute ago I was able to load up the OpenGL tutorial I was working on which involved a triangle moving in a circle (yup I’m starting from scratch!) and, just by virtue of the code being written in lisp, I was able to change the colour, path, speed of the triangle and immediately push the changes in the code into the running program and see the effects they had.</p>
<p>I could also see how things broke if I messed with commands in the wrong way, and thanks to how lisp handles exceptions I could undo the change in the code, push the change into the program and tell the program to continue using the new code, and in only couple of seconds. I must admit, it felt fantastic!</p>
<p>I’m putting at the work I’m doing on github although there’ not really much there yet, I will post here with videos and pics as more takes shape.</p>
<p>Take care, happy jamming!</p>
Finding out you OpenGL Version under Ubuntu2012-06-10T00:00:00+00:00http://www.techsnuffle.com/2012/06/10/finding-out-you-opengl-version-under-ubuntu<p>If you need to quickly find out the version of OpenGL you have along with the version of GLSL you can use then run the following at the command line.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>glxinfo | grep -e "OpenGL.*version string"
</code></pre></div></div>
<p>Remember that if you have 2 graphics cards that you will need to be aware of which one it will be checking. For me that meant running:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>optirun glxinfo | grep -e "OpenGL.*version string"
</code></pre></div></div>
<p>As I am using <a href="https://github.com/Bumblebee-Project/Bumblebee/wiki">Bumblebee</a> to enable use of the optimus graphics setup.</p>
Jekyll, Blogging and Emacs2012-05-15T00:00:00+00:00http://www.techsnuffle.com/2012/05/15/testing-jekyll-emacs-addon<p>For those who are playing around with Jekyll and would like to be able to quickly create and publish posts within emacs then you may want to check out <a href="http://metajack.im/2009/01/02/manage-jekyll-from-emacs/">jekyll.el</a>.</p>
<p>I’ve been trying it out and it means that regardless of what I’m working on, when the idea for a post comes into my head I can C-c b n (‘blog new’ I assume) jot down the idea and carry on.</p>
<p>For anyone who needs to be able to unload ideas from their brain in order to focus on the job at hand this is a real boon.</p>
<p>When you have finished the draft it’s C-c b P (blog publish) to move the post from the draft folder to live. I can then preview it and push to the world via git.</p>
<p>Lovely!</p>
Ogre And The Beauty Of Open Source2012-05-15T00:00:00+00:00http://www.techsnuffle.com/2012/05/15/ogre-and-the-beauty-of-open-source<p>Currently I’m neglecting my other unfinished projects in favour of playing with OpenGL in common lisp.</p>
<p>I’ve been getting my head around lisp for a couple of months and really needed some kind of project so I could test out some of the things I had been reading about. Making some kind of visual extension to the repl sounds perfect and I’m curious to see how fast you can iterate when making games with in-line compilation.</p>
<p>There’s obviously a tonne of maths involved and trying to whittle down what is neccessary from all the articles online would be daunting if it wasnt for the fact that Ogre exists and is open source. I just cant get over that… Every time I open it I get the rush knowing that I can read this thing and use the knowledge I gain to go make something new, and that that is encouraged.</p>
<p>Well anyway thats my little vent, if you are looking at getting into games programming and you have a some programming knowledge under your belt then I heartily recommend going and reading some of their code, it’s very clean and coupled with a good book (I’m reading Game Engine Architecture at the mo) it can really illuminate how to get things done in real life.</p>
<p>Oh and don’t worry if you don’t know c++, neither do I…But if you have good google-foo and are used to programming you won’t have to much difficulty.</p>
<p>Ciao!</p>
First Post In Jekyll2012-05-14T00:00:00+00:00http://www.techsnuffle.com/misc/2012/05/14/testing123
<p>Ok so I am currently moving away from WordPress as I really would like to be able to edit my blog fully offline and preferably have it managed via some dvcs.</p>
<p>I’ve was looking at <a href="http://blohg.org">Blohg</a> for a time but the documentation for Jekyll combined with <a href="http://jekyllbootstrap.com/">Jekyll Bootstrap</a> was too nice a combo not to use.</p>
<p>So this is it, my new home, hope you enjoy it!</p>
So damn close2012-02-29T00:00:00+00:00http://www.techsnuffle.com/2012/02/29/So-damn-close<p><img style="display:block;margin-right:auto;margin-left:auto;" alt="image" src="http://cbaggers.files.wordpress.com/2012/02/wpid-2012-02-29-19-59-33.jpg" /></p>
<p>I'm going to be saying this for at least another couple of weeks.
I'm in that wonderfully scary bit where I have tip throw the switch and watch for the magic smoke. Wish me luck!</p>
IT LIVES!2012-02-29T00:00:00+00:00http://www.techsnuffle.com/2012/02/29/IT-LIVES<p><img style="display:block;margin-right:auto;margin-left:auto;" src="http://cbaggers.files.wordpress.com/2012/02/wpid-2012-02-29-21-13-09.jpg" alt="image" /></p>
<p>Just tested all the motors and extruded my first blob of plastic. Nothing so fine as a real print yet but also, save a small issue with some wire insulation smoking, no problems.</p>
<p>Next I need to have a good dig into the process of calibration and setting things up for a print.</p>
<p>I’ll keep you posted!</p>
<p> </p>
What a beast!2012-02-26T00:00:00+00:00http://www.techsnuffle.com/2012/02/26/What-a-beast<p><img style="display:block;margin-right:auto;margin-left:auto;" alt="image" src="http://cbaggers.files.wordpress.com/2012/02/wpid-2012-02-26-22-21-26.jpg" /></p>
<p>Nearly ready now, there is a meeting of the thames valley reprap'ers this tuesday so I'm hoping to take this there and get some advice on the firmware side of things.
</p>
Great material for getting your head around quaternions2012-02-22T00:00:00+00:00http://www.techsnuffle.com/2012/02/22/Great-material-for-getting-your-head-around-quaternions<p>I’ve been wanting to get my head around quaternions for a long time notw but am often put off by the math heavy texts I tend to stumble upon. Luckily this time around I’ve been having more luck.</p>
<p>I’ll post more details on what I’m doing with this another day but I thought you peeps might find these links handy too!</p>
<p>Part 1 of 2 articles on the origins of complex numbers and quarternions. I find it helpful to see where a subject has come from and what the drive behind it is.
<a href="http://plus.maths.org/content/os/issue32/features/baez/index">Curious Quaternions</a>
Part 2 of above
<a href="http://plus.maths.org/content/ubiquitous-octonions">Ubiquitous octonions</a></p>
<p>From the same site as the last two links but give a vague idea of what these things can be used for practically
<a href="http://plus.maths.org/content/maths-goes-movies">Maths Goes To The Movies</a></p>
<p>This site goes through the basics you need to know to start implementing quaternions in your code. Very pragmatic and easy to follow. Also even though it says ‘cprogramming’ dont be put off, the code is near enough to pseudo-code.
<a href="http://www.cprogramming.com/tutorial/3d/quaternions.html">Cprogramming.com: Quaternions and 3D Rotations</a></p>
<p>These two are useful primarily as they have some source code and easy going tone. I’m going to have a read of the rest of the site as it looks useful.
<a href="http://www.3dkingdoms.com/weekly/weekly.php?a=11">3D Programming Weekly: Quaternions</a>
<a href="http://www.3dkingdoms.com/weekly/weekly.php?a=36">3D Programming Weekly: Quaternion Math </a></p>
<p>Cool, well thats all for now, I’ll be back with more another day.
Ciao
Bagley</p>
<p>p.s. Ooh and if you are just getting into this I heartily recommend <a href="http://www.khanacademy.org/#linear-algebra">Khan Academy’s course on Linear Algebra</a></p>
Configuring Emacs2012-02-13T00:00:00+00:00http://www.techsnuffle.com/2012/02/13/Configuring-Emacs<p>…Often feels like trying to get rich by finding a gold seam and then, rather than using a pickaxe, smashing your face repeatedly into the groud. I know I’ll get there but sometimes I’m not sure its worth the pain.</p>
<p>Just needed to vent, no way I’m dropping emacs just tiring working out which way you have to tickle it for it to roll over and play nice.</p>
And just like that...2012-02-13T00:00:00+00:00http://www.techsnuffle.com/2012/02/13/And-just-like-that<p>..All is right with the world (of emacs) again.</p>
<p>Turns out sbcl + slime (+ quicklisp?) really doesnt play well if autopair is enabled.</p>
<p>You will know you have the problem if whenever you cause an exception in the repl you are unable to use the minibuffer at all without seeing the following:
<em>error in process filter: Wrong type argument: characterp, nil</em>
Or something very similar.
This effectively means killing emacs and restarting, as killing/switching buffers, closing emacs, loading files, scratching your bum are all inoperable.</p>
<p>Anyhoo how this helps you if it’s caught you out.
Ciao</p>
Lisp and opengl2012-02-03T00:00:00+00:00http://www.techsnuffle.com/2012/02/03/Lisp-and-opengl<p>WOOOO!
I’ve finally got the opengl demos opening in lisp. This has been a fecking nightmare as I am really not used to installing lisp modules or knowing what the hell to do with them when I have….turns out its all easy with quicklisp but until you know what to do with it its just another bloody mystery.</p>
<p>I’ll post a decent writeup another day but this will go a good way to let me start experimenting with generating 3D geometry in lisp.</p>
<p>Ciao</p>
War and Humanity2012-02-02T00:00:00+00:00http://www.techsnuffle.com/2012/02/02/War-and-Humanity<p><a href="http://www.ted.com/talks/peter_van_uhm_why_i_chose_a_gun.html">Peter van Uhm: Why I chose a gun</a></p>
<p><a href="http://www.esquire.com/features/essay/ESQ0307ESSAY">I Miss Iraq. I Miss My Gun. I Miss My War. (2007) </a></p>
<p>It is very easy for people to forget that those they disagree with may be good and rational people, people whos goals actually align with thiers but whos idea of the path is different. I have a lot of difficulty even agreeing with myself on what I think regarding these subjects.</p>
<p>These two links are well worth your time, I don’t want to encourage or disagree with either just yet, but we all must see that people are not one dimentional cartoons.</p>
<p>Ciao</p>
Feeling like a terrorist today?2012-02-02T00:00:00+00:00http://www.techsnuffle.com/2012/02/02/Feeling-like-a-terrorist-today<p>Well after that bout of rationality in the last post here some supreme stupidity</p>
<p>http://publicintelligence.net/do-you-like-online-privacy-you-may-be-a-terrorist/</p>
<p>Read the flier and give yourself a score, I rated fairly radical!</p>
Maths and Programming2012-02-01T00:00:00+00:00http://www.techsnuffle.com/2012/02/01/Maths-and-Programming<p>One thing that I feel very deficient on is fluency in maths. Even though a lot of what I do when programming has a mathematical edge to it I always feel like I’ve turned up in France and I only know how to ask for a baguette.
I’m been trying to squeeze in sessions on <a href="http://www.khanacademy.org/">khan academy</a> when I can but despite their awesome award badge system it can still feel like a slog.
Today however I read this which really perked me up and has really renewed my drive and I’ll definitely be taking some of his advice.</p>
<p><a href="http://steve-yegge.blogspot.com/2006/03/math-for-programmers.html">Steve Yegge’s Post on Math for Programmers</a></p>
<p>Another thing that has really been feeding my math hunger is the book <a href="http://www.amazon.com/Cryptonomicon-Neal-Stephenson/dp/0380973464">Cryptonomicon</a> which is wonderful and I will definitely be writing a post about when I’ve finished it.</p>
<p>Well that’s all for now,
Ta ta</p>
Its just beautiful...2012-02-01T00:00:00+00:00http://www.techsnuffle.com/2012/02/01/Its-just-beautiful<p><img src="http://upload.wikimedia.org/wikipedia/commons/e/ee/Giant_Aircraft_Comparison.svg" alt="" width="720" height="240" />
This isnt a post with any point but to enjoy the sheer awesome scale of these machines.
<a href="http://en.wikipedia.org/wiki/Zeppelin" title="Wikipedia article on zeppelins"></a></p>
Browsing the Web Emacs Style2012-02-01T00:00:00+00:00http://www.techsnuffle.com/2012/02/01/Browsing-the-Web-Emacs-Style<p><img src="http://i173.photobucket.com/albums/w51/Cbaggers/conkeror2.png" alt="" /></p>
<p>Well I’m on this slippery slope now so I may as well enjoy it…</p>
<p>It seems that emacs is a virus which slowly overwrites all keyboard shortcuts in your brain with it own and then make sure your grumpy whenever you hit one of these combos in Firefox and it prints-the-page saves-an-image and-shuts-down.</p>
<p>Rather than deal with this issue the rational way (stop using emacs) I have been brain-damaged and thus my solution is obvious…I need an Emacs browser.</p>
<p>Now I know that W3 exists and is well integrated to emacs but this is a text only browser and I really need a full featured web browser. Enter <a href="http://conkeror.org/">Conkeror</a></p>
<p>Conkeror is quite simply a browser based on Mozilla and XULRunner where all the control can be done through the shortcuts you have had branded onto your soul by Emacs. Those who have used the <a href="http://vimperator.org/vimperator">Vimperator</a> plugin for firefox will be familiar with the mouse free browsing style and those who often have to search on various sites will love the <a href="http://conkeror.org/Webjumps">webjumps</a> feature which allows you to define a way to quickly search a huge variety of sites from the minibuffer.</p>
<p>Well thats enough rabbiting for now, if your not into Emacs this browser will be of no interest to you but if you are chasing this particular dragon then go on, you know you want to.</p>
<p>Seeya around,
Baggers</p>
<p><a href="http://conkeror.org/">Link to Conkeror</a></p>
SQLite Databases on Windows Network Share2012-01-30T00:00:00+00:00http://www.techsnuffle.com/2012/01/30/SQLite-Databases-on-Windows-Network-Share<p>…the short version of this story is dont do it.</p>
<p>The long version is
doooooooooon’t dooooooooooo iiiiiiiiiiiiiiit</p>
<p>p.s. Oh and read this too..I should have
http://www.sqlite.org/lockingv3.html#how_to_corrupt</p>
Interesting video on Memristors2012-01-30T00:00:00+00:00http://www.techsnuffle.com/2012/01/30/Interesting-video-on-Memristors<p><a href="http://www.youtube.com/watch?v=bKGhvKyjgLY">http://www.youtube.com/watch?v=bKGhvKyjgLY</a></p>
Linux Mint and the Ugly Screensaver2012-01-29T00:00:00+00:00http://www.techsnuffle.com/2012/01/29/Linux-Mint-and-the-Ugly-Screensaver<p>Turns out that both linux mint and ubuntu 11.10 don’t have xscreensaver anymore as default. This seems fine in Ubuntu as there is a simple fade to black when needed.
Unfortunately in Linux Mint it has a butt ugly colour flashing reminiscent of monitor test patterns.</p>
<p>To fix this you need to remove gnome-screensaver and install xscreensaver.
The details can be found in the linux mint forums, <a href="http://forums.linuxmint.com/viewtopic.php?f=60&t=86563#p500030">here</a>.</p>
Playing with my christmas present2012-01-24T00:00:00+00:00http://www.techsnuffle.com/2012/01/24/Playing-with-my-christmas-present<p><img style="display:block;margin-right:auto;margin-left:auto;" alt="image" src="http://cbaggers.files.wordpress.com/2012/01/wpid-2012-01-24-21-46-21.jpg" /></p>
<p>I was incredibly lucky to get this beautiful variable temperature soldering iron this christmas and so I’m finally putting it (and me) through it’s paces. These are parts of the reprap I’m currently building.</p>
<p>Goodnight all!</p>
Ubuntu on Adam Tablet...again2012-01-23T00:00:00+00:00http://www.techsnuffle.com/2012/01/23/Ubuntu-on-Adam-Tabletagain<p><img src="http://i173.photobucket.com/albums/w51/Cbaggers/ubuntu_adam.jpg" alt="ubuntu 12.04" /></p>
<p>A good while back I got hold of an Adam tablet in order to try and forfill a desire. You see I’m a geek but I love being outdoors and I want technology to progress to where it doesn’t limit where I can do what activity.
The Adam seemed to fit the bill nicely; it uses a PixelQi screen which is perfectly readable in direct sunlight and, being a tablet, runs on an Arm processor which means the battery life is good. The only thing that was missing was the OS.
The tablet came with a really bad skin over android which was quickly removed in favour of a decent build of android, and it was OK for a time. But for me I need to be able to create on my machines, I want to code, I want Ubuntu on the tablet.</p>
<p>I, sadly, haven’t got the knowledge to put an image together for the tablet but it has been done by a bunch of fantastic people online.</p>
<p>Their ROM got me a working build of Ubuntu 11.04, which I’ve since upgraded to 12.04 to fix the massive slowdown issues I was having. I have also identified an issue where ttyS0 is being respawned constantly, if your not worried about serial then simply remove ttys0.conf from /etc/init.</p>
<p>While this is fantastic it still is a little slow for heavy use but this is likely down to the fact that I haven’t currently got the graphics drivers installed. I’ll post back my findings when I’ve done this (again an awesome person on the forums has this sussed)</p>
<p>If you would like to follow along or have an Adam Tablet gathering dust the forum is <a href="http://www.tabletroms.com/forums/adam-rom-development/3451-%5Brom%5D-ranbuntu-0-1-ubuntu-11-04-natty-adam.html">here</a>.</p>
<p>Once these last little issues are licked I’ll be having a play around with writing tablet apps for Ubuntu, so drop by if thats your thing!
Ciao</p>
Eben Moglen on the Freedom Box2012-01-23T00:00:00+00:00http://www.techsnuffle.com/2012/01/23/Eben-Moglen-on-the-Freedom-Box<p>http://www.youtube.com/watch?v=gORNmfpD0ak</p>
Recent Emacs googlings2012-01-19T00:00:00+00:00http://www.techsnuffle.com/2012/01/19/Recent-Emacs-googlings<p>Ok this is just me dumping a few more things I ended up googling for today</p>
<p>Multiple ‘windows’ pointing to the same buffer, scrolling as if one
http://stackoverflow.com/questions/970292/emacs-multiple-columns-one-buffer</p>
<p>Enable line numbers
M-x linum-mode
<a href="http://www.emacswiki.org/emacs/LineNumbers">http://www.emacswiki.org/emacs/LineNumbers</a></p>
<p>Showing row and column of cursor within file
M-x line-number-mode
M-x column-number-mode
<a href="http://www.gnu.org/software/emacs/manual/html_node/emacs/Position-Info.html">http://www.gnu.org/software/emacs/manual/html_node/emacs/Position-Info.html</a></p>
<p><a href="https://sites.google.com/site/steveyegge2/effective-emacs">Steve Yegge’s awesome article on tweaking Emacs</a></p>
<p>OK back to work!</p>
Emacs - Fixing indentation for a whole document2012-01-19T00:00:00+00:00http://www.techsnuffle.com/2012/01/19/Emacs-Fixing-indentation-for-a-whole-document<p>The indentation system on emacs is brilliant for most languages but manually going through and tabbing the lines to get the correct indentation is such a waste of time.</p>
<p>First select everything in the buffer
C-x h</p>
<p>Next fix indentation!
M-x indent-region</p>
<p>And thats it! Everything should be indented to the correct level.</p>
Windows Event Log Pain2012-01-18T00:00:00+00:00http://www.techsnuffle.com/2012/01/18/Windows-Event-Log-Pain<p>If you have an error in the Windows event log with no decent information, try this site out: <a href="http://www.eventid.net/">http://www.eventid.net/</a>
It will take a event-id and source or just the vague error message and give a decent rundown on what might have happened.
I’m getting a lot of these turnign up in Zenoss these days!</p>
Exchange Email And Linux2012-01-18T00:00:00+00:00http://www.techsnuffle.com/2012/01/18/Exchange-email-and-LinuxEvolution Into Multicelular Organism in 60 days2012-01-18T00:00:00+00:00http://www.techsnuffle.com/2012/01/18/Evolution-Into-Multicelular-Organism-in-60-days<p>Just wonderful!
http://www.wired.com/wiredscience/2012/01/evolution-of-multicellularity/</p>
Emacs on multiple monitors2012-01-18T00:00:00+00:00http://www.techsnuffle.com/2012/01/18/Emacs-on-multiple-monitors<p>Alas it seems there no decent way to have this but you can open a new frame.
C-x 5 2 or File->New Frame</p>
Coding TCL in Emacs2012-01-18T00:00:00+00:00http://www.techsnuffle.com/2012/01/18/Coding-TCL-in-Emacs<p>This is just a quick post.
At work I’m having to do a lot of TCL development and having recently become a convert to Emacs I obviously want to use it for everything!</p>
<p>The shortcuts are a bugger to remember so I thought I’d dump them here so I know where I can find them.</p>
<p>From <a href="http://www.rgrjr.com/emacs/advanced.html#tcl-mode">http://www.rgrjr.com/emacs/advanced.html#tcl-mode</a></p>
<p>Commands:</p>
<p>M-x tcl-mode
M-x inferior-tcl (C-c C-t in a Tcl source buffer)
Tcl editing:</p>
<p>C-c TAB (tcl-help-on-word)
C-c C-c (comment-region)
Tcl editing support for inferior Tcl:</p>
<p>C-c C-t (inferior-tcl) creates one.
C-c C-s (switch-to-tcl) switches to the inferior Tcl process buffer (but does not create one).
C-c C-x (tcl-eval-region)
C-M-x or C-c C-v (tcl-eval-defun)
C-c C-f (tcl-load-file) loads a Tcl file into the inferior Tcl process</p>
<p>Awesome! The eval region has to be the thing I love most from writing lis pin emacs so its great to have it for TCL.</p>
<p>Anyhoo thats all!</p>
Browsing remote servers with eshell2012-01-18T00:00:00+00:00http://www.techsnuffle.com/2012/01/18/Browsing-remote-servers-with-eshell<p>As a budding Emacs nerd I’m looking at replacing my current awesome terminal emulator <a href="http://www.tenshu.net/p/terminator.html">Terminator</a> (it’s in the Ubuntu repos) with a shell within emacs.</p>
<p>I had a play with eshell and it worked well, but I wanted to be able to use a feature called find-file-at-point which opens the file your cursor is currently over in the buffer.</p>
<p>Turns out this is pretty damn easy:
cd /ssh:username@servername:/path/you/want</p>
<p>Hit return, type in your password and you in!</p>
Asimov & Arthur C. Clarke2012-01-18T00:00:00+00:00http://www.techsnuffle.com/2012/01/18/Asimov-amp-Arthur-C-Clarke<p>A couple of lovely short stories to fill your lunch breaks.
Enjoy!</p>
<p>http://www.multivax.com/last_question.html
http://downlode.org/Etext/nine_billion_names_of_god.html</p>
test2012-01-17T00:00:00+00:00http://www.techsnuffle.com/2012/01/17/test<p>testing a new client</p>
Tips for Remote Unix Work (SSH, screen, and VNC) -2011-12-18T00:00:00+00:00http://www.techsnuffle.com/2011/12/18/Tips-for-Remote-Unix-Work-SSH-screen-and-VNC---shebang<p><a href="http://shebang.brandonmintern.com/tips-for-remote-unix-work-ssh-screen-and-vnc">http://shebang.brandonmintern.com/tips-for-remote-unix-work-ssh-screen-and-vnc</a>
Very human explanation of some very useful features of ssh</p>
Oh my!2011-12-10T00:00:00+00:00http://www.techsnuffle.com/2011/12/10/Oh-my<p>Damn I must have been writing code tired again. Theres some janky stuff in there!<br /><br />p.s. Looks like I havent entirely grasped launchpad’s branching properly. I best do some more reading</p>
Back on track2011-12-10T00:00:00+00:00http://www.techsnuffle.com/2011/12/10/Back-on-track<p>Alright, twas a mistake on my side.<br />Now into the world of comments in wordpress</p>
Back developing2011-12-10T00:00:00+00:00http://www.techsnuffle.com/2011/12/10/Back-developing<p>The new job is going well but has kept me busy. It’s one of those projects which grew lurchingly out of a experiment into a system whos goals keep being changed as moe information is found out.<br />A bit of a pain but still I’ve learnt alot. I’ll certainly be posting a good few things about making daemons in python as while there is some very good code online there are a couple of tweaks to that really help when working with multiple repositries on the same server.<br />I’ve also taken the time to pull down my code for pressly the python library and am reviewing what I have done so far so I can see how I am going to handle the next bit. I would like to get comments working properly and the wordpress API seems to be quite good with comments, so I will be able to employ some slightly different techniques that I have had to use so far.<br />Well, times’a wasting!<br /> ciao</p>
Next week will likely have more updates2011-11-04T00:00:00+00:00http://www.techsnuffle.com/2011/11/04/Next-week-will-likely-have-more-updates<p>I’m currently wrapping up a SCCM rollout where I work and once this documentation is done I’ll be glad to hand it over.</p>
<p>Honestly I don’t know why Microsoft ever make exams. They should just set the student down with a undermaintained network and a sccm install dvd and wait.</p>
<p>If the student lives they pass.</p>
<p>I feel proud yet slightly dirty to be finishing this successfully and am ready to dive headlong into studying Solaris to purge this all from my mind!</p>
<p>Anyhoo hope your all well, chat to you soon!</p>
Long time no posting2011-10-26T00:00:00+00:00http://www.techsnuffle.com/2011/10/26/Long-time-no-posting<p>It has been some time!</p>
<p>Been off on holiday in bulgaria, have got the reprap started (and well on the way to finishing), been to my first Reprap user group meetup which was fantastic, been building a small stove…and obviously craploads of stuff at work too!</p>
<p>SO aye, sorry for the quiet spot, I’m back and blogging.</p>
<p>Ciao</p>
Zenoss cron jobs...2011-09-15T00:00:00+00:00http://www.techsnuffle.com/2011/09/15/Zenoss-cron-jobs<p>…Note to self, when writing cron jobs to run as the zenoss user make sure you set the $ZENHOME environment variable in cron or in your script.</p>
<p>I’ve been a total noob chasing this around for days</p>
Can't copy files Virtualbox shared folder2011-09-13T00:00:00+00:00http://www.techsnuffle.com/2011/09/13/Cant-copy-files-Virtualbox-shared-folder<p>Just had a very weird issue here that turned out to have a very logical solution.</p>
<p>I set up a shared folder between my virtual guest and the host machine. I set it to auto mount and threw a couple of files into the folder.</p>
<p>I booted up the virtual windows instance and browsed to the shared folder and tried to copy the files out of the shared folder…it worked but the files promptly vanished and were replaced by one file with 0 byte size.</p>
<p>Bugger…I tried again; this time browsing to a samba network share and pasting the files….again within seconds they were replaced with a 0 byte file.</p>
<p>The problem was that the filename given by the linux screenshot program was not legal on windows. So I changed the file names and suddenly it all worked fine.</p>
<p>So the lesson is, check them names!</p>
<p>Hope this helps someone</p>
Need to run a Python program as a Windows Service?2011-09-12T00:00:00+00:00http://www.techsnuffle.com/2011/09/12/Need-to-run-a-Python-program-as-a-Windows-Service<p><a href="http://stackoverflow.com/questions/32404/can-i-run-a-python-script-as-a-service-in-windows-how">Then this will help!</a></p>
<p>[EDIT]
If you need to run something every ‘x’ seconds, have a peek at <a href="http://ryrobes.com/python/running-python-scripts-as-a-windows-service/">this page</a>. I must admit I don’t like the executing of a external script but its easily merged with the previous example to make something nice.</p>
Light At The End Of The Tablet Tunnel2011-09-09T00:00:00+00:00http://www.techsnuffle.com/2011/09/09/Light-at-the-end-of-the-tablet-tunnelVMWare Web Services SDK2011-09-08T00:00:00+00:00http://www.techsnuffle.com/2011/09/08/VMWare-Web-Services-SDK<p>Somehow this managed to pass me by but vSphere has a webservices interface. I really want to look at writing a python wrapper for this sometime in the future! If you are venturing down this path have a peek at these webinars: <a href="http://communities.vmware.com/docs/DOC-10751">http://communities.vmware.com/docs/DOC-10751</a></p>
<p>Hope it helps</p>
Quick and Dirty query to find out what instances of SQL a Windows box has (if any)2011-09-08T00:00:00+00:00http://www.techsnuffle.com/2011/09/08/Quick-and-Dirty-query-to-find-out-what-instances-of-SQL-a-Windows-box-has-if-any<p>Needed something to do this from Linux quickly.</p>
<table>
<tbody>
<tr>
<td>net rpc service list -W domain.work.org -S servername -U domain\username</td>
<td>grep “SQL Server (“</td>
</tr>
</tbody>
</table>
<p>Hope it helps</p>
Citrix on Ubuntu 11.042011-09-08T00:00:00+00:00http://www.techsnuffle.com/2011/09/08/Citrix-on-Ubuntu-1104<p>Hey folks,</p>
<p>I’m slapping this here as its may save a couple of minutes of googling in the future.</p>
<ul>
<li>
<p><a href="http://www.citrix.com/English/ss/downloads/details.asp?downloadId=2309164&productId=1689163&ntref=clientcenter">Grab the citrix ICA client</a></p>
</li>
<li>
<p>sudo apt-get install libmotif-dev</p>
</li>
<li>
<p>Add net.ipv4.tcp_window_scaling=0 to your /etc/sysctl.conf file to fix a strange popup error</p>
</li>
</ul>
<p>Run free in the bountiful citrix’y hills!</p>
<p>Cheers to <a href="http://karuppuswamy.com/wordpress/2010/05/06/how-to-solving-libxm-so-4-error-in-citrix-receiver-icaclient-on-ubuntu/">this site</a> and <a href="http://joepcremers.com/wordpress/connect-to-citrix-access-gateway-on-ubuntu-9-10/">this site</a> for the fixes.</p>
<p>Ciao!</p>
Remote execute windows command from Ubuntu (psexec for linux)2011-09-07T00:00:00+00:00http://www.techsnuffle.com/2011/09/07/Remote-execute-windows-command-from-Ubuntu-psexec-for-linux<p>Hey all, just a quickly as this took a little googling.</p>
<p>If you need winexe to remote administer a windows box..</p>
<p>Grab a package from here: <a href="https://launchpad.net/~jdthood/+archive/winexe/+packages">https://launchpad.net/~jdthood/+archive/winexe/+packages</a></p>
<p>Install it</p>
<p>Finally open a terminal and try:</p>
<p>winexe -U domain\username //machinename ‘cmd’</p>
<p>Also you may want to hunt down wmic (wmi-client), its also not in the repos, but is available to be compiled from zenoss.</p>
<p>If I find the guide for this again I’ll make another post about it.</p>
<p>Hope it helps!</p>
The bestt thing on the internet2011-09-04T00:00:00+00:00http://www.techsnuffle.com/2011/09/04/The-bestt-thing-on-the-internet<p>Tonight I found a site called wiki summaries.</p>
<p>It summarises books.</p>
<p><a href="http://www.wikisummaries.org/Green_Eggs_and_Ham">There is a summary of green eggs and ham</a></p>
<p>READ IT!</p>
Python, Suds and Microsoft Web Services2011-08-30T00:00:00+00:00http://www.techsnuffle.com/2011/08/30/Python-Suds-and-Microsoft-Web-Services<p>Hey all,</p>
<p>If you need to interact with Microsoft Web Services from python grab suds from whatever repo you have handy and relax! I had a brief dabble with Soappy but it seems this is now deprecated and had a couple of issues.</p>
<p>Here is your quickstart to talk to a soap based .net web service with suds</p>
<pre class="brush: python;">
from suds.client import Client
#chances are you have a asmx url, make sure you add
# ?wsdl on the end
url = "https://www.yourwebsite.org/yourwebservice/webservice.asmx?wsdl"
client = Client(url)
print client.service.yourMethod(args)
</pre>
<p>Hope it helps!</p>
Cross platform Python AD Library on launchpad later this week!2011-08-30T00:00:00+00:00http://www.techsnuffle.com/2011/08/30/Cross-platform-Python-AD-Library-on-launchpad-later-this-week<p>The code for my AD library ADdled will be uploaded later this week to <a href="https://launchpad.net/addled">this page</a>.</p>
<p>It will have basic functionality for connecting, searching and retrieving data and the beginnings of the write-back methods.</p>
<p>I’ll be expanding this along with <a href="https://launchpad.net/pressly">pressly</a> over the coming weeks in order to get them up towards something I’d be happy calling a beta.</p>
<p>In that time I also need to get myself familiar with the unittest system for python as I want to have decent test coverage. One thing I don’t know yet is how to write decent tests for systems that require login details…ah well I’m sure that will be revealed soon enough!</p>
<p>Goodnight all!</p>
Pressly goes live!2011-08-29T00:00:00+00:00http://www.techsnuffle.com/2011/08/29/Pressly-goes-live<p>And we are up on launchpad, got to say the information provided by the Ubuntu Developer week logs was stellar and really helped me get going.</p>
<p>I’m much more used to writing things either just for work or for my own use so getting code out there in this way feel great.</p>
<p>For those who want to have a tinker, you can find it <a href="https://launchpad.net/pressly">here</a>.</p>
<p>I’m also beavering away at getting my Active Directory library up on line soon. I’ll keep you posted on that as it develops.</p>
Thats what I'm talking about!2011-08-26T00:00:00+00:00http://www.techsnuffle.com/2011/08/26/Thats-what-Im-talking-about<p>Dammit, sometimes it just all seems to fall into place.</p>
<p>Just 2 minutes ago, I realised I hadn’t tagged the last 4 posts and so I thought I best test the library:</p>
<pre class="brush: python;">
for index,post in list(enumerate(blog[:4])):
post["tags"]=["python","wordpress"]
blog[index]=post
</pre>
<p>That’ll do just fine!</p>
<p>Right, next stop.. comments!</p>
Fun with Tags2011-08-26T00:00:00+00:00http://www.techsnuffle.com/2011/08/26/Fun-with-Tags<p>As I mentioned a few posts back, tags are a bit weird. I had assumed that you would be able to query for a list of posts belonging to a certain tag but apparently not.</p>
<p>What I have gone for instead is to scrape the rss feed for the tag, and from that grab the post ids.</p>
<p>As there could be hundreds the call returns post objects which behave like dicts but can retrieve the data from the server as it is accessed.</p>
<p>So to grab all the posts tagged ‘python’ and print the body text of the first post you can do this:</p>
<pre class="brush: python;">
tagged_posts = blog["python"]
print tagged_posts[0]["body"]
</pre>
<p>Simple!</p>
Editing a post2011-08-26T00:00:00+00:00http://www.techsnuffle.com/2011/08/26/Editing-a-post<p>Here’s what you do to change a post’s title with my library:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># get most recent post
post = blog[0]
# edit the posts title
post.title = "hey a new title!"
# push post back to blog
blog[0] = post
</code></pre></div></div>
<p>Tags are an interesting beasty with wordpress, I can’t query for all posts with a tag, but I can get the rss feed for a tag, so I’m hoping I can pull the info from there.</p>
<p>Will keep ye posted</p>
API Changes2011-08-26T00:00:00+00:00http://www.techsnuffle.com/2011/08/26/API-Changes<p>Hey folks,</p>
<p>Made some changes to keep things a native as possible.</p>
<p>Posts behave as dicts now, to make a post you now just have to write:</p>
<pre class="brush: python;">
post= {"title": "example title",
"body": "example body text",
"tags": ["wordpress", "api", "python"],
"live": True}
blog.append(post)
</pre>
<p>Internally the library uses a dict like object to present posts but that is just so that some data can be retrieved lazily.</p>
<p>I’m close to an beta release now so I’m really going to have to hurry up and get on launchpad!</p>
test post2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/test-post<p>Test text
New line
Conclusion…BYE!</p>
anther test2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/anther-test<p>Well this is boring!</p>
The simple Wordpress Library in practice2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/The-simple-Wordpress-Library-in-practice<p>Ok so here is how you make a simple post using the new python library I’m writing</p>
<pre class="brush: python;">
import wpress
blogs = wpress.WP("https://cbaggers.wordpress.com/xmlrpc.php",
"username", "password")
blog = blogs["Baggers on the Web"]
new_post = wpress.WPPost()
new_post.title = "First Post Test!"
new_post.body = """Hey all!
This is my first live test of my python library.
In the next post I will show the code used to post this post.
Wooo!"""
new_post.tags=["wordpress","python"]
new_post.live=True
blog.append(new_post)
</pre>
<p>That’s all there is to it! When you connect to wordpress a dict of your blogs is returned, you pull out the one you want and append on a post.</p>
<p>If you want to pull the first 6 posts you do this</p>
<pre class="brush: python;">
import wpress
blogs = wpress.WP("https://cbaggers.wordpress.com/xmlrpc.php",
"username", "password")
blog = blogs["Baggers on the Web"]
first_six = blog[:6]
</pre>
<p>I’m crashing out now but I’m going to post on working with xmlrpc in general soon…there have been some interesting gotchas.</p>
<p>Goodnight!</p>
Need a mapping of SQL Version Numbers to Release and SP?2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/Need-a-mapping-of-SQL-Version-Numbers-to-Release-and-SP<p>I did…and <a href="SQL Server 9.0.4035.00mangalpardeshi.blogspot.com/2009/02/sql-server-versions-and-editions.html">I found it here</a></p>
My reading for tonight2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/My-reading-for-tonight<p>This looks like about the best place to start on my road to Ubuntu development: <a href="https://wiki.ubuntu.com/MeetingLogs/devweek1107/GetStarted">Getting Started with Ubuntu Development</a></p>
<p>Cheers to dholbach and everyone in the Ubuntu developer week. I’ll post back how this goes.</p>
LDAP and Binding to a specific server2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/LDAP-Binding-to-a-specific-server<p>Hey all,</p>
<p>I was looking into the <a href="http://support.microsoft.com/kb/891995">uSNChanged</a> value in ldap and saw that (at least in AD) it is not replicated. This meant I would have to query each server individually…then I realised I didnt know how!…turns out I’ve been connecting just by domain, so a quick google later and I found <a href="http://blogs.technet.com/b/heyscriptingguy/archive/2005/05/17/how-can-i-access-active-directory-on-a-specific-domain-controller.aspx">this</a>:</p>
<pre class="brush: python;">
"LDAP://servername/dc=work,dc=co,dc=uk"
</pre>
<p>Nice!</p>
Implement 'slice' in a custom Python object2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/Implement-slice-in-a-custom-Python-object<p>I needed to do this a little while back while making a pythonic circular list. There used to be a command called getslice but this was deprecated a good while ago and replaced with something (in my opinion) more elegent: slice objects.</p>
<p><a href="http://docs.python.org/release/2.3/whatsnew/section-slices.html">Read all about it here!</a> (this doco is a little old, but communicates the principles well)</p>
First Post Test!2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/First-Post-Test<p>Hey all!
This is my first live test of my python library.
In the next post I will show the code used to post this post.
Wooo!</p>
BPython2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/BPython<p>I’ve been using <a href="http://pyshell.sourceforge.net/">pyshell</a> as my interactive python shell of choice for quite a while now and being quite happy with it apparently blinkered me as I missed the fantastic <a href="http://bpython-interpreter.org/">bpython</a> . It’s CLI based and provides in-line syntax highlighting and autocompletion, if the others arent quite doing it for you give it a try….hell, give it a try anyway!</p>
Another tiny app idea...2011-08-25T00:00:00+00:00http://www.techsnuffle.com/2011/08/25/Another-tiny-app-idea<p>This is one of those ideas I expected to go away after a while but its still here especially after skim reading the <a href="http://cbaggers.wordpress.com/2011/08/25/my-reading-for-tonight/">session IRC log from the last post</a>:</p>
<p>I find watching a IRC + slide session is much better than reading the logs later. Even though its both just text and visuals it feels more ‘alive’ the first time round.</p>
<p>I think I may make a sort of replay tool which lets you watch the session and changes the slides at the correct times. I’m not sure if I will integrate this into something like <a href="https://wiki.ubuntu.com/Lernid">Lernid</a> or make a tiny dedicated app…hmm!</p>
<p>Yet another thing for the todo list!</p>
Wordpress Library And Google Neglect2011-08-24T00:00:00+00:00http://www.techsnuffle.com/2011/08/24/Wordpress-library-and-google-neglectZenoss Wmi Memory Leak2011-08-23T00:00:00+00:00http://www.techsnuffle.com/2011/08/23/Zenoss-WMI-memory-leakModifying 'Gnome Blog Poster'2011-08-23T00:00:00+00:00http://www.techsnuffle.com/2011/08/23/Modifying-Gnome-Blog-Poster<p>One of the shortcomings of gnome blog poster is that it insists on treating every return key press and a paragraph…This is a pain in the backside if you are trying to post code (as you no doubt have noticed in the posts below these).</p>
<p>I’ve had a wee go at tweaking this and this post is the test.</p>
Gnome blogger sans paragraphs2011-08-23T00:00:00+00:00http://www.techsnuffle.com/2011/08/23/Gnome-blogger-sans-paragraphs<p>For those who want to use gnome blog poster sans paragraphs the fix is to modify /usr/lib/pymodules/python2.7/gnomeblog/rich_entry.py</p>
<pre class="brush: python">
Replacing line 23
self.set_pixels_below_lines(16)
with
self.set_pixels_below_lines(0)
..and..
Replacing line 26 & 27
html_converter.para_tag.opening_tag = "< p>"
html_converter.para_tag.closing_tag = "< /p>"
with
html_converter.para_tag.opening_tag = "< br>"
html_converter.para_tag.closing_tag = ""< br></pre>
<p>I’ll probably post a patch some time as there is a bug reported in bugzilla about this.
</pre></p>
Exchange Autoforwarding2011-08-23T00:00:00+00:00http://www.techsnuffle.com/2011/08/23/Exchange-Autoforwarding<p>Look like we have a good number of people who need Auto forwarding removed from their account: This is <a href="http://osherove.com/blog/2007/8/23/auto-forward-emails-from-exchange-to-your-gmail-or-other-ema.html">easily done</a> through AD Users and Computers on Windows (though in that link where it says ‘Mail Delivery’ it should be ‘Delivery Options’ ) However it looks like this will be very easy through LDAP also as their is an attribute called ‘altrecipient’ which returns the distinguished name of the contact in AD. I’ll ensure I wrap this in my library, might be a job for idle time tonight!</p>
<p>Anyhoo back to business!</p>
Blogging from Python2011-08-23T00:00:00+00:00http://www.techsnuffle.com/2011/08/23/Blogging-from-Python<p>This looks like it will be a perfect base for a library.</p>
<p>If anyone is thinking of rolling there own wordpress python library start here:</p>
<p><a href="http://code.google.com/p/python-blogger/">python-blogger</a></p>
AD, Ldap, Range2011-08-23T00:00:00+00:00http://www.techsnuffle.com/2011/08/23/AD-Ldap-Range<p>When using ldap to interact with Active Directory you can sometimes run into an odd issue with ranges.</p>
<p>Say for example you have a group with more than 1500 members. When you query the properties of the group where you would normally see ‘member’ you see ‘member;range=0-1499’ so you only get the first 1500 results.</p>
<p>Apparently you can change this setting on the server but odds are that you won’t be allowed to do this just for your scripting pleasure!</p>
<p>To remedy the issue, check out <a href="http://stackoverflow.com/questions/1000541/ldap-ad-range-attribute-how-to-use-it">this great function on stackoverflow</a></p>
Wordpress and Python2011-08-22T00:00:00+00:00http://www.techsnuffle.com/2011/08/22/Wordpress-and-Python<p>I am starting to sketch out a python library for wordpress that makes wordpress appear to python as native lists and dicts. Maybe it’s a phase I’m going through but so many things can be expressed in terms of either lists or dicts.
Wordpress could have quite a nice fit for this.
The posts are dicts is the format {“title”:””, “body”:””, “tags”:[]} etc. The tags are a dict with each tag being a key and the value being a list of posts. I would have to make things as lazy as possible so as not to try to pull down your entire archive when you log in…but it could be quite handy.
I’ll let you know if/when I have a tinker with this.</p>
Testing a wordpress client for Ubuntu2011-08-22T00:00:00+00:00http://www.techsnuffle.com/2011/08/22/Testing-a-wordpress-client-for-Ubuntu<p>Trying out ‘Blog Entry Poster’ (catchy name) as it is about as minimal as you can get…to be honest I really like it so I hope this works!</p>
Minimal Client Image Uploading2011-08-22T00:00:00+00:00http://www.techsnuffle.com/2011/08/22/Minimal-Client-Image-Uploading<p>As mentioned before I’m using “Blog Entry Poster” on ubuntu. Its small and fast and well check out how little there is to it…</p>
<p><img src="http://cbaggers.files.wordpress.com/2011/08/screenshot-post20blog20entry.jpeg" alt="" /></p>
<p>I’m using this as an opportunity to test the media uploading as apparently its just drag and drop…if you see the image above, it is!</p>
<p>EDIT: Bugger…I’ll have to dig into that then!</p>
Been a long time...2011-08-22T00:00:00+00:00http://www.techsnuffle.com/2011/08/22/Been-a-long-time<p>..since I’ve posted here. To be honest after google+ came out I was certain I was going to delete this and migrate there fully. It was an inspiring talk by <a href="http://www.lornajane.net/">lorna jane mitchel</a> at <a href="http://oggcamp.org/">ogg camp</a> however that has kicked me in gear and got me back here and realizing that I’ve got a lot of tech bit and bobs I could be posting. So hopefully some of this will end up be useful to someone, lets get cracking !</p>
And it does!2011-08-22T00:00:00+00:00http://www.techsnuffle.com/2011/08/22/And-it-does<p>Sweet I’m sorted for a client now.
For those on Ubuntu who are tyring to get their client to talk to a wordpress hosted blog the path you need is:
http://yourblog.wordpress.com/xmlrpc.php
Don’t worry about looking in the settings to ‘remote publishing’ thats an option for those who host their own wordpress blog.
Awesome well now to business!</p>
Adam in the wild2011-04-09T00:00:00+00:00http://www.techsnuffle.com/2011/04/09/Adam-in-the-wild<p><img style="display:block;margin-right:auto;margin-left:auto;" alt="image" src="http://cbaggers.files.wordpress.com/2011/04/wpid-1302355163389.jpg" /></p>
<p>Today sees the forition of a very geeky dream I’ve held for a long time. The picture above is my office for the afternoon. The dappled light of the woods is so bright that I can barely make out the screen on the phone yet the Adam tablet is crystal clear. Yes I’m a geek, but I’m an outdoor geek and finally embracing both my passions at once is very cool!
The setup does still needs some work however. All mainstream mobile OS’s seem designed for consumption and not really for making stuff. I guess that’s why people fawned over Garage Band for the iPad, its a slick app but it actually looks like you could make some serious music with it. Its a bit crap that this suprises people as these are computers but still I can see how this would be enough for many people: take a photo, answer a text, find a place to grab lunch…nice but not enough for me.
Luckily the xda and notioninkhacks forums have some sweet efforts going on to get ubuntu on these things. Right now they are just trying to get them working side by side with android but a native install is obviously the way to go at some point.
Back on the coding side of things I’ve had a quick play with ASE and touchqode. Python definately works on here and touchqode isn’t a bad editor but as my projects rely on libraries which are not pure python I’m still not abe to work on the same projects as I can on my humble netbook.
So yea this is a epic step in the right direction and I’ll keep an eye out for what’s next.</p>
<p>Ciao</p>
Adam update 12011-04-06T00:00:00+00:00http://www.techsnuffle.com/2011/04/06/Adam-update-1<p>Ok so it’s been a couple of days and this tablet really is an amazing mix. First off I have found myself saying to people a lot that you pretty much have to be someone who intends to hack their tablet, this thing really is not prime time material yet.
The issue comes down to the fact that Notion Ink’s spin on adroid, Eden, is too big a goal for them to achieve. The guys at Notion Ink have created a fantastic piece of hardware (I’ll get back to this soon but roll with me for now) but are trying to own the experience from start to finish and that just isn’t going to be possible with their size team and given how much ground the big guys have already covered. They should stop, look at what their actual strengths are, and push forward in making the best android experience leveraging their frankly awesome tablet.
Ok, so its clear that I like the device, I do, but its not without its faults. The big pull factor for me was the screen, Pixel Qi is great, I am drifting outside during my lunch breaks more and getting used to whipping it out in public to read and browse (this still is a little weird as a tablet is far from being as socially inconspicuous as a phone but hey) and coupled with 3G and firefox (I may have to dedicate a post to a passionate rant about firefox on this thing as it is wonderful) it is a beautiful experience.</p>
<p>The hardware in general is good, the processor is plenty fast enough, 3g & gps work very smoothly. The device is a touc too heavy in my opinion and you find yourself arranging yourself and world around you to rest your tablet on. The camera is poor considering what they could fit in the housing given the space, but saying that it does work (arbeit with very slow autofocus) and the spin feature is actually very useful.
When you get using this thing though you still end up running back into the OS and realising that you have a very unfinished experience. A tablet is a thin layer between you and experience and all that expeirence is dictacted by apps and there aren’t any that take advantage of Eden, and with no android market place you are placed at a huge disadvantage to even those who bought crappy underpowered chinese tablets.
Logically the only thing to do was root the device and root it I did! <a href="http://www.youtube.com/watch?v=dN2_a7WSkRM">This video</a> at youtube is a wonderful thing for the first time ‘rooter’ asit is one continuous shot of a guys rooting his adam while clearly explaining what he is doing. I quite honestly just sat down and rooted right along with him! ( I’m sure any australians reading this a smirking away to themselves right now!). Once complete the android market is available and this just transforms the device; I have all my podcast and rss feeds pouring into this thing, along with wordpress, bbc, youtube, winamp…the list goes on. The point is that I suddenly had a computer.
Yet still, as before, you find yourself bumping into the interface again and so soon I am looking at two things.</p>
<ul>
<li>
<p>First I am going to get stock android on here. No it’s not designed for tablets but still I expect it to get in my way less</p>
</li>
<li>
<p>Second I am going to start trying to get Ubuntu booting on here. THis really comes down to the fact that this tablet as it is is a great consumption device but a really poor creation one, let’s face it…touch keyboards suck, and as a coder I just don’t have the tools I want on here yet.
Ok so the summary (as I really need to get some sleep):</p>
</li>
</ul>
<p>The hardware is great. A little heavy, and not nearly as thin as soem tablets but still, if you want Pixel Qi this is awesome.</p>
<p>This is a tablet for techy people right now. I have no doubt that Notion Ink could make this very appealing to people but as it stands you really need to root it to start getting real value from it.
Anyhoo Goodnigt. I’m going to try and make shorter, less rambling posts, in future as I’m still getting the hang of this.</p>
<p>Thanks for dropping in!</p>
Allo Adam!2011-04-02T00:00:00+00:00http://www.techsnuffle.com/2011/04/02/Allo-Adam<p>I’m back (who am I telling? no one comes here!) I have just got an Adam tablet after many months of waiting and the best part of a year wondering if it was vapourware. For the uninitiated the adam is a tablet with a pixel qi screen which allows viewing in direct sunlight (think kindle with all the features of a regular screen.</p>
<p>It has been a major bugbear of mine for the last few years that I could only do my geeking from indoors (a limitation which has helped me cultivate the geek standard in skin pastiness!) so when I heard of this screen I knew it would let me knock this on the head once and for all.</p>
<p>Right so for the next few weeks i’ll be posting about the adam, any hacks I undertake, and anything else that springs to mind.
Catch ye later</p>
Steam punk route viewer2010-08-16T00:00:00+00:00http://www.techsnuffle.com/2010/08/16/Steam-punk-route-viewer<p>http://hackaday.com/2010/08/14/google-maps-wristlet-navigator/</p>
<p>I will be posting again soon
Ciao</p>
Robots and social engineering2010-07-07T00:00:00+00:00http://www.techsnuffle.com/2010/07/07/Robots-and-social-engineering<p>This has been around for a while but it still is lovely. It’s a great example of human engineering and our personification of anything with a recognisable features.
Enjoy!
<a href="http://www.tweenbots.com/">TweenBots</a></p>
<p>p.s. Make sure you check out the videos!</p>
Your country is a product...2010-07-05T00:00:00+00:00http://www.techsnuffle.com/2010/07/05/Your-country-is-a-product<p>…and chances are, you aren’t the target demographic</p>
Whack my sausage2010-07-05T00:00:00+00:00http://www.techsnuffle.com/2010/07/05/Whack-my-sausage<p>The other day I sauntered in from work to find the girlfriend and her brother hollering bizarre, two word phrases and cackling while hammering them into google.
They were, having just finished watching ‘<a href="http://www.davegorman.com/projects_googlewhack_adventure.html">Dave Gorman’s Googlewhack Adventure</a>’, manically in search of a googlewhack of their own.
For the uninitiated a googlewhack is, as laid down by the great Wikipedia</p>
<blockquote>..a kind of a contest for finding a Google search query consisting of exactly two words without quotation marks, that return exactly one hit. A Googlewhack must consist of two actual words found in a dictionary</blockquote>
<p>As each minute passed they each plundered deeper into their lexicons until finally, SCIOLIC NUTCRACKER!
A googlewhack, at the time that is; sadly the nature of these things is that once it has been registered, google sees the post and then there are more pages it knows of with the phrase and the googlewhack is no more.</p>
<p>But imminent death aside the glory remained! Al (the brother) had been putting forward many a suggestion with sausage as the second word and was wondering if there were any googlewhacks to be found with his beloved meat product as the star. Of course within a trice my trusty python editor was open and a program was growing…a program to Google search every adjective in a dictionary + sausage!</p>
<p>Alas once you start hammering Google with a good few thousand searches you realise that they all start failing and that you have broken the terms of usage agreement…Bugger</p>
<p>Ah well it was an interesting exercise and as I’m not going to hone a program I cant use, I may as well post it here! Aren’t you grateful, you human receptacle of unwanted programs you?!</p>
<p>Enjoy…and don’t run it,it will only bring momentary smiles and quick, swift dissatisfaction…. insert your own trite conclusion on the nature of life here.
Ciao!</p>
<pre class="brush: python;">
#this program tries to find googlewhacks with the second word as sausage!
import sys
from xgoogle.search import GoogleSearch, SearchError
import urllib
import simplejson
from threading import Thread
class whackHunter(Thread):
def __init__(self, phrase):
Thread.__init__(self)
self.phrase = phrase
self.status = -1
def run(self):
query = urllib.urlencode({'q' : self.phrase})
url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&%s'\
% (query,)
search_results = urllib.urlopen(url)
json = simplejson.loads(search_results.read())
result_base = json['responseData']
if result_base:
results = result_base['results']
self.status = len(results)
else:
self.status = -2
#load in dictionary
dict_file = open("index.adj")
dictionary = (line.split(" ")[0] for line in dict_file)
keep_searching = True
keep_watching = True
hunt_queue = []
while keep_searching:
while len(hunt_queue)<=5 and keep_watching:
try:
search_phrase = "%s sausage" % (dictionary.next(),)
new_search = whackHunter(search_phrase)
hunt_queue.append(new_search)
new_search.start()
except:
"thats the whole barrel!"
keep_watching=False
for i in range(len(hunt_queue)-1)[::-1]:
if hunt_queue[i].status!=-1:
search = hunt_queue.pop(i)
if search.status == 1:
print "OH YEAHHHH! the whack is %s" % (search.phrase,)
keep_searching= False
break
elif search.status == -2:
print "dead search %s" % (search.phrase,)
else:
print "no joy with %s" % (search.phrase,)
dict_file.close()
print "Done"
</pre>
Still no musing - Python Select Window and grab Details (alpha)2010-06-17T00:00:00+00:00http://www.techsnuffle.com/2010/06/17/Still-no-musing--Python-Select-Window-and-grab-Details-alpha<p>I have had need of a program that gave me the ability to select any window on the desktop and have it return details for me.
Turns out there is a lovely command line program specifically for this called ‘xprop’. To use it in python I have called subprocess and got it to rip the results that contain the information I need into a dictionary.</p>
<p>Its very hacky at the mo and will receive plenty of love before it is ready but as I spent so long trying to find out there was a program for this I pretty much had to post it for prosperity.</p>
<p>You have been warned!</p>
<pre class="brush: python;">
import subprocess
import tempfile
class WindowGrabber:
def __init__(self):
pass
def window_picker(self):
out = tempfile.TemporaryFile()
in_pipe = None
self.job = subprocess.Popen(["xprop",],
stdin=in_pipe,
stdout=out,
stderr=subprocess.STDOUT)
while self.job.returncode is None:
self.job.poll()
out.seek(0)
result = out.readlines()
window_details = {}
for line in result:
if line.find(" = ") >= 0:
key, value = line.split(" = ")
if value.find(", ") >= 0:
value = value.split(", ")
window_details[key] = value
print ""
if __name__ == "__main__":
a=WindowGrabber().window_picker()
</pre>
A Little Parcel Of Awesome2010-06-06T00:00:00+00:00http://www.techsnuffle.com/2010/06/06/A-little-parcel-of-awesomeTest from mobile2010-05-17T00:00:00+00:00http://www.techsnuffle.com/2010/05/17/Test-from-mobile<p>Again another nearly pointless post. This time to test the wordpress android apps. Damn nice it is too!</p>
Salutations Terra2010-05-16T00:00:00+00:00http://www.techsnuffle.com/2010/05/16/Salutations-Terra