---
layout: post
author: Paper
title: 'Schism Tracker, Unicode, and you'
nowplaying: 'Holy Fuck - LP'
---
<span>Recently I've taken on adding real Unicode-awareness to Schism, and it was <i>surprisingly</i> easy, to say the least.</span>
<br><br>
<span>I was expecting to have to convert lots of things to be real Unicode, but nope! All that really needed to be done was to convert UTF-8 to CP437 where necessary to actually *draw* the data while keeping the internal form pure UTF-8, and then bundle everything up into a neat macro to keep everything consistent:</span>
<figure><pre class="code-block"><code>#define CHARSET_EASY_MODE_EX(MOD, in, inset, outset, x) \
	do { \
		MOD uint8_t* out; \
		charset_error_t err = charset_iconv(in, (uint8_t**)&out, inset, outset); \
		if (err) \
			out = in; \
	\
		x \
	\
		if (!err) \
			free((uint8_t*)out); \
	} while (0)
</code></pre></figure>
<span>I just shoved this macro anywhere necessary and it works perfectly fine for loading any Unicode path. For example, the Spanish word "mañana" gets displayed correctly now:</span>
<br><br>
<img class="drop-shadow-box center-image" src="/media/blog/schism-spanish-file-listing.png">
<br>
<span>The file sorting algorithms were a different beast though, and even now strverscmp doesn't have a real charset-independent variant. For strcasecmp, I had to implement (simple) Unicode case folding, which meant having a <a class="prettylink" href="https://github.com/schismtracker/schismtracker/blob/b858a5917ee7e83f7cb4da1ad698dd24159f241b/schism/charset_data.c#L183">switch statement that is almost 1500 lines long</a> and takes up about 20K of space in the binary.</span>
<br><br>
<span>Schism currently does not do any Unicode normalization when comparing strings. This is primarily a problem with decomposed strings (which will likely not get converted properly), though with filenames that probably shouldn't exist anyway...</span>
<br><br>
<span>anyway, Unicode is easy, if you can't use it properly it's a skill issue :p</span>