--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/_posts/2024-06-09-schism-unicode-and-you.html	Sun Jun 09 18:20:45 2024 -0400
@@ -0,0 +1,30 @@
+---
+layout: post
+author: Paper
+title: 'Schism Tracker, Unicode, and you'
+---
+<span>Recently I've taken on adding real Unicode-awareness to Schism, and it was <i>surprisingly</i> easy, to say the least.</span>
+<br><br>
+<span>I was expecting to have to convert lots of things to be real Unicode, but nope! All that really needed to be done was to convert UTF-8 to CP437 where necessary to actually *draw* the data while keeping the internal form pure UTF-8, and then bundle everything up into a neat macro to keep everything consistent:</span>
+<figure><pre><code>#define CHARSET_EASY_MODE_EX(MOD, in, inset, outset, x) \
+	do { \
+		MOD uint8_t* out; \
+		charset_error_t err = charset_iconv(in, (uint8_t**)&out, inset, outset); \
+		if (err) \
+			out = in; \
+	\
+		x \
+	\
+		if (!err) \
+			free((uint8_t*)out); \
+	} while (0)
+</code></pre></figure>
+<span>I just shoved this macro anywhere necessary and it works perfectly fine for loading any Unicode path. For example, the Spanish word "mañana" gets displayed correctly now:</span>
+<br><br>
+<img class="drop-shadow-box center-image" src="/media/blog/schism-spanish-file-listing.png">
+<br>
+<span>The file sorting algorithms were a different beast though, and even now strverscmp doesn't have a real charset-independent variant. For strcasecmp, I had to implement (simple) Unicode case folding, which meant having a <a class="prettylink" href="https://github.com/schismtracker/schismtracker/blob/b858a5917ee7e83f7cb4da1ad698dd24159f241b/schism/charset_data.c#L183">switch statement that is almost 1500 lines long</a> and takes up about 20K of space in the binary.</span>
+<br><br>
+<span>Schism currently does not do any Unicode normalization when comparing strings. This is primarily a problem with decomposed strings (which will likely not get converted properly), though with filenames that probably shouldn't exist anyway...</span>
+<br><br>
+<span>anyway, Unicode is easy, if you can't use it properly it's a skill issue :p</span>