|
85
|
1 ---
|
|
|
2 layout: post
|
|
|
3 author: Paper
|
|
|
4 title: 'Schism Tracker, Unicode, and you'
|
|
|
5 ---
|
|
|
6 <span>Recently I've taken on adding real Unicode-awareness to Schism, and it was <i>surprisingly</i> easy, to say the least.</span>
|
|
|
7 <br><br>
|
|
|
8 <span>I was expecting to have to convert lots of things to be real Unicode, but nope! All that really needed to be done was to convert UTF-8 to CP437 where necessary to actually *draw* the data while keeping the internal form pure UTF-8, and then bundle everything up into a neat macro to keep everything consistent:</span>
|
|
|
9 <figure><pre><code>#define CHARSET_EASY_MODE_EX(MOD, in, inset, outset, x) \
|
|
|
10 do { \
|
|
|
11 MOD uint8_t* out; \
|
|
|
12 charset_error_t err = charset_iconv(in, (uint8_t**)&out, inset, outset); \
|
|
|
13 if (err) \
|
|
|
14 out = in; \
|
|
|
15 \
|
|
|
16 x \
|
|
|
17 \
|
|
|
18 if (!err) \
|
|
|
19 free((uint8_t*)out); \
|
|
|
20 } while (0)
|
|
|
21 </code></pre></figure>
|
|
|
22 <span>I just shoved this macro anywhere necessary and it works perfectly fine for loading any Unicode path. For example, the Spanish word "mañana" gets displayed correctly now:</span>
|
|
|
23 <br><br>
|
|
|
24 <img class="drop-shadow-box center-image" src="/media/blog/schism-spanish-file-listing.png">
|
|
|
25 <br>
|
|
|
26 <span>The file sorting algorithms were a different beast though, and even now strverscmp doesn't have a real charset-independent variant. For strcasecmp, I had to implement (simple) Unicode case folding, which meant having a <a class="prettylink" href="https://github.com/schismtracker/schismtracker/blob/b858a5917ee7e83f7cb4da1ad698dd24159f241b/schism/charset_data.c#L183">switch statement that is almost 1500 lines long</a> and takes up about 20K of space in the binary.</span>
|
|
|
27 <br><br>
|
|
|
28 <span>Schism currently does not do any Unicode normalization when comparing strings. This is primarily a problem with decomposed strings (which will likely not get converted properly), though with filenames that probably shouldn't exist anyway...</span>
|
|
|
29 <br><br>
|
|
|
30 <span>anyway, Unicode is easy, if you can't use it properly it's a skill issue :p</span>
|
