{"id":23,"date":"2007-10-03T20:22:46","date_gmt":"2007-10-03T17:22:46","guid":{"rendered":"http:\/\/www.held.org.il\/blog\/?p=23"},"modified":"2007-10-03T20:22:46","modified_gmt":"2007-10-03T17:22:46","slug":"people-of-the-world-migrate-to-utf-8","status":"publish","type":"post","link":"http:\/\/www.held.org.il\/blog\/2007\/10\/people-of-the-world-migrate-to-utf-8\/","title":{"rendered":"People of the world: migrate to UTF-8!"},"content":{"rendered":"<p>The Unicode issue is usually irrelevant in countries which use Latin letters only, but is very relevant to the rest of the world.<\/p>\n<p>Every time when I saw 'Unicode support added' in OS\/software changelogs I've been thinking \"I couldn't care less\", but that was stupid. Because when 100% of the software would speak <a href=\"http:\/\/en.wikipedia.org\/wiki\/Unicode\">Unicode<\/a> (or more correct: <a href=\"http:\/\/en.wikipedia.org\/wiki\/UTF-8\">UTF-8<\/a>, which is the most popular Unicode implementation), the world would be a happier place to live in (but would still burn because of the global warming). Here's how I migrated:<\/p>\n<p><!--more-->So today I became a UTF-8 freak, and this meant I had to go through a few steps. I believe that trying to get UTF-8 support 5 years ago it would've been hell, today both Linux and KDE\/GNOME seem to have a very well UTF-8 support. So where were we? the steps:<\/p>\n<p><strong>1.<\/strong> <strong>X Support:<\/strong> I've manually added the line \"<em>export LC_CTYPE=en_us.UTF-8<\/em>\" into  <em>\/etc\/X11\/XSession<\/em> (any nicer place to put it in?) so everything that runs under X would know it should support UTF-8. I also exported <em>LC_ALL=en_us.UTF-8<\/em>, but I don't think it's mandatory. Then I restarted X, and ran the command '<em>locale<\/em>' to make sure it took effect.<\/p>\n<p><strong>2. <\/strong><strong>Converting file names to UTF-8:<\/strong> I had to convert all the file names from ISO8859-8 (hebrew encoding) to UTF-8. I used the <a href=\"http:\/\/www.j3e.de\/linux\/convmv\/\">convmv<\/a> script (available through apt-get\/yum in popular distros). Simply running the following line did the trick:<\/p>\n<p><em>convmv -f iso8859-8 -t utf-8 -r --notest \/path<\/em><\/p>\n<p><strong>3. Converting data to UTF-8: <\/strong><em>iconv -f iso8859-8 -t utf-8<\/em> should do the trick, but I didn't need it yet. I had some ID3 tags to convert, which could be done automagically by a small script using some id3tag tool + iconv, but I was lazy and re-typed manually. &lt;ashamed&gt;<\/p>\n<p><strong>4. Console support:<\/strong> Actually I was about to give up the console support, but it was so easy I couldn't resist:<\/p>\n<ul>\n<li>Set a Unicode console font<\/li>\n<li>Run the <em>unicode_start <\/em>command<br \/>\n(in Debian both are configurable in <em>\/etc\/console-tools\/config<\/em>)<\/li>\n<li>added <em>'export <\/em><em>LC_CTYPE=en_us.UTF-8'<\/em> in my ~\/.bashrc, and checked later with '<em>locale<\/em>' command after console login.<\/li>\n<\/ul>\n<p><strong>5. Testing:<\/strong> how to test if what I read is indeed UTF-8?<\/p>\n<ul>\n<li><em>unset<\/em> LC_CTYPE, LC_ALL,  so the <em>locale<\/em> command would show no sign of Unicode.<\/li>\n<li>Run from within this terminal a new xterm, check the <em>locale<\/em> command again, then this terminal should be unicode-disabled.<\/li>\n<li>If it's text, watch it through <em>hexdump<\/em>, if it's filename - watch it through <em>stat<\/em>. If it's Unicode, non-latin letters should be represented by <strong>two bytes<\/strong> (chars) each, first byte always tells the language was chosen (so first byte repeats itself quite many times) ..<\/li>\n<\/ul>\n<p>Hurray. Encoding problems no more.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Unicode issue is usually irrelevant in countries which use Latin letters only, but is very relevant to the rest of the world. Every time when I saw &#8216;Unicode support added&#8217; in OS\/software changelogs I&#8217;ve been thinking &#8220;I couldn&#8217;t care less&#8221;, but that was stupid. Because when 100% of the software would speak Unicode (or [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[75,14],"tags":[],"_links":{"self":[{"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/posts\/23"}],"collection":[{"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/comments?post=23"}],"version-history":[{"count":0,"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/posts\/23\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/media?parent=23"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/categories?post=23"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.held.org.il\/blog\/wp-json\/wp\/v2\/tags?post=23"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}