{"id":156,"date":"2013-12-27T17:34:59","date_gmt":"2013-12-27T16:34:59","guid":{"rendered":"http:\/\/blog.pjsen.eu\/?p=156"},"modified":"2013-12-28T21:40:12","modified_gmt":"2013-12-28T20:40:12","slug":"python-script-executed-by-cron-crashes-when-printing-unicode-string","status":"publish","type":"post","link":"https:\/\/blog.pjsen.eu\/?p=156","title":{"rendered":"Python script executed by cron crashes when printing Unicode string"},"content":{"rendered":"<p>I have written a little Python script for my personal purposes and scheduled it to be run on Raspberry Pi by cron. After some polishing work I was pretty sure it worked well and was successfully tested. What I mean by <em>to be tested<\/em> is to be executed\u00a0manually from the shell and to observe the expected results. So far so good.<\/p>\n<p>However when the script was run by cron, it failed at the line where it <strong>prints some string containing Unicode characters<\/strong>. The line executes normally when running from the shell. I suspected there is some issue with the standard output of the processes run by the cron, because in such case there is no meaningful notion of standard output.<\/p>\n<p>As it turns out, programs executed by cron have no terminal attached to their standard output. According to <a href=\"http:\/\/unix.stackexchange.com\/questions\/105058\/bash-if-script-is-called-from-terminal-echo-stdout-to-terminal-if-from-cron-do\">this Stack Exchange post<\/a> the standard output is sent to the system&#8217;s mail system and thus delivered to a user this way. One can easily verify this by issuing <code>tty<\/code> command in the cron and redirecting the output to a file. Something similar to <code>this is not a terminal<\/code> (message translated directly from my system with Polish locale) should be observed.<\/p>\n<p>The further explanation goes as follows: if there is no terminal attached to the process, the Python interpreter cannot detect the encoding of the terminal (it has nothing to do with the system&#8217;s environmental variables describing locale. They are all global and are part of the process&#8217; environment, but they do not affect the non-terminal device being attached to standard output). You can verify this by running a Python script that tries to output terminal&#8217;s encoding: <code>print sys.stdout.encoding<\/code>. The <code>None<\/code> can be observed. So, the interpreter falls back to Ascii encoding and crashes when printing Unicode characters.<\/p>\n<p>The solution in this case was to enforce the interpreter to use Unicode encoder for standard output.<\/p>\n<pre class=\"brush:py\">UTF8Writer = codecs.getwriter('utf8')\r\nsys.stdout = UTF8Writer(sys.stdout)<\/pre>\n<p>The output is discarded at all, but these lines prevent the interpreter from using default Ascii encoder which is not appropriate for printing Unicode string.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have written a little Python script for my personal purposes and scheduled it to be run on Raspberry Pi by cron. After some polishing work I was pretty sure it worked well and was successfully tested. What I mean by to be tested is to be executed\u00a0manually from the shell and to observe the<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,7],"tags":[],"class_list":["post-156","post","type-post","status-publish","format-standard","hentry","category-general-programming","category-quick-tip"],"_links":{"self":[{"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=\/wp\/v2\/posts\/156","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=156"}],"version-history":[{"count":0,"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=\/wp\/v2\/posts\/156\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=156"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=156"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.pjsen.eu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=156"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}