Export to GitHub

firephp - issue #166

Non-utf8 array values replaced with null


Posted on Feb 8, 2011 by Quick Wombat

What versions and operating system are you using?

OS:Win XP Firefox:Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 Firebug:1.6.1 FirePHP Server Library:0.3 FirePHP Extension:0.5.0

What is the problem?

Array/object fields that are non-ASCII strings and not encoded as UTF-8 are replaced with NULL when dumped to the Firebug console.

Array keys that are non-ASCII strings and not encoded as UTF-8 result in an error.

What steps will reproduce the problem? 1. FB::dump(array('foo'=>'bar')); 2. FB::dump(array('foo'=>'bár')); 3. FB::dump(array('fóó'=>'bar'));

What is the expected output? What do you see instead?

Expected:

array('foo'=>'bar') array('foo'=>'bár') array('fóó'=>'bar')

Actual:

array('foo'=>'bar') array('foo'=> NULL) ["There was a problem wri...ePHP/FirebugConsole/0.1", Object { name="SyntaxError", message="Bad string", more...}]

Please provide any additional information below.

The NULL part is a known bug of json_encode (array keys/values which are invalid when interpreted as UTF-8 are silently replaced with NULL); on PHP 5.3, you can detect the problem via json_last_error(), on earlier versions there is no way to detect it short of parsing the array/object yourself.

Comment #1

Posted on Feb 10, 2011 by Happy Camel

This seems to be only an issue when using FB::dump(). Try using FB::log() instead.

Comment #2

Posted on Feb 11, 2011 by Quick Wombat

The above was tested with $fb->log(). $fb->dump() seems to give equivalent results.

Comment #3

Posted on Feb 11, 2011 by Happy Camel

What encoding is your source file? Would that matter?

Any idea as to why I would not be able to reproduce it?

Comment #4

Posted on Feb 16, 2011 by Quick Wombat

The source file is in ISO-8859-1. It certainly matters - if it would be coded in UTF-8, FB would receive UTF-8 strings, and those work fine. Since the root of the problem is (probably) json_encode (running it on the three test arrays results in '{"foo":"bar"}', '{"foo":null}' and '{null:"bar"}', respectively), maybe that is PHP version dependent? I tested it on PHP 5.2.12 w/ Suhoshin.

Comment #5

Posted on Feb 17, 2011 by Happy Camel

(No comment was entered for this change.)

Comment #6

Posted on Feb 17, 2011 by Happy Camel

(No comment was entered for this change.)

Comment #7

Posted on Feb 17, 2011 by Happy Camel

The string value encoding code is here: https://github.com/cadorn/firephp-libs/blob/master/packages/core/lib/FirePHPCore/FirePHP.class.php#L1396-1400

Could you try and patch this to get it working on your system for string values. I can then port the patch to fix string keys.

Comment #8

Posted on Feb 18, 2011 by Quick Wombat

There seem to be two problems. The first is that array keys are not utf8encoded. Adding this after line 1392 fixes the third test case giving an error message:

if (!self::is_utf8($key)) { $key = utf8_encode($key); }

The second is that mb_detect_encoding is apparently not very reliable. On my system, mb_detect_order() returns [ASCII,UTF-8] (the default) so mb_detect_encoding should return false for an ISO-8859-2 string with accents, but it returns UTF-8 instead. After commenting out lines 1413-1415, the problem disappears.

Comment #9

Posted on Feb 18, 2011 by Quick Wombat

After some testing it seems that supplying the (rather vaguely documented) 'strict' parameter for mb_detect_encoding results in actually not detecting invalid utf-8 as utf-8: (line 1391) return (mb_detect_encoding($str, 'UTF-8', true) == 'UTF-8');

Comment #10

Posted on Feb 18, 2011 by Happy Camel

Great. Thanks for figuring this out. I'll patch it up and have you try it out.

Comment #11

Posted on Mar 4, 2011 by Happy Camel

Please verify that everything is working for you now:

https://github.com/cadorn/firephp-libs/blob/master/packages/core/lib/FirePHPCore/FirePHP.class.php

Comment #12

Posted on Mar 17, 2011 by Quick Wombat

Verified. Thanks!

Comment #13

Posted on Mar 17, 2011 by Happy Camel

Great. Thanks for checking. It will be part of the next release.

Status: Committed

Labels:
Type-Defect Priority-Medium Milestone-FirePHPCore-0.3.3