Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmp_ok($foo, "ne", "") gives the wrong expected value #6

Closed
schwern opened this issue May 27, 2010 · 10 comments
Closed

cmp_ok($foo, "ne", "") gives the wrong expected value #6

schwern opened this issue May 27, 2010 · 10 comments

Comments

@schwern
Copy link
Contributor

schwern commented May 27, 2010

  • GC-ID: 36
  • GC-Labels: Type-Defect, Priority-High, Milestone-Next-Stable
  • GC-Status: Accepted
  • GC-Attachments: No

What steps will reproduce the problem?

  1. $foo = undef; cmp_ok($foo, "ne", "");

What is the expected output?

#   Failed test at -e line 1.
#          got: undef
#     expected: anything but ""

That would be a start, but it loses the fact that this is an "ne" test. So
this would be better:

#   Failed test at -e line 1.
#     undef ne ""

What do you see instead?

#   Failed test at -e line 1.
#          got: undef
#     expected: anything else

Is this to be a special case for undef, or should it apply to all 'isnt'
comparisons? If the latter, should it apply to 'is' comparisons as well?


It will occur any time the thing on the left and right are not the same, and yet are
still equal. Another example is 0 and 0.0.

$ perl -wle 'use Test::More test => 1; cmp_ok("0", "!=", "0.0")'
1..1
not ok 1
#   Failed test at -e line 1.
#          got: 0
#     expected: anything else
# Looks like you failed 1 test of 1.

We got 0 but we expected 0.0.

The problem with the isnt() diagnostics is that it assumes that on failure what we
got is the same as what we expected. is() does not have this problem because it
shows what we got and what we expected.


Makes sense.

I've fixed this in http://github.com/gaurav/test-more/tree/issue36, but, since
'cmp_ok' throws a warning when 'undef' values are used, my new test produces these
warnings as well. Should I use $SIG{WARN} to surpress these warning messages, or
fix 'cmp_ok' to handle 'undef' correctly? I tried to do the latter, but t/undef.t
specifically checks for that warning, and I didn't want to break it in case somebody
depends on it.


@schwern
Copy link
Contributor Author

schwern commented Aug 14, 2016

Test2 has a new and different problem!

$ perl -wle 'use Test2::Bundle::Extended; my $foo = undef; cmp_ok($foo, "ne", ""); done_testing'
# Seeded srand with seed '20160814' from local date.
Use of uninitialized value $got in string ne at (eval in cmp_ok) -e line 1.
not ok 1
# Failed test at -e line 1.
# +-----+----+-------+
# | GOT | OP | CHECK |
# +-----+----+-------+
# |     | ne |       |
# +-----+----+-------+
1..1

@exodist
Copy link
Member

exodist commented Aug 14, 2016

ok, the table should show so that needs to be fixed.

not sure if you were implying the warning was a bug, the warning should be there, and as expected it is warning from the caller. trying to 'eq' against undef is usually a mistake and the warning is good.

@kentfredric
Copy link

My thoughts after seeing this is the warning aught to be part of the test output so its clear what its attached to.

# Failed test at -e line 1.
# +-----+----+-------+ ---
# | GOT | OP | CHECK | NOTES
# +-----+----+-------+ ---
# |     | ne |       | uninitialized value $got
# +-----+----+-------+ ---

Or something.

@kentfredric
Copy link

kentfredric commented Aug 14, 2016

Also in other cases where there's string overload, you'd probably want some sort of "no-overload" depiction of the terms for clarity. ( That is, showing both the stringified value in the comparison, and the sources of those stringified values for clarity )

@schwern
Copy link
Contributor Author

schwern commented Aug 14, 2016

The warning is fine. It's showing two blanks that doesn't help debug the problem. What I would expect to see is something like:

# Seeded srand with seed '20160814' from local date.
Use of uninitialized value $got in string ne at (eval in cmp_ok) -e line 1.
not ok 1
# Failed test at -e line 1.
# +---------+----+-------+
# | GOT     | OP | CHECK |
# +---------+----+-------+
# | <UNDEF> | ne | ''    |
# +---------+----+-------+
1..1

The most important part is to display the <UNDEF>. The display of whitespace should be discussed in another issue.

Anyhow, I don't think it's worth the hassle of fixing this in Test::More.

@kentfredric
Copy link

I thought that as well at first, but then I realised maybe that's not really optimal for this table.

Its optimal-ish for this case, but when you start getting into escaping, line feeds and UTF8 Characters, forcing it to be displayed in a dumper-esque format might be more trouble than its worth.

Another errant thought I had was to display a secondary table, and so horrible cases like:

cmp_ok( undef, 'ne', bless( %thing, "classname" )) where "classname" has string overloads would do:

# +-------+----+---------------------+
# | GOT   | OP | CHECK               |
# +-------+----+---------------------+
# |       | ne |                     |
# +-------+----+---------------------+
# | undef | ne | bless(...,classname)|
# +-------+----+---------------------+

Because clearly, neither the first row or the second row of this table is illuminating on its own.

@exodist
Copy link
Member

exodist commented Aug 15, 2016

I think empty strings, and strings that are only space characters (as in ascii space, not tabs, or other whitespace), should be quoted in the table. any other strings render sanely and I do not want to add quotes.

@kentfredric
Copy link

How are you going to differentiate between

$expected = q[''];

and

$expected = q[];

@schwern
Copy link
Contributor Author

schwern commented Aug 15, 2016

# +---------+----+-------+
# | GOT     | OP | CHECK |
# +---------+----+-------+
# | "''"    | eq | ""    |
# +---------+----+-------+

And before you ask.

# +---------+----+-------+
# | GOT     | OP | CHECK |
# +---------+----+-------+
# | "\"\""  | eq | ""    |
# +---------+----+-------+

We should move this discussion to Test-More/Test2-Suite#60

@exodist
Copy link
Member

exodist commented Aug 20, 2016

The test2 related bits of this should be moved to a ticket in Test2::Suite. This ticket only covers cmp_ok in Test::More.

exodist added a commit that referenced this issue Oct 30, 2016
    - Set the TEST_ACTIVE env var to true
    - Set the TEST2_ACTIVE env var to true
    - Fix the oldest bug still in the bug list (#6)
      This fixes cmp_ok output is some confusing cases
    - Update travis config
    - Add missing author deps
    - Fix handling of negative pid's on windows
    - Add can() to Test::Tester::Delegate (despite deprecation)
    - Fix some minor test issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants