Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fatal failure in SetUpTestCase() function should skip running tests. #247

Closed
GoogleCodeExporter opened this issue Jul 28, 2015 · 18 comments
Closed

Comments

@GoogleCodeExporter
Copy link
Contributor

I think googletest should skip running tests (or automatically mark as
failed) in a particular fixture if we call SetUpTestCase() function
and there is fatal failure in it. Suppose we create some shared object
in SetUpTestCase() function and try to access that in TEST_F. But the
failure to create that object in SetUpTestCase() cause to access NULL
inside TEST_F.

So if we make fatal assertion in SetUpTestCase() itself, then above
problem can be avoided.


Original issue reported on code.google.com by pvshew...@gmail.com on 11 Jan 2010 at 6:39

@GoogleCodeExporter
Copy link
Contributor Author

Original comment by w...@google.com on 11 Jan 2010 at 6:46

  • Changed state: Accepted
  • Added labels: OpSys-All, Type-Enhancement, Usability
  • Removed labels: Type-Defect

@GoogleCodeExporter
Copy link
Contributor Author

I am also facing this problem, when I add testing machoes (i.e. EXPECT_EQ) in 
SetUpTestCase errors are not reported on the console if there is any.

Original comment by l...@schouw.me on 28 Mar 2012 at 4:57

@GoogleCodeExporter
Copy link
Contributor Author

Is this problem resolved?

Original comment by irr...@gmail.com on 8 Aug 2012 at 1:42

@GoogleCodeExporter
Copy link
Contributor Author

Please, may you give an update of the issue? is it still under investigation? 
What about the TearDownTestCase()? I think that also errors in the 
TearDownTestCase() should be reported on the console and in some way invalidate 
the "Pass" TEST_F methods involved inthe fixture.

Original comment by sabrina....@gmail.com on 20 Nov 2012 at 1:06

@GoogleCodeExporter
Copy link
Contributor Author

Hi,

I'm also facing this issue now.
In my case, I'm setting a database connection so it's logical that if I can't 
set it up (e.g. if I don't have permissions to read the specified database) I 
shouldn't be able to even run my tests.
For now, I have a segmentation fault.

Original comment by ibizapea...@gmail.com on 17 Oct 2013 at 7:18

@GoogleCodeExporter
Copy link
Contributor Author

I'm having the issue also. We are testing code that interfaces with an external 
app and setup needs to fail properly if that app doesn't start.  

Currently, I have an ASSERT in setup that properly prevents running the tests, 
but the final output still says

[==========] 3 tests from 1 test case ran. (5004 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 0 tests, listed below:

 0 FAILED TESTS

But that is, in fact, incorrect. None of the tests ran and NONE of them would 
have passed if they did.

Original comment by JoeAndr...@gmail.com on 6 Nov 2013 at 10:51

@GoogleCodeExporter
Copy link
Contributor Author

I am still facing this issue. 
Is it considered to be fixed?
Could you please let us know of the status?

Original comment by trygi...@gmail.com on 17 Jul 2014 at 6:22

@GoogleCodeExporter
Copy link
Contributor Author

This is also on my wish list.

Original comment by ibizapea...@gmail.com on 12 Sep 2014 at 10:00

@hadrielk
Copy link

You can "fix" this yourself by using a test event listener, as described in the advanced user guide, by doing something like this in your derived test event listener class (which you would register in your main()):

class TestEventListener : public ::testing::EmptyTestEventListener
{
    // Fired before the test case starts and before SetUpTestCase() is invoked.
    virtual void OnTestCaseStart(const ::testing::TestCase& test_case) override
    {
        firstTestInTestCase = true;
        testCaseHadFailure = false;
    }

    // Fired before each individual test starts: before the test fixture is constructed
    // and SetUp() is invoked.
    virtual void OnTestStart(const ::testing::TestInfo& info) override
    {
        if (firstTestInTestCase) {
            size_t numFailures = numFatalFailures();

            if (numFailures > previousNumFatalFailures) {
                testCaseHadFailure = true;
                previousNumFatalFailures = numFailures;
            }

            firstTestInTestCase = false;
        }

        if (testCaseHadFailure) {
            // this will avoid running the test fixture's SetUp() and
            // TearDown() and TEST()/TEST_F(), but the test fixtures
            // constructor is still invoked (that's unavoidable)
            FAIL() << "The test case's SetUpTestCase() or executable had a fatal failure";
        }
    }

    // gets the number of fatal failures in the UnitTest's ad_hoc_test_result,
    // which is the number of fatal failures other than those inside of test
    // cases; fatal failures in SetUpTestCase() also count in this, but those
    // in SetUp(), TearDown(), and TEST()/TEST_F() do not.
    size_t numFatalFailures()
    {
        const auto& result = ::testing::UnitTest::GetInstance()->ad_hoc_test_result();
        const int numParts = result.total_part_count();
        size_t failures = 0;

        for (int i=0; i < numParts; ++i) {
            if (result.GetTestPartResult(i).fatally_failed()) {
                ++failures;
            }
        }

        return failures;
    }

    bool   firstTestInTestCase{true};
    bool   testCaseHadFailure{false};
    size_t previousNumFatalFailures{0};
};

And then register the event listener in your main(), with something like this:

    // Gets hold of the event listener list
    ::testing::TestEventListeners& listeners = ::testing::UnitTest::GetInstance()->listeners();
    // Adds a listener to the end.  Google Test takes the ownership.
    listeners.Append(new test::TestEventListener);

@behcetm
Copy link

behcetm commented Mar 2, 2016

Is there any progress on this issue? These seems like a major issue to me and should be prioritized.

tchaikov added a commit to tchaikov/ceph that referenced this issue Apr 11, 2016
see google/googletest#247, gtest does not fail
its tests even if SetUpTestCase() fails. after shutdown(), the behaviour
of RadosClient is undefined. so we'd better avoid using it anymore in
that case.

Signed-off-by: Kefu Chai <kchai@redhat.com>
@daaren-urthling
Copy link

daaren-urthling commented Apr 26, 2016

I also have this issue, partially fixed with the suggested workaround.
My problem now is that if I re-execute immediately the tests after a fail in the SetUpTestCase (i.e., to debug), none of them run, but however the general status of the test is "FAILED" (see below).
Any workaround to this?
EDIT: I re-execute the tests without terminating the .exe, as I have an (optional) UI that helps developers to select the test to be executed. The issue won't happen if the fail occurs in a TEST_F

[==========] Running 3 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] Global test environment tear-down
[==========] 3 tests from 2 test cases ran. (4 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 0 tests, listed below:

0 FAILED TESTS

@dsciebu
Copy link

dsciebu commented Apr 5, 2018

Guys! Any progress o this topic?? It's been hanging here since 2015...

@LazyWolfLin
Copy link

It's been hanging here since 2010...

I face this issue with gtest1.8, and when it will fix? I don't think test event listener is a good way to fix it by myself.

@ihsandemir
Copy link

Hello, any progress in this issue? I have the same problem.

@t-moe
Copy link

t-moe commented Nov 27, 2018

Here as well. Please take a look!

@gennadiycivil
Copy link
Contributor

Done, see 9ed99c6

@hbchai
Copy link

hbchai commented Dec 30, 2019

@gennadiycivil, can you clarify whether that change is expected to prevent test bodies from running (be skipped) when SetUpTestSuite() assertions fails? That is the crux of the current issue, so I'd like to understand whether it is actually fixed.

@panwarab
Copy link

wow , 10 years.

martiniil added a commit to PilzDE/psen_scan_v2 that referenced this issue Jul 1, 2021
Due to google/googletest#247
Can be reverted once googletest version is at least 1.11.0
martiniil added a commit to PilzDE/psen_scan_v2 that referenced this issue Jul 1, 2021
* Include bringup.launch
* Set unique udp port for data connection
* Only start laserscan validation when connected on topic
* Omit using SetUpTestSuite()
Due to google/googletest#247
Can be reverted once googletest version is at least 1.11.0
ct2034 pushed a commit to PilzDE/psen_scan_v2 that referenced this issue Aug 6, 2021
* Fix hwtest_scan_compare

* Include bringup.launch
* Set unique udp port for data connection
* Only start laserscan validation when connected on topic
* Omit using SetUpTestSuite()
Due to google/googletest#247
Can be reverted once googletest version is at least 1.11.0

* Update hwtest_readme.md

* Update forwarded udp ports
* Add description for running test with reference scan using industrial_ci
hanneskaeufler pushed a commit to hanneskaeufler/googletest that referenced this issue Jan 6, 2023
* Add rules_swift 1.2.0

* Add repository name to metadata

* Add prefix to build targets

* Remove strip_prefix since this is a release archive

* Add patch adding module extension

* Use clang on non-macOS platforms

* Download Swift for Ubuntu20.04

* Filter out broken grpc example on Linux

We do the same in the `rules_swift` repo: https://github.com/bazelbuild/rules_swift/blob/b6a111ea475da15793dfa5dda7e3e3646658f407/.bazelci/presubmit.yml#L24-L27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.