Any update on this? This is pretty important functionality for a testing suite...

How would you propose implementing this?
Googletest doesn't start threads, and it doesn't fork() unless you're dealing with a DEATH_TEST.

We could set a timer at the beginning of a test and then start the work for the test. When that timer expires, what exactly would you expect googletest to DO? If the code being tested isn't cancellable, what does it mean to halt that code with a timeout? What state is the test process in at that point? The test most likely will have objects that are unfinalized and abandoned, even if we could figure out how to stop the CPU from evaluating the testing thread's code.

I don't think it's an "essential" feature, so I'd like to just treat this as an ordinary feature request. I think it's nice to have if the client code has hooks to support it, but gtest is still quite useful without it.

Googletest does have test hooks in which a client might install its own timeout functionality in a way that the software-under-test will be able to respond to. That requires no changes to googletest, AFAICT. I don't see how googletest a generic one-size-fits-all hook for timeout cancellation, though.

Billy, could you point us to the "test hooks" you mention? I would like to give it a try. Thanks!

Sure. See if this does what you need:
https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#extending-google-test-by-handling-test-events

On Sat, Sep 26, 2015 at 3:10 AM, Marco Molteni notifications@github.com
wrote:

Billy, could you point us to the "test hooks" you mention? I would like to
give it a try. Thanks!

—
Reply to this email directly or view it on GitHub
#348 (comment).

Googletest does have test hooks in which a client might install its own timeout functionality in a way that the software-under-test will be able to respond to.

Billy, a sample test case such as this one (from the doc you supplied above) that demonstrate the use of test hooks to test for timeouts would be useful.

+1
I think it's an important feature. I currently run tests under Linux's timeout command, which is a crude way around this issue.

Answering @BillyDonahue - here's how I would implement this feature:

• Forking seems inevitable. So fork the main program before running any tests.
• The child runs tests, the parent monitor its progress.
• A shared memory area would be used to track the child's progress - what test is currently running, when it started and the expected end time. Use a simple structure that works lock-free.
• The parent would occasionally check if the current test has exceeded its expected end time. If yes - kill the child and exit.

Here's basically how I've implemented this for sub-second thresholds on a Unix OS :

#include "gtest/gtest.h"
#include <unistd.h>
#include <setjmp.h>
#include <signal.h>

static jmp_buf jmp_env;

static void catch_alarm(int sig)
{
    longjmp(jmp_env, 1);
}

#define ASSERT_USECS(fn, usecs) { \
    const auto val = setjmp(jmp_env); \
    if (val == 0) { \
        signal(SIGALRM, catch_alarm); \
        ualarm((usecs), 0); \
        { fn; }; \
        ualarm(0, 0); \
    } else { \
        GTEST_FATAL_FAILURE_(#usecs " usecs timer tripped for " #fn); \
    } }

For example the following should generate a fatal failure message: ASSERT_USECS(usleep(4000), 1000);

And the following should not: ASSERT_USECS(usleep(4000), 8000);

Note that this code has room for improvement. For instance: ualarm is "obsolete", and any previous signal handling for SIGALRM isn't restored.

@louis-langholtz, you suggest a nice implementation, but answers the needs very partially.

1. The biggest problem is that it works only when explicitly used. My main concern is that by mistake code will enter an infinite loop, which will hang the test instead of failing it.
2. I don't see how it mixes with other EXPECT/ASSERT macros. ASSERT_USECS(EXPECT_TRUE(foo()), 100)?
   a. It would be nicer if the fn and usecs parameters were swapped.

Perhaps an implementation based on the same idea, but implemented in the code that invokes the tests, would be an effective solution.

@ugoren longjmp past destructors is undefined behaviour in C++. I think fork (or moral equivalent on non-Posix) is going to be required for a fully robust solution.

@MartinBonner As much as I found it fun just using the alarm for this, I agree that it's not robust enough for a general solution. Too bad fnon-call-exceptions doesn't work for SIGALRM.

@louis-langholtz I found your suggestion really useful and I'm using it for a project I'm building.

I thought of extending it a bit so you can create a timed block. In the case of Boost's unit test, it provides a parameter for specifying a timeout for a test case. I'm not a C++ expert and don't know enough about google test to attach your code to the TEST macro. I thought of creating timed blocks instead.

#include <gtest/gtest.h>
#include <unistd.h>
#include <setjmp.h>
#include <signal.h>

static jmp_buf jmp_env;

static void catch_alarm(int sig)
{
    longjmp(jmp_env, 1);
}

#define ASSERT_TIMED_BLOCK_START(name, usecs) { \
    const auto val = setjmp(jmp_env); \
    if (val == 1) {\
      GTEST_FATAL_FAILURE_(#usecs " usecs timer tripped for " #name); \
    } else { \
        signal(SIGALRM, catch_alarm); \
        ualarm((usecs), 0); \
    } }
#define ASSERT_TIMED_BLOCK_END() { \
       ualarm(0, 0); \
    }

I found a better way to do this inspired by this post.

#define ASSERT_DURATION_LE(secs, stmt) { \
  std::promise<bool> completed; \
  auto stmt_future = completed.get_future(); \
  std::thread([&](std::promise<bool>& completed) { \
    stmt; \
    completed.set_value(true); \
  }, std::ref(completed)).detach(); \
  if(stmt_future.wait_for(std::chrono::seconds(secs)) == std::future_status::timeout) \
    GTEST_FATAL_FAILURE_("       timed out (> " #secs \
    " seconds). Check code for infinite loops"); \
}

We are not going to add timeouts to tests. This can be accomplished in many ways and there is very little benefit to support it in the framework itself

@gennadiycivil I use GoogleTest to autograde programming assignments, and am pushing for it to be adopted in our department. It would be much better if we didn't have to create our own timeout system that needs to track changes to and be maintained with google test but that timeouts be included in GT.

It may be easy for one company or project to have their own timeout facility and share it with each other, but getting anything adopted widely in academia is difficult. Unless it's built in, every prof. will end up having their own TA create this function. Sometimes their solution won't be very good, and that will turn into bad stories and hinder adoption of GT for grading programs.

@gennadiycivil Thanks for the quick reply.
Unfortunately I can't yet explain how it would work. I've been using the linux commanline "timeout" utility and test filters. However, that means I need to have a lot of post-processing scripts. No .json output is created when a timeout happens under that system.

I'm not really familiar enough with the inner workings of GT to say yet how it would or could be done.

Currently I have a small library of .hpps and things I can include that came from others that allow me to use GT with MPI and a quick-n-dirty timeout thing copied verbatim from here:

http://antonlipov.blogspot.com/2015/08/how-to-timeout-tests-in-gtest.html

Just trying to make it work ATM.

(In fact, that doesn't work for me. Time to learn all the newfangled bells and whistles added to C++ in the last 20 years since I learned it. ...)

Oh, sorry.

The simplest thing (from a user perspective) might be to add versions of all the TEST macros that include a timeout parameter.

Of course some tests may have memory management inside them, in that case, you might need an additional macro/something that behaves like the "finally" clause of exception handling.

Now that I've gotten the macros I linked to working, I do like them, because I can give a failure message when timeouts happen. Also this one lets me set the timeout for individual parts of the test, which is nice.

Anyway, I really do need timeouts, because students submit code that has infinite loops or communication deadlocks and it's added quite a lot of complexity to my autograders to deal with that.

Ah.

Students submit by checking their code into a git repo. Eventually we want to have continuous integration do this, but for the moment I manually submit grading jobs to our compute cluster. (And there are some security reasons to prefer this over CI.)

There is a set of scripts that checkout their repo, checkout my repo, copy only the files they were allowed to change into my fresh repo, build, run GT, run GBenchmark, and at the end a python script interprets the GT and GBench .json files to create a grade (also a .json file).

It's a parallel programming class, so there's still some difficulty in integrating this directly into a CI system.

Anyway, when the student code deadlocks, the first version of the grading system doesn't reach the end and I don't get output.

I have improved it to use the "timeout" commandline program. However, that adds several degrees of complication. I can no longer run GT once if it's an assignment with sub-parts, I have to run it for each sub-part of the assignment. It's not so bad, but now the grader has to be ready to process multiple parts, etc.

That's a whole lot of exception handling I have to write. Most of it is in BASH.

If GT had timeouts, I could at least trust that I would still get GT output in those cases.
(Of course, I probably still need a lot of exception handling for when they have segfaults. Since I am using MPI in my tests, it's harder than just spawning a sub-process and doing the tests there.)

I think this also has implications for any project running CI. With timeouts in GT, the continuous integration output would still reach the end and they would know precisely which function deadlocked. Otherwise, they just see that their CI system ran for too long.

@bryan-lunt if it helps, I tried addressing the timeout issue using the macro I posted earlier in the thread #348 (comment).

I then combine it with the usual google tests. Something like

// run some_function() and compared with some_value
// but end the function if it exceeds 3 seconds
ASSERT_DURATION_LE(3, {
  ASSERT_EQ(some_function(), some_value);
});

I made some other modifications if you are interested, but the gist of it is to run the same google tests, but have a timeout in place to end the function if it exceeds the threshold.

The issue with trying to add timeouts to the general-purpose testing framework is that it does not really "fit".
For example, take a look at the #348 (comment) earlier in this thread. In particular "So fork the main program before running any tests." - googletest is used by great many people on great many platforms, some of which would not even have forking available.

@bryan-lunt I believe your particular problem would be properly solved by doing proper CI , for example Travis has build-in timeouts for any runs.
Essentially, timeout belongs above general purpose testing framework such us googletest.

There are good suggestions in this thread. I would recommend either implementing them in your fork for your specific case and using your fork instead of the googletest master.

Thanks
G

@gennadiycivil Ok, Thanks. (And yes, automatically forking for each test would totally undermine my use-case. That will never play well with MPI. )

As to proper CI, there are a lot of institutional constraints I have to work within. In academia, we can never get what we really want.

Hmm this is odd, I copied the link to my previous post but it gave a link to another. Anyway, this is what I actually meant ... Create a new macro to address timeouts.

#define ASSERT_DURATION_LE(secs, stmt) { \
  std::promise<bool> completed; \
  auto stmt_future = completed.get_future(); \
  std::thread([&](std::promise<bool>& completed) { \
    stmt; \
    completed.set_value(true); \
  }, std::ref(completed)).detach(); \
  if(stmt_future.wait_for(std::chrono::seconds(secs)) == std::future_status::timeout) \
    GTEST_FATAL_FAILURE_("       timed out (> " #secs \
    " seconds). Check code for infinite loops"); \
}

Is it not possible to add --gtest_timeout=5000 option? (for setting maximum test case run-time for all test cases to 5000 milliseconds)

This will help avoid a test-case running forever and spamming the console. (which makes it hard to find out which test case is failing because all google test output messages are cleared)

How about an assertion that repeatedly retries until it passes? If it doesn't pass within a time limit or number of attempts then it gives up and the test fails. I've found that approach useful when testing some concurrent code.

The assertion might look like this:

ASSERT_EVENTUALLY_TRUE(condition, numAttempts, timeBetweenAttempts)

For example, let's say we've got a classic producer-consumer architecture. Something is produced in one thread and we want to ensure that it is later consumed in another thread. We don't know exactly when it will happen but we know it should happen within a certain time frame. The test might look like this:

Queue q;
Producer p{ queue };
Consumer c{ queue };
p.produceSomething();
ASSERT_EVENTUALLY_TRUE(c.hasConsumedSomething(), 10, 100ms);

The assertion will immediately evaluate c.hasConsumedSomething(). If it evaluates to true then the assertion succeeds. If it evaluates to false then it will wait for 100 milliseconds and execute it again. If it still hasn't passed after a total of 10 attempts then the assertion fails.

Doing it this way means the test author has complete responsibility for thread/process synchronisation and cleanup. It won't be able to detect a hang or deadlock, nor can it kill long-running code. However, hopefully it's still useful enough for some situations and fairly simple to implement.

I'd be happy to work on it if it sounds reasonable.

Hi people! I am looking for something similar mentioned by @daravi --gtest_timeout=5000 as I am building an automated test infrastructure which runs the tests on VMs with various OS (Linux/Windows). The tests require our USB device connected to the PC - which I achieved by sharing via USB redirector. All works fine which is fantastic but...

Unfortunately, I trusted developers that they write robust automated tests (which do not hang - lock up/waiting for user input) but unfortunately everyone is a human and makes mistaked 😅 Though it makes my interactive session hang forever and I cannot get tests output back even if I kill the session.

What I tried to implement was to run the tests individually and give them timeout (scheduling them on a Job) but unfortunately it does not close the device gracefully (sometimes the device is left in the unresponsive state if we just kill the process - needs reconnecting to reset the state which I cannot afford time-wise (we have thousands of tests to run)).
Hence now I am hoping to implement it within the tests framework itself as you do not have --gtest_timeout=xxx option - currently I am looking at @pinventado solution (thanks for sharing!) - I will give it a go now.

However I just wanted to add up to the request - it would be cool to get this option into your framework :) We use it not only for (quick) unit tests but also for integration and system tests with real devices (as described above) and it is extremely useful to us. Your framework is very powerful and I am glad we could use it but it could be even cool-er 😏
I believe, at least for my case, I would need a cleanup task (like current Teardown()) which would clean up the device state before terminating.

Cheers!

We are not going to add timeouts to tests. This can be accomplished in many ways and there is very little benefit to support it in the framework itself

How do we achieve those ways via command line? If I try to pass my own variable for specifying the timeout, it will not run and tell that I should use only the ones which GTest supports.

Do you think --gtest_print_time flag is more beneficial than the timeout flag? Please reconsider your answer.

Both Bazel and CMake (through CTest) support timeouts. This is how most people run their tests.

That would only apply to running a whole testsuite program not a single test. Using gtest_filter to run each individual tests would be too much boilerplate and overhead.

We need to run individual tests limited to 5 minutes each, as per client requirements. Then, this feature is indeed needed.

I managed to do some brutal and ugly solution encapsulating the tests execution from outside. A lot of refinement is needed on it, but this is what I came up with:

 #include "gtest/gtest.h"
 #include <chrono>
 #include <thread>
 #include <atomic>
 #include <string>
 #include <csignal>
 #include <iostream>
 
 int make_sum(int a, int b) {
     return a + b;
 }
 
 // Custom GTest listener to enforce timeout
 class TimeoutListener : public ::testing::EmptyTestEventListener {
 public:
     explicit TimeoutListener(std::chrono::minutes timeout) : timeout_(timeout) {}
 
     void OnTestStart(const ::testing::TestInfo& test_info) override {
         timeout_reached_.store(false);
 
         // Start the timeout timer in a separate thread
         timer_thread_ = std::thread([this, &test_info]() {
             auto start_time = std::chrono::steady_clock::now();
             while (true) {
                 auto elapsed = std::chrono::steady_clock::now() - start_time;
                 if (elapsed >= timeout_) {
                     timeout_reached_.store(true);
                     std::cerr << "[TIMEOUT] Test " << test_info.test_suite_name() << "."
                               << test_info.name() << " exceeded " << timeout_.count() << " minutes. Terminating.\n";
                     raise(SIGABRT); // Interrupt the test execution
                     break;
                 }
                 std::this_thread::sleep_for(std::chrono::milliseconds(100)); // Avoid busy-waiting
             }
         });
     }
 
     void OnTestEnd(const ::testing::TestInfo& test_info) override {
         if (timer_thread_.joinable()) {
             timer_thread_.detach(); // Detach if test ends normally
         }
         if (timeout_reached_.load()) {
             ADD_FAILURE() << "Test timed out after " << timeout_.count() << " minutes.";
         }
     }
 
 private:
     std::chrono::minutes timeout_;
     std::thread timer_thread_;
     std::atomic<bool> timeout_reached_{false};
 };
 
 // Helper function to add the listener
 void AddTimeoutListener(std::chrono::minutes timeout) {
     ::testing::UnitTest::GetInstance()->listeners().Append(new TimeoutListener(timeout));
 }
 
 // Example test cases
 TEST(MyTest, ExampleTest) {
     std::this_thread::sleep_for(std::chrono::minutes(2)); // 2 minutes, under timeout
     ASSERT_TRUE(true);
 }
 
 TEST(MyTest, LongRunningTest) {
     std::this_thread::sleep_for(std::chrono::minutes(6)); // 6 minutes, exceeds timeout
     ASSERT_TRUE(true);
 }
 
 TEST(MyTest, WhileLoopTest) {
     int x = 5;
     while (true) { // Infinite loop
         x = -x;
     }
     ASSERT_TRUE(true);
 }
 
 // Example test cases
 TEST(MyTest, TestSum) {
     std::this_thread::sleep_for(std::chrono::minutes(2)); // 2 minutes, under timeout
     ASSERT_EQ(4, make_sum(2, 2));
 }
 
 TEST(MyTest, LongRunningTestTwo) {
     std::this_thread::sleep_for(std::chrono::minutes(6)); // 6 minutes, exceeds timeout
     ASSERT_EQ(5, make_sum(2, 2));
 }
 int main(int argc, char** argv) {
     ::testing::InitGoogleTest(&argc, argv);
     AddTimeoutListener(std::chrono::minutes(5)); // Set the timeout to 5 minutes
     int result = RUN_ALL_TESTS();
     return result;
 }

The main problem I get here is that if the code reaches an infinite loop there is no other option but raising SIGABRT. No other tests are run after that. This solved for our needs, but better refinement is much appreciated.