Why your tests may pass locally but fail in Jenkins
The dreaded “works on my machine” test failure not only reduces trust in your automated builds and continuous integration, but could also be the sign of a bug that doesn’t manifest on your machine. I’ve run across many such cases over the years and hope to detail a few of them here, to reduce those inevitable headaches. I’ll be focusing mostly on JVM code here, but (hopefully) a lot of the principles can be applied universally.
Symptom: Tests using filenames are passing locally but failing in Jenkins
The problem is that your Jenkins instance is most likely running on a case-sensitive file system, while you’re on a case-insensitive one. For example, if a file is named Foo.txt and a test retrieves it via:
then that test will MOST LIKELY pass on Windows and Mac machines, but MOST LIKELY fail on a Linux machine (although case-sensitivity may be configurable for your file system). It’s probably best to match the case sensitivity of the file and the test anyway, so go ahead and fix that capitalization.
Symptom: Tests pass locally when run in a certain order, but fail in Jenkins when run in a different order (or tests pass/fail when run individually)
You probably have a test leak, and your Jenkins job revealed the problem. Hurray! Now, the cause of the leak may be a little harder to pinpoint, so here’s a few places to start:
- The first thing you want to do is determine the source(s) of your leak (the data that is persisting into other test cases). Yes, bear in mind that there may be more than one, which could make debugging even more difficult. What I normally do is compile a list of the tests that ran in Jenkins versus how they ran locally (potentially by grepping the logs), and see where the differences are. Anything in the Jenkins job that did/didn’t run in your local list is a culprit for the leak.
- Check any persistence layers for data that is carried over. Make sure you clear/remove/undo any operations that deal with a cache, database, file, etc.
- If you have any global contexts or properties (like ThreadContext, Spring sessions, Log4j’s MDC, SystemProperty, etc.) be sure to reset those to their original states after your test case.
- Any shared objects (say, through an abstract or static class) that are changed may also be leaky when running against other test cases. You don’t want your tests to have any side effects, so revert any changes done globally.
Don’t have tests that rely on other test cases. This makes trusting the tests difficult, and requires inherent logic that future team members may not be privy to. Furthermore, if a test case requires certain preconditions, then those should be explicitly set within the test case itself. If you find yourself duplicating a lot of code, then that’s a sign that the data should be bootstrapped officially at the beginning, or that the code should be refactored.
Symptom: Tests pass in my timezone, but fail on the remote/onshore/offshore Jenkins server (or when you work from home)
This one will most likely be obvious, as the time in the test will be off by a few hours (and maybe a half as well). The fixes for this are pretty widespread: you can explicitly set the timezone for your tests, you can use a relative offset, or you can move to a timezone-agnostic (and/or Daylight Saving Time-agnostic) approach for time comparison. If your tests start failing in March or November, this may be why.
Having reliable tests means keeping the data consistent. Unfortunately, this includes time (which changes very frequently!). It’s fine to use the actual time rather than changing it for the test case, just be sure to either standardize or account for, the timezone and DST.
Symptom: Tests that make async calls fail in Jenkins
Since the Jenkins machine/VM/pod/container is likely running different (slower) hardware than your local machine, tests dealing with multiple threads or inadvertent race conditions may run differently and fail in Jenkins. I’ve seen this occur when the test assertions are actually completed before the async calls finish. If possible, you should run your assertions based on a Future‘s completion, but if you must, you can also add a Thread.sleep(WAIT_TIME) before your assertions. Bear in mind if you’re using Spock that then blocks are run BEFORE when, so you’d need to Thread.sleep() call at the end of your when for it to fix this problem.
I don’t love putting a Thread.sleep() in my code, particularly because the exact duration is imprecise, but it’s the simplest approach I’ve found that doesn’t involve lots of internal mocking in Spock.
Symptom: “Random” tests pass in IntelliJ, but fail when run through the command line
OK, so this one is a bit of a cop-out since the failure may not be happening in Jenkins, but it’s a problem I’ve seen come up a lot and figured would be worth addressing in this post.
- IntelliJ actually has its own test runner for Gradle, and unless you specify it to do otherwise, it will use that runner, potentially bypassing your configuration. You can change the runner by going to the project Settings -> Build, Execution, Deployment -> Build Tools -> Gradle -> Runner and selecting “Gradle Test Runner” for “Run tests using.” You can change the build/run phases here too if you have hooks for those. Maven has its own configuration as well, but the test runner isn’t as drastically different and I haven’t had issues with it behaving differently than my local shell.
- Another possibility is that you are passing parameters or system properties that IntelliJ is not picking up on. You can change this by clicking on the test name at the top-right dropdown, selecting Edit Configurations… -> Defaults -> Maven/Gradle/Cucumber and then populating the parameters you need. Be sure to change the defaults and not the individual specs, or the test settings may not apply on future runs.
- One final note is that you may be using a different SDK version than your build tool specifies. To check this, right-click on your project name in the Project tab, then go to Module Settings -> Platform Settings and ensure that your SDKs and Global Libraries are at the versions you expect.
Don’t rely on IntelliJ as the end-all, be-all for testing. It’s a great tool, but since it does do things a little differently, I still like to do one test run through the command line before pushing my changes. Not everyone on your team (or in the future) may be using it, and Jenkins almost definitely doesn’t.
Symptom: The dependencies were updated and now the tests are failing in Jenkins, but they pass locally
This is most likely due to cached dependencies on the Jenkins server and the new dependencies didn’t have their versions updated. If you have internal libraries or domains, then they may be causing this issue. Depending on how you spawn your Jenkins agent, you’ll either need to do a clean before your next build, or delete the pod and create a new one. I actually don’t think this is something that is inherently wrong with the Jenkins build if it means faster run times, but you do need to be cognizant of the fact that the previous run may have already pulled the dependencies. That said…
Be sure to version your packages properly for easier auditing and to prevent caching problems. You may be overwriting the module locally, but this doesn’t mean Jenkins is following the same process. If your Jenkins job relies on cached dependencies, then be sure to do so intentionally and explicitly, rather than by happenstance of “the first build of the day.”
The mysterious test failures are an annoying problem, but hopefully with these steps you’ll be able to solve and prevent them!