The dreaded “works on my machine” test failure not only reduces trust in your automated builds and continuous integration, but could also be the sign of a bug that doesn’t manifest on your machine. I’ve run across many such cases over the years and hope to detail a few of them here, to reduce those inevitable headaches. I’ll be focusing mostly on JVM code here, but (hopefully) a lot of the principles can be applied universally.
Symptom: Tests using filenames are passing locally but failing in Jenkins
The problem is that your Jenkins instance is most likely running on a case-sensitive file system, while you’re on a case-insensitive one. For example, if a file is named Foo.txt and a test retrieves it via:
then that test will MOST LIKELY pass on Windows and Mac machines, but MOST LIKELY fail on a Linux machine (although case-sensitivity may be configurable for your file system). It’s probably best to match the case sensitivity of the file and the test anyway, so go ahead and fix that capitalization.
Symptom: Tests pass locally when run in a certain order, but fail in Jenkins when run in a different order (or tests pass/fail when run individually)
You probably have a test leak, and your Jenkins job revealed the problem. Hurray! Now, the cause of the leak may be a little harder to pinpoint, so here’s a few places to start:
Don’t have tests that rely on other test cases. This makes trusting the tests difficult, and requires inherent logic that future team members may not be privy to. Furthermore, if a test case requires certain preconditions, then those should be explicitly set within the test case itself. If you find yourself duplicating a lot of code, then that’s a sign that the data should be bootstrapped officially at the beginning, or that the code should be refactored.
Symptom: Tests pass in my timezone, but fail on the remote/onshore/offshore Jenkins server (or when you work from home)
This one will most likely be obvious, as the time in the test will be off by a few hours (and maybe a half as well). The fixes for this are pretty widespread: you can explicitly set the timezone for your tests, you can use a relative offset, or you can move to a timezone-agnostic (and/or Daylight Saving Time-agnostic) approach for time comparison. If your tests start failing in March or November, this may be why.
Having reliable tests means keeping the data consistent. Unfortunately, this includes time (which changes very frequently!). It’s fine to use the actual time rather than changing it for the test case, just be sure to either standardize or account for, the timezone and DST.
Symptom: Tests that make async calls fail in Jenkins
Since the Jenkins machine/VM/pod/container is likely running different (slower) hardware than your local machine, tests dealing with multiple threads or inadvertent race conditions may run differently and fail in Jenkins. I’ve seen this occur when the test assertions are actually completed before the async calls finish. If possible, you should run your assertions based on a Future‘s completion, but if you must, you can also add a Thread.sleep(WAIT_TIME) before your assertions. Bear in mind if you’re using Spock that then blocks are run BEFORE when, so you’d need to Thread.sleep() call at the end of your when for it to fix this problem.
I don’t love putting a Thread.sleep() in my code, particularly because the exact duration is imprecise, but it’s the simplest approach I’ve found that doesn’t involve lots of internal mocking in Spock.
Symptom: “Random” tests pass in IntelliJ, but fail when run through the command line
OK, so this one is a bit of a cop-out since the failure may not be happening in Jenkins, but it’s a problem I’ve seen come up a lot and figured would be worth addressing in this post.
Don’t rely on IntelliJ as the end-all, be-all for testing. It’s a great tool, but since it does do things a little differently, I still like to do one test run through the command line before pushing my changes. Not everyone on your team (or in the future) may be using it, and Jenkins almost definitely doesn’t.
Symptom: The dependencies were updated and now the tests are failing in Jenkins, but they pass locally
This is most likely due to cached dependencies on the Jenkins server and the new dependencies didn’t have their versions updated. If you have internal libraries or domains, then they may be causing this issue. Depending on how you spawn your Jenkins agent, you’ll either need to do a clean before your next build, or delete the pod and create a new one. I actually don’t think this is something that is inherently wrong with the Jenkins build if it means faster run times, but you do need to be cognizant of the fact that the previous run may have already pulled the dependencies. That said…
Be sure to version your packages properly for easier auditing and to prevent caching problems. You may be overwriting the module locally, but this doesn’t mean Jenkins is following the same process. If your Jenkins job relies on cached dependencies, then be sure to do so intentionally and explicitly, rather than by happenstance of “the first build of the day.”
The mysterious test failures are an annoying problem, but hopefully with these steps you’ll be able to solve and prevent them!