I’ve been using Shoulda for a while. And for my current project, I decided to go fixtureless with Shoulda + Factory Girl. All good, except one problem. Slow as fuck tests. So here’s fast_context as a solution for it. fast_context compiles all the ‘should’s within a context into a single test.
For example :
1 2 3 4 5 6 7 8 9 10 11 |
class UsersControllerTest < ActionController::TestCase fast_context "#new" do setup do @user = Factory(:user) get :new end should_render_template :new should_render_with_layout :users end end |
This will be compiled into something like :
1 2 3 4 5 6 7 8 |
class UsersControllerTest < ActionController::TestCase test "test: #new should run_fast" @user = Factory(:user) get :new should_render_template :new should_render_with_layout :users end end |
As your setup method just once per fast_context, the tests run much faster.
To install :
script/plugin install git://github.com/lifo/fast_context.git |
This was a result of a short time of hacking. So there may be bugs. Please comment here if you run into any issues.
Reduced functional tests time from 46 seconds to 15 seconds. Nice.
I really like this idea, have cleanly separated tests and then optionally join them together at test runtime. (and dont forget to use parallel_specs, to make it even less ;D )
@grosser : Check out tickle – which should be faster than parallel spec in theory!
I just did a find and replace of all context instances in my test folder (is that the idea?) Took my test suite from ~90ish seconds to ~60. That is a nice speed up.
@Brandon : Awesome! Yeah, that’s the primary idea. However, you cannot just blindly replace all the tests. It’s subjective. Some tests may fail when combined into one. In which case, you should not use fast_context.
Very nice. We’ve also been using Factory Girl and Shoulda, and have run into slow test speeds. One route I took to trying to solve the speed issue was to leverage Rails’ built-in test transaction support, and preload a set of common factory records before the first test runs, and then re-use them for many tests (rather than re-creating the records for each test, which is naturally much slower). The gem I wrote to help with this is at:
http://github.com/myronmarston/factory_data_preloader
One of my coworkers also tried to solve this problem in a manner similar to what you’ve done here, but created an entirely new test framework:
http://github.com/CodeMonkeySteve/mustard/
It’s just a proof-of-concept for now, but I think the idea shows promise.
@Pratik: You mention that some tests fail when combined into one. When would you expect this to occur? The obvious case is when a should has side effects (i.e. changes an instance variable or something in the database), but I’m also seeing some test failures when, as far as I can tell, there are no side-effects for any of the shoulds in my fast_context.
Also, does fast_context properly handle nested contexts, even when there are shoulds at various levels of the context tree?
@Myron : IMO, preloading a set up of known factories really defeats the purpose of using factory in the first place. I’ve been liking factories as they describe the data being tested. When you start preloading them, it’s pretty much like using fixtures w/ scenario plugin.
Also, fast_context should handle nested contexts just fine. But make sure you get the latest gist. Could you tell me a bit more about the failures you’re seeing ? If you could give me a reproducible example, I can try to debug.
Thanks!
I understand speed is a concern, but running the setup/teardown only once per context means your tests don’t run in isolation. One of the most annoying things I can think of is a test suite where the side effects of one test affect the outcome of another, and doing this is just begging for that to be the case.
@Myron: “One route I took to trying to solve the speed issue was to leverage Rails’ built-in test transaction support, and preload a set of common factory records before the first test runs, and then re-use them for many tests”
Isn’t that just fixtures, which is what factory_girl is intended to avoid?
Whenever I need to speed things up I rely on class variables. You know, in those cases where you just need “a user” (which you’re not going to modify, or you don’t care if another test modifies it).
setup do @@user ||= User.spawn(...) end
I find this approach simpler and clearer – it’s evident to anyone that all class variables are initialized only once per test case.
“I understand speed is a concern, but running the setup/teardown only once per context means your tests don’t run in isolation. One of the most annoying things I can think of is a test suite where the side effects of one test affect the outcome of another, and doing this is just begging for that to be the case.”
How could should_respond_with, should_render_template and should_render_with_layout possible affect each other? Rendering 3 times the same action to test response , template and layout is just waste of time
We seeing great results—thanks Pratik
I agree with @jon’s sentiments.
@george: The point is that a developer shouldn’t have to make the conceptual switch while writing a test to decide whether a should statement is going to change state.
I understand but this is sort of a trade off. If you don’t mind speed, isolation is something good to have. If your test suite runs in 300 seconds then you have to decide between running the tests in isolation and running them at all.
So, I guess it’s a matter of taste :)
Of course fast_context tests don’t run in isolation with this. And hence it’s not a replacement of the regular context.
HOWEVER, with over 90% of the controller tests that I write, I find it that it simply does not matter if the tests are run in isolation or not. And for those cases, I prefer not sticking to some slow as hell syntactical fetish, but rather go on to make sure those tests run 10x faster.
@Pratik/@Jon: If your primary reason to use factories is to create your records directly in the test, where you can easily see how the record got created, then yes, preloading a bunch of records defeats the purpose. If, however, your primary reason to use factories is to create your records through your models (with all the validations and callbacks), then preloading the records makes sense. The main reason I dislike fixtures is because it dumps the data directly in the database, without going through my model, which means I wind up with invalid data, false positives in tests, etc. Preloading the records through the model, and using the factories to help with that has helped tremendously with this. I would love it if I could create all my records through the factories directly in the tests, but preloading the records has been a big help with test speed, and speed is a more important concern for me than creating the records in the tests.
Pratik—I’ve had some other more pressing things to work on yesterday and today, but I hope to be able to get back to you later about where I was seeing the test failures.
Would be nice to mention which shoulda version these changes are based on. We were on ancient 1.1.1, which doesn’t work with this. Happily upgraded to thoughtbot-shoulda, version 2.10.2 and now it works great.
Also, not sure how others are doing this, but we have a common test_helper.rb that is referenced by both our shoulda and rspec tests. The thoughtbot-shoulda gem requires different things depending on if the test suite is Spec or not. So for rspec, it fails without explicitly requiring ‘shoulda/context’. Might be nice to have that requirement explicitly in ‘fast_context.rb’.
Thanks for solving one of our major annoyances with shoulda.
Much thanks to Myron Marston for factory_data_preloader. I think it’s a very underrated testing gem. In my case I use it to load “default” records. I use these default records when I don’t need to care about the internals of a record in a test. For example, if you have a user factory, and you end up calling Factory(:user) often, this is a good candidate for preloading the user factory.
factory_data_preloader is good because it just works, and you can use FactoryGirl alongside it.
I have identified a problem with fast_context.
I wrote a test: http://gist.github.com/192016
In case of should_change (http://rdoc.info/rdoc/thoughtbot/shoulda/blob/2e855629f2b04c63dc3cfd2383e581e52a6cd046/Shoulda/Macros.html#should_change-instance_method) the before value is stored in a attribute with a fixed name. When running all the before blocks in one go – as with fast_context – that value will only hold the last value of the last should_change before call.
Tis should be – fairly – easily fixed for should_change, however we might need to find a better way to store the values from before blocks.
This is cool, but having the same concern as @jon, I’d rather see something closer to rspec’s before(:all) setup blocks.
RSpec’s before(:all) would have the same set of problems. If any of the tests mutates the data populated in the before(:all) block, you lose the benefits of running tests in ‘isolation’.
I’ve monkey-patched Shoulda to fix should_change/should_not_change (using an expression-specific instance variable instead of the common ’@_before_should_change’). Code is here: http://gist.github.com/192203
I guess I’ll make it a plugin, so that it’s easier to use.
Pratik: true, before(:all) would have the same problems, but it’s explicit. Its intention is clear. Wrapping your existing tests with a fast_context block isn’t as clear, since the intention seems to be simply to make the tests faster.
I am not sure fast_context plugin is taking more time than normal context
Any Ideas?
I have changes all contexts in functional test including inner and outer contexts.
-thanks Tiny