It is my understanding that running rake qunit:test should run all of the qunit tests in Discourse, including those for any installed plugins. However, when I run the task in the docker development environment it only runs the core discourse tests (1980 of them).

When I run exactly the same command using my native mac development environment, it picks up the tests for the plugins (e.g. the 'poll' plugin), which results in more than 1980 tests, and I have no idea what's causing this difference.

This exact same problem seems to occur in the Discourse Travis build - looking at this recent log, it shows

Time: 151618ms, Total: 1980, Passed: 1980, Failed: 0

So it's not running the qunit tests for all the core plugins, even though it's running their rspec tests.

I've tried investigating further but don't really know where to look. I'm guessing there must be some kind of regex/filter which locates all the qunit test files?

RSpec tests are working fine, it's the QUnit ones that aren't working. I've tried putting LOAD_PLUGINS=1 before the rake command, but no luck

I don't think it's a typo, since it's an issue on Discourse's own travis builds.

It sounds like something that we might have disabled due to random test failures which are unfortunately more common on travis than I'd like. If we can figure out how to run them again I'd be happy to reintroduce them!

That makes sense, but I can't see any logic that disables/enables them... running rake qunit:test on my mac runs the plugin tests, running it in docker doesn't...

I’ve been doing a lot of experimenting with QUnit today, and have had another look at this issue.

discourse_dev

The issue was indeed with the LOAD_PLUGINS environment variable. When using the bin/docker/rake script, the LOAD_PLUGINS variable doesn’t get passed through to the container, hence why I couldn’t get it working with the discourse_dev image.

discourse_test

I also experimented with the discourse_test image. It is my understanding that the discourse_test image is used to test master before merging into tests-passed (according to this). In its default configuration:

docker run -it --rm discourse/discourse_test

it does not seem to run any plugin tests at all (either rspec or qunit). I am assuming that’s how broken core plugin tests managed to end up in tests-passed. I managed to get it to run both qunit and rspec plugin tests by running

docker run -e "LOAD_PLUGINS=1" -it --rm discourse/discourse_test

Maybe that’s something that could be used internally to make sure core plugins get tested properly? Or alternatively the LOAD_PLUGINS variable could be set in the Dockerfile?

Travis

Travis is currently running plugin rspec tests, but not qunit tests. I made a PR for turning on plugin qunit tests here:

Running that currently results in 11 qunit failures from a number of plugins: (Cakeday, Canned Replies, Details, and Spoiler)

Module Failed: Acceptance: Cakeday
  Test Failed: Anniversary emoji
    Assertion Failed: Error: Unhandled request in test environment: /t/11.json?track_visit=true&forceLoad=true&_=1499806929081 (GET)
      Expected: true, Actual: false    
    Assertion Failed: TypeError: undefined is not an object (evaluating '$posterIcons[0].title')
      Expected: true, Actual: false
  Test Failed: User is not logged in
    Assertion Failed:
  phantomjs://webpage.evaluate():49
Module Failed: Acceptance: Canned Replies
  Test Failed: Inserting canned replies
    Assertion Failed: Error: Unhandled request in test environment: /canned_replies?_=1499806929085 (GET)
      Expected: true, Actual: false    
    Assertion Failed: ReferenceError: Can't find variable: equal
      Expected: true, Actual: false
  Test Failed: Editing a canned reply
    Assertion Failed: Error: Unhandled request in test environment: /canned_replies?_=1499806929088 (GET)
      Expected: true, Actual: false    
    Assertion Failed: Error: Element .reply-selector #canned-replies-combobox not found.
      Expected: true, Actual: false
  Test Failed: Creating a new canned reply
    Assertion Failed: Error: Unhandled request in test environment: /canned_replies?_=1499806929091 (GET)
      Expected: true, Actual: false    
    Assertion Failed: ReferenceError: Can't find variable: equal
      Expected: true, Actual: false
  phantomjs://webpage.evaluate():49
Module Failed: Acceptance: Details Button
  Test Failed: details button
    Assertion Failed: it should contain the right output
      Expected: Before
[details=Summary]
some text in between
[/details]
After, Actual: Before 
[details=Summary]
some text in between
[/details]
 After
  phantomjs://webpage.evaluate():49
Module Failed: Acceptance: Spoiler Button
  Test Failed: spoiler button
    Assertion Failed: ReferenceError: Can't find variable: ok
      Expected: true, Actual: false

These are legitimate fails, which I can reproduce running the tests suite locally. I’ve already fixed all but one of the “discourse-details” tests, and the “Spoiler” test should be an easy fix. The others may need more investigation.

Edit: discourse-details PR here and spoiler alert PR here

@sam (I think discourse_test is your creation?), what do you think about loading plugins for the tests-passed pipeline?

The plugin qunit tests are now all fixed, but today many core plugin rspec tests were broken by the removal of the old markdown engine, yet the commit still ended up in tests-passed. (fails are in poll, discourse-details and discourse-narrative-bot).

I will have a look at this today… honestly I do not think the community can help out too much here, we have to plug this stuff in to our CI pipeline (we don’t really trust travis cause it is flaky and slow)… having a look at it later today.