Grails

Provide set of build events for test running

Details

  • Type: Improvement Improvement
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: None
  • Fix Version/s: 2.3-M2
  • Component/s: None
  • Labels:
    None

Description

This issue relates to https://issuetracker.springsource.com/browse/STS-551

In order to implement proper integration with STS Eclipse and providing GUI feedback about test runs (run via various forms of the "grails test-app" command, the event model needs to be extended.

Rationale: the Eclipse JUnit view works roughly as follows:

Phase 1: populate the view showing which tests will be run

Phase 2: while the tests, are running provide visual feedback

  • progress bar
  • view is update as each individual test starts completes/fails

The GrailsBuildListener infrastructure which publishes test events comes close to filling the needs to be able
to hook Grails up to the viewer. In particular, it provides enough information to implement what is required for
Phase 2.

However, it provides no information for Phase 1, to populate the view before the tests are run.

To meet the requirements of the UI it would be nice if the following changes were made to the "_GrailsTest.groovy" script.

1) One extra event type should be published:
event("TestSuitePrepared", [suite, type.name])

Here 'suite' should provide a reference to an instance of the Junit4 'Runner' type (e.g. a Suite instance).
type.name is the name of the 'test phase/type' e.g. "unit", "integration", ...

Alternatively (even better!) if the suite should contain "type.name" as its displayName the second parameter is not needed
(Currently, I believe the grails code sets this name to 'null').

2) change the processing order in the script

All suites for all phases should be constructed before any of them are being executed (so that UI view can be populated with all tests before anu tests are executed).

To be precise. Currently processing order is roughly as follows:

prepare unit tests suite
execute unit tests suite
prepare integration tests suite
execute integration tests suite
...

It should be as follows:
prepare unit tests suite
=> event("TestSuitePrepared", [suite, "unit"])
prepare integration tests suite
=> event("TestSuitePrepared", [suite, "integration"])
execute unit tests suite
execute integration tests suite

I have already implemented a modified version of the script and something that uses the extra events to drive the JUnit Eclipse UI, it works...
So I know that given something on the grails side that provides these extra events is sufficient to implement the integration with Eclipse.

However, I have little trust in the correctness of my hacked-up version of _GrailsTest.groovy. In particular, to be able to create the suites,
without making big changes to how they are represented, I had to change the order in which the gant 'prepare' 'run' 'cleanup' targets are called, and I fear that this means application contexts wouldn't be setup properly if there are more than one test-phase that sets up an application context.

I'm attaching my version here just to give an idea of what it is that I want. I've tried to include as much comments and explanation in the code to explain what I was trying to do as possible.

I'm available for further discussion and will be watching this issue.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Kris De Volder added a comment -

I thought a bit more about this. Implementing the requirements I have outline in this issue may be a bit tricky because of the reordering of prepare-execute-cleanup. As a compromise, it would be much easier to just add the extra "TestSuitePrepared" event. The changes to grails to accomplish that would be minimal. But it would be sufficient to get integration tests (by themselves) running nicely with the UI viewer. What it it will not do is allow running versions of the test-app command that execute tests from multiple test phases with the UI viewer.

So, if getting both of my requirements above implemented seems like too tall an order, just implementing requirement one would already be useful.

Show
Kris De Volder added a comment - I thought a bit more about this. Implementing the requirements I have outline in this issue may be a bit tricky because of the reordering of prepare-execute-cleanup. As a compromise, it would be much easier to just add the extra "TestSuitePrepared" event. The changes to grails to accomplish that would be minimal. But it would be sufficient to get integration tests (by themselves) running nicely with the UI viewer. What it it will not do is allow running versions of the test-app command that execute tests from multiple test phases with the UI viewer. So, if getting both of my requirements above implemented seems like too tall an order, just implementing requirement one would already be useful.
Hide
Permalink
Luke Daley added a comment -

Hi Kris,

I'll get something in soon. One complication is that the test system is not exclusively for JUnit, it's intentionally very abstract to support running other test frameworks such as easyb, fitnesse etc. We can try and optimise for JUnit as it's by far the most common but we can't do anything that inhibits other frameworks.

Show
Luke Daley added a comment - Hi Kris, I'll get something in soon. One complication is that the test system is not exclusively for JUnit, it's intentionally very abstract to support running other test frameworks such as easyb, fitnesse etc. We can try and optimise for JUnit as it's by far the most common but we can't do anything that inhibits other frameworks.
Hide
Permalink
Graeme Rocher added a comment -

Luke will you be able to look at this by Wednesday (1.3.5 target date)

Show
Graeme Rocher added a comment - Luke will you be able to look at this by Wednesday (1.3.5 target date)
Hide
Permalink
Kris De Volder added a comment -

Hi Luke,

It would be most convenient on my end if the events that are needed to pre populate the view come with a reference to a JUnit test suite. But this is not absolutely necessary. What is necessary is that we get access to information about the tests that are going to run in some form or other.

If it is a JUnit suite I have to write a lot less code. This is because there's some infrastructure in place that accepts a suite and encodes this into something it can send back to Eclipse over a socket. The Eclipse JUnit view uses a kind of 'client-server' architecture to receive information.

If the info came in some other form, it is a little bit more work but not infeasible to implement something that sends the info back.

So if you think it would be better to keep build/test-events more abstract and test framework agnostic I think that is fine. (And perhaps this is more logical and inline with the other test events, since these also do not send references to JUnit framework specific entities).

The main requirements are

  • we need info on all tests that will run before they start (i.e. a set of 'test-trees')
  • the info about the actual test runs (start/fail/end) has to be mappable onto the nodes of the test-trees.

If both of these requirements are met, I believe I can make it work.

Kris

Show
Kris De Volder added a comment - Hi Luke, It would be most convenient on my end if the events that are needed to pre populate the view come with a reference to a JUnit test suite. But this is not absolutely necessary. What is necessary is that we get access to information about the tests that are going to run in some form or other. If it is a JUnit suite I have to write a lot less code. This is because there's some infrastructure in place that accepts a suite and encodes this into something it can send back to Eclipse over a socket. The Eclipse JUnit view uses a kind of 'client-server' architecture to receive information. If the info came in some other form, it is a little bit more work but not infeasible to implement something that sends the info back. So if you think it would be better to keep build/test-events more abstract and test framework agnostic I think that is fine. (And perhaps this is more logical and inline with the other test events, since these also do not send references to JUnit framework specific entities). The main requirements are we need info on all tests that will run before they start (i.e. a set of 'test-trees') the info about the actual test runs (start/fail/end) has to be mappable onto the nodes of the test-trees. If both of these requirements are met, I believe I can make it work. Kris
Hide
Permalink
Kris De Volder added a comment -

Some further thoughts.

Keeping the event model abstract would be best.

Presumably, if I would then be running Grails tests with other frameworks, and assuming the same events are being published, and assuming I implement the Eclipse JUnit viewer 'test listener' based on those events, then the Eclipse JUnit view would also simply work for these other test frameworks. That would be very nice.

Show
Kris De Volder added a comment - Some further thoughts. Keeping the event model abstract would be best. Presumably, if I would then be running Grails tests with other frameworks, and assuming the same events are being published, and assuming I implement the Eclipse JUnit viewer 'test listener' based on those events, then the Eclipse JUnit view would also simply work for these other test frameworks. That would be very nice.
Hide
Permalink
Graeme Rocher added a comment -

Hi Kris we're not going to be able to get this into 1.3.5 as we have support customers waiting on that release and this is too big a change at this stage. We'll make sure it comes into 1.3.6 though.

Show
Graeme Rocher added a comment - Hi Kris we're not going to be able to get this into 1.3.5 as we have support customers waiting on that release and this is too big a change at this stage. We'll make sure it comes into 1.3.6 though.
Hide
Permalink
Kris De Volder added a comment -

Sure. I understand this change is probably quite involved.

I don't think there is extreme urgency in this. I already implemented something to run test-app commands that, while not as tightly integrated with the IDE, will probably suffice until we can do a better job with the extra events.

Show
Kris De Volder added a comment - Sure. I understand this change is probably quite involved. I don't think there is extreme urgency in this. I already implemented something to run test-app commands that, while not as tightly integrated with the IDE, will probably suffice until we can do a better job with the extra events.
Hide
Permalink
Luke Daley added a comment -

Moving to 1.4 as it is too big to go in the 1.3 line now.

Show
Luke Daley added a comment - Moving to 1.4 as it is too big to go in the 1.3 line now.
Hide
Permalink
Krystian Szczesny added a comment -

Hi,

I'm wondering how is this thing going?
Due to costs and due to STS being improved more and more recently I am considering switch from Idea.
However this little quirks lead to time consuming navigation through classes and test results.
Say what you want about Idea but going through test results is very fast.

Show
Krystian Szczesny added a comment - Hi, I'm wondering how is this thing going? Due to costs and due to STS being improved more and more recently I am considering switch from Idea. However this little quirks lead to time consuming navigation through classes and test results. Say what you want about Idea but going through test results is very fast.
Hide
Permalink
Kris De Volder added a comment -

Currently, in STS you can run tests from within the IDE by clicking on specific resources (e.g. a test class, source folder, package). STS will create an appropriate grails command to run the tests based on what you selected.

After the test run is complete the results will be opened in the standard Eclipse JUnit test viewer. You'll be able to see the results as well as click on the stacktraces of failed tests.

What you won't be able to do yet is see the green progress bar while the tests are running or re-run specific tests from the test result viewer.

So far we (STS-Grails) haven't really had much complaints or requests to expand on this basic support.

This issue (i.e. Grails-6755) is a request to expand Grails BuildListener infrastructure to enable us to show that progress bar while running. Certainly this would be nice, but it will also require a fairly large amount of work and restructuring in Grails. It isn't really clear whether getting this progress bar is really that important.

Anyhow, if you think it is you can vote for this issue. Or, if you have specific suggestions on what you would like from the STS Grails integration for test running, consider raising a Jira issue on the STS bug tracker.

https://issuetracker.springsource.com/browse/STS

Show
Kris De Volder added a comment - Currently, in STS you can run tests from within the IDE by clicking on specific resources (e.g. a test class, source folder, package). STS will create an appropriate grails command to run the tests based on what you selected. After the test run is complete the results will be opened in the standard Eclipse JUnit test viewer. You'll be able to see the results as well as click on the stacktraces of failed tests. What you won't be able to do yet is see the green progress bar while the tests are running or re-run specific tests from the test result viewer. So far we (STS-Grails) haven't really had much complaints or requests to expand on this basic support. This issue (i.e. Grails-6755) is a request to expand Grails BuildListener infrastructure to enable us to show that progress bar while running. Certainly this would be nice, but it will also require a fairly large amount of work and restructuring in Grails. It isn't really clear whether getting this progress bar is really that important. Anyhow, if you think it is you can vote for this issue. Or, if you have specific suggestions on what you would like from the STS Grails integration for test running, consider raising a Jira issue on the STS bug tracker. https://issuetracker.springsource.com/browse/STS

People

  • Assignee:
    Unassigned
    Reporter:
    Kris De Volder
Vote (4)
Watch (3)

Dates

  • Created:
    Updated:
    Last Reviewed: