My rough notes on talk & workshop conducted by Christopher Rex. Code most by Sahil. QA ThoughtWorks 7 years.

Met Narain*, consultant for banks. He said the design and implementation of software is most important followed by knowledge of languages.

Met Raju*, ThoughtWorks. General conversation about school and ThoughtWorks. They have about 80 people here. < 20% operations.

Met Ika*, ThoughtWorks testing. Brought me to office.

github

Office 97 Browser <-> Monolithic App <-> Db

API Gateway -> Still can get away with testing UI

Problem Function Testing

Alexa. Voice enabled device.

Apps put into Alex domain. How to test?

Basically, you can’t stick to one layer to test it. Just speaking to it doesn’t work. Too many variables/inconsistencies.

Goal: come up with sensible test approach for IoT devices.

Evolution of Automated testing

1. Manual testing
2. Automated testing
3. Continuous integration server: Build, test, result (feedback), deploy (maybes)

Problem Automation

Always Red, Painfully slow build

Rohit: Many problems… Sometimes have to wait for GUI. System upgrade then build fails.

Developer has to wait for the feedback for the build. He can’t go away… He would have already moved on to the new build.

We would like to have a more efficient automation suite.

Common approach

Using test pyramid. Using multiple layers. Go about testing in logic in each layer.

Stub testing is one layer in the test pyramid.

If I have a list of requirements how do I map them to tests in each layer? Christopher and Saye* came up with a github project with 4 layers of testing.

Goals today:

1. Building deeper understanding of AUT (layered approach to testing)
2. Testing for right logic in right layer (sensible functional testing, efficient automation testing)

Project

Simple application. sample-test-pyramid. Login, coupon page.

Tests

Login Functionality

Think of acceptance criterions. Expected functionality that is required to be built.

1. Should be able to login if account is verified. …

In extended case, super user going to the right page is a valid test scenario.

How to test?* You need data first (mock). After a long time, you get a lot of scenarios. To manage this you need to come up with a good testing approach.

Testing approach 1. How can we approach testing this user-auth feature? 2. Do we have enough information to to define…

First, thing to know… You need to know application architecture.

In this case, we have User interaction layer, Service layer

(End to end layer)

User interaction layer: - Web application API wire-up logic Presentation logic

Service Layer: - User Service - Coupon Service Domain Logic

process? lsof -i:4567 Check if jetty is still running on port 4567?

./up convenience script to bring up server. For java it uses gradle. For js it uses node.

Using postman without going through UI tests run much faster. You modularize your tests. Only when there is a change in the layer then you perform the tests for that layer or service. Don’t need to repeat tests unless you make a change. Advantage of making a test in seclusion.

Test UI layer without Service layer? Stub the response (for both stubbed user service and stubbed coupon service).

Tool used Stubby4j. Define YAML config. Says what to respond for a request (Seems like you need to record it).

There could be validation in presentation layer (hence, presentation logic).

Thick client, then there may not be a lot of communication going to the backend.

Do you want to be held up if the service layer is in the process of upgrading and not working? No. Stubs overcome this. You can continue developing and testing. The developers already know the contract the backend team is supposed to fulfill. Gives team autonomy to build their layer.

Choices: - E2E - Stubbed (Comprehensive layer that includes UI logic. Anything that doesn’t hit the server.) - Service

One convention we can use is the one used in cucumber.

The flow: Login -> Coupon page. Without stub you would not be able to go from login to coupon page.

Reason for “Verify Successful Login” as E2E because you need to communicate with server and you also need to show unauthorized on the UI.

…

Applying Layered Testing Approach

Don’t need to test Alex Service, Kitchen Light (Philips Hue) because they are well tested by their teams.

We just need to test the boundary. Then we need 1 more E2E to make sure our software works well in real life. With proper understanding of architecture of application you can simplify your tests. You need a deeper understanding of the application.

<< Ends functional testing portion >>

Now we move into automation. Before, going there… Zooming into user service there is a unit layer. (Router > Handlers (Ping, Login) > Repository). Unit tests uses mocks (e.g. mockRepository).

SparkServer?

Within integration tests, SparkServer is constructed as a functional component not the actual Db.

User journey better expressed in BDD so we use cucumber. Can run tests for each layer because they are tagged (e2e, stub).

Test Automation - Layered

Stubbed Webapp Tests most number of tests. Shows you it is a thick client. Lots of tests on UI layer.

CI Build Monitor

Unit tests failing. But rest green? Error has not propagated up.

Integration test failing. Rest green. Business logic in service layer is failing. Could be due to lift and shift of logic passing unit tests but haven’t actually updated unit tests. However, QA for integration layer could have updated it already.

E2E or stubbed test is failing. E.g. route has changed or response schema has changed. Schema upgrade. UI logic or presentation logic has gone for a test.

Just by looking at CI you know what the problem is at each time.

Interesting scenario for coupon

Verify coupon page is not accessible without login. (go to url directly)

Also, think about concurrent coupon validation.

QnA

(Q’s from narain)

Q: More expensive to have acceptance at UI level, cheaper at the unit level. If I have a thick client application, in terms of automation, and you’re refactoring the entire thing to get the right density of tests for each layer.

A: There might be tool to migrate tests to another layer. We use JUnit, unit and integration. Cucumber can also do unit and integration. If business level ppl then cucumber makes sense.

RephraseQ: Any other automation frameworks (mental frameworks)? What approach should you take?

A: It depends on how long lasting the code base will be. New? Many years in operation? If delivery is going to be real soon how much sense does it make to migrate the tests to lower layers… Maybe not much. If longer roadmap, probably good to invest in the automation framework.

Q: Application is more UI based but developers are not. This framework is unit up…

A: Built a suite, a framework, of 3000 tests. These tests were all on the UI layer. It was a big deal. When app grew it became expensive to manage. Industry has grown… determined its not wise to have too many tests in the UI layer. (Raju)

A: Root cause analysis for defects. Then categorized which defects can be defended in which layer. Then risk analysis to determine which defects to write tests for.

Q: Who should be writing these unit tests and integration? Large organizations don’t have agile. Whats the costs benefit analysis?

A: Yes its difficult to justify the cost of changing everything. Start building the right way. Old apps refactor to use the new way as you touch them. Progressive change. Present cost benefit to organization then let them decide.

A: Treat unit tests as black box. Focus more on integration test. If integration test failure then he goes to unit test to see if its covered in unit tests. Ask himself why not covered. Write it himself or ask a dev to do it.

(Interesting graphic: Ice-cream cone anti-pattern. watirmelon.blog) Christopher: Beware of recording too many request, responses. Try to prevent the anti-pattern from happening.

(Mention of slow tool. QTP. Falls into an anti-pattern. Write more tests to the extent that you dont know the architecture of your application. RFT. Migration tools to migrate QTP to RFT. All failing because they are building this ice-cream cone.)

Q (me): How would you test streaming data like stock information, audio or video?

A: Good question. Haven’t had experience there but theoretically you still can stub it. A (Ika): In my experience you can stub it.

A (Christopher): Has had a pre-prod scenario to test a few E2E scenarios.