top of page

Houzz API Tests at a Glance

As a fast-growing company, we have an increasing number of APIs consumed by our mobile apps, internal batch processes, and business partners. Manually testing these APIs each time has become almost impossible given our weekly release cycle. My first project after joining Houzz was to build our API test automation framework. With hundreds of test cases included, this API automation is now a critical part of our release validation. Not only does it save a lot of manual testing effort, it has also caught many critical bugs and has prevented production issues. This system has played an essential role in scaling product development with our growing team.

Where to start? With a large number of API endpoints, it could be difficult to figure out where to start at the beginning. After some discussion, we decided that it could be cost-effective to build a smoke test for the most frequently called API endpoints. I compiled a list from our production server logs, and built a smoke test to just make these API calls in batch and do some basic validations, such as HTTP status code and schema validation. Although it only took minimal effort to build, it turned out to be effective in catching a meaningful number of critical bugs before they reached production.

Next step is to build a more comprehensive regression test suite. Quite similar to end-to-end UI tests, API tests should also be based on user scenarios. The only difference is that API tests are simulating user scenarios at a lower level without UI interaction. To find out what APIs our mobile apps are calling, we have an internal tool to capture HTTP requests between our mobile app and API server, including HTTP method, url, query parameters, headers, request body, etc. As I go over each user scenario, the API call flows are being recorded. What’s left is to convert these recordings to API tests by extracting test data, calling APIs and adding validations. Please see the chart below.

image

Use test tool or write code? There are two popular approaches to automate API tests: one is through UI based API testing tools, such as SoapUI; the other is to write our own automation code with some open source libraries. I decided to go with the latter approach because of the following benefits:

– more reliable

– independent of OS

– better code reusability

– easier to debug

– easier to customize

– easier to integrate with internal tools and other projects

I chose Java to build the API automation framework, and use TestNG for test execution and reporting.

How to automate API calls? To quickly generate some automated API test flows out of the recorded request data, I wrote an utility to extract request data into JSON format. For example, an API call to get photo by id (GET http://www.example.com/photo?id=1001) could be parsed as the following JSON data:


{
    "method": "GET",
    "baseUrl": "www.example.com",
    "path": "photo",
    "queryParams": {
        "id": "1001",
    }
}

JSON data is later deserialized to RequestData Java objects, which could be used to build API requests. Definition of the RequestData Java class looks like this:

public class RequestData {
    private HttpMethod _method;
    private String _baseUrl;
    private String _path;
    private Map _queryParams;
    private Map _postData;
    private Map _headers;}

Often times, the same API endpoint will be called multiple times with different variations. JSON data generated from the first request of the same endpoint will be used as a baseline. We don’t need to record the full JSON for subsequent requests of the same endpoint. Instead it will be parsed as a JSON diff. Using the same example of “get photo” API again, let’s say we need to make the same API call with a different photo id “1002”, it will be parsed as:

[{"op":"replace","path":"/queryParams/id"