As a fast-growing company, we have an increasing number of APIs consumed by our mobile apps, internal batch processes, and business partners. Manually testing these APIs each time has become almost impossible given our weekly release cycle. My first project after joining Houzz was to build our API test automation framework. With hundreds of test cases included, this API automation is now a critical part of our release validation. Not only does it save a lot of manual testing effort, it has also caught many critical bugs and has prevented production issues. This system has played an essential role in scaling product development with our growing team.
Where to start? With a large number of API endpoints, it could be difficult to figure out where to start at the beginning. After some discussion, we decided that it could be cost-effective to build a smoke test for the most frequently called API endpoints. I compiled a list from our production server logs, and built a smoke test to just make these API calls in batch and do some basic validations, such as HTTP status code and schema validation. Although it only took minimal effort to build, it turned out to be effective in catching a meaningful number of critical bugs before they reached production.
Next step is to build a more comprehensive regression test suite. Quite similar to end-to-end UI tests, API tests should also be based on user scenarios. The only difference is that API tests are simulating user scenarios at a lower level without UI interaction. To find out what APIs our mobile apps are calling, we have an internal tool to capture HTTP requests between our mobile app and API server, including HTTP method, url, query parameters, headers, request body, etc. As I go over each user scenario, the API call flows are being recorded. What’s left is to convert these recordings to API tests by extracting test data, calling APIs and adding validations. Please see the chart below.
Use test tool or write code? There are two popular approaches to automate API tests: one is through UI based API testing tools, such as SoapUI; the other is to write our own automation code with some open source libraries. I decided to go with the latter approach because of the following benefits:
– more reliable
– independent of OS
– better code reusability
– easier to debug
– easier to customize
– easier to integrate with internal tools and other projects
I chose Java to build the API automation framework, and use TestNG for test execution and reporting.
How to automate API calls? To quickly generate some automated API test flows out of the recorded request data, I wrote an utility to extract request data into JSON format. For example, an API call to get photo by id (GET http://www.example.com/photo?id=1001) could be parsed as the following JSON data:
{
"method": "GET",
"baseUrl": "www.example.com",
"path": "photo",
"queryParams": {
"id": "1001",
}
}JSON data is later deserialized to RequestData Java objects, which could be used to build API requests. Definition of the RequestData Java class looks like this:
public class RequestData {
private HttpMethod _method;
private String _baseUrl;
private String _path;
private Map _queryParams;
private Map _postData;
private Map _headers;
…
}Often times, the same API endpoint will be called multiple times with different variations. JSON data generated from the first request of the same endpoint will be used as a baseline. We don’t need to record the full JSON for subsequent requests of the same endpoint. Instead it will be parsed as a JSON diff. Using the same example of “get photo” API again, let’s say we need to make the same API call with a different photo id “1002”, it will be parsed as:
[{"op":"replace","path":"/queryParams/id","value":"1002"}]The json-patch library (https://github.com/fge/json-patch) is very useful in this case to generate JSON diff and to reconstruct original JSON data.
Compared to a UI test that requires a browser, for API tests we need a HTTP client to handle request and response. I’m using Apache HTTP component for that purpose. In order to parse and validate API responses, I use JsonPath and XmlPath from REST-assured, which provides an easy way to access field values in API responses to verify its existence, data type, and expected value.
What to validate? Now that we have automated making API calls and parsing the response, we also need to perform some validations. Often times different API endpoints or test scenarios require different types of validations, but there are some common ones that almost always apply:
– Status code, schema validation
– Verify API response is what the client is expecting. Still using the example of “get photo” API, a client calling this API may be expecting a string field called “url” in the API response that points to the photo’s URL. Assuming a valid photo id is provided, the API test should check that “url” field exists, its data type should be a string, and its value is a valid URL (if you know what is the expected URL for a given photo id, you could also do an exact match on its value).
– Verify expected effect after making an API call. This is similar to UI functionality test. One example would be after calling an API to add a new photo; if the response is successful, you want to verify the photo is actually added on server side. Usually you can call an API to get the newly created photo to verify this.
– Cover negative cases in testing. Since we cannot control how clients make API calls once we expose them, we want to make sure API behaves correctly with invalid input such as missing required fields or incorrect data types, and API response should include an error message informative enough to tell client what is wrong with the API call.
Performance In addition to functionality, we also want to test API performance. One important aspect of performance testing is to monitor latency and resource usage of a single API call. API tests record latency of each call and flag calls with latency higher than a threshold. To monitor resource usages, our backend engineering team implemented a performance logging mechanism to log performance counters for each incoming request (examples include memory usage, and number of sql or solr queries). I integrated API tests with these performance counters to find out if there is any significant increase caused by code change. Basically an API test would make the same API call to two different builds – one is the current development build, the other is from the latest release – and will then retrieve the performance counters for each build and make a comparison. This has helped us detect several performance regressions such as increased memory usage or redundant sql/solr queries.
Backward compatibility We certainly don’t want to break a previously released mobile app when we roll out changes on the server side. Our users should be able to use the Houzz app regardless of whether they upgrade to the most recent app version or not. Therefore, we need to maintain API tests for some previously released versions. Since both API request and response could be different from version to version, there will be some additional maintenance effort.
To run API tests in different versions, test data is separated into resource files categorized by version if necessary. Based on system property, our test framework will simulate requests with different versions, and pick up different test data resource file accordingly. To determine which versions to maintain, I often need to check production logs to find out which versions are mostly being used by our users. We run API tests more frequently on current version and most used versions, and only check older versions on release day.
Internationalization With users from all over the world, we need to test our API endpoints in different countries. In addition to localized language in API response, functionality could also be slightly different in each country. For example, searching “red sofa” related photos could result in different content for different countries. Similar to how we handle versions, we also extract locale specific test data to files grouped by country code. Our test framework is able to pick up correct data file based on locale.
Integration with UI automation One additional benefit we realized after building the API automation is that, we can leverage them in end-to-end UI automation to speed up test data setup. For example, a significant percentage of our UI tests require a newly created user. Instead of signing up on browser, we now switched to creating users via API. This has several benefits:
– significantly reduced UI test execution time as API calls are usually a lot faster than UI flow.
– improved overall stability of the UI automation. One of the reasons is that the UI changes are usually more frequent. For example, if we are iterating different UI designs for sign up flows, all the UI tests that require user sign-up through UI could fail. This is no longer a concern after we switch to creating user via API. One caveat is that you should keep at least one test to cover the UI flow replaced by API elsewhere.
Summary It has been a very exciting experience to own a project from the beginning and build it from scratch. Of course I also received a lot of help from my amazing coworkers along the way, and it’s always fun to collaborate across the team. As Houzz grows, we will face new technical challenges that provide more exciting opportunities in the test automation area. If you’re interested in joining our growing team, please check out our jobs page at https://www.houzz.com/jobs