The underlying cause is pretty simple and it affects projects of all sizes (yep, including yours): users are becoming more demanding, less patient and not tolerant towards slow websites or applications. If your content takes too long to load, people will go somewhere else. Visiting a site that takes ages to open and navigate is a terrible user experience, especially on the dominant mobile environment where immediacy is crucial and battery life is precious.<\/p>\n
For that reason, website performance optimisation plays an increasingly important role in the success of any online property. All major browsers ship with tools that allow developers to keep an eye on some important performance metrics as the build progresses, but these are measured from the developer\u2019s own standpoint, which is not enough to see the full picture.<\/p>\n
Factors like geographic location, connection type, device, browser vendor or operating system can heavily influence perceived load times, so testing all these variables is the only way to get a (mildly) accurate representation of how a website is experienced by a broader audience.<\/p>\n
There are various tools and services<\/a> to approach that problem, but this article will focus specifically on WebPageTest<\/a>. We will look at it from a developer\u2019s perspective, in particular at using its RESTful API to extract vital information you can use to optimise the performance of your site.<\/p>\n <\/p>\n WebPageTest is an open source performance testing tool, maintained primarily by Google. It consists of one or multiple servers that act as web browsing robots, visiting websites and automatically collecting data about the experience in the form of a detailed performance report.<\/p>\n There is a public instance with a large pool of servers that is available for anyone to use for free, which is what we’ll be using in the examples throughout the article. Alternatively, the source code for the platform is available on GitHub<\/a>, should you want to host it privately on your own infrastructure.<\/p>\n When you open the WebPageTest website<\/a>, you’ll see the interface that allows you to run a test straight away. The first thing you need is the URL of the page to be tested. On top of that, there many parameters that can be configured, with the main ones being:<\/p>\n For this example, we’re testing https:\/\/css-tricks.com<\/strong> with the default settings: Chrome on a Cable connection from Dulles, VA, performing a single run with first and repeat view, with video capturing enabled.<\/p>\n After requesting the test, you will enter a waiting period, as the pool of devices is shared by everyone using the public instance of WebPageTest. For this reason, tests take an unpredictable amount of time to complete, depending on the number of people using the devices and the complexity of the test.<\/p>\n This link<\/a> contains a list of all available resources and their capacity at any given time.<\/p>\n The amount of information shown in the test reports can be a bit overwhelming, so it\u2019s worth having a look at some of the key metrics returned and what they mean:<\/p>\n The waterfall view is another key piece of the report. It shows a visualisation of the network activity over time, with each horizontal bar representing an HTTP request. The colours in the bars represent the five phases of a request: DNS lookup (teal), initial connection (orange), SSL negotiation (purple), time to first byte (green), and content download (blue).<\/p>\n It also shows vertical lines to mark key events in the lifecycle of the page, such as the time it takes for the browser to paint the first pixel (green), the point at which the DOM tree is ready (pink), or when the document is loaded (blue). Finally, it shows redirects (highlighted in yellow) and errors (highlighted in red).<\/p>\n We requested a video recording as part of the rest, so WebPageTest gives us a set of frames that visually show the page being drawn on the screen over time. We can use this data to generate a filmstrip view or an actual video.<\/p>\n With the basics of the platform covered, let’s dive into how we can interact with it programatically.<\/p>\n WebPageTest offers a RESTful API for public use. Because it’s a shared instance, usage is limited to 200 page loads per day \u2014 repeat views count as a separate page load, which means that a test with two runs and a repeat view would count as four page loads.<\/p>\n It\u2019s also worth mentioning that test results are only kept on the servers for 30 days, so make sure you save any data you might need for posterity (including images and videos) to your own infrastructure.<\/p>\n Anything you can do on the WebPageTest UI can also be done programmatically, as the site itself makes use of the RESTful API. You can request tests and obtain the results, which you can then feed into a variety of outlets, like a data visualisation tool, continuous integration process, trigger Slack or email alerts, or pretty much anything.<\/p>\n The code examples shown throughout the article are written in ES5 JavaScript for a Node.js environment, making use of the WebPageTest API wrapper<\/a>. But because the API is RESTful, it can be accessed using any language or environment capable of sending an HTTP request, so everything you’ll see here can be ported to your language of choice.<\/p>\n The first step is to request an API key<\/a>. After filling in your details, you should get a key straight away.<\/p>\n With that done, we can set up a new Node.js project and install the WebPageTest API wrapper.<\/p>\n The We’re going to repeat the test we did before, but this time programmatically using the API. We need the Remember that requesting a test puts you in a waiting list, so the response you’ll get from running the code above is not the actual test result, but more of a receipt that you can use to check on the progress of the test and obtain the results when they’re ready.<\/p>\n We’re particularly interested in Eventually (depending on how busy the platform is) you’ll get a response containing:<\/p>\n At this point, we know that the test results are ready and we can fetch them using This method of getting results involves some manual work on our end, as we need to keep calling The following example (using option 2) is a combination of all the steps we’ve seen, and uses one single call to I’m not going to include the full response here because it’s massive (375KB of data!), but you can see it in its entirety here<\/a>. Instead, we’ll see how we can drill down into it to find some of the metrics we described earlier.<\/p>\n The metrics shown above are just a small subset of everything WebPageTest captures, as you can see by dissecting the full result payload. But sometimes it’s important to measure other things, like metrics that are only relevant to the specific website being tested.<\/p>\n With WebPageTest, we can do this with custom metrics<\/a>, a feature that allows us to execute arbitrary JavaScript code at the end of a test.<\/p>\n For example, we could be interested in tracking the performance impact caused by the number of iframes that are being loaded, or by any ads being served from a specific provider. Let’s see how we can measure that.<\/p>\n Each metric is defined as a block of JavaScript preceded by an identifier in square brackets, separated by a line break.<\/p>\n To get the number of iframes, we’re simply finding and counting all Table of contents<\/h3>\n
\n
\n
\n
About WebPageTest<\/h3>\n
Running a test<\/h3>\n
\n
![\"\"](https://calendarstemplates.com/\"https:\/\/i0.wp.com\/css-tricks.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-07-31-at-00.21.04-300x225.png?resize=300%2C225&ssl=1\")
Reading the results<\/h3>\n
\n
![\"\"](https://calendarstemplates.com/\"https:\/\/i0.wp.com\/css-tricks.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-07-31-at-14.43.41-300x162.png?resize=300%2C162&ssl=1\")
![\"\"](https://calendarstemplates.com/\"https:\/\/i0.wp.com\/css-tricks.com\/wp-content\/uploads\/2017\/08\/WebPageTest-video.gif?ssl=1\")
WebPageTest API<\/h3>\n
Setting up<\/h3>\n
npm install webpagetest --save<\/code><\/pre>\nvar WebPageTest = require('WebPageTest')\r\nvar wpt = new WebPageTest('https:\/\/www.webpagetest.org\/', 'your-api-key')<\/code><\/pre>\nWebPageTest<\/code> constructor takes two parameters:<\/p>\n\n
Running a test programmatically<\/h3>\n
runTest<\/code> function, which takes two parameters:<\/p>\n\n
wpt.runTest('https:\/\/css-tricks.com', {\r\n connectivity: 'Cable',\r\n location: 'Dulles:Chrome',\r\n firstViewOnly: false,\r\n runs: 1,\r\n video: true\r\n}, function processTestRequest(err, result) {\r\n console.log(err || result)\r\n})<\/code><\/pre>\n{\r\n \"statusCode\": 200,\r\n \"statusText\": \"Ok\",\r\n \"data\": {\r\n \"testId\": \"160814_W7_960\",\r\n \"ownerKey\": \"ad50468e0d69d1e6d0cda22f38d7511cc4284e40\",\r\n \"jsonUrl\": \"https:\/\/www.webpagetest.org\/jsonResult.php?test=160814_W7_960\",\r\n \"xmlUrl\": \"https:\/\/www.webpagetest.org\/xmlResult\/160814_W7_960\/\",\r\n \"userUrl\": \"https:\/\/www.webpagetest.org\/result\/160814_W7_960\/\",\r\n \"summaryCSV\": \"https:\/\/www.webpagetest.org\/result\/160814_W7_960\/page_data.csv\",\r\n \"detailCSV\": \"https:\/\/www.webpagetest.org\/result\/160814_W7_960\/requests.csv\"\r\n }\r\n}<\/code><\/pre>\ndata.testId<\/code>, as it contains a string that uniquely identifies our test. We can pass it to the getTestStatus<\/code> method to check if the test is ready.<\/p>\nwpt.getTestStatus('160814_W7_960', function processTestStatus(err, result) {\r\n console.log(err || result)\r\n})<\/code><\/pre>\n{\r\n \"statusCode\": 200,\r\n \"statusText\": \"Test Complete\"\r\n}<\/code><\/pre>\ngetTestResults<\/code>.<\/p>\nwpt.getTestResults('160814_W7_960', function processTestResult(err, result) {\r\n console.log(err || result)\r\n})<\/code><\/pre>\ngetTestStatus<\/code> until we get a 200<\/code> response, and only then call getTestResults<\/code>. There are two alternative (and more convenient) ways of doing this:<\/p>\n\n
pingback<\/code> option to runTest<\/code>, containing a URL to be called by WebPageTest once the test is complete. This could be a route in your web server built specifically to handle test results. The test ID will be passed as an id<\/code> query parameter, which you could use to call getTestResults<\/code>.<\/li>\nrunTest<\/code> by providing a pollResults<\/code> option. Its value, in seconds, represents an interval used to poll the API for the status of the test. The execution will only finish once the test results are returned.<\/li>\n<\/ol>\nrunTest<\/code> to request a test, poll the API every 5 seconds until the result is ready, and finally output the results.<\/p>\nwpt.runTest('https:\/\/css-tricks.com', {\r\n connectivity: 'Cable',\r\n location: 'Dulles:Chrome',\r\n firstViewOnly: false,\r\n runs: 1,\r\n pollResults: 5,\r\n video: true\r\n}, function processTestResult(err, result) {\r\n \/\/ First view \u2014 use `repeatView` for repeat view\r\n console.log('Load time:', result.data.average.firstView.loadTime)\r\n console.log('First byte:', result.data.average.firstView.TTFB)\r\n console.log('Start render:', result.data.average.firstView.render)\r\n console.log('Speed Index:', result.data.average.firstView.SpeedIndex)\r\n console.log('DOM elements:', result.data.average.firstView.domElements)\r\n\r\n console.log('(Doc complete) Requests:', result.data.average.firstView.requestsDoc)\r\n console.log('(Doc complete) Bytes in:', result.data.average.firstView.bytesInDoc)\r\n\r\n console.log('(Fully loaded) Time:', result.data.average.firstView.fullyLoaded)\r\n console.log('(Fully loaded) Requests:', result.data.average.firstView.requestsFull)\r\n console.log('(Fully loaded) Bytes in:', result.data.average.firstView.bytesIn)\r\n\r\n console.log('Waterfall view:', result.data.runs[1].firstView.images.waterfall)\r\n})<\/code><\/pre>\nCustom metrics<\/h3>\n
var customMetrics = [\r\n '[iframes]',\r\n 'return document.getElementsByTagName(\"iframe\").length',\r\n '[ads]',\r\n 'return Array.prototype.slice.call(document.getElementsByTagName(\"a\")).filter(function (node) { return node.getAttribute(\"href\").indexOf(\"ad.doubleclick.net\") !== -1 }).length'\r\n]\r\n\r\nwpt.runTest('https:\/\/css-tricks.com', {\r\n custom: customMetrics.join('\\n'),\r\n connectivity: 'Cable',\r\n location: 'Dulles:Chrome',\r\n firstViewOnly: false,\r\n runs: 1,\r\n pollResults: 5\r\n}, function processTestResult(err, result) {\r\n console.log('Iframes:', result.data.average.firstView.iframes)\r\n console.log('Ads:', result.data.average.firstView.ads)\r\n})<\/code><\/pre>\n<iframe><\/code> elements in the DOM. As for the ads, we’re looking at all <a><\/code> nodes that contain ads.doubleclick.net<\/code> in the href<\/code> attribute. These are simplified examples for the sake of conciseness, but you can define metrics with routines as long and complex as you want.<\/p>\n