A few weeks ago, we hosted a conference on the future of testing for mobile applications. Almost 4000 people registered for the event, creating a fun and exciting atmosphere in the chat for each session as well as for the live Q&A that followed. There was a lot to learn and it was a great opportunity to engage with the testing community on such an important topic.

The videos are all available now in our on-demand library and can be watched for free. If you want to dive right in and watch them right now and skip this recap, go ahead, I won’t blame you 😉. You can check them all out here.

The Path to Autonomous Testing — Gil Sever

The opening remarks were from Applitools CEO and co-founder Gil Sever. In this ten-minute presentation, Gil delivers a strong primer on what autonomous testing really is — and how machine learning can help assist humans and make testing much, much more effective. Tune in for a glimpse at the autonomous future.

On the Same Wavelength: Adding Radio to Your Testing Toolbox — Jason Huggins

Jason Huggins was the opening keynote speaker at the event, and he gave a fascinating talk on where testing is headed. As Jason says, “testing is getting weird,” and is increasingly about things you can’t even see. He argues that it’s time to move past an understanding of testing as just simulating what can be seen and tapped. What mobile testers are ultimately interested in today is the triggering of radio activity. That’s the essence of how your app truly performs, isn’t it?

Jason is a founder of Selenium, Appium, and Tapster Robotics, so he knows quite a bit about where testing has been and where it’s going. Check out his talk to hear what he has to say.

Appium 2.0: What’s Next — Sai Krishna, Srinivasan Sekar

Appium is a very popular test automation framework, and the upcoming release of Appium 2.0 is highly anticipated. Sai and Srinivasan are both contributors to the Appium project as well as lead consultants at ThoughtWorks, and in their presentation you’ll find a preview of what’s coming with Appium 2.0.

For example, today you need to install a large number of drivers when you install Appium server — even ones you don’t need. With Appium 2.0, you can just install the ones you need. Another example has to do with bug fixes — a lot of fixes are added to betas but many people don’t install betas and miss out, so with Appium 2.0 the fixes will be attached to individual drivers and rolled out faster. There will be improved docs and it’ll be easier to build your own plugins… the list goes on.

Catch Sai and Srini’s presentation to learn all about it. And if you’re ready to try it out for yourself, read our blog post on Getting Started with Appium 2.0 Beta.

Coffee Break

You might think there’s not much to recap during a coffee break, but during the first coffee break of the conference the brand-new Test Automation Cookbook was introduced to the world. This is a collection of bite-sized recipes you can use to answer a number of specific and common questions you may have about test automation. This “commercial break” was very well received by the audience 😊.

Mobile App Testing & Release Strategy — Anand Bagmar

Your mindset needs to be mobile-first. That’s how Anand, a Quality Evangelist and Solution Architect at Applitools, opened his talk. He followed that up with an overview of the differences between web and mobile testing/releasing, including mobile test automation on a local/cloud device lab. Anand explains that even after all our hard work in continuous testing, sometimes visual tests can still come down to a game of manual “spot the difference.”

Visual AI is a difference-maker there, as Anand explains. He talks about the difference between Visual AI and pixel comparisons and how you can apply it yourselves. Take a look at this talk for a great overview of mobile testing and releasing.

Next Generation Mobile Testing with Visual AI — Adam Carmi

Adam Carmi, a co-founder and CTO of Applitools, picked up with Anand left off with a deeper dive into Visual AI. Adam walks through a live demo of Applitools Eyes so you can see it for yourself. He talks about the huge code reduction when you use Eyes — up to 80% — which also gives you increased coverage and no validation logic to maintain. He backs this up with hard data from a hackathon, highlighting the fact that many testers were completely new to Applitools and were able to pick it up quickly and get some really strong results.

Adam’s talk was full of examples of how Eyes can work in the kinds of scenarios you may be wondering about, including how Eyes deals with different mobile form factors and how it batches together similar errors that can be approved/rejected together. Check it out.

Expert Panel: State of the Mobile Frameworks

This panel gathered together three mobile development experts for a robust discussion of what life is like for developers using different mobile frameworks. Eran Kinsbruner, DevOps Chief Evangelist and Sr. Director, Product Marketing at Perforce Software, Eugene Berezin, iOS Developer at Nordstrom and Moataz Nabil, Mobile Developer Advocate at Bitrise shared a lot of great information about the frameworks they use, which included Flutter, Appium, Kif, EarlGray and of course XCUITest and Espresso.

The panel was moderated by Justin Ison, Sr. Software Engineer at Applitools. Justin led the panel through a conversation around framework limitations, how to make apps testable, and what could make mobile testing easier. You can check out the whole discussion below. And if you’re curious for a quick comparison, be sure to take a look at a recent writeup on our blog that tackles Appium vs Espresso vs XCUITest.

The Future of Multi-Platform Integration Testing — Bijoya Chatterjee, Rajnikant Ambalpady

Bijoya and Rajnikant work on testing for the new SONY PlayStation 5, giving them a unique outlook on what it takes to deliver strong integration testing across platforms. In this talk, they describe the challenge of having many standalone apps that require automated testing, when there aren’t any off-the-shelf tools that are built to test a PlayStation! They ended up customizing Appium and making use of many other tools in their stack (this might be a good place to mention that Applitools is part of it, which I did not know until I heard Bijoya tell the audience 😊).

They cover the challenges of testing numerous standalone components within apps that must talk to each other, as well as testing across platforms from console to web to mobile. For a discussion of the pros and cons of end-to-end integration testing and much more, be sure to check out this talk.

Let the Robots Test Your Flutter App — Paulina Grigonis, Jorge Coca

It’s not easy to organize code so that it’s A) maintainable and customizable by development teams, and B) still easily understood and readable by business stakeholders. In this presentation, Paulina and Jorge, who are respectively business and technical experts at Very Good Ventures, walk us through a methodology they call the Robot Pattern. This pattern separates the “What” from the “How” of testing and can result in some pretty spiffy code. Definitely very easy to read even for a non-technical user.

Want to learn how to implement this pattern in your own development? Check out their presentation below.

Your Tests Lack Vision: Adding Eyes to Your Mobile Tests — Angie Jones

As humans, we can only pay attention to so much at one time — and that means we miss things, even in plain sight. The closing keynote from Angie Jones makes this clear from the first moments with a great video clip. I won’t spoil it, but it reminded me a lot of another video when I saw it, so after you watch Angie’s talk go ahead and take a look at this one too and see how it goes if you want to laugh at yourself.

After helping us all understand our blind spots, Angie provides a lot of great examples of how visual bugs slip through traditional testing processes. She then walks us through a demo of a new app and show us how Applitools Eyes can help us make sure it’s visually perfect. In the live Q&A, Angie also answers a number of questions around handling multiple viewport sizes or when you have to scroll, and even testing variations like light and dark mode or dealing with pop-up notifications and alerts.

Angie also shared her inspiration behind launching the automation cookbook (hint: it’s making the life of fellow testers easier). If you haven’t taken a look at it yet, be sure to check out the automation cookbook here.

You can see Angie’s full talk below.

Thank You!

And with that (and a few closing remarks from host Joe Colantonio of TestGuild) the Future of Testing Mobile event ended. We want to extend a huge thanks to everyone involved in making this event such a success, from Joe for his incredible hosting to the amazing speakers for sharing their insights to every attendee for adding your voices and presence. The event could not happen without all of you.

If you liked these videos, you might also like our videos from our previous Future of Testing events too — all free to watch. Happy testing!

Originally published at https://applitools.com on April 30, 2021.

We are excited to announce “Future of Testing: Mobile,” a free, live-streamed virtual event dedicated to the current trends and innovations shaping mobile test automation. 🥳 Throughout the last year, thousands of live participants have joined host Joe Colantonio and thought leaders from around the world to discuss and share technology, trends and success stories focused around software quality. This iteration of the popular event features presentations that aim to upskill developers, test engineers and QA professionals responsible for ensuring the delivery of quality mobile apps and experiences.

“Staying ahead of the curve is critical as technology is constantly changing. Quality leaders need to have a 360 degree understanding of the trends, technologies, and approaches available in order to be successful,” said Parasar Saha, Director of Quality Assurance at SOTI. “Applitools Future of Testing events provide the perfect balance, ensuring quality professionals are equipped with the knowledge they need to excel in today’s fast-paced app dev environment.”

For more information and to register for the live stream, visit: https://applitools.com/future-of-testing-mobile-north-america/

The one day event features speakers from brands like Nordstrom, SONY PlayStation, ThoughtWorks, Very Good Ventures and more. Domain experts and test automation practitioners will lead sessions that cover the latest practices and tools in mobile testing, automation, release strategies and multi-platform integrations.

Keynote speakers include Jason Huggins, founder of Selenium, Appium, and Tapster Robotics, as well as Angie Jones, Principal Automation Architect and Senior Director at Applitools and Test Automation University.

“The Future of Testing: Mobile virtual event is an amazing opportunity to upskill the community on the latest frameworks, tools and strategies shaping mobile test automation,” said Angie Jones, Principal Automation Architect and Senior Director at Applitools and Test Automation University. “With compatibility and environmental stability a priority in the mobile testing space, it is important that we continue to improve testing efficiency and enable engineers to improve upon their own mobile test automation initiatives.”

Some of the exclusive, live stream content includes:

• Appium 2.0: What’s Next — Join Sai Krishna and Srinivasan Sekar, lead consultants at ThoughtWorks and contributors to the Appium project, to learn everything you need to know about Appium 2.0 — the first major release of Appium in 7 years.
• Expert Panel: State of Mobile Frameworks — Join a discussion on the latest trends with Appium, Espresso, and XCUITest. This candid conversation is perfect for any team researching mobile test frameworks and trying to understand the pros and cons of each.
• Let the Robots Test Your Flutter App — Paulina Grigonis, and Jorge Coca, of Very Good Ventures, discuss automation of mobile apps developed using Flutter-an open source UI software development kit created by Google to help develop applications across various platforms with a single code-base.

See the full agenda and register for free at: https://applitools.com/future-of-testing-mobile-north-america/

See you there! 👋

========================================

I stirred up a bit of controversy on Twitter by asking for unpopular opinions on software testing, and unsurprisingly, the community had some pretty hot takes! While I asked this some time ago, it appears that these takes are just as relevant today.

Angie Jones on Twitter: "Unpopular Opinion: Software Testing Edition pic.twitter.com/NxwYbEwY6V / Twitter"

Unpopular Opinion: Software Testing Edition pic.twitter.com/NxwYbEwY6V

Trish Koo says “You don’t have to argue with people all the time to be a good tester”.

Trish Khoo on Twitter: "You don't have to argue with people all the time to be a good tester. There are less combative methods of persuasion and negotiation that work more effectively and upset less people. https://t.co/Ib4I681v34 / Twitter"

You don't have to argue with people all the time to be a good tester. There are less combative methods of persuasion and negotiation that work more effectively and upset less people. https://t.co/Ib4I681v34

As testers, we are often thought of as the bearer of bad news and have accepted this as part of the job. We often find ourselves fighting for features and fixes in the name of customer advocacy. It doesn’t have to be this way. The best testers are the ones that people want to invite to the meetings. The best testers realize that their job is so much more than simply finding and logging bugs, as pointed out by Jason Phebus.

Mr. Wally Brown Complaint Department on Twitter: "finding/logging bugs is the least important/interesting thing about testing https://t.co/cAbqejQppu / Twitter"

finding/logging bugs is the least important/interesting thing about testing https://t.co/cAbqejQppu

In fact, the best testers have not only found a way to become a part of the team, but have gotten their teams to realize that testing is a team sport.

angela | Black lives matter on Twitter: "Testing should involve the whole team. https://t.co/5UkYCbLYer / Twitter"

Testing should involve the whole team. https://t.co/5UkYCbLYer

Speaking of team sport, this one is from a developer who says “If you don’t start with tests, the design is probably garbage. I can’t think of any APIs I am responsible for and satisfied with that didn’t start with docs and tests”. This developer gets it! He understands his responsibility for quality even as a developer.

But I also often encourage testers not to leave the testing solely to developers during the design phase. As Jason Phebus said in the earlier tweet, logging bugs after the fact is the least interesting thing about testing. Get involved in your design meetings and help the team spot flaws in their approaches before a single line of code is written. Now, that’s powerful!

Let’s head on over to test automation…

Paul Grizzaffi says “A failing automation script does not necessarily mean that the script is wrong, horror of horrors, the script may be accurate”.

Paul Grizzaffi on Twitter: "A failing #automation script does not necessarily mean that the script is wrong, horror of horrors, the script may be accurate / Twitter"

A failing #automation script does not necessarily mean that the script is wrong, horror of horrors, the script may be accurate

If you immediately question your test script when it fails, it could be a sign that you are not confident in this test. A lot of times people are not confident in the tests because they barely understand what it is that they’ve coded. As the director of Test Automation University, I have a lot of people ask me about learning Selenium or Appium, or some other automation tool. I get it, you need to learn the tool to be able to do the job. But can I tell you that you’re shortchanging yourself? Automation development is so much more than a tool. Learn the craft, learn the strategies, really learn to code! That way, when things fail, you know how to properly analyze what’s going on.

Juho Perälä on Twitter: "Too often testers don't really understand why automation failed and use the environment or tools as scapegoat. / Twitter"

Too often testers don't really understand why automation failed and use the environment or tools as scapegoat.

Just think about how much more value you can add if you’re truly code literate, as pointed out by Amber Race.

Amber Race on Twitter: "Testers should be code literate. Reading code is a crucial skill for any tester and writing code has so many uses beyond just boilerplate automation. https://t.co/Tts0rzHI4Y / Twitter"

Testers should be code literate. Reading code is a crucial skill for any tester and writing code has so many uses beyond just boilerplate automation. https://t.co/Tts0rzHI4Y

Even if you’re only able to read it, you now can understand much more about how your application works and sniff out poor coding practices that can lead to bugs.

And if you are writing code, you don’t have to stop just at automating tests. You can contribute so much more to automating processes, increasing test automatibility within the product itself, and even fixing bugs that are found.

You may be thinking “well Angie, if I’m doing all of that then I’m doing the job of a developer”. Yep, you absolutely are.

Daniil Marchenko on Twitter: "Writing code is a job of a developer. Therefore a person who writes automated tests is a developer. https://t.co/6apm5fcsat / Twitter"

Writing code is a job of a developer. Therefore a person who writes automated tests is a developer. https://t.co/6apm5fcsat

As Justin Ison so accurately says, quality managers expect you to test the entire site, write detailed bug reports, learn to program, automate all test cases, and setup CI/CD for half the salary of the developers creating the bugs!

Justin Ison on Twitter: "Manager: We'll need you to test the entire site & write detailed bug reports. At the same time learn to program and automate all test cases. Additionally, setup a CI/CD process. To thank you for all of your hard work we'll pay you 1/2 the salary of the dev creating the bugs! ;) / Twitter"

Manager: We'll need you to test the entire site & write detailed bug reports. At the same time learn to program and automate all test cases. Additionally, setup a CI/CD process. To thank you for all of your hard work we'll pay you 1/2 the salary of the dev creating the bugs!

So, to all of the quality managers and executives reading this, listen to David Burns and pay your testers the same salary as your developers, if not more!

David Burns on Twitter: "Pay your testers the same salary as as your developers, and sometimes pay testers more. They have amazing empathy for developers and users at the same time! https://t.co/Y7oYjKPCss / Twitter"

Pay your testers the same salary as as your developers, and sometimes pay testers more. They have amazing empathy for developers and users at the same time! https://t.co/Y7oYjKPCss

Because “a tester’s mindset is the most powerful thing in computing”, says Orandi Harris. Know this, believe it, and act accordingly.

Orandi on Twitter: "A testers mindset is the most powerful thing in computing. https://t.co/QwGIsw8O51 / Twitter"

A testers mindset is the most powerful thing in computing. https://t.co/QwGIsw8O51

Originally published at https://applitools.com on April 1, 2021.

Automated testing of PDFs has traditionally been a challenging task in test automation. Bitmap imaging, the legacy state-of-the-art approach, has proven unreliable. Most automation teams have focused on their end-to-end user tests, except PDF tests. Instead, they have relegated PDF validation to manual testers. And, alas, manual testing is prone to error.

In this article, we will review the requirements of PDF testing. We will also cover approaches to automating PDF tests using Applitools.

Why PDF Testing?

More and more organizations transform their business process to online applications. The digital operating model requires that documents be electronically produced and sent to their customers. Assume a customer visits an insurance company or a bank to open an account. Increasingly, this work occurs exclusively with electronic records. After a successful setup, the institution sends a digital copy of the record to the customer. PDF offers the most sophisticated document layout and necessary security to serve as an electronic record. Account statements, invoices, receipts, documentation, and disclaimers get distributed as PDFs.

Organizations cannot own all possible UI failures. When users generate their own PDFs via the local operating system, they may encounter errors. But, when organizations generate receipts, account statements, and other documents for customer download, the organizations clearly own problems with those. A nd, in regulated industries, PDF problems can incur regulator wrath in addition to customer frustration. Thus, testing the generated outputs is mandatory from both quality and legal perspectives.

Organizations use Applitools to uncover problems in their transactional or customer-related PDF documents

What to Automate?

In regulated sectors like insurance, medical, and banking, end-user documents need to be accurate. Regulators who uncover inaccuracy in published documents can impose fines and other penalties. Organizations need to ensure that the PDFs are fully tested before being published to recipients.

Consider an application producing customer letters using a PDF template. Various sections of the PDF are dynamically updated with the customer data. A test for PDF needs to verify that both the content and layout of the output document are correct.

When testing for layout, the document should be fully formed with:

• the specific sections present
• in the right location and
• in the right order.

When testing for content, testers need to ensure that:

A layout failure can impact the processing of the documents by downstream systems. Testers need to be able to check content on a page or on locations on all or selected pages.

How traditionally organizations did PDF Testing?

Let us consider the following (two pages) policy document which is published by a large telco provider with critical summary information of their services:

Usually, organizations take an approach of validating the data using API testing and finally using solutions such as PDF boxes to test them on a page. However, most organizations rely on manual testing to validate the full document output. As organizations generate increasing numbers of electronic documents, they see the scope of effort needed for comprehensive testing. These organizations test PDF a sample of their documents manually.

Application of Visual AI in testing PDF

Applitools is an AI-powered Visual testing platform. Using a range of algorithms, Applitools compares images with the context of the human eye and speed of computer.. Applitools can test any user interface with 99.99% accuracy. Applitools reports only real differences visible to the human eye. These include any changes to color, contrast, position, size, or content.

In the case of a PDF, Applitools lets users inspect the entire document, or just selected pages. The scope of comparison matches the needs of the test. Users can target specific sections of a PDF for testing and ignore sections that are not relevant to a test. And, armed with the Applitools layout algorithm, users can validate a PDF layout even if the internal text has changed.

PDF Testing solution

Applitools PDF Tester is a codeless utility for automating the PDF testing of any document using Visual AI. You can validate the content in a page or a region across selected pages or all pages of the PDF.

Look at how the previous example can be tested using Applitools PDF Tester. Configure the test job as an XML. Add the content validations as test assertions. You can create the XML manually, or you can build it programmatically using a script. Once completed, execute the PDF test using the Applitools PDF Test application. You can run the test from the command line using any batch process.

Following are the results:

The utility reports all the assertions in the PDF document and reports the result as ‘Passed’ or ‘Failed’. In addition, the utility tests the fully formatted output document against a baseline and reports any differences discovered.

Logging into Applitools dashboard we can review all the differences spotted by AI:

Here, AI is highlighting all the content, size, color, positioning, and font differences. Engineers can validate the differences, ensuring document accuracy prior to publication. We can further instruct AI to specifically test or ignore sections of the page by using annotations.

Conclusion

While organizations have largely automated web and mobile application tests, they have still struggled to automate testing of PDFs. Utilizing the capabilities of AI in testing for a completed document has proven to solve PDF test automation. By adding both full document and dynamic data tests, teams can incorporate PDF testing in their end-to-end test automation.

Originally published at https://applitools.com on March 5, 2021.

Introductions

Angie Jones serves as Senior Director of Test Automation University and as Principal Developer Advocate at Applitools. She tweets at @techgirl1908, and her website is https://angiejones.tech.

Brian Douglas serves as the Staff Developer Advocate at GitHub. Insiders know him as the “Beyoncé of GitHub.” He blogs at https://bdougie.live, and tweets as @bdougieYO.

They ran their webinar as a question-and-answer session. Here are some of the key ideas covered.

What Are GitHub Actions?

Angie’s first question asked Brian to jump into GitHub Actions.

Brian explained that GitHub Actions is a feature you can use to automate actions in GitHub. GitHub Actions let you code event-driven automation inside GitHub. You build monitors for events, and when those events occur, they trigger workflows.

If you’re already storing your code in GitHub, you can use GitHub Actions to automate anything you can access via webhook from GitHub. As a result, you can build and manage all the processes that matter to your code without leaving GitHub.

Build Test Deploy

Next, Angie asked about Build, Test, Deploy as what she hears about most frequently when she hears about GitHub Actions.

Brian mentioned that the term, GitOps, describes the idea that a push to GitHub drives some kind of activity. A user adding a file should initiate other actions based on that file. External software vendors have built their own hooks to drive things like continuous integration with GitHub. GitHub Actions simplifies these integrations by using native code now built into GitHub.com.

Brian explained how GitHub Actions can launch a workflow. He gave the example that a team has created a JavaScript test in Jest. There’s an existing test using Jest — either npm test, or jest. With GitHub Action workflows, the development team can automate actions based on a starting action. In this case, the operator can drive GitHub to execute the test based on uploading the JavaScript file.

Get Back To What You Like To Do

Angie pointed out that this catchphrase, “Get back to what you like to do,” caught her attention. She spends lots of time in meetings and doing other tasks when she’d really just like to be coding. So, she asked Brian, how does that work?

Brian explained that, as teams grow, so much more of the work becomes coordination and orchestration. Leaders have to answer questions like:

• What should happen during a pull request?
• How do we automate testing?
• How do we manage our build processes

When engineers have to answer these questions with external products and processes, they stop coding. With GitHub Actions, Brian said, you can code your own workflow controls. You can ensure consistency by coding the actions yourself. And, by using GitHub Actions, you make the processes transparent for everyone on the team.

Do you want a process to call Applitools? That’s easy to set up.

Brian explained that GitHub hosted a GitHub Actions Hackathon in late 2020. The team coded the controls for the submission process into the hackathon. You can still check it out at githubhackathon.com.

The entire submission process got automated to check for all the proper files being included in a submission. The code recognized completed submissions on the hackathon home page automatically.

Brian then gave the example of work he did on the GitHub Hacktoberfest in October. For the team working on the code, Brian developed a custom workflow that allowed any authenticated individual to sign up to address issues exposed in the Hackathon. Brian’s code latched onto existing authentication code to validate that individuals could participate in the process and assigned their identity to the issue. As the developer, Brain built the workflow for these tasks using GitHub Actions.

What can you automate? Informing your team when a user does a pull request. Send a tweet when the team releases a build. Any webhook in GitHub you can automate with GitHub Actions. For example, you can even automate the nag emails that get sent out when a pull request review does not complete within a specified time.

Common Actions

Angie then asked about the most common actions that Brian sees users running.

Brian summarized by saying, basically, continuous integration (CI). The most common use is ensuring that tests get run against code as it gets checked in to ensure that test suites get applied. You can have tests run when you push code to a branch, push code to a release branch or do a release, or even when you do a pull request.

While test execution gets run most frequently, there are plenty of tasks that one can automate. Brian did something specific to assign gifts to team members who reviewed pull requests. He also used a cron job to automate a GitHub Action which opened up a global team issue each Sunday US, which happens to be Monday in Australia, and assigned all the team members to this issue. Each member needed to explain what they were working on. This way, the globally-distributed team could stay on top of their work together without a meeting that would occur at an awkward time for at least one group of team members.

Brian talked about people coming up with truly use cases — like someone linking IOT devices to webhooks in existing APIs using GitHub Actions.

But the cool part of these actions is that most of them are open source and searchable. Anyone can inspect actions and, if they don’t like them, modify them. If a repo includes GitHub Actions, they’re searchable.

On github.dom/bdougie, you can see existing workflows that Brian has already put together.

Jump Into GitHub Actions — What Next?

I shared some of the basic ideas in Brian’s conversation with Angie. If you want to jump into GitHub Actions in more detail, you can check out the full webinar and the slides in Addie Ben Yehuda’s summary blog for the webinar. That blog also includes a number of Brian’s links, several of which I include here as well:

Enjoy jumping into GitHub Actions!

Originally published at https://applitools.com on March 2, 2021.

Applitools has reached 75,000 students enrolled in Test Automation University, a global online platform led by Angie Jones that provides free courses on things test automation. Today, more engineers understand how to create, manage, and maintain automated tests.

What Engineers Have Learned on TAU

Engineers have learned how to automate UI, mobile, and API tests. They have learned to write tests in specific languages, including Java, JavaScript, Python, Ruby, and C#. They have applied tests through a range of frameworks including Selenium, Cypress, WebdriverIO, TestCafe, Appium, and Jest.

75,000 engineers would exceed the size of some 19,000 cities and towns in the United States. They work at large, established companies and growing startups. They work on every continent with the possible exception of Antarctica.

What makes Test Automation University possible? Contributors, who create all the coursework.

Thank You, Instructors

As of this writing, Test Automation University consists of 54 courses taught by 39 different instructors. Each instructor has contributed knowledge and expertise. You can find the list of authors on the Test Automation University home page.

Here are the instructors of the most recently added courses to TAU.

Thank You, Students

As engineers and thinkers, the students continue to expand their knowledge through TAU coursework.

Each course contains quizzes of several questions per chapter. Each student who completes a course gets credit for questions answered correctly. Students who have completed the most courses and answered the most questions successfully make up the TAU 100.

Some of the students who lead on the TAU 100 include:

Join the 75K!

Get inspired by the engineers around the world who are learning new test automation skills through Test Automation University.

Through the courses on TAU, you’ll not only learn how to automate tests, but more importantly, you’ll learn to eliminate redundant tests, add automation into your continuous integration processes, and make your testing an integral part of your build and delivery processes.

Learn a new language. Pick up a new testing framework. Know how to automate tests for each part of your development process — from unit and API tests through user interface, on-device, and end-to-end tests.

No matter what you learn, you will become more valuable to your team and company with your skills on how to improve quality through automation.

Originally published at https://applitools.com on February 23, 2021.

======================================

A common question is “does Applitools visual testing work on mobile apps?”. The answer is a resounding YES! In this post, I’ll automate a couple of scenarios and show you the power of visual testing.

The App

We’re going to work with a Todo App which will allow us to add tasks. A cool feature of this app is the ability to add colors to tasks, which is a great way to designate categories for our various types of tasks.

Scenario #1: Add multiple tasks of different categories

Our first scenario is to add four tasks, one for each of the different colors/categories. So here, we declare the 4 tasks and add them to the app: one for purple, blue, yellow, and pink. Next, we verify that the tasks were indeed added.

However, a key aspect of this scenario is to verify that the tasks were added as the right color. This is a native app, so we can’t make a call to get the CSS value as that is unsupported. And when looking in the Inspector, there was nothing there that identified these tasks by color. Fortunately, visual testing can help us with this!

What is Visual Testing?

Visual testing is an automated technique for verifying the state of your application by using snapshots. This means your tests are not limited to interrogating the DOM to determine status, like what is done with most other automation tools. Instead, with visual testing, a screenshot of your application is taken on every regression run and compared against a baseline image. Therefore, viewing and verifying your application as your users would!

Applitools’ visual testing API is called Eyes. The Eyes API uses AI to compare the images and is much more reliable than all other forms of visual testing, namely the notoriously flaky pixel-to-pixel and DOM-based approaches offered by other vendors.

Adding Visual Testing

The Eyes API can be added to your existing tests so there’s no need to throwout your test suite and start over! Let’s add visual testing to Scenario #1 so that we can make sure the tasks were added with the right colors.

Here we’ve added visual testing with 4 lines of code! 😲

On line 21, we’re simply initializing the Eyes API. On line 22 is where we tell Eyes that we would like to begin visually testing.

On line 23 is where the magic happens — we take a screenshot of our app. If this is the first time the test is run, this will become the baseline image. If there is already a baseline image, a new image will be taken and compared to the baseline image to determine if there are any differences. Notice the arguments sent into this call. This is telling Eyes to ignore the status bar region of our app because it is dynamic and will contain a timestamp and notifications. So this part of the app will be ignored every time the test runs!

Finally, on line 24, we’re done so we can close our Eyes. At this point, the screenshots are sent to the Applitools cloud, analyzed, and stored in a nice dashboard. In the event of a failure, your team can easily review the images to determine where the application has changed, and annotate the image with bugs and remarks.

But wait…

We added visual testing to verify the colors of the tasks. But it’s doing so much more. It’s verifying the entire screen (sans the status bar which we explicitly said to ignore). So that means all of our assertions above are already covered by Eyes and we can actually DELETE CODE!!! 🤯 In fact, we’ve deleted almost HALF of the code from this one test alone — with MORE coverage!!! Woohooo 🎉

Let’s try another more scenario.

Scenario #2: Complete a task from the list

In our next scenario, we need to complete one of the tasks by clicking on it, and then verify that it is crossed off.

Now when it’s time to verify, we run into the same issue we did in Scenario #1. The strikethrough on the completed task is a CSS effect, and cannot be verified using the Inspector. So, let’s use visual testing.

Now we’re covered! Ok, one more scenario…

Scenario #3: Add long tasks

For the last scenario, we want to add long tasks. What is the purpose of this test? It’s to ensure that on this mobile view, the text is displayed properly and doesn’t get cut off or bleed off the edge of the page.

But how can we make sure the text is displayed correctly without using visual testing? We can’t! We can make sure it exists. We can make sure it’s visible. But only visual testing will make sure that it’s showing up just as we intend it to.

Visual Bugs Love Mobile Viewports

Mobile testing is such a challenge. In addition to not being able to access style attributes from the Inspector, the presentation of your apps can vary based on the various viewport sizes. This makes mobile apps ripe for visual bugs! To make matters worse, traditional mobile automation tools only offer so much and are incapable of catching these visual bugs, and heck, even some of the functional ones as we’ve seen in Scenarios 1 and 2. If you’re doing mobile app test automation, you need visual testing! Wait no more; create your forever-free Applitools account and improve your testing today!

Originally published at https://applitools.com on February 22, 2021.

Appium is an open source test automation framework for use with native, hybrid and mobile web apps.

It drives iOS, Android, and Windows apps using the WebDriver protocol and gives you the ability to automate any mobile app from any language and any test framework.

Appium released its first major version almost 7 years ago. Since then, Appium has rolled out a lot of new features and its automation backend architecture has evolved quite a lot.

That said, getting started with implementing automated tests and executing them using Appium the 1st time can be a daunting experience.

There are a lot of dependencies that need to be set up — node, android-command-line tools, appium, etc.

Appium Setup Scripts

In order to make this task easier, and avoid having to do this manually, one at a time, I have written a script that will help set the Appium-Android environment automatically for you. The script is available for Mac OSX and Ubuntu and will set up the latest available version of the dependencies & appium itself.

You can find the scripts here:

MacOSX: setup_mac.sh

Ubuntu: setup_linux.sh

The mentioned setup scripts install all dependencies required for implementing / running tests on Android devices. To do the setup for iOS devices, run appium-doctor and see the list of dependencies that are missing, and install the same.

To ensure the setup is working properly, you can clone / download this sample repository ( https://github.com/anandbagmar/AppiumJavaSample) which has 2 tests that can be executed to verify the setup.

Refer to the README file for specifics of the prerequisites to do the setup and also for running the tests.

To get full value of your functional test automation, add the power of Visual AI to your tests. You can refer to the relevant tutorial for adding Applitools Visual AI to your Appium tests from here.

Summary

Automate your Android and Appium test execution environment setup using an automated script on MacOSX ( setup_mac.sh) and Ubuntu ( setup_linux.sh).

Originally published at https://applitools.com on February 19, 2021.

Thunderhead is the recognised global leader in the Customer Journey Orchestration and Analytics market. The ONE Engagement Hub helps global brands build customer engagement in the era of digital transformation.

Thunderhead provides its users with great insights into customer behavior. To continue to improve user experience with their highly-visual web application, Thunderhead develops continuously. How does Thunderhead keep this visual user experience working well? A key component is Applitools.

Before — Using Traditional Output Locators

Prior to using Applitools, Thunderhead drove its UI-driven tests with Selenium for browser automation and Python as the primary test language. They used traditional web element locators both for setting test conditions and for measuring the page responses.

Element locators have been state-of-the-art for measuring page response because of precision. Locators get generated programmatically. Test developers can find any visual structure on the page as an element.

Depending on page complexity, a given page can have dozens, or even hundreds, of locators. Because test developers can inspect individual locators, they can choose which elements they want to check. But, locators limit inspection. If a change takes place outside the selected locators, the test cannot find the change.

These output locators must be maintained as the application changes. Unmaintained locators can cause test problems by reporting errors because the locator value has changed while the test has not. Locators may also remain the same but reflect a different behavior not caught by a test.

Thunderhead engineers knew about pixel diff tools for visual validation. They also had experience with those tools; they had concluded that pixel diff tools would be unusable for test automation because of the frequency of false positives.

Introducing Applitools at Thunderhead

When Thunderhead started looking to improve their test throughput, they came across Applitools. Thunderhead had not considered a visual validation tool, but Applitools made some interesting claims. The engineers thought that AI might be marketing buzz, but they were intrigued by a tool that could abstract pixels into visible elements.

As they began using Applitools, Thunderhead engineers realized that Applitools gave them the ability to inspect an entire page. Not only that, Applitools would capture visual differences without yielding bogus errors. Soon they realized that Applitools offered more coverage than their existing web locator tests, with less overall maintenance because of reduced code.

The net benefits included:

• Coverage — Thunderhead could write tests for each visible on-page element on every page
• Maintainability — By measuring the responses visually, Thunderhead did not have to maintain all the web element locator code for the responses — reducing the effort needed to maintain tests
• Visual Validation — Applitools helped Thunderhead engineers see the visual differences between builds under test, highlighting problems and aiding problem-solving.
• Faster operation — Visual validation analyzed more quickly than traditional web element locators.

Moving Visual Testing Into Development

After. using Applitools in end-to-end testing, Thunderhead realized that Applitools could help in several areas.

First, Applitools could help with development. Often, when developers made changes to the user interface, unintended consequences could show up at check-in time. However, by waiting for end-to-end tests to expose these issues, developers often had to stop existing work and shift context to repair older code. By moving visual validation to check-in, Thunderhead could make developers more effective.

Second, developers often waited to run their full suite of element locator tests until final build. These tests ran against multiple platforms, browsers, and viewports. The net test run would take several hours. The equivalent test. using Applitools took five minutes. So, Thunderhead could run these tests with every build.

For Thunderhead, the net result was both greater coverage with tests run at the right time for developer productivity.

Adding Visual Testing to Component Tests

Most recently, Thunderhead has seen the value of using a component library in their application development. By standardizing on the library, Thunderhead looks to improve development productivity over time. Components ensure that applications provide consistency across different development teams and use cases.

To ensure component behavior, Thunderhead uses Applitools to validate the individual components in the library. Thunderhead also tests the components in mocks that demonstrate the components in typical deployment uses cases.

By adding visual validation to components, Thunderhead expects to see visual consistency validated much earlier in the application development cycle.

Other Benefits From Applitools

Beyond the benefits listed above, Thunderhead has seen the virtual elimination of visual defects found through end-to-end testing. The check-in and build tests have exposed the vast majority of visual behavior issues during the development cycle. They have also made developers more productive by eliminating the context switches previously needed if bugs were discovered during end-to-end testing. As a result, Thunderhead has gained greater predictability in the development process.

In turn, Thunderhead engineers have gained greater agility. They can try new code and behaviors and know they will visually catch all unexpected behaviors. As a result, they are learning previously-unexplored dependencies in their code base. As they expose these dependencies, Thunderhead engineers gain greater control of their application delivery process.

With predictability and control comes confidence. Using Applitools has given Thunderhead increased confidence in the effectiveness of their design processes and product delivery. With Applitools, Thunderhead knows how customers will experience the ONE platform and how that experience changes over time.

Originally published at https://applitools.com on February 16, 2021.

In this series, we discuss the results of the Applitools Ultrafast Cross Browser Hackathon. Today, we will explain how those who were skeptics about cross browser testing would recommend that their organizations run Applitools Ultrafast Grid for cross browser testing.

Reviewing Past Posts In This Series

In this series we have covered the results of the Applitools Ultrafast Cross Browser Hackathon.

• My first post covered the overall results. I wrote specifically about the ease of creating cross browser tests with Applitools and introduced the topics to follow.
• My second post covered how the speed of Applitools Ultrafast Grid makes cross browser testing a reality within the application build process.
• In the third, I explained how Applitools Visual AI tests provide much greater code stability compared with legacy cross browser tests, making the code easier to develop and maintain over time.

With this post, I will cover the survey topic in the hackathon: would Hackathon participants recommend that their organizations adopt legacy cross browser testing approach to their peers, and would they recommend Applitools Ultrafast Grid for that purpose?

Methodology

For this survey, Applitools used an approach called Net Promoter Score (NPS). Net Promoter Score uses a survey question with the highest correlation to satisfaction:

“On a scale of 0 to 10, with 0 being not likely and 10 being highly likely, how likely are you to recommend [the survey object] to others?”

Researchers have shown that this question correlates most highly with satisfaction. Respondents who give a 9 or 10 (highly likely to recommend) get classified as “promoters.” Promoters have high satisfaction. Those who give a 7 or 8 get classified as neutral — they are neither satisfied or dissatisfied. Others with a 6 or below get classified as detractors. Detractors have been dissatisfied with something and have no willingness to recommend the survey object.

Break the survey responses into respondents by value and count. Add 1 for each 10 and 9. Give 0 for each 7 or 8. Subtract 1 for each 6 or below. Then, normalize the results to 100 by taking your count and dividing it by the number of respondents, and multiplying by 100.

Results can range from -100 to 100. According to Bain & Co, the developers of NPS:

• 80 rates as ‘world class’
• 50 rates as ‘excellent’
• 20 rates as ‘favorable’
• Around 0 rates as ‘neutral’
• -10 and lower rates as ‘negative’

Asking Hackathon Participants

Applitools surveyed the the 2,224 Hackathon participants about their willingness to recommend the use of Applitools Ultrafast Grid to their peers. The survey also asked their willingness to recommend the use of legacy cross browser testing. Of the participants, 203 engineers were able to run both the legacy and the Ultrafast cross browser tests.

Here were the survey responses:

Few Fans Of Traditional Cross Browser Testing

For legacy cross browser testing, using a traditional test application and validation process, 68% of participants got classified as detractors. 17% got classified as passives. Only 15% promoted the legacy approach. This gave a NPS of -54 (rounded down). Participants were, in general, not fans of legacy cross browser testing.

This result mirrors how much people use the legacy approach to cross browser testing. Companies listed in the graphic above provide the infrastructure to run tests. They don’t reduce the test load, or the code load. Their pricing reflects the cheaper cost for them to set up and maintain that infrastructure of devices, browsers, operating systems, and viewport sizes. Given the cost of setting up and maintaining legacy cross browser tests, it makes sense that not a lot of companies use cross browser testing actively.

Willing to Recommend Applitools Ultrafast Grid for Cross Browser Testing

Also, the survey asked participants their willingness to recommend Applitools Ultrafast Grid for Cross Browser testing. Here, 75% gave a 10 or 9 and got classified as promoters. Another 20% responded 7 or 8 and got classified as neutral.

The promoters valued:

• Speed of tests — noting they could run their tests in the build process
• Simplicity of management — no tests needed to be run and tuned across multiple plaforms
• Simplified code management — fewer locators meant easier to set up and manage
• More accurate — the underlying Visual AI wasn’t plagued by false positives and caught all the code errors

The promoters discovered the ease of creating and maintaining cross browser tests using Applitools Ultrafast Grid. The promoters also realized that, with tests completed and analzyed accurately in well under 10 minutes, Applitools Ultrafast Grid made it possible to run cross browser tests in the scope of a build or during unit tests. Legacy tests, even when run in parallel, took tens of minutes to complete and analyze.

Implications of Recommending Cross Browser Testing

If you read my earlier posts, you know that two camps existed related to cross browser testing. One camp ran cross browser tests because they had encountered issues in the past and saw cross browser tests as a safe approach. The other camp avoided it altogether and thought cross browser testing unnecessary.

There is a third camp using Applitools Ultrafast Grid. This camp recognizes that the combination of:

• test speed appropriate for software build processes
• ease of deployment
• lack of infrastructure to manage, and
• simplified test code management,

made cross browser testing feasible. This third camp can deploy Applitools Ultrafast Grid to run and validate rendering behavior for any combination of browser, operating system, and viewport size and run this test set quickly at the unit, build, merge, and final test timeframes.

What then are the implications for the Applitools Ultrafast Cross Browser Hackathon? 75% of the 208 engineers who completed both sets of tests could see the value of Applitools Ultrafast Grid just by using it. And, they would be willing to recommend it. They realized that, whether they had previously released a bug to the field that they could have caught with cross browser testing, that Ultrafast Grid changed their understanding about this kind of testing completely.

Applitools users know that Applitools Visual AI makes it possible to run visual tests as part of their unit tests. These users can incorporate visual tests at every build and merge. And with Applitools Ultrafast Grid, they can incorporate cross browser tests as well.

Importantly, Hackathon participants learned these lessons just through their time on the Hackathon.

What These Recommendations Mean For You

When you have 75% of engineers recommending something, it might be worth trying. Whether you classify yourself as a cross browser skeptic or a grudging participant, it is clear that Applitools Ultrafast Grid offers something that differs from your current conception of cross browser testing.

At the very least, read the Applitools results in detail. You will learn why the engineers gave these recommendations.

More importantly, why not give Applitools a try? Sign up for a free account? Or, if you prefer, request a demo from an Applitools representative.

Next Week

Next week, in my last blog post in this series, I will help you draw your own conclusions about Modern Cross Browser Testing.

Originally published at https://applitools.com on February 12, 2021.