Update e2e tests to take more screenshots

[brandyscarney]

Feature Request

Ionic version:

[x] 4.x

Describe the Feature Request
Update our e2e tests (specifically overlay components) to take screenshots of more examples.

Describe Preferred Solution
The action sheet splits all of the tests into multiple files:

However, alert loops through each button id to take the screenshots:

test('alert: basic', async () => {
  const page = await newE2EPage({
    url: '/src/components/alert/test/basic?ionic:_testing=true'
  });

  const alerts = [
    ['#basic'],
    ['#longMessage', 'long message'],
    ['#multipleButtons', 'multiple buttons'],
    ['#noMessage', 'no message'],
    ['#confirm', 'confirm'],
    ['#prompt', 'prompt'],
    ['#radio', 'radio'],
    ['#checkbox', 'checkbox']
  ];

  for (const [buttonSelector, message] of alerts) {
    await page.click(buttonSelector);

    const alert = await page.find('ion-alert');
    expect(alert).not.toBe(null);
    await alert.waitForVisible();
    await page.waitFor(250);

    const compare = await page.compareScreenshot(message);
    expect(compare).toMatchScreenshot();
    await alert.callMethod('dismiss');
  }

});

Meanwhile other components, such as loading aren't taking screenshots of anything but the basic one. I think we should be consistent here, and update the e2e tests for all of the overlay components to the same style:

• Action Sheet
• Alert
• Loading
• Menu
• Modal
• Picker
• Popover
• Toast

[brandyscarney]

cc @manucorporat @camwiegert

[manucorporat]

100% agree, i don't like how the action-sheet tests are spread across many fails, it makes it hard to add new tests! I like your format of using a loop, i would modify it to actually move the test() inside the loop itself, so we get better idea of what is being tested!

describe('alert: basic', async () => {
  const page = await newE2EPage({
    url: '/src/components/alert/test/basic?ionic:_testing=true'
  });

  const alerts = [
    ['#basic'],
    ['#longMessage', 'long message'],
    ['#multipleButtons', 'multiple buttons'],
    ['#noMessage', 'no message'],
    ['#confirm', 'confirm'],
    ['#prompt', 'prompt'],
    ['#radio', 'radio'],
    ['#checkbox', 'checkbox']
  ];

  for (const [buttonSelector, message] of alerts) {
    test(`click ${buttonSelector}`, async () => {
      await page.click(buttonSelector);

      const alert = await page.find('ion-alert');
      expect(alert).not.toBe(null);
      await alert.waitForVisible();
      await page.waitFor(250);

      const compare = await page.compareScreenshot(message);
      expect(compare).toMatchScreenshot();
      await alert.callMethod('dismiss');
    }
  }
});

[brandyscarney]

In looking to fix this I found a few problems:

• You can't loop over async tests in Jest, it all needs to be in one test block to loop or generated from a script, see is it possible to loop its in Jest jestjs/jest#1619
• Separating every compareScreenshot to its own test ends up with a lot of duplicated tests and increased time for the test because you have to redefine the page each time for each test so if you have a test that clicks multiple things on the same page and takes a few screenshots in between, you would have to repeat the previous test and incrementally add to it to get back to the same spot where the screenshot needs to be taken.
• When the expect fails on the screenshot, it stops taking the remaining screenshots - so if you have a mismatch on the first screenshot, take the alert for example, you won't get the remaining 7 screenshots in the test.

^ The last one is actually a much bigger problem for the tests because even when it fails on expects like the following it won't continue: https://github.com/ionic-team/ionic/blob/master/core/src/components/input/test/basic/e2e.ts#L15. This would mean that that screenshot won't ever be taken. So we need to somehow catch all of the errors from expects but continue and then throw them all at the end.

Here's a commit that attempts to address the problem of the expects failing on screenshots to at the very least avoid the screenshots being the source of a failure: 5511edb

cc @manucorporat @adamdbradley

[liamdebeasi]

Resolved by #17534