how to process 2 pages in diffrent threads?

[czcz1024]

i want to open 2 pages and create 2 thread,thread 1 process page1,and thread 2 process page 2.
i try this code

def run1(context):
    page = context.new_page()
    page.goto('https://page1')
    page.wait_for_timeout(5000)
    page.close()

def run2(context):
    page = context.new_page()
    page.goto('https://page2')
    page.wait_for_timeout(1000)
    page.close()

def main():
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=False)
        context = browser.new_context()
        t=Thread(target=run1,args=(context,))
        t1=Thread(target=run2,args=(context,))
        t.start()
        t1.start()
        t.join()
        t1.join()
        context.close()
        browser.close()

but first open page 1 and 5 seconds later ,it opens page 2.
it oepens the pages one by one
how can i process multi page in diffrent thread at same time

[kumaraditya303]

Playwright isn't thread safe so you need to start playwright separately for each thread or you can use asyncio

from playwright.sync_api import sync_playwright
from threading import Thread


def run1():
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=False)
        page = browser.new_page()
        page.goto("https://google.com")
        page.wait_for_timeout(1000)
        page.close()


def run2():
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=False)
        page = browser.new_page()
        page.goto("https://google.com")
        page.wait_for_timeout(1000)
        page.close()


def main():
    t = Thread(target=run1)
    t1 = Thread(target=run2)
    t.start()
    t1.start()
    t.join()
    t1.join()


if __name__ == "__main__":
    main()

[mxschmitt]

Closed as part of the triage and no response.

[sla-te]

Do I understand this one correctly, that if I create a new playwright object inside a newly opened thread it will be stable?

[mxschmitt]

exactly, this should work.

[sla-te]

Hmm, okay that confuses me because according to #470 playwright is not threadsafe, am I misinterpreting this issue?

[mxschmitt]

ah maybe I miss-interpreted it. If you create a new playwright instance in each thread, then it works fine. Sharing it with different threads does not work or is not stable.

[sla-te]

I understand you response as in, that in #470 the reply we got was incorrect, because as far as I understand the following snippet, - "create a new playwright instance in each thread" - is what we do:

from playwright.sync_api import sync_playwright
from concurrent.futures.thread import ThreadPoolExecutor

def sample_func():
    playwright_instance = sync_playwright().start()
    browser = playwright_instance.firefox.launch()
    page = browser.new_page()
    page.goto('http://whatsmyuseragent.org/')
    page.screenshot(path=f'example-{browser_type.name}.png')
    browser.close()

tpe = ThreadPoolExecutor()
for _ in range(100):
    tpe.submit(sample_func)
tpe.shutdown()

Is my assumption correct, that in contrary to the response in #470, that playwright is not threadsafe, that it is indeed threadsafe, if a new playwright instance is created inside each thread - As it is being done in the code snippet above?

[mxschmitt]

Yes this should work but I would not recommend spawning as many Playwright instances and do the scaling at that end. e.g. over multiple instances running on k8 and syncing over messaging queues like RabbitMQ would perform and scale better.

[sla-te]

Thank you for the quick responses.

I dont' fully understand what you mean by "do the scaling at that end, over multiple instances running on k8" - I am assuming, that you mean to not use Threading at all but create the logic in a way, that "if executed once it will execute one and only one instance" and then use kubernetes to have it run multiple times, like in the threading example?

[kumaraditya303]

Fyi: Achutha Balaji (@chwba) Thread safe means that it is not safe to call method or access attributes from a different thread. You can always create a playwright in TLS and it would be safe.

[sla-te]

Fyi: Achutha Balaji (@chwba) Thread safe means that it is not safe to call method or access attributes from a different thread. You can always create a playwright in TLS and it would be safe.

Understood. The reason my colleague had opened #470 was, that using playwright, as described on the code snippet(s), we were experiencing heavy "weird" behaviour of playwright, as in web-elements not being detected, which did indeed exist, inexplicable exceptions, that did not make much sense on first sight - None of this was happening if running only one instance.

Now I am looking for the best possible way to run multiple instances of playwright (roughly 100 simultaneously running instances at max, at the moment), but keeping playwright stable still - Now after Max Schmitt (@mxschmitt) 's response I have the feeling Threading would not be the right way to go, and Multiprocessing will eat up even more resources and expose additional challenges, especially if we create it async.

[kumaraditya303]

For using playwright in a multithreaded environment it is recommended to use TLS hence it would be safe and it will never cause wierd exceptions. You can avoid multithreading if you choose to use ProcessPoolExecutor and then use TLS per process to isolate it and hence it would be safe.

[sla-te]

For using playwright in a multithreaded environment it is recommended to use TLS hence it would be safe and it will never cause wierd exceptions. You can avoid multithreading if you choose to use ProcessPoolExecutor and then use TLS per process to isolate it and hence it would be safe.

What do you mean by use TLS? From how I interpret "use TLS", it means for me connect to the target website via "https" instead of "http", is this correct or am I misunderstanding what you mean?

Maybe I should clarify: The application will be running on a single dedicated server (32cores, 256gb ram), and we are not an organization but rather freelancers, who create automatic testing solutions for the websites we create.