How to scrape all thread links from a XenForo forum board? [closed]

I'm trying to extract all thread links from a specific board on a XenForo forum (similar to this one: Creative Writing, specifically this one: NSFW Creative Writing), but a simple script I tried stops working on the second page.

I'm using a JavaScript snippet that runs directly in the browser's console because I want to use my existing logged-in session. I would normally use Python but I don't know how to handle the login authentication from Python.

The script automatically clicks through all the pages of the forum board and collects every unique thread link it finds.

Here is the complete script. Copy, navigate to the first page of the forum board to scrape, open the browser's developer console (usually by pressing F12), paste the script, and press Enter.

(async function() {
    const allLinks = new Set();
    let currentPage = 1;
    let hasNextPage = true;

    while (hasNextPage) {
        console.log(`Scraping page ${currentPage}...`);

        // Get all thread links on current page
        const links = document.querySelectorAll('a[href*="/threads/"]');
        links.forEach(link => {
            if (link.href && !link.href.includes('#') && link.href.includes('/threads/')) {
                allLinks.add(link.href);
            }
        });

        // Check for next page button - XenForo usually uses rel="next"
        const nextButton = document.querySelector('a.pageNav-jump--next, a[rel="next"]');

        if (nextButton) {
            nextButton.click();
            // Wait for the next page to load
            await new Promise(resolve => setTimeout(resolve, 5000));
            currentPage++;
        } else {
            hasNextPage = false;
        }
    }

    // Display and copy results
    console.log(`Finished! Found ${allLinks.size} unique thread links.`);
    const linksArray = Array.from(allLinks);
    const linksText = linksArray.join('\n');

    // Copy to clipboard
    try {
        await navigator.clipboard.writeText(linksText);
        console.log('✅ All links have been copied to your clipboard!');
    } catch (err) {
        console.error('❌ Failed to copy links to clipboard:', err);
        // Fallback: log the links to the console
        console.log('Links found:', linksArray);
    }

    return linksArray;
})();

Why does the script fail on the second page, and how can I fix this script to reliably scrape all pages and get all the story links?

posted 30 days ago

CC BY-SA 4.0

30d ago

ShadowsRanger‭

224 reputation 16 3 11 61