Recently, while building a mobile app for my organization’s membership portal, I encountered an issue with YouTube API restrictions on private and unlisted videos. These restrictions prevent fetching of unlisted playlists, a significant issue if your organization uses YouTube to store its video content library. To circumvent these limitations, I developed a tool named PrivateYouTubeSync.

PrivateYouTubeSync is a Node.js application that enables us to fetch, save, and organize data from our organization’s YouTube channel, including private and unlisted videos.

In this post, let’s delve into the codebase and explore how it works.

Firstly, we need to import the necessary packages and define some constants. We will use Express.js for handling server-related tasks, Google’s APIs for YouTube data fetching, and fs and path for various file system and path-related operations. We also import json2csv for parsing json and converting it to csv, and our credentials.json file that we will create later.

const express = require('express');
const fs = require('fs');
const https = require('https');
const path = require('path');
const { google } = require('googleapis');
const { Parser } = require('json2csv');
const credentials = require('./credentials.json');

Then, we create our express application and define our port:

const app = express();
const port = 3000;
app.listen(port, () => {
  console.log(`Server is running on port ${port}`);
});

For OAuth2 authentication, we generate an authorization URL:

const { client_id, client_secret, redirect_uris } = credentials.installed;
const oauth2Client = new google.auth.OAuth2(client_id, client_secret, redirect_uris[0]);
const scopes = ['https://www.googleapis.com/auth/youtube.readonly'];
const authUrl = oauth2Client.generateAuthUrl({ access_type: 'offline', scope: scopes });

The credentials.json file should look like this:

{
    "installed": {
        "client_id": "your-client-id-here.apps.googleusercontent.com",
        "project_id": "your-project-id-here",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_secret": "your-client-secret-here",
        "redirect_uris": ["http://localhost:3000/oauth2callback"]
    }
}

To get the above information, you need to set up a new Google Cloud project, enable the YouTube Data API, navigate to the Credentials page in your project, and press “+ CREATE CREDENTIALS” at the top of the page, followed by clicking on “OAuth client ID.” Follow the process and copy-paste your credentials to the proper fields listed above in your credentials.json file.

Next, we define our app.get() function routes:

app.get('/', (req, res) => {
  res.redirect(authUrl);
});

The code above takes you to the OAuth screen to grant access to the related Google/YouTube account you are fetching the videos and playlists from.

Next, we define our /oauth2callback route, which is where the bulk of our application logic resides:

app.get('/oauth2callback', (req, res) => {
  const code = req.query.code;
  oauth2Client.getToken(code, async (err, tokens) => {
    if (err) {
      console.error('Error retrieving access token:', err);
      return;
    }

    oauth2Client.setCredentials(tokens);
    console.log('Successfully retrieved tokens');

    const youtube = google.youtube({
      version: 'v3',
      auth: oauth2Client,
    });

    async function fetchAllPlaylists(pageToken) {
      const response = await youtube.playlists.list({
        part: 'snippet,contentDetails',
        mine: true,
        pageToken: pageToken  
      });

      let playlists = response.data.items;

      if (response.data.nextPageToken) {
        const nextPagePlaylists = await fetchAllPlaylists(response.data.nextPageToken);
        playlists = playlists.concat(nextPagePlaylists);
      }
      if (playlists.length > 1) {
        playlists.pop();
      }
      return playlists;
    }

    const playlists = await fetchAllPlaylists();
    console.log('Successfully retrieved playlists');
      
    for (let playlist of playlists) {
      const playlistItemsResponse = await youtube.playlistItems.list({
        playlistId: playlist.id,
        part: 'snippet,contentDetails'
      });
      playlist.videos = playlistItemsResponse.data.items;
    }

    const dataToSave = playlists.map(playlist => {
      return {
        id: playlist.id,
        title: playlist.snippet.title,
        videos: playlist.videos
      };
    });

    fs.writeFileSync('playlists_with_titles_and_videos.json', JSON.stringify(dataToSave, null, 2));
    console.log('Playlists, their titles and videos saved to playlists_with_titles_and_videos.json');

    await convertJsonToCsv();

    fs.mkdir(path.join(__dirname, 'thumbnails'), { recursive: true }, (err) => {
      if (err) throw err;
    });

    dataToSave.forEach((playlist) => {
      playlist.videos.forEach((video) => {
        if (video.snippet.thumbnails && video.snippet.thumbnails.maxres) {
          const imageUrl = video.snippet.thumbnails.maxres.url;
          const videoId = video.snippet.resourceId.videoId;
          const imagePath = path.join(__dirname, 'thumbnails', `${videoId}.jpg`);

          downloadImage(imageUrl, imagePath, (err) => {
            if (err) throw err;
            console.log(`Downloaded image for video ${videoId}`);
          });
        } else {
          console.log(`No high resolution thumbnail available for video ${video.snippet.resourceId.videoId}`);
        }
      });
    });

    res.send('Playlists, their titles and videos saved. Images are downloading.'); 
  });
});

This code handles the YouTube API permission request and retrieves a JSON filecontaining all our playlists in the project folder. It also creates a folder called ‘thumbnails’ to download all of the maximum resolution thumbnails and store them with their relative video ID in jpg format. This is accomplished via our downloadImage() function:

function downloadImage(url, dest, cb) {
  const file = fs.createWriteStream(dest);
  https.get(url, (response) => {
    response.pipe(file);
    file.on('finish', () => {
      file.close(cb);
    });
  });
}

Lastly, we need to define our convertJsonToCsv() function, which is responsible for transforming our JSON data into CSV format:

async function convertJsonToCsv() {
  const playlists = require('./playlists_with_titles_and_videos.json');

  const flattenedData = playlists.reduce((arr, playlist) => {
    playlist.videos.forEach(video => {
      const thumbnailUrl = video.snippet.thumbnails?.maxres?.url || video.snippet.thumbnails?.high?.url || video.snippet.thumbnails?.default?.url;

      arr.push({
        playlistId: playlist.id,
        playlistTitle: playlist.title,
        videoId: video.contentDetails.videoId,
        videoTitle: video.snippet.title,
        videoDescription: video.snippet.description,
        videoThumbnailUrl: thumbnailUrl,
        videoPublishedAt: video.snippet.publishedAt
      });
    });
    return arr;
  }, []);

  const parser = new Parser();
  const csv = parser.parse(flattenedData);

  fs.writeFileSync('playlists_with_videos.csv', csv);
  console.log('CSV conversion complete');
}

This function flattens the data structure, ensuring each video has its own record, and then uses the json2csv library to convert the data to CSV. Once complete, the function saves the CSV data to a file named 'playlists_with_videos.csv'.

In conclusion, PrivateYouTubeSync is a handy tool that efficiently tackles the restrictions imposed by the YouTube API on private and unlisted videos. The tool’s ability to fetch, save, and organize data, including private and unlisted videos, is a significant relief for organizations that heavily rely on YouTube for their video content management.

Feel free to dive into the codebase and explore the tool further. You can access/clone the project on GitHub.

If you have any questions, feedback, or if you want to share your experience using this tool, I would be thrilled to hear from you. Feel free to connect with me on Twitter, Instagram or Threads, or visit my Website.

Happy coding!