Project Progress: Steam Backlog Helper

If you've ever had a huge backlog of potentially amazing games on Steam, look no further because I (might) have a solution for you :)

Introduction

Steam Backlog Helper is a website I made to solve an issue many people experience, myself included - the endless backlog of games on Steam you can never quite get around to finishing. Scrolling through my library, there are many games I haven't even touched, and every single one of those could be a memorable experience.

This is why I decided to start my own project to solve this issue, which is where Steam Backlog Helper comes in. It is being built completely in Angular 18, with a NestJs (TypeScript) backend.

How does it work?

Steam Backlog Helper (SBH for short) contains all of your steam library, along with game metadata and your personal stats regarding those games. This includes your favorite genres of games, your playtime for each game, time required to complete games, tags & categories of games and a recommendation page.

You initally log in with Steam from our Home Page, after which you are redirected to the dashboard/games page. Once you log in with Steam, the backend does several things:

1. Fetch user library data from Steam:

const response = await firstValueFrom(
      this.http.get<GameResponse>(
        'https://api.steampowered.com/IPlayerService/GetOwnedGames/v0001/',
        {
          params: {
            steamid: steamId,
            include_appinfo: 1,
            format: 'json',
          },
        },
      ),

Omitted the key (even though it's grabbed by the Config service) for security purposes. Unfortunately for me, Steam API returns a very surface-level response regarding the users' games - which makes sense, given that there is a LOT of data to process. We receive data in this format:

"game_count": 193,
        "games": [
            {
                "appid": 4000,
                "playtime_forever": 5794,
                "playtime_windows_forever": 0,
                "playtime_mac_forever": 0,
                "playtime_linux_forever": 0,
                "playtime_deck_forever": 0,
                "rtime_last_played": 1542490381,
                "playtime_disconnected": 0
            },...

As you can see, we only have playtime & appid, but for SBH we need more data: Title, Description, Genres, Categories, Completion status etc.

At this point, we move on to step 2:

2. Associate games with user in owned_games DB table:

for (const g of games) {

      await this.ownedRepo.upsert(
        {
          user: { id: user.id },
          appid: g.appid,
          playtime_minutes: g.playtime_forever,
          last_played: g.rtime_last_played
            ? new Date(g.rtime_last_played * 1000)
            : null,
        },
        ['user', 'appid'],
      );
    }

This is pretty self-explanatory, we just write the user's games to the owned_games database. With this data in place, we can move on to the next step.

3. Check and insert MetaData if required In this step, I go through all of the games and check them across another table: game_metadata. This table contains associations with games (appid) and their respective metadata which I collect.

In order to determine which games are missing metadata, I use a Set. I collect all the appids which are in game_metadata, and isolate them in this set, so that I could later compare them to all of the appids, thus effectively extracting missing metadata.

const appIds = games.map((g: Game) => g.appid);
    const existingMetadata = await this.metadataRepo.findBy({
      appid: In<Number>(appIds),
    });

    const existingAppIds = new Set(existingMetadata.map((m: GameMetadata) => m.appid));
    const missingAppIds = appIds.filter((appid: number) => !existingAppIds.has(appid));

    for (const appid of missingAppIds) {
      await this.metadataQueue.addFetchJob(appid);
    }

The last part is interesting -> await this.metadataQueue.addFetchJob(appid);. In order to keep the website stable, and also allow the user to use the website while loading, I delegate the job of scraping and finding metadata to a BullMQ worker! This worker, once assigned a job, goes on to find and insert metadata into the correct table, in the background.

In my worker service, I have defined the following function:

async getGameDetails(appid: number){
        const obs = this.http.get<GameDetailsResponse>(`https://store.steampowered.com/api/appdetails`, {
            params: { appids: appid, l: 'english'}, headers: {
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
                '(KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36',
                'Accept': 'application/json'
            }
        }).pipe(
            map(response => response.data)
        );
        return firstValueFrom(obs);
    }

All it does is it checks the game against the Steam Store API (api.steampowered != store.steampowered.com/api IMPORTANT), which does have a lot of the metadata I need. The extra headers are to avoid timeouts and blocks from the Store API (which has happened several times before).

Now that we understand where and how the worker gets the metadata, let's move on to the job itself.

async job => {
                const {appid} = job.data;
                await this.sleep(2000);
                let data: GameDetailsResponse;
                try {
                    data = await this.workerService.getGameDetails(appid);
                } catch (err) {
                    console.error(`Failed to fetch Steam API for appid ${appid}`, err.message);
                    return;
                }...

You might notice the 2 second sleep/delay, this is due to the API restrictions on Steam. After acquiring metadata from Steam Store, I then find the community reported time-to-beat. Originally, I've used IGDB and their API for this, however, it had plenty of gaps and was messy to use for me. Instead, I opted for RAWG. RAWG DB has a great amount of games and the data is excellent.

The worker fetches the time-to-beat data as following:

     const hltb = await this.hltbService.getGameTime(appData.name);

It's called

hltb

as an association to 'HowLongToBeat'. Next, it saves the metadata and calculates if the user completed the game (which it saves to the owned_games repo).

try {
                    await this.metadataRepo.save({
                        appid,
                        name: appData.name,
                        genres: appData.genres?.map((g: Genre) => g.description),
                        categories: appData.categories?.map((c: Genre) => c.description),
                        tags: [],
                        last_fetched: new Date(),
                        header_image: appData.header_image,
                        description: appData.short_description,
                        hltb_main_story: hltb
                    });

                    try {
                        const game = await this.ownedGameRepo.findOne({
                            where: {
                                appid: appid
                            }
                        });

                        if(!game){
                            console.log('Could not get game from ownedgames table');
                            return;
                        }

                        game.isCompleted = this.getIsCompleted(game.playtime_minutes, hltb);
                        await this.ownedGameRepo.save(game);
                    } catch (error){
                        console.log('Error saving isCompleted: ', error);
                    }
                    console.log('Saved metadata for: ', appData.name);
                    } catch (err) {
                      console.log(`Failed to save metadata for appid ${appid}`, err);
                    }

This pretty much concludes the core of the worker. There are also fail-safes and conditions for re-fetching in the service (once a week and if certain fields are missing).

The cool thing about this setup is that once a game's metadata is loaded, it's universal, meaning new users have to wait less time after signing up if their games' metadata is already collected.

After the initial log in / sign up, the user does not have to go through steam sign up until their token (given by SBH) expires. During this period, their games do not need to be assigned to the worker or fetched etc, instead, the data is pulled from the database, resulting in very fast response times.

Once in the dashboard, the user will see all of their games, paginated and sorted with appropriate tags: Genres, Playtime status (Never played, Less than 3hr played...) etc. Given the nature of Steam, online & offline play there is room for error when assessing whether or not a user has completed a game. To combat this, within the game card, the user is given an option to correct the status, marking a game either completed or not completed (in addition to the automatically calculated expectation).

Top Picks For You / Recommendations

This functionality extracts and ranks game choices for the user. Currently, it uses only vectors and weights to make this assumption, however, in the future it is very likely going to use some form of ML.

The logic for this is actually not complex and I will go through it quickly:

1. Get all user's games and extract genre counts

Simple, get all games and order genres in an Object which contains the number of instances a genre has appeared.

const genreCount: Record<string, number> = {};
    games.forEach(g => {
      if (Array.isArray(g.genres)){
        g.genres.forEach((genre: string) => {
        genreCount[genre] = (genreCount[genre] || 0) + 1;
      });
    }
    });

Now we calculate the total for each genre, and create a vector - this vector normalizes genre counts by dividing with total count. This gives us 'user preference weights', and looks a bit like {RPG: 0.5, MMO: 0.2...}

const total = Object.values(genreCount).reduce((a,b) => a+b, 0);
    const userVector = Object.fromEntries(Object.entries(genreCount).map(([k,v]) => [k, v/total]));

With this data in place, we then create a gVector which assigns the value 1 to each genre present in the game. It then calculates the users' preference and assigns score:

const score = Object.keys(userVector).reduce(
        (sum, genre) => sum + (userVector[genre] || 0) * (gVector[genre] || 0),
        0
      );

Effectively, the score is the sum of the user's preference weights. This score could potentially be RPG:0.5 + Action: 0.2 = 0.7 for example.

This worked fine, and the scores worked as expected. However, there was a slight issue: Many of the games recommended, despite having the "correct" genres, were also very, very small indie games or even abandoned projects. From personal experience, I found that in order to engage a user in a game, a big community and a good reception are paramount.

So, I had to introduce a new parameter: Rating. Thankfully, steam store does provide us with ratings, so after a slight modification to my table, I was able to provide ratings to game metadata. Now my goal was the following: after assessing the genre score, add the rating score to the mix as well. However, there's a catch here too: some games have an IMMENSE amount of reviews, which completely skewed the weights. I balanced it out by using logarithms on the ratings themselves, and even new weights on the two parameters: genre rating and review rating:

return scored
    .filter(g => !g.isCompleted)
    .map(g => {
      const ratingVal = g.rating && g.rating > 0 ? Math.log(g.rating) / 5 : 0;
      return {
        ...g,
        combinedScore: (g.score * 0.6) + (ratingVal*0.4)
      }
    })
    .sort((a,b) => b.combinedScore - a.combinedScore)
    .slice(0, Number(amount) || 10);
  }

This, in turn, has produced great results but there is a lot of room for improvement. There is much more code I haven't mentioned here as I'm trying to keep this as concise as possible. In the future posts, I will dive into explaining the Frontend (what I have so far) but also future features I have planned for SBH.

I also am very welcoming to any sort of advice or feedback. Stay tuned, until next time :)