Building a Video Games Recommender System with Milvus, FastAPI, and Docker
Published in
13 min readJan 29, 2024
Recommender systems play a crucial role in providing personalized content recommendations to users. In this article, we will explore the process of building a movie recommender system using state-of-the-art natural language processing techniques, scalable vector databases, and containerization. Our goal is to extract feature vectors from movie metadata, store them efficiently, and provide users with personalized movie recommendations.
All from one notebook!
You can use any dataset you want, for this tutorial, I will be using a steam games dataset obtained from kaggle.
The secret sauce making all this possible is SentenceTransformers. It’s like a magic tool that can understand the essence of sentences or paragraphs. It can take a piece of text and transform it into a unique representation (vector) that captures its meaning.
Think of games metadata as descriptions or summaries of games. SentenceTransformers is used to convert these descriptions into special numbers (vectors). Each game gets its unique set of numbers that somehow represents its “game personality.”
These game vectors create a magical “vector space” where similar games are closer together. It’s like grouping games based on their personalities. The closer two games are in this space, the more alike they are in terms of their descriptions.
First, we begin by loading our dataset:
After, cleansing the data, we load our transformer, and convert our dataset into embeddings:
transformer = SentenceTransformer("all-MiniLM-L6-v2")
def embed_game(data):
# Add the data you want to be embedded
embed = "{} Released on {}. Genres are {}. The price is {}. The Single Player's Main story takes {} hours to beat".format(
data["full_desc"],
data["date"],
data["genres"],
data["price"],
data["Single-Player_Main Story_Average"],
)
embeddings = transformer.encode(embed)
return embeddingsAfter we embedded the data, it’s time to upload it to our cloud vector database.
What makes vector databases so special?
Vector databases, such as Milvus, are employed because they excel at storing and searching high-dimensional vectors efficiently. In the context of a recommender system, SentenceTransformers generates feature vectors representing the semantic content of movie metadata. These vectors, serving as unique fingerprints for each game, are stored in Milvus.
Milvus leverages advanced indexing techniques to enable rapid similarity searches. When a user submits information about games they’re interested in, Milvus quickly identifies similar games based on vector proximity. This approach significantly speeds up the recommendation process, providing a powerful and scalable solution for handling large volumes of game data.
Let’s create a schema for our database:
id = FieldSchema(name="name", dtype=DataType.VARCHAR, max_length=500, is_primary=True)
field = FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=384)
schema = CollectionSchema(
fields=[id, field],
description="steam games recommender: game vectors",
enable_dynamic_field=True,
)
if utility.has_collection(COLLECTION_NAME): # For testing: drop the same collection created before
collection = Collection(COLLECTION_NAME)
collection.drop()
collection = Collection(name=COLLECTION_NAME, schema=schema)
print("Collection created.")
index_params = {
"index_type": "IVF_FLAT",
"metric_type": "L2",
"params": {"nlist": 128},
}
collection.create_index(field_name="embedding", index_params=index_params)
collection.load()
print("Collection indexed!")Then we convert our dataset into a dict and feed our embedded data onto Milvus’ cloud db in batches:
games_dict = df.replace(np.nan, "Unknown").to_dict("records")
# It doesn't accept nan values
j = 0
batch = []
for game_dict in tqdm(games_dict):
try:
game_dict["embedding"] = embed_game(game_dict)
batch.append(game_dict)
j += 1
if j % 5 == 0:
print("Embedded {} records".format(j))
collection.insert(batch)
print("Batch insert completed")
batch = []
except Exception as e:
print("Error inserting record {}".format(e))
pprint(batch)
break
collection.insert(game_dict)
print("Final batch completed")
print("Finished with {} embeddings".format(j))Now we create our functions for retrieving our top k results:
collection.load() # load collection memory before search
# Set search parameters
topK = 5
SEARCH_PARAM = {
"metric_type": "L2",
"params": {"nprobe": 20},
}
def embed_search(search_string):
search_embeddings = transformer.encode(search_string)
return search_embeddings
def search_for_games(search_string):
user_vector = embed_search(search_string)
return collection.search(
[user_vector],
"embedding",
param=SEARCH_PARAM,
limit=topK,
expr=None,
output_fields=["name", "full_desc"],
)
def convert_hits_json(hits_list):
hits = hits_list[0]
# Extract the list of dictionaries from the Hits object
games_info_list = [hit.to_dict() for hit in hits]
games_info_str = json.dumps(games_info_list)
games_info = json.loads(games_info_str)
return games_infoThat’s it! It’s that simple.
Now let’s test it:
search_string = "A football sports game."
results = search_for_games(search_string)
for game_info in results:
title = game_info["entity"]["name"]
overview = game_info["entity"]["full_desc"]
print(f"Title: {title}")
print(f"Overview: {overview}")
print("-----")Output:
Title: Draft Day Sports: Pro Football 2020
Overview: About This GameDraft Day Sports: Pro Football 2020 puts you in control of your favorite pro football franchise. You make the calls as you build your dynasty – build your roster through trades, the draft, and free agency. Analyze the impressive array of data to determine how to put together your own custom playbook and strategies to lead your team to victory. Even show off your coaching knowledge by creating your own plays! Watch the action unfold in dramatic 2D fashion where you can take control of the play calling and watch your calls play out in front of you. Play by yourself against a challenging AI or join an online multiplayer league and see if you have what it takes to outmanage your fellow gamers.
-----
Title: Behold the Kickmen
Overview: About This GameBEHOLD THE KICKMEN started as a silly Twitter joke, and evolved like a beautiful flower into the year's least-exciting and mostly-incorrect Football Simulation. It's football, as seen through the eyes of someone with no understanding of, or interest in, the Laws of Football.Start at the bottom of the Big Boring British Football Spreadsheet system and grind your way up, gruelling match after gruelling match, to become the best at the football that anyone's ever seen, and win the World Cup for your mantelpiece.FEATURES: Kicking, Passing, Tackling, Squad Management and Doing Goals over and over. There's even a full Story mode, crammed with overblown melodrama and inane decisions to be made.TWO WAYS TO PLAY! Hate football? No problem! The entire game can be re-skinned at the flip of a switch into a Dystopian Future Bloodsport!PLEASE NOTE: Kickmen has NO ONLINE MULTIPLAYER and NO LOCAL MULTIPLAYER, it's Single Player ONLY. Please don't buy it thinking it's at all sociable, because it isn't. Why, you ask? Because I wanted to make a Single Player game, I didn't want to make a Multiplayer game.
-----
Title: Mutant Football League
Overview: About This GameMutants and monsters unleash maniacal carnage on the gridiron in THIS action packed, arcade-style football game. Mutant Football League delivers intense online multiplayer action, deep strategy and deadly humor.Fantastical arenas full of ruthless fans and deadly traps set the stage for mayhem in every contest that allows players to win by highest score or by killing and eradicating their opponents - a feat made easier as teams use their unique “Dirty Tricks” such as “Bribe the Ref” to overturn a call or having a player become “Ginormous” to grow in size and pummel the puny combatants in their path. Whenever a game begins you’ll quickly discover the only predictable element is broken bones.Mutant Football League is the bloody-fun, next-gen, successor to arcade-style American football games, taking inspiration from over-the-top sports games such as NFL Blitz and the Sega Genesis classic, Mutant League Football. Digital Dreams' MFL game delivers deep strategic 7-on-7 gameplay, with bone-crunching, tooth-shattering, head splitting hits, and a roster of comical ghouls delivering tongue-through-cheek humor. Oh, and blood. Lots of blood.Two player online or four player local MULTIPLAYER gives you the choice to clobber one another or team up and crush opposing miscreants.Choose from 18 TEAMS made up of blood thirsty, wisecracking Skeletal Deadheads, Armored Bruiserbots, Criminal Aliens, Mutated Humans and Monstrous Orcs.NBA Jam and NFL Blitz hall of fame commentator, Tim Kitzrow gives unforgettable, adult-oriented play-by play commentary for every game.Use Dirty Tricks during game play to Bribe the Ref, throw Bombs, or equip Shotguns and Chainsaws to your players to inflict maximum carnage against the unsuspecting competition for real difference-making moments!18 unique fields across the league offer landmines, booby traps, fire pits, toxic moats and buzz saws as fun but deadly obstacles. Play as MFL All-Stars Wham Neutron, Bomb Shady, Ghoulio Bones, and more who dominate the league with their rising kill counts and ferocious abilities.Multiple punishing play modes like EXHIBITION, PRACTICE, PLAYOFFS, or try to survive a FULL SEASON and reach the ultimate glory: THE MAYHEM BOWL!Make the most of POST-PLAY MAYHEM by bludgeoning your opponents with brass knuckles, body slams, and rib-crushing kicks after the whistle blowsExploding body parts, comically horrific deaths, and blood splatters fuel the MFL GORE FX visual experience.
-----
Title: Front Office Football Seven
Overview: About This GamePut Yourself in the Front OfficeFront Office Football is a text-based sports simulation. It's a game for those of us who love the numbers in sports.In Front Office Football, you play the role of your favorite team's general manager. You determine your team's future through trading with opponents, negotiating contracts, bidding for free agents and discovering new talent through the annual amateur draft.You can also play the role of the armchair coach, setting game plans and depth charts, or even calling each play from an extensive playbook.You can determine ticket prices and submit stadium construction plans for public approval. You can move your team if the public won't properly support your franchise.Front Office Football Seven includes a multi-player career mode with full ftp support, allowing you to compete in a league with up to 31 other enthusiastic general managers. Multi-player mode runs independently of Steam and requires a commissioner to manage the web site.The original game, released in 1998, received an Editors' Choice award from Computer Gaming World and a 4 1/2-star review. It was nominated for numerous Sports Game of the Year awards. This is the Seventh full version.Front Office Football Seven features a 2015 player file and new careers with the default files begin in 2015 with the 2015 schedule. We're keeping this product on sale at half price to promote the series. For the latest in the series, Front Office Football Eight is available and features a 2018 player file and schedule.Your Goals in Front Office FootballIn order to succeed in Front Office Football, you need to perform as well as possible in four different areas.Team Performance. On the field, your primary goal is winning the coveted Front Office Bowl. Your fans, players and staff all want to see that championship banner raised to a new position in the ring of honor surrounding your stadium.Financial Performance. Off the field, your team needs to show a profit, or the owner will become angry and threaten your job. You need to control salary and staff costs while balancing the need to spend money to build and upgrade your stadium against the risk of facing stagnant ticket revenue with an aging arena.Roster Value. You need to negotiate contracts, sign free agents, make wise decisions in the amateur draft and outsmart opposing general managers in trade. Building a strong, capable roster means everything in Front Office Football.Franchise Value. The bottom line is that a happy owner has a franchise that's the envy of professional football. Nothing means more to the owners than seeing their franchise on the top of the list of most valuable franchises. You help put your team on that list by excelling in the three other categories, but the best general managers look for opportunities to move the team in order to find a home town with a strong economy that will support your team like none other.Major Features of Front Office FootballThe game concentrates on roster management and career play. There are several key elements emphasized in the game design:A realistic trading module. You can't simply take the players you want from other teams.Proper aging of players. Players at different positions age differently. Quarterbacks need a couple of more years to reach their prime, but their careers last several years longer, on average, than running backs.The amateur draft. Teams realistically assess their needs, and build through the draft.Statistics. All the major stats are tracked and are available at any given time. Career and full season-by-season statistics are tracked in 135 different categories, including Red Zone and Third Down numbers. You can view and sort statistics by team, category and position. It's fast and accurate. Front Office Football also tracks and displays 182 different team statistics and league totals.Play calling. Designed to allow quick selection of a large library of players, you can be the ultimate GM and wrest control of the play-by-play action from your coach. You can tailor your in-game strategy to your team's strengths without having to build each play from scratch.Free agency. Teams compete with you to sign the best free agents. Each player has his own idea of how much he wants to stay with his existing team, and how much he wants to play for a champion. But money is still at the root of all decisions.Home towns. Each player will have a home town from nearly 10,000 American cities. When deciding on teams during free agency, players may prefer a team closer to home.Depth charts. You set the depth chart at each position for your team, and fill out a play preference chart. Front Office Football simulates games based on these charts. You can choose different personnel depending on your choice of formations.Game plans. The game plan will allow you to specify different strategies depending on the score of the game and how much time remains. There are literally thousands of choices to make, or you can leave everything to your coaching staff.Player ratings. Each player is rated for 53 different skills. But you don't have access to the raw numbers. Where's the fun in that? You hire a coaching staff, with varied strengths and weaknesses. Your coaches tell you how good they think your players are - and how good they think your opponents' players are.The salary cap. It's an essential tool in keeping parity among professional football rosters. You'll have to cut your aging, high-paid veterans just like any ruthless general manager worth his weight in negotiations.City profiles. Submit a plan to build a new stadium to your voters. If they turn you down, you can propose a move to any of 169 cities modeled in the game. Each city is rated for several economic criteria, which affects its desire for a new team.Team chemistry. Players will perform better or worse in some instances, depending on how they feel about players in their group.Dynamic Quarterback learning process. As quarterbacks learn more about the game, they will have access to more plays during games, allowing the smarter signal-callers to better confuse their opponents.Record keeping. All team statistics are tracked for a manager's entire career. Team records, including all-time performance against every other team, are kept. A game-by-game performance breakdown is always available for individual players.Power ratings. You can see how your team ranks using Solecismic Software's custom power rankings. These ratings are used to set a point spread for each game.Enhanced replay value. Every time you start a new career, the core ratings for each player are randomly affected. For veterans, the random change will be very small. Established stars will always be significant players. For rookies, however, performance will vary significantly. This allows for a more challenging game and greater replay value.Multi-Player League Support. Choose a commissioner to run the games for your league. Your commissioner will simulate the games and process every team's instruction set for each stage during the game. Up to 32 people can compete in each multi-player league.Thanks for taking a look at Front Office Football.
-----
Title: Axis Football 2015
Overview: About This GameAxis Football OverviewAxis Football is a high-quality, 3D American Football Game. Choose from 32 team and play in either season mode or quick match against AI or local multiplayer. A great alternative to Madden, Axis Football features excellent controls and a unique aimed passing system that gives unprecedented accuracy and control of where the ball is thrown. Unlike the few other football games, Axis Football was specifically designed for play on a computer and takes full advantage of the mouse/keyboard controls as well as controllers.Teams and RostersAll 32 cities from the NFL are represented in Axis Football. Each team features its own ranking in passing, running, pass defense, run defense, and special teams. Additionally, each player in the game comes complete with their own stats for all of the various football metrics (speed, tackling, blocking, etc.). The names, numbers, and races of all players are stored in external files allowing for complete customization/modification of rosters. Complete mod support will be added soon!Axis Football 2015 Features and UpdatesSeason Mode with different schedules each year and season-long stats!Coach Mode Call the plays and watch the action! Available in both quick match and season mode!Controller Support. You can now use your XBox 360 controller to play!2-Player, Local Multiplayer. Using two controllersPlayer Mod Support You have control over the name, number, and race of each player in the gameUniform Mod Support Have complete control over each team's home and away uniformsSteam Achievements.Steam Trading Cards.I know Axis Football doesn't have everything that you want it to have - yet! It's going to take some time to get all of the features that you have come to expect. I'm asking for your support in helping build a football game franchise that gives players options. I'm in this for the long-haul. If you decide not to purchase the game, I ask that you at least mention it to a friend!
-----That seems about right.
Now the fun part, let’s convert this into an API microservice using FastAPI.
@api_router.post("/search")
async def search_game(search_input: SearchInput):
games_list = search_for_games(search_input.query)
games_json = convert_hits_json(games_list)
return games_jsonNow let’s see our api docs:
We can send a request from our api docs or via curl:
Perfect! It’s working like a charm.
Now we just have to containerize it via Docker Compose. Docker Compose is a tool for defining and running multi-container Docker applications.
It’s necessary if we also want a front end. That way we’ll need two Docker files, one for the front-end, and one for the back-end.
The Dockerfile for the back-end api:
FROM python:3.11.4-slim-bullseye as prod
RUN pip install poetry==1.4.2
# Configuring poetry
RUN poetry config virtualenvs.create false
# Copying requirements of a project
COPY pyproject.toml poetry.lock /app/src/
WORKDIR /app/src
# Installing requirements
RUN poetry install --only main
# Copying actuall application
COPY . /app/src/
RUN poetry install --only main
CMD ["/usr/local/bin/python", "-m", "recommender_api"]
FROM prod as dev
RUN poetry installNow our Docker compose file will look like this:
version: '3.9'
services:
api:
ports:
# Exposes application port.
- "8000:8000"
build:
target: dev
volumes:
# Adds current directory as volume.
- .:/app/src/
environment:
# Enables autoreload.
RECOMMENDER_API_RELOAD: "True"If we had a front end, we’d just simply add it to the services alongside the api service.
Let’s run it via docker compose:
docker-compose -f deploy/docker-compose.yml --project-directory . up --build
And that’s it! That’s all you need to get started into building a vector similarity-based recommender system.
The full code for this project can be found on my GitHub.
Stackademic
Thank you for reading until the end. Before you go:
