This project is intended to provide a visual tool for playing the trading card game Magic: the Gathering with physical cards online with someone else.
There are different forms of online magic already, but this project would allow personal collections of cards to be used, without the difficulty of reading cards from another's webcam feed.
Instead of a fuzzy image of your opponent's board, this program creates a virtual playing board to easier view each card on their physical board.
- Download the entire repository either by zip or cloning
- Check what sets your cards are from and see if there is a .json in the "dicts" folder for every set in your pool of cards
- If additional sets are required, see "adding new sets" for information on addition more cards to the recognition pool
- A webcam is required for this project. It is recommended to have an external webcam to be aimed down towards a flat surface
- For best recognition, aim the webcam at a bright, flat surface to maximize the contrast between the border of the card and the background
A note on adding sets: "why not just add every set in existence so you don't have to worry about adding more?"
The recognition algorithm depends on hashing the reference images in each set and comparing it to hashes of the input.
There are many magic cards with unique art, so the more cards added will increase the likelihood of false positives.
Unless an excellent camera is used, the hash of your input could be even more similar to another card than its actual reference.
- When adding new sets, we will be working with "hashing.py" to crawl the webpages of Wizards of the Coast for individual sets of cards
- There are two arguments for running hashing.py, the name of the set and the url to the webpage of each set
- The name of the set is not entirely necessary, but its acts as a descriptor when removing unneeded sets in the future
- The url for the set can be founded by searching online by searching for "[name of set] card list" and clicking the link by Wizards of the Coast
- run hashing.py with the above two arguments to add all cards of that set to the reference pool. nothing else is required
- example: python3 hashing.py -a "Modern Horizons" -b "https://magic.wizards.com/en/products/modernhorizons/cards"
will add the "Modern Horizons.json" to the dicts folder, and all cards from the set will be added to the reference pool
- example: python3 hashing.py -a "Modern Horizons" -b "https://magic.wizards.com/en/products/modernhorizons/cards"
In its current state, this program only provides a one-way service where the host submits cards to the server and someone else can remote into the server to view
For both players to view each other's board, in the current state each player must have an instance of the program running locally.
To host, an additional tunneling program is required to project the local server.
For this project, localtunnel is recommended. Ngrok is not recommended because of its limit to connections often maxes out when adding new cards to the board.
- install localtunnel
npm install -g localtunnel - host on port 5000
lt --port 5000 - run card_detector.py to start the local server
pythyon3 card_detector.py - copy the url from localtunnel and send to your opponent for them to view your board
At first I had hoped to find a card detection algorithm online to adapt to my purposes, but I was unable to find anything suitable for what I wanted.
However, this program's card detection algorithm takes inspiration from Timo Ikonen's card detector, but is less involved in pre-processing to maintain speed for the user experience.
In addition, the scope of this project assumes more ideal conditions than he accounts for.
The Hashing approach is similar, but I ran into the issue of compiling the necessary references to run the algorithm on any card wanted.\
I first started with determining the regions of interest from the camera frame. No frame will be perfect, so the first step is to separate just the card from the image before hashing.
openCV is used for all operations on the frame and a few methods for finding the bounding rectangle (because every card is a rectangle), adjusting for rotation, and setting bounds for the size of the card all serve to separate the card from the frame.
Note: the program is better suited to handle slight counter-clockwise rotations, but not clockwise rotations
After separating the card from the image, a hash is created and the difference between this hash and every card in the reference pool is taken.
If the minimum difference is less than the tolerance for a match, then the corresponding reference image is assumed to be the matching card.
In the image below, the program is reading a local image and comparing the card's hash to that of a few reference images.
After detecting the card from a single image, the next step is detecting a card in each frame of a video feed.
openCV is used to open the camera and read in each frame. The same algorithm is applied to each individual frame, relating it to reference images.
Manually inputing every card for the reference set is unreasonable, so I built a web crawler to store every card's hash and name in a dictionary for every published set.
This separate program builds a dictionary for these sets, and must be called for every set of cards required to be added.
This approach also helps efficiency because instead of hashing every image in the reference set, those hashes are already created and can be referenced.
The below image is detecting the card "Timberland Guide" with a reference set of "Modern Horizons" and "Iconic Masters"
At first I believed that the program would take a frame from a playing board and build the entire virtual board every frame, so there would be no input from the user.
Thus, I developed multiple-card recognition in, but later realized that this approach would be too computationally intense, as well as unreasonable for extremely cluttered boards.
In the current state, the program requires the user to place their card underneath a camera, then add the card to their board.
Once the card is detected, I wanted to provide the user with any information about the card that they hoped for.
Thus, I integrated the program with the Scryfall API to request any information about the card.
In the current state, the client cannot view this information, but I plan to provide it in a future version with card interaction.
I have no previous experience with hosting a server, so I first attempted hosting the project with node.js
This approach was somewhat successful, but implementing the python program to dynamically update the board and current card proved challenging.
However, in the video below, once the card was confirmed from the server side, the card would be added to the virtual board, shown below the life feed.
I decided that flask would be more suitable for my purposes, as I am dealing with the images and variables directly from the python script.
In addition to the virtual board, I added a preview section of the card and add/delete buttons to manage the board state.
This is closer to the future goal of having each player build their board from a connection to the separate server.
The current board currently has no interacction, so a text field is used instead to delete particular cards from the board.
Note: in the video a number of false positives are found and there are multiple instances of the found card.
These issues are a product of two factors, the lighting and hosting on flask.
The accuracy of the detection relies on the images taken being close to the lighting of the reference images, so in this video the lighting was different and thus the accuracy diminished.
As for the multiple instances of the card, I believe that the frames where the card is present on the board are stored and waiting in queue to be processed.
Thus, while the first frame displays the found card, there are still a few more frames with the card yet to be processed.
I hope to have an interactive board for both players to use, so the experience is more fluid.
A few features of this interactive board are:
* Dragging the card images along a virtual board to rearrange them
* Clicking on a particular card to have a larger view and possibly some information about the card
* Displaying both boards with player names side-by-side so that both players can view their board as well as their opponent's
* Chat feature
* Graveyard and exile stacks
As this program is meant to allow players to play a game of Magic online, I hope to implement some of the features of a normal game.
These features include:
* A life counter
* Different sided die
* Counters for card effects
The end goal for this project is to have up to four (or potentially more) players join a unique room hosted by the server, and play with their virtual boards against one another.
Thus, one player could create a room and others would enter the room id to enter into the unique room and start a game.
This would require a large overhaul of the current system, and I'm not sure if flask would be sufficient to create this server as well as creating the interactive boards.
If need be, I can go back to node.js and try to get better communication between the server and the main program.
As this project is meant to work as entertainment for players, the interface needs improvement.
Once multi-player support is added, I will work on improving the visual interface to make the user experience more enjoyable.
Once all of the above improvements are added, I will return to the card detection algorithm.
While the card detection program is mostly accurate, there are definite improvements to be made because the raw difference between hashes can be quite volatile.
Thus, I hope to implement deep learning to train the algorithm to be closer to 100% accurate.
This would most likely mean adding every set of cards to the reference set, and if successful, would remove the need for adding or removing sets for accuracy.