Develop new asyncio networking backend

[ntoskrnl4]

The current backend Quarry is good for quick and simple server implementations and handles nearly all of the networking for the server. However, I think we could benefit from developing our own networking backend, based off of asyncio.

AsyncIO in Python is significantly more performant in a single thread than a similarly written synchronous server implementation and can match the performance of a threaded server, all without the overhead of using separate threads (thread-safety, overhead, limited access to resources, etc). Other advantages of splitting from Quarry is that we are not bound to any of its constructs; for example its server classes, its API format, and its version support.

Per the starting discussion on Discord, here's my thoughts about how asyncio networking could be implemented:

The networking thread will consist of its own process (to avoid GIL bottleneck w/ main thread), and will run its own asyncio event loop to do networking. The networking thread will start the server, and create a networking queue for events. The networking queue will take in events that happen relating to players (eg. skeleton shot you, block was removed, etc). These will use internal event objects (probably dataclasses) and the networking thread will sort them out as needed and dispatch events to individual players' queues, at which point the event loop will pick up the events on each player and actually write the packet out to them and stuff.

Then, when a new player connects, the networking thread will manage the entire process of server list ping and login and such, until the player has mostly logged in. Then it will send an internal "player join" event back to the main server thread, and the server will give it the appropriate chunks to load.

Incoming data from the player relating to the main server thread (such as world movement or whatever) will be sent back to the main server with an event queue for the main server thread.

Things that don't involve the main server thread at all will never get passed to it; I can only think of this being chat and server commands, but perhaps PvP could be in this category as well.

That's my initial thought as to how async networking could be done. It passes a lot of the server functions into it but unless the server has 200 players on it I don't think it'd be too bad. Then again, we could also go with a simpler implementation where only incoming data events and outgoing data events are handled, and everything is done in the main server thread. This should also be something we consider in addition to my thoughts above- it would probably make the overall server implementation simpler and easier at the cost of more work in the main server thread.

Either way we take this, I am down to support the development and oversight of this project on a new branch of the repo (for example named asyncio-networking), as I have almost two years of prior asyncio experience developing complex Discord bots.

[randomairborne]

Based on my understanding, there would be more then one main server thread, with seperate ones for chunk loading, generation, and similar. Would this affect networking?

[Geolykt]

We are likely going to use Queues for the communication between net threads and other threads, where there will be no affect with the multithreading.

[ntoskrnl4]

Based on my understanding, there would be more then one main server thread, with seperate ones for chunk loading, generation, and similar. Would this affect networking?

The current setup I have in mind is that networking will be its own, single, separate process which runs an asyncio loop to manage connections into the server. Once players successfully log in, the networking thread (well, process) communicates with the rest of the server using internal events and multiprocessing.Queues.

I'm not sure if we have the main server thread in place yet, but other server threads shouldn't have any affect on networking since that's in a completely different subprocess.

Management of Player objects can be done in either the networking thread or server thread, so that is yet to be decided depending on whichever makes it more performant or easier.

[barneygale]

For what it's worth, I'd be interested in an asyncio port of quarry itself. The vast majority of quarry's code doesn't use twisted, but I don't understand asyncio well enough to port the bits that do use twisted to asyncio.

[ntoskrnl4]

Hey, cool to see you interested in this!

Lack of asyncio is definitely what we saw as the main problem, hence this branch. Without asyncio, we're a fair bit more limited in how we can write the server in terms of server thread / network communication, since it's otherwise blocking.

I've been completely blindsided recently due to life events and schoolwork, but I should be free now to work on this more.

If you'd like, I can also attempt a port of quarry to asyncio, although I personally am much less familiar with Twisted.

[barneygale]

I'll study your implementation and see if I can glean some understanding of asyncio. I previously gave up on asyncio when I read this. But it's time for another look!

You may still find the quarry.types and quarry.data packages useful, as they don't include any twisted stuff.

[barneygale]

I think I've got my head around asyncio, and I have a simple implementation of the Minecraft protocol. Here's a working server:

import asyncio

from mncrft.types import unpack_varint
from mncrft.server import listen

def status(version):
    return {'version': {'name': '1.16.4', 'protocol': 754},
            'players': {'max': 100, 'online': 0},
            'description': {'text': 'blah'}}


async def login(protocol, profile, version):
    while True:
        buff = await protocol.read_packet()
        ident = unpack_varint(buff)
        print(ident)        
        # etc


async def main():
    server = await listen('127.0.0.1', 25565, status, login)
    async with server:
        await server.serve_forever()


asyncio.run(main())

Code coming soon.

[barneygale]

Here we go: https://github.com/barneygale/quarry-ng

Using minecraft-data for packet structs now, too.

May change a fair bit as I build on it. Let me know how you get on!