Multi-threaded Hercules

[pan 87]

Long time no see!

I've been dabbling with adding multi-thread support to Hercules for quite some time and even started quite a few server projects from scratch, and having failed in most of my attempts - last year I tried again with a new core design and it seems to be working so far. A lot of the systems that the server core is reliant were reworked, I believe some of these changes can even be used upstream but they would need to be revised first. This is mostly a proof of concept.

I'm still planning on finishing the map-server but as of now only login and char servers have multi-thread enabled - I didn't test with multiple users yet. There's no linux / FreeBSD support because I want to have everything in working condition before trying to add another I/O paradigm, the architecture is very reliant on the way that this module works so I thought it'd better to implement the whole server before trying to add other paradigms.

Lately I haven't got much free time to finish the map-server, but sometime later this year I think I can finish it.

The code is at https://github.com/panikon/HerculesMT , I forked from stable v2021.08.04

 

Server architecture

Hercules employs a multi-server approach in order to better load balance player requests, so for each game world there'll be at least three separate program instances running at all times. These instrances are: login-server, char-server and map-server. Each of which can only access a subset of the world's SQL tables and is responsible for a different step of the game experience.

Authentication and account database changes are segregated to the login-server, while character selection and character specific information are to the char-server, and the actual gameplay to the map-server.

The login-server is the player entry-point and can be responsible for multiple different world instances, so it's possible to use the same account table (login) for multiple worlds (char-server + map-server) and to dynamically change their IP without needing to push a new patch. After authentication the player is queried for which service they'll use and then the server relays the proper address so the client can connect.

In the char-server the player usually is queried again for a four digit PIN (PACKETVER >= 20180124), after which they can select which playable character will be used to play. The server then relays the address of the proper map-server to the client.

A single world instance can have multiple map-servers where each is responsible for a zone (set of maps), all inter map connectivity is managed by the char-server. When the player requests to change character the char-server IP is then relayed.
A single map-server can be responsible for multiple different zones, and each will be a different thread.

 

Relationship of each of the possible server instances

 Brown - Game world | White - program instances

 

Core design

All servers are multi-threaded. Basic server functions are assigned to different threads:

• Console (../src/common/console.c):
  This thread is responsible for all console input operations.
• Timer (../src/common/timer.c):
  The timer thread manages the `timer_heap` and dequeues timer actions from the `timer_queue`. Each timer can be ran in a different thread depending on the `timer_target`, this is accomplished by either queuing an action in one of the available action queues (`target_id`) or by running the specified function in the main timer loop (`do_timer`).
  Other threads queue different timer actions (add, delete, set tick) to the `timer_queue`.
  The timer module is also responsible for tick acquiral via (`timer->gettick` and `timer->gettick_nocache`).
• Message(../src/common/showmsg.c):
  The message thread is used to synchronize the output of multiple different threads, each call to `Show*` instead of being directly outputted to `STDOUT` is queued in the `showmsg_queue` and then dequeued by the main thread function `showmsg_worker`.
• I/O Worker(../src/common/socket.c):
  All I/O operations are asynchronous and implemented via I/O completion ports and a thread pool. The main workers are defined in `socket_worker` and after a operation is dequeued it's converted into an action (`socket_operation_process`) that is then queued to the proper action worker (each session has a responsible worker, and it's obtained by the I/O worker via `action->queue_get`).
  These workers don't execute any "business logic" so we can minimize the time that they are blocked by any given operation, and also so we can better isolate different session groups without having to take into account in I/O operations (e.g. in map-server each zone has a different action worker and multiple actions can be queued simultaneously without having to block any of the threads for a longer period of time).
• Action Worker(../src/common/action.c):
  Action workers perform all the "business logic" of each of the servers, and each is responsible for a different session group. After every action loop all send actions are queued to the completion port.
  
  
  • Login-server:
    Each char-server connection has a different worker. Player connections are randomly assigned.
  • Char-server:
    Each map-server connection has a different worker. Player connections are randomly assigned.
  • Map-server:
    Player connections are assigned depending on the current map according to the different configured zones. The char-server connection is randomly assigned.
    
    

Thread relationships

[Sephus 203]

It is very challenging to design and architect a multi-threaded application. You will need to make sure that your architecture doesn't result in any dead-locks, or that the code doesn't violate any critical sections and cause race conditions. It is also important to provide a clear understanding of why you need threading in certain areas of code when you are mutually excluding functionalities in threaded architectures. Sometimes people use threading without analyzing why they need multi-threaded performance in areas of the application. I believe that Hercules or eA should have been designed with a multi-threaded architecture from the start, since the current state of the project doesn't consider more than 1 thread and most features are designed to work on a single thread as well, making it more complicated to accurately structure and compose safe concurrent logical behaviour within the application.

 

My emulator follows a similar structure where the I/O is independent of the main application and multiple threads pass on the process from the networking infrastructure to the main component which splits the processing of logic performed on maps multiple maps per thread. I'm following a Hercules style of emulating in C++ and LUA. Maybe it can help you understand where concurrency is needed or if you wish to join in on the composition of a new emulator cleanly designed for modern day development, feel free to contact me on Discord.