[SOLVED] Clustering/Scaling/General Questions

[manuel.gysin]

Good morning

 

I saw that there seems to be some knowhow in the forum about network code programming. So maybe I get some questions answered about scaling and possibility.

For our mmo project, maybe you saw it in the Showcase forum, I’m developing the network part. I took the NetworkViewer sample project as base for the understanding how raknet is used within unigine. In the meantime I setup the basic server/client stuff with object synchronization.

So far all is good, but at the stage now I have to solve some troubles before they occurs.

 

Clustering

We plan to run the whole mmo in one big virtual world. So the funny part is to split the whole world in sub worlds (nodes) and get them synchronized. While I do not like loading screens the nodes must be synchronized such a way, that a player can change between two nodes without any notice. That leads to a node to node synchronization with overlapping areas. Has someone implemented such a system? Theoretically it should not be a big deal, at the end it is an one way synchronization with master and slave nodes. The problem I see is, that I have to swap a player at a given time to the other node, which needs a new connection initialization. The world synchronization breaks in this moment, too the player gets out of sync. This should take 1-2 seconds, till the connection is back and stuff get back synchronized. But I’m not sure if I’m doing the right, when I solve the problem in this way. An other way is to start the connection as soon as the player is near a node swapping area, so he has two synchronizations at a time, and when he switches the node a flag is set to ignore synchronized object from the old node and take the new node as reference for the synchronization.

 

Performance/Scaling

Performance/Scaling on a server is too a very interesting part and it would be very nice if someone can provide some statistics about how much player you got on a node and how much objects you can sync within one node.

 

Cheating

A big problem with online games today is cheating. So a main goal is the keep calculations on the server and do only as much as needed on the client. How you solved this problem? Did you implemented an anti cheating software or just do normal controls within the virtual world?

 

 

Thank you very much in advance

Manuel

[alexei.garbuzenko]

Hey, Manuel

As I see, you are in the very beginning on the lo-o-ong MMO server development road. First of all, I recommend you to read some books and articles or huge server development, take a look at open-source MMO servers (especially RunUO), think of scalable architecture for multi-threading, etc.

 

About your questions:

1. Clustering

I'd suggest you to use external `controller` node that keeps connection alive and forward packets to one or another cluster node. On the other hand, "Loading screens" is not really a huge problem for MMO.

 

2. Performance

Performance depends on game logic type, architecture flexibility and hardware. I.e. our server middleware (Runserver) is expected to handle up to 5000 active players in MMORPG on Dual Intel Xeon 5130. That number was produced by observing CPU load with 2500 active connections - it was about 45% (averaged on four cores with separate DB and controller nodes). I really doubt that RakNet is able to do something like that - it is more network engine than server core.

 

3. Cheating

All calculations and game logic should be performed on server, use encrypted protocol, a lot of checks on server side for malformed packets, perform client integrity validation on start. It is also wise to leave a "backdoor" on client to scan active processes and loaded modules on server request.

[manuel.gysin]

Hey, Manuel

As I see, you are in the very beginning on the lo-o-ong MMO server development road.

Hi Alexei

Thank you very much for your answer! For sometime I found runserver and thought about to test it or get more information. But it was a bit complicated because I can not read Russian. But a mail will follow to runserver.

Yes, I know. 

 

First of all, I recommend you to read some books and articles or huge server development, take a look at open-source MMO servers (especially RunUO), think of scalable architecture for multi-threading, etc.

Do you know some good books? I’m searched around in some book shops but found nothing.

I will sure have a look at RunUO and some other open source projects.

 

1. Clustering

I'd suggest you to use external `controller` node that keeps connection alive and forward packets to one or another cluster node. On the other hand, "Loading screens" is not really a huge problem for MMO.

This is a good idea! I can develop something like a load balancer with extended functionality and connection management. We are a more “action” orientated mmo, so loading screens in the free world are a problem for us.

 

2. Performance

Performance depends on game logic type, architecture flexibility and hardware. I.e. our server middleware (Runserver) is expected to handle up to 5000 active players in MMORPG on Dual Intel Xeon 5130. That number was produced by observing CPU load with 2500 active connections - it was about 45% (averaged on four cores with separate DB and controller nodes). I really doubt that RakNet is able to do something like that - it is more network engine than server core.

That seems to be a good value. Thanks a lot for the information!

 

3. Cheating

All calculations and game logic should be performed on server, use encrypted protocol, a lot of checks on server side for malformed packets, perform client integrity validation on start. It is also wise to leave a "backdoor" on client to scan active processes and loaded modules on server request.

Thanks too for this very usefully information! I will take care of this.

[manuel.schmidt]

As being in the year 2013, you should think about splitting up your servers into two node types: Database Nodes and Application Nodes.

Database Nodes are private and can only be accessed by Application Nodes after Verifying an users request.
Application Nodes are horizontally scaling non-data nodes close to the database node for low latency. They can be scaled up or down dynamically based on the Network Load (for example DDos, which is popular in shutting down Online Game Services). I would use Amazon EC2 for these nodes.

Once you have done that, you might review if you really need multiple database nodes, based on the data size. You might be interested in Couchbase, an DB which scales well to multiple nodes as needed and already got built-in backup/restore routines. I have tested it myself to handle around 32.000 Small Operations per Second (READ or WRITE with 10 byte each) or still many hundred to a few thousand Operations per Second with 10kb each. I will use these in my own project.

 

One last sentence about MMORPGs. There have been quite a hype for sometime and everybody wanted to create his own MMORPG. Evaluate, if your team is big enough to create a game with all its needs for Content, Website, Programming, Marketing and so on. I've seen many projects failing in the last year because the under-estimated the amount of work or their team just wasnt big enough. Good Luck with your project anyways.