If you want to discuss anything on these blogs, create a post on our forums
and I will reply when I can, also please note at this current time most of these are just ideas and might not get added.
And we are back.
Post by TheMightyDude, Sat, October 13, 2018
Development Planning Server Linux
Been a while since my last development blog, I did post a couple of blogs which was lost when we had a Server Hardware failure where one of our SSD Drives died, sadly we were running these SSD Drives in RAID 0 (Disk Striping) so we lost absolutely everything.
We did use to do backups, sadly the last one was done over a month ago, so anything we did within that time was lost, so all my main new code within that time for Blacknova Realms was lost. I do have different versions here locally, just not an up to date version due to I was editing and compiling code remotely.
Sooooo, I now need to see how old the local versions are and work from that, I know I left some sticky notes here in the change of direction on the code.
Server Console - Screen Buffer
Post by TheMightyDude, Thu, June 14, 2018
Development Planning Feature Server Linux
Well been looking into the server side of things so it will be written for a Linux Server.
I am aiming to have the server application as a console-based application so that the admin can run it via a remote terminal.
So, I really want it to display a title, body, footer and a section where you enter admin commands.
So, I had to create my own Screen Buffer code which allows me to create segments on the screen and display information on just that segment.
Sure, I could have looked for an already created one and hope for it to be free, but then I have to hope that developer maintains it, I have been burnt too may times "ADODB" etc.
Granted its not perfect, it might have some issues, but I haven't seen any yet.
Network Update 6
Post by TheMightyDude, Sat, April 14, 2018
Development Planning Feature
Well the new version of the Network Library is starting to look rather promising so far, I have got it to a stage where it was when I was adding support for different METs (Message Event Type) like ConnectEvent, DisconnectEvent, DataEvent, ServerNoticeEvent etc.
So far, the very basic server network code is about 40KB in size and the client network code is also about 40KB in size.
So, I am currently working on the MessageManager which works out the type of message and sends it off to the required handler and queues it up.
Each handler processes their own queue on its own thread so that it doesn't slow anything game wise down, so all this is done Asynchronously.
Once this Message Manager is complete I will start on the following Stages:
Stage 1 - Authentication Security Feature
Add support for SRP (Secure Remote Password) where the client can authenticate with the server without sending any passwords, basically the user enters their username and password, but only the username and loads of random hashes are sent back and forth over a few stages where the server knows that the client knows the password and the client knows that the server also knows it.
Stage 2 - Add Message Support
Add the support for Authentication in the MessageManager by adding an Authentication Handler.
So, users at this point should be able to log into and out of the empty SandBox.
Stage 3 - Server Notification Messages Support
I really need to add support for this early on, I could also use this as feedback debug message which in development.
That's it for now.
Network Update 5
Post by TheMightyDude, Thu, March 22, 2018
Development Planning Issues
Well I seem to of hit a brick wall again to do with the Networking code that I have been creating.
It seems that the way that I have designed it makes it a tad bit difficult to do some lookup stuff.
And having to go through loads of code which needs to happen every threaded network loop, this is causing a CPU performance hit which is impacting the number of connected users that the game would be able to support.
So, I have once again stepped back to see what every part needs to do and I think I might have it this time.
I know the Threading side of the code is fine, so it just falls down to the actual networking side that is the issue.
It seems that I might have just over complicated things with my code an that simplifying it might end up being better.
But we will see.
Data Storage [server side]
Post by TheMightyDude, Wed, January 17, 2018
I am sorry I have neglected this blog section for a while, TBH I had forgotten about it due to on and off ill days.
I had been working on Octree and Quadtree stuff last year, but where its going to be like the current BNT for this version there was no need to use those.
I will be using HashSet<...> to store the information which does take a little more memory than actual List<...> but the lookup is supposed to be faster when using HashSet<...> so I will try those.
At the moment the default storage size for a Universe consisting of 2,500 Sectors is currently taking about 36 MB of RAM and 35,000 Sectors takes about 53 MB of RAM.
Out of curiosity I tried 100,000 Sectors an it took about 94 MB of RAM, so not too bad.
I am also still working on the Database side of things, changing stuff here and there.
I will try to add more stuff here.
Planting an Octree [server module]
Post by TheMightyDude, Tue, July 25, 2017
Well I have been looking into Octree's for the last couple of weeks now and I think I am starting to understand it.
So, I will be using Octree's to store all objects that have a 3D location in space like Ships, Stations, Asteroids, Planets, Moons and so on.
Hopefully this will all help to reduce all the database calls etc due to its all loaded in RAM at runtime and we only update the database if anything changes in RAM and which it checks every X mins (15 mins by default).
Apart from that not much has happened.
Network Server Update [small update]
Post by TheMightyDude, Mon, July 03, 2017
Another short update...
Cleaned up some of the Network Server code, removed redundant code.
Added in support for message handlers, so now we can add in different message handlers, like PING, Chat Messages, Movement, Combat and so on.
Also optimized the network code to do stuff when it's really required to do so.
Put in place holders for future stuff.
Well that's it for now, well I did say it was a short update.
Network Update 4
Post by TheMightyDude, Thu, June 15, 2017
Well, due to not touching the code for a while because of RL stuff, most of the code was stopped in mid coding, so wouldn't compile.
So, I decided on stepping back to see what I really wanted the Networking code to do.
I have now got the Client and Server Network Code to both support Multiple Threads, so it can now support loads of Threads for the NetworkLoop, ReadQueue, WriteQueue, so those sections should never bog down the rest of the server's code.
Using my own Threads takes the Networking Side of the code away from the rest, i.e. Unity for the Client side and the Linux Server code.
A lot of the stuff is hardcoded in, so that will need to be changed.
At this point of time I am not sure on what exact amount of CCU (Concurrent Connected Users) this code will be able to handle, but I am sure we won't be reaching it for a while, I am aiming for 1,000 to 2,000 CCU players, but we will see how it goes.
And then look into creating Node Servers, which will be a learning curve for me.
So, so far the Client can connect to the Server, and they can send and receive Network Messages (formatted data) back and forth, this data is meaningless at this stage, but both ends can extract the individual messages and process them, so it is still work in progress.
Back on Track (sort of)
Post by TheMightyDude, Wed, June 14, 2017
Sorry for being a bit A.W.O.L, health issues happened again to-do with the hot weather here at the moment in the UK, as well as several Hospital Appointments going on all over the place.
I am still looking into different ways to re-write Blacknova Traders.
Like I have already said before on the forums, it being a browser based game really does limit what you can add to the game due to it would be a Request and Response type game, so the game wouldn't be responsive enough to add all the features I wanted to.
Now a full-blown TCP or UDP Socketed type game would allow me to add all the features that I wanted plus more and also makes it more future proof.
Now the status of development and the Dev Team, as far as I am aware we haven't stopped work on the new game, whether it will be just a complete recode from scratch or a completely new concept based on BNT, we have yet to discuss this, this is however possibly due to start sometime next year.
Network Update 3
Post by TheMightyDude, Mon, March 20, 2017
Well I have just done some more of the Client and Server Network Code, so we can now send and receive basic hardcoded Network Messages in a non-secure manner.
It's still not perfect, but we can send and receive stuff now :)
I think some of the code may need to be changed to support the Secure SSL / TLS Mode, but I assumed that would be the case.
Apart from that, not much has gone on.
Network Update 2
Post by TheMightyDude, Thu, March 16, 2017
Just a short quick update this week.
I have got about 60% of the Client Network code done so far, the remaining 40% should get done when I complete the last parts of the Server Network code.
As for the last blog "Network Update 1" to-do with the secure side, I now have two options, have it like I have said using HTTPS for just the secure side like logging in, or have a separate service for just the authentication stuff and services for the actual Game Lobby (Server List / Character Select) and the actual Game Server.
I might combine the Game Lobby Service with the Authentication Service for simplicity.
So, to confirm what's "hopefully" going to happen...
1: Create a Secure (SSL / TLS) Service for Authentication and Lobby area.
2: Create a Service for the actual Game.
3: Create a Client that first connects to Authentication Service and if valid login send Lobby Information.
4: If Valid login and Game Service selected in the Lobby Service, connect to selected Game Service.
So it's one step closer :)
Network Update 1
Post by TheMightyDude, Sat, March 11, 2017
Well a bit late adding this blog, been having some bad days, so haven't been in a mood to do coding.
I am still working on the Networking side of the code where I have had to code from scratch a TCP Server Code that "should" handle thousands of connected users using multiple threads.
I have started this code just as a backup in case the NetworkTransport which is built into the Unity Engine has any limitations or any future restrictions like them requiring a subscription to use their networking library.
Plus, in my code I have more control over how it works.
I am currently working on its Message Queuing System which is about halfway done atm.
I am still on the fence whether to use SSL/TLS over TCP Sockets for everything other than just for authentication.
The issue with using SSL/TLS over a TCP Socket is that once you start it you have to keep using it and using SSL/TLS for every network message would impact the server resulting in supporting less connected players.
So, if we want to only use SSL/TLS for authentication we would have to use Secure WebSockets (i.e. HTTPS) for the handling all of the logging in and normal TCP Sockets for the actual game.
Also, using Secure WebSockets it makes it easier to allow for 3rd party servers if I decide to release the server binaries (Linux using Mono) later on, leaving just the Authentication side being run by us.
So, to answer some of your questions that you sent via email, yes, you create your account on our servers and you then use the Game Client to connection to our network where you will / should be presented with a list of servers to play on.
Game Servers will not store any private information like email address, passwords etc, only we store that encrypted.
The Game Servers request for a hash which is created just for that server, so a player can play on different servers and have all different reference in that server's database.
Once this Server Code is complete I will be working on the Client Network Code and once that has completed I would be at a point where I can start adding game stuff in.
Well that's it for this week.
Post by TheMightyDude, Mon, February 27, 2017
Space to planet surface transitioning
This will allow for the player controlling his ship to fly about (like Elite, No Mans Sky) and then head for a planet and fly through the atmosphere of the planets right down to the planet surface where you can fly and land.
This will be all done at run time, we are still trying to get this to work, its partly done.
Now here is the killing blow, players will not own planets, end of.
There is nothing stopping players build bases, structures on a planet, but planets are mostly there for resource gathering and some combat missions.
Post by TheMightyDude, Mon, February 27, 2017
I thought I would give another update on how things are going.
At the moment things are going rather well now, but I did have some issues a few days ago, when I tried to implement database layer to the server code.
Well as I said in my last development blog I was going to use Unity's LLAPI (Low Level API) Networking which uses UDP Sockets which means fast transferring of network traffic.
So, that was going very well so I thought ok, now to add the database side of things to the server code, now this is where things went screwy, I downloaded MySQL Connector for Mono & .NET which Unity imported fine, now when I tried to use it I got some errors to do with missing references so I resolved that issue to then hit another.
I was almost ready to just give up on everything when I started playing about where I then got MySQL working fine, I thought great, but what other Databases it could use, so at least I would have a choice which one to use if the other proved problematic down the line.
So, after several hours faffing about I can use either MySQL or SQLite.
There are some pros and cons with both of them.
The database is only used for the initial loading of contents into physical memory and the server then only uses the content stored in memory instead of doing loads of queries resulting in a HUGE performance boost.
The server updates the database every X (15 mins by default) mins with the live data it its own Thread so that it doesn't impact the server's performance.
This is how most MMO's do it, so we are on the right track here :)
Long delayed update...
Post by TheMightyDude, Mon, February 20, 2017
Well I started work on my Network Library a couple of weeks ago, that will be used for all the WebRequests which used Threads, Message Queues etc, this seemed to work great.
This was around the time I got an Eye Infection, so there was me trying to use my PC with a one of my eyes completely blood shot and un able to focus onto anything without pain, so this slowed me down somewhat, hence me not being online much.
Anyway, I thought I would add the next stuff to the library which was the background requests that get called every X seconds for any updates that might have happened between the player clicking on stuff.
This is to support features that notify users that haven't clicked on anything for a while, like for example a friend logs in and also when you are being scanned and or attacked, it will pop up a notification window or change to a combat scene when being attacked.
Anyway, I added in this background requests which seemed to work ok, well until I simulated more and more players playing and also running the background requests, I actually managed to kill Apache twice, MySQL wasn't too happy either but I could resolve that with SQL changes.
Which leaves me to the Apache Issue...
I really want these features to be added, sadly if it's going to result in Apache locking up and crashing due to an over load in requests per second, then I am then going to have to find another route to take.
I am already currently using the Unity Engine for the client side application on the PC and later on Linux, Mac and possibly Tablets depending on the screen resolution.
Anyway, since Unity is being used for the game client I thought why not write the server using Unity as well.
So, I started coding another Network Library from scratch, oh what fun that was, Threads not shutting down correctly resulting in me creating a Thread Manager to manage all the used Threads and so on.
I got about a third of the way through it when I thought about looking at the built-in versions of networking that come built into Unity.
So, I checked the easiest version first, had too many restrictions like the maximum amount of ccu (concurrent connected users) players and that was very low, so I checked the new HLAPI (High Level API) was is still being worked on, sadly like the first it also has the same ccu restrictions and to overcome those restrictions I would have to charge a monthly rate to pay for those services.
But due to not really wanting to make the game a subscription based game to cover the costs and also wasn't really too sure on how many people will even play the game to cover the costs, then looked into the LLAPI (Low Level API) which also uses UDP Packets just like the other types that I tried before this one. But this one you have no restrictions as far as I can tell so far, the only restrictions I can see are there will be loads more coding to be done which isn't too bad and the maximum ccu was 65,535 a lot better than the 20 odd that you get on the other two.
Now if I was ever lucky enough to have a game that had that number of connected players I would probably have some income whether it's by micro-transactions or nicely placed ads etc, I would just setup another server to run a new universe.
So here is the gotcha, the only people that would be able to setup a server would need to have root access and own the entire root server, so no cheap GoDaday hosts.
I know this isn't what you really wanted to hear, but I really did try the easy routes first, sadly this doesn't work out how we wanted it, to be honest I am glad I found all this out now then a year or two down the line.
I will try and keep this game as close to as possible as the current BNT, but "some" stuff will be changed, we have to be realistic here, there is no way in hell in you getting millions for fighters in your little ship: P
Well I think this individual blog has gone on long enough this time, so I will be cutting it here.
Like as always, if you have any questions, just email me or post on the forums.
Post by TheMightyDude, Thu, January 26, 2017
Well had to redo all my JSON Serialization code due to compatibility issue, which worked fine on Windows but had issues on Linux, Mac etc.
So, I thought I would try the built-in version, which works, however it requires loads more code to work and wasn't compatible with what I already had coded so far.
So, for the Client to send a request to the server in a JSON formatted request, I have to create a class container which is more work to do, but it works and doesn't require a 3rd party library.
So, like I said before I will be creating a Windows version first and then try working on the other platforms (Linux, Mac, Tablets), sadly Smart Phones will not be supported due to the small screen size.
Post by TheMightyDude, Thu, January 12, 2017
Well not done much as late apart from working on JSON Serialization which will be used for all of the Locale (English only for now) and Configuration files, its not perfect but it works.
JSON Serialization is also going to be used to format the information between the Client and Server.
Like I said I haven't really done much since the last blog.
Hopefully more will get done over the next few weeks.
Just a quick update.
Post by TheMightyDude, Mon, December 26, 2016
Just thought I would say Merry Christmas to everyone.
Ok, I have been doing some huge thinking on ways to do this game and have come up with this.
And if we are support loads of players we are going to have to reduce the amount of MySQL Queries, so we will need to do some caching of some sort of our database records.
I looked into Memcache, Memcahed, MySQL Caching and so on, all having limitations with either the amount of data as well as the size of each data, so it seems that caching is out of the question here.
I also looked at MySQL's Memory Engine which by reading it sounded exactly what I wanted, sadly the performance was either the same as the InnoDB Engine or it was slightly slower which I thought was strange, however I may come back to try this again later on.
Now knowing the above I started looking into tmpfs and ramfs for a type of caching and for simplicity I chose tmpfs.
Using tmpfs will allow me to create a filesystem in a chunk of allocated ram, so it will be fast at reading and writing, which is exactly what I wanted.
So basically, every record in the database will be stored as a file in file folder fashion for simplicity,
So, when the server is started it loads all the required records from various tables and stores them individually in the required folders on the tmpfs partition and the game server will use that copy of the data.
There will be a scheduled background task that runs a Script that locks a record (i.e. tmpfs file), reads its content and if it's time to update then [update the database with its content], unlocks record (i.e. tmpfs file) move to next record and so on, this is done every 10 or so mins, so if the server crashes and needs rebooting all that will be lost is 10 or so mins of play.
I tried it last night loading up all 161 ship accounts that I had in the previous game that was run and it was fast, so this might be the right direction.
I was having some thoughts about lookups that return multiple records, like returning all the planet records that are located in sectors 35 to 97, in a SQL Query that would be simple, whereas my way I would have to for loop through folders to get the same result, it is doable, but I just wasn't expecting it that's all.
Well that's it for now, I hope you all have a nice new year.
2 Pass Verification (part 2)
Post by TheMightyDude, Tue, November 22, 2016
Well I had a play with the 2 Pass Verification and it didn't go as planned.
It seems that the code that I got from GitHub is using Google's charts API for the QR Code generation which seems to be deprecated and on top of that the code was spread over loads of files, where as the new code I have found is all stored in a single file and is also easy to follow in the code.
So I looked else where and found some code that can create a QR Code either using a table (YUCK) and each cell is a pixel, or it uses the GD library to generate an image.
I have tried two Authenticators (The Google Authenticator and the Microsoft Authenticator) and both tests are the same; they both display a 6 digit code that changes every 30 seconds.
So I might use that code for the QR Code Generation along with my own TOTP (Time-based One-time Password Algorithm) code.
2 Pass Verification
Post by TheMightyDude, Sun, November 20, 2016
Well I have been thinking about this for a while now, but due to the current security issues I have seen on Sony's PSN where users have had their accounts hacked and Sony refusing to resolve the issue, I have therefore looked into a 2 Pass Verification.
I will be using Google's Authenticator for this feature.
At this point I haven't tested it yet, but I am sure it will be fine and work :P
Added DB Support for blogs
Post by TheMightyDude, Sat, November 12, 2016
Well I finally got around to added database support for the blogs, so now they are stored in the database.
So this will help edit blogs and support tags etc.
There be code here captain
Post by TheMightyDude, Thu, November 10, 2016
Well I thought it was time to setup a development blog, so that I can show how the progress is getting on. There may be some images, YouTube Videos and so on, but at this current time I am just in the planning stage, basically seeing what works and what doesn't.
At this current stage I am playing about with Smarty Template, Bootstrap 3, JQuery just to see how they work together. They seem to work ok, sadly there is an issue, Smarty Template can only load the page once, basically it loads it the once and if you want to change a huge chunk of its content the page needs to be reloaded.
Now by using Smarty Template we are able to parse information to the template file to only display certain information, like only display the admin section when an admin logs in and so on.