This has to be the best Citcon to date. The very best. They just blow you away in ever panel. The first time ever the ship team really was not front and center. Ships were not a big deal. The actual game foundation was front and center. So good!

The lag spike when Benoit transitions from green to red IS the proof of server meshing. Plus running 3 servers and the replication layer on a single system with multiple clients connected, you would expect some latency.

Your channel won't die, man. Folks don't watch your channel because you're cynical, they watch bc you are being honest. You're not trying to hype or de-hype anyone.

Ships having their own dedicated server will bring literal meaning to large scale server battles.

I was talking some shit coming into this, given the glacial pace of meaningful updates and the necessary tech coming together for the PU systems to function, but I am so happy to eat my words. This CitCon hasn't totally allayed all of my trepidations about release dates etc, but the spark of hope is there again and it's so nice to see the team showcasing new tech, managing expectations in a healthy way and finally demoing the vision that all of us want to take part in. The legacy of boundary-pushing PC games might actually have a new yardstick by which to measure contemporaries. It actually seems possible again. What a wild ride.

It pleases me to see someone actually appreciate just how important that silly little demo was. It's literally the missing foundation corner that this whole house of cards is built on. We've officially arrived at "There's a chance". I think scaling this will be a lot easier than creating it. I bet this time next year we have full OCS and SM.

The thought of having two capital ships flying through space fighting each other and each one is its own server is mind boggling.

It was definitely a exciting moment. Now, lets see it in game. Then the dream will be a bit more real

If they really can scale it up it's absolutely massive, it will completely change gaming as a whole. Most games are limited to 100-200 players MAX but with this it will be possible to have thousands of players in the same area. It's one of the biggest breakthroughs made in gaming ever.

I dont think you can quantify how much hate and BS that team deals with from literally everyone. I am excited for them to have some of this figured out partly. Bc it benefits me, but also for their own sanity.. good for them

yup this is huge. With 10 years into development I was beginning to doubt server meshing would ever become functional. Massively impressive to see they did it. This is not your average online game server.

Perhaps we need a "SugaRMike" channel as well :)

Just another 10 years.

I seriously wish more people understood exactly how freaking MAGICAL this technology is. Sadly, most people won't get it, won't understand, won't notice, or won't care.

"I cried" Yeah... The memes will be legendary about this Citcon in 12 months.

Mike I've always liked how genuine you are. You make some great content. Keep up the awesome work.

Mike i watched you live reaction yesterday. And i dont blame you, the emotion of what this panel meant was palpable

Not only is this a milestone but the tech developed, if it were to be licensed to developers for other games, be it starengine or this server meshing, would likely be worth billions.

This just makes me happy for us and for them, man what an incredible step forward, people just dont understand this level of complexity from IT side of things. It´s absolutely amazing, and more important seeing how the gaming industry is moving the last five years... I mean yes we are just used to delays and bugs, but seeing how proud they feel on this, being able to feel how slowly performance is becoming better in PU, just respect.

Watched the whole stream with you and Levelcap. Just wanted to say, that you guys were a great combo and if you should repeat that next year, I will drop in again. 👍

Its amazing to watch the hype cycle live. Ill believe it when i see it in game

It's hard to believe they finally made it. And I want to be happy and want to believe in this game, because this is a game with a huge potential. But I'm still somewhat cautiously happy. And I want to see it coming to life. I still consider it took too long time to develop this core feature. But if we have it in the end, I'll be happy.

It’s cool Salt. I hope they really get this done, this time. They have so much practice , practice over years and years of hyping us up, then more years go by. But I will hope with you Salt.

Being a technical guy, this is an amazing thing. HOWEVER... They had to make everything "physical" as single entities. This makes me think of beam weapons and tracktors. So there must be overlapping between zones. I'm sure they have thought about it though and the lag between them. The question of lag between the servers is very important but, again, they have thought about it.

Yea, and I remember a "jump to Pyro" When it is deployed to the PU...great. I'm not going to hold my breath.

This is either SC's most important breakthrough or another 2016 CIG sandworm.

Wow. Three weeks since they got it working, that had to be nerve racking coming up to CitizenCol

CC was awsome glad I could talk to you guys after Day 2. Regards from xEtarYx

There are a lot of people who don't understand what the "server restart" scene represents. This will not happen during an error of this type. There are always backup servers that can take over, there is no need to restart one manually (this is a demo). This means that on 2 servers (main and backup), one can "transfer" (the replication layer does this constantly during the game) the data to the other and the backup becomes the main server. Thus the main, becoming the new Backup, can be diagnosed and restarted while the players continue to play. Later becoming a new backup. And so on. All this will not be visible.

The only worrying part: According to him, if you are in green that is next to purple, green is authoritative over entities in green only… BUT green simulates everything in purple. So while green is not authoritative over purple it still has to spend resources communicating with purple, and simulating what happens there. This is why it is seamless, both green and purple are simulating one another and do a simple authority ‘hand off’ when an entity crosses. It also means green is doing a lot more work than if it was only authoritative/simulating itself. So a few things: Latency is bound to be considerably higher interacting between server boundaries at least, i.e. rubber banding while guy X strafes left and right, and will cause freezing at worst. What makes me optimistic is despite this the server is responsible for a much smaller area. So while it will have extra work simulating areas next to it, the overall area it is simulating is smaller than e.g. the entirety of stanton. I expect some big teething problems with this but still considerably better than today.

I think the import part in the transition is that the purple and red sever are out of POV from each other because of the corner. So if all that is disappearing is stuff out of POV then we will never notice.

hi mike, at 22:30 paul mentions he’s running everything on the one pc and so it’s a little slow, this would include the server as he’s able to kill/revive it all on the one screen, that’s probably why it takes a few seconds. on a computer that’s only running the client it’s probably wayyy faster at reviving a server hope that puts your worries at ease :)

Paul Reindell is a freaking legend, people should learn his name, if this tech works out he will have been one of the major players who pushed game development into a true next generation

implementation without any issues and bugs approx 2037.

best citcon ever. Full stop. The wait has been long but they were right in working every feature on SQ42 first

On a big picture scale this is just a small baby step in all honesty. I'm glad they now have a working foundation for static servermeshing. But when we're talking about DYNAMIC servermeshing we're literally talking a universe away still. Technology will continue to hold that reality back for many years to come still. But at least they finally have "something" to show finally for our hard earned money right? And SQ42 looks simply amazing now as well. So yeah the dream is still alive but sadly for the Verse side it's still just going to be a lucid dream for a long time to come still. 36:29

Mike, I'm with you, the whole dang, two day show, was full of great presentations but that server meshing demo was the show stealer for me. This is incredibly tough technology to crack and they did it, even if it was only on a small scale for the demo, it still shows they have accomplished something great and I look forward to seeing it scaled up to the PU. Freakin awesome!!!

This reminds me of how nuclear fusion researchers recently achieved ignition for the first time. It doesn't achieve anything on it's own, but it proves that it's actually possible.

Wow!! Mike I can't believe you made time to create a React to this. Hopefully you got some sleep 😝

You explained it very clean

Big Deal - Mike great content you are really taking your content to the next level

Good that CIG could show a proof-of-concept demo of static server meshing, including something of a failover/failback test for both 3.22 (DGS + separate replication layer) and a comparison with the static server meshing. - I'm not doing somersaults yet, since separate replication layer is neither in the PU or "standard" PTU yet. - I expect - even more so than PES - that there will be a raft of issues with both replication layer and static server meshing at scale - Would be nice to see static server meshing in PTU by end 2nd quarter of 2024 ... as always ... we'll see!

Running a tiny environment on a single PC is still a ways from what needs to happen on the PU. But damn it is pretty amazing!

One thing I noticed a lot of people in chat seemed to get caught up on, is the term server, and having multiple servers. It's indeed multiple servers, but the servers job is only to do the simulation, as in the physics of what is happening. All entities ( that means any object can be moved or interacted with, so basically everything but the ground) is simulated by a server. The information about the simulation is then fed to the replication layer that sends the data to all the other servers. The reason you can see and shoot something from one server from an other one, is because every thing exist on both servers, at all times, it is only that server A is simulating person A and server B is simulating person B. In SC even bullets are entities and objects, so when you fire a bullet it will travel though it's server, and then when it gets to the boundary of that server/container it will be handed over to the new server where it will continue as all the information about the bullet is stored in the replication layer, so the new server can pick up the bullet exactly where the other server left of. All this ends up that for the user there are not really "multiple servers" or containers or instances. A user only connects to a server ( what they call a shard ) and that can consist of anywhere from 1 server to a thousand physical servers all running together to form that server. And when running this system with the replication layer not tied to the server, and having multiple servers you can indeed have a server per ship. They can also have multiple servers per zone if that is needed, as it's only a matter of assigning what server has the authority of that entity. (Talked with Paul at the CIG After party for quite a while about the tech and the dude is a wizard, but take my memory and understanding with several grains of salt as I'm just going of my own limited understanding of everything)

Relentless optimism guys, we must hold the line😂

This is the most important aspect for the SC ‘Verse and we just re-Watched it in real time. Fuck yes 🤙🏻

This is a milestone in gaming.

Paul is an incredible engineer. He was also involved with the introduction of planet tech. He is so passionate about this stuff, you can see how proud he is that they pulled this off. And rightfully so.

You're the man Michael!

@SaltEMike, In relation to your question at around 19:55 in the vid: The tech you experienced with your mining example is a persistence example. A server crashes and its state is restored when the server is brought back up. What the second layer (the replication layer) does is run the same instance over multiple servers Like: Your on orison, there are for example 2 major zones and 20 ish small zones, the 2 majors being the city and its area's and the small ones are player streamed hangars. These are "meshed" together to behave as a single game session. And multiple servers handle all the load, dependent on the load required and the server config a single server could run anywhere from 1 zone to several thousand. Now when 1 server dies, the replication data on the replication server remains intact, and your entity (your player) is migrated to another server with the same replication layer as the one your are on. You might experience a hickup of down time, and if the crash is big enough, lets say a physical server broke down rather then a virtual server then new servers will need to be allocated for more data and no existing server might have your particular replication layer active, in that case you could get 10-20 seconds of lag and then pop back in. But I expect they will split single instance loads over multiple physical devices so a single device crash cannot cause a major issue. What this means is, when your server crashes, your experience of that crash is the same as in the example in the video, your character moving from Purple to Green zone. AKA. you dont even see it happen. = No more 30K, now if the replication server crashes.. (and i expect them to run concurrent replication for that as well) we might get a crash. But a handler server like that has a extremely low load and complexity, allowing it to be highly stable. What currently however, and i did not see indicated here, happens is that it takes a while for the server to "crash" with the server running slower and more poorly over time and then crashing. I expect them to put in play pre defined metrics for a servers health that either the replication server checks for or a controller does. So that when a server starts to under-preform you are moved to another server in situ, and that server you where on is rebooted preemptively . The tech itself is not that different from current day database servers, where a single stack of physical servers behaves as a single server, and any virtual clients, or datasets running on the server exist as shared data. giving it high redundancy. Current server replication on data center level is a replication of the physical servers content to another site or server, replicating it over 2 physical stacks of servers often running dozens of server instances on each. Never seen it in games like this. A game like WoW only has a dataset for player locations if your server crashes you are moved to another seamlessly , but any entities on the original server are lost (creeps/bosses etc) It allows them to layer multiple servers over each other to allow for redundancy and more players in a given location at a given time without the server struggling to handle it.

Probably the most important feature and panel this past weekend. Hope? Yes! Hyped.. not yet. Reality will set in as the months go bye and Mike will remain salty I have faith lol

Also having large fleet battles should be possible (space or planet side or both) without time dilation like in EvE as each ship (and perhaps even compartments in ships) will have their own dedicated containers linked via the server meshing. So having 100 players on a one Idris or Javelin and other Idris/Javelins having their hundreds can all be managed by their own servers dedicated to their clients and only having to track entities that cross into their container. (Missiles, bullets, lasers, other players or vehicles etc). At that point the only thing holding back the game would be client side rendering of all that going on but at the distances and scales that these sorts of battles will play out at most of the time, I imagine that will also not be a problem. It s a very long way off (perhaps several years just for the foundational tech here to be optimised and matured enough) let alone actually implementing the final result and testing. But its now actually in sight and far more realistic. It would be great if the logic could anticipate large fleet battles when huge numbers of entities start to gather and start spinning up fresh servers on the fly, transitioning the clients to these so its in pristine condition for the actual slog it's about to go through. Ah, I can but hope and wait. I need the SQ42 fleet but in SC on multiple player sides sides squabbling over contested space in unclaimed systems, with a looming Vanduul invasion just casually happening mid battle as we all react to the chaos and some join up as others exploit the madness for easy kills and loot. An actual epic clusterf**k with lasting ramifications on the in game universe. Imagine the salvage and pirating and ripples through the economy. The UEE Navy showing up to lay the smack down, albeit a little late.

*servermesh is born and looks at saltEmike* server mesh: "daa daa"

I believe this would hold up at longer ranges as I dont think its intended to have 3+ servers literally running areas right next to each other. As of right now, we have individual servers handling the 700k+/1M+ entities of the entire system. If there was a massive 200 player, multi cap ship battle forming around Ghost Hollow, I'm assuming a server would spin up to handle the traffic of maybe the 60km region around it. That would only be the 200 players, their ships and gear, and like a handful of NPCs. For larger battles, I'd assume they would further nest spheres of influence so if you got within visual range, you would appear.

Let’s go saltemike sunday

As a programmer, I have to say, these developers are extremely capable and have impressed me in many ways. Server meshing is just one of the many things that was thought to be almost impossible until the Star Citizen people solved every one of it. 2024 may be a very interesting year. :)

Booting up a new server will be much quicker, as the new server only needs to be updated for its immediate surroundings (entities streamed in), not the whole gameworld. The less a server has to keep track of, the quicker it can be started.

Honestly phenomenal reactions agreed 1000%

This is all great, but in RL a lot of the potential issues will be avoided because the boundaries between servers will be in space between planets and/or moons. If you assigned one server per celestial body, you'd scale up the capacity of Stanton by what, 16x. With a little conservative margin, let's say a player cap of 1200+, instead of 100. But it would be easy to split it up even further - say, one server per major LZ, one per orbital station and LaGrange point station. None of those would require much in the way of viewing/shooting across server boundaries. In other words, what they're showing is worst-case. If they can come up with a good algorithm for redistributing server boundaries, then dynamic meshing will take care of the rest. At that point CIG will have an asset on their hands that they can license to other game companies for hundreds of millions of dollars. And if S42 is good, then CIG will be funded for the foreseeable future. Of course those are two big 'if's.

After a Server crash there is no need to repopulate, the replicationlayer is still there. A server basicaly just manages authority at that point, so a server switch shoul be very fast

My concerns are that there is notable desync even with 3 containers running on the same host. Being on the same host is the best case scenario when it comes to networking. Theres no traversal of intermediate networking devices like firewalls, routers etc Now translate this to Cloud based containers in multiple regions and continents and latency will be a big issue. Also if you look at the resource usage from the tool int he bottom right corner when he spins up the 3 servers, the total RAM consumption is around 30GB. Now lets assume they have debug logging enabled and have done minimal optimisation to give them worse case. Still this is for a tiny area footprint with 2 players and a buggy. Now scale this to the aspirational size of Star Citizen....the infrastructure footprint and cost to operate will be Huge. I dont think many people realise just how expensive its going to be to run this, which suggests a significant revenue stream will be necessary to maintain this on an ongoing basis. Needing an ongoing revenue stream will mean both a subscription model will be essential and microtransactions too. Which will inherently make the game unbalanced whale Vs avg Joe.

People seem a little concerned about the size of the zones in the example. But the size doesn't matter (in this case). The servers will scale up and down based on demand and the zones will be defined/configured dynamically. Previous plans for server meshing were based on the idea that certain servers covered certain zones, like there could be a server for each planet, each sector of space, maybe a dedicated server would spin up for a huge ship or battle, etc. But in this example of the modern iteration, those zones could be as big or small as you want/need. All three of these zones could be controlled by one server, split between 100 servers, replicated/duplicated to parallel servers (i.e., people in the same place/zone, but on different servers, seeing the same things, interacting with the same environment and with each other), or areas can be completely turned off when there is no demand. Plus, since the replication/PES handles all the persistence and streaming, the AI, dynamic economy and other background simulations/mechanics will be able to run independent of the game servers. An AI miner could be out mining somewhere and if there's nobody there to see it, no server needs to be spun up. It's just a matter of how many physical servers you have and you could theoretically have a million people in the same place at the same time. Not that anyone's PC could handle it....

That may have been the first SC demo that didn't bug out Live lol

The reason it took so long for the server to come back after he killed it was simply that it took a while to re-start the server. If there are multiple servers already running and ready to take over the simulation, it won't even take a noticeable time. The reason it takes so long for you to get back to the same shard in the current PU is because it takes so long for the normal server-rotation restart cycle in the current version before the shard is restarted.

It's important to understand the difference betwen a server as an application and a server as a piece of hardware. In this demonstration's context, a server is an application. Multiple server applications can run on a single piece of server hardware.

I agree with you, Mike. I pledged for the PU. I will play SQ42, but I am passionate about the PU. This presentation was monumental.

We were told this is what we will get, it's funny how all the messaging before doesn't seem to count for much as a physical demo, skeletal as it was did far more for building confidence than everything that came before

Man, I have felt so doomer about this project for years, but this is amazing!! I have never been so hyped for SC and S42. Best CitCon ever!

They might be able to prioritize entities near players to get replicated first as the ‘new’ server spools up. That way the process of streaming in entities from the replication layer will seem faster than 10-30 minutes from the player’s point of view?

I've done it immediately and got back in. two days prior to this post. pulled back into my ship after a game crash. a friend experience also. also, good for you for standing up for server meshing. i know you're salty, and ive been a critic, but you have every right to defend your beliefs and shouldn't have your trust questioned. disappointment comes from knowing someone/something can be better, and confidence is trust being rewarded. CIG has disappointed you, but you're confident they can do better. and they have with this 👍

This technology will not change at all how fast a server comes back after a 30k. Mike.

Server Meshing is what Worlds Adrift needed, but couldn't accomplish with the small team that Developer had. This success is amazing.

The difference between this spinning up and a current 30k server, is the current server has to populate the entire Stanton system but with server meshing each planet could be its own server so it only has to spin up that planet and not the entire system at least that’s how I’m understanding it

I think the idea is that if a server 30Ks lets say, with meshing one of the other servers would take over rendering right away until the server that 30Ked is back up

This is WAAAAAY more impressive than that fancy cinematic they released, but obviously need to see how it works when deployed to live to be 100% this is actually achieving what they believe it is achieving. Just, wowee though, extremely impressive tech demo.

The "magic" is the replication layer. That is what allows the smooth transition between servers. That and super low latency networking. I'd imagine they'll want all the servers colocated on the same network. Curious to see how it will work with clients over Internet. That's probably the biggest remaining hurdle.

I see manny comments here regarding lag between servers and etc. with a little bit of planing the will probably calculate where these server edges. I mean I cannot think these will be dynamically set at any points.

Answer for 20:23 , the slow speed is because currently the server has to stream every single entity in the entire solar system. a 30K with static server meshing will mean that a 30K server in these containers only has to stream in the entities of that server from the replication layer when needed, because the server that 30K'd would be a small size. IE.. the entire solar system does not 30K, only a small aspect of it does. The rest of the entities are already streamed to their servers.

makes me wonder how extremely large objects (javelin) would behave when 49.99% of it is in one server and the rest in another

Mike has been desalinated.

The stuttering at 30 min mark, you have to remember he is running 4 virtual servers (3 zone servers + 1 replication server) plus a game client on one desktop, that's not going to be representative of a single dedicated zone server at our current level of players if one server can currently Handel the whole damn solar system.

Call me skeptical if you like but this is not the first time dev tools were highlighted at a Citcon lauding some feature that we have yet to see. I'll believe it when I see it

I about broke my TV when Server Meshing worked! I was like "LETS GO!!!" because as soon as they put it in PU I think SC will take off!

Replication layer: it should only need to "propagate" the entities where players are currently at. In theory you could have a very large area running on a single server and it could have players extremely spread out and in areas with tons of entities, however... a good amount of the time it may only need to stream entities to a server where players are, and being able to push additional entities to that server as needed. "Hey server, you have players at X, Y, and Z, here are all those entities. Ask me when they move to new areas." Basically: just like how a client only needs its local entities streamed to it, so too does the newly activated server only need those fewer entities. It doesn't (shouldn't) need the entire 700,000+ entities of current day servers to be streamed over before it can start handling gameplay.

Yeah instead of drawing thought maps about how they think it will work they actually have a working scalable system. We’re past the 1000 ways to fail to invent a working light bulb and into the fine tuning and delivery phase. Exciting stuff.

19:20 The difference between this and that is that you never actually get logged out because you are on the Replication-Layer. So the question is how long will the lag persist (the time in which you are in a zone without a server running that zone). I honestly can't say that as well (obviously since I don't work at CIG 😂) but I will say that two aspect I think will shorten that spin-up time: 1. If we actually have ServerMeshing the whole point of that is that one server should work on less objects so the amount it needs to load in startup will naturally also be decreased 2. I can only speculate but I think the time will shorten at least a bit because right now the Server also INCLUDES the Replication-Layer which for sure also takes at least a bit of time. I can't really tell how much work starting a Server has to do with the Server (the simulation) and how much with the Replication-Layer (the object-tracking) but I can't believe all of the time is only the server and not also a bit the replication-layer. I hope this explanation was at least somewhat understandable and reasonable. I think this tech is so interesting and it honestly is one of the reasons I still follow StarCitizen (I am not super old as a backer but I have been with the project since 2018) LASTLY: For anything who doesn't believe this is real here is a simple explanation why this tech is actually possible and works: With ServerMeshing we are still playing on ONLY ONE SERVER for the entire universe, only that now this server is the Replication-Layer and not the game-server meaning that the server all clients are connected to (which gives the clients information about the ENTIRE UNIVERSE) is only giving database-entries to clients, servers and the entity-graph (database) which is a very easy and lightweight task. The real game-logic which is very heavy and takes most of the performance is now only applied to specific regions and when we are entering that region your physics and world-interaction is calculated by that server BUT we never actually switch any server-connection when transitioning between zones. THE PLAYER IS ALWAYS CONNECTED TO THE SAME SERVER VERY LAST: I THINK I KNOW WHY THE STUDDER HAPPENED. The player JUMPED at the exact point of the transition (which he only did that one time and only that time we actually saw a stutter). My prediction is that this is similar to desync. While he transitioned there was a very short time where he did not move (for the replication-layer) and that was transmitted to the client we are looking through. That client then wanted to smooth over that very small stutter so he walked in the same direction BUT in reality he already jumped. Shortly after that the replication-layer gave that information to your client and it now needed to change what it had shown before (walking to jumping). At least something similar to this is I think the reason that lag was there

Lol. I skipped this presentation cause I thought it was just PES. Server meshing. Amazing

Everybody wants every player in the same world. But even if this is possible, that would quickly feel overcrowded. Imagine taking the elevator with 2000 other players..

You won't get huge fleet battles until they get to dynamic server meshing. What they have shown is static server meshing. With static server meshing, each server is permanently responsible for a certain volume of 3D space. If there are more than 100 people in that space, the server will bog down. However, if there are too few people in a 3D volume, they are wasting server resources. So they will probably have something like one or two servers per star system or so, because we know that this is what the servers can handle right now. Possibly they do one server per planet to give us a bit better performance than we have right now. In the future they are going to build dynamic server meshing. With dynamic server meshing they adjust the 3D volume that each server handles in real-time so that it is a region that holds 100 people. That means you can have two rooms right next to each other with 100 people in each and the servers will adjust to have one server per room instead of one server per star system. This will allow an unlimited number of people to be as close to each other as they like because the region for each server will dynamically adjust to be small enough to hold only 100 people. This is the point in time when we can have huge space battles with unlimited numbers of players fighting each other.

BWIN stutter was likely do to IOPS of the workstation, basically had to reload every object when the sever streamed everything back in.

I don't think most people realize what they saw in that demo. CIG has done what most thought was impossible. They've found a way to merge individual servers in a way that they essentially become one big seamless megaserver. Never been done before. The way it's described that tech is infinitely scalable. The only real limitation is the number of servers you have. Let me put it in a way that everyone will understand... With this server tech the Oasis from Ready Player One just became possible... Let that sink in.

This was THEEE ABSOLUTE BEST Citizencon

They should've done a 1 billion stretch goal

Greybox also you need to see the zones they have to be visually distinguishable. Also see that those three boxes are not the same size. My guess this super important too.

when benwa stuters comming into the server , nothing wrong with that its the first meshing these things can be fixed the better the tech gets, hell yes it looks real to me boys, excited about this.

8 months later, Server Meshing, PES, Replication layer still Unicorns with no proven improvement to the game.

I will try and explain and this is from a server architect perspective. Normal games meshed or otherwise will store the user state on a server. Normal hosted servers eg. client connects to server. In wows case client connects to server and the server moves the user from one to the other i.e if the server you are on crashes the client crashes and the db will restore you to a server in the state you were in the last db update or stream.(Role back). This is how it works today Now CIG is doing something very different to everyone else and here is where the brilliance is. Just a note when I refer to the entity layer think of it as a db cluster (db clusters are multiple servers running the same db) when I refer to the environment it is a set of servers running the environment. This allows CIG to separate the character (held in entity layer) from the server (environment layer) so the client will connect to the entity layer not a server and will be moved to another serve when it degrades. So each geographic region will have Entity environment and a server environment and all players in that region will be on the same "server" as we know it today. So in the case of your story with the mining you will be able to be moved from one server to another without impact because your user and items will be held on a separate environment to what you are playing on. To all those that say server meshing has been done before, there is a huge difference between a tent held together by ductape and a house.

Yup don’t care about any little test until the feature is on my pc

I'd like to assume that the server replication system makes a copy of each currently running server to keep on standby, that is then updated every couple minutes. If the active server is 30K'd, the rep system uses the stored image to create a new one. Totally a guess on my part, but that would seem like the logical ideal reality for us.

I think we will still have 30ks even with dynamic meshing and replication layer, but it will be extremely rare and only happen in a “perfect storm” scenario. Best example I can give is my experience with FFXIV, I’ve played it for 9 years and I can say I have experienced server crashes less than 10 times. So it will be a more acceptable experience. It will still suck when it happens but it’s no longer going to be a question “when” it will happen during a gaming session. I’m so here for it