About the Ingress Server - History and Plans
Hi, I’m @ofer2 and I’m an engineer on the Ingress team in LA. In an effort to be more transparent and open with the agent community about recent server issues and maintenance windows, I wanted to share a behind-the-scenes look to help clarify where things are today, and where we’re heading.
First, a little bit of history: Ingress started out as a small team within Google. We were originally an autonomous unit trying to answer the question, “What if Google Earth were a videogame?” The community shared fan art for game invites, captured portals seeded from the Historical Marker Database (HMdb.org), and the game began to grow. The game was built on Google’s infrastructure, and it’s what we refer to as our classic servers.
Later, Niantic spun out from Google as an independent startup. While Niantic transitioned to its next phase, our game team used lessons learned from Ingress and created new infrastructure for our second game: Pokemon GO. This new set of platform servers was designed to support even more players, and is both high performance and low cost. Meanwhile, Ingress continued to run on our classic servers.
Shortly after the successful launch of Pokemon GO, our Ingress team was tasked with rewriting the client because its foundational core was no longer supported. Like the classic servers, the original client was also designed in a way that made continued development more difficult and costly. On our server team, we made the decision to also move from the classic servers to our platform servers, and write new Ingress-specific features (like links and fields) on top of it. Unfortunately, both client and server transitions took much longer than expected, with the client being the most pressing need due to the risk of losing iOS support. .
Once Ingress Prime was released, the team began refocusing on fixing bugs and server updates. However, the new server still appeared to be roughly a year away from completion leading to a difficult question: Should we continue moving towards the new server at the risk of further disappointing the community and delaying new significant game features?
We decided to prioritize our community and the implementation of new features, resulting in a complete shift to finding cost reductions on the classic server rather than move onto the new architecture. This decision did still come with a hefty price. For example, Ingress cannot take complete advantage of awesome features that are automatically built-in for our other games utilizing the new server, so we must manually implement all of the missing functionality.
In order to reduce the costs of our classic servers, we chose to move our servers out of a high-cost compute environment into a low-cost one instead. Changing environments is like playing a game of spot the difference with those two pictures: we’ve fixed all the things we found, but with a picture large enough, changes small enough, and some changes mattering while other don't matter, it has been very difficult to anticipate what impacts will occur, which has led to various cases of testing and then rolling back features such as Intel. Intel was deployed into the new environment but the team discovered a central piece was needed first, resulting in the delay. We are optimistic that the central piece is very close but still need to hammer out some remaining issues.
All of the code for the changes have already been written and our team is currently focused on re-deploying the classic servers in this much more cost-friendly environment as well as evaluating the new impacts on the software side. We still need to test, fix bugs, and then publish. We will be starting our next round of testing on Tuesday. Once done, we will go back to testing Intel and eventually, the client servers. At that point, we will be running on a much lower cost for the classic servers which enables us to focus on features that will revitalize the game. New feature specs are already in the works that we hope the community will be excited for once it’s ready to be shared.
I hope this update has provided more clarity into the decision-making of our team and clearer transparency into how we’re actively shaping a new future for Ingress.
Comments
Thank you so much for providing so much insight into what's happening behind the scenes. Knowing we have new features to look forward to once the process is complete makes it much easier to accept the bumps along the way.
Thanks for the backstage view! This is really helpful information, and it not only puts ongoing issues into context but also gives us more information we can use for rumor control (i.e., to combat the conspiracy theories and such that some players concoct about goings on at Niantic). There's nothing harder than working on software that is meant to be "invisible:" if it works, no one notices, and if it doesn't, everyone yells at you. But without that work, everything collapses. Thanks for your efforts in support of Ingress, and the community!
Thanks again for the transparency. So much better than players (with varying levels of actual knowledge) trying to read the tea leaves of Niantic's intent.
Thank you.
This kind of information is great. Thanks
Thank you so much! Little posts like these go a long way, belive me. Working on a product is hard, even more so if you can't really talk about future plans without the community ripping into you constantly asking "what about X"
Speaking of... ;) You wrote that the development of new features instead of switching to the new server was prioritized. I might read a little too much in between the lines, but the way you wrote it, it seems like the classic servers are here to stay for a bit longer, even though it seems harder to code stuff around it. Out of curiosity, at which point would you personally consider the move to the new server imperative? (And if there is a rough time plan, when? ;))
Thank you for the information. It really means a lot and helps and is fun to read!
I'm a bit confused on a part. You spoke about the team working slowly on the platform enhancements to support Ingress specific functionality, but due to that being too complex or not in focus, it was dropped out of focus due to a large time frame for release, stagnating development in the search for server parity.
Now prioritizing cost enhancements on the legacy infrastructure almost reads to me about cutting costs to let it peacefully lay without burning too much $$$. I can't imagine how much money has been spent on rewritten clients for Prime, but it must not match the effort required for platform support for Ingress, as simply gaining the features that other games have seem like a win-win.
Does that mean the priority now is making Ingress cost less to run and make more money, or is there still a small part of the team working on the platform features for Prime? I'll admit I got excited when I saw the application slip protos in for "ServerMigrationFeatureKnobsProto", so I thought endpoints were moving to a new server one at time with same data-store.
Though, now I wonder if what I was watching was a less costly legacy infrastructure build out, not the true platform client integration. You probably can't comment on that though.
If you are able to answer some real technical deep dives for my own curiosity - I was always curious with:
Thanks for the informative post!
One question, does that mean we won't be moving to RWP, or that it's delayed in favor of getting Ingress moving forward for a while first?
So, that was the news information about the history and plans of Ingress Server!
Thanks and Happy Valentine's day!♥️♥️
Does that mean the priority now is making Ingress cost less to run and make more money, or is there still a small part of the team working on the platform features for Prime?
@iBotPeaches Yes, our focus in on getting Ingress to a sustainable place. That means having players who are engaged and having a game that can support itself financially. In order to get there, we may implement some platform features ourselves (i.e. in the classic codebase) or we may not implement them at all and instead focus on new game features. This is why I said that the choice to stay on classic for the time being is not an easy one.
As for how linking works....this is one of the crazier parts of the codebase. The server basically locks everything in a region and filters out as many irrelevant things as possible based on geometry. Then whatever remains, it checks does not intersect the link one by one. This process as you can imagine is super costly and only allows one check at a time. Therefore, the check is performed once and then is attempted to be written separately. This is the cause of ghost fields: if the check passes and then someone else does something to change the state (i.e. add link / destroy portal / add field / etc.) the link is still done afterwards because the check isnt performed again. This problem was solved in the prime server implementation, but unfortunately the prime server is still a ways away from being usable.
Out of curiosity, at which point would you personally consider the move to the new server imperative? (And if there is a rough time plan, when? ;))
@InvestigateXM this is something that we're keeping in the back of our minds, and will want to get to at some point. Current thought is we might try to incrementally switch over, but there are no concrete plans. As for timeline, still up in the air. As I said above, our focus in on getting Ingress to a sustainable place. That means having players who are engaged and having a game that can support itself financially.
@Perringaiden not sure what RWP is, but I assume that you are referring to the prime servers. Your statement is correct, at least for the time being. We may still move in the future, or move incrementally, but for now we are staying on classic servers.
This transparency is greatly appreciated. I'm excited about the new features and hope that it's not too rough of a process for y'all haha
Great post, very, dare I say, Enlightening!
Like the classic servers, the original client was also designed in a way that made continued development more difficult and costly.
Palms are sweaty
Knees weak, tech debt heavy
No comments on my code already
Hanke's spaghetti
Seriously though, thanks for this write up! I hope it's not the last of it's kind, and I'm sure many of us would appreciate deeper dives into what goes on behind the scenes!
Big thanks for sharing!
As far as I know, the 'classic' server methods are basically using the google cloud data servers. So... does this mean we'll be moving to the slower servers now that it's an external entity? Will the game start slowing down because of these cost saving changes?
Can we ask how the cost numbers are looking? Revenue is basically CMU conversions and that's about it, right? How are anomalies faring as far as revenue vs cost? Is that why we're down to just one 'big party which is a net loss but engages players'?
@ofer2 thanks for the update, looking forward to new features and game mechanics.
But I can't help thinking, that working with the old, and VERY slow for what is actually needed, is the right approach.
Could you please say if this current implementation is going just an intermediate step to the higher performing NGP (Prime Server)?
Thank you so much for providing this insight towards an otherwise black box of information regarding the development of this cherished game.
Sidenote/question: Any timeline for when the game's OSM overlay to be updated? The most recent one is dated from 17 December of 2017 and quite a few improvements/road updates have been implemented since and a regularly scheduled update to the overlay would drastically improve game quality.
My opinion is the maps used for calculating scores NEED updating first.
Thanks for all the work, we enjoy ingress.
@ofer2 this post was amazing THANK YOU for providing this feedback for us, players. Sometimes (most of it) we really just think Niantic is decommissioning Ingress, because how slow things move.
But with that letter we can have a backstage view and feel Niantic is yet caring about us, their early players.
I really think that even this upgrade challenge can become a HUGE case of success by Niantic. Really, and then you guys can present how you did it. Me, as also a software engineer, would be very curious to know how you guys made it.
Now, talking about present, I would like to address this to all ingress team: I know now its not easy, but I know you guys are doing your best, so THANK YOU!!! I believe that you guys can handle this!
And finally, lets hope this Server Change doesn’t take so long, because if there is too many features tangled with legacy infra, it will be hard to decouple, A LOT of re-work. So besides only hope, lets also try to build in a way aiming for that future upgrade.
CHEERS for a prosperous future 🍻!
While I appreciate lip service as much as every other paying customer...
Can you finish fixing what you broke before you go writing lengthy fanmail?
My account has been unplayable for 3 days. Inventory is locked behind a known server issue.
I responded to Support's request for the error code in my scanner 48 hours ago.
NIA started a shiny new fielding event that I can't even participate in.
Bad move, trying to engineer-speak your way outta the mess that's been made.
You guys doing an amazing job, but you still issuing heavy bugs.
Firstly, some services goes off and makes Ingress unplayable.
Second, account issues. My actual IGN is EMRE0YUN but forum shows wrong account name due email.
Thirdly(also connected with 2nd issue), some services not lets in due account issues(Facebook Login i mean)
Forthly, support team is bit slow. They answers us too lately.
Otherwise, Niantic is success project with Ingress, Pokémon GO and Harry Potter Wizards Unite. Keep it up, we are always supporting you.
@ofer2 Thank you so much for openly talking to the community. It's much appreciated.
This is a very nice share, thank you. A part from the server cost, how is the scalability of the backend now? I mean, for what I understand, PoGo, Harry Potter and Ingress now rely on very similar (yet not identical) environments. Will there be major updates in the foreseable future for the platform?
why isn't your name @Nianticofrr2 ? 😆
Thanks for the insight.
Finally some transparency!
Great update from you guys. We were expecting transparency from you guys from day one. It is way less frustrating if you at least know what is going ond why this decision has been made that way. I am sure that things are moving forward (sometimes slower than expected) but it is getting into better. As I can imagine you have migrated all server side into container based environment (would be most logical thing to do to reduce hosting overhead and costs for it too). If I can throw a tiny idea: If Niantic events would be scheduled to take turns in different games (Ingress, PoGO, HPWU) and they would not affect other game's events. Reason being that Ingress event is clashing with HPWU and PoGO sometimes which makes it more challenging to achieve best results on either of them. Just to clarify, there is significant user base playing 2 out of 3 or even 3 games at the same time and we got only limited ammount of time available for activities. Scheduling not to overlap events would make it lighter for the load balance of DB server (as assuming it is using same db for all 4 games (Field trip, Ingress, PoGO and HPWU). In big events of Ingress (Anomalies, Mission days and other events) quite often you can feel it becomes "laggy" on the client site for users regardless of which game you are playing. Hope that helps. If there is needed for some insight and on field data I am more than happy to provide that in future. Feel free to get in touch if you would like me to provide some help and other insights.
One really important thing, we are here to help and I believe it is many of us!! :)
Keep it going guys! We all love the game!
I can’t login right now. Hard to accomplish a live event like that
😘
Neat... Although this post doesn't actually directly answer any questions. At least as far as I could understand it.
It Does do a wonderful job of appearing to tell us things in a weird sort of proprietary tech-speak... But after reading it I don't actually know anything new....
Except at some point in time Ingress may or may not be moving to the same servers that PoGo and WU run on... And until that time, instead of moving to those servers, we'll get game updates... But a lot slower, and not as good as updates Could Be once Ingress is running on the new servers... Which it may never do, unless it does.
And these updates... Might lock Ingress into using the old servers? Maybe?
Which Ingress might still move to, or not. Which there is absolutely no timeline for.
Is that basically it? It was Very hard to tell what the post Actually meant.
So what exactly about all that was Transparent?
You may or may not do something, and anything you Do do will be slower and not as good?
It seems like more of a veiled blanket pre-excuse than actual development transparency.
I'm pretty sure the OP said Ingress is Not on the same servers... But may or may not move to them at some point.