Dev Diary: Player Portal History
NianticThia
Niantic › admin
Hi everyone - I'm excited to share insight into the Player Portal History that was added back in February.
This entry of the Dev Diary was provided by @ofer2 , Senior Software Engineer.
===========================
Hi everyone, my name is Ofer and I was the Feature Lead for Portal History. With this Dev Diary entry, I wanted to give you all a bit of insight into the development process for this feature.
Competition, exploration, and social are three major aspects to Ingress. When I worked on Battle Beacons, I got into the competitive aspect. But with COVID and staying at home, this was put on the backburner. After being stuck indoors for so long, I was really looking to lean into the exploration aspect of the game. The main problem here, for me, was that I love exploration, but I need a goal and I need to see progress towards completing that goal. Therefore, like many of you, I wanted to be able to see the portals I had been to and go to new ones until I CAPTURED THEM ALL!
The opportunity to dig into this arose last year when there was a call for ideas for a team-wide hackathon. Our pitch was to add code to the server to track all the Portals you’ve captured and show it to you on the app’s Map Screen. In this proof of concept, this wouldn’t include previously-captured Portals because that was a whole other can of worms, which would definitely not be doable within the scope of a hackathon.
The main challenge with this limited feature was that we needed to mix spatial data with Agent-specific data. Normally, when you make a query for entities on the map, all of that data is the same for everyone. Therefore, the code responsible for doing that query has no idea which Agent is the one that is asking for the data. Secondly, we needed a way to ask for that Agent’s specific data in an area without having to look through all the data.
The basic plan was to create an entity (a storage container for data) that was keyed by player ID and S2 cell ID. An S2 cell is a way of dividing the world into chunks and labeling it. You can read more about it here if you are interested: https://s2geometry.io/devguide/s2cell_hierarchy.html. Then, when you query for entities on the map, we would route that through a new interface which also looks up the player data using the S2 cell and “mixes” it in to the Portal information on the way out.
With this plan in mind, I recruited Andrew, another engineer on the server team, who was interested in the feature and set off to work. At the end of the hackathon, we all presented our projects and I had the honor of presenting my team’s:
We used red dots at the bases of Portals and only showed unique Portal captures here, but programmer art aside, I think we made a really awesome feature, and wanted to work towards building a real version for production.
After the hackathon concluded, we returned to our normally scheduled work, but the team saw the results and the opportunity to bring to life a feature that Agents have been asking for for years. Brian, however, wanted the whole enchilada: prior Portal History included together with all Agent history from the beginning of the game AND to top it all off, he wanted it on the Intel Map, too.
Brian tasked the Engineering team with getting this feature out. Fortunately, we had a strong foundation with our hackathon project. Alas, it turned out to have a few kinks: returning Agent-specific data for every Portal is often unnecessary as the data hasn't changed since the last time you interacted with the Portal. To tackle this, we return the player data as a completely different piece of data from Portal data, and then put it together on the client. Unfortunately, the Intel Map doesn’t have real-time updates like the client does (it refreshes all data), so it still had to have the player data attached to the Portal.
The second major part of this was pre-populating the data with the entire history of Ingress gameplay. When we initially looked into this, we saw that the records storing actions that Agents have taken have a field for the entity. This field is hashed, meaning that we would have to reverse the hash to find out the actual entity that the record corresponds to. In order to do this, we created a process that would go through every single Portal ID and hash it so that we could use that hash to compare against the Agents’. Then we would go through each Agent’s records and find all the Portals they’ve visited and captured, and write that data to the structures that the client will query. After we finished writing this, we ran it in the dev environment and it wrote out the results to the datastore.
When we checked our results, they didn’t match the Portals we expected. It turns out that we made a mistake when examining the code that wrote out the tracking records. The function takes in a string which it calls ID, hashes it, and writes it to that field. The thing we missed was that this function wasn’t called in the case of visiting and capturing Portals. We instead called a different function first, which combined the ID of the Agent with the ID of the Portal. This means that the single list of hashes that we made before wouldn't work. We had to recreate the list from scratch for each individual Agent.
The cost for doing that process once was high, though reasonable; but multiply that for every Agent and the end-to-end process wasn’t feasible. The team put our heads down and went to work trying to reduce the cost. After some diligent effort and time optimizing the pipeline, we succeeded in reducing the cost significantly. One of the optimizations that helped us to do this was by making the flow run on a single machine per player instead of spreading it out to multiple machines. This involved some creative ways of reducing the size of the data that each machine would need to have in order to generate the results. Then, by specifically choosing to run the optimized backfill for Agents Level 3 and higher, we managed to get the cost into a reasonable range. The cost reduction of the engineering team here allowed us to give this feature to all Level 3 Agents and higher without making it a paid feature.
At this point, we had a working client and server, but we needed a clear way of displaying all of this data. We came up with a clever concept of “Layers.” This enabled us to only show Agents the information that they care about. The Art team took this on and came up with an initial concept, which everyone was so excited to try out. The first iteration was hard to read, so we continued to test different designs. The Art team quickly took that feedback from the team and from Vanguards, and increased the size of each Portal History ring and added spacing to the Portal, which made the rings easier to see at different zoom levels.
We have iterated quite a bit on this feature from the hackathon to during development before the launch all the way to development after launch as well. There were some additions that we knew players would want (e.g. invert on the intel map) but even without that, we knew that players would really enjoy this feature, so we elected to ship the core feature and then continue to iterate after launch. This iterative design process enables us to draw feedback from Agents and our teammates, and help guide the direction of the Ingress roadmap.
Player portal history was always something that we wanted to put on the roadmap as it was one of the top requested features from players, so we found a creative way to get it out to players. We’ve since expanded the Layers button to include Drones so Agents can see more activity going on around them. We hope to continue to make further enhancements in the future!
Comments
Hope we can get version 2 of portal history on Intel. Now it works to slow and I must restart map often to get it work on 1 lication.
Thank you for all of your work on this feature!
@NianticThia love these sorts of things, especially when @ofer2 goes into some of the nitty gritty things, that even a non-programmer such as myself can enjoy.
Thank for the background. As an avid unique hunter this was a very welcome feature.
Thank you Thia and Ofer for the write up! I love seeing some screenshots from some early Ingress features, you don't happen to have a few more, do you? 😉
Thanks for the great write-up, @NianticThia ! Retrieving those actions sounds a lot less straight-forward than I expected. 😫
This! I came here to write these exact words!
Thank you for the writeup Ofer, and Thia! 🙂
If you are playing the same game as the rest of us and interacting with the same play field as the rest of us, how is this a fair gameplay?
How can you know how to balance this game if you don't experience the same thing as normal players when it comes to inventory management?
It's a test server, which I think is quite evident by the multiple shards and anomaly ornaments in what seems to be San Francisco. As far as I'm aware, no anomaly happened there within the last year, so either he traveled back to EXO5 to test the portal history feature, or he's simply on a test server ;)
Thank you so much for this feature and getting it across the finish line!
CAP ALL THE PORTALS!!
Thanks for the write-up!
Is this the first time niantic have publicly stated they use S2 cells for game functions? (It's long been assumed but never confirmed I believe)
The screen shots were from a testing site, not the real game.
All hail @ofer2!
It's interesting to hear about how they develop these features and the challenges faced. A simple feature might sound easy to introduce but behind the scenes there are challenges to bring these features to life Hopefully to see more insights of ingress in the future (eg. drones, kinetic capsules,lore) as well as other entries from other developers so that we get to know the team better.
I really appreciated the history of this feature. Thank you!
I'd always wanted to try for my Onyx Explorer and Pioneer badges but without knowing which portals I had visited, I knew it would take a lot of wasted portal visits and captures to achieve.
Since the portal history feature was enabled, I've captured and/or visited thousands of new-to-me portals! I'm now just 735 unique captures away from Onyx Pioneer and 5,985 away from Onyx Explorer! This new feature has invigorated my gameplay and I thank you for it!
@Mxx these screenshots are from an internal-only test environment that we use for development purposes only. The production server is capped to 2,500 inventory for C.O.R.E. subscribers.
Thank you @ofer2, Andrew, and @NianticThia for building the Portal History feature and for sharing this post!
This is hot! Love the dev diary-- Keep em' coming!
Thanks!
Development Diary? Excellent!!
I asked me what will come next?
We will have to be aware. 👀
Gotta capture'em all!
Thank you for sharing! It's an amazing feature. It was incredibly helpful in my push toward onyx explorer and pioneer.
Perhaps a hint at the end with "further enhancements" mentioned after discussing the drone layer ;)
I didn’t expected such a difficult development of this new feature.
As a all-scope completist I really appreciated it, I missed some capture near my home and many scout control before the introduction of the feature.
Do you plan to expand this feature to drone visited portals? I’d use that layer
Very interesting read and thanks for doing the work. It's a feature that I use daily. Thanks too to @NianticBrian for insisting that you included old captures and visits.
Have you seen a drop-off in portal scans now that the ornamentation for them is hidden in a map layer? I know I haven't scanned anything in ages because I simply forget they are there.
That's really interesting.
Thanks for posting.
We are waiting... https://community.ingress.com/en/discussion/15707/community-manager-where-are-you-we-are-waiting
Thanks for the unique layer. Love it. Hope to get unique drone hack layer as well
I really like this kind of dev diaries, thank you all very much!
That was really interesting reading how much effort went into developing Portal History layers. This is the best new feature since Power Cubes. My favourite part of Ingress is exploring and collecting uniques, I love a road trip and have been on quite a few since Portal History was introduced. This feature was instrumental in helping me find all those new portals and portals that I had missed before and to finally getting Onyx Explorer. I really hope to see a Unique Drone Hack layer in the future.
Thanks so much @ofer2 and all else involved.
Very much enjoyed hearing the background on this feature. Whatever you’re paying @ofer2 & their colleague Andrew, it’s not enough.
I also found it a little odd, the acknowledgment that the game uses S2 cells. We all know it does, but it’s nice to see it officially said.
Gran trabajo y continuar así.