They had a 'speculative/tentative' fix for one part of the problem, the recharging. All other 'latency issues' haven't even been addressed. The fix they did make, didn't work.
It's been like this forever though. The blank map on startup had a 'tentative fix' like 3 years ago, and it's still an issue today.
So, this issue definitely isnt being ignored. Its just hard to figure out due to various reasons. The first reason is that it just doesnt happen all the time. We've spent full days with no reproduction on our end. The second reason is that we suspect that the issue lies in some of the shared low level code provided by the platform and not ingress-specific code (as there seem to be reports of similar hangups in other Niantic games) which means that people on this team dont have insight into that code. Next, that code is built differently for development and production environments due to anti-cheat and other such mechanisms which makes it harder to find and even try out a fix. Lastly despite all of these complications and difficulties, at the end of the day it is on us to fix these issues, I just ask that you understand that we are trying and while it may seem like a trivial issue, it really isnt.
Pokemon Go added a Log feature which saves the latest log of a Raid or Trainer Battle. Maybe that would be an idea for Ingress as well? Save the last X minutes and as soon as an agent notices lag, they hit the button in the settings to capture everything that led up to the lag?
I think the problem started for real when u switched to the low-end servers before that this issue wasnt present but maybe its not because if that but i dont know really. Should we send more reports when the lag happens is it of any help saying the game lagged x time for x min ?
I hope u can figure out whats causing it, usually i notice it when fully deploy a portal many times it get stuck on second last resonator sometimes can be so for a minute or two before finally it lets me deploy fully.
My experience is as follows: When switching between LTE and 5G, the game hangs and you can't set resonators or anything. The speedlock also causes the game to hang in that this also occurs. No I am not a cargresser, but I am a tramgresser. In constant networks, i.e. area-wide, the game runs relatively briskly. But if this is not the case, these problems occur. The time of day also plays a role. During the day, when many people use the same dial-up point, for example o2 or Vodafone, it hangs. In the evening it is less the case. If it happens that you can't do anything, you sometimes have to restart the scanner 2 or 3 times. Maybe someone has had similar experiences. It may be that it has to do with the server change, but can also have other causes.
Over the weekend with the low drone times. Each time a drone is ready (while actually playing, deploying/glyphing/linking/fielding), effing laggg.
Deploy on a portal, glyph it for keys/resos, and effing laggg.
Capture a portal, start spamming (deploying resos) and after 2-3 resos, effing laggg.
Recharging multiple portals, effing laggg.
Glyphing a bunch of portals quickly (or trying to), effing laggg.
Maybe you need someone who experiences it frequently to come out and show those who are working on the issue how frequently it happens? (I'm sure there are enough folks near HQ that could demonstrate the issue) I was playing yesterday evening (after 7 pm CST) in a park, getting the effing laggg multiple times. It got so frequent I almost quit playing.
I'm pretty sure it was actually the switch to lightship when issues started. There's probably some hangup in fetching some status on the portal itself.
When we locked up yesterday the rechargers were separated on the map far enough with comms range set to 5km that we would not have had the same comms activity feed, the only thing we shared was the remote view of the portal and interacting with it via recharging. This is consistent with every lockup we've had with our recharge team. We all say in the slack at the same time that we locked up.
There's going to be remote recharging for Kythera, so we will see if more rechargers and attackers makes the problem worse I guess.
We have a feature in Ingress called "4 finger tap." Basically, touch the screen with 4 fingers and the logs get copied to the clipboard and you can paste them somewhere. Generally this is useful to help debug the issue, but we suspect that its actually happening at a lower level than what these logs capture. If anyone is technically inclined and knows what logcat is (the command is "adb logcat -v time" but you have to have the utility installed and dev mode enabled on your device) that would probably give us more insight, but even then its pretty hard.
We're fairly certain that switching to the new servers didnt create the issue (also, the servers we switched to are actually MUCH more powerful than the old servers) since we saw latency + errors go down after we switched (once the bugs were sorted). That being said, we wont entirely rule it out.
Switching networks can very well be part of the problem. If you switch IPs it would almost guarantee it (i.e. if you switch cell towers and a cell tower gives you a new connection, then all of the connections to the server would have to be reset).
This is unlikely since the Prime client has always been on the same base. That being said, it is entirely possible that when we upgraded to a new version of the platform, the bug was introduced.
That's sounds like a fun thing to do while recharging, maybe I'll catch a lag :D Still, if you're truly in the dark and think this more detailed logging will help, maybe implementing such a game logging feature will give you a ton more data from out in the field, and not just from the few technically inclined while they're hooked up to ADB. But hey, I'm no developer, I'm just good at using Google lol
I'm not confident on the cell switch theory. I often recharge at home, not moving, on a set cell tower.
There appears to be two kinds of lag.
First, recharge lag. This has been happening way before the server switch. Symptoms are power bar usage shows, but lag before it "drains" down to the used level.
This has been happening basically since switching to Prime. (Check all the recharge issues logged in Bug section of forum).
The second is deploy/mod deploy/cube usage lag.
These occur usually when deploying resonators or mods or triggering a power cube. Long delays. Purple circle for cube, spinning spirals for mods.
adb is short for Android Debug Bridge, as ofer said its not dependent on internet connection - it's the command line for anything running within Android
Parent of RectTransform is being set with parent property. Consider using the SetParent method instead, with the worldPositionStays argument set to false. This will retain local orientation and scale rather than world orientation and scale, which can prevent common UI scaling issues.
@ofer2 Is it possible to turn on debugging in the client so we can log it locally, then have some place to send the logs to? (outside of the adb logging) or release a BETA client to some users to collect some data for you/Ingress to use to help narrow down the issue?
We are willing to help. Just need the right tools to be able to do so.
That's super common. Dev is usually far more heavily instrumented, but doesn't perform suitably for production. That way they can get more info on a problem. Additionally, often dev for cloud work will be run in a local container to allow debugging.
So.... The dev environment with tons of debug code inserted runs better than the stripped down prod code ;)
Naw just kidding. We do the same here and it can be tricky to find issues. However I think the longest was 2-3 weeks in my 30 years there with devs working overtime to find issue and fix it. (Team of about 7).
With the types of lag- something kicked in around early/middle of last year. Can we not roll back code, add required new code and try it in closed beta or similar. Narrow it down to whether it's servers/non Ingress code/Ingress code. (I am not a dev so.. ignore me if this is stupid, I'm IT systems engineer).
We have a "cannot reproduce" that's had various companies report it once or twice over the last 15 years. Enough that it's a real thing, but so infrequently it isn't worth chasing and we can't reproduce it.
However, the lag issue is far more common and actually a significant issue, so maybe they need to add some of the instrumentation to the live system to see where processes are slowing.
@ofer2 I can replicate laggg. Though it might not be the "right" laggg. Make a field of your faction a few km long on each side. Link to one of the anchors of said field.
Can also link from other portals under the field to portals under the field. It might not be the actual lagg issue though.
It's more likely me just causing extra cpu cycles while ingress servers are trying to calculate if I can actually throw a link that normally shouldn't happen.
Comments
They had a 'speculative/tentative' fix for one part of the problem, the recharging. All other 'latency issues' haven't even been addressed. The fix they did make, didn't work.
It's been like this forever though. The blank map on startup had a 'tentative fix' like 3 years ago, and it's still an issue today.
So yeah...
Nothing is more permanent than a temporary solution ;)
So, this issue definitely isnt being ignored. Its just hard to figure out due to various reasons. The first reason is that it just doesnt happen all the time. We've spent full days with no reproduction on our end. The second reason is that we suspect that the issue lies in some of the shared low level code provided by the platform and not ingress-specific code (as there seem to be reports of similar hangups in other Niantic games) which means that people on this team dont have insight into that code. Next, that code is built differently for development and production environments due to anti-cheat and other such mechanisms which makes it harder to find and even try out a fix. Lastly despite all of these complications and difficulties, at the end of the day it is on us to fix these issues, I just ask that you understand that we are trying and while it may seem like a trivial issue, it really isnt.
Thank you for explaining.
Is there anything we, as ingressplayers, can do/ help out with?
Pokemon Go added a Log feature which saves the latest log of a Raid or Trainer Battle. Maybe that would be an idea for Ingress as well? Save the last X minutes and as soon as an agent notices lag, they hit the button in the settings to capture everything that led up to the lag?
I think the problem started for real when u switched to the low-end servers before that this issue wasnt present but maybe its not because if that but i dont know really. Should we send more reports when the lag happens is it of any help saying the game lagged x time for x min ?
I hope u can figure out whats causing it, usually i notice it when fully deploy a portal many times it get stuck on second last resonator sometimes can be so for a minute or two before finally it lets me deploy fully.
My experience is as follows: When switching between LTE and 5G, the game hangs and you can't set resonators or anything. The speedlock also causes the game to hang in that this also occurs. No I am not a cargresser, but I am a tramgresser. In constant networks, i.e. area-wide, the game runs relatively briskly. But if this is not the case, these problems occur. The time of day also plays a role. During the day, when many people use the same dial-up point, for example o2 or Vodafone, it hangs. In the evening it is less the case. If it happens that you can't do anything, you sometimes have to restart the scanner 2 or 3 times. Maybe someone has had similar experiences. It may be that it has to do with the server change, but can also have other causes.
Having alot of "problems" recently.
"Mod does not exist" message when trying too deploy a Green shield, that one was new last week.
The client hangs when opening a kinetic capsule. - Works again after 5-10 minutes.
Overlapping UI.
I simply cant motivate people too give 5 stars on the App store after a year of the same lag issues / keep their subscription up for Core.
Places/times I've seen it recently.
Over the weekend with the low drone times. Each time a drone is ready (while actually playing, deploying/glyphing/linking/fielding), effing laggg.
Deploy on a portal, glyph it for keys/resos, and effing laggg.
Capture a portal, start spamming (deploying resos) and after 2-3 resos, effing laggg.
Recharging multiple portals, effing laggg.
Glyphing a bunch of portals quickly (or trying to), effing laggg.
Maybe you need someone who experiences it frequently to come out and show those who are working on the issue how frequently it happens? (I'm sure there are enough folks near HQ that could demonstrate the issue) I was playing yesterday evening (after 7 pm CST) in a park, getting the effing laggg multiple times. It got so frequent I almost quit playing.
I'm pretty sure it was actually the switch to lightship when issues started. There's probably some hangup in fetching some status on the portal itself.
When we locked up yesterday the rechargers were separated on the map far enough with comms range set to 5km that we would not have had the same comms activity feed, the only thing we shared was the remote view of the portal and interacting with it via recharging. This is consistent with every lockup we've had with our recharge team. We all say in the slack at the same time that we locked up.
There's going to be remote recharging for Kythera, so we will see if more rechargers and attackers makes the problem worse I guess.
We have a feature in Ingress called "4 finger tap." Basically, touch the screen with 4 fingers and the logs get copied to the clipboard and you can paste them somewhere. Generally this is useful to help debug the issue, but we suspect that its actually happening at a lower level than what these logs capture. If anyone is technically inclined and knows what logcat is (the command is "adb logcat -v time" but you have to have the utility installed and dev mode enabled on your device) that would probably give us more insight, but even then its pretty hard.
We're fairly certain that switching to the new servers didnt create the issue (also, the servers we switched to are actually MUCH more powerful than the old servers) since we saw latency + errors go down after we switched (once the bugs were sorted). That being said, we wont entirely rule it out.
Switching networks can very well be part of the problem. If you switch IPs it would almost guarantee it (i.e. if you switch cell towers and a cell tower gives you a new connection, then all of the connections to the server would have to be reset).
This is unlikely since the Prime client has always been on the same base. That being said, it is entirely possible that when we upgraded to a new version of the platform, the bug was introduced.
That's sounds like a fun thing to do while recharging, maybe I'll catch a lag :D Still, if you're truly in the dark and think this more detailed logging will help, maybe implementing such a game logging feature will give you a ton more data from out in the field, and not just from the few technically inclined while they're hooked up to ADB. But hey, I'm no developer, I'm just good at using Google lol
@ofer2 A million thank you's for replying!
I'm not confident on the cell switch theory. I often recharge at home, not moving, on a set cell tower.
There appears to be two kinds of lag.
First, recharge lag. This has been happening way before the server switch. Symptoms are power bar usage shows, but lag before it "drains" down to the used level.
This has been happening basically since switching to Prime. (Check all the recharge issues logged in Bug section of forum).
The second is deploy/mod deploy/cube usage lag.
These occur usually when deploying resonators or mods or triggering a power cube. Long delays. Purple circle for cube, spinning spirals for mods.
The last issue. Portals not loading or displaced.
Again thanks, it's a bit depressing you have not found the issue yet.
This updates so welcome it's made appearances on several Ingress news feeds...
I have it happen multiple times a day every day. Might try the log idea.
You dont need to be connected to get the adb log. You can be out and about have it happen, then go home and the adb log should have it there for you.
Thanks. We are definitely still trying to figure it out, and I know it sucks, but it will probably take more time still :(
@ofer2 Would it be too much to ask for a monthly update on this? Even a "No real update on this" is appreciated rather than silence for ages.
adb is short for Android Debug Bridge, as ofer said its not dependent on internet connection - it's the command line for anything running within Android
This after we’ve passed the 1,5 year mark. To me it feels too little too late. The fact that you are posting this now, it just feels unreal.
We have logged the issue is a separate topic for months. People offered to help, offered technical insights, there was NO response to ANY of it!
Anyway, I hope you find it and a solution soon, the game has been steadily dying whilst you guys “investigate”.
There’s three types. And I should hope they at least know what the scope of the problem is by now, but I’m reading “we don’t have a clue” :/
Funny, four finger tap also works on iOS.
I see this error repeated a bunch of times….
Parent of RectTransform is being set with parent property. Consider using the SetParent method instead, with the worldPositionStays argument set to false. This will retain local orientation and scale rather than world orientation and scale, which can prevent common UI scaling issues.
Yes the issue is on the platform code, on the comm layer, where there is data corruption happening.
Or even the scheduler
A B C is supposed to happen
A C B is received, due to data packets taking different times to travel across the globe. The platform does not know how to handle this,
As You pointed out, and myself and many others have reported that it is a widespread issue across multiple game.
Surely there is a good deal amount of work with the platform devs to sort it out.
Only niantic products have this specific issue, other games do not have this problem..
There are also known data corruption bugs with some versions of protobuf
production is built differently than dev? um, okay - how differently?
@ofer2 Is it possible to turn on debugging in the client so we can log it locally, then have some place to send the logs to? (outside of the adb logging) or release a BETA client to some users to collect some data for you/Ingress to use to help narrow down the issue?
We are willing to help. Just need the right tools to be able to do so.
That's super common. Dev is usually far more heavily instrumented, but doesn't perform suitably for production. That way they can get more info on a problem. Additionally, often dev for cloud work will be run in a local container to allow debugging.
So.... The dev environment with tons of debug code inserted runs better than the stripped down prod code ;)
Naw just kidding. We do the same here and it can be tricky to find issues. However I think the longest was 2-3 weeks in my 30 years there with devs working overtime to find issue and fix it. (Team of about 7).
With the types of lag- something kicked in around early/middle of last year. Can we not roll back code, add required new code and try it in closed beta or similar. Narrow it down to whether it's servers/non Ingress code/Ingress code. (I am not a dev so.. ignore me if this is stupid, I'm IT systems engineer).
We have a "cannot reproduce" that's had various companies report it once or twice over the last 15 years. Enough that it's a real thing, but so infrequently it isn't worth chasing and we can't reproduce it.
However, the lag issue is far more common and actually a significant issue, so maybe they need to add some of the instrumentation to the live system to see where processes are slowing.
Is it possible to link portals like this? Oh sure.
Haha sweet.
Upcoming feature, cross links possible up to 500m.
@ofer2 I can replicate laggg. Though it might not be the "right" laggg. Make a field of your faction a few km long on each side. Link to one of the anchors of said field.
Can also link from other portals under the field to portals under the field. It might not be the actual lagg issue though.
It's more likely me just causing extra cpu cycles while ingress servers are trying to calculate if I can actually throw a link that normally shouldn't happen.