A Thursday afternoon all hell broke loose. We experienced a security incident on our online platform.
I started receiving phone calls with no significant information. I took it quite easy, having worked for this client for a long time, my experience has taught me not to get dragged into the well of mass panic.
The problem was simply described as: “Customers can see each others confidential data”.
So without any major overview of what the problem was, what the impact was, I followed orders and disabled the authentication on the platform, not allowing customers access to confidential data.
The rest of the Thursday afternoon and evening was a combination of phone calls, RT commenting, log examination and source code skimming, with absolutely no idea of what to look for. Attempts to recreate the problem were without success.
A very positive thing was that one of the other developers had taken lead on the assignment, so when people called me I could simply just refer to him, so all information would be going through a single point.
Without being anywhere closer to a solution I went to bed, I did not sleep particularly well, my brain working overtime on the problem.
Friday morning I got in early, we started brainstorming and attempting to gather as many facts as possible. We were unable to get a complete overview of the customer impact. A lot of misunderstandings on the nature of the problem where flourishing in the organization, since everybody wanted to participate and the information we had was scarce and showed no useful patterns.
We tried to gather as much information and started laying out scenarios, investigating dark corners of the systems. The online platform is some 7-8 years having run since 2001 and we had never had any serious issues particularly not of this kind, much of this due my very competent colleagues, no longer employed with the client.
I called the most knowledgeable of my former colleagues and we discussed the problem, he informed me that it would be possible to create the incident we had seen if the customer’s network administrator was either evil or stupid.
Everything pointed to some sort of proxy/caching mechanism.
We got clearance to call the customer from the security people and we interviewed one of the involved customers. The information we got here was again misleading and leading to a dead end.
Saturday we took the day off, even though orders from the corporate powers that be, meant that we should be working our asses off 24/7, we where nothing closer to a resolution.
Maybe I was not at the keyboard Saturday, but my brain was pretty preoccupied with the incident.
Sunday morning we met at the office, we started brainstorming again. The customer we had talked to previously had called our manager with more information. And had stated that he was unable to replicate the problem from home, only when using the work VPN. He had first seen the problem at work. Also the name of the other customer rang a bell since he had seen her name on the work intranet.
During our first interview, I had specifically inquired about possible relations, work/family wise. And whether they were on the same network.
So now things were finally coming together. We had a cache or proxy fooling with our data on a corporate LAN.
I laid out a plan to enable SSL/HTTPS for our authentication, since the session would then be between the client and us. So SSL would act as ice breaker for this piece of equipment on the LAN of the customer’s place of work, doing something we did not anticipate.
After a few changes and some testing on our test environment, I created a list of changes to make and applied them in production. We called up the customer, asked him to test and he was unable to recreate the problem, success.
I still have no idea what exactly was causing the problem, whether it was a badly configured cache or proxy, some security auditing tool or whatever and we simply left it there, after having reestablished the system for all customers.
I was then asked to write a report for the client on what the problem was and what we did to make it go away. At the same time we had identified several places, where improvements could be made to make auditing and logging easier on our side.
So now we have a bunch of stuff to do, including a report for the security department.
I learned a lot from this, like what questions to ask a customer, that it was a good thing to have a single lead on the group of people pursuing the issue and finally, take time and think hard, do not be stressed out by panic struck people. Be hard on the information you get, get at much as you can but be critical, people start to see ghosts everywhere and if you are really unlucky, they also look for somebody to blame. The latter was not the case this time, but it could might as well have been.
0 Comments.