Daily Archives: June 12, 2009

Mind the Gap

Posted by on June 12, 2009 No comments

One of my latest blog entries have been about a security issue.

The sort of issues are as such fixable and often very easy fixes. We have both the knowledge and we have the tools to do so. As I have lined out on that earlier occasion, our organizations sometimes lack the process and QA it takes to make sure that we do not leave our sites vulnerable.

Anyway – working on the issue did however demonstrate some other problems, I had not anticipated.

Picking some modules from CPAN to help me out seemed like a good idea and the component in question was already relying heavily on CPAN modules.

When I first started out, I had separated out the component in question from our core software, the actual portal software. Since the component was actually just a plugin (we call these services), so this was quite easy. This would mean that I could work on it and deploy it without having to deploy a whole lot of unrelated software, which current state I was not totally updated on. Personally I would prefer to isolate and focus on the security issue at hand.

So I cooked up a new distribution, copied the stuff I needed (not much history lost there since we only had the initial revision), deleted it from the core trunk, no dual life for me no sir, and moved on…

I then started building a proper distribution. I threw in my scripted installation process based on a subclass of Module::Build, updated this to reflect that the service was a core server not living on the ordinary application hosts, but on the core hosts, this seemed like an easy fix, documenting it was the hardest part, but I was to become wiser.

I am of that opinion that we should not have first and second rate citizens in our system, a service is a service and should be agnostic to where it is invocated. There is however a big difference between the installations. Where our application servers have a designated special directory outside the control of the local Perl configuration, the core hosts install much of their stuff under the control of the local Perl configuration, this was a bit annoying, but certainly fixable, but it does mean that the two rates of services cannot rely on the same build tool as is. Even though they are both just services.

Another problem with the build tool was that it utilized newer features so I was not compatible with the installed Perl and I was unable to install a newer one, note on revisiting this was made.

Not having a proper build tool to assert my configuration and the sanity of what I was trying to do. I simply had to install the module, by overriding the existing one and trying it out.

First shot (my release 0.02) failed utterly. My use of XML::Simple was simply too fancy, so I had the same problem as I had experienced for Module::Build.

I rewrote the fix using a simpler strategy. This time with the 0.03 release candidate, it at least compiled.

So after a lot of hoop jumping it worked out after some more testing together with a colleague, we nailed the actual security problem.

So to get back to the intro in this blog entry.

We have the tools and we have the knowledge, but as this demonstrates I ended up implementing a less generic solution using a regular expression. Just running the data through some encoding schemes would keep me happier, since regular expressions are often culprits of bugs and the first solution was much more generic and would handle changes better.

So the hindrance for doing the right thing, was actually the platform. Time is slowly catching up with us and the technical debt put upon by using a platform, which has not been kept up to date is starting to interfere with our work and solutions. The platform primary goal is to bring leverage, but now it is putting constraints on us and the work we do.

It would be easy to update the Perl interpreter, but working under time constraints to get a security issue fixes is not the time and place. Updating the Perl interpreter would require extensive regression test, something I am not sure our platform is ready for.

The lesson learned must be that a platform should be maintained using baby-steps, this strategy has proven useful in other situations. This mean that you update in small gradual steps so you do not have to take huge giant leaps, when you finally get to a point where updating is unavoidable.

In the example above I simply added more to the technical debt than was necessary, but I was forced to. I did not make the giant leap, which would probably be a good idea, so the next issue would be easier to address using contemporary versions of the tools at hand.

Do not get me wrong I really like old platforms and in particular this platform, but things should be maintained, because if not they become a nuisance and get exchanged for something new and fancy, which might introduce new bugs, which had been addressed over time on the old platform.

A story about a XSS vulnerability

Posted by on June 12, 2009 No comments

I got a mail forwarded from my current manager. A security scan in relation to our PCI certification had flagged a functionality as insecure, on a medium level.

The scanning tool was able to post URI encoded strings, which could be evaluated as working Javascript. This would enable a malicious user to manipulate with the functionality on the page. The functionality in question has nothing to do with the PCI related parts directly, but if your site has a vulnerability, you can loose your PCI certification all together and the people receiving the scans are not so interested in the technical details, they just react to a red flag.

The fix seemed quite obvious after having understood the problem. I would decode all URI encoding to UTF-8 and then I could encode everything as HTML entities prior to propagating it.

The actual implementation did however give me some headaches. To begin with the data was XML. The data was passed untouched through and processed using XSL to render the HTML result for the client.

So in order to perform the decoding and encodings I had planned, I started by transforming the original XML to a native Perl data structure using XML::Simple for easier access.

I implemented the operations and started working on the translation of my cleaned data structure to XML again.

This was not quite as easy as expected and it took several attempts to get the XML in a condition where it resembled the original, this was a goal in itself, since I wanted to avoid making changes to the XSL. The XML did not completely resemble the original, but it was close enough for the XSL to work.

I then set out to test the complete setup on one of the designated hosts, then things really started to go the wrong way. The build tool I am using is based on Module::Build. The Module::Build version required was not compatible with the perl interpreter installed. After a brief moment of disappointment, I just decided to work around the scripted installation process and target the task at hand.

Next disappointment came swiftly, the new version of the software simply did not work. My use of XML::Simple was based on newer features, not supported by the currently installed version. So I attempted an upgrade. The same problem, the contemporary version of XML::Simple was not compatible with the installed perl version.

After some cursing (which I will leave out here), I discussed the problem with the colleague sitting opposite me. Explaining him the problem gave the solution. I would create a simpler fix, avoiding all the data structure transformations and simply using a regular expression.

I revisited my test suite and mocked important components and finally I was able to see the fix working.

This is the second time I experience and XSS vulnerability. The first time, it was introduced by a freelancer I employed and we got a lot of heat from the developer receiving the PCI scan.

As things often go, nemesis sets in and the new vulnerability I am describing here was actually introduced by the same developer who complained over our previous guest appearance in the security scan report.

The most important lessons learned, must be that we will keep introducing new holes. Things that can help us too keep this to a bare minimum must be a combination of the following:

  • Peer review of the source code
  • Education in understanding holes
  • Security Testing
  • Introduction of general components to close holes and education in their use

We are not particularly good at the above, since it is not practices required by our organization, so everything is left up to the developers and hence their knowledge, time, workload and interpretation of priorities. This leads us to a new problem, which is that we are bound to developers, meaning that when they leave/disappear the knowledge goes.

I will get back to knowledge in organizations in a later blog entry.

Suggestions for additional practices are more than welcome, since this is just one war story and not a particularly interesting one, but I am sure I am not the only one who has a story or two on XSS vulnerabilities or similar and the following reflections we make as developers dealing with these.

I also expect to write up a blog entry more on the actual functionality in use, to share my experiences.