Monday, December 31, 2012

Project schedule risks & best practices

In his book, Steve Mc Connell has made a useful compilation of project mistakes: http://www.stevemcconnell.com/rdenum.htm

From this list he compiled the top 10 project schedule risk and the means of controlling them such as:

Feature creep (never ending requests from customer for new/change features): change board, design for change, incremental development
Requirement gold-plating (wasting time to polish features while the added value of the extra efforts is minimal): scrub requirements, time boxing, cut the feature list based on time/cost limit
Over optimistic schedule: multi estimation (e.g. ask estimate from several people), negotiate the schedule, cut the feature list
Silver bullet syndrome (a fallacy that a software/methodology can solve everything): be sceptical of claims, measure/test the software.
Short charged quality: allow time for QA (review/test) activities
Inadequate design: have explicit design activity, schedule time for design, design review
Contracting failure: check references, access the contractor ability before hiring
Weak personnel: hire top talents, training, teambuilding

He compiled also a list of best practices based on their efficacy (reduce schedule, reduce risk, increase progress visibility) such as reuse, outsourcing, time boxing, evolutionary delivery, prototyping, top 10 risk list.

One of the interesting best practices is the "top 10 risk list":

1. make list of project risks (e.g. feature creep, the personnel with required skills is not on board, the external party hasn't deliver their interface/WSDL specification)

2. rank the list based on the risk exposure = probability occur * impact on schedule e.g. 20% * 2 weeks = 0.4

3. risk monitoring: update the ranked list periodically (e.g. weekly) with resolution and status.

Rank this week	Rank last week	How long this risk has been in the list	Risk	Resolution	Status
1	2	4 weeks	The personnel with Oracle SOA skills has not been acquired.	Hire an external consultant. We're expecting to have the resource on board in the week 22.	The hiring budget has been approved last week. This week we're starting with interviewing candidates.

The illustration below interestingly describes different options for risk resolution:

Any comments are welcome :)

Source: Steve's blogs http://soa-java.blogspot.com/

Reference:

Rapid Development: Taming Wild Software Schedules by Steve McConnell

Web Service Security: message level vs transport level

Why message-level security (e.g. WS-Security) is better than transport-level security (e.g. TLS/SSL):

End-to-end security: message-level XML-Encryption protects sensitive data also in the intermediaries / external proxies. The point-to-point security TLS/SSL doesn't prevent the intermediaries to read the sensitive data.
With WS-Encryption it's also possible to encrypt only a part of the messages for flexibility (e.g. in case the intermediary proxy need to peek the unencrypted part) or performance (it's cheaper to encrypt/decrypt only portions of the messages).
The message-level security (e.g. WSS Authentication, XML-Encryption, XML-Signature) is independent to the protocols thus it offers more flexibility to send SOAP messages across different protocols (e.g. http, jms, ftp).

On the other hand, message-level security has also disadvantages:

Performance (encrypt/decrypt, validate): processing time & increased message size
Configuration & Maintenance (but can be easier using declarative policy)
Can not peek the message values during development & debug
More complex, more difficult to find developers who master

Transport level security (TLS/SSL)

Security protocol for internet communication with web protocol (e.g. web application, web services SOAP/REST)

Authentication

The basic authentication scheme is by passing the credentials (userid & password) in the http header. This can be improved using password digest: the credentials are hashed (so that the attacker can not read the password) & using nonce (to prevent reply attack)

SSL

Certificates can be used for authentication, encryption and signature (non repudiation)

How

by setting in the web server (e.g. Weblogic, Apache, Tomcat): basically enabling the https listening port and register the location of keystore/certificates.

Message level security

Message standard for SOAP web services security e.g. WS-Security (WSS), WS-Policy.

How:

Java: using handler/adapter to insert WSS header in the request and remove the WSS header in the received response. The handler also encrypt/decrypt the data. Futher info: read book by Kanneganti.
Java using Rampart/Axis2 framework: set security context (e.g. keystore) in the request, define security policy in the wsdl. Futher info: read book by Tong.
OSB: using OWSM by defining policy. Futher info: read OSB development cookbook by Schmutz et.al.

Messages examples:

WSS Authentication using user password token

<soapenv:Header>

<wsse:Security xmlns:wsse=".../oasis-200401-wsswssecurity-secext-1.0.xsd">

<wsse:UsernameToken>

<wsse:Username>myname</wsse:Username>

<wsse:Password>mypasswd</wsse:Password>

</wsse:UsernameToken>

</wsse:Security>

</soapenv:Header>

WSS Authentication using password digest

<wsse:UsernameToken>

<wsse:Username>myname</wsse:Username>

<wsse:Password Type=" http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordDigest">thepasswoorddigest</wsse:Password>

<wsse:Nonce EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary" >thenonce</wsse:Nonce>

<wsu:Created>2011-01-11T11:11:11.111Z</wsu:Created>

</wsse:UsernameToken>

WSS Encryption and Signature

The encrypted data is stored in <xenc:CipherData> inside <xenc:EncryptedData> in the body. The <ds:KeyInfo> inside <xenc:EncryptedData> in the body refers to the key information <xenc:EncryptedKey> in the header. So in this example the key is also sent in the message (in the <xenc:CipherData> inside <xenc:EncryptedKey> in the header) but there is also a scheme where you can reuse the key so you don't have to resent it everytime.

In this example we show also the use of signature, the signature is in the <ds:Signature> and the client key signature information is in the <wsse:BinarySecurityToken> both in the message header.

<?xml version="1.0" encoding="UTF-8"?>

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"

xmlns:xenc="http://www.w3.org/2001/04/xmlenc#">

<soap:Header>

<wsse:Security xmlns:wsse="..." soap:mustUnderstand="1">

<xenc:EncryptedKey xmlns:xenc="http://www.w3.org/2001/04/xmlenc#"

Id="EncKeyId-9B0F450EB80863260412615456546075">

<xenc:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />

<ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">

<wsse:SecurityTokenReference

xmlns:wsse="...">

<ds:X509Data>

<ds:X509IssuerSerial>

<ds:X509IssuerName>CN=CA...<ds:X509IssuerName>

<ds:X509SerialNumber>4</ds:X509SerialNumber>

</ds:X509IssuerSerial>

</ds:X509Data>

</wsse:SecurityTokenReference>

</ds:KeyInfo>

<xenc:CipherData>

<xenc:CipherValue>Pgdgg...cpL4gIc=</xenc:CipherValue>

</xenc:CipherData>

<xenc:ReferenceList>

<xenc:DataReference URI="#EncDataId-2" />

</xenc:ReferenceList>

</xenc:EncryptedKey>

<wsse:BinarySecurityToken .

ValueType="...oasis-200401-wss-x509-token-profile-1.0#X509v3"

wsu:Id="CertId-9B0F450EB80863260412615456546075">M34634347D4=

</wsse:BinarySecurityToken>

<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"

Id="Signature-1">

...

</ds:Signature>

</wsse:Security>

</soap:Header>

<soap:Body xmlns:wsu="..." wsu:Id="Id-16874657">

<xenc:EncryptedData xmlns:xenc="http://www.w3.org/2001/04/xmlenc#"

Id="EncDataId-2" Type="http://www.w3.org/2001/04/xmlenc#Content">

<xenc:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#tripledes-cbc" />

<ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">

<wsse:SecurityTokenReference xmlns:wsse="...">

<wsse:Reference xmlns:wsse="..."

URI="#EncKeyId-9B0F450EB80863260412615456546075" />

</wsse:SecurityTokenReference>

</ds:KeyInfo>

<xenc:CipherData>

<xenc:CipherValue>tdfgdfgdg...yk5H</xenc:CipherValue>

</xenc:CipherData>

</xenc:EncryptedData>

</soap:Body>

</soap:Envelope>

References:

• SOA Security by Kanneganti

Oracle Service Bus 11g Development Cookbook by Schmutz & Biemond et.al.

Developing Web Services with Apache CXF and Axis2 by Tong

Sunday, December 30, 2012

Concurrency risks: thread safety, locking problems, deadlock

Concurrency risks

thread safety / correctness e.g. race condition (incorrect behaviour/result because a shared variable/resource is modified by another thread)
liveness hazard e.g. deadlock (the thread block by a lock), starvation (the thread doesn't chance for its turn since other threads have higher priorities)
performance hazard e.g. locking problems, too much context switching
difficult to test, debug, maintain

Thread safety

The main source of thread safety problem is the shared variable/resource. The solutions:

don't share variable (e.g. stateless servlet)
make variable immutable
use lock (synchronize in Java)

Race condition is the most common concurrency correctness problem. Pay attention for the compound actions (e.g. check then act, read then write). Using thread safe atomic objects don't always guarantee the correctness, the compound actions themselves need to be atomic (e.g. using synchronize).

Deadling with performance problems due to locking

Improper / excessive lock usages can lead to liveness hazard & performance hazard. Tips to work with locking:

reduce the lock scope, for example: synchronize a block of codes instead of the whole method, delegated lock (e.g. using thread safe objects)
avoid locking lengthy operations (waiting user response, contended resource, long distance web service)
don't nested lock
Lock splitting / reduce lock granularity: use different locks for each object instead of using one lock in a long code.
use alternatives to locking: confinement / don't share object (e.g. use local variables, deepcopy to create local copy for each thread, use ThreadLocal), immutable object (final), volatile object.
provide deadlock resolution (e.g. timeout and then choose a victim)

Deadlock causes

Inconsistent lock ordering (e.g. thread 1 acquires lock on resource A then lock on resource B while thread 2 acquires lock B then A, so thread 2 will wait for thread 1 and vice versa) . Solution: provide global ordering (e.g. bash on hash code of the objects) instead of using dynamic/input-dependent order.
Calling an locking alien method within a lock (double locking). Solution: open call (call without lock)
The limitation of resource capacity (e.g. we have 100 requests while the resource pool limit the connection to 10 so the other 90 have to wait). Solution: bigger capacity resource, clustering & load balancer, request throttling.
The thread is waiting for another thread result (that might a lengthy operation or even never finish). Solution: time out, refactor the flows to avoid waiting if possible, use FutureTask so that the computation can be started even before it's needed.

So why learning multithreading?

One famous book in Java suggests that we need to avoid using multithreading in our program due to its intricacy. Well, this is fine if you want to learn Java just for certification or to pass college exams, but in real life as a software developer there are many situations where you can't avoid to deal with multithreading situations, for example:

when you build a web application, typically the web server will manage many sessions serving requests from different clients so you need to be aware for example the effect of static variables in your servlet.
when you implement business process (since typical business process are parallel) e.g. exam papers processing.
when you modify data (e.g. in database) in multi user cases (e.g. via web application that will be accessed by multiple users), typically you need to implement lock during data modification.
multi threading programming is inherent in many domains such as: database clusters, algorithm trading (dispatch multiple buy/sell orders), games programming, multi agents, simulation, peer to peer protocol, high performance scientific computing, multi core programming and many others.

Please see also Software guidelines for performance: http://soa-java.blogspot.nl/2013/01/software-guidelines-performance.html

Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)

References:

Java Concurrency in Practice

High-Performance Java Platform Computing: Multithreaded and Networked Programming

Effective Java

How to reduce risk in big projects

A big project has bigger risk, so to reduce the risk a big project requires:

Better project planning & team coordination. A big project has more dependencies between teams so you need to smoothen the workflow. Pay more attention to communications between teams.
Better to avoid technology risk (use familiar technology & development methodology, hire consultants that have experiences with the technology.) You better to try out new technologies/methodologies in smaller pilot projects instead of a big project. The use of familiar technology/design will also prevent hours of argumentations over architecture details.
Change one thing at a time. Don't try to do too much things once. Use incremental development, deliver smaller improvements early & often.
It's more critical to follow best practices (e.g. standard/guidelines, architecture governance, domain modelling, design patterns, design & code review, refactoring, requirement scrub, test driven development). You can't get away with breaking the rules or take shortcuts such as in smaller projects.
Pay more attention for reuse, avoid duplication of works by different teams.
Pay more attention to QA (e.g. continuous integration test, automatic build & test scripts).
Use available tools/framework for your infrastructure (e.g. Trac for project/bug tracking, SOAPUI for web service tests, Selenium for GUI tests, OSB or Spring Integration for ESB). Don't build your own tools since it will add more risk and maintenance burden.

Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)

Reference:

Surviving Object-Oriented Projects by Alistair Cockburn

Rapid Development: Taming Wild Software Schedules by Steve McConnell

Tuesday, December 4, 2012

Oracle database integration using Advance Queuing (AQ)

AQ is a proprietary messaging technology from Oracle that can be handy for integrating Oracle databases with applications (e.g. OSB, BPEL, Java, even VisualBasic) or other Oracle databases.

The advantages of using AQ (messaging integration style) compared to the traditional jdbc connection (database integration style):

reliability (AQ supports message persistence, retry)
avoid sharing/locking problem of database integration style
messaging integration style is more scalable than point to point integration using jdbc connection

Architecture example

A typical use case is a data push scenario: a change in the database (e.g. employee data) needs to be propagate to different applications/databases.

a queue message is generated in a Oracle database (e.g. using plsql / trigger when an update is occurred)
a jca adapter is made using JDeveloper/SOA Suite (AQ adapter)
using this jca adapter the queue can be consumed by OSB, Soa Suite BPEL or Java code.
the queue can be also consumed by other Oracle databases

The AQ also supports more scenarios such as publish-subscribe or sending messages in opposite directions than in the usecase above (the message is produced by applications and then consumed by Oracle database).

How


Create the payload object using CREATE TYPE payload_name AS OBJECT
Create the queue table using DBMS_AQADM.CREATE_QUEUE_TABLE 
Create the queue using  DBMS_AQADM.CREATE_QUEUE

To use:


Start the queue using DBMS_AQADM.START_QUEUE
Create a payload
Send the queue payload using DBMS_AQ.ENQUEUE


You can see the numbers of messages in the queue using: SELECT COUNT(*) FROM queue_table_name;
You can view the messages payload using: SELECT user_data FROM queue_table_name;

For examples, please refer to http://www.oracle-developer.net/display.php?id=411

For generating jca adapter using JDeveloper and the example of consuming the queue using OSB, please refer to the book" Oracle Service Bus 11g Development Cookbook" chapter 7.

Notes
You need these priveleges: AQ_ADMINISTRATOR_ROLE to create the queue and EXECUTE ON DBMS_AQ to execute the plsql to send the queue.

Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)

References:

Sunday, December 2, 2012

SSO Shibolleth

Shibolleth is a popular open source single signed on (SSO) framework. Using SSO, we avoid too many password accounts for different applications that will increase the phishing risk, ear drop risk and revocation problem (you need to cancel/change all the password accounts in all applications).

Other features of Shibolleth:

federated e.g. you can use a national-wide IdP such as SURFnet in the Netherlands and then SURFnet will communicate with your local (university) IdP to authenticate you.
the application (via SP) can obtain user metadata (e.g. employeeID, email) from IdP
SP & IdP are communicated using standard SAML format

Shibolleth consists of 2 parts: the Service Provider (SP) an Apache-module that resides next to your web application in the Apache web server and the Identity Provider (IdP) server that authenticates the user based on authentication source e.g. LDAP, database, password files, SAML security provider (e.g. Aselect), OAuth security provider such as Google, etc. The SP consists of shibd daemon and shib apache module.

The flow of Shibolleth is as follows:

1. The user browse a web application. In the apache config, this web application is declared as protected using Shibolleth.

2. The user will be redirect to the IdP for authentication. The IdP will use an authentication source (e.g. LDAP) to authenticate the user.

3. The IdP will send the security assertion to the SP (via client session cookie).

4. The SP can obtain user metadata (e.g. employeeID, email) from the IdP.
5. The SP grants access to the application

SSO for webservices

An interesting idea about using SSO Shibolleth for web service:

http://predic8.com/shibboleth-web-services-sso-en.htm

Since Oracle OSB has functionality to access http header, so it's possible to use Shibolleth SSO in OSB web services proxies.

Links

Exercise SP install & config to connect to Dutch SURFnet IdP using Shibolleth:

https://wiki.surfnetlabs.nl/display/surfconextdev/My+First+SP+-+Shibboleth

Shibolleth security checklist:

https://wiki.surfnetlabs.nl/display/surfconextdev/Shibboleth+SP+security+checklist

Want to establish your own IdP? SimpleSAML is one of the easy framework:

Install http://simplesamlphp.org/docs/1.5/simplesamlphp-install

IdP config http://simplesamlphp.org/docs/1.4/simplesamlphp-idp

Source: Steve's blogs http://soa-java.blogspot.com/

Any comments are welcome :)

SOA, Java, Software Development

Monday, December 31, 2012

Project schedule risks & best practices

Web Service Security: message level vs transport level

Transport level security (TLS/SSL)

Message level security

Messages examples:

See also:

References:

Sunday, December 30, 2012

Concurrency risks: thread safety, locking problems, deadlock

Concurrency risks

Thread safety

Deadling with performance problems due to locking

Deadlock causes

So why learning multithreading?

Effective Java

How to reduce risk in big projects

Tuesday, December 4, 2012

Oracle database integration using Advance Queuing (AQ)

Architecture example

How

Sunday, December 2, 2012

SSO Shibolleth

SSO for webservices

Links