A Simple Introduction to the Architecture of Salesforce Platform Encryption

The architecture of the Salesforce Platform Encryption solution is described here.

I thought I’d have a go at writing a simplified version in a way that’s easy for me to understand, starting with the encryption of the data and then moving out to key management.

Encryption Basics

In this post I’m going to assume a certain amount of knowledge about encryption but let’s start with some simplified basics.

Symmetric encryption is where you have the same key to both encrypt (for privacy) and decrypt the data. This is the fastest way to encrypt/decrypt but it is also the easiest to crack and if you lose the key then you’re in trouble e.g. if you encrypt something with the key and someone else wants to decrypt it then they need to have the same key and then there’s nothing to stop them imitating you.

Symmetric Encryption

Public key encryption (PKI) addresses this issue using key pairs. The key that does the encryption is different to the key that does the decryption. The key that does the encryption (the public key) can be made public, anyone can use it to encrypt but only the holder of the other half of the pair (the private key) will be able to decrypt it.

Public Key Encryption

The same public key technology can be used for signing (for authentication). Someone can use their private key to sign something and people with the corresponding public key will be able to verify that the sender used that private key. Public key encryption is sometimes called asymmetric because the encrypting/decrypting keys are different.

Public Key Authentication

Asymmetric security is more secure than symmetric because you don’t have to share the encrypting key and it takes longer to crack but it also takes longer to encrypt and so sometimes the performance impact can be too high. So, typically, a combination of the two is used. The symmetric key is used for the encryption/decryption but its distribution and storage is protected using the public key technology.

Salesforce Security

Salesforce has always been a very secure platform, using a range of services such as encryption of the data in transit, two factor authentication, verification of login address, profiles, permissions and penetration tests. They are now adding to this a new feature called Platform Encryption which allows customers to optionally encrypt some fields at rest i.e. while they are stored in the Salesforce database.

How does Salesforce Platform Encryption Work?

Salesforce uses a symmetric encryption key to encrypt the customer data that it stores. (The symmetric encryption used is AES with 256-bit keys using CBC mode, PKCS5 padding, and random initialization vector (IV).) The symmetric mode gives the performance benefit but means that the key needs to be closely protected. For this reason the Data Encryption Key (which is also the decryption key) is never transmitted or even written to disk (persisted). It is created/derived in the Salesforce platform and never leaves. It is created in a component of the platform called the Key Derivation Server.

Platform Encryption Architecture

So this brings us to the question of how is it created, and how can we ensure that it’s the same when it’s recreated to do the decryption? Also, given that this is a multi-tenant environment, what is the customer specific component? The answer is that the encryption key is derived/created from a combination of a Salesforce component and customer/tenant specific component. These are called secrets. Sometimes they are also referred to as key fragments.

The encryption key is generated from the master secret (Salesforce component) and the tenant secret (customer component) using PBKDF2 (Password-Based Key Derivation Function 2). The derived data encryption key is then securely passed to the encryption service and held in the cache of an application server.

Key Derivation Server

The Write Process

So, to write an encrypted record, Salesforce retrieves the Data Encryption Key from the cache and performs the encryption. As well as writing the encrypted data into the record it also stores the IV and the id of the tenant secret.

The Read Process

Similarly, to decrypt the data Salesforce reads the encrypted data from the database and if the encryption (decryption) key is not in the cache then it needs to derive it again using the associated tenant secret, and then it decrypts using the key and the associated IV.

So, we’ve established that the data can’t be accessed without the data encryption key and that this key can’t be accessed without the master and tenant secrets, but how do we know that the secrets are secure?

Generation of Secrets

Remember that for this discussion, there is one master secret for Salesforce itself, and a tenant secret and key derivation server for each customer. Actually these secrets are regularly replaced, which is why we need to keep their ids.

The master secret is created by a dedicated air gapped HSM. It is then encrypted using the key derivation server’s public key (tenant wrapping key) and signed with the HSM’s private key (master wrapping key) and transported to the key derivation server where it is stored.

Master HSM

The tenant secret is created on the key derivation server, with a different HSM. This is initiated by the customer who connects using their usual transport level security. It is then encrypted with the tenant wrapping key (public key) and stored in the database. The tenant secret never leaves the key derivation server and can only be accessed using the tenant wrapping key private key which also never leaves the key derivation server.

The Transit Key

A unique transit key is generated on the a Salesforce Platform application server each time it boots up. The transit key is used to encrypt the derived data encryption key before it’s sent back from the key derivation server to the encryption service. The transit key is a symmetric key but itself is encrypted with an asymmetric key, created by the master HSM, to get it to the key derivation server.

That’s Enough For Now

There’s a lot more that can be explained. There are more keys for more parts of the process. There are more distribution processes, and processes for updating the keys and keeping the system working using updated keys. There are processes for archiving data and keys, and for destroying the archives. But for now, I think I’ve understood enough to be comfortable with the way platform encryption works and the extra layer of security that it provides. Please let me know if you spot any glaring errors. For more detail please see the original document or suggest future posts.

My Thoughts on Songwriting and Pop Song Structure

A pop song is supposed to carry the listener.  It is supposed to instill emotion into him, to take him on a journey.  In my opinion the most important thing is the structure.  Obviously you can say that you shouldn’t stick to rules, and rules are made to be broken which is perfectly true.  Records that make it are usually ones with a difference (interesting chords, new sounds, clever lyrics) but they can’t be completely different they have to comply in some way with what the listener expects.  You shouldn’t try to break rules before you know what they are.  Music is about psychology, about knowing what the listener is expecting and how to manipulate their emotions.

The average successful pop song is 3 minutes 20 seconds, has 120 beats per minute, and has structure intro, verse, verse, chorus, verse, chorus, breakdown, chorus, chorus, chorus.  You can vary this.  I prefer songs that are slightly faster.  120 bpm is the speed to get girls in white stilettos dancing around handbags.  You can make it faster and people like me or people on ecstasy will like it more, or you can make it slower and people on cannabis will like it more but either way it will probably result in a drop in record sales.

So you have to decide what market you’re going for. My order of priority is

1) It fills me emotion when I hear it

2) Lots of people will like it

3) I can dance to it

4) It is original

English: The Beatles wave to fans after arrivi...

English: The Beatles wave to fans after arriving at Kennedy Airport. (Photo credit: Wikipedia)

Decide what drug you’re aiming at. If you’re into cannabis the chances are that you’ll like a song that drags on and on and doesn’t actually go anywhere. If you’re into ecstasy you’ll like a song that loops pretty fast and goes on forever as you’ll have forgotten what came before within a few seconds. Both the Beatles and the Jesus and Mary Chain have said that they’ve tried writing and recording songs when under the influence of drugs and when they’ve listened to the songs while sober they think they’re terrible. Both recommend drugs as a form of inspiration but that when you actually write you should be sober.

So let’s discuss the structure of a song.  First we need to define ‘up’ and ‘down’.  Going up raises the listeners emotions and going down lowers them.  The best way to do this is with the music.  The standard chords used in pop songs are I IV V. I know there are others but for the sake of simplicity let’s just talk about these.  For example in the key of C the chords would be C, F and G, the first, fourth and fifth notes in the scale.  A song usually starts on chord I where the listener is comfortable, their position of equilibrium.  Chord V is usually the place where they get excited and feel good.  Chord IV is used to lead them on and to prepare them so they’re ready for when they get to V. Obviously this is just a generalisation but we’re starting from the most traditional place as that’s where pop music itself came from.  So the chords of a 12 bar blues song (which doesn’t have a chorus per se but still fits with what I’m talking about) are I, IV, I, IV, V, IV, I. In other words you start from the position of equilibrium and make a couple of attempts to get to the position of happiness, the fact that it takes a while makes the final achievement even better and then you gradually come back down again only to do it again.  Right that was an explanation of 12 bar blues now let’s get back to modern pop music again.  (An example of pop music chords is that the verse is I IV I IV and then the chorus is V IV V IV.)  As I was saying the best way to control the listener’s emotions is with music but there are other ways.  You can gradually introduce more instruments, often at higher octaves.  Bands such as Primal Scream do this.  They take loop or verse or chorus and nothing else they just play it over and over again introducing and removing instruments to take the listener through the 3 min 20 secs of stages that I’ll describe later and it’s an adequate thing to do but personally I find it unsatisfying in the end.

Manic Street Preachers live in London

Manic Street Preachers live in London (Photo credit: Wikipedia)

The majority of successful bands, even ‘guitar’ bands like the Manic Street Preachers use the music to carry most of the emotion and then introduce things like strings at the end to sustain the song a little longer, to raise the emotion a little bit more.  Another way to sustain the song a little longer is to raise the key, so instead of playing CFG you start playing DGA or FCD.  This is often done in Bros songs and in ‘My Girl’.

Primal Scream

Primal Scream (Photo credit: Wikipedia)

The actual structure of a song should do things to the listeners emotions on a bigger scale but similar to what I described above for chords.  The minimum you need for a song is a verse and a chorus (unless it’s 12 bar blues).  You might say that Primal Scream can get away with just one but they only just get away with it.  A band trying to break through has to be better than that.  You might think you need a middle 8 (a completely different tune in the middle).  You can have that if you want but it hasn’t been used much since the sixties.  The listener isn’t really expecting it and as I’ll explain in a minute even though it’s probably a good middle 8 in itself it actually spoils the song.  If you’ve got a good different tune then save it and build a new song around it don’t waste it.  The chorus is the up bit and the verse is the down bit.  The chorus is what the audience wants to hear and the verse is what you use to make them wait, to introduce desire and to explain the song with some lyrics so they have something to think about so the emotion given by the music means more.  A chorus should have a good hook that the audience can learn quickly and sing along with. By the end of the first hearing of the song they should know the chorus and they should want to hear the song again.  Songs based on a simple loop have the problem that the listener doesn’t feel that he has moved anywhere so although he may have like hearing the song he doesn’t need to hear the song again.

So this is it. This is the structure:


All of these horizontal lines must be joined together by smooth lines. Their positioning depicts ‘up’ and ‘down’ as described above. This is the most important thing. The order of chorus and verse etc don’t matter, or the order of notes or chords or instrumentation but what does matter is that you take the audience through this journey of emotion. You might argue and say some records don’t do it but 99% of songs in the charts do it and I would expect that 99% of the songs you like do it. If you do something different to what the audience expects they’re just going to say ‘what the hell was that!’. You probably won’t understand why because the song will do exactly what you expect as you’ll know it but if they don’t know what to expect then this is what they’ll expect. You may say that your song is better and that they’ll like it more when they’ve heard it a few times but at the starting point of a band’s career the fact is that they won’t hear it more than once. In fact on tape they won’t listen to the whole song at all. They’ll listen to the first minute and a quarter or so


and then when it goes to its first repeat they’ll wind it on to breakdown (the most ‘down’ part of the song). If it doesn’t have a repeat (if the chorus goes on too long or if it introduces a third and superfluous tune) then you’ll be starting to lose them. So they’ll listen to the breakdown to see what the rhythm track’s like (just like they’ll listen to the start to see what the singer’s like) and then they’ll listen to a few choruses. If there aren’t enough the listener will feel cheated, if they don’t build up or if there are too many then they’ll feel bored.

So you start with the intro. This is optional. The Jesus and Mary Chain often completely miss out the intro and just leap straight into the song. There are various problems with this. Firstly, when playing live it is very difficult to get all instruments to start playing simultaneously.  Secondly you lose 30 seconds off the length of the record.  You do need the song to be as long as possible as you need the time to get the listener interested in your song and to get it fixed in their head.  Also you get paid by the minute when the song gets played on the radio so the longer the better as long as you keep it interesting. Songs of half an hour are OK if you can keep them interesting but it is hard enough to keep a verse and chorus interesting for 3 minutes 20 sec let alone half an hour.  The intro can be a stripped down instrumental chorus to give them a taster and make them wait but don’t blow it all at once.  Alternatively it can be based on the verse, an instrumental build up to it, but don’t make it too long or they’ll get so used to the music that by the time the singing starts they won’t be listening to it they’ll just be singing the rhythm track and they’ll get very bored.  The music and singing should complement each other. Even if you play a solo over an intro verse which you may think would stop them getting bored with the rhythm track they’ll lose track of where they are in the song and just get disoriented.  They’ll be so into it that they won’t want to settle down and listen to a nice ‘down’ bit of singing.

The next bit is the first verse. This is usually quite simple or held back in the instrumentation for instance the guitar chords won’t ring on or have notes played between them, the drums will not use something e.g. snare or hi-hat so the listener concentrates on the singing.  The music for the verse is not really important in itself although obviously it should be good and have a certain shape to it (usually a IIVI kind of thing).

Next is the second verse.  Establish the tune of the verse, have some more lyrics to give it more intellectual content and let the music go a bit more but still making them wait for the excitement of the chorus.

Next is the chorus. This is the hook. This is what they’ve been waiting for. Don’t try and get any complicated sentences across. The best ones are things like ‘hello, I love you, won’t you tell me your name’. This is really short and yet conveys so much.  Because of the way it scans and the music behind it the listener starts singing along with it long before they understand what it means and yet it is particularly good because it does mean something.  When the listener thinks about it on the 100th hearing they’ll get renewed interest because they’ll realise it has more depth to it than they originally thought and then they’ll go back and listen to all the lyrics again.  The chorus is usually the title.  The title could be something completely different but if someone hears the song on the radio without hearing the title they’ll go to the shop and ask for ‘hello, I love you’, for instance and the shop assistant will say ‘no, we’ve only got ‘made-up name’ and the listener will say ‘I’ve never heard of that’ and go away again.  Similarly if two people are discussing a song one will say ‘what did you think of ‘made-up name’ or even if you just look at the cover and ask yourself the question you won’t be able to remember the song but if you can hear the title in your head with its associated music then you’ll start singing the rest of the song too and you’ll know exactly what it is.  Sometimes people are attracted to a song because of its title because it means something to them already so it won’t take 100 listens to hear the words ‘hello, I love you’ but that still leaves the rest of it to hear.

As I said earlier the chorus should be more up than the verse. So the chords should be higher. If you feel that you’ve used up your quota in the verse then raise the key for the chorus. It should also be more up in that it can have crashing cymbals and may have backing singing (there can be a certain element of backing singing in the verse, but preferably not the first verse, but make sure you don’t use it all up so you have nothing to raise the chorus above the verse).  Backing singing is not only an extra instrument to raise it but it also strengthens the singing making it easier for the listener to latch onto it and get the idea that they’re supposed to sing along.  This should only be one verse long or if you wanted to do it after the first verse then it should only be half a verse.  Musically you may find it necessary to have a bridge between the verse and chorus.  This is perfectly OK as it gives the feeling of building up even more (coming back down shouldn’t take so long though).  The most important thing to remember is that the music is the most important thing.  Don’t add extra song sections to elaborate on the lyrics as the listener won’t hear many of these on first hearing and you want to make sure they do hear it again.

Next a verse again. This can be a go-for-it verse. As much backing singing and guitar riffs as you like as we won’t hear the verse again in the song and due to the music it still won’t be able to compete with the choruses. There’s no harm in repeating the first verse if you’re stuck for lyrics, in fact this may even be better as repetition helps get the listener into it and singing along as soon as possible. While I’m on this subject it’s OK to have a verse that has the same lyrics for the first three lines if the music’s changing underneath it. It’s good for the listener to get a feeling of stability, something to hold onto, while the music is actually progressing. Similarly it is good to keep the music quite monotonic in the verse to give a feeling of tension and wanting to break free and this is OK if the lyrics are changing.

Now we have a chorus again. You should have held back a bit on the earlier chorus so that this is a bit more, but if the music is good enough it won’t matter, but try using a different cymbal or something to give variety.

Now is the breakdown. This is the point where they used to have middle 8’s or guitar solos. Nowadays this is considered to be masturbation, however if it’s a fast song you’re going to need to pad it out a bit more so you could put a guitar solo in before the breakdown.  To strengthen the impact and keep it going without vocals you could double the speed of the hi-hat.  Some people say that a verse is twice the length of a chorus, some say that it’s the same length but the bit before the first chorus should be the length of two verses.  It doesn’t matter but the breakdown should be the length of two choruses.  It is basically just the rhythm track of the verse i.e. just bass and drums.  If it’s simple someone might sample it and then you’ll get loads more money.  You can do other things over it like say a few vocals or make feedback noises but the important thing is that it is ‘down’.  It lets the listener have a rest.  If they’ve been dancing and singing they can calm it down a bit but even if they’ve just been listening you need to get their adrenalin down a bit so that it has more impact when you bring the choruses back in.  The trouble with middle 8’s as opposed to breakdowns is that you don’t let them come down so they don’t feel like they’ve gone on an emotional journey.  The second half of the breakdown can start to build up.

Right now the audience knows the song, you’ve given them enough foreplay and leading on, they’ve had the rest. Now just go for it. Play the chorus over and over so they can sing along, let themselves go, reach their peak, and really drum the hook and title into their head so they’ll ask for it again. First play the chorus just like you did for the previous one. It’ll have more impact than that one did because it’s straight after the breakdown. Then play another with more instruments, and then another with more instruments. Keep going until you’ve filled up the 3 mins 20 secs. You might be getting bored with it and want to stop but it takes the listener that many minutes to get into it no matter how fast the song is. The guitarist might be repeating one simple loop four times for each chorus so it’ll seem like an eternity to him but the singer and listener will just hear a few choruses. You can do other things to make it more interesting like singing new words or the verse or rapping over the music of the chorus.  After that you’ll have built the listener up to such a high that he’ll want to hear it again.  Really all he’ll want is the peak at the end but if he kept hearing that over and over he’d get bored but if he keeps playing the whole emotional journey he’ll carry on wanting to hear it.  So if a band’s songs all sound the same it’s not really a problem as the beginning is still different to the end.

So that’s the structure of a song. It is also the structure of a set.

Connecting Salesforce to a Heroku Database

A popular use of Salesforce is as a front end system of engagement application, using a lookup integration to the more static system of record data in a back-end such as SAP. I wanted to set up a demo to show this but I didn’t have access to an SAP environment so I decided to simulate the back-end by creating a Postgres database in Heroku.

I decided to use some publicly available open data on properties as the database and for this to be looked up dynamically from the account record in Salesforce.

Here are the steps I went through.

First get a Heroku account. Go to heroku.com and ‘sign up for free’


Then login.
On my computer I installed node.js from https://nodejs.org/

And npm from https://github.com/npm/npm

And the Heroku toolbelt from https://devcenter.heroku.com/articles/getting-started-with-nodejs#set-up

Once installed, you can use the heroku command from your command shell. Log in using the email address and password you used when creating your Heroku account:


Execute the following commands to clone the sample application:


You now have a functioning git repository that contains a simple application as well as apackage.json file, which is used by Node’s dependency manager.


Now we create an app on Heroku, which prepares Heroku to receive the source code.


When you create an app, a git remote (called heroku) is also created and associated with your local git repository.

Heroku generates a random name (in this case glacial-sierra-8855) for your app.

The package.json file determines both the version of Node.js that will be used to run your application on Heroku, as well as the dependencies that should be installed with your application. When an app is deployed, Heroku reads this file and installs the appropriate node version together with the dependencies using the npm install command.

Run this command in your local directory to install the dependencies, preparing your system for running the app locally:


Now we add a free Heroku Postgres Starter Tier dev database to the app.


Now we need to add Postgres to the path:


I found some sample data of properties here.

Download the CSV file to your current directory and delete first line so there’s only one header.

Now add an index column; so in Excel insert a new column, put 1 and 2 as the first two entries and then highlight these and drag the mouse down so that it auto populates the other records with increasing indices:



Now we create the database table in the same format as the spreadsheet.


Run a select statement to check it worked:


Set the date format to be UK format like the spreadsheet.


Then copy the spreadsheet into the database:


So now we have an app, a database and a table. In order to access it from Salesforce we need to request that the database be enabled as an external object by raising a ticket at https://help.heroku.com/

You will need to provide Support with the application name (in this case glacial-sierra-8855).

Once external object support has been enabled go to https://connect.heroku.com/

and set the app as a Heroku Connect Instance and get the username, password and url for it.

Select the ’emptyproperties’ data source to share:


Now, to connect to this from Salesforce you need a Salesforce org with Lightning Connect/OData enabled. If you request a developer environment from here then you will get this functionality automatically https://developer.salesforce.com/signup

In your Salesforce org:

  • Click Setup (upper right corner)
  • Click Develop > External Data Sources (left navigation)
  • Click New External Data Source
  • Enter OrderDB as the Label. As you click or tab away from the label field, the Name field should automatically default to OrderDB.
  • Select Lightning Connect: OData 2.0 as the Type.

(OrderDB doesn’t have to be the name, choose something meaningful for you).

Enter the url, username and password from heroku


  • click ‘validate and sync’
  • and select the ‘emptyproperties’ table
  • and select ‘sync’


Then click into the ’emptyproperties’ external object


You should be able to see that all the fields have been picked up from the database:


You can now create a Custom Tab to Easily Access properties

  • Click Setup (upper right corner)
  • Click Create > Tabs
  • Click the New button next to Custom Object Tabs.
  • Select properties as the Object.
  • Click the selector next to Tab Style and choose whichever style you like.
  • Click Next.
  • Click Next to accept the default tab visibility settings.
  • Choose the apps that you want the tab to be included in.
  • Click Save.


Now there will be a new tab:


Click on ‘go’ to view all. All the records are now accessible in the database by clicking through the external ids.


Now go back to ‘external objects’ where you were just before creating the tab.

Now we want to make the index an external lookup. Click ‘edit’ next to index.


Select “change field type”:


  • Select External Lookup Relationship and click Next. An external lookup relationship can link any object to an external object.
  • Select emptyproperties as the value of Related To and click Next.


  • Enter4 as the value for Length and click Next.
  • Enable theVisible checkbox to make the relationship visible to all profiles, and click Next.
  • Click Save to accept the defaults – you definitely want an ‘OrderDetails’ related list on the Orders page layout!

Now go to the properties tab and select an external id and the full property detail is displayed:


Now let’s assign properties to accounts. Se we’ll edit the account record and add a property

  • Setup, customize, accounts, fields
  • New custom field
  • External lookup
  • Select the properties but change the field label just to Property
  • Step through and save it
  • Now when we go to an account we see an empty field for Property


If we edit the field and put an index in it becomes a link to the Heroku properties database


Now the property also shows a link back to the account from the property:


And that’s it. We now have accounts in the Salesforce CRM system with real-time lookups to the system of record in a Heroku database.

A Guide to IBM Bluemix Resiliency and Security

This post was originally published on ThoughtsOnCloud on February 7th, 2015.

I’m pleased to say that it was also published for the 20,000 attendees at IBM Interconnect on Feb 26th.

B-yCORyUcAA8Gq1IBM Bluemix is suitable for high performance, high input/output (I/O), high availability or latency-sensitive production applications, as well as development and test deployments. This is due to the IBM Bluemix configuration of Cloud Foundry within its data centers and the underlying strength of the IBM SoftLayer cloud infrastructure platform.

All Bluemix applications have their infrastructure automatically deployed as required and in real time. For example, if an application is dynamically scaled because it requires extra capacity, Bluemix handles it automatically. There is a full web-based management console and programmable management interfaces, which enable completely flexible monitoring of users’ applications.

IBM Bluemix configures Cloud Foundry in a highly available topology within the IBM SoftLayer data center. All Cloud Foundry components have been replicated to avoid any single point of failure (SPOF). These components include Droplet Execution Agent (DEA), Cloud Controller, router, Health Manager and login server. If any component fails it will be restarted within the data center while the remaining components provide continued availability. Other deployments can become available for the purposes of disaster recovery for IBM Bluemix applications.

IBM Bluemix exploits the IBM SoftLayer cloud infrastructure platform, hosted in data centers with Tier 3 resiliency. IBM SoftLayer provides a compelling set of service level agreements (SLAs) which in turn provide a strong platform for IBM Bluemix technology.

IBM Bluemix is able to exploit IBM SoftLayer’s triple network, which isolates public Internet, private application traffic and infrastructure management traffic. Together with highly redundant servers, each of which has five network cards, and the ability to seamlessly integrate with secure client private networks, IBM Bluemix applications benefit from a highly available and resilient network.

A large catalog of application services is available, each of which typically provides an appropriate range of priced service levels. The service plan will document a priced service level as well as the free service tier. While the free tier provides the ability for developers to try out the functional behavior, the priced levels provide increasing operational quality of service. This

service plan is fully documented with the details of the service performance and capacity, as well as specifying high availability and disaster recovery options. This flexible service approach enables departments to match their development and operations with the appropriate service plan to ensure the most economical mix of service levels.

The IBM approach to information assurance is to provide evidence according to government security principles. IBM Bluemix and its underlying cloud platform infrastructure, IBM SoftLayer, are designed to comply with these 14 principles for all security elements including people, process and technology.

The IBM SoftLayer cloud infrastructure platform has already demonstrated compliance with SOC2 Type II, EU Safe Harbor, and CSA STAR CAIQ and CCM self-assessments, as well as the ISO 9000 quality assurance standard. These standards represent the ongoing commitment to the European Commission data privacy requirements.

From an engineering and support perspective, IBM Bluemix and its underlying cloud infrastructure technologies undergo continuous rigorous security testing in accordance with IBM Secure Engineering development practices. If a security exposure is identified by IBM or a third party, then IBM Support will use the IBM Product Security Incident Response Team (PSIRT) process to apply appropriate and timely updates to ensure the overall system security and integrity is maintained.

As you can see, the security and compliance offered by Bluemix is attractive and comprehensive. Do you think Bluemix is right for you?

Sending SMS messages using Twilio and Bluemix


Here’s an excellent post on setting up a Bluemix app to send SMS messages.

Originally posted on Martin Gale's blog:

I’ve been tinkering with an Internet of Things project at home for a while which I’ll write up in due course, but in the course of doing so have knocked up a few useful fragments of function that I thought I’d share in case other people need them. The first of these is a simple Node.js app to send an SMS message via Twilio using IBM Bluemix.

There’s lots of material on Twilio and Bluemix but by way of a very rapid summary, Twilio provide nice, friendly APIs over telephony-type services (such as sending SMS messages), and Bluemix is IBM’s Cloud Foundry-based Platform-as-a-Service offering to enable developers to build applications rapidly in the cloud. Twilio have created a service within Bluemix that developers can pick up and use to enable their applications with the Twilio services. One of the things I wanted for my application was a way of notifying me that…

View original 805 more words

Sam’s Views on Cloud for Government Policy Makers

I was honoured to be asked to present yesterday on “Cloud Skills, Flexibility and Strategy” at the Westminster eForum Keynote Seminar: Next steps for cloud computing.

English: The Palace of Westminster from Whitehall.

English: The Palace of Westminster from Whitehall. (Photo credit: Wikipedia)

As explained on its website, Westminster Forum Projects enjoys substantial support and involvement from key policymakers within UK and devolved legislatures, governments and regulatory bodies and from stakeholders in professional bodies, businesses and their advisors, consumer organisations, local government representatives, and other interested groups. The forum is structured to facilitate the formulation of ‘best’ public policy by providing policymakers and implementers with a sense of the way different stakeholder perspectives interrelate with an aim is to provide policymakers with context for arriving at whatever decisions they see fit.

The abstract to the session asked about the extent to which Government departments embracing the cloud, what progress is being made in achieving the UK’s Data Capability Strategy on skills and infrastructure development, whether organisations are doing enough to address the emerging shortfall in skills and also asked about the contradiction between mobile device power and cloud.

I was part of a panel and the following was my five minute introduction.

In my five minutes I’d like to talk about the power of cloud and within that to address three areas raised in the abstract to this session – shared services and shared data; mobile; and skills.

We see cloud as being used in three different ways – optimisation, innovation and disruption. Most of what I’ve seen so far in cloud adoption is about optimisation or cost saving. How to use standardisation, automation, virtualisation and self service to do the same things cheaper and faster.

What’s more interesting is the new things that can be achieved with the innovation and disruption that this can provide.

I’ve been working with various groups – local authorities, police forces, and universities, discussing consolidating their data centres. Instead of each one managing their own IT environment, they can share it in a cloud. They justify this with the cost saving argument but the important thing is, firstly, that they can stop worrying about IT and focus on what their real role is, and secondly that by putting their data together in a shared environment they can achieve things that they’ve never done before.

English: The road to Welton, East Riding of Yo...

English: The road to Welton, East Riding of Yorkshire, just south of Riplingham. Taken on the Riplingham to Welton road at MR: SE96293086 looking due south. This is typical south Yorkshire Wolds country. (Photo credit: Wikipedia)

For example, Ian Huntley would never have been hired as a caretaker and so the Soham murders would have been less likely to happen if the police force had access to the data that he was known by a different force.

And we wouldn’t have issues with burglars crossing the border between West and North Yorkshire to avoid detection if data was shared.

In Sunderland we predict £1.4m per year in cost savings by optimising their IT environment but what’s more important is that this has helped to create a shared environment for start up companies to get up and running quickly so it’s stimulating economic growth in the area.

Another example is Madeleine McCann. After her disappearance it was important to collect holiday photos from members of the public as quickly as possible. Creating a website for this before cloud would have taken far too long. Nowadays it can be spun up very quickly. This isn’t about cost saving and optimisation, it’s about achieving things that could never have been done before.

This brings me to the question in the abstract about mobile: “As device processing power increases, yet cloud solutions rely less and less on that power, is there a disconnect between hardware manufacturers and app and software developers”. I think this is missing the point. Cloud isn’t about shifting the processing power from one place to another, it’s about doing the right processing in the right place.

English: GPS navigation solution running on a ...

English: GPS navigation solution running on a smartphone (iphone) mounted to a road bike. GPS is gaining wide usage with the integration of GPS sensors in many mobile phones. (Photo credit: Wikipedia)

In IBM we talk about CAMS – the nexus of forces of Cloud, Analytics, Mobile and Social, and we split the IT into Systems of Record and Systems of Engagement. The Systems of Record are the traditional IT – the databases that we’re talking about moving from the legacy data centres to the cloud. And, as we’ve discussed, putting it into the cloud means that a lot of new analytics can happen here. With mobile and social we now have Systems of Engagement. The devices that interact with people and the world. The devices that, because of their fantastic processing power, can gather data that we’ve never had access to before. These devices mean that it’s really easy to take a photo of graffiti or a hole in the road and send it to the local council through FixMyStreet and have it fixed. It’s not just the processing power, it’s the instrumentation that this brings. We now have a GPS location so the council know exactly where the hole is. And of course this makes it a lot easier to send photos and even videos of Madeleine McCann to a photo analytics site.

We’re also working with Westminster council to optimise their parking. The instrumentation and communication from phones helps us do things we’ve never done before, but then we move onto the Internet of Things and putting connected sensors in parking spaces.

With connected cars we have even more instrumentation and possibilities. We have millions of cars with thermometers, rain detection, GPS and connectivity that can tell the Met Office exactly what the weather is with incredible granularity, as well as the more obvious solutions like traffic optimisation.

Moving on to talking about skills. IBM has an Academic Initiative where we give free software to universities, and work with them on the curriculum and even act as guest lecturers. With Imperial College we’re proving cloud based marketing analytics software as well as data sets and skills, so that they can focus on teaching the subject rather than worrying about the IT. With computer science in school curriculums changing to be more about programming skills we can offer cloud based development environments like IBM Bluemix. we’re working with the Oxford and Cambridge examination board on their modules for cloud, big data and security.

Classroom 010

Classroom 010 (Photo credit: Wikipedia)

To be honest, it’s still hard. Universities are a competitive environment and they have to offer courses that students are interested in rather than ones that industry and the country need. IT is changing so fast that we can’t keep up. Lecturers will teach subjects that they’re comfortable with and students will apply for courses that they understand or that their parents are familiar with. A university recently offered a course on social media analytics, which you’d think would be quite trendy and attractive but they only had two attendees. It used to be that universities would teach theory and the ability to learn and then industry would hire them and give them the skills, but now things are moving so fast that industry doesn’t have the skills and is looking for the graduates to bring them.

Looking at the strategy of moving to the cloud, and the changing role of the IT department, we’re finding that by outsourcing the day to day running of the technology there is a change in skills needed. It’s less about hands on IT and more about architecture, governance, and managing relationships with third party providers. A lot of this is typically offered by the business faculty of a university, rather than the computing part. We need these groups to work closer together.

To a certain extent we’re addressing this with apprenticeships. IBM’s been running an apprenticeship scheme for the last four years This on the job training means that industry can give hands on training with the best blend of up to the minute technical, business and personal skills and this has been very effective, with IBM winning the Best Apprenticeship Scheme from Target National Recruitment Awards and National Apprenticeship Services and Everywoman in technology.

In summary, we need to be looking at the new things that can be achieved by moving to cloud and shared services; exploiting mobile and the internet of things; and training for the most appropriate skills in the most appropriate way.

Using a Cloudant database with a BlueMix application

I wanted to learn how to use the Cloudant database with a BlueMix application. I found this great blog post Build a simple word game app using Cloudant on Bluemix by Mattias Mohlin. I’ve been working through it.


I’ve learned a lot from it – as the writer says “I’ll cover aspects that are important when developing larger applications, such as setting up a good development environment to enable local debugging. My goal is to walk you through the development of a small Bluemix application using an approach that is also applicable to development of large Bluemix applications.” So it includes developing on a PC and also setting up Cloudant outside of BlueMix.

So here’s my simplified version focusing purely on getting an application up and running using a Cloudant BlueMix service and staying in DevOps Services as much as possible.

The first step is to take a copy of Mattias’s code so go to the GuessTheWord DevOps Services project.

click on “Edit Code” and then “Fork”


I chose to use the same project name GuessTheWord – in DevOps Services it will be unique as it’s in my project space.


This takes me into my own copy of the project so I can start editing it.

I need to update the host in the manifest file otherwise the deployment will conflict with Mattias’s. So in my case I change it to GuessTheWordGarforth but you’ll need to change it to something else otherwise yours will clash with mine. Don’t forget to save the file with Ctrl-S, or File/Save or at least changing file.


Now I need to set up the app and bind the database on BlueMix so I click on “deploy”. I know it won’t run but it will start to set things up.

At this point I logged onto BlueMix itself for the first time and located the new GuessTheWord in the dashboard.


I clicked on it and selected “add a service” and then scrolled down to the Cloudant NoSQL DB


and click on it. I clicked on “create” and then allowed it to restart the application. Unsurprisingly it still did not start as there is more coding to do. However the Cloudant service is there so I clicked on “Show Credentials” and saw that the database has  username, password, url etc so the registration etc on the Cloudant site is not necessary as this is all handled by BlueMix.

image015image017Clicking on Runtime on the left and then scrolling down to Environment variables I can see that these Cloudant credentials have been set up as VCAP_SERVICES environment variables for my app. So I just need to change the code to use these.

I switch back to DevOps Services and go to the server.js file to modify the code for accessing this database.

I change line 27 from
Cloudant = env[‘user-provided’][0].credentials;
Cloudant = env[‘CloudantNoSQLDB’][0].credentials;

So we’re providing the high level environment variable not the name or the label.

Unfortunately there is also an error in Mattias’s code. I don’t know whether the BlueMix Cloudant service has changed since he wrote it but he builds the url for the database by adding the userid and password to it but actually these are already in my environment variable url

so I change line 30 from

var nano = require(‘nano’)(‘https://’ + Cloudant.username + ‘:’ + Cloudant.password + ‘@’ + Cloudant.url.substring(8));
to simply
var nano = require(‘nano’)(Cloudant.url);

Now save the file and click deploy. When it’s finished a message pops up saying see manual deployment information in the root folder page.


So I click on that and hopefully see a green traffic light in the middle.


Click on the GuessTheWord hyperlink and should take you to the working game which in my case is running at



However there are still no scores displayed as there is no database table or entries.

I spent a long time trying to do this next part in the code but eventually ran out of time and had to go through the Cloudant website. If anyone can show me how to do this part in code I’d really appreciate it.

So for now, go to the GuessTheWord app on BlueMix and click on the running Cloudant service


From here you get to a Launch button


Pressing this logs you on to the Cloudant site using single sign on


Create a new database named guess_the_word_hiscores. Then click the button to create a new secondary index. Store it in a document named top_scores and name the index top_scores_index. As Mattias says, the map function defines which objects in the database are categorised by the index and what information we want to retrieve for those objects. We use the score as the index key (the first argument to emit), then emit an object containing the score, the name of the player, and the date the score was achieved. Following is the JavaScript implementation of the map function, which we need to add before saving and building the index.

function(doc) {
emit(doc.score, {score : doc.score, name : doc.name, date : doc.date});


Again, we should really be able to do the following as part of the program startup but anyway, the following should add an entry to the database, replacing guessthewordgarforth in the URL with the host name you chose for your application:


You should see a success message. Enter the following URL, again replacing guessthewordgarforth with your application host name.


The entry you just added should appear encoded in JSON e.g.


So, the code and the database are working correctly. Now it just remains to play the game. Go to


(replacing guessthewordgarforth with your hostname)

This time it will include Bob in the high score table


and click on “Play!”


My twitter feed


Get every new post delivered to your Inbox.