Cognizant Certified DevOps White Belt

 Achieved the DevOps White Belt Certificate at Cognizant

Cognizant Certified Enterprise Developer

 Achieved my Cognizant Certified Enterprise Developer certification

Which witch is which - Availability, Reliability, Durability, Resilience, Responsiveness?

How do you tell how good your software application is?

How do you tell how good your car is? Is an F1 racing car better than your modest homely one? That depends on the purpose it’s being used for. Is a sedan better than an SUV? Is a digital dashboard better than an analog? Again, depends on the users' taste and purpose. However, if you do want to compare cars, there are any number of attributes to consider — quality of build, power, efficiency, looks, price, handling, after sales service etc. Car makers are constantly improving these traits. Are there similar attributes that you must consciously improve when building software? Sure, read on…

Well Architected, well implemented, and well run software

Software is everywhere. How good a software app is, can be gauged through some quantifiable and some abstract parameters. Just as there are many aspects defining how good a car is, there are aspects defining characteristics of software too. I have tried to briefly talk about some of the important ones here — namely — Availability, Reliability, Durability, Resilience, Responsiveness, Performance, Usability and Accessibility.

In many a technical discussion, I have seen people use these terms interchangeably, but they are quite distinct and must be treated so, w.r.t software applications or data. This is a brief article about the distinction between them at a very high level.

Again, drawing an analogy of software with the more familiar car -

Availability

Your car is easily ‘Available’ if it is in a ready-to-go condition, located close enough that you can get to it quickly when needed. You would never have to wait to get going on it. A chauffeur who is ever ready to drive you around and keeps the car squeaky clean makes it a bit more Available :)

In software terms, availability is the ratio of uptime vs. downtime of the application. For example, if your website is down for 1 minute in a hour, it is unavailable for 1.6% of the time (1/60). In other words, it has an availability of 98.4%. H.A (High Availability) is a general term used for applications that have extremely high values. On a cloud platform such as AWS or Azure, the expected availability for HA applications/services is at least 99.99% (“four nines”), which translates to a downtime of not more than 52 minutes per year.

For your car to qualify as H.A (highly available), it would need to have multiple engines (maybe a gas and an electric hybrid?), additional batteries and tyres on each axle, to compensate for failure of anything critical! Alternatively, you would need to have multiple cars, possibly in multiple garages, to compensate for geographical disasters, and chauffeurs working in shifts :) You would also need a second car to drive alongside the one you’re in, so if something happens to the one you’re driving, you can hop into the second car while you’re still rolling without a stop. Big deal huh? But this is how most software is made H.A!

How is this achieved? Typically using the ‘Design for Failure’ principle — do not strive to make your application infallible — instead, accept that failure is normal and design around it. The most common way to increase availability is to have redundant copies of your application running in geographically distributed areas, which may be offline (cold), ready to spin up at short notice (warm), or actively online (hot).

Reliability

Reliability is a quality of being predictable and consistent. If your car needs to be refueled after say 4 hours of driving, you can rely on it if you need to drive for 3 hours, but not 6 hours. In this case the Reliability of your car is 4 hours. If your car feels moody, that is, it handles well on some days and not so well on others, or maybe on some days it gives more mileage than normal, then it can be said to have low reliability.

A software application cannot afford to be inconsistent. You can measure a system for reliability based on how long you can depend on it to serve its purpose correctly without buckling out. The system is expected to not fail, or serve faulty or incorrect data. If it works, it works well, or not at all.

Reliability is measured in terms of how long a system can continue to be available for use and behave as expected (producing accurate data). One way to express this is in terms of MTBF (Mean Time Between Failures). For example, if your website is down for some time every Sunday at 11pm for weekly deployment or optimization activity, then its reliability is about a week (it can go a week without failure).

Reliability is partially related to Availability. H.A systems typically can go several months between failures.

Durability

Durability of a car is well understood — how long you can continue to use it before the engine coughs and wheezes. While reliability is more concerned with temporary recoverable unavailability such as refuelling or changing a flat tyre, durability is more concerned with long term non-recoverable depreciation. Most modern cars have a durability well beyond 100,000 kms or miles, after which their resale value decreases due to expected mechanical troubles.

A car with easily replaceable spare parts is more durable than one whose parts are hard to replace. Regularly replacing the worn out parts of a car makes it endure longer. Similarly, in terms of software, Durability tells how amenable it is to be serviced to continue to meet the requirements and stay relevant (Serviceability). Durable software remains trustworthy, accurate, and usable for a long time. Durability of Software is an abstract concept and is not usually expressed numerically. However, Durability of Data is precisely defined and measured. It is the possibility of losing a file or DB record over time. For example, if one out of a thousand files on a server get corrupted in a year (despite backups), then the probability of losing a given file is 0.1% (1/1000), and consequently its Durability is 99.9%.

Replacing one part in a thousand per year may be acceptable degradation for a car, but not for software. Typical cloud based platforms are expected to offer a durability upwards of 99.9999999% (“nine nines”). For example, AWS S3 offers a Durability of 99.999999999% (“eleven nines”), which is equivalent to saying that you may see, in about 10 million years, 1 file getting lost/corrupted out of a million stored files!

Resiliency

Resiliency is most commonly confused with reliability. Resiliency does contribute to a system being reliable, but is still an independent measure. Your car is ‘Resilient’ if it can work well in multiple conditions- summer, winter, rain, smooth road, muddy tracks, high or low altitude, ethanol or gasoline fuel, with a spark plug worn out, a headlamp fused etc.. Similarly, a software application is resilient if it can work around limitations, deal with situations, and still deliver expected functionality.

A typical example is when a Mobile App allows you to enter information even when your phone is offline, and quietly syncs up with the server later when network is available. Another example is when a system finds its database is down and is unable to fetch data requested by a user, but queues up the requests and responds later through email. When your phone doesn’t have the latest weather info, instead of showing nothing it still shows the last updated weather along with a timestamp. A resilient system can function with same or reduced functionality when things go wrong, instead of hanging itself on a hook or showing a blank screen.

Responsiveness/ Performance

If your car swerves a few seconds after you turn the steering wheel, or responds to the accelerator and brake pedals after a few seconds, would that be acceptable? So, the responsiveness expected from a car’s navigation is in the order of microsecs and not seconds. Its performance, on the other hand is measured by many parameters such as how much load it can carry, how fast it can go from 0 to 60, how efficient it is with the fuel etc.

In Software terms, responsiveness is the ability of the system to complete the requested task in acceptable time. For example, the expected responsiveness of a webpage is typically less than 3 secs for lean pages and 5 secs for rich ones, before the user starts to lose patience and navigate away. It is important to distinguish two similar sounding terms here which should not be mixed up. Responsiveness is a measure of how quickly your app responds to user actions, while Responsive Design s a completely different concept which tells about the ability of the website to present itself optimally in multiple devices of differing forms- desktops, tablets and mobiles.

Usability

You connect with your car through the dashboard. As a good doctor looks into the patient’s eyes to gauge their health, you look into the gauges and feel confident about your car. Do you prefer a minimalist dashboard in your car or an advanced one that makes you feel like a Pilot? Do you think the old school analog dials are good or the digital ones? It’s a personal choice, but brings in a comfort between you and your car. A cluttered dashboard that shows irrelevant info or is difficult to understand, is not useful and takes the pleasure out of the drive.

Likewise with Software — Usability is the ease with which a software can be used by its users. It is an unquantifiable measure of the happiness or satisfaction your users experience upon interacting with your application. Usability is best evaluated by putting yourself in the place of the end users of your system and critically reviewing it — the UI is expected to be uncluttered, easy to navigate, with intuitive organization of content, visually appealing, handleable on mouse and touch screen devices, following a consistent branding pattern, and most importantly helps users accomplish useful work comfortably. Responsiveness helps achieve better usability.

Accessibility

Accessibility is a specific form of usability which is aimed at improving the user experience of differently abled persons interacting with your software. This includes considerations such as using clearly legible fonts, sizes and colors, providing text alternatives to media, making all features available through the keyboard, use compatibility with assistive technologies, and many more. The W3C WAI group defines standards for accessibility levels as well as guidelines for implementing them, and most standard websites built nowadays are compliant to some level of Ax. If you use a standard UI library such as Material UI to build out your UI components, you may already be partially compliant with ax standards.

~~~~~~~~~~~~~ _()_ ~~~~~~~~~~~~~ 

NOS Appreciation Certificate

Sharing an appreciation I received from NASSCOM for the work done last year for developing National Occupational Standards in Web and Mobile App Development areas. If you're looking for Job Standards and Model Training Curriculum for Project Manager, Application Architect, Application Developer, and UX Designer roles, please read here my detailed article about this.

Read This Before Backing up Your Files on Clouds such as Google Drive

Did your laptop just crash? Do you think you can easily restore your files from a backed-up copy on Google Drive? Read on...

In this article I want to share my recent experience on Google Drive, which I think may prove useful to those who need to restore files on their crashed system from Google Drive.

It is not my intent to find faults with G Drive - it is an indispensable service - but merely to highlight learnings that may benefit others in a similar situation.

Will it happen?

For many years I have been relying on a local external hard drive to backup data from my laptop. Recently I purchased a Google One plan and made a copy of my backed-up data on cloud with their easy Google Backup and Sync software. Voila, I had my whole hard drive online! I was delighted that I could now access all my files from anywhere, anytime and from any device, share instantly with anyone, and have a reliable backup in case the laptop or external hard drive failed. I regularly backed from my laptop to my external drive (for portability and handy retrieval) and configured Google Drive to backup the external hard drive on cloud. The minimalistic interface of Backup and Sync software was a breeze. I could even edit my files on cloud directly using the Web Interface or the Mobile App and the changes would sync up with the files on my external drive.

Then it happened!

Three months ago, my laptop conked out. I kept calm and buried it with honors :) I was saved because I had saved (pun :)), redundantly, on the external hard drive as well as on cloud. I got a new laptop and restored the data from the external drive. There you are - all back to normal, or so I thought. While I had not installed Google Backup and Sync software yet, I continued to update my cloud data directly using the Google Drive web interface (https://drive.google.com/).

After I had settled down with my new laptop, I decided to install Backup and Sync, connect my data to the cloud and get back to a sync-up routine like before. Like the rest of our universe, G Drive also follows Murphy's Laws - expect the unexpected!

The data is there...

When Backup and Sync software runs on your laptop, it creates a folder called My Laptop on your Google Drive, and backs up all the data inside it:

So I had my precious data from my previous laptop all backed-up in a cute little My Laptop folder on cloud. When I installed G Drive backup software on my new laptop, I expected it to connect the folders of my external drive to the folders already existing on cloud. Sure :)

...but cannot be accessed on your new laptop!

Guess what, the backup software completely ignored the previous backup, creating a new folder called My Laptop(1) on Google Drive for my new laptop and started uploading the entire data once again from scratch to the new folder!

Old backup doesn't matter anymore

I couldn't find a way to tell Google Drive that my data was already backed up, and that all it needed to do was just connect the local folders to ones already existing on cloud. The simplistic, abstracted user interface doesn't allow for much configuration; most aspects are defaulted with no way to change. Too much simplicity didn't help here!

Drive began uploading the gigabytes all over again to the new My Laptop(1) folder!

Meanwhile, all the updates done to my files on the cloud using my mobile app and/or the web interface (https://drive.google.com) were left dangling in my old My Laptop folder!

Helpdesk were helpless

At this rate it would take me well over 3 weeks of constantly keeping the laptop on and connected, to re-upload a TB of content. Even then, it would not have the updates made directly on the old Backup. That was extremely frustrating and, in my opinion, negated the very purpose of having a cloud backup!

My first action was the most obvious - reach out to customer support. Both on chat and on email, Google Drive customer support confirmed that each new device will necessarily create a separate folder and there is no way to connect it to a previous backup.

They also confirmed that they didn't have a tool that could compare the old and new backups and highlight the differences. Basically they said - backing-up is our baby, restoring is yours!

A feature request to Google

(This part is a bit technical) It would be great if Google Drive could just have folders on the cloud instead of scoping them to devices. Making the backup mechanism device-specific defeats its own the purpose, IMHO. All that the G Drive needs to do is provide a configuration in the backup client that asks whether a new My Laptop folder should be created or an existing one used. Simple and logical.

Ok, so until Google figures it out

I tried out a few ways to work around the problem:

One - Restore from a local copy, wait for Drive to upload it to cloud, then manually move changes

This is the default approach, but not necessarily efficient.

1. Restore data on your new laptop device from a previously made local copy (such as external drive)
2. Install Backup and Sync software
3. Wait for it to backup (upload) all the data to a new My Laptop(1) folder
4. Open the old My Laptop folder online, and check the Activity section. Here you will find a stream of all updates that you made directly using the web interface. It should look something like this:
   

   

5. From this list of changes, consider only the ones that you did AFTER your old computer crashed (those are the ones that did not sync up with your local copy). Download these changed files or move them manually to the new My Laptop(1) folder. These will then be synced up to your new laptop as well.
6. When you are done with moving all changes, delete the old My Laptop folder on cloud.

Note that this needs a long time (even days or weeks) depending on your net speed and data size, as all the data needs to be uploaded.

Two - No local copy? Restore by downloading from Google Drive

If you never made a local copy of your data (such as using an external drive) before your machine crashed, you can-

1. Download all the data from your old My Laptop cloud folder to your new laptop device
2. Install Backup and Sync software
3. Wait for it to backup (upload) all the data to a new My Laptop(1) folder
4. Delete the old My Laptop folder on cloud

The advantage of this approach is that you don't need to move changes manually - they are included in the download from the old My Laptop folder. The downside is that it needs double the time and bandwidth as compared to approach One, because the data needs to be first downloaded, then uploaded in entirety.

Three - Restore by moving the old backup into new folders

This approach involves only download and no upload, and hence should be much faster than One or Two.

1. Create empty folders on your new laptop with the same names as the ones you backed up to your old My Laptop folder on cloud. Basically you replicate your previous folder structure except that they don't contain any files.
2. Install Backup and Sync software
3. Sync up the blank folders to cloud (maybe with just a dummy file in each). This creates an empty folder structure in your new My Laptop(1) folder that replicates your old My Laptop folder.
4. Open your old My Laptop folder on the web interface, get inside your folders and Move each one manually to the new My Laptop(1) folder. This should be easy because you are likely to have only a handful of root level folders 
   
5. Now you will have moved your latest files, including any direct updates, in your new My Laptop(1) folder. They will eventually be synced (downloaded) to the empty folders that you created on your laptop device.

Note that this approach also involves downloading your entire backup over the internet, which may require a long time depending on your speed and bulk, although it will be faster than approaches One or Two.

Four - Hybrid approach

This is probably the quickest approach, and because it is not officially documented by Google, is the reason for writing this article.

1. Follow approach Three all the way until the penultimate step - you will then have the latest files in your new My Laptop(1) folder on cloud
2. The difference is in the last step - restore the files in each of the empty folders on your laptop device from a previously made local copy (from the external drive)
3. Now the backup software will recognize that most files exist both locally and on cloud, and will sync (download) only the differences. The difference will only consist of direct updates made using the web interface or mobile app.

This should be significantly faster than other approaches as this involves moving changed files only.

Summing it up

My takeaway from this experience is that using a cloud backup does not exempt you from creating a local copy. The local copy is highly recommended so as to make restoring easy, even if you are backing up on cloud.

Hope you found this useful! This article may become outdated eventually as new features are added to G Drive, but for now feel feel to comment and leave suggestions if you have a better way of doing this, which may help others. Have a great day!