How knowledge becomes a blocker when you design products for end users

If you are the one having a background of working for big companies in the past, having worked on large projects, having met a variety of customers, written 100s of functional, architecture documents, travelled across the globe, having built small and medium sized products, having handled small to medium sized teams, having worked on numerous state-of-the-art technologies,

does it all help when you build a brand new product for end users like you and me?


I feel they block you from seeing the simplicity you need to reach ultimately in your design.


They block you to the extent that your brain constantly brings in ‘complexity’ of extreme nature ultimately making you roll into your comfort zone of what you were always doing. If you are a techie, you think the ‘How’ to solve it rather than what is needed for that end user or who is that end user. If you are a business guy, you weigh everything with money and business models and break even and so on without understanding how beautiful something can be conceptualised which makes life easy in selling.

The end user is that ordinary person who would like to look at things that are ‘cool’. Things that give them some value no matter how big or small. For example, my 5 year old goes on using the ‘Talking tom’ app pushing the cat, rubbing it and seeing the facial changes and the sounds coming out of the cat in the App. This is such a thrill for him. It is not to say you design everything like a game or entertainment, but you got to find those ‘nice’ and ‘beautiful’ things beyond the mundane even if you are designing a spread sheet.

The small things that are haunting have a big effect than the big things that you have no choice but to use. You want to get back to those as often as possible.

This is why I feel you need to completely unwind and become a ‘empty cup’. This is not a button you press and you are degaussed. This takes time and an enormous amount depending on how much assumptions you have built about yourself over time. If they are too many, unwinding becomes that difficult. It may sound like the philosophy of OSHO.

Dropping what you have done and taking a fresh look at the problem in hand, every day, every moment is all that is needed. Constantly questioning your own assumptions and trying to break free and see you are in an unbiased state is often needed. If you continue to do this you will eventually have unwound with a pair of fresh eyes.

Why is it difficult to understand code?

If you are asked to read and understand a software that spans 10000s of lines of code, it is impossible to do it. Even if you spend months, you may be able to fit some pieces but will not end up getting a hang of what the code does.

This is the real bane of software. In fact this is true even for some one who has had a long gap and end up staring at his own code.

This is true, no matter how much of annotation, commenting, documentation that is there.

Code commenting and documentation is only a small step in the direction of understanding one’s code. The reason is, a code is something that changes so drastically during its early phases. Almost all the parts of the code is volatile like a molten lava. Needs change, perceptions change, how you solve something change. It is hard to keep up the documentation or commenting in line with the code change and it will almost always be erroneous.

Coding a large software is like carrying a model in your head which is having several moving parts. It is like how Einstein would have carried his view of the universe and all the lingo of general relativity and associated visuals, albeit in a small way. No other person can understand it unless they get this model right. No matter how much of a documentation is out there.

Over time, if the product survives, portions of the code gets stabilized. If the design decisions, the perception of customer needs are reasonably accurate, the initial core of the code remains somewhat intact. This is really the time to ‘document’ the code. This is when the model in the head of the programmer can be laid out in diagrams and words and comments and algorithms to make sense to others who may enter to maintain or enhance it.

I would say the initial period of development of anything should be by a team which is sitting next to each other. Until it reaches the ‘inner core’ level of stability there is no need to document anything. Everyone simply codes. Everyone talks and listens to one another and grow the model in their heads.

Teams spread across require specs to be developed and integration etc. to be planned. These are like turning an already muddy water. If there is a choice, never have your initial development spread across teams. This is because, unless two brains are in synch in terms of the language spoken, visualizing the model there is always a gap to be filled up and time to be wasted and mistakes to be made and frustrations to be built up. And two brains to be in synch is not a question of skill, but a question of attitude. Hard to get.

Even if there are documentation after reaching the stability, it is nothing like having the original team around. If you happen to lose the key members, ensure their video lectures are captured that gives the ‘model’ that was visualized, key algorithms used, key decisions, assumptions made all along. This can help the developers to latch on to the context of looking at the code.

What is so difficult in building a successful product?

After spending years on building a product, pivoting, iterating, which is yet to be launched, I can share some of my thoughts about this question.

Idea is the first step

You have a interesting thought of doing something in a way that is elegant, cool and effective over how it is being done today.

This is the starting point.

Exploring the idea

Here you start expanding, imagining the different aspects of the idea itself. It is a very free format thinking that happens throughout the lifetime. This can branch off into other interesting ideas and often you may leave the original idea and find something more meaningful and interesting.

This phase can involve prototyping, talking to customers, finding new use cases, finding customer segments, finding new ways to solve the problem, innovative thoughts, usability aspects, customer delight, engineering challenges, marketing challenges, scale and so on.

This phase can go round and round in iterations. Often people say to get traction, you should start building something with few potential customers in mind and then keep iterating until you can see that the existing set of customers are able to use it easily, find it is useful and move on to increase the base.

It is often a way you iterate your product while listening to its first set of users and adjust / re-adjust to see that it is exactly meeting the real problems of the users. It is quite possible that you find the problems you originally thought or not the real problems and the real problems are way different.

For example, when I built a collaboration tool for businesses and showed it to few business people whom I had in mind while building it,  I realized most of them do not use computers and they just keep track of their biggest orders without worrying about other smaller ones. So the problem of a what I perceived as a collaboration on various orders that are happening simultaneously suddenly fell apart as they just focus on few big ones for which they do not need a tool.

Of course, it is quite possible that such a need may be there with a retailer who is dealing with a huge amount of inventory and collaborating with his suppliers. But you suddenly find that the product you envisioned as a broad collaborative tool that can work for any one is narrowing to a segment.

This cannot be figured out easily until you are deeply connected to a domain like healthcare or education or others. If you are a generalist being a software guy, your field of vision does not have those specific problems outside computers. You end up imagining those domains and often get them wrong in terms of the real problems faced by the people who are there.

That is why products that are built for domains do not work for other domains. Or those that are generic in nature like a email which works for everyone does not solve the domain specific problems without adequate customization.

It all depends on which direction you decide to travel. A Google or a Facebook deals with the domain of information and communication which is a horizontal concern. If done well, they can be used in a domain agnostic way. But the catch is, finding such a broad based problem and seeing it work for all requires understanding the most fundamental needs of humans and trying to find the most simplest way of solving it. This can be done but requires a lot of research, lot of experimentation and perseverance.

So this part of exploring the idea and bringing it to a way for end user consumption is a long iterative phase. You cannot call your successful here just by raising funds or bringing in the first 100 customers. It can die even after getting your first 100 customers.

The most important thing to do in this phase is to constantly iterate and validate until you are confident that it works for the small subset and it can scale to a larger set.

Building the product

The actual product comes into existence only when there are customers already using it and demands more and more out of that little thing you originally started with. This is the phase where things start to get bigger. Engineering , customer support comes to the fore. Till this point, you have to be nimble and should not bother about anything else other than getting the product and the market fit right.

This should be the least worrisome phase in my guess if I ever get to this.

Some mistakes we make

We often think that the product should be built with a lot of focus on engineering. While this is not wrong, it is important to see who the hell is going to use this and how big is that ? If this is not addressed the effort put in to make something goes waste. Early on getting the feedback of target users is a great thing to have.

If you are building something very generic, often you will struggle to find this target user. You end up thinking every one can be a user. For example, if you are designing a new email thing, you may think everyone who is using email today can use it. But where you may fail is, if you are not showing something unique and differentiated then existing people would not move into yours. On the other hand, if you build something special , say it is more useful as a CRM type of email solution, then you better focus on businesses who are overloaded by their customer communications and would like to try your thing.

Going for funding initially is also a waste of time. This can distract you a lot from what you want to build. The best is to build something, find some target customers who can try out and correct things, iterate and keep trying to find what is so unique about what you have built and how easy it makes the life of your target users. Because time is a precious commodity, you may want to use it sparingly for stuff which drives you out of the main goal which is stated in the beginning of this paragraph.

Going for patenting is a waste of time. How much ever innovative your idea is, being a software thing, unless you have invented a algorithm that is so fundamental to solve that problem, spending time and money on patenting stuff on the overall idea and the way you solve it is a absolute waste of time. You may find that there are so many variants of your idea already floating in several states of maturity. Patenting is a time sucker. If you have time, then go for it. It helps to understand and broaden your thoughts without a doubt, but it does not help you to validate your idea ever. I also feel being a startup, patenting cannot be enforced even if some one violates because you simply do not have that financial muscle. On the other hand, if you have done something smart and even if you do not have customers, you can be a potential buy out by a corporation that is desperate to get a head start in that direction.

Update 1 on patenting - One of my friends told he had to face a lot of trouble when they designed a internet based backup product from lawyers representing other companies as they did not patent some of them. It was a horrendous experience. So, I would say that if you have a non-obvious way of solving a problem which gives the best result, then it should be considered for patenting.  So, my original statement of 'patenting as a waste of time' should be figured out by you as to how non-obvious/novel or trivial the thing you are trying to do.


Giving up too early - Either you do not keep good health or a good bank balance or you simply do not have the persistence to try out things. It is important to keep yourself steady on health, finance and mental strength to be there , float, try out and see what opens up for you . If you read Papillon or South you will know what I am talking about. It is all about endurance and your ability to float against all odds. You need a large dosage of hope and be able to guide you to see the light at the end of the tunnel. You may be lucky to start seeing it early or can go on forever. That is the reason why you start somewhere , read the pulse of the target customers, pivot, iterate and so on. Often it is possible you may land up with a big idea in the process if you keep at it. Or in the least, you may know the area better than any one else leaving you other possibilities.


I feel if you keep at it, you will get it right some day.

Listening to the wrong folks for advice.

Like the way you figure out the right customers for your product, you also need to figure out the right set of people who can advice you on. You need to keep miles away from people who have never built a product or taken it to customers all by themselves. Every one has a view of the world and it is completely your discretion to identify the noise from the signal. Most of it is noise. Almost all of them. The signal can only be seen by you.

No one can tell that a idea will bring you million bucks or it will fail. It is as hard as finding water on Mars.

Glamour quotient

Building a product that works for a broad set of people in the world is not a glamorous thing. If you came in looking at the glamour of a startup as touted everywhere, you are doing a disservice to yourself and to the world. I feel it is a deep meditative thing. You are absorbed entirely and you cannot be filling your brain with anything else other than getting the best out of your brain for the problem in hand. It is really tough and cannot be anything that you have ever accomplished before.

It is the next toughest thing than all the previous things you did in your life.

s/w product development and the building block analogy

Developing a software product is akin to building a house. You have a plan when you start building the house. With computer models, you can pretty much visualise the entire house, space, things that will be there like sofas or ward robes or others and get a feel for the entire house before you construct. 

Why is it not the same with software? We still have this practice of drafting requirements in English which the author himself would find it hard to read after it is written. 

There are models based on UML as well which may be better than text. There are user interface mock ups which give an idea of the front end. But still the programmers who are essentially the masons and carpenters and plumbers of the code that builds the product has no way to visualise the entire house the way the plan shows. 

You may say that the masons of a real building may not know the entire thing and it is the civil engineer or the architect who has this mental model of the entire building. Agreed. But this is where a big hole is left open in software. Building a wall which forms part of a bigger plan is known up front when a building is constructed. However, in software this is not really true. The reason being the requirements themselves are vague many a time on a document. Further, the design based on UML or other diagrams are not frozen before coding begins. Though the waterfall models strictly mandate this, there is always a overlap of phases and with agile models and go-to-market strategies things have gotten really worsened to the point that you often end up scrapping the whole effort done earlier to begin a brand new build out. And we come out unscathed giving justifications for such acts eventually growing the mistrust on the programmers guild.

The biggest difference I see is, the requirements for a building are easily visualisable. And hence a wall that is done somewhere in one corner of a building is easily visualisable as well. If we ask why requirements for a building are easily visualisable, it is primarily due to the fact that you see houses all around you all the time. People right from when they are kids have a sense of buildings around. They play with building sets, they build castles on the beach sand and we live in them. So the sense of buildings are easily ingrained in us. It is not something like an effort has to be made to bring it into a visual form. Of course, complicated structures, aesthetics and so on may require hard thinking. But a engineer or an architect can do this in his head and bring it in a virtual reality model using a tool that can be seen and visualised easily by others. So it is easy to freeze the needs between the consumer and the producer.

If you take the case of software, the consumer varies in degrees in terms of the knowledge of the domain and may not know what software can or cannot do. Same is the case with the software guy who does not understand the business side to be able to speak that lingo with the consumer. There are nuances in every domain which only the experts in that domain understand. It is mostly a case of people, processes and the way businesses have been going on which is what the software tries to replace in parts or whole. How do you model this like the way you visualise a house?

If we take the different components of a software, each of them may expose interfaces for a consumer. Say, we are designing a software that brings together people in healthcare like patients, doctors, hospitals, diagnostics labs and so on into a platform where they exchange things. Now to solve this, if a graph database was chosen, which exposes APIs that allow the user to add, link , traverse and so on. All of this makes sense for the software guy. But how about the healthcare folks? 

If there is a visual representation of the 'business data' of the health care domain shown to enter a box representing this graph database and the data coming out of it or the ability to see the connected graph of the business information for one or more use cases that is produced as a result of interaction with that box which is the graph db, then everyone involved including the person constructing it will not miss the point. However, this can still be missed by the domain people who do not get what a graph db is or how it helps their case. How do we bring these two people together? 

This is where I feel we can use the analogy of a building itself which is familiar to both the parties and which in parts can be likened to software construction.

What I say here is a sort of convergence of the business domain and the software architecture / design. The software architecture is not done by having components and connections and interactions and flow diagrams independent of the domain experts. The business domain experts can make sense of the flow of the information into the 'user interface' which is the most visible part and be able to see changes in the user interface. 

The underlying parts which result in the computation are all now shown as a person walking in a house from one room to another, from one floor to another and coming out. It is sort of a model of a house with floors and rooms where the data payload is a person who starts somewhere say ground floor in a room which is akin to the component that exposes a REST API and starts going into the different parts of that room say cubicles that does some transformations to the payload which is the person himself. The person's look and feel changes as he walks through cubicles and rooms and floors which are the different layers of the software. The person may grow in size that is akin to the payload growing while traversing a room of a certain layer for example. The person may come out of a cubicle or a floor or a room based on O(n) computations that happen within them. If he takes more time, we know there is a large amount of computation necessary or unnecessary as a next step. 

This kind of visual models like a house, building and people where we are more comfortable to associate with than the flat diagrams with its own notations can often give a quick sense of what is going on for every one right from the business folks to the coder responsible for a module. I some how feel that this can be done with Lego bricks. The only difficult part is the morphing of the person walking along as he visits a room or leaves a floor with the computations or impacts created in that place. Even if there is a 3D model of this in software it looks like a interesting and highly potential software tool that can pretty help to accelerate the way we architect software in the future. 

How SMBs can grow bigger?

I have spent a lot of time observing the way small businesses work in the Garment sector in Tirupur, India. There are 1000s of small and medium sized units that churn out millions of T-Shirts, tracksuits and the likes in a day for their customers predominantly in the US, Europe, Japan and others.

For an outsider, it is a wonder how these units work together to come out with a shipment that includes quality checked products. It is a massive team effort crossing different skills, different domains and different cultures to mass produce, ship from India and sell them in Europe.

Communication happens across board right from physically visiting a 'Dyeing unit' to check whether the cloth that was given for a certain order has been dyed to electronic medium like email, voice, spreadsheets, specialised Apps for communicating the designs, optimising the cutting of the reams of cloth or making the designs on the T-Shirts like embroidery or printing and so on. There are innumerable possibilities that arise with every order which can be as crazy as a certain type of button to be fixed somewhere near the pocket with a certain size and colour. I am just sounding an example to show you how 'things of art and fashion' identified and designed by some one , some where in the world is translated to 'things of labour' and 'mass production' and the numerous steps involved in getting them to be shipped finally.

I believe the above is the nature of most of the SMBs in various other domains, be it, ancillary parts manufacturing for automobiles or even a logistics company sending off its trucks to deliver goods. They all have innumerable interactions every day within the business, across businesses.

Many of these SMBs go through the cyclicals in revenue realisation, labour management or government policies on exports or others. Because of the ups and downs, the sector (mostly from what I have seen in India) is highly unorganised, chaotic and most often huddled into groups that support one another informally often disallowing new entrants.

Most of the SMB owners (at least the ones I have seen) are not tech savvy. They absolutely rely on paper and pen and mostly memory for information management. They appear with flashy phones and tablets mainly to show off their status in a business meeting with others. But the owners are completely ignorant of how software can be used to scale their operations.

It is also partially true that the software folks who have created things for them have failed to understand the exact needs that would make the life of these business owners easy. Many a time, the software is extremely unfriendly with 100s of options and features and menus that acts as a deterrent. On the other extreme lies SAP which covers a lot of ground, but again lacks the ease of use or the tons of configuration to set it up apart from the cost. Custom software suffers from too much of pointed solutions often requiring the owners to go back to the software company to keep patching things. The tools that seem to be used widely are Email and spreadsheets and PDF files.

It is very clear that a lot needs to happen here. Especially software that is not as hard and pricey as a SAP while not as specific and patchy as a custom build or as generic and lot of labour like Email or Spreadsheets to come to the rescue of these people.

Collaboration like a social networking tool, Organisation of information and retrieval for both structured and unstructured data from documents to pure structured business data that can be done by any one easily like the way they organise files and folders in the paper world but having the advantages of a digital world which can relate things and search or query, access controls at different levels within and outside of the business and discretion in sharing of information....

This is a serious want, a big want and a complex one to accomplish.

This can alter the way SMBs work enabling a whole new information supply chain, taking to a new level the scale and mainly ease of doing things. A tighter integration of information and data collaboration and standardisation of processes and communication across the loosely coupled SMBs can make them suddenly look like a single entity that can stand up to challenge larger businesses. This is a great model to achieve as it has the power of loosely coupled independent entities that are geared to achieve bigger goals.

coding can be punishing at times...with a stress on AngularJS

What I mention here must be extremely familiar for coders. I come from the world of 'C' language in the early nineties. There were very few tools to use. I had to code a linked list for every small thing. I happily did it. Did not complain. It may appear stupid now. But that was how the world was. You had to write every bit of your code.

The scene now in coding is a lot different. There is a lot more collaboration due to open source, stack overflow and of course Google. Every question is out there asked by some one and answered by a bunch. While this is all good time for coders, the number of frameworks and tools that are available are immense.

I tried my hand with Servlets, JSP, JQUERY, JS, AngularJS and so on and on. I have spent more than a year with AngularJS now. I have very mixed opinion about it not from what all it can solve but the information that is available out there for coders from the community in variety of ways. So many different voices and so many different ways of explaining the same thing. I am afraid what I write here can again spin off into discussions. Contrast this with K&R C language. One and only authoritative, precise and clear book to learn 'C' language. Of course, there were other books as well. But language which solved pretty much the syntax and semantics of programming constructs could very much be expressed in a small book. How to solve problems with that was elucidated in many other books taking real world examples. And that's it.

Now, with the web, every one has become an author. And there are these 1 million plus links thrown by Google for everything. So how on earth you figure out simple, clean answers for your questions? The answer is fairly simple. Just get lost. I mean get lost in the sea of information out there. The price you pay for democracy. Millions of voices, zillions of things. See already I digress from what I want to say.

AngularJS has code examples and demo examples everywhere. It seem to be doing wonderful if you are person who has done enough Javascript in life and can understand the design patterns of Javascript, IoC, scope hierarchy, dependency injections, Object oriented JS, pub-sub-notify etc. However, there is no single thing which explains you the fundamentals of the these things and then talk about how AngularJS has brought in these aspects. There is a sea of demos and specific examples. The angularJS book by O'Reilly is definitely not something that covers the fundamental aspects in a simpler way for all to understand. That is the only book I have seen. The syntax is very convoluted and unreadable many times. It only makes sense after you spend a year on it. That too you get confused with the constant mixture of square brackets, curly braces and the $ signs.  It is definitely not for the weak hearted.

With so many different variations available to solve some problem, it amplifies the pain when someone's aim is to solve a business problem burning in their head. The syntax can be frustrating and can cause significant delays in schedules if you are new and picking up. And no one who comes on the videos from the AngularJS team or others who have supported this seem to bother about making life simple with easy to read syntax.

Added to this is the constant chatter about testing. It is great that testing everything eases your life. Towards it AngularJS has a steady supply of tools from community like Yo, Grunt, Bower, Karma, Jasmine and so on. I have been of late trying to use some of these. And specifically I have found that like using the AngularJS framework, these tools are also have lack of simple documentation explaining why they are there, how they are related to one another.

For example, Yo is a scaffolding service to create a boiler plate layout of your angular files plus it interacts with Bower to download dependent libs and Grunt  to run tasks. This can be explained with some simple visual diagrams somewhere rather than reams of text or demo examples that dive into the configuration aspects immediately.

There is no doubt that they make your life simple once you are familiar with them. But as I said, it is not for even for the newbies. They can shy away seeing the number of things to try out and the time it takes initially.

I request the authors of these tools to put out simpler and more fundamental descriptions with simpler English , visuals and examples rather than just pieces of code with demos with absolutely no explanation of them expecting the reader to make out things on their own.

To give an idea of a better explanation I give this link on AngularJS scopes. This is more fundamental, clear with pictures and covers a number of scenarios. https://github.com/angular/angular.js/wiki/Understanding-Scopes

I am not against AngularJS if you feel I sound so. I salute the team for their efforts, the community for jumping to solve a problem free of cost and the numerous ways they have made life better in solving the HTML/CSS/JS entanglement. I use them, have benefited by using them. The only complaint I have is the initial complexity in understanding them (when I say understanding, I mean making sense of them in a fundamental way) to be able to code complex behaviours for a newbie is bewildering with too many jargons and the issue with internet scale collaboration.

There needs more improvement in the content of the documentation around these new tools which are coded by some of these smart people around who end up answering questions on stack overflow rather than putting out documentation on why they envisioned such a thing and how they have solved in a way everyone can understand.

To end this, I will give a example of what took me a day to fix. I was using Yo/Grunt to run Karma tests on a Angular directive. The mistake I did was in the karma.conf.js I kept the order of the files with Angular JS files after my own scripts. This ended up not $compiling my directive and as a result there is no HTML generated for me to test with JQUERY find etc. This is probably a simple , stupid error for the people who have authored these tools. Also, this may not be anything to do with the AngularJS or Karma. It is probably the way dependencies get loaded and resolved in the JS world. However, since I was new to all of these, I went hunting into everything from the syntax to config files to stack overflows to Google and trying out tirelessly, mind numbingly the different options I had until it struck me from the error that the HTML of the directive is not getting generated.

Man! Life should be better than this any day.

The Broken WEB

It is just that I have been intrigued by the divisions I see around in life and how the mind thrives in these divisions. Divisions are dangerous and we are seeing it everyday in politics, office and even among us in friends, family. Humans have this penchant for dissecting and making sense. Dissecting is a double edged sword. It can be used as a tool to analyze and at the same time, it can arouse passions and lead you into unwanted confusions. The problem is a weak mind cannot know the difference between when it ended the analysis part and when it entered the realm of the unwanted. This sounds like philosophy right?

I see the same thing in software. Microsoft, Google, Apple all have been delivering this so called Apps which are divisive. They focus on a single function. They deliver that effectively with a great looking UI/UX (the Apple way). They go on developing the Apps like this, millions and zillions.

Now compare this to the WWW of the past when it originated. It was a beautiful, marvelous discovery in my opinion. Hyper links unified the way you read documents. Even today, unless you watch yourself, you can get thoroughly lost in your browsing. I have seen this happen with me many times. I do not know why or how I ended up reading a page which was never what I started with. Hyper links unified the sea of documents.

Unfortunately the documents are a weak form of representing things. It caters to the form and loses the underlying structure from where it originated (the databases). In the quest to present in a form that is readable, the documents lose vital information. Thus the very strength brought in by the hyper linking which links documents breaks mysteriously because there are these finer things in the documents which cannot be linked to precisely.

With time as the documents started representing not merely some text, but company financials, books, weather and so on, the hyper link broke and Google came in :-) Google emerged as the great leveler, the one that fixes the breakage. It brought in search which kind of fixes the hyper link problem indirectly.

I will give you an example, I look for 'lambda calculus', then I find the profile of the lecturer, OK I find a link and I go to his page, then I find some other subjects, say 'Haskell' which are mentioned there and get interested, but I cannot go to them. I go to Google and look for those. Google has figured out the linking between these disconnected pages and it has allowed me to browse again with a brief interruption. You may think it is brief. But I feel it is really annoying. I use a App (Google) to fix my linking when it should have been possible for the lecturer to have linked it or for that matter if it was publicly editable it should have been done by some one. But how does the lecturer know the web site that has the information about 'Haskell' without doing a search? Why was this not solved by the inventors of the WWW which seems vital for the hyper links to function? OK if I do not want to go there, I will have to still answer how will the lecturer or some one else know which web site to link?

I feel the only way is when you have the web itself converted from the web of documents to the web of things. Things are not just data or text. They embody the description about the data as well. They carry meaning. One aspect of describing a thing is also the 'categorization' of the thing as to where it belongs to in a hierarchy of things in the world. If only we had a web of things, then it was fairly easy for the lecturer or any one to look for things that belong to the category of functional programming and find pages of Haskell categorized there and be able to make that Hyperlink.

This way, the web would have remained powerful and would have been a truly knowledge base of the human beings or in short the collective consciousness of us. But as history would want it, we had a more ugly way of fixing this by giving the rights to fix this to the great company called Google. I admire them as they probably were truly annoyed by this breakage as well and looked to answer this sincerely.

However, I feel the root cause is in the way we create things. It was form based and not structure based. It was about putting up a document in the human cognizable way and leaving it for Google to discover it than being able to present the underlying structure. Form falls into the Apple space. That is what they mastered and took it very far. And many others followed. But Form is as I said very divisive. It caters to the human mind very well. It allows you to see things distinctly without a way to know that underneath every thing is related to one another. Form destroys structure in the process of delivering something comprehensible for the human mind. I am not against forms. I feel they are good, but not to the extent where they can destroy the underlying structure of things nor the connection between the things.

I see this world crafted by these two big companies. On one side is Apple which has focused on Form , brought App store, zillions of Apps to cater to every need of you not allowing to realize the connectivity across these sea of Apps because they focused on the Form and make you focus on the Form.

On the other hand, we have Google which is helping you relate the underlying structure, but in the process it did it in a way allowing the lazy human way of typing text for everything and destroying the underlying structure. It did the grunt work of analyzing things and allowing the humans to continue in their realm of text or Form which they are comfortable with.

Thus one has blatantly sidelined with the Form and another has not looked at alternate means and allowed things to continue as is. The net result is, we have a BROKEN WEB and that continues.

I believe the original inventor of the web had realized this in some sense and brought in the Semantic Web and all the W3C work that followed it. However, what they miss is that the damage has happened and I see light only in the micro formats. But that is not again going to be seen as important for a long time to come by the web authors as it looks too late into the game and there are already zillions of web pages without them.

Coming back to philosophy, the divisiveness has to be fixed within. Because it originated within you. That does not mean you stop seeing forms and only see the underlying structures and connections. You will still divide. You will still see forms. But now you are with the awareness that you are dividing and you are admiring the form. The awareness is not something that is turned on in a day for people who had been dividing and seeing things for long. The awareness was there and is there and always there. It has just got faded over time and has remained muted. The realization that divisions are a property of the mind and it should be done only when needed is the awareness aspect of it. With awareness, practice is needed to glide through life's challenges that forces you to be divisive. With awareness one has to watch the divisions happening around. Being in steadfast awareness is the key. It is harder than a small reed withstanding a whirlwind. It is the way and the only way for living. A unified YOU is the goal. Then it is easier to drop YOU.

We have applied our divisive mind to designing the WEB. The spread of apps or the searching of documents is a testimony to that. By raising the awareness level of the WEB which represents the collective consciousness of all of us, we bring a seamless connected web without the distractions of the Apps, but not ignoring them altogether for the function and focus they bring in to solve the problems. A cohesive WEB is the goal. Then it is easier to drop YOU.

PS: I have voluntarily not added any hyper links in the above text to maintain that the web is broken. I have added tags though to show that that is the way to go forward, though it is not going to solve the mess we are already in.

URL encoding, decoding and the confusions you may have with Servlets

I had a lot of trouble with receiving HTTP URLs in my servlet (written in java) code.

Specifically when I do Ajax calls from Javascript (AngularJS) , the URLs were encoded and sent by the browsers with the space being replaced with %20 and probably some others which when I used the HttpServletRequest object with getRequestURL() to get the path segment of the HTTP URL ,

in my case it was like 'http://localhost:8080/abc/xyz/tags/Address of my home'. This URL is returned as 'http://localhost:8080/abc/xyz/tags/Address%20of%20my%20home'. This is something I thought was a trivial one and looked for answers in the world wide web. However after reading this post ,

http://blog.lunatech.com/2009/02/03/what-every-web-developer-must-know-about-url-encoding

which is written very well, I felt there is no point relying on the Servlet decode methods and waste my time. I decided to have my own little parser that essentially replaces the encoded characters as above to the spaces. I have some more in the URL query portion also.

Thought that if you are onto something like this, this may be of help.

Why Provisional patent?

I had been alone doing some part-time development of a product for the last two years. The product itself helps collaborating on rich data mainly targeted at the SMB.

The idea has some novelty and it has made me look into several technology areas like Semantic web, text mining, Software architecture, Visualization etc. At the end of it all before I put it out on the web I thought I had some interesting way to solve some of these problems and approached a patent attorney only to know that I cannot apply for a patent if it is published on the web. Or rather if I wanted to patent some of the ideas I need to do it before I put it out even if it is a beta.

The next thing was, I paid close to 500+ USD in Indian rupees to go for a prior art search. And boom came a bunch of patents, one of them had 100s of pages of write up which were all laid before me with some sections across each of these patents marked to be resembling the write up I gave about my invention. My attorney had advised that if my invention (as given by me) is found in one or more patents in a scattered manner and there is nothing non-obvious over and beyond what is already there the chances of getting a patent granted would be negative.

I have been reading them for a week now. I have gone nuts literally trying to understand the soupy noodles like language only to ask myself 'What the heck are you trying to say at last?'. I did find some parts of those patents talking similar terms and methods of solving it. But I did figure out that either some of them are overtly broad and vague to even realize them as them or they ended up solving different problems, even though some parts of the methods are similar. I got a feeling that I still have something exclusive that can be patented.

But now I started to think if I should do a 'Provisional patent' which is like adding myself to the beginning of a queue which denotes my invention allowing me to go back and file a proper patent application within a year. This gets me the right to say 'Patent Pending' and probably prevents some one else coming out with similar stuff and claiming rights over it. A non-provisional one is a proper one. The cost of a provisional one is another 1000+ USD more (over and above the prior art). Non-provisional would be like 4500+ USD (over and above prior art). Which one should I choose? You can help me with your experience and thoughts. However, here is what I am thinking about this.

I will go for a provisional one. Mainly because, I feel I have to do a clean job of depicting the problem space and the method to solve it, highlighting the novelty, adequately bringing out the differences from existing patents. This is a time consuming affair. Also, I would be able to understand what I have been doing even clearer. If you ask me the question of how can I do something in the first place without knowing what I was up to, the answer would be like 'not all problems are definite. Some times, you start building stuff with an idea of solving a class of problems, which can cut across domains and as you do it, you may find more interesting ways to solve even a broader set'. This may go well against the notion of a start up which should always be closing on their goals and coming out with an answer quickly. I somehow could not do this party due to my part-time work and partly because it was a broader, generic problem cutting across domains.

Anyways, coming to the issue of prov or non-prov.., I would like to get a year's time to better understand the work I have done, the future expansions to it and the work that has already been done by others. This may mean I blow away 1000 USD. However I do not feel it to be something big given the value I may end up generating on this one. 

The other thing is about the fact that I need to get some funding going forward mainly due to the need for me to realize a domain specific use case over the generic tool I have been building. This phase 2 requires business partnering, marketing, some hands to help me out on the tech side and others. So, I have come to realize that I cannot move forward without a VC funding on this phase 2 thing. I believe a patent pending status gives a very good value in terms of some one who has thought through a lot about the problem area and also have a beta ready application. It will help me strike the right notes.

On the other hand, say I do not see a traction in what I have done in the next one year and if I can evaluate it to be negative, I will drop the idea of patenting without incurring a major cost. It allows me time to toy over the idea, improve it and present it when the context around you begins to show the right way.

Let me know what you think.

Conceptual architecture or technology stacks?

A lot many times I end up seeing designers / architects drawing technology stacks when showing the architecture.

Architecture to me is always independent of technologies. Architecture is a sort of blue print addressing the concerns. It has many views and a technology stack that fits the blue print is one of the views.

Mainly the blue print should be able to have blocks which addresses the different functional requirements - sort of a broad sweep. For example, if the functional need is "Integration with one or more southbound systems" then a block which maps to it having inner blocks representing "Adapters, Transforms, Endpoints, Routes" as some of the labels of it would give an idea that the southbound integration is addressed by this block.

Now there can be many blocks representing different functional needs. It is important to be able to specify the interconnections between these blocks and specify the type of protocol.

The entire system comprising of the different functional blocks should be given a view in terms of the ecosystem where it resides. The systems and applications that connect to this system whose architecture is being defined. The ingress, egress points have to be specified along with the protocol.

As another view, the physical view or deployment view can show these blocks as processes, libraries, clusters, physical interconnect etc. This gives an idea of how the above functional blocks are realized and represented in the real world. The deployment view can also label the exact technologies like Hibernate or Drools or J2EE or others to give a better idea of the physical view. The physical view typically shows the single node and cluster deployments.

As another view, the most significant call flows through the architecture can be highlighted by way of a sequence diagram across the architectural blocks.

Essentially what we are showing is the way the overall system under question is viewed from different angles like the way a house is seen from the front or top or dissected to see the internal arrangement of the rooms or the passages connecting them and a even high altitude view to see how the other houses and street are where the house under question is situated. It is quite similar.

This way of representing in different views or dimensions help answer different aspects of the architecture early on allowing others to comment and refine the picture. It gives the broad guideline for design and shows the way all through implementation. If the original blocks have not been violated over course of time in spite of the varying requirements through a project, its a sign of good abstraction of the problem in hand.

The key here is the blocks should not be overtly high level and general that it is useless to relate the problem to it. It should have sufficient detail to give a clear understanding of how the underlying design will work at a high level. At the same time, it should stay away from too many details which if you think will be better served to be handled during the elaboration of a block during design.

Architecture should not be seen as mere pictures. They are the outcome of a experienced person solving the entire complex problem with its multitude of requirements in one go at a broad level. It should address all the key requirements and should not leave anything. It is one broad stroke with a paint brush covering the final outcome. Details are missing but can be filled in during the course of design.

Application architecture concerns

Sponsors interests

Know who the sponsors are. What their motivations are in bringing a software product. Is it a brave new thinking and disruption in terms of a idea that they want to realise or is it something that they want to bring in as a refinement over a legacy product that lacked architectural thinking. 

Will architecture change because of the above?  Architecture for a new idea or concept is not a real need in my opinion. A brand new product which is yet to see its customer base has other challenges like bringing a early version to the market as a Beta even to understand and refine the thought. Architecture for this would come in the way of realising this early versions. Short cuts even if they mean rapid prototyping with minimal design thoughts are good enough. When there is a demand for the product and it is steadily growing, then things like scalability, performance, availability all comes to the fore. What this means is architecture comes at a much later stage in this case.

The second category is a total product rewrite to replace a legacy product. This comes with a clear idea of existing customers and potential new customer base. This definitely require a clean architectural thinking right from the beginning. Architecture helps to avoid making mistakes of the legacy product that would have made it a burden to maintain, evolve or penetrate new markets. That normally would be the reason why sponsors would want to bring a architecture base. 

Bringing in a architecture early on structures the thinking around all the issues that floats on the surface of the legacy product over time. Also, a new architecture helps to get a leap to a different level taking advantage of the technical, technology evolutions since the legacy product. 

A third category is about re-factoring an existing product. This also require architectural thinking. But this most often may not be easy to have a cleaner structural arrangement as a total product rewrite. Re-factoring is mostly the case with legacy products as against a complete rewrite due to the cost, time pressures to bring out a better version to the customers. Often if the original product was shoddily designed, customers would bring in the pressure to have a better version as they would be bearing the brunt in terms of frequent patch upgrades. This triggers a re-factoring rather than a rewrite to allow for a quicker and better release. 

A re-factoring is always tricky than a rewrite because of the fact that it is always difficult to make sense of some one else's code. It is always rare to see a design or architecture document that is up-to-date reflecting the code which makes it even more difficult to achieve a clean partitioning. Thus re-factoring majorly is always a dream unless the original design team is remaining intact in full or part.

Application architecture concerns

Continuing the key concerns on the application architecture,


Cost of product development plays a big role

If the proposed architecture results in a lot of labour it is usually shot down no matter how good it is. This can also happen due to lack of skill thereby requiring more time to realize the architecture by the team. Budget is always constrained in organizations, especially for new ventures. Architecture will be even more constrained when developers decide to code something quick and be able to handle the short term need. If the product becomes a hit, demand picks up and all the issues come to the fore.

If it is a web application, this model works perfectly fine as the initial budget is invariably small and nothing in the sense of a scalable, reliable architecture can be done. The broad outlines can be specified in terms of the components, interfaces, functions and then the plane takes off. If the product is a hit, more funds are available and scaling can be addressed at that time. Most of the time the users are also individuals and the fee collected per them is not significant to be precise in terms of delivering a fool-proof architectural answer.

However, if it is a enterprise application paid in millions by the customer and used by millions of people of the customer such as a telco application, then short-term solutions turn into nightmares later driving the project into a death spiral and ending up having a irated customer. Architecture in such a case should be given a serious thought. In this case, the cost of having a architectural underpinning is usually insignificant compared to the size of development. Or in other words the cost of architecture should not exceed 10% of the product development cost to give a rough idea. This I believe if executed correctly should give the benefit of reduced labour resulting in lesser cost of incremental development of the product and a quicker turnaround time in releasing new features.

 It would be suicidal to go with a non-architecture approach for a large enterprise application. As the number of users increase, the size of data goes up, the rate of failures due to bugs, the predictability of performance and scaling comes to the fore without a sound architecture. Bad architecture always comes to haunt a product team at a later stage when the number of users increase. Citing cost or schedule pressures in avoiding architecture is not a sensible thing in this scenario as cost due to adhering to a architecture is always lesser when compared to the cost of an adhoc development and resulting rework.
 
A well thought out architecture results in the simplest and best way to address a problem. However, this means the problem should be correctly stated as much as possible. If this involves a cost, it is  money well spent. The hassles of spending money on a imaginary problem is far worse. Often cost overruns happen due to incorrect perception of the problem and a improper answer. For example, a requirement that says the requests will hit the application at 10 TPS (transactions per second) if in reality is at 100 TPS,  not having a Cache or a light weight framework in the architecture may cost the day for you.

There is no idealistic architecture. It evolves over time. Thus, it is important to spend money to refine the architectural thinking with prototyping. If multiple options can be tried quickly and the best technology or approach is identified, then it saves a lot of money.

Application Architecture concerns

Not all the applications need a architectural thinking. If you are building a product that has to stand the test of time or go for a long innings then you better get an architectural underpinning for that. If it means a large number of end users or it is going to run on the cloud, then it is a must to have this aspect nailed down. Most of the web scale products we see like Google or Amazon platform would not have been developed with a scalable architecture on day one.

But what I have seen is, even if you want an architecture underlying your software, and even if some one is deputed on that task from day one, the final architecture is proven only when it meets the desired scale, performance, flexibility over time. This will not happen in the initial days or months.

Architecture cannot be fixed in the initial weeks or months of a product and then onwards we can see the product following the same. It is absolutely not possible. Of course, the initial structure of the product can be laid out based on the requirements available at that time. However, as we have seen, requirements are not cast in stone. It is almost close to making the writing on water. It changes over time so drastically. There is nothing wrong about it if you leave out the 'get me this out by Monday' type of pronouncements. That's the way things are. In a way the 'agile' methods make more sense to the sponsors due to this. They do not need to wait infinitely to see the product in its full glory. Because requirements can change, architecture, design changes, implementation changes, if some one decide to wait for the first perfect release it becomes a mythical thing. So, give me value for the money I put in. Show me something that is in tune with the current stated requirement. That seem to be the mantra. In a way, the architecture also has to be moving in this path. It is true that architecture cannot be so damn fluctuating as a product evolves. The basic structure should be in place to address the key concerns. Then it is a matter of how course corrections are done to accomodate the NFRs like performance, scalability etc. That does not mean the initial architecture focuses on functionality only. Functionality should be obviously addressed. But care should be taken to ensure non-functional aspects like performance is also discussed even in initial stages.

So it becomes important to focus on the key concerns and what could they be?

Before even looking at technical or technological considerations, some of the key aspects that has to be understood as a architect is

- What kind of audience will be using this product?

In most cases, applications are used by people who may be well versed in the business domain. For example, if it is a application that automates the processes in a automobile manufacturing industry the product should be talking about their final product which are the cars and how things move around to get the processes involved in getting a car ready. The domain people who are well versed in a manual setting in manufacturing a car are the audience, apart from administrators who are a little bit more tech savvy and who may define workflows or rules or the CEO who is more interested in a summary outcome. Thus it is important to understand these people who will expect different things from the product you give out. The product should be able to meet all of these diverse view points and their skill or capability to comprehend out of the product and map it to their own needs. You cannot ask a supervisor of the plant to define rules. This has to be done by some one who is computer savvy, trained to handle the rules and processes. The CEO cannot be asked to define rules as well :-)

These are very finer details not touching any of the details of the product you make but more from the view points of its users.


- The ecosystem where the product will be deployed

Normally with tiered architectures, the deployment is going to be clients and server side. However, there could be people on the move such as salesmen or people who need to monitor things around the manufacturing facility. This may require mobility involving smart phones or tablets and others. This will impact the amount of data you can push around or other latency considerations. The product may have to be running on multiple sites across a country with local caching and ability to handle synchronize it with a centralized DB. The product may be a carrier grade one requiring geographical redundancy requiring the architecture to bring in questions early on during architecture definition such as the choice of cache or messaging technologies.

Will discuss more in the next blog..