The Broken WEB

It is just that I have been intrigued by the divisions I see around in life and how the mind thrives in these divisions. Divisions are dangerous and we are seeing it everyday in politics, office and even among us in friends, family. Humans have this penchant for dissecting and making sense. Dissecting is a double edged sword. It can be used as a tool to analyze and at the same time, it can arouse passions and lead you into unwanted confusions. The problem is a weak mind cannot know the difference between when it ended the analysis part and when it entered the realm of the unwanted. This sounds like philosophy right?

I see the same thing in software. Microsoft, Google, Apple all have been delivering this so called Apps which are divisive. They focus on a single function. They deliver that effectively with a great looking UI/UX (the Apple way). They go on developing the Apps like this, millions and zillions.

Now compare this to the WWW of the past when it originated. It was a beautiful, marvelous discovery in my opinion. Hyper links unified the way you read documents. Even today, unless you watch yourself, you can get thoroughly lost in your browsing. I have seen this happen with me many times. I do not know why or how I ended up reading a page which was never what I started with. Hyper links unified the sea of documents.

Unfortunately the documents are a weak form of representing things. It caters to the form and loses the underlying structure from where it originated (the databases). In the quest to present in a form that is readable, the documents lose vital information. Thus the very strength brought in by the hyper linking which links documents breaks mysteriously because there are these finer things in the documents which cannot be linked to precisely.

With time as the documents started representing not merely some text, but company financials, books, weather and so on, the hyper link broke and Google came in :-) Google emerged as the great leveler, the one that fixes the breakage. It brought in search which kind of fixes the hyper link problem indirectly.

I will give you an example, I look for 'lambda calculus', then I find the profile of the lecturer, OK I find a link and I go to his page, then I find some other subjects, say 'Haskell' which are mentioned there and get interested, but I cannot go to them. I go to Google and look for those. Google has figured out the linking between these disconnected pages and it has allowed me to browse again with a brief interruption. You may think it is brief. But I feel it is really annoying. I use a App (Google) to fix my linking when it should have been possible for the lecturer to have linked it or for that matter if it was publicly editable it should have been done by some one. But how does the lecturer know the web site that has the information about 'Haskell' without doing a search? Why was this not solved by the inventors of the WWW which seems vital for the hyper links to function? OK if I do not want to go there, I will have to still answer how will the lecturer or some one else know which web site to link?

I feel the only way is when you have the web itself converted from the web of documents to the web of things. Things are not just data or text. They embody the description about the data as well. They carry meaning. One aspect of describing a thing is also the 'categorization' of the thing as to where it belongs to in a hierarchy of things in the world. If only we had a web of things, then it was fairly easy for the lecturer or any one to look for things that belong to the category of functional programming and find pages of Haskell categorized there and be able to make that Hyperlink.

This way, the web would have remained powerful and would have been a truly knowledge base of the human beings or in short the collective consciousness of us. But as history would want it, we had a more ugly way of fixing this by giving the rights to fix this to the great company called Google. I admire them as they probably were truly annoyed by this breakage as well and looked to answer this sincerely.

However, I feel the root cause is in the way we create things. It was form based and not structure based. It was about putting up a document in the human cognizable way and leaving it for Google to discover it than being able to present the underlying structure. Form falls into the Apple space. That is what they mastered and took it very far. And many others followed. But Form is as I said very divisive. It caters to the human mind very well. It allows you to see things distinctly without a way to know that underneath every thing is related to one another. Form destroys structure in the process of delivering something comprehensible for the human mind. I am not against forms. I feel they are good, but not to the extent where they can destroy the underlying structure of things nor the connection between the things.

I see this world crafted by these two big companies. On one side is Apple which has focused on Form , brought App store, zillions of Apps to cater to every need of you not allowing to realize the connectivity across these sea of Apps because they focused on the Form and make you focus on the Form.

On the other hand, we have Google which is helping you relate the underlying structure, but in the process it did it in a way allowing the lazy human way of typing text for everything and destroying the underlying structure. It did the grunt work of analyzing things and allowing the humans to continue in their realm of text or Form which they are comfortable with.

Thus one has blatantly sidelined with the Form and another has not looked at alternate means and allowed things to continue as is. The net result is, we have a BROKEN WEB and that continues.

I believe the original inventor of the web had realized this in some sense and brought in the Semantic Web and all the W3C work that followed it. However, what they miss is that the damage has happened and I see light only in the micro formats. But that is not again going to be seen as important for a long time to come by the web authors as it looks too late into the game and there are already zillions of web pages without them.

Coming back to philosophy, the divisiveness has to be fixed within. Because it originated within you. That does not mean you stop seeing forms and only see the underlying structures and connections. You will still divide. You will still see forms. But now you are with the awareness that you are dividing and you are admiring the form. The awareness is not something that is turned on in a day for people who had been dividing and seeing things for long. The awareness was there and is there and always there. It has just got faded over time and has remained muted. The realization that divisions are a property of the mind and it should be done only when needed is the awareness aspect of it. With awareness, practice is needed to glide through life's challenges that forces you to be divisive. With awareness one has to watch the divisions happening around. Being in steadfast awareness is the key. It is harder than a small reed withstanding a whirlwind. It is the way and the only way for living. A unified YOU is the goal. Then it is easier to drop YOU.

We have applied our divisive mind to designing the WEB. The spread of apps or the searching of documents is a testimony to that. By raising the awareness level of the WEB which represents the collective consciousness of all of us, we bring a seamless connected web without the distractions of the Apps, but not ignoring them altogether for the function and focus they bring in to solve the problems. A cohesive WEB is the goal. Then it is easier to drop YOU.

PS: I have voluntarily not added any hyper links in the above text to maintain that the web is broken. I have added tags though to show that that is the way to go forward, though it is not going to solve the mess we are already in.

URL encoding, decoding and the confusions you may have with Servlets

I had a lot of trouble with receiving HTTP URLs in my servlet (written in java) code.

Specifically when I do Ajax calls from Javascript (AngularJS) , the URLs were encoded and sent by the browsers with the space being replaced with %20 and probably some others which when I used the HttpServletRequest object with getRequestURL() to get the path segment of the HTTP URL ,

in my case it was like 'http://localhost:8080/abc/xyz/tags/Address of my home'. This URL is returned as 'http://localhost:8080/abc/xyz/tags/Address%20of%20my%20home'. This is something I thought was a trivial one and looked for answers in the world wide web. However after reading this post ,

http://blog.lunatech.com/2009/02/03/what-every-web-developer-must-know-about-url-encoding

which is written very well, I felt there is no point relying on the Servlet decode methods and waste my time. I decided to have my own little parser that essentially replaces the encoded characters as above to the spaces. I have some more in the URL query portion also.

Thought that if you are onto something like this, this may be of help.

Why Provisional patent?

I had been alone doing some part-time development of a product for the last two years. The product itself helps collaborating on rich data mainly targeted at the SMB.

The idea has some novelty and it has made me look into several technology areas like Semantic web, text mining, Software architecture, Visualization etc. At the end of it all before I put it out on the web I thought I had some interesting way to solve some of these problems and approached a patent attorney only to know that I cannot apply for a patent if it is published on the web. Or rather if I wanted to patent some of the ideas I need to do it before I put it out even if it is a beta.

The next thing was, I paid close to 500+ USD in Indian rupees to go for a prior art search. And boom came a bunch of patents, one of them had 100s of pages of write up which were all laid before me with some sections across each of these patents marked to be resembling the write up I gave about my invention. My attorney had advised that if my invention (as given by me) is found in one or more patents in a scattered manner and there is nothing non-obvious over and beyond what is already there the chances of getting a patent granted would be negative.

I have been reading them for a week now. I have gone nuts literally trying to understand the soupy noodles like language only to ask myself 'What the heck are you trying to say at last?'. I did find some parts of those patents talking similar terms and methods of solving it. But I did figure out that either some of them are overtly broad and vague to even realize them as them or they ended up solving different problems, even though some parts of the methods are similar. I got a feeling that I still have something exclusive that can be patented.

But now I started to think if I should do a 'Provisional patent' which is like adding myself to the beginning of a queue which denotes my invention allowing me to go back and file a proper patent application within a year. This gets me the right to say 'Patent Pending' and probably prevents some one else coming out with similar stuff and claiming rights over it. A non-provisional one is a proper one. The cost of a provisional one is another 1000+ USD more (over and above the prior art). Non-provisional would be like 4500+ USD (over and above prior art). Which one should I choose? You can help me with your experience and thoughts. However, here is what I am thinking about this.

I will go for a provisional one. Mainly because, I feel I have to do a clean job of depicting the problem space and the method to solve it, highlighting the novelty, adequately bringing out the differences from existing patents. This is a time consuming affair. Also, I would be able to understand what I have been doing even clearer. If you ask me the question of how can I do something in the first place without knowing what I was up to, the answer would be like 'not all problems are definite. Some times, you start building stuff with an idea of solving a class of problems, which can cut across domains and as you do it, you may find more interesting ways to solve even a broader set'. This may go well against the notion of a start up which should always be closing on their goals and coming out with an answer quickly. I somehow could not do this party due to my part-time work and partly because it was a broader, generic problem cutting across domains.

Anyways, coming to the issue of prov or non-prov.., I would like to get a year's time to better understand the work I have done, the future expansions to it and the work that has already been done by others. This may mean I blow away 1000 USD. However I do not feel it to be something big given the value I may end up generating on this one. 

The other thing is about the fact that I need to get some funding going forward mainly due to the need for me to realize a domain specific use case over the generic tool I have been building. This phase 2 requires business partnering, marketing, some hands to help me out on the tech side and others. So, I have come to realize that I cannot move forward without a VC funding on this phase 2 thing. I believe a patent pending status gives a very good value in terms of some one who has thought through a lot about the problem area and also have a beta ready application. It will help me strike the right notes.

On the other hand, say I do not see a traction in what I have done in the next one year and if I can evaluate it to be negative, I will drop the idea of patenting without incurring a major cost. It allows me time to toy over the idea, improve it and present it when the context around you begins to show the right way.

Let me know what you think.

Conceptual architecture or technology stacks?

A lot many times I end up seeing designers / architects drawing technology stacks when showing the architecture.

Architecture to me is always independent of technologies. Architecture is a sort of blue print addressing the concerns. It has many views and a technology stack that fits the blue print is one of the views.

Mainly the blue print should be able to have blocks which addresses the different functional requirements - sort of a broad sweep. For example, if the functional need is "Integration with one or more southbound systems" then a block which maps to it having inner blocks representing "Adapters, Transforms, Endpoints, Routes" as some of the labels of it would give an idea that the southbound integration is addressed by this block.

Now there can be many blocks representing different functional needs. It is important to be able to specify the interconnections between these blocks and specify the type of protocol.

The entire system comprising of the different functional blocks should be given a view in terms of the ecosystem where it resides. The systems and applications that connect to this system whose architecture is being defined. The ingress, egress points have to be specified along with the protocol.

As another view, the physical view or deployment view can show these blocks as processes, libraries, clusters, physical interconnect etc. This gives an idea of how the above functional blocks are realized and represented in the real world. The deployment view can also label the exact technologies like Hibernate or Drools or J2EE or others to give a better idea of the physical view. The physical view typically shows the single node and cluster deployments.

As another view, the most significant call flows through the architecture can be highlighted by way of a sequence diagram across the architectural blocks.

Essentially what we are showing is the way the overall system under question is viewed from different angles like the way a house is seen from the front or top or dissected to see the internal arrangement of the rooms or the passages connecting them and a even high altitude view to see how the other houses and street are where the house under question is situated. It is quite similar.

This way of representing in different views or dimensions help answer different aspects of the architecture early on allowing others to comment and refine the picture. It gives the broad guideline for design and shows the way all through implementation. If the original blocks have not been violated over course of time in spite of the varying requirements through a project, its a sign of good abstraction of the problem in hand.

The key here is the blocks should not be overtly high level and general that it is useless to relate the problem to it. It should have sufficient detail to give a clear understanding of how the underlying design will work at a high level. At the same time, it should stay away from too many details which if you think will be better served to be handled during the elaboration of a block during design.

Architecture should not be seen as mere pictures. They are the outcome of a experienced person solving the entire complex problem with its multitude of requirements in one go at a broad level. It should address all the key requirements and should not leave anything. It is one broad stroke with a paint brush covering the final outcome. Details are missing but can be filled in during the course of design.

Application architecture concerns

Sponsors interests

Know who the sponsors are. What their motivations are in bringing a software product. Is it a brave new thinking and disruption in terms of a idea that they want to realise or is it something that they want to bring in as a refinement over a legacy product that lacked architectural thinking. 

Will architecture change because of the above?  Architecture for a new idea or concept is not a real need in my opinion. A brand new product which is yet to see its customer base has other challenges like bringing a early version to the market as a Beta even to understand and refine the thought. Architecture for this would come in the way of realising this early versions. Short cuts even if they mean rapid prototyping with minimal design thoughts are good enough. When there is a demand for the product and it is steadily growing, then things like scalability, performance, availability all comes to the fore. What this means is architecture comes at a much later stage in this case.

The second category is a total product rewrite to replace a legacy product. This comes with a clear idea of existing customers and potential new customer base. This definitely require a clean architectural thinking right from the beginning. Architecture helps to avoid making mistakes of the legacy product that would have made it a burden to maintain, evolve or penetrate new markets. That normally would be the reason why sponsors would want to bring a architecture base. 

Bringing in a architecture early on structures the thinking around all the issues that floats on the surface of the legacy product over time. Also, a new architecture helps to get a leap to a different level taking advantage of the technical, technology evolutions since the legacy product. 

A third category is about re-factoring an existing product. This also require architectural thinking. But this most often may not be easy to have a cleaner structural arrangement as a total product rewrite. Re-factoring is mostly the case with legacy products as against a complete rewrite due to the cost, time pressures to bring out a better version to the customers. Often if the original product was shoddily designed, customers would bring in the pressure to have a better version as they would be bearing the brunt in terms of frequent patch upgrades. This triggers a re-factoring rather than a rewrite to allow for a quicker and better release. 

A re-factoring is always tricky than a rewrite because of the fact that it is always difficult to make sense of some one else's code. It is always rare to see a design or architecture document that is up-to-date reflecting the code which makes it even more difficult to achieve a clean partitioning. Thus re-factoring majorly is always a dream unless the original design team is remaining intact in full or part.

Application architecture concerns

Continuing the key concerns on the application architecture,


Cost of product development plays a big role

If the proposed architecture results in a lot of labour it is usually shot down no matter how good it is. This can also happen due to lack of skill thereby requiring more time to realize the architecture by the team. Budget is always constrained in organizations, especially for new ventures. Architecture will be even more constrained when developers decide to code something quick and be able to handle the short term need. If the product becomes a hit, demand picks up and all the issues come to the fore.

If it is a web application, this model works perfectly fine as the initial budget is invariably small and nothing in the sense of a scalable, reliable architecture can be done. The broad outlines can be specified in terms of the components, interfaces, functions and then the plane takes off. If the product is a hit, more funds are available and scaling can be addressed at that time. Most of the time the users are also individuals and the fee collected per them is not significant to be precise in terms of delivering a fool-proof architectural answer.

However, if it is a enterprise application paid in millions by the customer and used by millions of people of the customer such as a telco application, then short-term solutions turn into nightmares later driving the project into a death spiral and ending up having a irated customer. Architecture in such a case should be given a serious thought. In this case, the cost of having a architectural underpinning is usually insignificant compared to the size of development. Or in other words the cost of architecture should not exceed 10% of the product development cost to give a rough idea. This I believe if executed correctly should give the benefit of reduced labour resulting in lesser cost of incremental development of the product and a quicker turnaround time in releasing new features.

 It would be suicidal to go with a non-architecture approach for a large enterprise application. As the number of users increase, the size of data goes up, the rate of failures due to bugs, the predictability of performance and scaling comes to the fore without a sound architecture. Bad architecture always comes to haunt a product team at a later stage when the number of users increase. Citing cost or schedule pressures in avoiding architecture is not a sensible thing in this scenario as cost due to adhering to a architecture is always lesser when compared to the cost of an adhoc development and resulting rework.
 
A well thought out architecture results in the simplest and best way to address a problem. However, this means the problem should be correctly stated as much as possible. If this involves a cost, it is  money well spent. The hassles of spending money on a imaginary problem is far worse. Often cost overruns happen due to incorrect perception of the problem and a improper answer. For example, a requirement that says the requests will hit the application at 10 TPS (transactions per second) if in reality is at 100 TPS,  not having a Cache or a light weight framework in the architecture may cost the day for you.

There is no idealistic architecture. It evolves over time. Thus, it is important to spend money to refine the architectural thinking with prototyping. If multiple options can be tried quickly and the best technology or approach is identified, then it saves a lot of money.

Application Architecture concerns

Not all the applications need a architectural thinking. If you are building a product that has to stand the test of time or go for a long innings then you better get an architectural underpinning for that. If it means a large number of end users or it is going to run on the cloud, then it is a must to have this aspect nailed down. Most of the web scale products we see like Google or Amazon platform would not have been developed with a scalable architecture on day one.

But what I have seen is, even if you want an architecture underlying your software, and even if some one is deputed on that task from day one, the final architecture is proven only when it meets the desired scale, performance, flexibility over time. This will not happen in the initial days or months.

Architecture cannot be fixed in the initial weeks or months of a product and then onwards we can see the product following the same. It is absolutely not possible. Of course, the initial structure of the product can be laid out based on the requirements available at that time. However, as we have seen, requirements are not cast in stone. It is almost close to making the writing on water. It changes over time so drastically. There is nothing wrong about it if you leave out the 'get me this out by Monday' type of pronouncements. That's the way things are. In a way the 'agile' methods make more sense to the sponsors due to this. They do not need to wait infinitely to see the product in its full glory. Because requirements can change, architecture, design changes, implementation changes, if some one decide to wait for the first perfect release it becomes a mythical thing. So, give me value for the money I put in. Show me something that is in tune with the current stated requirement. That seem to be the mantra. In a way, the architecture also has to be moving in this path. It is true that architecture cannot be so damn fluctuating as a product evolves. The basic structure should be in place to address the key concerns. Then it is a matter of how course corrections are done to accomodate the NFRs like performance, scalability etc. That does not mean the initial architecture focuses on functionality only. Functionality should be obviously addressed. But care should be taken to ensure non-functional aspects like performance is also discussed even in initial stages.

So it becomes important to focus on the key concerns and what could they be?

Before even looking at technical or technological considerations, some of the key aspects that has to be understood as a architect is

- What kind of audience will be using this product?

In most cases, applications are used by people who may be well versed in the business domain. For example, if it is a application that automates the processes in a automobile manufacturing industry the product should be talking about their final product which are the cars and how things move around to get the processes involved in getting a car ready. The domain people who are well versed in a manual setting in manufacturing a car are the audience, apart from administrators who are a little bit more tech savvy and who may define workflows or rules or the CEO who is more interested in a summary outcome. Thus it is important to understand these people who will expect different things from the product you give out. The product should be able to meet all of these diverse view points and their skill or capability to comprehend out of the product and map it to their own needs. You cannot ask a supervisor of the plant to define rules. This has to be done by some one who is computer savvy, trained to handle the rules and processes. The CEO cannot be asked to define rules as well :-)

These are very finer details not touching any of the details of the product you make but more from the view points of its users.


- The ecosystem where the product will be deployed

Normally with tiered architectures, the deployment is going to be clients and server side. However, there could be people on the move such as salesmen or people who need to monitor things around the manufacturing facility. This may require mobility involving smart phones or tablets and others. This will impact the amount of data you can push around or other latency considerations. The product may have to be running on multiple sites across a country with local caching and ability to handle synchronize it with a centralized DB. The product may be a carrier grade one requiring geographical redundancy requiring the architecture to bring in questions early on during architecture definition such as the choice of cache or messaging technologies.

Will discuss more in the next blog..