Principled Creativity

To create is to bring into existence. It is a human endeavor which — by necessity — we are perpetually bound to. Solving technical problems is a creative endeavor. But complex problems can overwhelm us into losing sight of this connection. We may navigate complexity effectively by adopting a principled approach.

The process of creation reshapes a collection of resources into a solution. It is the human element — not the set of resources — which determines the ultimate utility of a solution. We can squander resources or apply them masterfully toward an end. How may we accomplish the latter?

Efficiency is a measure of how many resources we require to solve a problem. We should strive to use as little resources as possible. The natural state of the world is toward inefficient processes. Simple problems don't require much efficiency. But difficult problems demand a greater degree of efficiency is the pursuit is to remain feasible.

The ammount of efficiency and our understanding of the problem are intricately connected. Poor understanding means we will end up doing unnecessary work. Greater understanding means that we can do only the necessary work. We are in a perpetual state of ingorance of the problem we are trying to solve. We decrease the amount of ignorance in each step if we move carefully.

The first approximation to a solution is a leap of faith. In the best case, a domain expert makes a guess in to what the best solution to the problem is. There is no guarantee that our approximation will be sufficient. For difficult problems, we will invariable need multiple iterations, with better approximations each time. Here we are prepetually guessing our best approximation to the solution. In a way, it is reflective tof the scientific method. We take our understanding of the problem and guess the best solution we can. Then this concrete solution must come into contact with the real world in order to collect higher quality data. We repeat this process until our approximation is sufficient.

Solutions to complex problems are systems. A system is a set of components organized by a set of rules. For complex problems, we can not iterate on the whole problem at once, so we iterate on subsystems. The same principle apply as in the general case.

further study

How to split the system into subsystems?
How to know when progress is sufficient?

xxx

Outline

a human endeavor
characterization of the problem
description of efficiency
connection between efficiency and understanding of the problem
connection between the prototype and understanding
systems and pareto

We will require some metrics to disect the issue. Value measures how well a solution solves a problem. And efficiency is the ratio of value to resources allocated. This truth is captured in Parkison's Law which states "work expands so as to fill the time available for its completion".

We must appreciate the variablility of acceptable solutions to a problem.

Efficiency is connected deeply to our understanding of the problem.
We are in a perpetual state of ingorance of the problem we are trying to solve.
Sometimes solving the root problem sidesteps entire solutions/endeavors.
Not needing something is as good as having it.
We need a real prototype to study the problem further.
We iteratively take guesses to walkt toward a solution.
we mimic the scientific process
increasing our understanding is the same as creating
- our understanding of the problem is sufficient when the creation sufficiently solve the problem
talk about minimalism?

Value is a spectrum! The value we seek from a problem is dependent on context. If we are starving, any source of calories will do. If we are well fed, then we might seek a chef to cook an artisinal meal. our understanding of the problem is sufficient when the creation sufficiently solve the problem Keeping this context in mind is a powerful tool. We can become wasteful if we do not limit ourselves. Even the most trivial of problems can expand endlessly if the scope is not kept in check. To be efficient, we must push until enough value is delivered and no further. It can be our natural tendency to strive toward perfection, but this is at odds with efficiency.

Solutions to difficult problems can be described as systems. These systems may be further decomposed into subsystems. Not all components of a system deliver equal value. A car's engine delivers more value than its seat material.

How can we move forward without being wasteful? Apply the pareto princple! It gives a rough idea of the difference in magnitude. Take an accounting of all the components that the system would have to reasonably solve the problem. Order these components based on value. Roughly the top 20% will deilver 80% of the value. In most cases, the first pass is sufficient. If not, iterate.

Limitations

Determining value of component is best left to users/domain experts.

Pro Tips

This is a pro tip.

Conclusion

This is the conclusion.

XXX

A solution is not complete until it crosses some threshold amount of value. What does?

To dissect this issue, we need another metric. An efficient process should yield sufficient value while limiting the necessary resources.

Creation begins with an often vague sense that something will be useful if brough into existence.

The world is littered with projects which were doomed to fail because of mismanagement of resources. Resources alone do not determine the ultimate value created.

Our first goal is to create a solution with sufficient value.

Experience has taught me that a few simple observations can help naviagate this sea of tradeoffs.

For the purposes of this discussion efficiency can be defined as the ratio of value created to resources used. As a creator you should endeavour for maximum efficiency.

The first observation is that systems can be arbritrarily and recursively decomposed into subsystems. A car is a composition of an engine and a transmission and a shell and an interior.

These simple observations can be grouped into what is popularly known as the Pareto Principle. The most fundamental observation is that not all features of a system deliver the same value.

When navigating the complexity of a large project, it is easy to not move with as much force. We can lose sight of the basic princpiles which we might understand intuitively but are obfuscated by complexity. Here I cover the Pareto Principle, a suprisingly simple observation which can yield substantial gains in efficiency.

The fundamental truth to the Pareto Principle is that not all features deliver the same value. Think to any technology that you use. Out of the many features that a technology offers, most people only rely on a subset of features most of the time.

value proposition: apply a simple principle to increase efficiency
rule of thumb for organizing work
simple truth: not all features deliver the same value
much effort is wasted from a lack of clarity
a little planning goes a long way
a danger of complex projects is to lose sight of the guiding principles
- it helps to make the process explicity and to set aside time for it
applies to large/small projects, big/small teams
order features by value
determining value of a feature is a black art, domain experince is helpful
- users are the ultimate indicator of value
value := utility as percieved by user
feature := subsystem
- subsystems are arbritrarily nested, so this principle can be recursively applied
should stop at a reasonable point of progress to collect feedback
constant interplay between planning and progress
- feedback guides planning
- getting overzealous with "work" can lead to wasted effort
- too much planning and not enough effort is spent "moving the needle"
constraints always exist
- resources (time/human effort/money) are converted to value
lean process is a manifestation of this principle
pareto principle is a simple observation that is generally true
what are the limitation of the pareto principle?
- sometimes value is not obvious. domain experience helps in this regard
- dogfooding also helps to determine value
determining value always comes down to the user
sometimes usage patterns surprise you
goal is efficiency maximize value while limiting the amount of resources dedicated to an endeavour
clarity drives efficiency
- make your goal simple and easy to understand
- the more clear you are about your purpose the more uncessary work you can avoid
tunnel vision: when diving deep into technical details, it is easy to miss the forest for the trees
constraints are a good thing! they drive focus
- lack of focus is symptomatic of lose contraints on resources
example: google landing page. search bar is front and center. they understand what benefits their company
works at all scales of the project. from simple function to the whole project to the whole company
drives data modeling in a technical project, the data flow is often clear when the top 20% of features is commpleted