IT Aphorisms #

Aphorisms on IT, problem solving, and related.

On project manglement #

The scope is the project.

Or, do you even know what your project is, if you haven’t scoped it?

Failure to scope your project adequately will result in failure to deliver said project. A poorly defined scope or project description that’s chock-full of buzzwords is vague enough as to be virtually useless (i.e. limitless or without scope). There’s no way to complete a project with no clearly defined scope because there’s no way to reasonable be able to say when it’s “done.” Define the scope and you’ve defined the project. Said in other words, if you don’t have a clearly defined scope what you have is an “idea” not a project. This is also why many people have a “million dollar idea” but they need a “technical co-founder” to bring it to fruition. They have failed to define a scope and therefore a project. Adequately scoping a project requires technical knowledge and domain experience, and the effort involved shouldn’t be taken lightly. This may in fact be more than half the battle when it comes to completing a project.

Non-technical PM’s who are also poor managers are worse than no manager at all when it comes to technical projects.

There I said it. But also, if you’re not technical and you’re also not a good manager, what business do you have managing a technical project?

Non-technical PM’s who are also poor managers are worse than no manager at all. Non-technical PM’s who are good managers are slightly better than no manager at all as they can act as a “buffer” between the customer and the IT professional and perform a “diplomatic function”. (see runaway problem cascades, below). Best is a PM who’s both technical and possesses adequate people skills to interface with the customer. This is understandably rare and worth every penny (although C-levels rarely seem to see it this way). It’s difficult to apply quality control to a project without a technical PM.

Projects must be rooted in “reality” #

I feel like this shouldn’t need to be said, but experience dictates otherwise…

Projects must obey the physical laws of the universe.

Projects must be rooted in reality

This means first and foremost obeying the physical laws of the universe. This includes the laws of thermodynamics and therefore no project is immune to entropy. Therefore, bugs are inevitable. If bugs could be predicted and mitigated that would mean time-travel is possible, and this breaks all of the known laws of the universe! Customers who expect this, are dilusional and need to be “educated” (see below).

Complexity is the devil and is best avoided if at all possible.

Unfortunately, some complexity is necessary. Efforts to reduce complexity and provide simple clear and ledgible solutions is honorable and deserves more recognition than it seems to get.

The difficulty in producing a simple universal solution to a class of problems should not be underestimated. Depending on how universal the solution, this can literally be a “work of genius.” Einsteins general theory of relativity is an example of a simple solution to a universal class of problems that was literally a work of genius.

Inalienable Truths #

Something must serve as the single source of truth. Always. It doesn’t necessarily matter what it is, but it must exist. Everything else should be treated as “fiction”.

If it’s not a ticket, it’s not getting done.

Just because it’s a ticket doesn’t mean that it’s a problem that needs to be solved. (See law of parsimony below)

Not solving a problem can be a valid solution. It’s always worth consideration.

Time spent in meetings is inversely correlated with productivity.

Computers do everything you tell them to exactly as they’re told, nothing more nothing less.

Do things the “right” way the first time.

There’s usually a “right” way to do something, or at least a “more right” way to do something. It’s many orders of magnitude easier and faster to do something right the first time than it is to try to fix it later (if it’s even possible to fix it later). Alongside lack of documentation this may be the 2nd most abundant reason for “technical debt”.

DNS is almost always the issue.

Most developers and software admins don’t truly understand DNS.

Any developer who understands DNS and networking is actually “DevOps” despite what their title might say.

Human-centered “issues” #

Communication overhead grows exponentially in relation to the number of people on a team. There’s a point beyond which no useful production occurs beyond maintaining the status quo. If combined with mis-management it’s in-fact possible to move backwards on a project under certain circumstances.

Knowledge work is non-linear and hours in doesn’t equal hours out in terms of production. At the extreme end of performance knowledge work is more like being an athlete. Top chess grandmasters is a case in point.

Productivity obey’s the pareto principle in that 80% of the work is being done by 20% of the workers. This an immutable law of the universe. You might as well use the 80% of your work-force to do menial tasks to free up your top 20% of performers to do their best work. Nobody manages their work this way though, because “fairness” and “equality” usually plays a part. Optimizing for productivity above all else would look a lot different.

Attitude is more than half the battle.

Sub-class of humans known as “customers” #

Customers are often wrong and if not handled carefully may introduce more technical debt, burry techs in “non-problems”, or result in more communication overhead than can be reasonably managed by the IT resources available. If left completely unchecked this could easily result in a runaway problem cascade that can’t be resolved with the IT-resources available in the known universe.

Educating the customer must be achieved “diplomatically” or the IT professional may find themselves saddled with “financial” problems instead of our preferred “technical” ones.

There exists space in a venn-diagram of customer “asks” and “solvable problems” that’s neither the “most right” solution or impossible. This is a balance that must be delicately maintained. (see runaway problem cascade above)

On the unreasonable usefulness of documentation #

Understanding undocumented systems is an art, but more importantly a “science."

Doing “science” as an IT professional is not necessarily a good thing.

If it’s not documented somewhere, “science” must be conducted to understand the system through rigorous positing of hypothesis and developing models (i.e. theories) via testing, and observation. Only once concrete evidence is obtained through observational tools and a useful model developed (i.e. theory) can the system be reliably fixed when it inevitably breaks. This is an arduous, time-consuming, and inefficient process. It’s often many orders of magnitude less-effort to rebuild the system, documenting as you go, than it is to try to understand an undocumented system that already exists. This is the origins of most “technical debt.”

Lack of documentation is probably the single greatest source of “technical debt” beyond poor design or development choices.

Documentation is a moral obligation to your fellow IT professionals. It’s unequivocally ethically “good” to write documentation and worse than mere laziness but for all intents and purposes “evil” to fail to produce adequate documentation alongside delivery of any sort of developed system, body of work, coded project, or other necessarily complex system. Not only will future techs thank you, but your future self will thank you as well. It’s irrational to assume your memory is as good as you think it is and that you won’t refer to your own carefully crafted docs 6-months from now.

Visual aids such as lovingly crafted network diagrams are a love-letter to future sysadmins. This is the art of documentation. It’s intrinsically good to produce this form of documentation even if nobody ever see’s it other than you.

Failure to document is failure to embue your thing with enough evolutionary relevance to survive the first iteration of said thing. It will cease to exist beyond it’s first iteration since it will be too difficult to understand and it will instead be replaced by a newer heartier version of the thing. The only exceptions are a thing that’s so business useful it’s worth infusing inordinate amounts of resources (money) to “life-support” said thing, but even these systems eventually are overcome.

Getting to the “root” of the problem #

Context is everything.

Problems don’t exist outside of the context they live in. Failure to provide context is failure to provide a problem rooted in reality. Problems not rooted in reality are fictitious and not real problems. (See laws of physics above)

A well-defined problem is a solvable problem.

You can’t solve the problem before defining the problem in concrete terms. If you can’t define the problem in concrete terms you don’t understand enough about the system to define it. If you don’t understand the system, either read the docs or if those don’t exist, do more “science” (see above) to understand the problem space. Apprenticeship from a wizard is the only “short-cut” to RTFM and/or doing “science”. But it’s not a substitute for those two things and should be used sparingly.

Accurately defining the problem is more than half the battle.

Sometimes if you try to formulate your problem in such a way that you can describe it to someone else you’ll solve your own problem. This is because you’ve only now “defined your problem in concrete terms” (see above). Reaching out to others and formulating your problem is a valid problem solving method.

Problems are only truly resolved from at the root.

Solve root issues not symptoms. Solving symptoms is little better than a “bandaid” in most cases, and in some cases creates more “problems” to be solved. The only right solution is solving the root issue. QED

The law of parsimony and problem solving. #

The simplest solution is almost always the best solution. The absolute simplest solution is no solution at all. For this reason, it’s worth putting serious thought into if your thing should exist at all. After all, existence creates problems that then need to be solved. Entropy dictates this.

On the observability of problems #

AKA quantum problems

Some problems are in a special class of problems analagous to quantum particles in that observation of the problem affects the problem itself. You can know either a particle’s position or velocity, but not both. Observation of the problem will change the nature of the problem. The good news is the solution is usually static even if observations are subject to change. The wise sysadmin knows this, and is thus unsurprised when different observational tools produce different results for the same problems. Only through testing and continued observation can the problem space converge towards a “limit.” Using the calculus analogy most problems converge, but some classes of problems (i.e. impossible problems) diverge. Experience is required to identify divergent probems, and mastery required to be able to set aside such problems.

Don’t go looking for problems.

In a similar fashion, some problems don’t exist until they are observed. These are the “if a tree falls in the forest but no one is there to hear it, does it make a sound?” class of problems. For this reason it’s important to make sure that monitoring and alerting tools are calibrated effectively such that you’re not creating a limitless supply of “problems” for yourself and your team. Problems are literally infinite if you look hard enough, but the solution space is for all practical purposes finite.

Triage problems ruthlessly or forever be chasing the receding horizon that is your problem set.

Triaging problems is essential. Some problems are not really problems in that the system is robust enough to survive uncritical “issues”, in many cases indefinitely. Given this, are they really problems? Other problems kill the host and should be treated with utmost urgency. Most problems are somewhere in-between these two extremes.

Premature optimization #

Performance optimization is a class of problems that may be both uncritical or critical depending on the context. At some point performance optimization becomes “yak shaving” and is best avoided.

Fear of failure #

Backups are only as valuable as the regular testing and documentation of your disaster recovery solution. Without testing, your backup solution is little more than worthless.

HA is only as good as it’s implementation and in practice only as good as the sysadmins maintaining it. Overly complex or difficult to maintain HA systems may in fact be less reliable and result in more downtime in practice than a single monolithic system. Think carefully before introducing this level of complexity to your system. HA is by no means a silver-bullet and should be implemented sparingly. An undocumented HA solution may as well not exist. Anybody who asks for HA as a solution to the class of “all problems resulting in downtime” doesn’t truly understand HA. (See educating the customer above)

DR tests don’t always go according to plan, that’s why they’re called DR “tests”. If a customer gets upset when a DR test doesn’t go according to plan they need to be “educated” (see human problems above).

Theory of knowledge (aka learning) #

If don’t truly “know” something until you can describe it in a way that a 5 year old can understand.

You don’t really understand something until you can build something with it.

Learning vs. “Education” #

There’s no substitute for experience.

You can’t learn how to be a developer/sysadmin/devops in a 12-week bootcamp, despite marketing to the contrary. You can, however, get a job after, and then “learn” those things.

Formal education in computer science is weakly correlated with future success in the field, if it’s correlated at all. It could be argued that IT is really more of an “apprenticeship” and for most practical purposes it’s best approached with “hands-on” learning.

Certs are resume builders.

IT certificates are little better than bullet points on a resume that get your foot in the door with recruiters. Most IT professionals could care less and the proof is the the pudding so-to-speak. It’s possible to study for the test and not really understand a lot of what you “learned.” It’s even more common to cram for the test and forget 90% of what you learned.The only exception is skills learned in a lab that closely approximates real-life scenarios. We only care about the skills you can demonstrate and being able to solve real problems in the field. Willingness to learn, attitude, and analytical thinking skills are far more important than any certification. It’s your approach to a problem that’s unfamiliar to you that’s much more important than whether or not you get the right answer at least as far as interviewing is concerned.

How to learn #

Learn by doing is best.

Portfolio projects are a much better indicator of true skills and knowledge.

Project-based learning that solves real-world problems is the best way to learn IT skills. The best lab for the IT professional is the homelab. The monetary investment in a homelab is completely dwarfed by future salary compensation. In the long-term it’s a negligible expense.

If you’re not interested in building a homelab and learning in your off-hours, seriously reconsider whether or not this is the right field for you. It’s difficult to progress (at least at first) without a natural affinity for learning and being driven by curiosity. Later in your career you may be able to afford to “rest on your laurels” and coast in your career. But expect to put a significant amount of free-time towards learning and skills acquisition your first few years.

Security #

People are usually the weakest link when it comes to security.