* Bonus points for tools whose authors have made the effort to explicitly compare it with competitor tools, particularly ones that acknowledge points where the competitor might have the advantage. ("Our new tech is better than existing old tech in every possible way" gets the side-eye from me; "Our new tech is better than existing old tech for these particular purposes, but old tech may still be more appropriate for these other purposes" goes a tremendous way towards confirming that the new tech has a real reason to exist.
* Is there documentation? Is it any good? This is a really low bar, but far too many new tools have no documentation at all ("just check out the source code") or have minimal, incomplete, or tautological docs ("bar.foo(): executes the foo method of bar"). A message board or IRC channel is nice, but not a substitute.
* How big is the API surface? Does it need to be that big? I tend to avoid tools where there are six different ways to do the same thing -- looking at you, Angular -- it suggests the developers are unfocused or in disagreement, and makes it harder to find support or documentation on any particular issue. Same thing if the API has undergone major breaking changes or paradigm shifts between versions (looking at you, Angular...)
* What does the tag look like on stackoverflow? This serves as a good indicator of whether the tech is too new or obscure to bother with, what the common pain points are, the average skill/knowledge level of its users, and whether help will be available if I get stuck when using it.
* Is there a relatively simple way to try it out? I'm much more likely to experiment with something where I can clone a repo and get going with simple but nontrivial example code; if I have to reconfigure half the settings on my machine just to get a hello world, I'm not going to bother.
Maybe?
Oddly enough, as I'm rereading through my own list I'm just now noticing how many of them really boil down to assessing the quality of the tech based on the humility of the tech's inventors: if they understand that their code isn't self-explanatory, and that good documentation is important, and that it isn't necessarily perfect for everything just because they invented it, then that implies a lot more care and deliberation in the code itself.
I definitely understand your reason for prioritizing documentation but I wanted to point out that "documentation" actually accomplishes the opposite of the OP's concerns about warts/disadvantages/gotchas/etc:
>, but also I want to describe stuff that I just hated or disliked. Because I've noticed that people always try to praise the new tech they are using, but rarely point out bad things.
As we know, documentation is typically written by people who are generally positive about the programming language or technology. Hoping for documentation authors to point out the flaws is like asking a mother to list the reasons her son is defective and girls shouldn't marry her.
To echo the specific issue the OP mentioned, my [totally unrealistic] dream documentation would be written by a crusty skeptical person that was an expert in the technology but became disillusioned by it. They'd point out all the flaws and pathological (but realistic) use cases where the technology fails or is inappropriate.
Since official documentation doesn't include the contrarian viewpoint, the newbies trying to evaluate new technology have to synthesize the "negatives" from other sources. E.g. I've typed the following phrases into google search:
"golang sucks"
"disadvantages of golang"
"rust sucks"
"disadvantages of rust"
"disadvantages of python"
"disadvantages of functional programming"
"disadvantages of Linux"
"disadvantages of <anything>"
By all means, read the official and blessed documentation of the technology but be aware that it's a very biased viewpoint.
EDIT: added "[totally unrealistic]" to prevent misunderstanding of my point
I want to know when method X of interface Y is called by the framework, with what parameters, and what it’s expected to return. This is a low bar that shockingly few software projects meet.
I agree that evaluating the contrarian side is quite valuable, however both sides need to be taken with a grain of salt. I think that sometimes reading the negatives is more helpful, because people tend to be a little less careful in exposing their prejudices, or tend to be drawn to the more clickbait-y side. This makes the shallow criticisms more obvious, except for the cases where it manages to exploit your own prejudices.
Or you just imagined your ideal choosing process?
Or perhaps you use what you employer's previous employee chose to use.
My time is limited for learning any new to me technology. If it doesn’t either lead to me making more money in the future or remaining competitive, I don’t learn it.
Java and C# have been the go to enterprise languages for at least 15+ years. Sql Server/Oracle/MySQL have been three of the most popular database for more than 15 years. JavaScript has been in demand forever.
AWS adds stuff all of the time, but the basics still work like they always have.
I avoid trying to keep up with the latest $cool_kids_front_end_framework for just that reason. It’s about knowing where a technology is in the hype cycle.
I also acknowledge that one day, I might wake up to a stinking pile of tech-crap, but even then at least I know that at the time it felt right.
For example, many years ago, in a decade far far away (The 90s), I went with ColdFusion as a tech stack. Back then it was that, Perl or maybe TCL. ColdFusion felt right because it allowed very rapid prototyping with a clear syntax and batteries included. There was nothing like it. Fast forward a few years later and that tech was so smelly it made everyone nauseous but I knew at the time it made perfect sense, and by then other options presented themselves.
I mean, the whole time I knew it was a doomed language, far too philosophically pure to be practical, and the signs that XML was a mass hysteria were there from the beginning -- but it was exactly the tool I needed at the time I needed it.
Never mistake popularity for quality. Most of the time they have negative correlation.
Just take SAP, probably a pile of garbage in terms of quality, but it's paying its consultants handsomely and customers keep rolling in.
Just because a lot of people use it, it does not mean that any good people do.
Some technologies are very good in selecting novices only, and some others in selecting experts only. The first ones will always be more popular.
In fact, the average "expertness" of the documentation for the tech and its ecosystem is one of the things I pay most attention to when selecting tools. I've said "expertness" here because I have no better term for it, it does not mean extensive, correct, complete, or any such usual quality. It's more related to "are advanced uses documented?", "is it at the correct level of abstraction?", "are the examples for doing hype or real things?", "do the descriptions pose the proper point of view so they become simple?", and stuff like that.
It is implied here that the product must have issues which are so complicated that you need experts to solve them.
I think it's often the case that hype is not correlated with quality, it's correlated with marketing, social networks, perverse financial interests, click farms, voting rings, etc...
"quality: general excellence of standard or level."
I have built this standard up after a few mistakes, where we chose a tool just to realize pretty much everyone working on it was a extremo ultimate rockstar ninja, better known as first-year CS dropout with a god complex.
NB: I'm not saying all first-year dropouts are bad, or that having a degree is in any way mandatory in this field. All I'm saying, is that when our site is down, I'd like to have a few people with 1980s MIT Electrical Engineering degrees on my team. The kind of people who know what a process is, how TCP passes packets, et cetera.
Me? I'll keep doing distributed and low level programming in Haskell.
Thus, with PHP versus Python battles, for example, looking at libraries matters more. (PHP tends to be easier to set up for web-oriented projects. Something to consider for small or often-changing teams.)
Just type in the names of the languages/frameworks into a job search engine with "London" as the location. It'll show up in their hiring ads. If they aren't hiring, they're irrelevant.
When I'm trying to choose between competing technologies, I often choose based on popularity rather than isolated merits, as want to build upon something which other people will be contributing to, rather than something marked for death.
There are few cases where one technology is objectively and universally better than another. Most of the time, each has strengths and weaknesses. Often there is overlap between use cases but one is better for one type of usage and the other for another type (such as Redis/Rabbit). In other cases they’re pretty much interchangeable, but one may match more closely to my particular mental model (my experience with React/Vue/Angular); or one may be easier to get started with.
With that in mind, I’m usually looking for balanced information about specific trade-offs. What is this technology good at that the most similar alternatives aren’t as good at? Where do the alternatives excel that this tool struggles. How do the philosophies of the tools differ? What’s the learning curve like? What limitations did you stumble over only after using them for a while? How do the ecosystems compare?
Pick Django? You're also buying into the system of Django packages vs. those for Rails. Same with all the others. Although I think for the most part in the examples you selected, both choices could be equally valid for different projects which is why there are multiple strong choices for those application areas. I think that also means you probably won't go too far wrong picking one vs. the other. I find the choice is often made by how well the stack integrates with other stuff I'm already working on.
Or that the trade-offs have to do with differences in project philosophy (Rails’ convention over configuration vs Django’s explicit over implicit), which in turn impacts in what ways they’re flexible vs rigid in subtle ways, or what sorts of ecosystems grow up around them.
Often there isn’t a right choice or a wrong choice, but there are still trade-offs. And you or your particular team may be able to more easily absorb certain trade-offs than others.
Even if two technologies are fully equivalent, you’ll need some basis on which to base your decision and to use to build consensus so that you don’t spend the next three years fielding “we should have gone with Laravel” complaints twice a week.
Which is why those in-depth comparisons up front can be so valuable — if you can find them.
Elixir is great for a chat application. At the same time it's probably not the right choice for a Machine Learning project.
Of course there are a number of absolutes that I look for regardless of the context:
* How good is the documentation?
* Is it actively maintained?
* Is it mature?
* etc.
Relatedly, while I wouldn't use either for many types of ML...I wouldn't use Python either. The only benefit to using Python in this space is it has libraries bound atop C/C++ implementations. Erlang/Elixir doesn't, that I know of, but that doesn't prevent it from being done if someone wanted to. In terms of actually building something from the ground up...well, it depends. It isn't fast...but it does make the concurrency part easy to model ( https://www.amazon.com/Handbook-Neuroevolution-Through-Erlan... for instance), and for learning/prototyping that might be what you're prioritizing for. Certainly, Erlang/Elixir have seen use in high speed trading and ad bidding platforms, things known for needing low latency.
2. Is it elegant? In the general sense: things that are elegant are often also good designs.
3. Do people use it? This is a hard one, and stems from 25+ years of experience. No matter how beautiful or impressive the new technology is, there will be problems, and I am too old to iron them out myself. I've got things to do. So, tough as it may sound, these days I will not start using things that have not seen reasonable adoption. This does not mean I won't read about them, just not use them. And my criteria for "reasonable adoption" are not "everybody and their dog uses it", I just need to see a user base. Also, the threshold is higher for databases. Practical examples: Clojure and ClojureScript are fine, Datomic is not. RethinkDB just fell below the threshold and I have to migrate; FoundationDB is barely getting to the threshold.
Taking Elixir as an example, it fulfills all three criteria. I know what it is, I've seen it in action, I read about it and I keep in the back of my mind as a tool I might want to use when needed.
2. What makes it better/worse than other solutions?
3. How is the project run/what is the maintainer situation like?
4. What is the community like? Will I have a hard time finding someone else to work on this project competently if I use this tech?
5. How long has it been around / will it be around in 5 years?
Not too long ago, that statement would've been seen as satirical. Microsoft has left a huge graveyard of technologies behind. They seem to be on a better track now, but I'd still be wary.
If you are interested in my blog you can contact me: adam@wrestlerman.me and I will send you a link when it's ready:)
And now this might seem weird, but I have had good results following tech trends that attracted the widespread interest of hobbyists. These tend to be things that are easier to obtain, install, and use, and that have communities built up around them. The hobbyists will tend to weed out the things that are just too painful to use.
Examples over the years include Turbo Pascal, Visual Basic, PIC microcontrollers, Arduino, and Python.
Some of these are proprietary technologies, of course, but their vendors didn't abuse us too much (until Visual Basic went overboard with dot-net, whereupon I switched to Python). I got a good solid decade or more out of each of these things.
Of equal or greater importance to the use of these tools is what I can learn. I don't mind throwing a vendor a few bucks if I will use their tool to expand my knowledge, and if that knowledge is applicable to a broader range of things. For instance through Python and the community of developers, I've learned new disciplines that have made me a better programmer in any language.
If it’s for a task, I’d say:
1. Really spend some time to understand what you’re trying to solve 2. (If applicable) What pain point are you experiencing with existing solution? 3. Find tech (old or new) that might be a good solution, and understand their limitations/trade offs that you'll be making by choosing this tool.
Then, make a decision.
If it’s for learning, it ultimately is what are you trying to get out of the learning experience, and see if it fits your goal. (It is totally OK to learn a new tech just because it sounds cool - Learning more about something cool is a type of goals as well.)
The most important is an analysis of what use-cases is Technology X good for, and why? Every technical decision is a list of pros and cons. If it fits the use-case perfectly, that is the most important factor.
After that the most important factor I consider is community and momentum.
It's possible a technology is immature and lacks good documentation. BUT - if it has a rapidly growing community and momentum, these 'cons' will disappear rapidly.
1. Is it road-tested for a few years at least? Unless you are in R&D or a high-risk start-up, don't volunteer to be a guinea pig. 5 years is my rule of thumb.
2. Are there successes in similar organizations? One size does NOT fit all. Make sure it's useful in your particular organization in terms of domain (subject matter), culture, and company size.
3. Do the benefits over-emphasize a few narrow factors while ignoring others? There's rarely a free lunch; most decisions are balancing various trade-offs. There are probably down-sides that vendors or fans don't want you to know about or failed to notice due to enthusiasm bias.
4. Does it over-extrapolate current trends? For example, just because more UI's are going mobile does not mean every business is throwing out their mouse and big monitors. You may be limiting yourself by trying to make your UI's both mouse-friendly and finger-friendly even though most actual business users will be on a desktop. It's not always good to keep up with the Tech Kardashians; they are not always rational or timely.
5. Does it require a big learning curve or lots of prerequisites? If the new technology turns out to be mostly a fad instead of a real improvement, a long learning curve or expensive investments will drain your time and budget. Look for incremental improvements first.
6. Vague buzzwords or promises: Lack of specifics and realistic examples is a sign you are being had.
7. Experimenting is fine & recommended, but don't do it on production projects. If possible, introduce it to production gradually.
* are there any tutorials / code examples? (to see if I like its API/philosophy AND there is a way to pick it up)
* are there any practical, working projects out there (used by companies, etc)? (otherwise it may be not useful for bigger projects)
* is it in active development? (otherwise there is a risk that it will cease to be useful)
* how does it match against other tools (e.g. maybe it is easy to pick, but so are all other frameworks).
Also, I did write some comparisons. Vide: https://deepsense.ai/keras-or-pytorch/ (got popular here).
I look for a book. Books in general are usually miles better than technical blogs and you can find a ton of information addressed about a subject in one place, so there's the convenience factor (not that you couldn't create some program that indexes your bookmarks, but still have the problem of providing meaningful titles and organization for your bookmarks).
I also find that books are generally peer reviewed, especially textbooks, so the BS is kept to a minimum. Makes for a more boring read, but its more accurate.
That being said, I like language analysis posts. People tend to bring up a lot of things I haven't thought of, and there was one done on C a while back on HN that looked very good: https://eev.ee/blog/2016/12/01/lets-stop-copying-c/ (tbh, I skimmed the article because I didn't have a lot of time to read it at the moment).
This applies to everything open source, including languages. If you open up the codebase and go “wtf” that’s a problem. If you open up the codebase and go “I don’t understand this, but it looks clean and with some effort I could understand this” that’s gold.
This is especially true for libraries, you get a sense for the right size and right amount of complexity in dependencies.
One, usability over existing systems. It is a difficult one to actually answer but I find people talking about new technologies all the time and when you ask them - Okay, what can we do with it which older technology couldn't? Mostly I hear murmurs or barely justifiable answers. But, if the explanation is sound, I go for number 2.
Does the person talking about it has significant exposure into the problem space? Mostly you will see people talking about how X is great but never having to work with the nuances of an older tech Y.
Now this process has some bias built-in. As a supporter of older tech Y it is entirely possible to never find a reasonable explanation. The only way around it is to talk to as many people as you can.
Rationale is that in a world with limited time, I'd rather focus on solving problems external to the technology itself. If I have something in my toolbox that will work, then it's generally the easiest thing to use, rather than learning something new. There's less of an immediate learning curve and fewer of the issues associated with being an early adopter.
Where this changes are in situations where either the investment to learn a new technology is low enough or the potential for return high enough that the learning might be expected to produce a high ROI.
So what that means practically is that I'm looking for things where I either have an immediate commercial need to know it or a strong feeling it's likely to be useful in a way that none of my existing toolset will fulfill.
From the perspective of something like a programming language, this is part of the reason I've tended to like Lisp-family languages as an adjunct to C-family languages. They're different enough that they're more likely to be complementary to each other and it's likely to be easier to make choices about what code goes in which language.
and of course compare with existing and known. Very rarely does an old or new tech measure up against comparison... unless it offers something not present elsewhere.
And of course, bug reports (). If there's a long queue of bug reports I won't touch it until a significant number are addressed. (when I worked in commercial drupal dev, this is how we picked modules to use). If there are no bug reports or any kind of online reputation of ignoring or denying bug reports, it gets avoided. This is not unusual in project with particularly fragile egos in charge and therefore unreliable. () bug reports include feature requests, documentation requests and similar as well.
I'm also learning Elixir and writing about it at https://inquisitivedeveloper.com/. In the first couple of posts, I talk about what Elixir is really good at and what it isn't good at. I'm generally positive about it but I do grumble when I encounter something I feel could use improvement or doesn't make sense to me.
For example, Elixir is great for concurrent and scalable software. I'd use it to build a web service or game server, but it is unsuitable for a game client, physics simulations, or OS development. It's just not low-level enough for those purposes.
To sugar coat it is to just set up your reader for disappointment further down the line when they hit the limitations and realize that it's not suitable for their needs.
I also keep a general awareness of what technologies are out there and what they're good for. I've never used Redis, for example, but I know what it's good for. The same applied to RabbitMQ. I knew what it was good at even though I didn't use it. That lasted until I encountered a situation where it would be useful and I ended up introducing it into my organization.
How Eternal is the knowledge? Math is Eternal in the true sense, while that new webpage framework will be around for maybe just 2-5 years, and is thus not worth learning. (Although I am sometimes happy that people sacrifice their lifes on stuff like that, I would never do it.)
Vendor lock-in? I steer clear from that.
Is it being maintained? (Some things are okay not being maintained though. It depends on the type of tech.)
Surveillance potential. I am somewhat paranoid in my own personal life.
When procuring network-attached tech I usually dig through exploit-db and the CVEs and try to make myself a picture of how their security is being handled. Small companies are difficult to judge from this because they have little recorded history. (So I didn't buy any mikrotik router because of their shenanigan attitude to security, for example.)
Do the creator of the tech try to keep my hands away from digging into the machinery? Like hiding that it is really just a Linux/bsd box underneath? I stay away from that, if possible. (It's not always possible, almost all tech runs on Linux nowadays.)
I can be bribed to ignore any of these things that I usually avoid. For example, I learned MS Windows and some of their tech because I got bribed.
* Look at the open issues -- Is it full of buggy edge cases and "3d chess"?
* Quality and maturity -- Stuff works, as documented, under all possible conditions. Maintaining existing features / "tech debt" is prioritized over new features.
* What problems does it solve? What problems might it introduce? Does it actually solve the problems it's claimed to solve?
* Is it designed in a way that will totally break a possible future use case?
* How much of a "custom ecosystem" is there (bad), and how much of it is "open standards based" interfaces (good)
* Is it getting more or less complicated to use it over time? Can you learn it as a "simple cognitive model" or do you learn it as a "huge collection of random facts" e.g. "a cognitive edge case LUT"
* Documentation must be okay (no elasticsearch)
* History and sustainability -- How long has it been around, is it updated regularly (regularly != often), do updates break stuff (no pipenv)
Programming languages are their own category. They tend to not just be a project investment but a career investment. I generally at least get a feel for the language by reading some projects written in it and looking at code that does things I know how to do in the languages I already know. If there's some known strengths to the language (eg. "Go is good for servers", "Rust is good for system programs", …) I may be tempted to use the language for one such project as my first experience with it.
Software tech, things like what framework/DB engine to use, proprietary services such as AWS/GCE services and what not: Those I'll find when looking up how to solve a technical challenge I'm having at a particular moment. Reading up on it (documentation, known use cases, success stories, failure stories) is the only way to really decide. Then it's a matter of choosing the best fitting solution from the ones I found.
Important criteria:
- How well does this solve the problem at hand?
- Is it open source?
- If closed source, is it hard lock in or does it have open source tooling? (eg. Redshift uses postgres-compatible tooling/syntax, a huge benefit)
- How good is the code? (And what's it written in?)
- How good is the documentation?
- How popular is it? (= how likely am I to find help when faced with an obscure bug) / Is it an easy tech to hire for?
- How much do I trust its maintainers? (Especially: How much do I trust them to keep maintaining it and keep the project healthy)
- Is the pricing/licensing compatible and affordable for my use case?
The most difficult part of this process is knowing when not to use your own shitty hammer when there's much better hammers available for the current nail. It's easy to get stuck in a mindset of "The current tech, which I know, doesn't work really well for this use case but it can be made to work somewhat and that's good enough, no reason to look up better alternatives".
I'm always blown away when I discover a new piece of equipment I might have completely missed, were it not for actually googling/asking around for solutions to a new problem. Immediate discoverability of solutions kinda sucks when you aren't sure what to look for.
Documentation specifically addressing how to use in multiple different tech stacks. More generally, not assuming that everyone has the exact same pipeline as the core developer(s), and some evidence that they've made good faith efforts for the tool to work in other situations.
A lot of the things people hate about a tech is workable. It's better to pick something unusually good at something with multiple flaws (e.g. PHP/JS) rather than something with no flaws but isn't particularly good at anything.
It's the same thing with any other piece of technology for me. It's easy to build tech. But In my opinion, when I look at new tech, here is when it is genuinely better tech for me:
- Cross platform, solves toolchain issues
- Provides a great debugger/observability interface
- Integrate well with native/os/hardware. Open interface
- Has decent basic primitives and not the kitchen sink
- Well designed that the core is stable and is compatible for 5 years. "It's the design, stupid."
What scope does the technology cover. I believe in the rule of "Do one thing and do it well". If it suppose to do A, but can also be used for B and C is a warning sign to stay away.
Is it something a engineer built or a hacker built. Not saying a hacker can't be a engineer and such, but long term vs short term goals for technology need to be well planned out and executed. I use a ton of technology built by hackers, but anything I put in production is built by engineer.
Actor model? Type-classes? Algebraic data-types? Immutability by default? Type-level programming?
Second, what is the tool support? Do I get an IDE? A languace server? A config for vim/emacs? At least syntax-highlighting? How do the error messages look like? Compiler? Testing frameworks? Lint?
Third, I like to evaluate practicality. Can I imagine maintaining a cli app in this? A web server? Would I be able to connect to databases? Easily serialize data-structures?
Edit: also, active reddit community. I don't ask a lot of questions there, but it's a reasonably good metric of the health of something IMO.
Plus:
* is it tech i wanted before i knew it existed.
* tech clear on limitations / being unfit for case XYZ.
* community with low frustration levels.. where clear answers get clear questions.
Con:
* tech that is just the opinionated alternative of the month. Even if it catched on.
* tech website cant (or forgets to) explain solved problem in simple laymans terms.
* not actively maintained, must if facing external dependencies. (includes os, browsers, libraries, etc)
* docs are inconsistent/unversioned, or old versions are not kept available.
The features? I don't care. I need to understand the problem(s) that it solves. I need to understand the benefits __to me__.
Time is important. Absolutely! And while a free trial is helpful it is not a replacement for crisp and transparent communication. I cannot afford to spead half a day trialing only to find out Solution X isn't a good fit.
* Does it introduce a new way of doing something compared to how I'm currently doing it and is it plausible that it's substantially better.
* Does it do something similar to what I'm currently using but with significantly higher performance or reliability?
If any of the above are true, I will investigate and follow it and try it out on tests/projects matching its maturity.
Having less IDE features available, lack of graphical debugger, additional build systems, having to manually write FFI wrappers, not exposing all the features available in the platform languages, impedance mismatch to create libraries to be consumed by the platform languages are all issues that I might consider negative when evaluating new tech.
There's way more tech out there than I have time to learn. If it doesn't make some of my problems go away, then I don't have time for it...
... unless my employer pays for me to learn it.
I examine how easy it is to start playing with the capabilities of the technology today and the possibilities that it could be used for.
Can't learn swimming by study, research and reviews alone.
* How big is the learning curve?
* How is better than the competitors?
* How costly is to use it?
* Is it mature enough?
If not, I pass.
2. Is it actively maintained? (How) are issues dealt with?
3. How popular is it and what is the trajectory?
I don't want to be the first person to run into all the issues. I'll let the kids do that, they love shiny new technology.
I don't want to invest into a stopgap technology (like Coffeescript). My gut feeling is that Elixir is one of these technologies, it has some good ideas that will probably show up elsewhere in due time.
I do value "XYZ sucks" posts to some degree, but unless I already know the technology reasonably well I probably won't be able estimate the impact of the issues.