When resources are scarce, interesting stuff happens. Nowadays most outages are either due to expired payment, disk full, or provider outage. That's a good thing I guess? Just much less fun.
It seems nearly everyone goes through a “How hard can it be?” phase where self-hosting seems like it will be a trivial way to save a lot of money. Then you get to learn about all of the things that can, and will, go wrong over time. Great learning experience.
And then there are the rose-colored glasses people who seem to forget all of the troubles they go through with self-hosting. A few companies back we had a guy who insisted on self-hosting some things on premises to save on SaaS costs. The amount of time we spent waiting for him to figure out why the server was misbehaving each week easily cost several times more than any hypothetical savings. Yet when you ask him, he’d claim it was virtually hassle-free. Management liked it because it “saved a lot of money”… as long as you ignore the cost of the time we lost to dealing with it.
bean counters tend to not evaluate your time as lost money. they are paying you anyways, so sunk cost fallacy or something. as long as they are not having to pay an invoice, they are "saving" money
I’m not going to argue with your points, but we’ve spent maybe a months worth of man hours figuring out why some SaaS service wasn’t working this year. Only for it to be because some Azure or Office365 or Sharepoint online service wasn’t healthy, and, to be fair, because some of our systems aren’t build very maintainable so that it takes a ridiculous amount of time to figure out what is wrong. Still, it would’ve probably cost us maybe a work-weeks worth of time if we were more competent, which sort of goes against the SaaS idea when the licensing is also approaching the cost of self-hosting and hiring a Sharepoint person. Well, if a Sharepoint person existed of course, but I bet you get my point.
I wonder how much we spend babysitting slop (either directly or indirectly through SaaS) vs how much it would cost to invest in engineering around automated and reliable deployment and resilient software?
Highly polished distributed file systems, databases, and orchestration tools that incorporated automatic replication and fail over would be a great start.
Don’t just say “it’s hard.” Of course it is. But is it cheaper to do this or to babysit rickety piles of junk?
Another possible way out is AI sysadmins. I wonder how far we are from AIs that can admin a cluster including upgrades and disaster recovery?
The house's floor slab had a crack that wicked up groundwater when it rained too much, so my carpet would get mushy. Electronics off!
The anti-cheat software I ran (PunkBusters) went crazy after an update and banned anybody who connected. I noticed when my MRTG graphs flatlined.
Girlfriend once yanked out the Ethernet cable when we were having adult time. I didn't bother explaining that one to the players.
Heavy web browsing would cause lag, so I ended up learning how to do QoS via iptables (or its precursor, whose name I forgot).
At the time, the server used an astonishing 25 gigabytes of bandwidth a month. Always raised an eyebrow with the ISP tech when I called in, but they never complained.
Thanks for reminding me I'm getting too old. :V
1200 bps Unix shell access from TENet (Texas Educators’ Network) - thank you, Mr. Horner, my 9th grade US History teacher, for telling that nerdy girl who turned in all her papers in dot matrix print about what she could get her mother, a fellow teacher, to pay all of $5/year for.
Oh, and daring to have us look at Reconstruction’s sudden cut-off, Jim Crow and the fact that we were less than 30 years from its legal end in Texas, the Bonus Army, our rather late entry into WW2, the Japanese-American internment camps, and various shenanigans to do with the Vietnam War - you taught us that it’s possible to both love your country and to accept that it is by no means perfect.
I hope you’re enjoying a good retirement.
I still have ADSL. It's now 5 Mbps, but I don't recall what it was when dial up was common.
They had a message up for a while that said they were unavailable because their servers were either underwater, or on the back of a looter's truck that was itself underwater.
Wow, I haven't seen or heard that name since the 2000s. Good times.
So did my work computer;)
I wonder if people will experiment more?
- https://www.eevblog.com/forum/repair/trick-the-hotspotphone-...
- https://jibout.com/verizon-7730l-mifi-hotspot-battery-bypass...
- https://old.reddit.com/r/Calyx/comments/lorkrv/running_mifi_...
I feel like there's some liability on Netgear's part here: People can't be expected to know they can't leave it connected and a charging circuit should not constantly feed the battery.
It's a shame there aren't more affordable connectivity options for projects like this. Hotspots with batteries tend to be a lot cheaper than battery-less routers and USB dongles. The latter of which isn't even available for 5G.
I "fixed" it by plugging the charger into an old school mechanical timer. Every six hours, it runs off the battery for 30 minutes. Has been working great for 2 years.
There's tons of options, and the "professional" grade routers aren't much more expensive than that consumer grade AC800s. With the one I have I get a removable SIM, dual SMA antenna connectors for MIMO with the ability to have an external high gain antenna, 5 ethernet ports, and a box that runs a version of ddWRT that I have full control over.
The AC800s is $200, my modem was $350.
To me this seems like a common misconception. I can’t see it being true - we’d see so many more battery fires out there if this was true, including in this very case (his setup would’ve burnt down within days in that case).
I wrote a script that scrapes the hotspot web interface to turn off a smart plug when it's at 90% and turn back on at 60%.
Rocket sticks come to mind.
What are the statistics on lithium-ion battery failures? Are they dangerous full stop, dangerous in certain scenarios (e.g. heat), or do we only hear about the failures when 99.9999… of the time they are fine as they are everywhere?
Are all lithium-ion batteries created the same?
Think about it in terms of "A 33kWH battery roughly equals 1 gallon of gasoline". This is the most paranoid possible way to look at it, but it does help to establish some perspective. Energy is energy and if you don't really think it through, you may wind up with much worse than LTE modem/router fires.
Tesla power walls are an example of something I could never install in my home due to this simple math. I would not be able to sleep at night knowing there is a chance they could go up while bolted to my home. I've got one of those EcoFlow batteries (2kWH) for emergencies, but I keep it on a concrete slab (protected from weather) outside my home.
Go look at how a datacenter does their batteries if you want to get a sense of this very same paranoia at scale. All the DC-scale UPS systems I've seen typically consist of gigantic, low voltage cells installed in a manner expressly designed to deal with the worst case.
I think the density of the energy is what is most dangerous in my mind. A whole room full of lead acid batteries might not be very portable, but its a hell of a lot easier to inspect room-scale components, prove they are safe, and deal with emergencies before they turn into catastrophes.
Nah, that's the wrong way to think about it. Oxidation (well, burning) of lithium releases far more energy than the electric charge of the battery. Electric (dis)charge merely gives the initial heat to break structural integrity and set lithium on fire.
And even then the problem is not in the amount of energy, but in that it's almost impossible to extinguish a lithium fire. And the fumes from battery are thick and toxic.
> I think the density of the energy is what is most dangerous in my mind. A whole room full of lead acid batteries might not be very portable...
Nah again. Lead-acid just does not burn so spectacularly. And about the energy density - filled fuel tank of a regular car has much higher energy density than Tesla battery, but is significantly harder to ignite, easier to extinguish, and when burning, releases the energy slower.
I am honestly amazed the whole setup lasted for so long.
Regarding temps, the modem itself, I seem to remember, never reported a temp higher than 55 degrees C. I don't live in a very warm place though...
Battery management aside, those portable 12v fridges are great at using a fan to heat or cool
No
They have different chemistries with different characteristics. Some are safer than others
See here https://en.m.wikipedia.org/wiki/Comparison_of_commercial_bat...
Beyond the actual chemistry there's regulators of various quality and strategies that can be found in many modern battery packs.
I'm not a professional in battery manufacturing but I'm sure there's somebody around here who is that can give more details
If you wanna get more paranoid: https://www.reddit.com/r/spicypillows/
Jokes aside, hard to say, but it's mostly scenarios.
Some batteries have proper venting (or puffing), protection circuits and sensors to prevent thermal runaway, and they are very unlikely to fail catastrophically, they would stop working before, generally.
Now those take more space, so such features are often offloaded onto the PCB designers to make them fit, and you may trust them more (or less) than the battery manufacturer.
More or less the same can be said about the charging circuit, and how it might handle low voltage or trickling/stop charge.
Personally I've seen flashlights that burn through the 18650 without respecting the minimum voltage whatsoever, and that would put 5V to charge. There a proper battery will refuse to charge or just die, but an unprotected one might catch fire.
As anecdotal data, after dozens and dozens of devices, I've only owned two batteries that became spicy (without catching fire), a Chinese-brand laptop and a cheap quadcopter battery.
The cheap ZTE devices in particular seem especially bad at keeping the thermals within a safe range. Even with a 100% full battery they'll let the electrolyte boil if left plugged in for their charging logic to do as it wishes in a hot environment.
What's so frustrating is practically every consumer electronics device seems to now have some form of LiPo pouch cells in them. I wish it were normal for manufacturers to offer chunkier variants using NiMH AA/AAA cells to consumers. Instead I've been resorting to "industrial" stuff having no battery at all like the Gl-Inet X300B [0] I eventually ended up with.
For a while I used a hacked together pack of four NiMH AAs with a battery board stolen from the LiPo on a ZTE/AT&T hotspot that kept swelling its pack. The battery board kept the device happy enough to believe it had the proper LiPo connected. The now external AAs stayed charged more-or-less, and didn't care how hot the ambient temps got through summer. But it was a sprawling mess of wires, soldered AAs arranged like an 80s-era RC-12L saddle-pack, kept ~together with Duck tape.
They had enough voltage to let the thing turn on and weren't going to catch fire, problem solved. Left it that way for years.
When aggressively overcharged they may rupture, but they don't go up in flames like Li-ion does.
One brand I use is ATP for my ruggedized systems[0]. Yes, they may be an order of magnitude more expensive than even a high endurance from SanDisk or Samsung, but I started using these in 2015 or 2016 and found they hold in harsh terrain, tolerate brown outs (these devices are solar powered and run off of super capacitors), and JustWorks(tm).
[0]: https://www.digikey.com/en/products/detail/atp-electronics-i...
(/sarcasm, obviously, I have no idea how those things are legal)
Where they differ is how the cells are organized, safety measures taken outside of the cells themselves and things like cooling, mechanical construction of the assembly and so on. And even big manufacturers get things wrong, notably: Bosch. They have a pannier mounted e-bike battery pack that has serious problems with water ingestion, boards frying and balancing wires catching fire.
I'm looking for some kind of tablet that is wall-mountable but doesn't contain a Lithium battery. Some other kind of safe battery (NiMH?) that can last through momentary brownouts and power cuts of a few minutes would be nice.
The tablets are wall-mounted in multiple places elsewhere in the apartment. I don't want them to go into a full reboot cycle if power is lost only for a couple of seconds or even minutes.
Maybe there was a fault in the circuitry to begin with.
https://hackaday.com/2021/02/12/pcb-mods-silence-voltage-war...
I believe the RPi and the car battery they were using are still fine.
[1] https://www.netgear.com/images/datasheet/mobile/ac800s_optus...
But you have two tiny potential bombs inserted into your head.
[1]: https://louwrentius.com/my-solar-powered-blog-is-now-on-lith...
But, this is not really a failure of solar power system, just failure of a Netgear device. It seems that it used it's battery as a high current source for when it needed more peak power for wireless transmission, thus the instability and failures when just plugged to USB charger. I don't think such constant drain and constant recharge was really healthy for the battery. I wonder if it was/is stated in manual that it was supposed to be used charged and unplugged and not really as full time usage device.
This really isn't on them at all, a properly designed Li-ion system will be absolutely fine left plugged in 24/7. Netgear either doesn't understand how to design a safe battery powered device, or this was a rare case of a bad battery just going off regardless of how you treat it.
Of course the router isn't as nice of a software environment as a pi but that's part of the fun, no? If you want easy just use GitHub pages or square space or whatever.
