undefined | Better HN

0 pointsDrPhish8y ago0 comments

It was cheap to set up, all the cost is in time to learn to implement in a secure and performant fashion

Hardware actually cost money. Here's a breakdown:

> 4 pcengines alix boxes for openbsd router/firewall appliances

These were around $120 each with 4GB flash storage at the time. They're half that now. Low power, no cooling required, x86, 3 ethernet. You could buy APU2 now for more power

> 3 supermicro Opteron servers with KVM/corosync/Sheepdog/csync2 for hosting all VMs

I used cheap cases, eBay MB/CPU/RAM, tiered storage (green/black/ssd) to keep costs down and infiniband for 10gbit interconnects ($15/card on ebay!). I made sure to get quality components (esp MB/power supplies). One of the servers is also my desktop. I'm guessing they were about $800 each. Having the 3 node cluster is nice. When we had a forest fire threaten our town and we were ordered to evacuate I just grabbed one box and all my data was already replicated to it. When we returned home I plugged it back in and it re-synched back up.

Maintenance has been a non-issue. I haven't had any components die except the occasional HD. Power costs are the main thing. Estimate about $500/yr at $0.10/KWh

> PF + CARP + pfsync + OpenBGPD for routing

I use a local indy ISP that gives me a bunch of static IPs and lets me route a /29 with BGP. I know them fairly well, and get a sweet deal. Doesn't cost more than a regular consumer connect, though it is slower. I've used OpenBSD since the early 2.x days, and find it very easy to administer. The release and documentation quality are second to none, and I've found the community to be very helpful as long as you've tried to help yourself first.

> Unbound + NSD + Bind for DNS I keep my Bind server vlanned off and serve everything out thru unbound/nsd. Both of those programs are very easy to set up, the real beast being bind. I know there are better alternatives out there, but I know Bind well and have lots of custom config I don't want to throw away.

> SSH/OpenBSD ipsec/apache Guacamole for roaming and permanent site-to-site VPN (pcengines ALIX hosted at my inlaws in Japan)

If I were to pick one outstanding program on this list it'd probably be Guacamole. Pure HTML5 rdp/vnc/ssh/telnet/etc client that is seriously amazing. I've set it up at a half dozen places now, and it's never so much as hiccuped. OpenBSD IPSEC is VERY easy to set up, if you've had nightmare experiences with other packages!

> Apache + Lets Encrypt + awstats + relayd for serving web pages and analysis

Apache is the old standard, and awstats is cool for keeping tabs on what is going on in the logs (geoip as well). Lets encrypt was amazingly easy. I'm using certbot and set it up in under an hour. I'm forcing SSL on all my web services now. Relayd is another "so simple and it just works" package from OpenBSD. I use it as a front-end load balancer.

> ZoneMinder for video monitoring. Tied into legacy security system for automation

On Debian (my Linux distro of choice), this was simple to set up. Perl scripts to integrate it into my DSC security system. Auto arm/disarm camera recording and relay light control required the IT serial integration board and programmers manual for the system

> Postgres for database work. Some mysql/redis

I've been using postgres forever, so setup and use are second nature. An absolutely incredible piece of software engineering.

> NetDisco + Nagios + NagVis + NFSen + MRTG + Smokeping + PNP4Nagios + NUT + Splunk + Racktables for monitoring. All configs are dynamically generated from netdisco db

This is another stack I've set up at many locations (including businesses). They are a real timesink to integrate together. I have MANY custom scripts to make the config generation from netdisco work properly, but once setup you have total insight into every aspect of your network (and I forgot to list RANCID!). Netdisco/NFSen on their own are still a killer combination, and work as well or better than packages that cost tens of thousands of dollars. I'm happy to help any one trying to set these up if you PM me.

> OpenSMTPD + Citadel (webcit) for email delivery and webmail

Citadel is maybe the weakest thing I have in my stack. I'm looking at the other webmail solutions in this thread carefully

> Minetest server for kids. We use this tons as a family, and the kids spend lots of time modding. TW2002 server. TShock server.

When your kids are asking to learn LUA, you know something is working!

> OpenELEC for diskless netboot KODI machines around the house

Amazing and easy to set up if you already have your own DHCP server you can modify. Just need tftp and nfs after that. Using OLD desktop PCs for this works great. I'm using cast off dell gx290s

> Samba4 Domain controller + NFS for sharing files in different applications

I've been using Samba4 since pre-alpha (TP series) when you had to run your own LDAP server. Things are so easy now its hard to overstate. Using Bind makes it a bit trickier since I need to add some magic entries, but if you use the builtin its a single python script between you and a full SSO AD domain.

> SVN for source control and Config diffs for all servers/tools/network devices

I found SVN config to be a bit of a head-scratcher. I think this is another one where other tools are probably better nowadays. I'm looking at some of the other things people are suggesting.

> Asterisk via FreePBX / NCID for all phone/CallerID services, including remote handsets at VPN locations.

Another timesink. PBXs are hard to configure, and I'd move to another system if there was something less esoteric.

As to why I do it? I find it satisfying to learn how things work, like the idea that I'm master of my own destiny and know how my data is being used

Any specific questions, let me know!

0 comments

ge968y ago

Not original asker

How do you secure your publicly accessible IP at home from the web. I see you use static IP's I guess it's no different than renting a sever? You probably have separate networks (local home vs. serving)

DrPhishOP8y ago

Looking at Netdisco I have 12 VLANs. ISP1, ISP2 (from Dad's house across the valley), OSPF backbone, DMZ, LAN, Guest, Cameras, Servers, Japan, Kids, Management and Test. Everything passes thru PF and only exposes services as required. The Guest VLAN, for example, has no way to get to the rest of the VLANs. Camera VLAN can only talk to the DVR software, etc.

The important thing with securing my public IPs (as with securing anything) is to understand the surface area and minimize risks. With the number of services I'm exposing, I have to be careful. First thing is to keep all OSs up to date. OpenBSD every 6 months and Debian on an ongoing basis.

Next, whenever possible use single-purpose proxies that have been well audited. In my case OpenSMTPD, Unbound and NSD protect the always popular mail and dns servers from attack.

Keeping complicated things off the internet is important. Big CMS packages are constantly under attack. I only run either static pages on my webserver, or very carefully audited custom PHP pages (written with an attacker's eye to exploit injection)

When I expose something more complicated like Guacamole or Citadel that might end up with a known hole in the login screen, I put it behind both SSL and an HTTP simple auth login prompt. It's ancient, and so well tested that it is unlikely to end up with an exploit against it. Or at least less likely than the app its protecting. I've actually been toying around with doing 2 factor auth on these services with a dynamically generated simple auth sent via SMS bridge or IFTTT...

On my internal network, I have my family running linux wherever possible, and keep Java and Flash off the windows PCs. Everyone runs with a nonprivledged account and the OS/software packages/PDF readers are all kept up to date. I think the wife and kid's PCs are probably the most likely things to be compromised on the network, so I keep a close eye on them.

I also take the reactive approach of log alerting, change monitoring and general diligence to the state of my network. If you know what has changed in the last 24 hours it tends to be a small enough dataset to eyeball, and will tend to tip you off when things are wonky. I also have enough logging and data sources (syslog, configs diffs, apache logs, nfsen data, mrtg data) that it would be exceptionally difficult for an attacker to wipe out all trace of their presence.

I do not know of a successful attack against my network, tho I'm not naive enough to think that it hasn't happened. I just haven't realized it if it has.

Its a great hobby if you like this stuff!

ge968y ago

Wow thanks a lot for the details this is not my field haha regarding the tech you mentioned in the beginning.

Curious what your family thinks having to run Linux. It's my primary os myself with i3-wm. People usually complain "install apps command line" ui, etc .. whatever Ubuntu being as supported as it is that UI is so slow/bloated IMO.

I have Windows as well but use Virtual box and Linux to access the web if it's not just YouTube/Gmail/regular pages with adblock/uorigin running.

I don't even know how many ports there are on computers, I mean in my experience I've seen up to port 10000, I think you have to enable them? I usually use 21/22/80/443 though I've seen 3000 (websocket), 4200 (angular), 246 or 286 (windows RDP) I don't know...

It's a lot to keep track of web security. OSWAP

Anyway thanks for your time rambled on my end.

3 more replies

atmosx8y ago

Sounds like a full time job to me. I mean, I'm getting paid, to monitor less than this...

j / k navigate · click thread line to collapse

0 comments

ge968y ago

Not original asker

DrPhishOP8y ago

Next, whenever possible use single-purpose proxies that have been well audited. In my case OpenSMTPD, Unbound and NSD protect the always popular mail and dns servers from attack.

I do not know of a successful attack against my network, tho I'm not naive enough to think that it hasn't happened. I just haven't realized it if it has.

Its a great hobby if you like this stuff!

ge968y ago

Wow thanks a lot for the details this is not my field haha regarding the tech you mentioned in the beginning.

I have Windows as well but use Virtual box and Linux to access the web if it's not just YouTube/Gmail/regular pages with adblock/uorigin running.

It's a lot to keep track of web security. OSWAP

Anyway thanks for your time rambled on my end.

3 more replies

atmosx8y ago

Sounds like a full time job to me. I mean, I'm getting paid, to monitor less than this...

j / k navigate · click thread line to collapse