;)
My daughter loves looking back at home videos of the family. So embarrassing, so beautiful, so many little things about their personalities that would otherwise be forgotten.
Just a few hours ago I was checking backups and stumbled into a nest of folders that had some more home videos in it, and of my son just being his quirky little kid self. Gives my life meaning, shows me I did stuff ok.
Here's a practical example of it coming in handy: back in university I moved to another city for an internship and took some framed photos I had on my walls. When I moved back into my parents' house when the internship was over, the nails were still in the walls and I wanted to hang them the same way I had before but couldn't remember which photo went where. So I went into my Google Photos, typed "photo frames," and was able to find pictures in my library with the framed photos in the background, and piece together which photo went where to get everything back to where it had been before.
Yearly, we create printed photo albums that we look at regularly and show to our kids.
But all this only works after lots and lots of hours of manual work filtering the photos.
Still, over the last 15 years or so for two people we have way more than 100k photos stored on the NAS.
If I don't have any real reference points, its hard to measure things. Its hard to improve things you cannot measure. And sometimes it just feels good to help remember the good times with people no longer with us.
The thing is also to delete photos you don't want to keep long term. Most of my photos are only synced when I am on my local wifi, so until then having a good hygiene of deleting badly taken photos right after taking them.
Having said that I have recently started (late december) to do a weekly printing session of my key moments of each weeks. Well there have been weeks in january and february I didn't print anything. I sometimes only have one or two photo per day/event but since I put the date on the notebook next to the photo and keep all other photos it helps me find back the whole event if I want to see them all. I am using smaller photo format than 10x15cm, mostly zink 2"x3" adhesive photos and kodak instant print 3"x3" from instant cameras that have a printer function.
I should blog about that.
But you're right, otherwise I would never have time to dig back into old photos.
Geographical separation is important if you're serious.
I think that I'll give DigiKam another go, against the NAS loaded up.
Thanks for the comments!
That ARM machine also has the responsibility of making backups, local to a USB drive, as well as to another cloud. Not mirrors, but proper versioned backups (as in Restic, Borg, Arq, Duplicacy, Kopia, etc).
I also maintain a couple of USB drives with yearly updated mirrors of the entire photo library. The drives are stored at geographically different locations, and surface scanned, updated and rotated yearly.
And finally, as a "last ditch recovery", i maintain an archive of M-disc Blu-Ray discs that contain a complete copy of our family photo library. Every year i make an identical set of discs containing the past years photos, and these sets are stored alongside the USB drives.
I don't bother archiving documents as everything that is important is stored on government servers anyway, or exists in hardcopy. Also, if every step in my normal 3-2-1 backup scheme has failed and i need to recover from the archive, i probably have bigger issues than retrieving my budget for this years finances.
Isn't that 2 or 3 orders of magnitude less space though?
I do understand where you're coming from for sure.
I'm also looking for something FOSS that can do basic face recognition + maybe even more, but last time I checked DigiKam's detection didn't work so well, or maybe I got spoiled by the detection in Google Photos. If you do give it a go, would be nice if you reported back on your experience :)
The quality of this software is overall extremely good. It is a solo-developer as well so you might want to consider sponsoring them if you end up using it.
I am not affiliated, just a happy user :)
https://github.com/LibrePhotos/librephotos
I may have missed some: https://github.com/awesome-selfhosted/awesome-selfhosted#pho...
That being said - I think the best additional backup is to store important things on BluRay - producers estimate they should last 80-100 years and so far never had a bad disk, even those burned years ago.
It will resume when you hook the drive back up, and work around dead sectors.
It was slow processing the images, so took very long after adding new pictures to them turning up in searches.
One of the main attractions, searching geotagged images by location, wasn't smooth at all and quite clunky to use.
Tried it a few times last fall but gave up.
You are right about file systems doing a lousy job of helping you organize and locate photos, especially when they are spread across many different HDD and SDD drives attached to various computers. They are equally bad at organizing other forms of content (documents, videos, logs, music, software, etc.)
File systems were invented decades ago when the biggest hard drives could only hold a few thousand files at most, and drives were also so expensive that most people only had one of them. Traditional file systems are antiquated and need to be replaced with something better.
* A nextcloud+recognize (for face recog) instance running on Hetzner. $20/month (I co-host other services on the VM). This is attached to a Hetzner storage box.
* Syncthing pushes all of nextcloud to a zfs.rent (great service BTW) machine, I purchased my disks up front, so it's $cheap/month. ZFS snapshots are taken.
* Local RPi NAS. Syncthing up to the other two.
About $40/month in total. How valuable are 240k photos to you?
Uploading photos is a matter of sending them to Nextcloud. Syncthing does the rest.
I agree with all the points about having backups and/or using better service/self host etc., but IMO it's too early to conclude anything about this incident.
I do wish I have such luck when posting on HN though, often ended up with zero upvotes and no replies.
My experience couldn't be further from that.
When I look through my "Watch Later" list, probably 10% of videos are missing. I think you have to choose "show deleted".
I've been planning to set up a script that uses yt-dlp to auto-download the new videos on my Watch Later list every night, but haven't got around to it yet.
240k photos is 66 photos per day, every day, over a full decade.
If you're spending time taking 66 photos a day, I think you have bigger (personal) issues to contend with than your image storage solution.
Even on vacation, relaxing, with my kids and wife, visiting all manner of picturesque spots, 66 photos would be an enormous feat of narcissism.
The author is either lying about the photo library size to garner sympathy, or has a gargantuan psychological issue to address.
Time lapse photography is taking a photo every X seconds/minutes/days. If you want the video to look smooth, you need 24-30 shots per second, so a 30 second video has 720 exposures. So again, failed three time lapse attempts and I'm at 1% of OPs forever exposure count.
There's an awful lot of judgment in your comment, and the issue certainly seems like a lack of understanding on your part, rather than OP's "psychological issue" or "lying." I'll refrain from my own judgements, and suggest you watch David Foster Wallace's "This is Water," and when you're done, please watch it again. https://www.youtube.com/watch?v=eC7xzavzEKY
If you pay $15k a month for AWS enterprise support, you will find someone who may or may not be able to help you within 6 months after misinterpreting the problem and having to raise your account manager and wave your hands around.
A few years ago my (very non-technical) mother asked me to make and send her a photo album of recent pictures. My wife and I had so much fun doing it that we kept doing it. Choosing the best pictures, arranging them and making nice layouts and titles itself is a fun walk down memory lane.
Now we have a shelf with about a dozen volumes of memories and we found we are far more likely to look at these than we ever were with photos in cloud storage.
I like to think that if I lost all the digital copies, I would still be happy with what I have in these bound albums. I highly recommend this to everyone.
1) use iCloud Photos
2) find a Mac with a very large system drive (1tb or above)
3) open the photos app on mac and ensure settings is configured to “download originals” not “save space”
4) make sure the computer is online on a regular basis. You should see ~/Pictures/Photos.app becoming massive over time.
5) setup Time Machine for that computer
6) success! You have a second copy of your photos library on Time Machine now.
7) Use Arq.app to backup your Time Machine drive to AWS glacier. Nice! Now you have 3 copies of your library.
An alternative path is making sure you have both a giant iPhone storage and giant Mac storage. Make sure photos on phone is set to “download originals”. Manually backup your iPhone to Mac monthly. These backups include photos library. I recommend encrypting your backup and keeping a key in safe and 1Password. Continue with Time Machine and Arq.
My parents have photo albums going back a few generations. The family tradition is that they are passed to the cousin with the most children, to maintain for future generations. If I were handed 500K+ images in a drive or a cloud account from each family member, I doubt I would ever look at most of them. With a bound and curated album, I know they are full enjoyed by every family member with an interest in the past.
Hold Alt (Option) when starting the Photo app to invoke a dialog where you can select custom photo library location.
You can have multiple photo libraries in different locations and switch between them during Photo app start.
https://support.apple.com/guide/photos/create-additional-lib...
My wife and I now have an annual tradition of making physical albums of memories from the past year.
Making the album in January is not just fun - it makes you mindful of fond memories at a time when you're likely to think "Geez, another year has passed so quickly".
My only regret is realising I should have started the habit much earlier.
Professional photographers take a thousand photos, but they only use one. Quality, not quantity.
Wonder if there are stress tests on cloud storage that account for this possibility by means of adequate insulation or whatever it is that can counteract a solar flare.
Oh, one solar flare can solve global inequality in an instant! There money in all bank account would dissapear and the recently rich would have to pull themselves up by the bootstraps!
Besides that i keep USB drives and M-disc archives of photos, identical sets stored in geographically separate locations. Optical media should be resilient to most natural phenomenon like flooding and solar flares. They're not resilient to fires, which is why i keep duplicate sets.
https://spp.fas.org/starwars/congress/1999_h/99-10-07wood.ht...
Note: reader mode on Firefox Mobile makes this more legible for me.
Google is especially frustrating for me. By default, an Android tries to sync all videos and photos up to Google photos. You can turn that off, but not in any sort of granular way. For example, you can't say "sync photos, but don't sync videos". You USED to be able to do this, but it's all or nothing now. Which I suppose drives revenue for the paid storage plans?
It's annoying to have to do this, but at least I can finally choose what gets synced and what doesn't.
-- edit
I originally called the app "private cam" and said it was from fdroid, apologies! Forgot I had aliased it.
Getting users to add photos and then charging (while also using them as image sets for training) was the strategy from the beginning. They didn't resort to anything. Keeping your photos stored somewhere is very sticky, particularly for non-software developers. And even so how do you even transfer them to another service without a huge bill?
I did spell out what one of the changes was. I used to be able to sync photos, which for me isn't a lot of space...but also NOT SYNC videos, which do use a lot of space. That option was removed, and syncing can only be turned off/on globally now.
Separately, there was also this change, which is notable: https://support.google.com/photos/answer/10100180?hl=en
So, yes, I believe they once offered more storage and flexibility, then took that away.
I don't trust to open my network to the internet though. So I've created a VPN to connect with my ubiquity network from mobile. Sadly, I can't share photos with other folks or allow my family to participate easily.
I'm a layman in terms of networking. I always wonder how people can sleep if they open their home network. In this case either by using QuickConnect or forward ports and access the machine through a custom domain secured by TLS. I mean that would be perfect but I just can't convince me to take that burden of maintaining that open door into my network.
I've read about creating sub networks with the power of ubiquiti (or any other router), but I will never trust anything I configure there either because I can't confirm it's secure.
So I'm stuck with Google Photos additionally to Synology Photos because of the convenience of sharing.
It does fit some workflows very well though, you just have to assess if your needs over the next few years fit their business intentions.
I run a Mac VM on mine but via iscsi as the actual VM host hasn’t enough storage to hold my library.
There is only one significant issue that I found: Backing up large videos fails If I put the Syno behind Cloudflare. Because the photo backup app uploads it as one large HTTP form data and CF does not like it. So now I need to make do with basic firewalling at my place.
No local backup on a drive or memory-card? Local backups shall be stored in different locations (depends on importance) not reachable by network.
The cloud is the server of someone else and not a backup.
That's common knowledge among tech people but blaming the victim is not the solution here. These services are marketed as safe, reliable, and not needing backups (in fact sometimes they make it difficult to take backups, even more so for a non-technical user).
Maybe the regulation should be changed that any consumer-grade storage service either needs to provide adequate SLAs & compensation, or have to advertise in bold and prominent font (at least as prominent as their selling points) "WE MAY EAT YOUR DATA AT ANY TIME, PLAN ACCORDINGLY".
I don't think people care enough about SLAs to pay a premium, until something happens to their data. But by then it's too late. The bold warning is cute, but have you seen cigarette packaging in Europe lately?
PS: I should better do another backup…
This is basically you signing that we have failed as an industry.
No other industry expects it's users to operate with tbe diligence of a bank.
I'm a bit older than most, but that would be a photo every other hour since the minute I was born...
Kind of a digital hoarder moment. Which is relevant because my entire archive fits on a flash drive and I have a couple flash drives and its no big deal at that small of a scale. But a quarter mil pics will take some serious storage.
Now I could imagine if you owned five security cameras and stored a snapshot every minute for a month, that's about a fifth of a million photos, but that's all autogenerated and if you lose your complete set you'll have a new set in a month anyway.
Now, we select less than 10% for a private Flickr album that we actually enjoy looking at and sharing with the grandparents, etc. But so far it's been easier and safer to store the remainder instead of deleting them. I've gotten into trouble before with my wife when I deleted a photo that I thought was crummy, but she loved for some reason.
Edited to add: We also photograph all of the kids' artwork as a compromise instead of trying to save all that paper or hang it up somewhere in our house, which would be their preference. Those go into their own albums, which has actually turned out to be pretty neat.
But those nose-picking moments can be so easily utilised...usually when the kids get to 17-20yo and their partner 'Of course wants to see pics of when their loved one was a child'.
I’ll maybe take a couple a day when out for a walk. 10-50 for an event. 20-30 for a big family gathering. My kids were always a good excuse to pull out the camera. It all adds up.
If you shoot with a big camera, people frequently opt for “JPEG+RAW” which is two image files every time you tap the shutter release. That adds up.
Also, if you use(d) google photos, or any other service that downscaled your originals, and then _also_ directly take a backup of your DCIM directory off of your phone, there’s another source of duplicate images.
I wrote more about this here: https://photostructure.com/faq/what-do-you-mean-by-deduplica...
Why do they keep everything? Idk, /r/datahoarder is a subreddit that exists though so the mindset is out there
What I need to do though is get them all copied to my home backup server (the HTPC) on the regular...
That's insufficient. If one of then screws up, deletions will likely propagate to the other.
That's not the setup described at least. It's both of the services withing in parallel. > I use both Amazon Photos and Google Photos simultaneously
Your scenario would depend on Google being a backup for Amazon or vice versa.
So, yes, they could propagate but you have time to correct the mishap.
And if it’s a local backup, that local backup needs to be taken offline, to avoid the same issue if things get deleted in the cloud
I guess I don't extend much trust to the companies that provide these services, especially if they are free. But then I don't take many photos nor share them around much. That might change the equation a bit if I was more into that. What photos I do take that I want to keep I download to my desktop.
Maybe I'm just too damned cynical.
FWIW, I don't think so. There's such a thing as healthy paranoia.
Privacy considerations definitely factored into my decision to move my library to iCloud Photos from Google Photos a couple years ago.
I still maintain more than one backup of that library — one local, and one to a cloud backup provider.
Imagine if the nazis had access to the data that people store online now...
There wouldn't be any Schindler's list, he'd already have been in jail because the algorithm said so.
You can't opt out when everyone else has already opted you in.
Amazon Alexa for voice recognition and eavesdropping
Amazon subsidiary Ring.com to spy on your neighbor's driveway
AWS pulling it all together to send this panopticon to your local police department.
[0]: https://aws.amazon.com/blogs/storage/how-amazon-photos-uses-...
Still it sucks and it shouldn’t happen, hopefully it’s just a reporting error in the app.
- Synology NAS is accessible on LAN & phones sync photos when at home
- Nightly rsync from internal NAS storage to 1 external (non-raid) hard drive
The nightly rsync does not delete files, so the external hard drive is pretty close to an off-site backup even though it's still attached. The only better thing I've thought of doing is mounting it ro after backups + re-mounting before backups.
At the end of the day we get 3 copies of data: the photo on your phone, the photo on the shared NAS, and the photo on an external hard drive that spends 99% of its life spun down.
Imagine uploading all your photos to a cloud provider, then having 5% of your files disappear. How would you even know your photos are missing if its a small %? At least when your local filesystem is corrupted, you know what you're losing.
What if you never found out in your lifetime?
(seriously, I have dire doubts about the usefulness of our hoards ...)
Not cheap, but it’s a beautiful thing.
Aww, I got all excited because I read that as 3.5" FDDs, and got nostalgic for the noise... and installing Windows from like twelve of the bastards.
http https://www.amazonforum.com
http: LogLevel.ERROR: ConnectionError: HTTPSConnectionPool(host='www.amazonforum.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10b551e10>: Failed to establish a new connection: [Errno 61] Connection refused')) while doing a GET request to URL: https://www.amazonforum.com/
It's not, I'm in EU as well, can connect without any issues
I don’t like relying on monolithic monopolized corporate services as much as the next hacker— but I think there should be an easy to understand service for the 98% of basic users.
If someone could make a service that detects and requests permission to connects to all of your cloud services and provides full backup service that includes whatever the best practices are it would seem like a win for those who don’t want to roll their own.
A marketable “sticker” Ala “intel inside” ie - “open backup compatible” would be good to have with a simple but serviceable api.
I think today there is little financial incentive for storage providers to make the data easy to access via machine.
Ideally an api would not only allow incoming data to be quickly compared and synched… but would allow graceful and intelligent transfers or splits or duplication levels among stores, in the case that you want to change services or have your own backup.
None of this is that hard to do, not one just wants to put a minimal amount of effort into learning how to do backups, or automate them or deal with DR scenarios. If this information is important to you it would behoove you to learn how to protect it. Otherwise its not that important to you. Obviously IMHO.
I think it's likely worth it considering the value of basically the history of our lives.
Additionally, I've heard SSDs aren't great long term backup mediums, compared to mechanical hard drives. I personally use both.
Less than USD$50 for a 2TB disk. So buy 3, post one to a family member when its full. Double the price it for SSD?
How much Amazon Photos storage do you get for $150? Ok for this guy, none, and I get connection refused from the link above so I got no details. But assuming it worked as advertised how much would you get?
I agree they should also have their own physical backup if the photos are important, even at ~10x the storage requirements you estimate, but I'd also say any complaints about Amazon losing their photos are valid.
None of this data is stored in a FAANG company's DC.
I've taken to essentially backing up all my RAWs to Amazon Photos - even the ones I would typically discard.
This may be user error? It's possible the photos are still there, or if they are gone, perhaps Amazon can restore.
OP, did you try to call / contact Amazon before you posted this?
...which is listed as the first port of call for support.[0]
> For more help, try our Amazon Drive & Amazon Photos forum
But hey, maybe I missed the prominently displayed phone number somehow, so if you can help me find it, it'd be much appreciated.
[0]: https://www.amazon.com/gp/help/customer/display.html?nodeId=...
EDIT: I misremembered the details, actually the email says:
> Beginning December 31, 2023, files stored on Amazon Drive will no longer be available to customers. We will continue to support Amazon Photos and ensure customers can safely back up, share, and organize photos and videos with the Amazon Photos service.
That's an interesting choice of phrasing they're using.
1. First, all photos are on a fat external harddrive connected to a homelab server. When I come home with photos, I dump them onto this (TODO to find a way to automatically copy them into the proper spot off the camera upon the camera being plugged in, or a different spot if the gopro is plugged in)
2. Once a day, these are rsynced to a separate set of harddrives, which serve self hosted photo software I use (similar functionality to google photos. Looking for a replacement that lets me create arbitrary albums of photos, and then create web pages pointing to those arbitrary albums, that's FOSS. Currently evaluating photoview and librephotos)
3. Once a day, all photos are backed up to a backblaze b2 bucket using duplicati. with several previous versions also kept.
4. Once a year, a fucknormous external drive is purchased, all photos dumped onto it, and mailed to my parent's house, where they put it into a firebox.
5. Oh, and all my photos are also uploaded to google photos, because why not
I'm very much a scrub when it comes to sysadmin tasks and was able to set up all of the above with about... I don't know, no more than 100 hours of effort I'm sure, probably less. I recommend at least looking into consumer backup services like the one by backblaze!
(here I was hoping that the Docker thing filled the surprised Pikachu face quota for the week’s frontpage)
The app is just a front-end, deleting data (with Amazon photos and even S3) is not easy to do by accident without user input.
Is it just resynching the application (metrics updating) ? Super curious the issue.
Cloud storage is just as volatile as your physical storage breaking or getting lost.
Although, you should also think about whether you really need to keep more than a few dozen or at most a few hundred photos of your total life.
and I do not have backups of them all (which is on me)
55 photos a day, every day, for 12 years, does not sound "carefully curated" to me.
Advantages: unlimited full resolution backup
Disadvantages: Poor facial recognition Unable to manually add tags Poor folder /album management Android app has stopped uploading photos and support hasn'tbeen able to fix it Upload from PC only using their app (which isn't great) Upload from PC has "lost" the database and subsequently uploaded at least 2 versions of the just under 1TB / 50k images / videos from my family
The result is that actually finding or doing anything with the photos in the storage is not very easy, but at least everything is backed up there!
PiBox w/ Mirrored SSDs -> Server w/ RAIDZ2 (at another wing of the house) -> Windows NAS -> Backblaze Unlimited
Multiple SSDs and HDDs would have to go down at the same time, and there are hourly snapshots up to a month in case a roll back is needed.
If you're relying on a third party in any case except total catastrophe, you're just waiting for that moment to come! This is why Backblaze is at the end, I depend on them when I have failed to provide availability myself.