undefined | Better HN

0 pointsWalterBright5y ago0 comments

> indexes them in a local SQLite DB.

I don't really understand this. I've been in computing for 45 years now. Programs come and go. I store files I want to keep in the most generic format practical. Storing a file in some program's special format is not a good plan for reading it 40 years from now.

For photos, I store them as jpgs in folders named after the year. Within those folders, there might be sub-folders named with a topic, like "disneyland" or "christmas". I'll "tag" photos by selecting a name for the jpg, like "bob and sue.jpg".

If more is needed, I'll just add a "notes.txt" file in the folder, with whatever text seems appropriate.

I have no worries about ascii text becoming unreadable, and few worries about jpgs becoming unreadable.

0 comments

mholt5y ago

Please don't misunderstand, with your 45 years of computing experience. :) (which I highly respect) I never said the files are stored in sqlite. They are just indexed in sqlite. The files are stored on disk like normal.

peterwwillis5y ago

> Please don't misunderstand

If there's one thing people will always do regardless of age, it's that.

phoe-krk5y ago

SQLite is an ubiquitous and well-tested piece of software that is going to stand the test of time and survive the next twenty years. There are very few pieces of software that I am able to praise like that with a clean conscience; SQLite is definitely one of them though. You might find it worthwhile to read e.g. about its memory allocation strategy, or its testing strategy.

Also, I don't think that the image files themselves are stored in the SQLite DB file; likely just their indices and metadata.

WalterBrightOP5y ago

> SQLite is an ubiquitous and well-tested piece of software that is going to stand the test of time and survive the next twenty years.

My timeline is much longer than 20 years. I have family photos going back to the Civil War.

Besides, Wordstar and Wordperfect files used to be ubiquitous. Good luck with those today.

I switched my mail from Outlook to Thunderbird because the former stores the email in some undocumented unreadable binary format. TB stores the mail as text, so I can recover the mail without needing TB. I have mail going back 25 years now. Some of my earlier mail is now lost because, surprise surprise, the mail program no longer works and the data is stored in a proprietary format.

I unzipped all my old file archives a few years back out of concern that some of the old DOS archive software would disappear.

williadc5y ago

> Besides, Wordstar and Wordperfect files used to be ubiquitous. Good luck with those today.

Here's a Hacker News story from 10 days ago about an update to Wordperfect: https://news.ycombinator.com/item?id=24959090

I haven't tried it myself.

1 more reply

fomine35y ago

I agree to store metadata inside the file is better, but FYI: SQLite is expected to be supported very long.

https://www.sqlite.org/lts.html

knyazhefilms5y ago

>My timeline is much longer than 20 years.

What storage format do you use then? Are you concerned about Hard drives format becoming obsolete? Are you using Mac or Windows? What if tomorow Apple or Microsoft decide to stop support for their formats? What would you consider as a reliable Storage format? Do you trust linux formats ext3 ext4? What would be your ideal choice looking from your perspective which I really share. It's really a big question for me how reliably store things for long periods of time.

1 more reply

stjohnswarts5y ago

I hope you at least have at least 1 on site and one off site and 1 physical media backup of all those if you want to keep them for the next few generations...

1 more reply

depr5y ago

Wordperfect files can be opened in MSFT Word ;)

1 more reply

WalterBrightOP5y ago

> likely just their indices and metadata

notes.txt will work just fine, then, even for several thousand photos. I don't see a need for a database until you've got far more than that. The tree file system works tolerably well as a "database".

As for searching, I know how to use "grep" and "locate".

joshuamorton5y ago

Tags are more flexible than folders. An actual index let's you search by tag. Folders don't let you search for all photos of your mother, or in 1987, or in Alabama, or some combination thereof.

Folders are a limiting api and we shouldn't limit ourselves to strictly hierarchical organizational structures for non hierarchal data.

1 more reply

megablast5y ago

45 years experience, and your first thought is that they are being stored as a binary blob in a database??

WalterBrightOP5y ago

Yes. Binary file formats are common (like Microsoft Outlook, pdf files, etc.), and are very fast to read/write. Text files have to be parsed. I've written lots and lots of dumpers for binary file formats generated by other programs.

mholt5y ago

Oof, I feel the trauma on this one. I've been bitten by too many binary file formats that I don't care to store aggregate amounts of content in them (unless it's just metadata).

ForHackernews5y ago

TBF SQLite format is also extremely unlikely to become unreadable: https://www.sqlite.org/fileformat2.html

Retric5y ago

Metadata is more useful with older photos. Right now it’s obvious who’s in them, but in 40 years being able to just put names to faces can be difficult.

WalterBrightOP5y ago

I just note their names in the file name. If there are too many, I make a copy of the photo, bring it up in an image editor, and type the names of the people next to their heads. It works like a champ.

zeroxfe5y ago

Bits rot. If you want your data to last the next 40 years, plan for spending a few hours every 5 years migrating your storage. It's a far better plan than trying to predict what the world will be like 40 years from now.

WalterBrightOP5y ago

Absolutely. I buy new drives every year or so and copy the stuff forward. Many of my 10+ year old drives no longer function.

e_y_5y ago

It would be easy enough to use the SQLite DB for normal read/write and search, and also save a human readable .txt dump periodically such as when making backups, or asynchronously on every save.

kristopolous5y ago

Creating a cadence that also stores the data in EXIF isn't really that hard if that's your concern. You could even get SQLite triggers to do external calls to update the tags if things change. This is all pretty trivial.

j / k navigate · click thread line to collapse

0 comments

mholt5y ago

peterwwillis5y ago

> Please don't misunderstand

If there's one thing people will always do regardless of age, it's that.

phoe-krk5y ago

Also, I don't think that the image files themselves are stored in the SQLite DB file; likely just their indices and metadata.

WalterBrightOP5y ago

> SQLite is an ubiquitous and well-tested piece of software that is going to stand the test of time and survive the next twenty years.

My timeline is much longer than 20 years. I have family photos going back to the Civil War.

Besides, Wordstar and Wordperfect files used to be ubiquitous. Good luck with those today.

I unzipped all my old file archives a few years back out of concern that some of the old DOS archive software would disappear.

williadc5y ago

> Besides, Wordstar and Wordperfect files used to be ubiquitous. Good luck with those today.

Here's a Hacker News story from 10 days ago about an update to Wordperfect: https://news.ycombinator.com/item?id=24959090

I haven't tried it myself.

1 more reply

fomine35y ago

I agree to store metadata inside the file is better, but FYI: SQLite is expected to be supported very long.

https://www.sqlite.org/lts.html

knyazhefilms5y ago

>My timeline is much longer than 20 years.

1 more reply

stjohnswarts5y ago

I hope you at least have at least 1 on site and one off site and 1 physical media backup of all those if you want to keep them for the next few generations...

1 more reply

depr5y ago

Wordperfect files can be opened in MSFT Word ;)

1 more reply

WalterBrightOP5y ago

> likely just their indices and metadata

As for searching, I know how to use "grep" and "locate".

joshuamorton5y ago

Tags are more flexible than folders. An actual index let's you search by tag. Folders don't let you search for all photos of your mother, or in 1987, or in Alabama, or some combination thereof.

Folders are a limiting api and we shouldn't limit ourselves to strictly hierarchical organizational structures for non hierarchal data.

1 more reply

megablast5y ago

45 years experience, and your first thought is that they are being stored as a binary blob in a database??

WalterBrightOP5y ago

mholt5y ago

Oof, I feel the trauma on this one. I've been bitten by too many binary file formats that I don't care to store aggregate amounts of content in them (unless it's just metadata).

ForHackernews5y ago

TBF SQLite format is also extremely unlikely to become unreadable: https://www.sqlite.org/fileformat2.html

Retric5y ago

Metadata is more useful with older photos. Right now it’s obvious who’s in them, but in 40 years being able to just put names to faces can be difficult.

WalterBrightOP5y ago

zeroxfe5y ago

WalterBrightOP5y ago

Absolutely. I buy new drives every year or so and copy the stuff forward. Many of my 10+ year old drives no longer function.

e_y_5y ago

It would be easy enough to use the SQLite DB for normal read/write and search, and also save a human readable .txt dump periodically such as when making backups, or asynchronously on every save.

kristopolous5y ago

j / k navigate · click thread line to collapse