Cool, thanks for the link! I don't mind the tags being public, but it seems Hypothes.is now also supports private/group annotations.
Unfortunately you can only annotate text (so far), but they might add deeper integration in the future to e.g. tag a section of a video.
In the meantime I started writing a Chrome extension based on Firebase. Having never used Firebase before I'm very pleasantly surprised how easy it is to get something up and running!