As far as your SQL client is concerned, data.splitgraph.com:5432 is a giant Postgres database with ~40,000 tables in it. You can query and join across them using your existing tools. Behind the curtain, we'll forward your query to the upstream data source, translating it from SQL to whatever language it expects. (We can also ingest delta-compressed versioned snapshots).
> We can understand structured data in Web pages about datasets, using either schema.org Dataset markup, or equivalent structures represented in W3C's Data Catalog Vocabulary (DCAT) format. We also are exploring experimental support for structured data based on W3C CSVW, and expect to evolve and adapt our approach as best practices for dataset description emerge. For more information about our approach to dataset discovery, see Making it easier to discover datasets.
For more info on those:
- W3C's Data Catalog Vocabulary: https://www.w3.org/TR/vocab-dcat-3/
- Schema.org dataset: https://schema.org/Dataset
- CSVW Namespace Vocabulary Terms: https://www.w3.org/ns/csvw
- Generating RDF from Tabular Data on the Web (examples on how to use CSVW): https://www.w3.org/TR/csv2rdf/
I keep getting errors from Google that some of my dataset’s descriptions are over 5,000 characters even though dcat:description does not have a size limit.
Of course it’s impossible for me to report a bug in how they index.
1. #StructuredPremises:
> (How do I indicate that this is a https://schema.org/ScholarlyArticle predicated upon premises including this Dataset and these logical propositions?)
2. #LinkedMetaAnalyses; #LinkedResearch "#StudyGraph"
3. [CSVW (Tabular Data Model),] schema.org/Dataset(s) with per column (per-feature) physical quantity and unit URIs with e.g. QUDT and/or https://schema.org/StructuredValue metadata for maximum data reusability.
4. JupyterLab notebooks:
4a. JupyterLab Metadata Service extension: https://github.com/jupyterlab/jupyterlab-metadata-service :
> - displays linked data about the resources you are interacting with in JuyterLab.
> - enables other extensions to register as linked data providers to expose JSON LD about an entity given the entity's URL.
> - exposes linked data to the user as a Linked Data viewer in the Data Browser pane.
4b. JupyterLab Data Explorer: https://github.com/jupyterlab/jupyterlab-data-explorer :
> - Data changing on you? Use RxJS observables to represent data over time.
> - Have a new way to look at your data? Create React or lumino components to view a certain type.
> - Built-in data explorer UI to find and use available datasets.
This is really really cool. Between this and Hugginfaces Dataset and models hubs, AI/ML is really getting easier to use.
I've found https://modelzoo.co/ but it seems more like a currated list of models (some incomplete) rather than a community where users share trained models.
https://www.arcgis.com/home/item.html?id=9850793c688e4eebaab...
Can anybody explain why this showed up in a dataset search and what exactly the data is?
One cool aspect of this metadata is that it allows a dataset to have multiple sources. So if two sites index the same dataset, there is no duplicate content penalty like there might be with textual content. If you search for a dataset, it will include links to all its sources (whether canonical or otherwise).
For most of the data we index at Splitgraph, the canonical source is an open government data portal powered by Socrata (e.g. data.cdc.gov). We noticed that Socrata powered a lot of portals, so we wrote a Socrata plugin for Splitgraph, along with a scraper to index the metadata. The plugin basically implements a Postgres FDW so that Splitgraph can translate from SQL to the upstream query language. In this case, the plugin translates to Socrata's bespoke API language. But for private deployments we also have plugins for Snowflake, Postgres, some SaaS services, etc.
If you find some data on Google Dataset Search with Splitgraph listed as a source, please take a look! Our "Data Delivery Network" (DDN) is implemented on top of the Postgres wire protocol, so you can connect with any Postgres client (or use our web editor). All the Postgres query syntax is available to you; you can even JOIN across any of the other 40k+ datasets indexed at Splitgraph. That includes "live data" like Socrata portals, but also versioned snapshots of data called "data images." Here's an example of a point-in-time query across two snapshots (basically a diff) [1], and another query that joins across tables at data.cityofchicago.org and data.cambridgema.gov [2].
[0] https://www.splitgraph.com/cdc-gov/distribution-of-covid19-d... – "View Source" to see the Schema.org metadata
(Sorry for the bit.ly links. The URL for our query editor includes the full SQL string, and I don't want to mess up HN formatting.)
I certainly tried my current projects on it, and found some useful stuff (most of which I've seen before).
[0] - https://datasetsearch.research.google.com/search?query=penis... [1] - https://data.world/jemus42/world-penis-data
Doesn't take a genius to predict, but there ya go! Governments are assembling datasets in a very fragmented way. It'll take a private company to provide one single website to explore and find all datasets from around the world, making it easier to look at holistic patterns that are happening around the world, or compare patterns between countries.
Though, I would expect a much better UX from Google nowadays. This site has more in common with Google Scholar than Google Search.
And ultimately I'd like to see them build something where people don't need to download datasets in order to make use of the data.
I compare the state of open data to the state of mapping software before Google Maps. You needed to download map files and open them on special software that you open on your computer to make sense of the data. And then Google Maps came along and flipped that whole model. Open data needs the same leap forward in order for more people to make greater use of open data.
Discussion from Sept 2018: https://news.ycombinator.com/item?id=17919297 | 76 comments
A common barrier to making FAIR datasets is that not all data lends itself to be schema.org compliant. The idea is that instead of enforcing one schema to rule them all, we allow people to make their own schemas by extending existing ones, and register them in an API to be easily discoverable.
