The killer feature I haven't seen with many of these solutions is easy, out of the box integration with internal systems (Atlassian Confluence, JIRA, Remedy, SharePoint, FileSystem, Intranet). When you have a SaaS search engine it's difficult to export that data... Even worse to secure it. Ironically, Plumtree Software (bought by BEA -> Oracle) had all of this in their product in 2001. What's old is new again... Those features are prime for a comeback.
I think this is a space where Elastic can do well with an on-prem or managed cloud offering that is "behind the firewall", integrated with customer's environment. Add in term vector search support, ML for document/query understanding, and integration with customer's security model (Active Directory) and it would be compelling.
And most of the time, while not indexing, the hardware would be sitting there sleeping. Probably not very cost-effective for enterprises.
Not to be condescending, but idle hardware isn't even on the radar as far as waste goes in enterprises. An on-prem solution that is idle for 364 days of the year is completely fine for most of these companies.
For the ones that do care, that's what they make virtual machines and over-subscription for if they even care the slightest about that.
This isn't true.
I've build (neural-network) vector based search extensions for search. You don't train the model - you use a pretrained model (that understands English in your domain) and then use it as an encoder.
Sometimes there is once-off pretraining process for domain adaptation, but honestly this isn't a big deal. Even on a CPU based machine you could do this overnight or over a weekend, and since it is once off that time doesn't really matter.
We'd love to talk to you if you're interested in using Kendra. We're also wondering if there's more value on the Question Answering side of things, or the document retrieval side of things? Would love your thoughts!
In fact, while I do notice people doing question answering, users are also exceedingly lazy and want even more out of a search UI with fewer keywords. I just went to an e-commerce search UI and searched for backpack, and got something closer to search-y recommendations targeted around the kinds of backpacks I might want.
Building a similar enterprise search product at http://landria.io/ that has a lot of additional features & enhancements over a unified keyword index + ML.
We also have a terraform config if you would like to boot it up within your own private cloud!
Any feedback would be great appreciated
> Kendra’s preview will not include incremental learning, query auto-completion, custom synonyms, or analytics. The preview will only offer connectors for SharePoint online, JDBC, and Amazon S3. It will be limited to a maximum of 40k queries per day, 100k documents indexed, and one index per account.
Hopefully Amazon moves faster and offers more out of the box data sources. They are missing G Suite content that a lot of orgs are relying on these days. Would be interesting to see what's their strategy there.
kendra (IndE)
noun C
a centre for some activity (research, study, business, art, etc.)
The main issue is giving access to documents, which most Enterprise customers do not want to do... Further, most info is in employees heads, not in documentation.
Seems like this could integrate well with an enterprise wiki (attempt to document what is in the employees heads).
Google disrupted the market by factoring in links into its algorithm, something that is rather meaningless in proprietary context.
Agreed relevancy is a problem. PageRank works well for public content but internal search has plenty of relevancy problems. Having little control over this certainly hurt.
Managing on-prem hardware/appliances is a difficult business. I don't fault them for moving to a more scaleable model.
The more niche/higher level services like kendra are priced based on the value to a medium to large company.
They don’t expect individual developers to use this, or build anything on top of it. They expect a partner or employee of the company to do a pilot on the developer pricing, then convert to the enterprise pricing.
It’s a somewhat annoying trend but imo Google Cloud is a much worse offender here, everything new from them seems to be on prem “call sales for pricing” aimed at the enterprise.
* 3x m5.12xlarge (192GB RAM) = $6.90 / hr
* Kendra Enterprise 150GB "documents" = $7 / hr
But for the AWS offering, you get less than one mean query per second (capped per day). I would think ElasticSearch on the same hardware would offer a couple orders of magnitude more throughput.
The AWS pages talk about "document scanning," so perhaps this product is poised more towards replacing an office full of humans and filing cabinets, which most definitely costs more than $7/hr. This is the gateway product to wanting ElasticSearch.
If you are interested in a search solution like Biome, please feel free to reach out so we can talk more and learn the best way we can empower your team to be more productive.
What's good:
==========
- Focused search for question and answer databases (such as customer FAQs)
- ML-based semantic search without requiring any explicit configuration
- Connectors for S3, AWS-hosted MySQL/PG, Sharepoint. Searching data already in the AWS ecosystem (S3, Aurora) is now easier, and likely faster and cheaper too in some aspects like saving incoming/outgoing bandwidth
- Document-level access control at all pricing plans
- Managed search (similar to Algolia)
What's similar to existing search systems (Solr / ES / Algolia):
==========
- Indexing: All data has to be processed into "field:value" structure prior to indexing
- Indexing file formats: Plain text, HTML, PDF, MS DOCX, MS PPT
- Searching: Usual boolean filters and faceting but only at field level.
- Searching: Field and value boosts for relevance, but only at index-time
- Results: Highlighting support
What's missing:
===========
- No multi-lingual support. Only English. Given that it's AWS, I'm very surprised by this actually (or I've missed out something in their docs)
- Can't configure text analysis for English. I feel this'll return relevant results for formal-style content, but probably not for informal-style content like emails.
- No connectors for common internal systems: Outlook, JIRA, Confluence
- No built-in support for CSV, XLS, JSON (that one's odd!). They'll all require preprocessing which means additional infra costs.
- Doesn't seem to support range- / query- facets. I feel lack of range facets is a big problem, especially for numerical data.
- No query-time relevance tuning
- No field-level access control
- Scores are not returned in results
- Common post-searching functionality is missing: rescoring, grouping, clustering
What's unknown:
============
- I don't see any information about phrase or proximity searches. Of course, they are usually relevance hacks in keyword-based systems, but sometimes users really need exact phrase matches. Does their ML backend handle this somehow?
- All search systems fall short while handling proper nouns - names, places, things, scientific names. It's possible to alleviate it to some extent using part-of-speech aware indexing. Not sure if Kendra does it in its ML backend.
