acidity

Hello HN,

We use PubSubLite but the same can be applied to any queue system. The events are then processed by service running in K8S.

I am looking for strategies/patterns on how do folks partition their data so that each worker unit can get consistent performance with optimal CPU/memory usage.

Our data is global and temporal i.e. different regions will have peak/off peak data volume based on their time zones.

We have done performance test and we know how many events / second / partition we can process per worker units.

The issue we are currently facing is that when we get auto assigned partitions and based on randomness, we can get multiple heavy traffic partitions thus causing resource contention at a worker and causing slowdown.

Some options we have considered:

* running one worker unit per partition - this results in wastage as we have to use constant resources even in off peak time. this also results in using coordinater to make sure everybody is subscribing to correct partitions.

* group tenants into their traffic size and setup workers based on that. This has side effect of again extra maintenance on our side and bit of coordination with tenant provisioning/deprovisioning.

* order traffic to certain partitions based on tenant_id but then some partitions become too big and others dont.

What other strategies you use?

I have experience doing traditional web apps using PostgreSQL and some bit with SQL Server - back in 6.5 days.

Lately, I have been hearing about OLAP/OLTP/OLAP cubes etc at various meetings.

Doing a search does not seem to throw up a canonical reference on all these.

What would be a good way to go about learning these things with my background - which would be more traditional RDBMS based webapps/business apps.

My father (based out of India) is a retired professional who wants to start an educational video site (e.g. KhanAcademy) but in specific subject (geared towards Accountants). He is what one would consider a non technical guy.

Initially, the website/content would be free but maybe he wants to monetize it in future.

For a start, I setup an account at wordpress.com and even that interface is kind of intimidating. Sites like Netlify requires a Git based workflow which again is not something he is adapt at.

What other options do I have? I could but I dont really want to reinvent the wheel to build another custom CMS site for him.

This is similar to topics like: https://news.ycombinator.com/item?id=20765553

I am a full stack and general software developer (have developed various Desktop apps using Win32 APIs).

After move to web development (backend, frontend, infra etc) and using various distributed apps (Kafka, Redis etc), I am a bit curious how does one start developing such apps.

I know you have to apply some sort consensus protocols (e.g. RAFT) but I could not find any practical tutorial in it.

How would one go about learning this? I know I can always try and read various tools code but I was more hoping if there is some tutorial/book where the author/developer has implemented a tiny/toy project to show how they work.

Thanks.

Recently, I started playing with GCE and have been pretty impressed by the Cloud Shell implementation. They even make VI bindings work.

We could use something like this at work too so I was wondering, what would be a good resource to start if I have to go on implementing something like this? Our front end app is in Angular and APIs are served by Python/Java services but we are language/library agnostic.

Topics that I can study?