psankar

I want to start a hobby project which needs to have as less development time as possible. I would want social media login (sign up via email, fb, google accounts, etc.), user management, connecting to a posgres (or even sqlite) db, deployments (perhaps to a paas like cloudrun, heroku, aws beanstalk etc.), db migrations etc. all already automated. If there is some automation for the API body and parameter validation from a openapi spec that would be even better.

I just want to write the core business logic for the API handlers and leave everything else to the framework. I do not mind (and actually would prefer) if the framework is opinionated and works well for its intended goals.

I prefer Typescript/JS, Golang, Kotlin in that order. If no other option, then python3 would be an acceptable choice.

Any recommendations ?

tldr: I want a batteries-included API framework with users management, session management, db management, openapi integration. Help ?

There are plenty of debates and talks about monolithic vs microservices touting the benefits of each. Most such talks seem to indicate that monolithic apps scale badly. By that they probably mean horizontal scaling && developers/teams scaling, but we have good options for vertical scaling these days via monstrous VMs with high RAM/CPU from almost all cloud providers.

What is the biggest (scale) application(s) that you have personally seen or worked with in a single VM or baremetal server ?

The scale could be in terms of maximum requests per second, volume of data transferred, maximum parallel connections, etc.

The most popular example that I could think of is Expensify where they handle 4million qps in a single monster machine with 1TB RAM via Bedrockdb https://blog.expensify.com/2018/01/08/scaling-sqlite-to-4m-qps-on-a-single-server/

I am looking for similar examples of big products on a single machine or VM. Not on a super-computer, mainframe, specialised appliances (like SAP HANA), etc. but normal X86 boxes or cloud VMs. I am curios to know how much of HN crowd choose these over complex things like Kubernetes and their thoughts on these.

I have been working from home for the last six months or so, since the COVID related shutdowns began. I have been having terrible hand, shoulder and upper-back pain since then. It has increased to an extent that my productivity is getting affected.

The right thumb (used for the mouse), the right wrist, the shoulders are the one that pain the most. Sometime I feel like I cannot even use the spacebar with my thumb because of the pain and use my index finger.

I am a touch typist and sit straight in an ergonomic chair. I use a thinkpad. My monitor elevated and is at eyelevel. I do not have any problems in my lower back. It is the upper back and the right hand that gives problem.

This could possibly be because, I have a small frame and can only rest my left elbow in the hand-rest but not my right elbow as the chair is wide. I do not have narrower sized ergonomic chairs where I live (Some small town in India). We get standard one-size-fits-all chairs. I sit either cross legged or at the 90 degree angle with the help of a foot rest. I sit in front of the computer for about 10 hours a day. I take frequent walks and breaks too (thanks pomodoro).

Do any of you have any recommendations for eliminating this pain ? I am considering buying some kind of vertical / ergonomic mouse and keyboard. I am eyeing the kinesis split keyboard so that I can rest both my hands in the handrest. I am thinking if I should get a trackball instead of the mouse, as some people are recommending that. Youtube has contradictory recommendations (ex: should [not] rest wrists while typing). What do you HN readers do for improving ergonomics or steps you did to reduce/eliminate hand/wrist/shoulder pain ? Thanks.

We have a telemetry gathering system. We get about 250rps to our backend systems, which dump this data on a database. We get ~25million records per day, occupying a disk space of ~12.5GB per day (inclusive of elastic index). We use elastic as the backend.

We chose elastic because we did not what fields we may get. There are ~40 fields (per record). Some fields may be missing in some records. Coming to the reads, the data thus saved would be queried in a timeseries way. We will access data for last 1 hour, 1 week, 1 month mostly.

The data will be queried with a lot of complex aggregations and joins too. For ex: What is the avg response time (which is a field) of all the requests that talked from node (another field) appServer1 to node appServer2, last month.

With elastic we are quite happy with the data until there are ~10 million records, beyond which response times just lag. It takes 10+ seconds for each query after 10 million records. We thought it could be due to reader starvation (because of incoming writes) and created a read-only replica and did our queries in that replica. But even there the slowness is felt.

In terms of memory, We have given about 1GB for esmaster, 2GB for esdata and another 1GB for esclient. All persistent disks are normal HDDs and not SSDs. We do not have much worries about diskspace and can add more diskspace (but not SSDs). But we will not be able to provision more RAM.

Also, we want our system to add columns/fields without any stop-the-world migrations. This is what makes me nervous about choosing databases that want a schema ahead.

The writes can take (update multiple indexes on each write) but the reads have to be fast. We are also fine with choosing a hybrid approach where one system receives the writes and another system can do the analytics. The system has to work in kubernetes across clouds. So single-cloud solutions (like dynamodb) are ruledout.

Given the load and the above constraints, what kind of a database solution would you choose ?

If you use kubernetes (micro-services) for your application deployment, how do you handle database migrations ? Do you have any downtimes ? The choice of blue-green deployments, rollbacks etc. could cause an issue and I am trying to accumulate what the best practices are, tools/processes that are useful and what lessons you have learnt.

We are using kubernetes. We have about 15 or so pods/services, about 3-4 statefulsets. Things work fine and we are happy with kubernetes so far.

However, for our dev setups and also for CI, we are looking for a better solution. These 15 services have a few hierarchical dependencies. For example, the postgres stateful set should be brought up first, followed by the users service and then the auth service.

The users/auth service will work fine (albeit returning 5xx errors) even without the other services being up. However, if all the services start together, they will all compete for the tiny amounts of CPU that we have given and eventually some starve and kind of gets into weird unrecoverable situations (in CI and test clusters where we can't give max cpu/mem).

I know of ways to increase the cpu/mem limit but keep the initial allocations, small. But I believe that since I know about service dependencies better, if I bring up kubernetes pods/services in a hierarchical way, it would be far more useful for me.

Do any of you that use kubernetes, use any tools/hacks/techniques to bring up pods/services in particular order (without lots of kubectl waits) ? In the past, with `docker-compose` I was able to achieve this via service dependencies but can't find similar in kubernetes. Any pointers ?