Evolving CRUD (part 2)

Serving content at scale without complexity

 Command Querying Responsibility Segregation

Martin Fowler and Udi Dahan have in the past described this approach at length. This is obviously not a new concept and what follows is only a possible implementation of it.

Command Querying Responsibility Segregation CQRS

A fancy name that hides a simple concept: read traffic (GET) has different requirements compared to the write traffic (POST/PUT/DELETE) and should be handled possibly by different systems.

This is by far not a new concept. Already in production with big players but never standardised.

This design splits read and write requests according to the following philosophy:

 Read requests (GET)

 Write requests (POST/UPDATE/DELETE)

The concept of message queues is not new and is at the very heart of the Enterprise Integration Patterns concepts, minus all the bloatware.

 CQRS+

In a CQRS system eventual consistency for upsert operations is a fact.

This may not always be desirable: not everything can be done later, something needs to be done upfront in order to return necessary information to the user (ids, errors if any, etc)

Command Querying Responsibility Segregation CQRS Plus

Writing to the serving layer synchronously is a small variation that can work well with Redis given its high read and write throughput.

 Advantages of this approach

 No more Cron jobs

Cron jobs and scheduled tasks are no longer necessary as background tasks are run on demand when needed rather than at a time based schedule.

 Automatic Error Handling

Messages are continuously reprocessed until no error is encountered.
This introduces resiliency in they way data is syndicated to 3rd party platforms, which can be down or not always available.

Task idempotency is crucial to this end

 Push approach to caching

Instead of calculating the views on demand we now precompute them at ingestion time. This allows us to scale out writes by beefing up our queue processing cluster.

 External system integration

Syndication and data ingestion are handled transparently in the same way

 Scale out more easily

By dividing internet facing read and write requests from queue task we can now scale them independently.
In the most common scenarios we can simply resort to single threaded task processing depending on the ingestion traffic and this can greatly simplify our code.

Also as tasks are processed in groups smart optimization techniques can be easily applied: for example when multiple updates to a resource are sent in a series, only the last one can be applied and the others discarded.

 Disavantages of this approach

 Data loss is possible

 Limited by the Redis memory

 Non trivial data modelling in the serving layer

 Eventual consistency

 Integral error

 Bottom line

This approach is suitable when

 
3
Kudos
 
3
Kudos

Now read this

Nginx by examples: naxsi WAF

Naxsi is an open source WAF module developed by NBS System and released under GPL v3 In the past a nginx-naxsi standard Ubuntu package was available from the official repositories. Unfortunately this package is no longer maintained so we... Continue →