Evolving CRUD (part 2)

Serving content at scale without complexity

Command Querying Responsibility Segregation #

Martin Fowler and Udi Dahan have in the past described this approach at length. This is obviously not a new concept and what follows is only a possible implementation of it.

Command Querying Responsibility Segregation CQRS

A fancy name that hides a simple concept: read traffic (GET) has different requirements compared to the write traffic (POST/PUT/DELETE) and should be handled possibly by different systems.

This is by far not a new concept. Already in production with big players but never standardised.

This design splits read and write requests according to the following philosophy:

Read requests (GET) #

Write requests (POST/UPDATE/DELETE) #

The concept of message queues is not new and is at the very heart of the Enterprise Integration Patterns concepts, minus all the bloatware.


In a CQRS system eventual consistency for upsert operations is a fact.

This may not always be desirable: not everything can be done later, something needs to be done upfront in order to return necessary information to the user (ids, errors if any, etc)

Command Querying Responsibility Segregation CQRS Plus

Writing to the serving layer synchronously is a small variation that can work well with Redis given its high read and write throughput.

Advantages of this approach #

No more Cron jobs #

Cron jobs and scheduled tasks are no longer necessary as background tasks are run on demand when needed rather than at a time based schedule.

Automatic Error Handling #

Messages are continuously reprocessed until no error is encountered.
This introduces resiliency in they way data is syndicated to 3rd party platforms, which can be down or not always available.

Task idempotency is crucial to this end

Push approach to caching #

Instead of calculating the views on demand we now precompute them at ingestion time. This allows us to scale out writes by beefing up our queue processing cluster.

External system integration #

Syndication and data ingestion are handled transparently in the same way

Scale out more easily #

By dividing internet facing read and write requests from queue task we can now scale them independently.
In the most common scenarios we can simply resort to single threaded task processing depending on the ingestion traffic and this can greatly simplify our code.

Also as tasks are processed in groups smart optimization techniques can be easily applied: for example when multiple updates to a resource are sent in a series, only the last one can be applied and the others discarded.

Disavantages of this approach #

Data loss is possible #

Limited by the Redis memory #

Non trivial data modelling in the serving layer #

Eventual consistency #

Integral error #

Bottom line #

This approach is suitable when


Now read this

Nginx by examples: naxsi WAF

Naxsi is an open source WAF module developed by NBS System and released under GPL v3 In the past a nginx-naxsi standard Ubuntu package was available from the official repositories. Unfortunately this package is no longer maintained so we... Continue →