profile

Changelog

Amal Invest - May 11 incident (what happened and what's fixed)


Salaam,

On Monday May 11 at 13:30 UTC, a backend worker on Amal Invest executed sell orders for 40 users without the 24-hour cancellation notice the platform requires. 263 sells went through. Some were of stocks that were and remain Shariah-compliant β€” they should not have been sold. Others were valid liquidations of non-compliant stocks, but the affected users were owed a 24-hour preview email and didn't get it. Either way, the sells happened without authorization or notice.

We stopped the worker within the hour. No further unauthorized trades since. The platform has been in maintenance while we reconciled and built the fix.

If you were one of the 40, you'll get an individual email from me today with what was sold, where the cash is, and what are the next steps going forward. If you don't get that email today, your account wasn't affected and you don't need to do anything.

Why it happened:
​
Over the past few weeks we've been rolling out a substantial rewrite of how the platform tracks trading data, separating what a user intended, what we sent to the broker, what filled, and what each portfolio holds, as clear and distinct records. Before, those overlapped and made things hard to debug when something went wrong mid-process.

The incident happened during that rollout. A database migration created a new positions table but didn't populate it in the same step. For about 5 minutes, scheduled liquidation jobs ran against an empty version of it, saw "nothing to liquidate," and silently skipped the 24-hour preview emails, while the execution side read a different (correctly populated) data source and queued the sales for Monday.

When the worker restarted, those sales fired. A second issue compounded it: stale jobs from earlier dates, scheduled when certain stocks had been temporarily flagged as non-compliant, were still in the queue. Those stocks had since become compliant again, but the worker didn't re-check before executing.
​
In short: our preview check and execution check were reading two different sources of truth, and the queue had no idea it was carrying jobs that should never have run.
​
We're sorry this happened. Trading actions taken against your account without your consent or notice is the failure mode we work hardest to prevent, and on Monday we didn't prevent it. The fix shipped today is the floor, not the ceiling, of what we owe you on this.

What we're shipping today:
​
The rewrite is now in production in full. The worker can no longer execute a sale unless a durable record proves the preview was sent, the cancellation window elapsed, the user didn't cancel, and the underlying compliance state is still current. If any of those is missing, the worker skips instead of trading. Stale or replayed jobs are now harmless. The platform comes out of maintenance shortly after US market open today, we're bringing it back during live hours so we can watch it directly, instead of letting the open hit a freshly restarted system.

Nothing changes in how you use Amal. The work was entirely backend.

Reply to this email with any questions.

best,
Omar

Changelog

Earn more, while investing in line with your faith. We share updates twice a month, with a list of all the things we've released. Join below πŸ‘‡

Share this page