Pre Vita.

or, post-post mortem

I’d like to preface this article with the simple fact that we won’t know if our plan will work until it’s actually done. Proofs of concept are one thing, but rescuing 2 million dollars is no easy task. We are extremely optimistic, but until everything is in place and we execute the rescue logic, it’s all theoretical.

September 12th. The Reaper team investigates a problem with their Scream contracts and discovers a pair of bugs that locks $2,000,000 without any way for the strategy to retrieve them. After several hours of hacking, the situation is deemed hopeless. A post-mortem is written, the team is devastated, and we begin to brainstorm a path forward.

September 13th. Our community throws their support behind us. Tears of sorrow turn to tears of joy, and we attack the problem with renewed vigor. While brainstorming ways to make all of our users whole again, Reaper decides to put up a $100,000 bug bounty for help in rescuing the funds.

About 5 hours after the bug bounty is announced, discord users Zarigis and Jeffery enter our general channel with some intriguing ideas.

Zarigis, the legend

Before I get started on this, let me circle back to the 2 bugs that made this situation so complicated:

First: the logic necessary to maintain enough reserves for withdrawal was accidentally deleted from the contract, meaning users with significant sums wouldn’t be able to pull their funds out without manual intervention from the Reaper team.

Second: our pause failsafe was incomplete, meaning we could remove ERC20 allowances from the strategy but not give them back, making repayment of our loan impossible.

The issue with reserves was an easy fix, but in pausing our contracts so we could migrate to new vaults, we’d removed any chance for the strategy to repay its debt and pull the funds back in.

Happy Hacking

I fired up my console, the team piled into the office with Zarigis and Jeffery, and we got to work.

After hours of effort, I fired off the final transaction, which would prove whether his idea was possible. We all sat waiting with bated breath as the EVM went to work figuring out if execution was possible. I stared at the blinking cursor on my console…

Success!

I leapt out of my chair and cheered, ready to tell the community what had happened. We all shouted praise for Zarigis and gave him the honor of withdrawing the first batch of rescued funds…

failure.

We try again and again and again, only to be met with more failure. We look at the strategy and realize all the funds we’d pulled back in were added to reserves, without a means to make them withdrawable.

Round 2

The only function in our entire system that would deleverage without attempting to lever up again is our upgrade system, which retires the strategy and switches to a new implementation after a 5-day timelock. It seemed promising — Zarigis was convinced it would work and at this point we were inclined to believe whatever he said.

We iron out our plan and re-create the bugged system in production with smaller amounts of money: depositing into the broken strategy, calling our broken pause function, repaying the strategy’s debt from a separate account using repayBorrowBehalf, initiating the upgrade, retiring the strategy, and recovering all the funds into a brand new implementation.

It works.

Goober stays up all night refining our test into a contractual proof of concept while I pass out like a wimp.

The Recovery Plan

In addition to this, we are essentially performing an upgrade that would be malicious in any other context, so we want to ensure that user funds are safe throughout every step of the process. We are reaching out to community leaders to join us on a multisig and working on raising funds for the rescue operation. Every step of the recovery effort will be performed with smart contracts to ensure total safety and transparency.

I’d like to give my humble thanks to Zarigis, Jeffery, Goober, and the rest of the Reaper team and community. Your support yesterday gave us the strength we needed to push through and find a solution. I will remember these past couple days for the rest of my life — it was a crazy rollercoaster of emotion but right now it seems like we might come out alright.

tl;dr

We believe we’ve found a way to recover the lost funds and will be spending the coming days organizing the money and multisig signers necessary to do so. It will also take 5 days for the timelock to execute our upgrade so expect compensation in the next couple of weeks.