Pre Vita.
or, post-post mortem
I’d like to preface this article with the simple fact that we won’t know if our plan will work until it’s actually done. Proofs of concept are one thing, but rescuing 2 million dollars is no easy task. We are extremely optimistic, but until everything is in place and we execute the rescue logic, it’s all theoretical.
September 12th. The Reaper team investigates a problem with their Scream contracts and discovers a pair of bugs that locks $2,000,000 without any way for the strategy to retrieve them. After several hours of hacking, the situation is deemed hopeless. A post-mortem is written, the team is devastated, and we begin to brainstorm a path forward.
September 13th. Our community throws their support behind us. Tears of sorrow turn to tears of joy, and we attack the problem with renewed vigor. While brainstorming ways to make all of our users whole again, Reaper decides to put up a $100,000 bug bounty for help in rescuing the funds.
About 5 hours after the bug bounty is announced, discord users Zarigis and Jeffery enter our general channel with some intriguing ideas.
Before I get started on this, let me circle back to the 2 bugs that made this situation so complicated:
First: the logic necessary to maintain enough reserves for withdrawal was accidentally deleted from the contract, meaning users with significant sums wouldn’t be able to pull their funds out without manual intervention from the Reaper team.
Second: our pause failsafe was incomplete, meaning we could remove ERC20 allowances from the strategy but not give them back, making repayment of our loan impossible.
The issue with reserves was an easy fix, but in pausing our contracts so we could migrate to new vaults, we’d removed any chance for the strategy to repay its debt and pull the funds back in.
Happy Hacking
Zarigis’ first big idea was that we could bypass the need for allowances by making sure the strategy had a 0 balance. This would give us the ability to deleverage incrementally by paying off the loan using repayBorrowBehalf from a different account. This way, we’d be able to use deleverageOnce in lieu of the troublesome _deleverage.
I fired up my console, the team piled into the office with Zarigis and Jeffery, and we got to work.
After hours of effort, I fired off the final transaction, which would prove whether his idea was possible. We all sat waiting with bated breath as the EVM went to work figuring out if execution was possible. I stared at the blinking cursor on my console…
I leapt out of my chair and cheered, ready to tell the community what had happened. We all shouted praise for Zarigis and gave him the honor of withdrawing the first batch of rescued funds…
failure.
We try again and again and again, only to be met with more failure. We look at the strategy and realize all the funds we’d pulled back in were added to reserves, without a means to make them withdrawable.
Round 2
We had proven a lot of ideas correct while testing out Zarigis’ plan and weren’t about to give up. Looking through code, we realized we’d probably be able to call _deleverage once the loan was fully paid off, but it’s internal and every public function that calls it also calls something that would revert due to our revoked allowances. We think and think and think until Goober excitedly shouts “we could retire the strat!”
The only function in our entire system that would deleverage without attempting to lever up again is our upgrade system, which retires the strategy and switches to a new implementation after a 5-day timelock. It seemed promising — Zarigis was convinced it would work and at this point we were inclined to believe whatever he said.
We iron out our plan and re-create the bugged system in production with smaller amounts of money: depositing into the broken strategy, calling our broken pause function, repaying the strategy’s debt from a separate account using repayBorrowBehalf, initiating the upgrade, retiring the strategy, and recovering all the funds into a brand new implementation.
It works.
Goober stays up all night refining our test into a contractual proof of concept while I pass out like a wimp.
The Recovery Plan
An astute reader may have realized by now that repaying 2 million dollars of over-leveraged debt will be expensive. Using the deleveraging trick described in the “Happy Hacking” section we can lower the amount needed, but it’ll be difficult nonetheless (flash loans solve this btw).
In addition to this, we are essentially performing an upgrade that would be malicious in any other context, so we want to ensure that user funds are safe throughout every step of the process. We are reaching out to community leaders to join us on a multisig and working on raising funds for the rescue operation. Every step of the recovery effort will be performed with smart contracts to ensure total safety and transparency.
I’d like to give my humble thanks to Zarigis, Jeffery, Goober, and the rest of the Reaper team and community. Your support yesterday gave us the strength we needed to push through and find a solution. I will remember these past couple days for the rest of my life — it was a crazy rollercoaster of emotion but right now it seems like we might come out alright.
tl;dr
We believe we’ve found a way to recover the lost funds and will be spending the coming days organizing the money and multisig signers necessary to do so. It will also take 5 days for the timelock to execute our upgrade so expect compensation in the next couple of weeks.
