The game logic itself took less than an 20 minutes.
Adding little elements takes me a a hell of a lot longer..
The replay system works most of the time. It seems that every once in a while the timing is off. I think the fix is to not rely on the replay collisions and instead record them and manually trigger them.
(NOTE) Turns out it was just a timing issue in my code.