rtmp: Clean up properly if the handshake failed