Good work. Running 60 fps in the 40x40 cells demo, needing 1.5 seconds to re-calculate.
I don't know your algorithm, but make it flexible enough to only calculate rows and columns, instead of the whole scene. Then, while moving the camera only calculate the next one or two (depends on the speed of the camera movement) rows/columns.
If the camera goes to the right, you only need to calculate one or two columns to the right, if the movement hits the next column (grid-based, in this case column 41 would be needed, etc.). Same for all other directions. The heaviest impact would be a diagonal movement, where both, row and column needs to be calculated.
EDIT: Forget about it. Was writing quicker than thinking. It doesn't solve the problem of light cells that may influence the rows/columns. Sorry.