Layout size - Size of let's called this "area" where your game happens - think of it as a level size
Windows size - Size of your game window. How big, in pixels your game will be on preview and after export. For example 1280x720 pixels means that your game window in browser,mobile or desktop will have 1280 pixels wide and 720 pixels tall.
Margin - it's just a "free space" outside the layout in layout editor. make yourself a layout of size 400x400 and set margins to 0,0 - try pan around layout editor with Middle mouse button (or holding spacebar). Then set margins to 1000,1000, pan around, and see the difference.