Im currently working on a dungeon generator, but im far from an expert on the subject.
But I started out trying to make one with path finding.
Demo of the dungeon generator - Alpha: Demo of the dungeon generator - using path finding
1. Path finding
The idea behind it was to add X amount of rooms at random locations with random sizes. Each room then get a minimum of 1 door and as far as I remember the larger the room the more doors. These doors then work as connection points, which are added to a list. When all the doors have been added, it take a random door and path find to another random door on the list. And it just keep doing that until there are no more doors. The path that the path finder find between the doors are then turned into halls, and when its done it create walls around all floor tiles. The problem with this one, was that I found it a bit slow, even though I could have optimized it. But it also was quite complex to make and it does contain some problems where certain rooms might end up not being reachable. So I dropped it due to these things and then because it didn't really fit what I wanted for my game when I thought about it.
2. Node system
So I took a different approach which is a node system, that is easier to make. The idea is that each tile only have so many possibilities when it comes to how you can actually place them.
To start the dungeon I start a random place, and then expand the dungeon from there.
And it will keep expanding that way.
Then I added a functionality that will either force the dungeon to stop expanding or force it to keep growing, which is done so im able to create different sizes.
Then when its done making the whole dungeon, there are actually a lot of tiles that are messed up, because you cant be sure how it expand, so I complete it by running a clean up that correct all tiles. And when this is done its pretty simply to pick random tiles and replace them with room tiles since I only need rooms that are one tile I don't need to worry about where the rooms are connected as they will always be connected correctly, compared to the one with path finding, where im not sure where the doors will actually be placed.
Demo of the dungeon generator version 2.1 - Alpha: Demo of the dungeon generator version 2.1 - node system
But for both types or any time I think, its pretty much about setting up restrictions and rules for how things should work. But a good idea I think, at least it helped me a lot, is to take a piece of paper and then try to draw how you imagine it should work or make the tiles and just add them manual on a tilemap while trying to figure out what rules should apply and how you want it to work. But personally I think its one of the most difficult things to do, because there are so many rules and when bugs happen it can be quite hard to figure out what wrong.