Another way that should give you basically the same results as DavidT's answer, but perhaps with simpler code:
Step 1: Divide your square formation into four rows. Each row starts at a corner position and extends from there in a consistent direction (let's say clockwise, it doesn't really matter much) around the square.
Adapting DavidT's ASCII art notation, here are the four lines of 3 units each in formations of N = 4, 8 and 12 units. I've labeled the units in each line as F(front), L(left), R(right) and B(ack):
N = 4 N = 8 N = 12
F R F-F R F-F-F R
| |
L B L R L R
| | |
L B-B L R
|
L B-B-B
Step 2: Arrange all the units in each line so that the spaces between them and between the last unit and the next corner are equal.
(This effectively hides the asymmetric choice of whether to extend the lines clockwise or counterclockwise from the corners. Either way, the corner spots will always be filled as long as N ≥ 4 and the rest of the units on each side of the square will be evenly spaced between the corners.)
Step 3: Decide how many units to put in each line.
If the number of units is divisible by 4, you can (and probably should) just put the same number of units in each line. If not, you will have to add one extra unit to some lines.
The pictures in your question effectively match a filling order of F(ront) -> R(ight) -> B(ack) -> L(eft), and if that's what you like, it should work perfectly fine.
DavidT, however, suggests what's effectively an alternative filling order where you start by putting the first extra unit in the F(ront) row and the second in the B(ack) row, but if there's a third extra unit, you remove the second unit from the back row and move it and the third unit into the L(eft) and R(ight) rows. This has the arguable advantage of always keeping the same number of units on the left and right sides of the square and thus keeping the formation always symmetrical. (Well, at least for N ≠ 3.)
Of course there are also other options, like F(ront) -> R(ight) -> L(eft) -> B(ack), which doesn't quite always maintain symmetry but avoids moving units between rows as new units are added to the formation.
Or you could even let the player explicitly choose how many units they want on each side of the formation. If most threats are coming from the front, maybe they'd rather have only one or two units in the back and a bunch of extra units in the front, for example. Of course whether it makes sense to offer the player such flexibility (and the associated UI complexity) depends on the type of game you're making.
Step 4: Decide how big the square should be.
It probably makes sense to make the size of the square proportional to the number of units in the longest line. That way, the minimum distance between adjacent units in the square will be constant.
The distance between the units and the PC in the middle of the square will also always be at least 1/sqrt(2) ≈ 0.7 times the minimum distance between two units. If this feels too crowded, you could make a special case for N ≤ 4 and multiply the size of the square by some scaling factor between sqrt(2) ≈ 1.4 and 2 in that case. To me, a factor of 1.5 suggests itself as a pretty simple and natural approximation of sqrt(2), but you can decide for yourself what looks and feels best to you.