You are correct... those tiny screws can be easy to drop and are nearly impossible to find thereafter. I have a Z4 G4 Xeon version (its motherboard has two M.2 sockets at its bottom front corner whereas the Core motherboard has only one). The female-threaded hexagonal post that allows you to screw into for a standard M.2 2280 NVMe stick is attached to the metal case beyond the motherboard, and can be repositioned to allow use of a shorter 2260 M.2 stick (rare). That post has a little round protrusion at its top to help center the rounded groove built into the M.2's PCB. The screw to use is a wide head metric M2 short machine screw. Here are tips from experience over the years:
1. It is hard to get easy access to that post if a front case fan kit is installed. For my first install of a M.2 drive I took that kit out and put it back in later. The M.2 stick wants to spring up to an angle. Best to temporarily tape it down so it is flat against the post before you insert the hold down screw. You want to magnetize the screwdriver tip so the screw can be lowered vertically... rub the tip one way only over and over with a magnet to do that. Hence, you don't want to use a stainless steel or brass screw... use a ferrous one which is usually what you'd get from eBay or Amazon. You want a long narrow screwdriver... I've attached a pic including the Lutz 4-in-1 Pocket Mini Screwdriver (larger PH0 tip) I currently used to get the spare screw out of my secondary M.2 post in my Z4 G4. It is the silver screw on the right in the picture. I remove the caps from both ends of the screwdriver to get near vertical access with the front case fan kit remaining in place. The black screw on the right is also an equivalent metric M2 short screw harvested from a Dell laptop... its shaft looks a bit bigger diameter as an optical illusion... it fits perfectly too. For these tiny screws don't crank down on them... just tighten to gentle snug. You can gall the threads if you tighten with too much force.
2. McMaster.com is a great source for all kinds of unique fasteners, but they don't carry what you need. Go to Amazon or eBay and search for M.2 screws which almost always are metric M2 (2mm) size. The heads on the McMaster metric M2 screws top out at 4mm. Same for their metric M2 washers. The HP silver and the black M2 screws shown both have heads that measure 5mm diameter... 5 or even 6mm is the right size head. The original silver one and the black one has threaded shafts 3mm long, but the hexagonal post can accept a metric M2 male threaded shaft of up to 5mm.
3. HP made their special brass/black plastic M.2 hold downs for the Z Turbo Drive G1 and G2 and those use a special shaped internal metric M1.6 screw. The Z Turbo Drive Dual Pro G1 and G2 use the same basic hold down but with brass/blue plastic hold down, and those use a special shaped internal metric M2 screw, shown. Those won't work in your Z4 G4. Some McMaster M1.6 screws are shown on the left.
4. The Z4 G4 Xeon motherboards have a primary (lower) and a secondary (upper) M.2 socket. Ideally, you should attach your boot/apps M.2 stick into the primary socket. Of interest in the Z4 G5 the primary socket is upper and the secondary is lower. The primary generally had direct PCIe lanes to the processor vs the secondary to the PCH. There is a Z4 G4 HP White Paper that shows the difference.
5. I've attached in the picture a nice HP heatsink that I've used in our Z4 G4s and our Z Turbo Drive G1 cards that originally came without a heatsink... search eBay for 919952-001, about $10.00 USD via eBay. Those work fine for two being inserted side-by-side in the Xeon Z4 G4, and in the ZTD G1. Note they have a wider footprint, with those spring steel side arms. More recent HP tower and SFF workstations may not have enough room for them... HP thus came out with a narrower version but it has less heatsink surface area and are hard to find, and are almost $100.00 via HP PartSurfer (!). There are also many non-HP options but I like these for HP quality, and they maintain single PCIe slot width.
Here's the picture:
