I believe for Adjustments, quantity represents the number that changed in this specific adjustment. i.e if it was from IN_STOCK to SOLD and quantity=5, then that means this particular adjustment had 5 of these items that were sold (from in stock).
For Physical Counts, I believe quantity represents the literal physical count for the particular count.