The answer to how many quarters fit in a mason jar really depends on the shape of the jar and how you want to organize your coins. If you want to maximize the number of quarters in the jar, you should stack them diagnolly (side by side). This will allow you to fill up the whole jar without leaving any empty space between each coin.
The other way to calculate the maximum amount of quarters in a jar is to use the volume of the jar itself. This method is more accurate if you are working with the same sized jar, but it can still be tricky to get exactly right. The best way to do this is to measure the jar's diameter and height in inches, then square that number and divide by pi (which is estimated at 3.14, or 22/7 as a fraction).
You can also estimate how many quarters you have in your jar by weighing the jar itself. This can be a bit difficult because you'll have to separate the coins before weighing them, but there are scales out there that will let you do this and even automatically count your pennies for you! Modern US pennies are 2.5 grams each, so a jar full of them will weigh about 1779 grams. Subtract the jar's tare weight and you'll have your estimate in pounds of how many quarters are in your jar.
Lifehacker offers another handy tip that involves weighing the jar's contents and comparing it to others. This method isn't as precise as the other two, but it can help you quickly gauge how much your jar of coins is worth without having to count every penny by hand.