I think I know what the difference is: one works and the other doesn't, at least for playback on my setup (see Playback, below). Apart from that, there appears to be no detailed description of what these two options do, either in the Premiere PDF or anywhere online. Unless you count this as description (p471 of PDF):
For single-layer or dual-layer Blu-ray Disc, select either MPEG2 Blu-ray or H.264 Blu-ray.
Really informative stuff (please forgive my sarcasm). So, other than peering at the settings that come up, I can't work out why there are two H.264 options.
PLAYBACK
I have a 60-second test sequence, consisting of an animated still with audio and titles, 1920 x 1080, 23.976 fps. I exported two versions, at maximum quality settings:
1. H.264: VBR, 2 pass, Target 32.00 Mbps, Max 40.00 Mbps, AAC, 192 kbps, 48kHz, Stereo.
2. H.264 Blu-ray: VBR, 2 pass, Target 25.00 Mbps, Max 30.00 Mbps, PCM, 48kHz, Stereo.
(1) came out as a 20.7 MB, mp4 file, 2.8 Mbps. Played in Quicktime, VLC and Premiere.
(2) came out as a 20.6 MB, m4v file, ? Mbps. Would not play (video or audio) in QT or VLC. In Premiere, the file lacked audio.
QUES 1
Why are there two H.264 options? In what situations do you use one and not the other?
QUES 2
The bit rates of the exported files are very low, less than 10% of the target. Is this because low bit-rates are all that was needed for maximum quality for this simple file?
QUES 3
Why doesn't the H.264 Blu-ray file playback properly? VLC can normally play almost anything you give it. Even Premiere couldn't play a file it had just exported.
I've uploaded both files to a 40 MB zip file on MediaFire.