Here is another idea: Treat the intervals as sets and compute their intersections.
Start with the first interval and compute the intersection with every other interval. Store the intersections inside a second vector. The second vector should now have n-1 elements (intersection of 1st with 2nd, 1st with 3rd, 1st with 4th, ..., 1st with nth, but not 1st with 1st). Now you take the intervals in this second vector and intersect them with all original intervals again (first interval in the second vector is for 1st with 2nd, so we only need to start intersecting starting from 3rd, for the second interval we start from the 4th interval, and so on). The intersections are then stored inside a third vector. If there are only empty intervals inside the third vector, we can stop for now. Otherwise we throw away the second vector and repeat this process with the third vector. (The second vector is intersections containing 2 intervals, the third vector contains intersections of 3 intervals, so we don't need the second vector anymore if there are non-empty intervals in the third vector.) From the last vector we get from this process we keep one intersected interval (just take the first non-empty intersection) and remember it. It is the intersection with the most intervals including the first interval. Now, we repeat this for the second interval, and so on.
This should give as for each interval the maximum of intersections containing this interval and the interval limits of the intersection. You can just pick the intersected interval with the highest number of intersections and use the minimum of that interval as the number.
It is a little tricky to wrap your head around this (initially I thought it would be little easier). The way I described it runtime would be at least O(n³). I'm not sure if this is good or bad.