liballoc_system's unix __rust_reallocate doesn't handle allocation failures for over-aligned types

[froydnj]

The Unix implementation of __rust_reallocate in liballoc_system boils down to:

    pub unsafe fn reallocate(ptr: *mut u8, old_size: usize, size: usize, align: usize) -> *mut u8 {
        if align <= MIN_ALIGN {
            libc::realloc(ptr as *mut libc::c_void, size as libc::size_t) as *mut u8
        } else {
            let new_ptr = allocate(size, align);
            ptr::copy(ptr, new_ptr, cmp::min(size, old_size));
            deallocate(ptr, old_size, align);
            new_ptr
        }
    }

The else branch of the conditional doesn't properly handle the case where allocate fails. For comparison's sake, the Windows implementation says:

    pub unsafe fn reallocate(ptr: *mut u8, _old_size: usize, size: usize, align: usize) -> *mut u8 {
        if align <= MIN_ALIGN {
            HeapReAlloc(GetProcessHeap(), 0, ptr as LPVOID, size as SIZE_T) as *mut u8
        } else {
            let header = get_header(ptr);
            let new = HeapReAlloc(GetProcessHeap(),
                                  0,
                                  header.0 as LPVOID,
                                  (size + align) as SIZE_T) as *mut u8;
            if new.is_null() {
                return new;
            }
            align_ptr(new, align)
        }
    }

[alexcrichton]

Oh dear, seems bad!

Out of curiosity, was this discovered from reading or from a bug? We could consider a backport to beta if that's necessary

[froydnj]

This was discovered when reading through liballoc* to write a custom memory allocator for Firefox. I suspect it's hard to make it trigger due to overcommit policies and the general lack of usage of over-aligned types.