About tracerealloc()

[Damon Caskey]

pls stop discussing about that bullshit, or i have to invite the trolls from the ##C channel or reddit to teach you

And if you do, I'll put you and them in the same place. This kind of talk doesn't fly on Lavalit.

DC

[SX]

Back to the subject at hand....


If its a fragmentation issue.  It might be worth putting in the time to try and get newlib compiled for windows.

A user from the ps2dev.org forum posted the following:

hrimfaxi:  NewLib-1.17

Hi, folks. Recently I found out if we update newlib-psp malloc version to the latest malloc-2.8.4, the memory fragment after a mass of allocating-and-freeing memory operations will significantly reduce.

I can verify that his statement holds true.... here is my own quote from that forum:

The Good News:
I went from 6246484 Bytes of fragmented memory to 4759612 Bytes. That's about 23.8% Savings!

The Bad News:
I still have quite a bit of fragmented memory...

Here is the link to the conversation: Click Me!


Bottom line here is that OpenBOR does cause memory to become fragmented.  You will definitely see this issue on embedded hardware such as the PSP, but If your running a Windows machine that is light on Ram....  The same issues are bound to happen.

Linux users by default have the latest and greatest version of newlib, and is probably why fragmentation does not hurt them as much as Windows with the same hardware specs.

[Damon Caskey]

"significantly reduce"...

That doesn't sound like a great solution. There is a big difference between reduce and eliminate. If we still have fragmentation, even "significantly reduced", it's going to come back and bite us.

DC

[SX]

"significantly reduce"...

That doesn't sound like a great solution. There is a big difference between reduce and eliminate. If we still have fragmentation, even "significantly reduced", it's going to come back and bite us.

DC

There is no great solution.

On one hand we use arrays, and have the space pre-allocated for the strings we use.  But then it just blindly increases the amount of total memory OpenBOR uses.  (Old days of OpenBOR)

On the other hand we allocate strings as needed.  Less memory is used, but fragmentation now occurs. (OpenBOR Today)

One user on that forum made an interesting statement.  I should use memory pools for specific areas.  Which is starting to make more sense.  For strings where we allocate 5, 10, 32 bytes or so, we should use a small pool dedicated for that very behavior.  

This will guarantee the heap does not get fragmented.   Same can apply to other areas, like sprites, etc... each get their own pool.


Lastly,
It might be worth spending the time to review all instances where tracemalloc is called.  I'm betting in some areas we can rely on the stack instead of the heap.  This will lower fragmentation as well, utilizing the stack whenever possible.

[Damon Caskey]

Mind meeting in chat a sec?

DC

[SX]

Sounds good.

[Plombo]

Okay, my current theory is that Windows + realloc() == fail.  Do you know if Windows has any issues with realloc() usage?

Um.................right.

come on, if you take this vague feeling you had at the time serious, there's something wrong

Great, you decided to dig out some old posts I made in a different thread.

The first one is an incomplete theory I had three months ago.  I haven't actually believed that for 3 months.  It was a tentative guess until further evidence turned up, which it did.

The second post had absolutely nothing to do with realloc().  It was talking about something completely different (my personal modding project), and I have no idea why you quoted it.

and YES, even Microsoft uses realloc, because it simplifies memory management a lot, and leads to compacter and cleaner code

No, they use HeapRealloc, which is basically the same thing.  That's what I said before.


But whatever, I'm tired of talking about this crap, and I could be wrong.  If you're really that bothered by not being able to reallocate buffers in theoretically possible recursive functions without keeping track of their lengths ( ) then let's just leave the stupid function like it is.

In fact, it's not like the following (or something with similar logic) is rocket science, or even close:

Code: [Select]

void* recursive_something(void* buf, size_t curlen)
{
    static size_t buflen;
    void* buf2;
    buflen = curlen;
    // [...] (change buflen at some point to new desired buffer size)
    buf2 = tracerealloc(buf, buflen);
    if(buf2 == NULL) { tracefree(buf); return NULL; }
    // [...]
    if(whatever_condition)
    buf2 = recursive(buf2, buflen);
    // [...]
    return buf2;
}

But if this theoretical function really is that painful, then like I said, have it your way.  I'm going to leave the function alone and not argue about it any more unless prompted.

If its a fragmentation issue.  It might be worth putting in the time to try and get newlib compiled for windows.

Using a non-default C library would not solve the problem...

Linux users by default have the latest and greatest version of newlib, and is probably why fragmentation does not hurt them as much as Windows with the same hardware specs.

Linux users don't have newlib at all.  We use glibc.  Newlib is primarily for embedded systems.

Also, on platforms that use, the important parts of memory management are handled by the kernel, not by the userspace C library.

"significantly reduce"...

That doesn't sound like a great solution. There is a big difference between reduce and eliminate. If we still have fragmentation, even "significantly reduced", it's going to come back and bite us.

It's already been biting us for years.  It's actually gotten better since the beginning of 2010, at the very least.  I don't see how it's going to "come back and bite us" any more than it already has unless we introduce even more fragmentation.

[utunnels]

Bump*

It seems I missed something big. 
I want to know, how do we check memory leak in this new layout?

[Plombo]

Bump*

It seems I missed something big. 
I want to know, how do we check memory leak in this new layout?

By default, nothing.  I think there's a tracemalloc replacement that you can enable at build time in debug mode.  I still wish tracemalloc hadn't been removed.