Crash Heap in NTDLL when server running

i have problem about it and log it:

Code:

---

(1970.161c): C++ EH exception - code e06d7363 (first chance)
(1970.161c): C++ EH exception - code e06d7363 (first chance)
(1970.161c): C++ EH exception - code e06d7363 (first chance)
(1970.161c): C++ EH exception - code e06d7363 (first chance)
(1970.161c): C++ EH exception - code e06d7363 (first chance)
(1970.161c): C++ EH exception - code e06d7363 (first chance)
Heap corruption detected at 5F5EB3F8
(1970.1974): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=5f5eb3f8 ebx=00003982 ecx=00000000 edx=5f5eb3f8 esi=5f5eb3f0 edi=096f0000
eip=7c82b5a3 esp=0013d5d0 ebp=0013d5e8 iopl=0        nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000            efl=00010246
ntdll!wcscpy+0x108:
7c82b5a3 8901            mov    dword ptr [ecx],eax  ds:0023:00000000=????????
Missing image name, possible paged-out or corrupt data.
Missing image name, possible paged-out or corrupt data.
Missing image name, possible paged-out or corrupt data.

---

And i see when it appear error . It call RTLALLOCATEHEAP

Code:

---

ntdll!RtlRestoreLastWin32Error+235
7c82a38b 8b39            mov    edi,dword ptr [ecx]

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c82a38b (ntdll!RtlRestoreLastWin32Error+0x00000235)
  ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
  Parameter[0]: 00000000
  Parameter[1]: 00000000
Attempt to read from address 00000000

---

Anyone have error the same to me. and anyone can fix it done ^^. Thank

@JayMoko i hear you say. You can fix it.

Clearly you are hitting exceptions your server has registered to the system that it will handle, but is not doing so.

Check the code sections which deal with issuing SEH. (Structured Exception Handler) I think you will find that one (or more) of them has had a pointer to an error handling routine corrupted. (moved or replaced the routine? Search & Replace too much? Unbalanced PUSHes an POPs? Something of that nature.)

When server runing for a time. It appear error. I think everyone have problem about it.^^ And you think SEH????...

I've never seen this problem. So I guess not "everyone" has it. :wink:

It could be connected to the issues 35hit is struggling with though. Similar level of problem, but quite different end result.

Structured Exception Handlers are chained... like Interrupt Handlers used to be in DOS, or you can do with API Overrides in Windows.

There are default ones built into Windows, and the NT Kernel... those produce "Fatal Exception", or "Illegal memory access", "Illegal instruction / Privileged instruction attempted" (Ring 0 Opcode executed from Ring 3 code) type messages, and force the application to terminate and unload.

C++ automatically adds Structured Exception Handlers for common run-time errors. All of the common Runtime Error messages from your C++ library are produced this way. "Divide by zero", "Invalid pointer assignment" and so on.

But any time you program a try... catch... finally... type sequence in C++ it's also (usually) generated by utilising the OS Structured Exception Handling System.

In Win32, this is managed by the KE (Executive Kernel) which is NTDLL.dll, and Kernel32.dll. This is the underlying Micro-Kernel below the Win32 OS (as it also supports, or has supported OS/2 and Services for Unix, and supports the MinWin, now WindowsPE and Recovery Console mini OS), so we don't usually hit errors in it when programming against the Win32 environment... but that only applies because we write in high level languages which are targeting either Services for Unix or the Win32 platform.

The only time we should get errors that deep is if there is a bug in the C Runtime Libraries (MSVCRT* and the CRT .lib files) or if we are targeting the DDK, which operates much closer to the NT Micro-Kernel than Win32 programs.

When we mess around with the assembler, it's very easy to "break" standard C++ run-times which are built into the executable and should ensure this never happens. But if you know your C++, imagine that you compile a try... catch... finally block of code, and then go in with OllyDbg, and NOP out the code that equates to the "finally" part. Try to imagine, what would happen?

Most C++ programmers can't imagine, because the compiler wouldn't allow the executable to be built until they fixed their code. XD

This is the sort of error you are hitting.

There are glimpses of this process explained quite expertly by Arun Kishan in this C9 Interview Video. (there's a lot more too, but since he owned the Process Management part of the Vista KE team at the time his words are "from the horses mouth")

Ntdll.dll Problems, Process Information - Ntdlldll.com (Register cleaner) try this or gregoo guide http://forum.ragezone.com/f562/guide...-crash-660353/

gl

I use http://forum.ragezone.com/f286/crash...-crash-660353/ for 2 year ago. But it still error :D

Yes, I bet the only thing jaymoko cant do is answer his PMS, right jay?

@zaharavn

Which operation system you run the server on?

Quote:

---

Originally Posted by Rovarav

@zaharavn

Which operation system you run the server on?

---

Windows Server 2003 Enterprise ^^.

32bit or 64bit

Quote:

---

Originally Posted by Rovarav

32bit or 64bit

---

Windows Server 2003 Enterprise 32bit

zaharavn u got disconect from server , but server still online but cant login in game ?

No. It crash server and exit server application

I don't want to say "everyone's wrong" (especially as I don't actually have the right answer) but, JayMokos answers relate to DEP (which has very little to do with ntdll.dll, except that it is also implemented in the KE, and is one of the many "Exceptions" that it implements) and "Missing ntdll.dll" errors, which are caused when the system registry no longer points the boot loader to the correct %systemroot% directory... that would stop most of your OS booting up, so you'd get between 5 and 20 of those messages just as a matter of trying to log in... if NTdll.dll is giving "ntdll!RtlRestoreLastWin32Error" errors, then the SEH heap is corrupted in a specific process, and NTdll.dll has been found, and injected into the process at start time without difficulty.

@ 35hit: I said the problems seem to be at a similar level, but the end result is quite different. Your problem seems to be either memory or thread allocation and de-allocation mismatch, Zaharas' seems to be Exception Handler registration and / or release which is corrupted. That might suggest that you have both tried to implement a similar patch to your server, and slight differences in your base, or methods of working have produced similar levels of error, but very different actual working problems.

So where Zahara has a 'try' that's missing a 'finally', you have something like a malloc() without dealloc() or 'new []' without 'delete []' or CreateThread() without TerminateThread().

Actually, since your problem seems to happen, even if you change your entire executable, and dump the optional DLLs, it could even be a bug that has always been there, but was cleaned up (automagically) by previous system libraries in a way that a recent Windows System Update has "broken". (They are surprisingly resilient to "bad code", and attempts to avoid DDoS, Data Execution or Buffer Overrun attacks do reduce this resilience.) And the same could be true for Zahara... But I don't know if he's tried switching server EXEs.

Either way, both these issues come up when you corrupt the standard runtime library routines which are built into the executable, and not part of the server source code.

By which, I mean that it would be hard (and in Zaharas' case, impossible... aside from the Spoiler below) to write code in C / C++ that would cause these errors. The compiler would never build and link such code into a .exe file. It would throw up compiler / linker errors before it got that far.

Spoiler:

Actually, when I say "impossible" I know people like SoyEdu will tick me off, because if you write "self-modifying code" in C, or C++, or you write "in-line assembler"... you can bypass all the normal compiler checks. So you can fool the compiler into allowing such nastiness.

I just mean that following normal coding practices, you wouldn't see it. You'd have to be trying to code a self-packing demo, an Exe compressor or encrypter, or a code injector to get results like this in high level programming.

.NET managed code is different again, because the JIT compiler will give a high level debugger style response if something like this happens at run-time.

But my experiences of MSIL executables (Processor independent .NET managed code) is that they do this all the time, even when you write your code perfectly, and never modify the executable... simply because some dependant package on the users system is not 100% identical to the development version you use. >.<

I gather I have unusually bad experiences from .Net Managed code (MSIL) and an thus prejudiced. After all, thousands of XBox 360 users have no problems with their games. All of which are compiled to MSIL. XD

@Bobsobol

From all what I read, i understood nothing. Can you give solutions instead of write too much information that others don't understand and/or don't know about. All he wanted is to fix this problem not to read a book of many information he doesn't need to solve it. Just suggesting.

Anyway if the server.exe exits, you can add a auto restarter to auto restart it everytime it crashes.

No. I can't give solutions to problems I don't know the answer too... I can only suggest where to look for solutions.

If *you* don't understand, that's not my problem. If Zahara and 35hit don't understand me, I'm sure they will ask me to clarify a section they don't know how to follow.

Using auto restart is a *very* small bandage on a *big* wound, and if (like 35hit) the server doesn't quit, it just stops letting players log on until someone kills it and starts it up again, auto restart is useless... if you leave a crashing server crashing and keep developing it anyway your server will get much worse *very fast*.

So from my point of view, "Use an autostarter" is an admin answer, it's not a developer answer. You can't develop a server that is already crashing regularly.

You got better solution than auto restarter? if yes just tell it clearly.
I know that auto restarter will work only on server.exe that closes (Zahara menioted that his server.exe closes, and thats why i offered him this solution untill we find a fix to the crashes).

I can give you a example, FroggPT. they have crashes each day (which they use auto restarter to put server back up) almost and they are up since 2006.

Yes... I do.

Clearly? Fix the problem in the server.exe code. o.O Simple enough?

All the information you give is simply not important to fix the problem. You can ask the people here if all this information you write (and you are the only know what you are talking about) is helpfull or not.

Quote:

---

Originally Posted by Vovozozo

@Bobsobol


Anyway if the server.exe exits, you can add a auto restarter to auto restart it everytime it crashes.

---

If you use it. Server will rollback item, exp.... I want to fix it. not use auto restart ^^. I think Sandurr know fix it. :))

It's okay... for some (other, personal) reason I fell for that bait. XD

No disrespect to Frogg, I know their server and team are good, and I'm sure they do use some Auto Restart, but if their server crashes on a daily basis, I would suggest it is a sign of their popularity, and it is shutting down in an orderly manner in order to prevent those who would otherwise take advantage of it.

I know Zahara has been running E-XPT for at least 2 or 3 years, and I'm reasonably sure he will have something in place to re-initialise a failed service. But what he is describing is not an orderly shutdown. Rather, it is quite a serious crash. If the poor kernel is suffering regularly with this kind of abuse, it is likely to leave the system unstable, and an automatic Server Reboot would be more in order than simply starting the Server executable up again.

Having said that, 35Hit could probably implement an automated server shutdown and then restart every 12 hours or so, as a temporary fix until he figures out where his virtual memory leak (or whatever it turns out to be) is. Then at least players would know 'when' they are going to DC. XD (@35hit, call it "regular maintenance hour" to your players. :wink: Maybe 12 O'Clock every 12 O'Clock. 12:00 & 24:00 hrs server time.)

Back to the main thread issue. I do not see this issue on most servers, so has your server, Zahara always done this, or is it a recent event?

I don't think it's inherent to all PT Servers, and I certainly don't think it's present in the original jPT server... but swapping out server executables for a while would prove that.

Having said that, I know that the original jPT server has clearly been modified to take sql.dll, Clan.dll & URSLogin.dll... these aren't loaded by normal C++ means, so it could be as old as that.

If it's not, finding out which group of server executables do it, and which don't... how wide spread the issue is. Would be a great help. I had assumed it was unique to the E-XPT current server, but if you think Sandurr would have any knowledge of that, you must think the problem is more wide spread than just something you've added your self. (I guess?)

it's normal crash for other PT. So we have Auto Restart ^^. If PT Server running as offical server, won't have auto restart. I think Turn On DEP, It only terminate msg box error ^^

thats what im doing restart server every 12 hour

HeapAlloc, HeapCreate, HeapFree ... ??? That's problem ? Or overflow ??

Well... I always run with UAC and DEP fully enabled on my servers, if they fail either then they are not running securely and need fixing IMHO.

It's not difficult, and very wise to comply with DEP... specifically because it can run you into problems like this. Remember what it stands for "Data Execution Protection". If you clearly define to the OS what is data, and what is code (which normally linked EXEs do anyway with the .text and .data sections) then the OS will ensure that you don't try to Execute instructions in your data memory, and don't try to write data over your code memory. A pretty sound policy to have enforced, don't you think?

The problem is early teams like Global Fantasy adding a single section to the executable which the declare as Readable, Writeable, Executable, Pre-initialised and uninitialised data. :s It's just not safe to have such an insecure memory section in an executable. You can put the code and static, unchanging data in one section and only dynamic and state based data in another, or your can separate them all out into code, static (read only) data and changing data... but don't put them all in one memory block. XD You can also add no new sections, but instead rewrite the import directory for the executable, add everything that is in your GFantasy section into a .DLL (in appropriate .text, .data .rdata sections, "GFantasy.dll" maybe) and access functions via the normal DLL methods.

I'm fairly sure Sandurr wouldn't have a GFantasy section, as I know he was involved in their developments, and can therefore re-create all of that code on his own, and in a much more system friendly manner.

Heap operations are performed by the Server Executable, and by the OS it's self when a new thread is created or destroyed.

The solution I would propose is that you find an exe which doesn't have this problem, and find out which change between it and your server executable causes it... then you know the code which is written badly.

Short of that, you are in an awful sticky mess of trying to match up pairs of such statements, or others that might lead to them.

I've said this would almost never happen in high level programming, so debugging it is going to be very tricky... but: it may be possible to spot such anomalies by globally patching and logging any and all such API calls. There are some tools to monitor Windows APIs generally, but IDK if they would give you enough information.

Blade API Monitor - Trace and log API and ActiveX interface with parameters
API Monitor: Spy on API Calls and COM Interfaces (Freeware 32-bit and 64-bit Versions!) :: rohitab.com
API Monitor - Spy and display Win32 API calls made by applications
Dev Stuff

The last is probably the most comprehensive free solution, and comes with an SDK that would allow you (if you are skilled enough) to write a custom solution to this debugging problem. (It kinda "did my head in" though, so be warned.)

What you would need to know, is each Allocation API called, from where, why, and on what handle, or address.

Specifically:-

• you need to match up allocations of a specific block with their de-allocation. That's why you need the handle or memory block address.
• you need to know where (in running memory) each of these calls is made, so you can find the code in Olly.
• You need to know where the last call was made to it, because of wrappers and because many of the allocations are made by higher level APIs, which will require more in-depth debugging

It's a nasty issue, if it's not due to a code change you made. o.O