[branch] merge trunk head

svn path=/branches/jcatena-branch/; revision=45592

_jc_changelog.txt

@@ -0,0 +1,301 @@
+***** 45059 branch creation
+
+***** 45567
+
+[branch regression] revert branch to 44678
+- the msvc port based in 45263 (including new traps etc) is booting to desktop,
+but contains many hacks and require cleanup. For example new hal initialization
+enables the timer interrupt before setting the vector, while ntos had a vector
+to unexpected interrupt -> bugcheck. It is a miracle that trunk boots, just
+depending that no timer int happens since the int enable to the vector init in
+a later phase. I'm not building hal yet, so for now I added in ntos the timer
+interrupt stub that calls KeUpdateSystemTime in a test file. I'll need some time
+to clean those hacks properly, so I decided to revert to 44678, and later update
+stuff orderly and in a proper way. Also the new stuff from Sir Richard is WIP
+and it is being a huge waste of time for me fixing it before it is finished.
+Anyway while fixing it i made vast improvements to trap, interrupt and mainly
+syscall handling, so i intend to commit it later.
+- 44678 is previous to freeldr and hal changes, it needs those from 0.3.11
+relase or builds previous to 44678 (probably it is compatible with those until
+44900 or so).
+- The change log for this submit is very long, because it contains everything
+done fom trunk 44678 to a bootable msvc build of ntoskrnl.
+An effort has been made to fix incompatibilities in proper ways, improving
+general portability of the whole system instead of adding unnecessary compiler
+dependent conditionals spreaded through the sources. Now most of these
+conditional compilation are replaced by the definitions in a very small set of
+compiler and platform dependent includes, placed at include\platf.
+Thanks to this further ports will be greatly simplified, and code is much
+cleaner. 
+
+[ntos] msvc projects
+- added msvc projects, .vcproj and .vsprops files for ntoskrnl and kddll,
+and libraries used by these.
+- different sections of ntoskrnl are built as libraries to ease separate
+testing of updates. It is just how the vcprojs are made, it does not
+change the tree structure or source files.
+- msvc projects use forced *_bld.h includes to eliminate the need of changes
+in the sources as much as possible. 
+- The root.vsprops file sets root properties for all projects. Here you have
+to set the ros and objrbe macros to match the root path for reactos source
+tree. objrbe is explained below.
+- ntoskrnl uses files are autogenerated, for now i'm using those generated
+by rosbe. Since they are autogerated, i am not commiting them, but are
+necessary to build.
+root.vsprops defines the objrbe macro as .\obj_rosbe.
+you may do either:
+a) create a .\obj_rosbe junction pointing to where you have the rosbe's object
+dir.
+b) redefine the objrbe macro to point to the rosbe's object dir. 
+c) copy the required autogenerated files there from rosbe's object dir.
+These are:
+.\include\
+.\ntoskrnl\include\
+.\ntoskrnl\ex\zw.s
+- currently the autogenerated zw.s file is not assembled correctly by the
+as rule, because $KGDT_R0_CODE is not changed to $8 by the preprocessor.
+So for now i copied it to ntoskrnl\ex\i386, edited it, and added for commit.
+I'll try to find how to change the as rule so that it behaves as when using
+gcc as preprocessor. If someone has a better idea... It is the only file
+that the as rule doesn't assemble well untouched.
+- The import libraries are searched in libobj\ros first and then in
+libobj\ms (check the root properties). I have there the rosbe built libraries
+renamed to .lib. Only hal.lib and bootvid.lib are required. But because
+ntos imports kdcom and kdcom imports ntoskrnl, intitially you need also one
+of these two libs there, although the generated ones will take precedence
+after building the first time, the result of the first link may not be
+good if the import lib used is not right, but the generated import libs
+can then be used to link correctly.
+I also tested linking with ms libs without trouble, (that's why libobj\ms
+is there), but none from ms is required.
+- A new folder ntoskrnl\kdw1 contains the kd64 debugger, the one compatible
+with ms debuggers like windbg. This is a provisional location, basically 
+a copy of kd64 which i didn't want to break during testing. It is working.
+Now it is a static library, so that the desired debugger can be selected
+at link time, but could also be built as dll to make it selectable at run
+time. I'm not motivated to to sync builds of kdbg and dbg, and until a
+decission is made about this idea of isolating the different debuggers is
+made or its integration in trunk or any other request, i'll keep it in this
+location.
+
+[tools] rules files
+Rules files for msvc added in tools/msvc, adding support for as and nasm.
+The one to support masm is copied from msvc.
+
+[global] platform abstraction includes
+- created dir include/platf with includes providing abstraction of compiler-
+specific directives and platform-specific low level basics.
+- platf.h has the abstraction for compiler-specific directives and is intended
+to be inclued in every c project or file before crt, sdk, etc. Some have psdk
+equivalents, code should use psdk definitions in those cases, while psdk,
+ddk and crt will define theirs based in platf.h instead of compiler-specific
+directives. There are separate versions of this file under the msc and gnuc
+subdirs. The gnuc one is currently untested.
+- cpu.h has low level definitions for the cpu, and cpu_c.h has the compiler
+dependent part, intended to replace common inline assempler in conditional
+blocks. Some of the funtions are equivalent to intrinsics and are just
+defines for them, while many others are inline or macro implementations for
+required functions that are not available as intrinsics and were prevously
+implemented as inline assemply spreaded along the kernel and hal sources.
+cpu_c.h under msc is complete and tested, the version under gnuc needs to be
+completed and tested.
+The defs that have equivalents as intrinsics are there for consistency and
+convenience, and keep in mind that different compilers may have different
+intrinsics, and it's better to write a cpu_c.h for a new compiler than
+having to add inline assembly everywhere...
+These are used in kernel and driver modules.
+- utl.h contains some general purpose utility macros.
+- intrin.h for each compiler contains the intrinsics definitions. For the
+intrinsics the naming of those of msvc are used, since these are what are
+used by the ms sdks. For example sdk defines InterlockXX _InterlockXX.
+inting.h declares, defines or implements these _xx.
+Currently i have there some that was causing problems, but i will complete it
+so that sdk and ddk can completely rely in _xx intrinsics being available
+or defined to define theirs, so eliminating also compiler and platform
+dependency in the sdks. After fixing a few, everything works, but these
+defs are a mess, the sdk has implementations for some available as 
+intrinsics, others are lacking or incorrect for some platform. etc.
+I will be fixing that mess by defining all those based in the compiler and
+platform specific intrin.h, so the sdks will be much cleaner and portable.
+- Many occurences of inline assembly inside conditional blocks, where
+very often the msc part was wrong, and compiler specific directives, 
+have been replaced by instances of the macros defined in these includes.
+Many in ke\i386\cpu.c but also many others spread among many files.
+
+[ntos] DbgPrint improvements
+- DbgPrint support moved to ntoskrnl/dbg.
+- DbgPrint now uses a function pointer to the actual handler, so that it can
+be changed as necessary in different phases or by the kernel debuggers.
+The initial DbgPrint handler in ser1.c is a very simple one that writes
+directly to the com port, allowing early and transparent debug logging right
+at KiSystemStartup.
+When enough of the system has been initialized, the pointer is changed to
+the right handler. The debuggers can also change the pointer to their own 
+handlers, improving debugger module isolation.
+Note that ser1 assumes that there is a com1 (@0x3f8) present. If this is not
+the case #define EARLY_DEBUG_PORT 0. If DBG is 0 it won't be used as well.
+I could add port presence detection very easily, but i don't think it necessary.
+Let me know otherwise.
+I use windbg in com2 through named pipe in vmware, and works nicely.
+Currently the initial mapping to ser1 is not changed later, i have yet to
+change the pointer when appropiate during the boot, so if the
+debugger is not activated all DbgPrint output is always handled by ser1,
+and if the debugger is activated output from kernel goes to com1 and from
+user to windbg, not a big problem anyway.
+
+[sdk] sdks fixes
+- some defs like #define NTKERNELAPI DECLSPEC_IMPORT has been made conditional as
+#ifndef NTKERNELAPI
+#define NTKERNELAPI DECLSPEC_IMPORT
+#endif
+Obviously original ms ones were not intended to build ntoskrnl, but we do so we need
+to define NTKERNELAPI as empty (or dllexport) when building ntoskrnl, for sdk compatibility.
+- FAST_MUTEX.Gate changed to FAST_MUTEX.Event, as in ms sdks, for sdk compatibility.
+- KIPCR.Tib changed to KIPCR.NtTib, as in ms sdks, for sdk compatibility.
+- KIPCR.Self changed to KIPCR.SelfPcr, as in ms sdks.
+Actually, i copied the KPCR def from ms, since it coincided but was more complete
+with an union and better documented.
+
+Changed some argument types to match ms definitions. In every case the type is
+compatible, so the change is safe. Eliminated some warnings.
+NTSYSAPI ULONG NTAPI vDbgPrintExWithPrefix(PCCH Prefix, ULONG ComponentId, ULONG Level, PCCH Format, va_list arglist);
+NTSYSAPI ULONG NTAPI vDbgPrintEx(ULONG ComponentId, ULONG Level, PCCH Format, va_list arglist);
+ULONG _CDECL DbgPrint(PCSTR Format, ...);
+NTSYSAPI ULONG _CDECL DbgPrintEx(ULONG ComponentId, ULONG Level, PCSTR Format, ...);
+NTSYSAPI ULONG _CDECL DbgPrintReturnControlC(PCCH Format, ...);
+NTSYSAPI NTSTATUS NTAPI DbgQueryDebugFilterState(ULONG ComponentId, ULONG Level);
+NTSYSAPI NTSTATUS NTAPI DbgSetDebugFilterState(ULONG ComponentId, ULONG Level, BOOLEAN State);
+
+[ntos] added lacking global
+ntoskrnl\cc\cacheman.c
+BOOLEAN CcPfEnablePrefetcher = 0;
+
+[ntos] ntoskrnl\ex\work.c: Int32x32To64 with all constant arguments is silly,
+let the compiler do it instead of invoking a runtime multiplication.
+
+[ntos] ntoskrnl\ke\dpc.c 
+FIXME: KiTimerExpiration causes a crash if global optimization (MSVC9)
+difficult to trace, it works most times, but sometimes make a called dpc routine
+crash.
+Added _OPTIMIZE_OFF_GLOBAL and _OPTIMIZE_DFT before and after the function.
+_OPTIMIZE macros are defined in platf.h, appropiately in msc version and empty 
+in the gnuc one.
+
+[ntos] ntoskrnl\ke\freeldr.c KERNEL_DESCRIPTOR_PAGE macro bug fix
+// #define KERNEL_DESCRIPTOR_PAGE(x) (((ULONG_PTR)x &~ KSEG0_BASE) >> PAGE_SHIFT)
+#define KERNEL_DESCRIPTOR_PAGE(x) ( ( (ULONG_PTR)x > KSEG0_BASE ? ((ULONG_PTR)x-KSEG0_BASE) : (ULONG_PTR)x) >> PAGE_SHIFT)
+
+[ntos] TSS needs alignment
+changed globals definition order
+_ALIGN(0x80) KTSS KiBootTss;
+_ALIGN is appropiately defined in platf.h for msc and gnuc
+
+[ntos] mm/arm3
+#line 15 in removed in all files
+These play havoc with msvc and ultracompare. I don't think
+they are necessary in any case so i removed them.
+If there is a good reason to have them, plz put them in #ifdef ARM block,
+or much better, define a macro in the platf.h for arm
+
+[ntos] ntoskrnl\ps\win32.c
+PsConvertToGuiThread calls KeSwitchKernelStack, which changes the stack and ebp
+added _OPTIMIZE_OFF_STKF and _OPTIMIZE_DFT before and after the fn, so it
+works regardless of the optimizations chosen.
+
+[rtl] GetGlobalFlags GetVersion moved to separate files so that are only
+linked if actually used.
+
+[ntos] ntoskrnl\include\internal\ke_x.h KeGetPreviousMode
+Changed to macro without outer cast.
+Don't use wrong cast or inline return type, it is unnecessary and
+can generate unnecessary rt conversion.
+
+[ntos] use section macros in platf.h instead of compiler specific
+directives defs of PLACE_IN_SECTION, INIT_FUNCTION, PAGE_LOCKED_FUNCTION,
+PAGE_UNLOCKED_FUNCTION removed and refereces changed to macros based in 
+platf.h that are suitable for both gnuc and msc, unlike the previous ones
+that couldn't be defined for msc coz it needs the function name and is used
+only in definition, not in declaration.
+
+[ntos] msc part in intrin_i.h removed
+cpu_c.h contains everything intrin_i had, and much more it didn't.
+gcc doesn't need it either if cpu_c.h is included instead, so this
+file should dissapear.
+
+[ntos] PETHREAD and PEPROCESS ambiguity
+ddk may define PETHREAD as KTHREAD * or ETHREAD * depending on
+what specific includes are used.
+To avoid the ambiguity i replaced in some places PETHREAD for ETHREAD *,
+and so removing the need of previous recast hacks.
+The ambiguity is also in ms ddk, so i don't think we should change it,
+but be aware of the ambiguity and use * instead of the pointer type to
+avoid mistakes.
+
+[ddk] #define KERNEL_STACK_SIZE
+this has been put inside an #ifndef KERNEL_STACK_SIZE block
+we may need to increase the default stack size, i did, since i
+detected occasional crashes debugging boot due to insufficient stack.
+
+[ddk] seg fields in struct _CONTEXT changed to USHORT
+and USHORT reserved added to keep the same alignment.
+ms ddk also used ULONG, but it is unforntunate since it may generate
+unneeded rt size conversions. I can't think any way the change could break
+anything so i considered the change a safe improvement.
+
+[ndk]
+include\ndk\extypes.h
+casts in EX_PUSH_LOCK_ defs removed, totally unneeded and undesirable.
+KGDT_
+include\ndk\i386\asm.h
+removed parentheses in KGDT_ defs, totally unnecessary, and wrong, since
+depeding on the assembler they can be interpreted as memory addresses instead
+of immediate values. as was failing to interpret them correctly!
+
+[rtl] ROUND_ macros
+were giving redefinition warnings, as these were implemented differently
+in different includes
+added #ifndef block, changed the implemantation, (older used multiplication
+what's unnefficient), and eliminated unneeded and undesirable casts.
+#ifndef ROUND_DOWN
+#define ROUND_DOWN(n, align) \
+	(n & ~(align-1))
+#define ROUND_UP(n, align) \
+	((n + (align-1)) & ~(align-1))
+#endif
+
+[ntos] declaration of KeInvalidateAllCaches lacked NTAPI, necessary since it's
+an export. The problem manifested only if not using stdcall as compiler default.
+
+[crt] lib\sdk\crt2
+- created lib\sdk\crt2, initially a copy of crt, to avoid breaking
+the current crt while testing changes.
+This is the one being linked in the msvc ntoskrnl.
+Many functions have been put in separate files, to avoid linkage of unused
+functions. A few of them have been improved. vsprintf was taken from
+the rtl.
+- currently contains at least all the functions needed and exported by ntoskrnl,
+some others are missing.
+
+[crt] lib\rtl2
+- created lib\rtl2, a copy of rtl, to avoid breaking current rtl while testing changes.
+- minor changes and some fix. Use diff to check changes, not big deal.
+
+
+***** 45572
+
+[ntos] msvc: path to rules files in vcprojs fixed
+
+
+***** 45591
+
+[branch] revert to 45059
+First of a sequence of commits to heal branch and sync to trunk head, prior to commits of msvc builds based in current trunk.
+45467 reverted the whole branch to 44678 to allow commits of msvc build based in 44678, problematic coz it is prior to branch creation.
+45572 will be the latest bootable msvc build based in 44678 until future commits complete the new msvc version (log will indicate that).
+
+****** 
+
+[branch] merge trunk head
+
+
+